Rate Limiting and Retry Strategies for Data APIs in Agents

A single agent task doesn't make one API call. It makes a reasoning turn, then a tool call, then a retrieval step, then a summarization pass, then another reasoning turn to decide what to do with all of it. Multiply tasks per minute by average steps per task, then add a retry multiplier for when something fails, and one user action turns into dozens or hundreds of provider requests. Traditional rate limiting was built for humans clicking buttons, and there are natural pauses between their requests. Agents don't pause. Ten agents can each stay politely within its own limit and still blow through an org-level quota that no single agent came anywhere near. That's the structural break: the old assumption was that request volume tracked human behavior, and agentic traffic has no human behavior to track.

Three distinct throttle signals agents encounter demand different responses

Treating every 429 the same way is the first architectural mistake agents make in production, because not every 429 means the same thing. Two signals in particular require opposite handling. An HTTP 429 means the client went over its own quota, a client-side problem that calls for backing off and fixing the rate limiter. A 5xx response means something different: the server itself is overloaded, and retrying into that same overloaded dependency just adds more load to it. The correct response to a 5xx is to trip a circuit breaker toward a fallback, not to back off and try again.

Third-party SaaS tools that agents call introduce a layer of signal chaos on top of that. Agent tool code either has to speak every one of these dialects natively or delegate the translation to a normalization layer sitting in front of the tools. OpenAI's guide is specific that these require a person to take action and should not be retried automatically.

The practical consequence is that before any backoff or retry code gets written, an agent needs a signal classifier sitting ahead of it. That classifier maps every provider-specific response into one of three buckets: back off and retry, fail over to an alternative, or escalate without retrying.

Exponential backoff with jitter as the baseline retry mechanism

Exponential backoff is the right starting point for handling transient 429s, and it works by waiting a base delay after a failed request, then doubling that delay on each subsequent failure up to some ceiling. A simple version looks like this:

delay = min(base_delay * (2 ** attempt), max_delay)
delay = delay * (0.5 + random())  # full jitter
sleep(delay)

The doubling gives an overloaded service room to recover before it gets hit again. The jitter matters just as much as the doubling. OpenAI's guide tells you to add a small random delay on top of whatever Retry-After value comes back, so multiple clients don't retry at the same instant.

Backoff only fixes retries for a single agent working alone. Closing that gap takes three more layers: retry budgets to cap total attempts, circuit breakers to stop retrying into a dependency that's actually down, and concurrency semaphores to limit how many in-flight requests a pipeline can hold regardless of backoff timing.

Retry budgets: capping total retry volume per task regardless of per-request success

A retry budget is a different control from a rate limit. A rate limit caps request volume over time. A retry budget caps the number of retry attempts a single task is allowed, independent of whether each individual request technically complied with the provider's rules. That distinction matters because an agent can exhaust its entire retry budget without ever tripping a rate limit, simply by spreading its failures out across time inside the provider's quota headroom.

Implementing this is a matter of attaching a counter to the task record itself, not to the individual request, and incrementing it on every retry across every step of that task. Once the counter hits the ceiling, the task stops. It either gets checkpointed for later resumption or routed to a human queue. Budget size should match the task. Short, single-step tasks do well with a small ceiling, three to five total attempts, keeping failure fast and visible. Multi-step pipelines need a budget allocated per stage, not globally per task, so a flaky early step can't eat the whole budget before the critical later steps even get a chance to run.

Non-idempotent writes deserve their own handling, and this is one of the most overlooked details in practice. The fix is either to attach a unique idempotency key that makes the write safe to retry, or to allow exactly one attempt and send any failure straight to a review queue.

Retry budgets can also be tiered. Each priority tier got its own non-overlapping time band for retries, so critical tasks retried first in an earlier window before background tasks were even allowed to start retrying. That way, low-priority retries couldn't eat the quota that high-priority tasks needed.

Token buckets and concurrency semaphores as the proactive layer before requests are sent

Everything described so far reacts to a failure that's already happened. Token buckets and concurrency semaphores sit in front of that, controlling what gets sent. A token bucket calibrated to roughly 80 percent of a provider's stated limit, rather than 100 percent, leaves headroom that absorbs the retry traffic generated by backoff and retry logic without pushing total throughput over the line. At 100 percent of the limit, any single retry attempt immediately triggers another 429, and the system never gets ahead of itself.

If twenty agent workers each run their own token bucket calibrated to a "polite" individual rate, their combined traffic can still blow past the org-level quota that none of them approached on their own, because rate limits enforced separately by each agent never add up to the shared quota they all draw from. On top of that, a priority queue decides what happens when the semaphore is full and the bucket is empty: interactive, user-facing tasks need to jump ahead of background batch jobs rather than wait in line behind them, which keeps user-visible latency low without changing total throughput.

Circuit breakers: stopping agents from retrying into a dependency that has failed structurally

A dependency that's structurally down doesn't recover because an agent keeps retrying against it. Those retries just add load to something already broken, and every pipeline step waiting on that dependency sits stalled behind it. Circuit breakers solve this through three states. Closed is the normal operating state: requests flow through, and failures get counted against a threshold. Failure resets the open timer and the cycle repeats.

For agentic pipelines, that open timeout should line up with the provider's own rate limit window. If you pair a 60-second timeout with a per-minute limit, the circuit stays open for exactly one full rate limit cycle before it probes for recovery.

The distinction between 429 and 5xx responses carries through here too. The stakes for agents are higher than for single-step systems. A failure at step three of a seven-step pipeline wastes every bit of quota and credit already spent on steps one and two. Tripping the circuit at the dependency level, not at the task level, stops the pipeline from even starting downstream work once an upstream dependency is known to be down.

Adaptive throttling: adjusting concurrency in real time as quota headroom changes

Static token buckets and fixed semaphore limits set a safe ceiling, but they leave throughput on the table whenever more quota happens to be available. Adaptive throttling reads the quota signal a provider already sends back and adjusts concurrency against it in real time. The mechanism is to read the ratelimit-remaining or x-ratelimit-remaining header on every single response and feed it into a controller that raises or lowers the semaphore limit based on how much headroom is left. High remaining quota permits more concurrency. Low remaining quota tightens the semaphore before a 429 ever arrives.

AIMD, additive increase and multiplicative decrease, is the standard algorithm for this: concurrency goes up by one unit on a success, and gets cut in half the moment a throttle signal shows up. It's the same approach TCP has used for congestion control for decades, applied here to API quota. None of that required extra instrumentation. The signal was already present in every API response.

Dollar caps and kill switches as the enforcement layer that rate limiting alone cannot provide

Staying within every rate limit a provider enforces still leaves room for catastrophic spend. Throughput controls and cost controls have to be built as separate layers, because a provider's per-minute request limit caps volume, not dollars. An agent making requests just under that limit, each one carrying a large context window, can burn through budget at a pace the rate limiter has no way to see.

You get the best cost control by stacking layers on top of each other. An org-level kill switch sits above both: a configuration flag checked at the gateway on every single request, flippable in under a minute, that halts all non-critical agent traffic once aggregate spend crosses a multiple of expected daily spend. That threshold needs to be set before launch and tested as a real code path that actually executes, not a dashboard someone glances at occasionally.

Cost tracking has to happen incrementally, on every response, not in a single check at the end of a task. Neither changes throughput behavior, but both lower the effective cost of each task.

Enforcement location: application layer versus gateway or mesh layer

A fair objection to everything described so far is that retry and rate logic inside application code is the wrong place for it. Both layers have a real role, just different scopes. Structural limits, requests per second, concurrent connections, total spend, belong at the gateway, because they apply uniformly across every agent regardless of what the application code does, they're harder to bypass by accident, and they save every team from reimplementing the same controls. For agent pipelines specifically, a mesh or proxy layer can provide per-agent isolation and circuit breaking that frameworks like LangChain and CrewAI don't ship with out of the box.

Behavioral limits belong at the application layer instead. Retry budgets, step caps, task-level dollar caps, escalation to a human queue, none of these can be expressed as gateway rules, because they depend on task-level state the gateway simply doesn't have visibility into. The agent enforces behavioral limits, and it keeps waste down inside those bounds. Neither layer covers both failure modes on its own.

Checkpointing and pipeline design to prevent retry work from replaying completed steps

A failure at step N in a multi-step pipeline shouldn't force a replay of steps one through N-1. If you skip checkpointing, every retry of a failed step reissues every API call that came before it, wasting quota on work that already succeeded, and it can hit provider limits faster on the retry than the original run ever did.

Checkpoint design means persisting intermediate results at each stage boundary: retrieved documents, extracted schemas, partially assembled outputs, all written to durable storage as the pipeline moves. On retry, the pipeline resumes from the last successful checkpoint. For scraping and crawling pipelines in particular, splitting the work into two distinct stages makes this concrete. A first pass discovers and stores URLs through a map or crawl operation. A second pass then runs extraction against a filtered subset of those URLs. If extraction fails partway through, the URL list is already safely stored, so the pipeline resumes extraction without re-crawling anything. The same split that enables checkpointing also lets you batch the extraction calls with rate limits in mind, since you can throttle extraction independently of discovery.

Freshness as a constraint that intersects with rate limiting in RAG and live-web retrieval pipelines

For agents grounded in RAG or live web data, every rate limiting decision is also a freshness decision. How aggressively a system retrieves, and how often, sets the upper bound on how current its answers can possibly be. Pure embedding-based RAG only retrieves from documents already indexed, and for anything that changes faster than the indexing cadence, security vulnerabilities, regulatory updates, pricing, software library versions, the system hands back answers that were already stale the moment they were generated. That's not a retrieval failure. Retrieval worked exactly as designed against an index that was out of date.

Live-web retrieval keeps answers current, and every retrieval call draws down the same quota and budget as an LLM call, adding its own per-request cost and latency. Every pattern covered in this piece, token buckets, retry budgets, circuit breakers, adaptive throttling, applies to the retrieval layer exactly as it applies to the LLM layer. LinkedIn's agentic customer support system exposed RAG as a tool the agent calls within its own reasoning loop, rather than running retrieval on a fixed schedule, letting the model decide when to retrieve and how to query. That turns retrieval from a predictable batch process into a spiky, task-driven one, which is exactly the traffic pattern the client-side controls described throughout this piece are built to handle.

The earlier distinction between client-side and server-side throttle signals applies to these data APIs as well. One scraping or extraction service might return a clean Retry-After header, another might embed a retry timestamp inside the response body, and a third might bury a structured error inside a 200 response. That normalization layer gets simpler the fewer providers an agent has to reconcile. Context.dev's architecture is built around exactly this workload: a single REST API that turns URLs into LLM-ready Markdown, crawls full sites, and extracts structured data against a JSON schema, so the retrieval layer returns one consistent, predictable response format. Agent-side rate limiting and retry logic can reason against that one format, so you don't need a separate normalization path for every provider, and the same token bucket, semaphore, and circuit breaker configuration covers both web retrieval and LLM calls. Teams building grounded agents need to treat the retrieval layer as a first-class rate-limiting concern from the start, with the same discipline applied to the LLM calls sitting next to it.