Handling LLM API Rate Limits (HTTP 429): Troubleshooting and Fault-Tolerance Practices

    This article explains what HTTP 429 means when calling large language model (LLM) APIs and why LLM endpoints hit rate limits so easily. It shows how to distinguish quota-based limits from capacity overload, how to implement exponential backoff with jitter (and respect Retry-After), how to control context size and token budgets to reduce triggers at the source, and how to add multi-model failover, circuit breaking, and graceful degradation. It also covers observability and capacity planning so you can build resilient client SDKs and middleware that stay stable when traffic spikes.

    1. What HTTP 429 Is Actually Telling You

    In LLM API call logs, developers often see errors like RateLimitError [HTTP 429], together with a business error code (for example, code: 1305) and a message such as “The model is currently receiving too many requests, please try again later.”

    HTTP 429 (Too Many Requests) is a standard HTTP status code. Its meaning is: “Within a given time window, the number of requests the client has sent exceeds the limit the server allows.” It is not a simple network error, but an explicit backpressure signal: the server is saying “I’m too busy right now, please slow down.”

    It helps to distinguish two different kinds of causes behind a 429:

    • Quota-based rate limiting: the request count per unit of time (QPS/RPM) or the token throughput (TPM) has reached the limit for the account or the model. These limits are usually predictable and tied to your account tier.
    • Capacity-based overload: the server temporarily rejects some requests because its overall load is too high. This is a form of elastic protection on the platform side. The “too many requests” wording in the logs is closer to this category.

    For developers, the two share a common client-side approach—both require “slow down + retry”—but each has its own focus, and the strategies below address them separately.

    2. Why LLM APIs Hit Rate Limits So Easily

    Compared with traditional REST APIs, three characteristics make LLM calls especially prone to 429:

    1. Each request is resource-heavy. A single inference can consume thousands or even tens of thousands of tokens, so the cost of serving it is high and the platform is more sensitive to throughput protection.
    2. Long context amplifies the pressure. A log line like Context: 64 msgs, ~43,303 tokens shows a request carrying more than forty thousand tokens of context. The longer the context, the more compute and GPU memory a single request occupies, and the more likely it is to be rate-limited first during peak periods.
    3. Traffic spikes driven by word of mouth. When a model suddenly becomes popular, concurrent requests surge in a short time. Even if the platform scales capacity, there is a lag window, and during it 429 becomes a routine protection mechanism.

    Understanding this makes the key point clear: hitting a 429 is not a “failure”—it is the system protecting itself as designed. The developer’s goal is not to eliminate 429, but to let the client handle it gracefully.

    3. Backoff and Retry: Don’t Be the Client That Pours Fuel on the Fire

    The original log shows a typical naive retry: wait 2.2s after the first failure, then 4.8s after the second. This “fixed interval or simple incremental” approach is actually harmful when the server is overloaded—if all clients retry at the same pace, they form a “retry storm” that further overwhelms an already overloaded service.

    The correct approach is exponential backoff with jitter:

    1. import random
    2. import time
    3. def call_with_backoff(request_fn, max_retries=5, base=1.0, cap=32.0):
    4. """
    5. Retry on retryable errors (such as 429) using exponential backoff with jitter.
    6. base: initial backoff base (seconds)
    7. cap: per-attempt backoff ceiling (seconds), to avoid unbounded backoff growth
    8. """
    9. for attempt in range(max_retries):
    10. try:
    11. return request_fn()
    12. except RateLimitError as e: # catch only retryable rate-limit errors
    13. if attempt == max_retries - 1:
    14. raise
    15. # Prefer the wait time suggested by the server (Retry-After header), if present
    16. hinted = getattr(e, "retry_after", None)
    17. if hinted is not None:
    18. delay = hinted
    19. else:
    20. # Exponential backoff: base * 2^attempt, capped
    21. delay = min(cap, base * (2 ** attempt))
    22. # Add random jitter to spread out concurrent clients' retry times
    23. delay = delay * (0.5 + random.random())
    24. time.sleep(delay)
    25. raise RuntimeError("Retry attempts exhausted")

    A few key design points:

    • Exponential growth instead of a fixed interval: doubling the wait each time gives the server a real recovery window.
    • Add jitter: scattering the retry moments avoids many clients retrying at the same instant.
    • Respect Retry-After first: many well-behaved APIs return a suggested wait time in the 429 response; use it when present.
    • Distinguish retryable from non-retryable errors: 429, 502, 503, and 504 are usually retryable, while 400 (bad parameters) and 401 (auth failure) are pointless to retry and should be raised immediately.

    4. Context and Token Budget: Reduce the Trigger Rate at the Source

    The failed request in the log carried about 43,303 tokens of context. Since long context is one of the triggers of rate limiting, managing context size is a way to relieve pressure at the source.

    • Context compression and trimming: keep only the history messages strongly related to the current task, and summarize older dialogue rather than concatenating everything in full. For multi-turn conversations, a “sliding window + summary” strategy can keep the total under control.
    • Tiered context: manage system prompts, long-term memory, recent dialogue, and current input in separate layers, prioritizing high-value information and dropping low-value intermediate turns when necessary.
    • Proactive token estimation: before sending a request, use a tokenizer to estimate the tokens this call will consume, and combine that with your account’s per-minute token limit (TPM) to throttle on the client side—slow down before hitting the wall.
    • Batching: combine small requests that can be merged into batch submissions, reducing the impact of request count on RPM.

    Example: a simple client-side rate limiter

    1. class TokenBucket:
    2. """A simple token-bucket limiter used to proactively control send rate on the client side."""
    3. def __init__(self, rate_per_min, capacity):
    4. self.capacity = capacity
    5. self.tokens = capacity
    6. self.rate = rate_per_min / 60.0 # tokens replenished per second
    7. self.last = time.time()
    8. def acquire(self, need=1):
    9. while True:
    10. now = time.time()
    11. self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
    12. self.last = now
    13. if self.tokens >= need:
    14. self.tokens -= need
    15. return
    16. time.sleep((need - self.tokens) / self.rate)

    5. Multi-Path Fault Tolerance and Graceful Degradation

    When a single model stays overloaded, retry alone is not enough—you need higher-level fault-tolerance design:

    1. Multi-model / multi-route fallback: where the business allows, configure backup models or backup endpoints for the primary model. When the primary model returns 429 and retries are ineffective, switch to the backup route to keep the business running.
    2. Circuit breaker: when a link’s failure rate is too high over a short period, temporarily “break” that link, fail fast, and go to the degradation path, avoiding endless ineffective retries that drag down the whole system.
    3. Graceful degradation response: for non-critical scenarios, return a cached result, a simplified result, or a friendly message (such as “Peak hours now; you’ve been switched to lightweight mode”) instead of throwing the error straight at the user.
    4. Request queuing and prioritization: separate critical from non-critical requests, and prioritize the quota for critical paths during peak periods.
    1. def robust_call(primary, fallback=None, retries=3):
    2. try:
    3. return call_with_backoff(primary, max_retries=retries)
    4. except RateLimitError:
    5. if fallback is not None:
    6. # Primary path is rate-limited and retries failed; switch to the fallback path
    7. return fallback()
    8. raise

    6. Observability and Capacity Planning

    The final piece of 429 governance is visibility. It’s recommended to instrument the call chain thoroughly:

    • Monitoring metrics: request success rate, 429 share, average retry count, P95/P99 latency, token consumption trends, and the error distribution across models and routes.
    • Alerting integration: when the 429 share or overload errors stay above a threshold, notify on-call staff through monitoring alerts and trigger scaling or route adjustments.
    • Structured logging: record error codes, latency, context size, retry counts, and other fields in a structured way, making post-incident review and capacity forecasting easier.
    • Capacity planning: forecast peak demand from historical token-consumption curves, confirm quotas with the platform in advance, or use general cloud capabilities such as container platforms, message queues, object storage, and logging services to build asynchronous, peak-shaving pipelines—turning synchronous real-time calls into buffered queue consumption so pressure is smoothed during peak periods.

    7. Summary

    HTTP 429 is not an anomaly but a normal protection mechanism of LLM services under heavy traffic. Building a highly available call chain comes down to four things: retry correctly with exponential backoff plus jitter, relieve pressure at the source with token-budget management, ensure business continuity with multi-path fault tolerance and degradation, and make problems visible and controllable with observability. Codifying these capabilities into a standard calling SDK or middleware layer lets your application stay stable and graceful even when it faces “too many requests”—and that is the key step from “it runs” to “it holds up.”