Don’t Mistake Rate Limiting for an Outage: A Better Way to Probe API Availability
A single 429 response does not necessarily mean an API is down. This article explains how to distinguish transient rate limiting from genuine unavailability by using delayed, multi-round probes, separating access failures from retryable signals, and tracking latency as part of endpoint health. It also includes a practical Python implementation and a status classification table.
Recently, I ran an availability check on a batch of third-party model inference APIs. The same 15 endpoints were tested again six days later. At first glance, only a few endpoints had disappeared from the list. But the more important finding was that two endpoints were almost incorrectly classified as unavailable.
The incident exposed a common monitoring mistake:
Using a single probe result to decide whether an endpoint is alive or dead is highly unreliable.
1. The Trap of Single-Probe Checks
Here is what happened.
During the first scan, one endpoint returned:
429 Too Many Requests
Based on my usual practice, I would have placed it in the “down” category. It had worked normally six days earlier, with a response time of just over 2,000 ms. The day before the scan, it had also handled 15 consecutive requests without a single failure. A sudden 429 looked like the account had simply exhausted its quota.
This time, however, I waited a few minutes and tried again. Three consecutive requests all succeeded. I then ran another probe approximately every 45 seconds for four rounds, and all four rounds passed. The fastest response took about 1,600 ms.
The conclusion was clear:
The 429 was caused by a temporary rate-limiting window. The endpoint itself was not down.
A similar situation occurred with another endpoint. During an earlier check, it had returned “rate limit reached,” so I had categorized it as unavailable. On this occasion, the first request was rate-limited again. But after four additional rounds, all four requests succeeded, with response times of just over 500 ms.
Ironically, these two endpoints that were almost declared dead were the two fastest endpoints in the entire scan.
The underlying problem was that our definition of “failure” was too coarse. A 5xx response may indicate a genuine upstream outage, while a 429 may only mean that the service is temporarily busy. Treating both signals as equivalent makes monitoring data much less useful.
2. The Two Faces of 429: Free and Paid Tiers
To understand why a single probe can produce a false positive, it is important to consider what a 429 response means in different contexts.
A 429 on a paid tier often indicates that an account quota has actually been exceeded.
When an account is subject to a defined quota or plan limit, the service may continue returning 429 responses after the limit is reached. In that case, retrying shortly afterward will probably produce the same result because the rejection is based on a deterministic account-level quota.
A 429 on a free tier often means that the service is temporarily busy.
Free capacity is commonly shared among multiple callers. When demand spikes, the server may use 429 responses to temporarily reduce load. Once the current traffic peak passes, the same endpoint may respond normally again. This is a transient, probabilistic condition rather than a hard account-level quota failure.
This distinction is critical. Treating a free-tier 429 as a permanent failure can incorrectly eliminate healthy endpoints. The method for distinguishing the two is simple but effective:
Wait several dozen seconds and retry multiple times.
3. A Multi-Round Probing Strategy
I eventually added retry logic for every endpoint that returned a 429 or another rate-limit signal:
- Do not make a final determination after the first rate-limit response.
- Retry once approximately every 45 seconds, for a total of four rounds.
- Use the number of successful rounds divided by the total number of rounds as the decision signal, rather than relying only on the first response.
The results fell into several clear categories:
- First round returns 429, followed by four successful rounds → healthy, indicating a transient rate-limit window.
- First round returns a rate-limit signal, followed by four successful rounds → healthy.
- First round returns 429, followed by four failed rounds → unavailable.
- All retry rounds return a payment-wall or subscription-level message → insufficient access, and therefore unavailable for the intended use.
If only one probe had been used, the first two categories would have been incorrectly classified as outages. These were also the endpoints that proved to be the most valuable because of their fast response times.
4. Implementation
The following logic can be adapted directly into an endpoint inspection script. The key idea is to retry after a rate-limit signal and use the success rate to determine the endpoint state:
import timeimport requestsdef probe_endpoint(url, headers, payload,max_rounds=4, interval=45, timeout=30):"""Run a multi-round availability probe for a single endpoint.Retry only on transient signals such as 429 or rate limiting.Stop early for hard access failures.Returns: (status, successes, latencies_ms)status: 'ok' | 'unavailable' | 'no_access'"""successes = 0latencies = []for r in range(max_rounds):if r > 0:time.sleep(interval) # Allow the rate-limit window to recovertry:t0 = time.time()resp = requests.post(url,headers=headers,json=payload,timeout=timeout)latency = int((time.time() - t0) * 1000)except requests.RequestException:# Network-level failure: this round fails, but later rounds continuecontinue# Access-related rejection: retrying is unlikely to helpif resp.status_code in (401, 402, 403):return "no_access", successes, latencies# Rate limiting is treated as a transient signalif resp.status_code == 429 or _is_rate_limited(resp):continueif resp.ok:successes += 1latencies.append(latency)# Classify based on the success rate rather than the first responseif successes == max_rounds:status = "ok" # All rounds passedelif successes > 0:status = "ok" # Partially successful: available but unstableelse:status = "unavailable" # No round succeededreturn status, successes, latencies
The important point is to process different signals differently:
- Access-related rejections, such as subscription restrictions or payment walls, should converge immediately because retrying does not change the access level.
- Rate-limit signals should trigger delayed retries.
- Only signals that may be transient should go through multi-round validation.
This separation helps distinguish “temporarily busy” from “genuinely unavailable.”
5. Status Classification and Signal Semantics
A decision table makes the monitoring logic easier to understand:
| Initial signal | Retry result | Actual status | Explanation |
|---|---|---|---|
| 429 / rate limited | 4/4 successful | Healthy | Temporary rate-limit window, not an outage |
| 429 / rate limited | Partially successful | Healthy, but unstable | Intermittent behavior under load |
| 429 / rate limited | 0/4 successful | Unavailable | Genuine service failure or persistent inability to respond |
| Access or payment message | 0/4 | Unavailable | Insufficient access; retrying is not useful |
One point deserves special emphasis:
A free-tier 429 and a paid-tier 429 do not necessarily mean the same thing.
The former often means “the service is busy right now and may recover shortly.” The latter often means “the account has exceeded its quota.” Applying a universal rule such as “mark the endpoint as dead whenever it returns 429” will inevitably produce false classifications.
The better approach is to treat 429 as a signal that requires verification, not as a final state.
6. Additional Practical Recommendations
Separate transient signals from hard failures.
Network timeouts, 429 responses, and 5xx errors may be transient and therefore retryable. Access denials, invalid parameters, and nonexistent models are deterministic failures for which retries are usually meaningless. Monitoring logic should take different branches for these categories.Retry before declaring an endpoint unavailable.
A single probe only captures the endpoint’s condition at one moment. Use several probes spaced by dozens of seconds to filter out temporary fluctuations before triggering an alert or removing the endpoint from the available pool.Track response latency as a health indicator.
Even when requests continue to succeed, latency rising from 500 ms to 5 seconds may indicate increasing load or an approaching failure. Latency should be monitored alongside success and failure rates.Track upstream errors separately.
Some endpoints fail because the upstream service returns an error rather than because of rate limiting. In some cases, different models in the same family may fail at different times. Recording these failures separately helps determine whether the issue affects one endpoint or an entire service line.Watch for a sustained reduction in available endpoints.
The failure of one endpoint may be incidental. But if the total number of available endpoints keeps declining, the overall supply is tightening. That is a sign to reassess the resilience of any architecture that depends on this pool of APIs.
Conclusion
The most important lesson from this availability check is that the key question is not “What happened during one probe?” but rather:
What pattern emerges across multiple probes?
A single 429 is a signal that requires verification, not a final verdict. By incorporating delayed retries, multi-round validation, signal classification, and latency tracking into monitoring scripts, teams can avoid incorrectly removing healthy endpoints while still detecting genuine failures.
That makes availability data more trustworthy—and operational decisions based on it much safer.
