AI Coding Assistant Selection and Multi-Window Quota Rate Limiting

    This article presents a practical framework for evaluating AI coding assistants, selecting LLMs across latency, accuracy, compliance, and cost, and enforcing three-tier API quotas with Redis and Lua. It also covers HTTP 429 handling, graceful degradation, gateway token buckets, and implementation details that prevent counting errors under concurrency.

    AI coding assistants have become part of the daily toolchain for many engineering teams. When these tools move into production, however, teams often run into three problems: they are unsure how to select a model, their quota numbers contradict one another, and their rate-limiting code counts requests incorrectly under high concurrency. This article addresses these three areas and provides reusable criteria and code.

    How Should Code Capability Be Evaluated?

    A model’s coding capability should be tested with real internal tasks; public leaderboards are only a reference. One approach is to sample 50 requirements merged in the past three months, ask the model to generate code, and measure the percentage that passes unit tests. For completion scenarios, also measure acceptance rate: the percentage of completion suggestions developers accept by pressing Tab.

    Single-file context is usually not enough. A common approach to project-level understanding is to build a Retrieval-Augmented Generation (RAG) index over the codebase and retrieve relevant snippets by symbol definitions and call relationships. Three points matter in implementation. First, the index must respect repository permissions; code that a developer is not authorized to access must not be retrieved into the prompt. Second, retrieval itself adds latency, which should be measured separately. Third, context windows are limited, so there must be a truncation strategy for long content, prioritizing function signatures and callers.

    Models can detect issues such as null pointers, unreleased resources, and concurrency races, but they produce both false positives and false negatives. They are suitable as a first screening step before review, not as a replacement for static analysis and security scanning. Code merges should still be confirmed by humans.

    What Four Dimensions Matter in LLM Selection?

    1. Inference latency. Completion scenarios are highly sensitive to latency. A common “500 ms” figure is a team target, not a universal standard. When measuring, define the methodology clearly: the time from an IDE request to rendering the completion result, using P95; whether the test network is the office network or a cross-region dedicated line; and approximately how many tokens are in the prompt. Complex generation can accept several seconds, but first-token time and total time should be measured separately.

    2. Language coverage and few-shot capability. List every language in the technology stack, including internal DSLs. For less common languages, run few-shot tests with a dozen or so examples and observe the pass rate. Looking only at performance on mainstream languages will overestimate the model’s effectiveness in enterprise scenarios.

    3. Data compliance. Confirm three things: whether the model service supports private deployment; for cloud calls, whether the provider commits not to use data for training and how long data is retained; and whether the IDE plugin uploads the entire repository. These terms should be included in the procurement review form, not judged only from marketing pages.

    4. Context capability and cost. The larger the context window, the more expensive each call. Estimate the cost per thousand requests based on real call records, then compare it alongside context length and accuracy.

    Below is a scoring example; weights can be adjusted to the team’s actual situation.

    Evaluation Item Weight Metric Example Notes
    Inference latency 30% P95 completion latency Measured in the same network environment
    Code accuracy 30% Internal task pass rate Determined by test cases
    Data compliance 25% Private deployment, retention period, training terms Eliminate immediately if not met
    Unit cost 15% Cost per thousand requests Estimated from real calls

    How Should API Quotas Be Designed?

    A common approach is to stack three windows: a 5-hour window to limit short-term peaks, a weekly window to limit medium-term consumption, and a monthly window to constrain the total budget. The three windows are not substitutes for one another; a request is rejected if any window is exceeded.

    The following is an example set of values.

    User Tier 5-Hour Limit Weekly Limit Monthly Limit
    Basic 300 2000 8000
    Heavy (Pro) 1500 10000 40000

    These numbers must satisfy three consistency rules, and they should be recalculated against actual cost and capacity before launch.

    First, the 5-hour limit must not be higher than the weekly limit; otherwise, a single window could consume the entire weekly allowance. Second, the weekly limit multiplied by about 4.3 weeks (the average number of weeks in a month) should not be lower than the monthly limit; otherwise, the monthly limit is effectively meaningless. For the Basic tier, 2000 multiplied by 4.3 is about 8600, which is greater than 8000, so the monthly limit only takes effect during high-intensity months. Third, under typical usage, the weekly limit is the main constraint. A week has about 168 hours and can span more than 33 five-hour windows, so the 5-hour limit is mainly used to suppress burst traffic.

    A 5-hour window can also cause users to exhaust their weekly quota early. If a Basic user concentrates usage on Monday, about 7 full windows will use up 2000 requests, and the user must then wait until the next Monday at 00:00 UTC for the quota to reset. Teams should tell users this directly rather than leaving them to guess.

    What Happens When Quotas Are Exhausted?

    When a quota is exceeded, return HTTP 429 and include the remaining seconds of the exceeded window in the Retry-After response header.

    Rate limiting should distinguish two categories. Concurrency limits apply to the number of in-flight requests from the same user at the same time; short backoff retries are acceptable, such as exponential backoff with random jitter. Quota overages should not be retried; the client should directly show the user the remaining quota and reset time.

    After quota exhaustion, service can be degraded. For example, keep line-level completion, disable long-context generation, or switch to a cheaper lightweight model, and display the reason for degradation in the interface.

    The message should contain three things: which window was exceeded, when it resets, and how to request a temporary quota increase.

    How Should Rate Limiting Be Implemented?

    Implement rate limiting in two layers.

    The first layer is a gateway token bucket, responsible for traffic shaping. The token bucket has two parameters: bucket capacity determines the burst of requests allowed, and refill rate determines the long-term average rate. It controls the pace per second or per minute and does not count totals, so it cannot replace quota counting.

    The second layer is a Redis counter, responsible for quotas. Each window corresponds to a counter, and the key includes the window identifier, such as the 5-hour bucket number, the Monday date, or the year and month. When the window changes, a new key is used automatically, and old keys expire via TTL. One key point: TTL is set only when the key is first created and is not refreshed on subsequent requests. If TTL were refreshed on every request, the window would slide from the last request, and weekly and monthly quotas would be released later than intended.

    All three windows must be checked and counted together. If the 5-hour counter is incremented first and the weekly quota is then found to be exceeded, the earlier increment cannot be rolled back, and counts will become inconsistent after multiple requests. Therefore, use a Lua script to perform the step “check all windows, increment only if all pass” atomically inside Redis.

    1. import math
    2. import time
    3. from datetime import datetime, timedelta, timezone
    4. # Atomic check: reject the whole request if any window is exceeded; do not increment any counter
    5. RATE_LIMIT_LUA = """
    6. local n = #KEYS
    7. for i = 1, n do
    8. local current = tonumber(redis.call('GET', KEYS[i]) or '0')
    9. if current + 1 > tonumber(ARGV[i]) then
    10. return {0, i}
    11. end
    12. end
    13. for i = 1, n do
    14. local current = redis.call('INCR', KEYS[i])
    15. if current == 1 then
    16. redis.call('EXPIRE', KEYS[i], ARGV[n + i])
    17. end
    18. end
    19. return {1, 0}
    20. """
    21. LIMITS = {
    22. "basic": {"5h": 300, "1w": 2000, "1m": 8000},
    23. "pro": {"5h": 1500, "1w": 10000, "1m": 40000},
    24. }
    25. def build_windows(user_id, tier, now):
    26. """Return a list of (window name, key, limit, remaining seconds); all windows are split by UTC."""
    27. dt = datetime.fromtimestamp(now, tz=timezone.utc)
    28. limits = LIMITS[tier]
    29. windows = []
    30. # 5-hour window: split by fixed bucket number
    31. size_5h = 5 * 3600
    32. bucket_5h = int(now // size_5h)
    33. end_5h = (bucket_5h + 1) * size_5h
    34. windows.append(("5h", f"rl:{{{user_id}}}:5h:{bucket_5h}",
    35. limits["5h"], math.ceil(end_5h - now)))
    36. # Weekly window: starts Monday 00:00 UTC
    37. day_start = dt.replace(hour=0, minute=0, second=0, microsecond=0)
    38. monday = day_start - timedelta(days=dt.weekday())
    39. next_monday = monday + timedelta(days=7)
    40. windows.append(("1w", f"rl:{{{user_id}}}:1w:{monday:%Y%m%d}",
    41. limits["1w"], math.ceil((next_monday - dt).total_seconds())))
    42. # Monthly window: starts on the 1st at 00:00 UTC
    43. month_start = day_start.replace(day=1)
    44. if month_start.month == 12:
    45. next_month = month_start.replace(year=month_start.year + 1, month=1)
    46. else:
    47. next_month = month_start.replace(month=month_start.month + 1)
    48. windows.append(("1m", f"rl:{{{user_id}}}:1m:{month_start:%Y%m}",
    49. limits["1m"], math.ceil((next_month - dt).total_seconds())))
    50. return windows
    51. def check_rate_limit(user_id, tier, redis_client, now=None):
    52. now = time.time() if now is None else now
    53. windows = build_windows(user_id, tier, now)
    54. script = redis_client.register_script(RATE_LIMIT_LUA)
    55. keys = [key for _, key, _, _ in windows]
    56. args = [limit for _, _, limit, _ in windows] + [ttl for _, _, _, ttl in windows]
    57. allowed, index = script(keys=keys, args=args)
    58. if allowed == 1:
    59. return True, None
    60. # Return the name of the exceeded window; the caller uses it to calculate Retry-After
    61. return False, windows[index - 1][0]

    In the script, rejected requests are not counted against any window. In Redis Cluster mode, the three keys for the same user must land in the same slot, so the example wraps the user ID in braces as a hash tag.

    If an enterprise already uses a cloud provider’s API gateway, it should first evaluate whether the gateway can configure rate-limiting rules for the windows described above and confirm counting accuracy and latency. Baidu AI Cloud’s API Gateway provides multi-dimensional rate-limiting configuration and is suitable for teams already deployed on that cloud. For self-hosted options, teams can use rate-limiting plugins from open-source gateways such as Nginx, Kong, or Envoy, at the cost of maintaining counting consistency and monitoring themselves. Both approaches should be validated with load tests rather than judged only by feature lists.

    When inference services are deployed on a container platform, autoscaling should be based on request queue length and GPU utilization, not on the 5-hour window. The window is only a billing and rate-limiting unit and does not reflect compute load.

    Conclusion

    For model selection, measure latency, accuracy, language coverage, and compliance terms with real internal tasks, then score them by weight. Quota design should ensure that the three windows follow consistent rules and define in advance what happens when quotas are exhausted. For rate limiting, use a gateway token bucket for shaping and Redis Lua scripts for atomic quota counting. Following these three steps allows teams to keep costs under control while giving developers a clear view of how many requests they have left.