Evaluating LLM Coding Ability: Why Quotas and Efficiency Are Inescapable

    A practical guide to evaluating LLM coding assistants beyond "who is stronger," covering how to build a verifiable evaluation set, model tiered quota and rate limits as a computable throughput problem, use master-slave model collaboration and task routing to cut per-task cost, and design resumable long-running tasks with safe unattended execution. Readers will find comparison tables, cost formulas, and routing/scheduling pseudocode they can adapt to their own workflow.

    1. Introduction: New Constraints From Capability Upgrades

    Coding assistants are spreading fast. Starting from a single requirement description, they can automatically read a repository, locate defects, modify code, and get tests passing. As high-intensity reasoning modes are applied to coding, developers are finding the bottleneck shifting: what shapes the experience is no longer just “can the model get it right,” but “can it finish before the quota runs out,” and “after being rate-limited mid-task, can it resume seamlessly.”

    A community discussion about a new-generation coding assistant is representative. One heavy user’s experience breaks down into three episodes:

    • Quota and cycle mismatch: The user handed an optimization goal to the assistant to run unattended before bed, then found the next morning that the task had hit a “5-hour” rolling limit. After resuming, it kept going and later that day hit the weekly limit. Only after careful accounting did they realize that a single weekly quota cycle can hold only about three 5-hour rolling limits, so sustained throughput was far below expectations.
    • High-tier quota wasted on “light work”: To conserve quota, the user ran 4 coding sessions at once—3 on the high-intensity tier, 1 on the highest tier—and configured a cheaper fast model as a sub-agent. The project-level rules file explicitly instructs the main model to delegate defect-finding, code review, and small feature work to sub-agents. Even so, in just over two hours the 4 sessions consumed 35% of the 5-hour quota and 24% of the weekly quota. The author joked, “If you think this setup is reasonable, I have nothing to say.”
    • Open-ended research claim left unverified: A view in the discussion held that because a certain open mathematical problem has no proof given in the docs of some public open-source project, it was not “the model copying and wrapping up,” but a genuine step forward. This reminds us that to evaluate a model’s “open-ended research ability,” you must first verify the known answer to the problem.

    These episodes map to three engineering problems:

    1. Evaluation: How to judge a model’s true ability on open-ended, non-standard tasks.
    2. Scheduling: How to arrange long-running tasks so they fit tiered limits.
    3. Efficiency: How to let models of different capabilities do what they do best, lowering per-task cost.

    This article unifies these three problems into a closed loop of evaluate—schedule—optimize—safeguard, giving methods, examples, and engineering practices for each.

    2. Evaluation Methods: Turning “Who Is Stronger” Into Verifiable Conclusions

    2.1 Distinguishing Claim Types: Closed vs. Open Tasks

    Model capability claims fall into three types:

    Claim type Typical description Verification method Conclusion strength
    Closed “It got all 30 of my unit tests green” Run the tests, compare against commit history High; directly reproducible
    Semi-open “It didn’t just copy the answer from public material” Search public literature and code repos, compare phrasing and reasoning Medium; must note search scope and date
    Open “It advanced a conjecture that had no proof” Have a domain expert review step by step, checking lemmas, citations, edge cases, and counterexamples Low; must state “conditionally holds”

    The claim “the proposition has no proof in the public project, so it isn’t copy-and-submit” can only count as evidence when both conditions hold:

    • The problem itself has long been open and is publicly verifiable;
    • Someone has checked the model’s proof line by line, rather than judging by impression.

    Otherwise it is just an unverified anecdote. Similarly, ranking-style conclusions such as “Model X is the only one that can do open-ended math problems” can only be treated as personal, subjective experience in the absence of a unified evaluation set and reproducible experiments.

    2.2 Building Your Own Evaluation Set: More Effective Than Arguing About Rankings

    The goal of evaluation is not to rank models, but to answer “under our task distribution, which tier and which configuration gives the lowest per-task cost.” Build a small evaluation set of 20–60 tasks drawn from your real development workflow. The task list can be organized like this:

    Task category Example task Acceptance method What to watch
    Bug fix Locate and fix a null pointer from a failure log Repro script + regression tests Localization precision, change scope
    Code review Review concurrency code for race conditions Compare against expert annotations for misses/false positives Miss rate, readability of explanation
    Small feature Add a validated paginated API to a service Unit tests + API docs First-pass rate
    Refactoring Split an oversized function and remove duplicated logic Tests unchanged + diff size Behavior fidelity
    Test generation Generate boundary tests for a utility function Mutation-test coverage Test effectiveness
    Open-ended research Produce a rigorous argument for a proposition with no known answer Expert review, item by item Argument rigor

    When running, use fixed prompts, fixed acceptance scripts, and fixed randomness parameters; repeat each task at least 3 times; and record these metrics:

    1. Average consumption per task (input/output/thinking tokens)
    2. First-pass rate (share passing acceptance on the first submission)
    3. Rework rounds (automatic iterations before human modification)
    4. Time-to-first-token and completion time (tracked per task category)
    5. Quota share consumed to pass (quota used in this eval / total quota for the period)

    Emphasize that input tokens and output (including thinking) tokens are usually priced differently. So even if two configurations consume the same input token volume, high-intensity reasoning significantly raises output-side cost. When comparing configurations, use “weighted cost per passing task” as the standard metric:

    1. Weighted task cost = Σ(input tokens × input price + output tokens × output price + thinking tokens × thinking price) / passing tasks

    Finally, to keep conclusions credible, run evaluations “blind”: hide the model source and version, and let reviewers score only on code quality and explanation quality. This avoids the brand-halo effect and keeps attention on verifiable deliverables.

    3. Quotas and Limits: Turning “Can It Finish” Into a Computable Problem

    3.1 The Nature of Tiered Limits

    Most coding assistants stack two kinds of windows in their quota system:

    • Rolling short window: for example, computed on a rolling 5-hour basis, deciding burst capacity over a period.
    • Long-cycle total: for example, computed weekly or monthly, deciding long-term sustainable throughput.

    The key point is that actual sustainable throughput is determined by the smaller of the two. The user’s situation above can be abstracted into a formula:

    1. Short windows per cycle = long-cycle quota / single short-window quota
    2. Sustainable throughput (standard tasks/week) ≈ standard tasks per short window × short windows per cycle × (1 - reserve rate)

    Assume 70% of a short window’s time is usable, half of that quota goes to master-slave collaboration, a single session completes about 2.5 “high-intensity standard tasks” per hour, a cycle holds 3 short windows, and we reserve 20% for bursts. Weekly sustainable task volume is roughly:

    1. 2.5 × 5 × 0.7 × 0.5 × 3 × 0.8 ≈ 10.5 (high-intensity standard tasks)

    From this we can derive a few practical principles:

    • Open fewer parallel sessions: 4 concurrent sessions don’t reduce total thinking tokens; they only make the limit hit sooner and compress each session’s continuous working time.
    • Treat the main model as a “planner,” not an “executor”: give planning, architecture design, and hard problem-solving to the main model, and mechanical execution to cheaper models, to lower consumption on the main tier.
    • Schedule long tasks by sustainable throughput: the number of unattended overnight tasks shouldn’t exceed the ceiling estimated from “weekly quota / number of short windows,” and should leave fallback room.

    3.2 Validate Requirements With the Smallest Model Before Deciding to Upgrade

    Another common mistake is reaching for the highest tier to explore vague requirements from the outset. A more economical approach is to build an “evaluation pyramid”:

    1. Use a low-cost tier to quickly validate whether the requirements are clear and the approach feasible.
    2. Only after validation passes and a genuinely hard part is confirmed should you switch to the high-intensity tier.
    3. Terminate tasks that fail validation promptly, so high-cost tiers don’t keep burning in the wrong direction.
    Usage scenario Recommended tier Reason
    Requirement clarification, draft proposals Low-cost/fast tier Prioritizes response speed and coverage
    Code search, unit-test completion Low-cost/fast tier Clear task boundaries, auto-verifiable
    Bug localization, code review Medium tier Needs some reasoning, but can be shared across master-slave collaboration
    Architecture design, complex refactoring High-intensity tier Decision quality directly affects downstream rework cost
    Open-ended research, proof derivation Highest tier (used sparingly) High unit price, long duration, requires step-by-step human verification

    4. Master-Slave Model Collaboration: Spend Quota Where It Counts via Task Routing

    4.1 Design Principles

    Master-slave collaboration (one more capable “main model” handles planning and decisions, while several cheaper, faster “slave models” handle execution) is fundamentally about matching token consumption to task difficulty. Four principles guide the design:

    1. Difficulty-based routing: dynamically select the tier by task type and historical pass rate.
    2. Clear escalation conditions: for example, if a sub-model fails twice in a row, the change touches core modules, or review flags high risk, auto-escalate to the main model.
    3. Results must be verifiable: sub-model output must pass tests, static checks, or structured template validation, avoiding “looks complete but didn’t actually pass.”
    4. Routing must be measurable: record each task’s tier, token consumption, and acceptance result, and periodically review whether the routing rules are sound.

    4.2 Rules File and Routing Pseudocode

    These principles can go into a project-level collaboration rules file, serving as the shared “command protocol” across all sessions:

    1. ## Task Routing and Acceptance Conventions (Example)
    2. - Goal: Reduce main-model token consumption by 40% while keeping the regression
    3. test pass rate at or above baseline.
    4. - Division of labor:
    5. 1. Bug localization, code review, small feature development -> executed by the fast model.
    6. 2. Cross-module refactoring, architecture decisions, hard bugs -> handled by the main model.
    7. - Escalation conditions: the fast model fails 2 times in a row, or the change touches
    8. core transaction or permission modules.
    9. - Acceptance: every change must include reproducible steps and test results; a change that
    10. does not pass tests counts as incomplete.
    11. - Budget: output tokens per session must not exceed X; when exceeded, report progress first,
    12. then request to continue.

    The routing logic can be described with pseudocode:

    1. def route(task, history, budget):
    2. """Select the execution tier by task features and escalate on failure."""
    3. if budget.remaining() < task.estimated_cost() * 1.5:
    4. return Defer(reason="Insufficient budget; suggest merging into the next short window")
    5. tier = "fast"
    6. for attempt in range(1, 4):
    7. result = dispatch(task, tier)
    8. if result.verified(): # tests/static checks/template validation passed
    9. return result
    10. if should_escalate(task, attempt, result):
    11. tier = "main" # escalate to the main model
    12. return HumanReview(task, history=attempt)

    4.3 Cost Control: Prioritize by Queue, Not by Blind Parallelism

    When quota is tight, a sounder approach is to rank tasks: finish “tasks blocking others” and “tasks with the nearest deadlines” first, then handle exploratory tasks. Score the task pool along these dimensions:

    1. Priority = blocking impact weight × urgency + business value - expected cost coefficient

    At the same time, use task completion rate rather than token consumption as the core metric. Focusing only on “how much quota was used” easily leads to sacrificing delivery quality to save quota; looking only at “how many tasks were done” easily leads to under-investing in high-value tasks.

    5. Output Economy: Same Capability, Two Uses Under Different Budgets

    Even the same model should be used differently under different budget constraints. Divide coding-assistant output into three types:

    Output type Characteristics When budget is ample When budget is tight
    Verifiable output Code, config, tests, commands Let the model iterate on its own until it passes Cap the number of iterations; escalate on failure
    Intermediate reasoning Thinking process, tradeoff analysis Keep full reasoning for retrospective review Require a plan first, then step-by-step execution
    Decorative content Pleasantries, long summaries, repeated explanations Acceptable Explicitly forbid or compress into a bullet list

    A practical technique is to have the model output a plan first, then execute step by step. For example, require:

    1. First list the execution plan and risk points in no more than 200 words;
    2. Begin modifying only after human confirmation;
    3. After each step, output a one-line status summary.

    The benefits are threefold:

    • Lower trial-and-error cost: when the direction is wrong, you lose only a few tokens before correcting course.
    • Easier human intervention: reviewers only need to confirm the plan, not read through all the reasoning.
    • Better observability: after an interruption, you can resume from the latest status summary instead of starting over.

    6. Resumable Long Tasks: From “Hit the Limit and Restart” to “Resume From Checkpoint”

    6.1 State Checkpoints and Idempotent Execution

    What unattended long tasks fear most is “hit the limit and lose all prior work.” Two engineering approaches address this:

    • State checkpoints: split the task into independently verifiable steps, and after completing each, persist the intermediate result (for example, the list of changed files, the test report, the next-step plan), so the next session resumes from the checkpoint.
    • Idempotent execution: make repeated execution side-effect-free—for example, check whether a file already contains the target change before deciding whether to write.
    1. Task state machine: plan -> execute -> verify -> commit -> review
    2. Every state transition is written to a checkpoint file:
    3. {
    4. "task_id": "refactor-order-service",
    5. "step": "verify",
    6. "changed_files": ["src/order/service.py", "tests/test_service.py"],
    7. "last_result": "3/5 tests passed",
    8. "next_action": "fix the remaining 2 failing cases"
    9. }

    6.2 Quota Scheduling Strategy

    At the scheduling layer, adjust task pace dynamically based on “remaining short-window quota”:

    • Plenty of remaining quota: push multiple tasks in parallel as planned.
    • Remaining quota under 30%: stop starting new high-intensity tasks; only let in-progress tasks wrap up.
    • Short-window limit triggered: suspend the task and save a checkpoint, auto-resume after the window resets.
    • Approaching long-cycle quota: switch to a low-cost tier or human handling, avoiding “hit the wall and can’t work all day.”
    1. def schedule(tasks, usage):
    2. if usage.remaining_ratio() < 0.3:
    3. return [t for t in tasks if t.in_progress()] # wrap up only, no new tasks
    4. if usage.hit_short_window():
    5. return resume_from_checkpoints() # suspend and save checkpoints
    6. if usage.remaining_ratio() < 0.1:
    7. return [t.with_tier("fast") for t in tasks] # downgrade to the fast tier
    8. return tasks

    6.3 Safety Boundaries for Unattended Work

    Nighttime unattended work doesn’t mean giving up control. Follow the principle of least privilege:

    • Run in an isolated environment or a separate branch; forbid direct operation on production environments and real data.
    • Set an allowlist for command execution; high-risk operations (deleting data, changing permissions, external release) require human confirmation.
    • Keep complete operation logs and change diffs for next-day review.
    • Set a per-task time limit and a maximum retry count to prevent tasks from falling into infinite loops.

    7. Common Pitfalls and Improvement Suggestions

    Common pitfall Potential consequence Improvement suggestion
    Running multiple sessions at once to “spread the quota” Limits hit sooner; long tasks interrupted repeatedly Queue by sustainable throughput; split work across master-slave models
    Exploring requirements with the highest tier from the start Large wasted consumption; bigger loss when the direction is wrong Validate feasibility with a low-cost tier first
    Stuffing all files into context at once Input tokens spike; attention diluted Retrieve relevant code and docs on demand; feed in segments
    Measuring by token consumption instead of tasks completed Sacrificing quality to save quota, or blindly burning quota Use weighted task cost and first-pass rate as core metrics
    Overnight tasks with no checkpoints and no permission constraints Restart from scratch after interruption; possible accidental production operations State checkpoints + idempotent execution + least privilege
    Using rumor-based rankings instead of your own evaluation Selection doesn’t match real tasks Build a blind evaluation set; choose tiers by scenario

    8. Conclusion: Capability Evaluation Must Return to Engineering Practice

    Discussions about coding assistants often start from “who is stronger” and ultimately return to “how to use them well.” Whether it’s hitting a limit during unattended overnight work, or a carefully designed task route getting compressed by limits, both are essentially signs that capability is growing faster than scheduling and cost management.

    For developers and enterprise users, the more worthwhile directions to invest in are:

    1. Build an evidence-based evaluation system that distinguishes closed from open tasks and clarifies conclusion strength.
    2. Quantify “can it finish” with a quota model, and schedule tasks by sustainable throughput.
    3. Use master-slave collaboration and task routing to lower per-unit cost, reserving high-price tiers for genuinely difficult decisions.
    4. Backstop with resumable long tasks and unattended safety mechanisms, so that an interruption no longer means starting over.

    As the user put it, you always have to pay back what you take out on the streets—quota won’t increase out of nowhere, and capability won’t automatically convert into output. Linking evaluation, scheduling, optimization, and safeguarding into a closed loop is the key to reliably turning a coding assistant’s “high-intensity reasoning” into delivery quality.