Evaluating LLM Math Capabilities and Managing Inference Quota in Practice
Leaderboards and single anecdotal runs can't tell you whether an LLM can actually solve open-ended math problems, or how to spend a limited inference budget. This article lays out two engineering practices: building a layered, reproducible math-capability evaluation—formal reasoning, modeling, and open exploration—with automatic verification and repeated sampling to separate capability from luck; and treating call quotas as a scheduling problem across overlapping short- and long-window limits through explicit budget modeling, tiered routing, an agent-collaboration convention file, and observability. It includes Python code skeletons, a routing table, a usage dashboard, and a pre-launch checklist for turning capability selection and quota management into measurable, optimizable engineering work.
After intelligent coding assistants and LLM inference services became a staple of daily development work, two questions moved to the center of engineering practice. The first is how to judge whether a model genuinely can solve open-ended math problems, instead of making that call from leaderboards and word of mouth. The second is how to allocate a limited inference budget to the tasks most worth spending it on, when call quotas are constrained by time windows. The first is a capability-evaluation problem; the second is a resource-scheduling problem. They look unrelated, but both require turning “vague experience” into “measurable, reproducible, governable metrics.” The two threads below follow these questions.
Why Leaderboard Rankings Aren’t Enough to Judge Math Capability
Most public benchmarks lean on multiple-choice, fill-in-the-blank, or standard-solution questions. Their boundaries are clear and their answers are decidable, so they naturally reward “pattern-recognition ability.” Open-ended math problems in real research and engineering have three different traits: the problem statement has to be formalized by yourself, there is no single solution template, and verifying the conclusion is expensive. A model’s performance on these three kinds of tasks can diverge from its rank on public leaderboards.
Two more realistic confounds:
- Data contamination. Overlap between training corpora and evaluation sets inflates scores and hides true reasoning ability.
- Sampling variance. Run the same problem multiple times at different temperatures and the conclusion may differ. A single successful case says nothing about stability.
So the first principle of evaluation is: tasks must resemble real usage, scoring must be reproducible, and results must be statistically significant.
A Reusable Method for Evaluating Math Capability
1. Layered Task-Set Design
Split the evaluation set into three layers and track the pass rate of each separately:
| Layer | Task Form | What It Tests |
|---|---|---|
| Formal reasoning | Proofs, inequality derivations, structural theorem application | Logical closure, no skipped steps |
| Modeling & derivation | Probabilistic modeling, optimization derivations, numerical analysis arguments | Whether the modeling assumptions are sound |
| Open exploration | Open problems with no standard solution, requiring multi-round tool calls | Autonomous planning and self-correction |
The third layer has the highest discriminating power and is the only one that can’t be scored by keyword matching.
2. Scoring and Judgment
- Auto-score anything verifiable: hand conclusions that can be checked by symbolic computation or numerical validation to a program.
- Route what can’t be auto-verified to sampled human review: have two reviewers score independently and record the disagreement rate.
- Also record three derived metrics: reasoning-chain length, number of self-corrections, and number of tool calls consumed per problem. These three directly determine cost.
A minimal evaluation skeleton:
def evaluate(model_client, tasks):results = []for t in tasks:runs = [model_client.solve(t.prompt, k=1) for _ in range(t.repeat)]passed = sum(verify(t, r) for r in runs) # verify: symbolic/numeric checkresults.append({"task": t.id,"pass_rate": passed / t.repeat, # stability"avg_steps": mean(len(r.trace) for r in runs),"tokens": sum(r.usage for r in runs),})return aggregate_by_layer(results) # aggregate results by layer
The key is t.repeat > 1. Without repeated sampling, you can’t separate “capability” from “luck.”
The Nature of Quota Constraints: Resource Scheduling Across Two Overlapping Windows
Much of the frustration users feel comes from the same phenomenon: the short window still has quota left, yet the long window gets exhausted first. This isn’t a product defect—it’s an inevitable outcome of the quota model.
Viewed through queueing theory, short-window limiting controls instantaneous concurrency pressure, while long-window limiting controls cumulative resource consumption. When a task is neither interruptible nor compressible, long tasks and long windows inevitably conflict: you can’t squeeze five hours of inference into a shorter span, and you can’t interrupt a long session that’s already halfway through.
The right response isn’t to complain about how the windows are divided, but to model the budget explicitly:
class Budget:def __init__(self, short_window, long_window):self.short = TokenBucket(short_window.capacity, short_window.period)self.long = TokenBucket(long_window.capacity, long_window.period)def try_acquire(self, est_tokens, priority):if self.short.remaining() < est_tokens or self.long.remaining() < est_tokens:return enqueue(priority) # degrade or queue, not fail-and-retryself.short.consume(est_tokens)self.long.consume(est_tokens)return granted()
In practice you can reuse well-established general patterns: token buckets for rate control, sliding windows for cumulative volume, and a priority queue for preemption. This is the same model as rate-limiting components and message-queue consumer backpressure.
Spending the Budget in the Right Place: Tiered Routing and Agent Dispatch
The gains from quota governance come mainly from task dispatch, not from compressing individual calls.
1. Tier by Task Nature
- Flagship reasoning tier: architecture design, cross-module refactoring, hard defect root-cause analysis, open-ended math derivations.
- Balanced tier: single-module implementation, test-case generation, routine code review.
- Lightweight/fast tier: formatting, renaming, adding comments, bulk mechanical edits.
In practice, mechanical tasks often make up more than half of a code-agent session. If all of them draw on the flagship tier, the long-window quota drains fast.
2. Fix Dispatch Rules in a Collaboration Convention File
Maintain an agent-collaboration convention file at the repo root so the main controller agent explicitly hands subtasks to sub-agents:
# Agent Collaboration Convention (excerpt)## Task Dispatch- Mechanical tasks (bug localization, code review, formatting, small feature increments, adding unit tests)→ Dispatch to sub-agents using the balanced/lightweight tier- Architecture design, cross-module refactoring, complex root-cause analysis→ Handled by the main controller agent using the flagship reasoning tier## Budget Constraints- Default per-session budget: 500K output tokens / 3000 tool calls- When a sub-agent's single change exceeds 400 lines, it must first be split into independently verifiable subtasks- When a budget threshold is reached, write a checkpoint and report; never expand task scope on your own## Recoverability- Before each subtask starts, record: goal, files involved, expected verification method- After an interruption, resume from the checkpoint without repeating completed verification steps
The real value of this file is turning “saving quota” from a verbal request into an explicit instruction to the agent, instead of relying on the user to remind it every time.
3. Isolate Model Differences with a Routing Layer
Add a thin adapter between the application and the models so routing policy stays decoupled from business code:
ROUTE = [("reasoning-heavy", "flagship"), # complex derivations("code-review", "balanced"), # review("mechanical-edit", "fast"), # mechanical edits]def route(task):return pick(ROUTE, task) # weights can be tuned dynamically by measured pass rate
The benefit: when a higher-capability tier becomes available, you only adjust the weights without touching business logic, and it also naturally supports multi-model canary rollouts and automatic fallback.
Observability: Making Quota Consumption Explainable
Budget governance starts with visibility. Collect at least four dimensions and feed them into logging and alerting:
- Call dimension: task type, tier selected, input/output tokens, tool-call count.
- Effectiveness dimension: first-pass rate, rework rounds, human-takeover rate.
- Time dimension: per-task wall-clock duration, queue wait time, failed-retry count.
- Quota dimension: real-time remaining balance of short and long windows, estimated time to exhaustion.
A usage view ready to drop onto a dashboard:
| Task Type | Tier | Sessions | Share | First-Pass Rate |
|---|---|---|---|---|
| Bug localization | Balanced | 3 | 42% | 71% |
| Code review | Balanced | 2 | 18% | 84% |
| Refactor design | Flagship | 1 | 25% | 55% |
| Mechanical edits | Lightweight | 1 | 15% | 93% |
When the “flagship tier first-pass rate” drops below some threshold, it means the tier is being used in the wrong place—not that the model is inadequate. This kind of data guides model selection better than any subjective ranking.
Pre-Launch Checklist
- Does the evaluation set include at least one class of open tasks with no standard solution, each sampled multiple times?
- Are all automatically verifiable conclusions handed to program validation, with humans reviewing only what can’t be auto-judged?
- Are dual thresholds for tokens and tool calls set at the session level, with a checkpoint written when exceeded?
- Have mechanical tasks been dispatched to low-cost tiers through the collaboration convention file?
- Is there a priority queue to handle long tasks, so the long-window quota isn’t fragmented and drained by short tasks?
- Are there fallback paths and a routing adapter layer to avoid a full-chain blockage when a single tier is unavailable?
- Are usage and effectiveness metrics reported together, so you can answer “where did the quota go, and was it worth it”?
Conclusion
Comparing model capabilities ultimately comes down to two numbers—task success rate and unit cost—while quota constraints ultimately come down to scheduling and dispatch strategy. Once you turn evaluation into a repeatable experiment, budget into an explicit scheduling object, and dispatch rules into something baked into the repo rather than held in your head, both selection debates and quota anxiety become engineering problems you can compute and optimize.
