Evaluating LLM Math and Coding Ability and Planning Token Quotas in Practice
This article explains how to evaluate large language models on math and coding tasks using verifiable task sets instead of subjective impressions, and how to plan token quotas once usage is billed by weighted consumption. It covers layered evaluation of formal proofs versus open research problems, a three-stage acceptance loop for code tasks, why high reasoning tiers exhaust quotas quickly, capacity models for multi-window limits, prefix caching as a cost lever, strong/weak model routing across planner, executor and reviewer roles, and quota governance through task prioritization, automatic downgrade paths, checkpoint resume, and dual-threshold alerting. Readers get practical metrics, a routing table, and a minimal dual-window budget ledger implementation.
1. Start with verifiable tasks, not subjective impressions
Most arguments about whether “a model is good enough” fall apart for the same reason: they treat a single non-reproducible experience as a capability conclusion. A reusable evaluation has to turn impressions into a set of verifiable tasks first.
Verifiable means the output has an objective criterion. A mathematical proof can be checked line by line to see whether each inference holds and whether each citation is legitimate. Code can be run against unit tests, static analysis, and a build pipeline. Once you have criteria, sampling the same prompt multiple times yields a mean and a variance, instead of treating one impressive output as the ceiling of a model’s ability.
A private task set should cover at least four categories of samples:
- Real workload tasks: drawn from your own repositories, documentation, and bug tickets, not from public benchmark questions;
- Boundary tasks: very long contexts, cross-file dependencies, and problems that require disproving a claim or constructing a counterexample;
- Anti-contamination tasks: internal business context that is almost impossible to find in public corpora;
- Repeated-sampling tasks: run the same question 3–5 times and observe both accuracy and style stability.
Evaluation metrics should also go beyond “did it get it right.” In practice, these are more discriminating:
- first-pass rate;
- how many revision rounds are needed before a change passes;
- auditability of the reasoning (can each step be checked, or is the conclusion correct while steps are skipped?);
- readability of the output text (deliverables such as lecture notes or review comments especially reward output that “reads like a human wrote it”).
2. Look at math ability in two layers
Treating “math ability” as a single dimension is the most common confusion in current discussions. It has at least two layers, and they differ completely in where the ability comes from and how it should be evaluated.
The first layer is decidable formal derivation: given an explicit proposition, complete a step-by-step proof without introducing new assumptions, or show that the proposition is false and provide a counterexample. These tasks suit automated evaluation — the reasoning chain can be checked step by step, and errors surface as an illegal jump at some specific step.
The second layer is open-ended research problems: the question may have no accepted solution, and the model’s value lies not in “giving an answer” but in proposing testable ideas, narrowing the search space, and producing local lemmas. Scoring these as right or wrong is nearly useless; a more reasonable assessment looks at the density and falsifiability of the hypotheses the model produces across multiple iterations.
An easily overlooked fact: not every proposition in a public problem set comes with a proof. When a task asks the model to fill in a proposition that has not been fully proven, it cannot retrieve and reuse an existing answer — it has to do the reasoning itself. That makes such tasks closer to the real capability boundary than “transcription-style” tasks, and more suitable for a high-discrimination evaluation set.
The engineering conclusion is to evaluate in layers: run formal derivation as an automated regression suite continuously, review open-ended problems with human reviewers on a quarterly sample, and report the two sets of findings separately rather than merging them into one score.
3. Coding evaluation: switch from “can it write code” to “does it pass acceptance”
Code tasks have the clearest criteria and are the easiest to misuse — judging by “it produced some code” badly overestimates ability. A better evaluation covers a three-stage closed loop:
- The output compiles or parses: no syntax or type errors. This is the basic gate.
- Acceptance tests pass: new cases cover boundary conditions, not just the happy path.
- Regression and static checks pass: the change does not break existing behavior, and it conforms to style and dependency constraints.
The third point is skipped most often and shows the real gap most clearly. Whether a model can fix a small feature without quietly introducing a regression usually affects engineering experience more than whether it can write an algorithm. So putting static analysis, a dependency allowlist, and change-scope restrictions (which directories may be touched) into the evaluation pipeline is more informative than adding more algorithm problems.
4. Reasoning tiers and weighted consumption: the real reason quotas “shrink”
For most models, the resource bill is not based on the number of requests but on weighted token consumption: a request costs a weighted value of the sum of three token categories — input, output, and cache hits — and the weight grows as the reasoning tier rises and the thinking per step gets longer.
This explains a common puzzle: why a quota can be drained in a few hours on a high reasoning tier. A quick quantitative estimate makes the structure obvious:
cost per request ≈ (input tokens × w_in + output tokens × w_out + cache-hit tokens × w_cache) × tier weightsustainable requests ≈ period quota / cost per request
Consider the price structure common across the industry, where input is cheap, output is significantly more expensive than input, and cache hits are cheapest. In long-chain reasoning scenarios, output tokens dominate absolutely, and multiplying by a high tier weight can inflate the cost of a single request by an order of magnitude. At that point, “how many sessions I have open” says almost nothing about real load; what actually determines consumption is the average thinking length per request.
The practical implication is direct: treat the reasoning tier as a cost knob, not a performance switch. Use high tiers for planning, proofs, and hard defect localization; use low tiers for format conversion, bulk renaming, and comment completion.
5. A capacity model for multi-level limits
Beyond per-request cost, the time structure of the quota matters. The mainstream approach is two or even three tiers of limits coexisting: a short sliding window (for example, 5 hours) caps burst rate, while a long-period window (a week, a month) caps total volume. Both apply at the same time, and either one hitting its ceiling makes the resource unavailable.
Their relationship can be judged with one simple expression:
if long-period quota ≈ N × short-window quota, then under a "continuously maxed out" strategy,the whole long period only covers roughly N full short windows.
Example: if the long-period quota is about 3× the short-window quota, then sustained high-intensity use means roughly 15 hours of “maxed-out” usage will exhaust the entire period quota — there is quota left in the remainder of the period, but the rate is blocked. This is not a miscalculation; it is the design intent: short windows control peaks, long windows control totals.
So capacity planning cannot look at the total alone. Both constraints have to be computed:
- Rate constraint: short-window quota ÷ average cost per request = the maximum number of requests within the window;
- Volume constraint: long-period quota ÷ average daily consumption = how many days it can be sustained;
- take whichever is tighter as the real ceiling, and reserve a buffer at a 85%–95% utilization level.
6. Cache-hit billing: a stable prefix is the core of cost engineering
Cache hits on input tokens are usually billed at a significantly lower price. That yields a highly leveraged optimization principle: put unchanging content at the front and volatile content at the end.
# ---------- Stable prefix (unchanged over time; aim for continuous cache hits) ----------Role and constraints: you are a senior engineer, follow the rules below...Code standards: naming, directory layout, commit message format...Repository map: module breakdown and key entry points...Common commands: how to run build / test / static checks...# ---------- Volatile suffix (differs on every request) ----------This turn's task: <specific requirement or defect description>Relevant file diff for this turn: <diff>Session ID / timestamp / random ID: <keep these last>
Any volatile field that leaks into the prefix — a timestamp, a random ID, a summary of the previous answer — invalidates the cache for every token after it. In practice, moving session metadata out of the system prompt often improves the hit rate immediately.
In addition, replacing whole-file pastes of long documents and large code blocks with “retrieved snippets plus a stable explanation” both raises the hit rate and reduces per-request context cost. Context is not better when it is longer; it is better when it is more stable and relevant.
7. Strong and weak model collaboration: plan, execute, verify
When available resources are limited, the optimal solution is usually not “use the most expensive tier for everything” but division of labor. Split the work into three roles and let the strong model handle only the parts that cannot be replaced:
| Role | Responsibility | Resource strategy |
|---|---|---|
| Planner | Break down requirements, design the approach, define acceptance criteria | Strong model + high reasoning tier |
| Executor | Mechanical rewrites, bulk bug fixes, adding tests, running checks | Fast model + low tier |
| Verifier | Review the diff, judge whether it meets the bar, decide whether to retry | Strong model, but only reads the summary diff |
A routing table you can put to work (illustrative):
# Task type -> (assigned model, reasoning tier)ROUTER = {"architecture": ("strong", "high"), # Solution design, technology selection"proof": ("strong", "high"), # Proofs, root cause of complex defects"review": ("strong", "mid"), # Code review, regression judgment"edit": ("fast", "low"), # Bulk edits, comment completion"test": ("fast", "low"), # Generating and running test cases}
This also needs an orchestration note file in the repository root (plain text is fine) that states explicitly: which tasks must go to the executor model, which directories changes may touch, which check commands must pass, and after how many failures to escalate. That file is also part of the stable prefix, so you get both cache hit rate and enforceable constraints from it.
The verification loop must not be skipped: every change produced by the executor model has to pass tests and static checks before entering the verification stage. Otherwise, “saving cost” degrades into “accumulating rework.”
8. Quota governance: prioritization, downgrade, and observability
A quota is a finite resource and needs to be managed as one, rather than reacting only when an error appears. Three things have the highest priority:
- Task prioritization: P0 blocks production, P1 is required within the iteration, P2 is optimization work. P2 tasks queue automatically at high utilization and do not consume the high tier.
- Downgrade paths: when utilization crosses a threshold, downgrade in order — first lower the reasoning tier, then switch to the fast model, finally move to an asynchronous queue. Write the downgrade policy down in advance instead of deciding under pressure.
- Long-task splitting and checkpoint resume: cut long jobs into small steps that can be accepted independently, and drop a checkpoint at each step (commit, documentation, test results), so that once the quota is hit you resume from the checkpoint instead of starting over.
A minimal usable quota ledger (illustrative):
import timeclass QuotaBudget:"""Two-tier sliding-window ledger: short window caps peaks, long window caps totals."""def __init__(self, short_sec=5 * 3600, short_quota=100.0,long_sec=7 * 86400, long_quota=300.0):self.short_sec, self.short_quota = short_sec, short_quotaself.long_sec, self.long_quota = long_sec, long_quotaself.events = [] # (timestamp, weighted cost)def _used(self, window, now):return sum(c for ts, c in self.events if ts >= now - window)def estimate(self, in_tok, out_tok, cache_hit_tok=0, tier_weight=1.0):# w_out is usually much higher than w_in, cache hits are cheapest;# calibrate against the actual billing ratesw_in, w_out, w_cache = 1.0, 8.0, 0.25cost = (in_tok * w_in + out_tok * w_out + cache_hit_tok * w_cache) / 1e6return cost * tier_weightdef admit(self, cost, now=None):now = now or time.time()return (self._used(self.short_sec, now) + cost <= self.short_quota * 0.95and self._used(self.long_sec, now) + cost <= self.long_quota * 0.95)def record(self, cost, now=None):self.events.append((now or time.time(), cost))
On observability, at minimum do three things: emit the weighted cost of every request with per-session and per-project dimensions into a log service; set two alert levels at 80% and 95% for the short window and the long period separately; and correlate “consumption against output” (how much quota each accepted change consumed) so you can use it to recalibrate task prioritization and tier policy. Whether you self-host an inference cluster or use a cloud provider’s general compute, object storage, logging, and monitoring and alerting capabilities, this instrumentation scheme is the same — it solves a resource governance problem and is independent of the vendor.
9. Practice checklist
- Build a private, verifiable task set; evaluate formal derivation and open-ended problems in layers and report the findings separately;
- Replace “can it write code” with “first-pass rate + regression pass rate,” and put static checks into the pipeline;
- Treat the reasoning tier as a cost knob: high tiers for planning and verification, low tiers for bulk execution;
- Do capacity planning against two-tier limits and take whichever of the rate constraint and the volume constraint is tighter;
- Stable prefix plus volatile suffix, with no volatile field allowed to contaminate the prefix;
- Strong/weak model collaboration across three stages, and enforce that executor output must pass checks;
- Task prioritization + automatic downgrade + checkpoint resume, with 80%/95% dual-threshold alerts and consumption instrumentation.
The core judgment fits in one sentence: model capability evaluation belongs to a verifiable task set, and resource usage belongs to a quota ledger and a scheduling policy. Get both right, and you will not discover at the moment a limit is hit that what was truly scarce was never the quota, but the unplanned way of calling the model.
