Evaluating LLM Math Reasoning and Quota Scheduling in Practice
This article explains how to evaluate large language models across several dimensions rather than judging them by a single task. It covers math proof tasks, model routing and sub-agent division in engineering work, quota observability, and cost review through A/B testing.
1. From Single-Task Experience to Multi-Dimensional Evaluation
A common mistake in discussions of LLM capability is treating the result of one task as an overall judgment of the model. For example, on advanced math tasks, asking a model to reconstruct a proof when only the proposition is given is very different from asking it to restate an existing proof. The former tests concept retrieval, argument organization, counterexample exclusion, and formal expression. The latter is closer to information organization. If conclusions are drawn from a single success or failure, variables such as task type, prompt quality, context length, and quota status are easily overlooked.
A more reliable evaluation framework should separate at least the following dimensions:
- Proof reproduction: Given a theorem and its proof, assess whether the model can accurately rewrite it, fill in missing steps, and identify hidden assumptions.
- Proof reconstruction: Given only the proposition and background, assess whether the model can independently organize a proof path.
- Open-problem exploration: No standard answer exists, so experts judge the value and verifiability of the proposed approach.
- Engineering code ability: Defect localization, code review, small feature implementation, and test addition.
- Writing and readability: Whether the chain of thought is concise and whether explanations read like human technical writing.
- Cost and quota efficiency: Credits, time, and queue wait consumed to complete the same amount of valid work.
These six metrics cannot substitute for one another. Strength in math reasoning does not mean code review is cheap. A long chain of thought does not necessarily mean more rigorous reasoning. Public discussions often mention “capability tiers,” but the better approach is to treat tiers as hypotheses and verify them with a local benchmark set rather than adopting published rankings directly.
2. Math Evaluation: Evidence Matters More Than Labels
Advanced math tasks in particular require distinguishing between “can produce an answer” and “can produce a correct answer.” We suggest building a tiered benchmark set:
- Basic reproduction: Rewriting and checking known proofs, used to assess faithfulness.
- Missing-proof problems: Give the proposition without a proof and ask the model to complete the argument, focusing on implicit conditions and edge cases.
- Counterexample construction: Test whether the model blindly tries to prove a false proposition.
- Open exploration: No unique answer exists. Experts score novelty, verifiability, and potential for further development.
- Long-chain reasoning: Observe whether the chain of thought is readable and whether circular reasoning appears.
Open-problem ability is a scarce skill and should not be replaced by simple automated scoring. For these tasks, keep human review and traceable records. Another point is often overlooked: writing that reads like a human and mathematical correctness are not equivalent. A concise chain of thought may be easier to audit, but complex tasks still need enough intermediate steps. The evaluation sheet should record accuracy, human scores, average cost, and failure types.
3. Model Routing and Sub-Agent Division in Engineering Tasks
In real engineering work, high-capability models should not handle every task. A more reasonable approach is task tiering, model routing, and sub-agent division. For example, defect finding, code review, and small feature additions can go to low-cost models, while complex proofs, architecture design, and hard defect localization go to high-reasoning tiers. The task orchestration configuration should define clear responsibility boundaries to keep high-cost models away from low-value repetitive work.
An illustrative configuration follows:
routing:math_proof_reconstruction:tier: high_reasoningrequire_human_review: truecode_review:tier: low_costsmall_feature:tier: low_costsecurity_patch:tier: high_reasoningrequire_human_review: trueguardrails:max_budget_per_task: "placeholder"fallback_on_queue: "downgrade_or_defer"log_fields: ["task_type", "tier", "tokens", "latency", "quota_window"]
The key to routing is not fixing one particular model. It is matching task type, risk level, and budget ceiling. High-risk tasks keep human review. Long tasks support resumable execution. Queued requests may be downgraded or deferred. Sub-agent configuration should be auditable, so that the setup does not declare a division of labor while a high-cost model still completes all the work.
4. Quota, Queuing, and Credit Consumption Observability
In fixed-fee or credit-based services, short-term and long-term quota limits may apply at the same time. A common pain point is that after a short-window limit is hit, requests queue, the long-window quota drops quickly, and the effective usable time ends up lower than expected. Solving this requires observability into quota usage rather than relying on intuition.
We recommend logging each request’s task type, model tier, input and output tokens, latency, queue time, and quota window. Then calculate the consumption ratio for the short and long windows, and set three alert levels at 50%, 80%, and 95%. Example pseudocode:
def check_quota(usage, limits, task):alerts = []for window in ["short_window", "long_window"]:ratio = usage[window] / limits[window]if ratio >= 0.95:alerts.append((window, "critical"))return "reject_or_downgrade"elif ratio >= 0.80:alerts.append((window, "warning"))elif ratio >= 0.50:alerts.append((window, "notice"))return alerts
Supporting strategies include setting budget ceilings for long tasks, caching results for repeated queries, deferring non-urgent tasks to off-peak hours, automatically downgrading queued tasks to a lower-cost tier, and applying exponential backoff to retries to avoid wasted consumption. The goal of cost control is not simply to use less. It is to increase output per valid task.
5. Evaluation Experiments and Cost Review
We suggest running A/B comparisons across tiers on the same set of tasks. Record pass rate, human scores, average tokens, average latency, and quota consumption, then calculate the “cost per valid task.” Math proof tasks should keep expert review. Open-ended problems should not be scored automatically. Code tasks should be paired with test cases and static checks. After a model upgrade, run regression tests to catch degradation in any task category.
The final result should be a three-dimensional conclusion: whether capability meets the target, whether cost is acceptable, and whether quota usage is sustainable. Leaderboards or single experiences alone are not enough. Through task tiering, model routing, quota dashboards, resumable execution, and human review, teams can reach a stable balance between advanced reasoning and everyday engineering work.
