A Practical Guide to Evaluating LLMs for Math Reasoning and Coding Agents
This guide explains how to evaluate large language models for math reasoning and coding agents as complete systems rather than single models. It covers how to set evaluation goals across capability, engineering, and cost layers, how to build a small test set for open-ended math problems, how to judge coding agents by the testable changes they deliver, and how to design task routing and quota controls. It also outlines a three-stage validation process: offline golden sets, online gradual rollout, and cost regression testing.
1. Define the Evaluation Goal First: Avoid Leaderboard-Style Conclusions
Math reasoning and coding agents are different capabilities. Math reasoning emphasizes abstraction, proof, counterexample construction, and whether the process can be verified. Coding agents emphasize repository understanding, tool use, delivering changes, test verification, and code review readability. If you judge with a single overall score or one trial experience, you can easily reach wrong conclusions. For example, a model may score well on standard problems but fail on open problems, produce code that cannot be reviewed as a patch, or stay stable on short tasks while frequently breaking on long ones.
We recommend a three-layer evaluation matrix:
- Capability layer: open math problems, proof rewriting, code localization, defect fixing, test completion, and review suggestions.
- Engineering layer: tool-call success rate, long-context stability, retry cost of failures, human takeover rate, and reversibility of changes.
- Cost layer: token consumption per task, the speed of short-cycle quota consumption, the share of long-cycle budget used, and queue wait time under concurrency.
Only when capability, engineering, and cost sit in the same table can a selection decision be explained.
2. Math Reasoning Evaluation: Open Problems Reveal Differences Better Than Standard Problems
Open math problems are hard to evaluate. They usually have no single standard answer, verification is expensive, and process quality matters more than the final answer. During evaluation, check whether the final conclusion is correct, whether key steps can be verified, whether the reasoning chain contains redundancy, and whether failures produce intermediate states that can be diagnosed. A long and complex chain of thought does not necessarily mean stronger reasoning. A concise, verifiable chain usually fits engineering use better. At the same time, a concise chain does not mean weak capability. Judge it together with task difficulty and verification results.
In practice, build a small “golden set” with competition problems, proof rewriting, counterexample construction, numerical estimation, and boundary-condition analysis. For each sample, record the final answer, key steps, verification method, and failure modes. For high-end math tasks, do not rely only on past model tiers or marketing claims. Run sample tests and observe stability across long reasoning chains. If a task needs multiple rounds of verification, set checkpoints so that a single failure does not waste the entire budget.
3. Coding Agent Evaluation: From Writing Code to Delivering Changes
The core of a coding agent is not producing code snippets. It must complete changes that can be reviewed, tested, and rolled back. Evaluation dimensions should include requirement understanding, repository localization, defect reproduction, patch quality, test coverage, review readability, tool-call stability, and human takeover cost. A common mistake is to measure only generation speed and ignore review cost. Longer and harder-to-read patches raise future maintenance costs.
For task orchestration, use layered routing with a “planner + worker” structure. High-capability models handle architecture decisions, security review, and complex defect localization. Lightweight models handle code search, log triage, unit test drafts, and low-risk small features. The configuration should state that parallelizable, low-risk search and review tasks go to lightweight models where possible, while high-risk decisions stay with strong models. The illustrative configuration below conveys only the routing and budget ideas. It does not correspond to any specific API.
routing_policy:planner: high-capability-modelworker: small-fast-modelrules:- when: task.type in ["code_search", "unit_test_draft", "log_triage"]route: workermax_tokens: 8000- when: task.type in ["architecture_decision", "security_review", "math_proof"]route: plannermax_tokens: 32000budget:short_window_ratio_warn: 0.7weekly_ratio_warn: 0.6hard_stop_weekly_ratio: 0.9
The key to this configuration is not the individual fields. It binds task types, model tiers, and budget thresholds together, so that not every request takes the highest-cost path.
4. Quota and Cost: Design Rate Limits as System Constraints
Short-cycle quotas suit controlling burst traffic. Long-cycle quotas suit controlling total budget. In practice, if short-cycle consumption rises clearly faster than long-cycle quota consumption, check for long sessions, repeated retries, sub-agent amplification, or poorly split tasks. Capacity planning should estimate tokens by task type, and long tasks should have checkpoints, failure backoff, and human takeover thresholds. Non-critical tasks can be queued for off-peak hours, while critical tasks keep budget headroom.
Observability should cover at least consumption curves, retry counts, cache hit rate, task interruption rate, human takeover rate, and unit change cost. Only when these metrics are stable can you tell whether the problem is insufficient model capability or poor orchestration and quota policy.
5. Conclusion: Replace Single-Point Experience with a Systematic Method
For math capability, look at verification quality and process readability on open problems. For coding capability, look at end-to-end delivery, test pass rate, and review cost. Quota policy must be designed together with task orchestration. A strong model is not the only answer. Layered routing, budget control, and observability often determine efficiency more than the model alone. We recommend a three-stage validation process: an offline golden set, online gradual rollout, and cost regression testing. This turns subjective experience into reproducible, comparable, and optimizable engineering conclusions.
