A Practical Guide to LLM Evaluation: From Math Reasoning to Quota Management

Introduction: What “Evaluation” Really Means

For developers, the question “Is the new model any good?” is hard to answer—not because information is scarce, but because there is too much of it. Public benchmarks keep hitting new highs, yet real workloads often feel “a bit different.” Between benchmark scores and actual experience lies a systematic evaluation gap—no one else can answer “how does it perform on your problems, your codebase, and your budget?” This article distills evaluation practice into an executable methodology, breaking down what a rigorous evaluation should cover across four dimensions: mathematical reasoning, chain-of-thought quality, multi-level engineering collaboration, and quota cost.

1. Why You Need Your Own Evaluation System

Public benchmarks suffer from three inherent flaws. First, question leakage can inflate scores, making top leaderboard results hard to fully trust. Second, the distribution of benchmark problems is misaligned with real business needs—benchmarks lean toward “verifiable closed-form problems,” while real R&D involves many tasks that are “open-ended with unclear paths.” Third, a single score erases critical differences: for the same problem, one model converges with a concise chain of thought, while another wanders through an extremely long one. The former is cheaper and more stable; the latter may have a higher ceiling but poor reproducibility—and none of these differences are visible on a leaderboard. Therefore, a usable evaluation must happen on “your own problems, your own environment, your own budget.”

2. Mathematical Reasoning: Using “Open-Ended Problems” to Test the Upper Bound

Closed-form problems (with standard answers) and open-ended problems (with no known answer, or answers the evaluator cannot easily verify) test two different capabilities:

• Closed-form problems test convergent execution: step-by-step decomposition, segment-by-segment verification. Mainstream strong models generally score high here, and discrimination keeps fading.
• Open-ended problems additionally test three higher-order abilities: finding direction (proposing reasonable conjectures and routes from limited conditions), switching strategies (actively changing paths after a failure rather than grinding in place), and self-verification (reviewing intermediate results and correcting upon discovering contradictions).

Decisive gaps between models mostly surface on open-ended problems. Practical suggestions:

1. Prepare 10–20 open-ended problems from your own domain—do not use public problems, to avoid leakage interference;
2. Use a four-level scale: solved independently / solved after hints / not solved but with productive exploration / completely off track;
3. Record results by problem and by model to form a longitudinal comparison table, updated continuously as models iterate.

This problem set is your most important technical asset: leaderboards come and go, but your problem set and scoring criteria remain stable and comparable.

3. Chain-of-Thought Quality: Length Is Not the Only Criterion

Most people evaluate reasoning ability only by “right or wrong,” but chain-of-thought quality is an equally important observation dimension—it directly determines the actual cost and stability of using a model:

• Overly long, overly complex chains of thought may indicate the model needs multiple rounds of search and backtracking to converge. The ceiling isn’t necessarily low, but token consumption, latency, and error points all increase—and once the chain goes off track, it’s hard to pull back.
• Concise chains of thought indicate the model has already formed a fairly mature decomposition pattern for that class of problem, with clearer structure and a more “human-like” solution path, resulting in lower marginal cost on the engineering side.

A reference scoring rubric for chain-of-thought quality:

Dimension	What to Observe
Structure	Is the step decomposition clear? Is each step’s responsibility singular?
Branching strategy	Does it actively choose paths? Are branch goals clear?
Backtracking and correction	Can it find errors and self-correct? Is the correction effective?
Verification awareness	Are there intermediate checks and a final review?
Redundancy	Proportion of repeated attempts and irrelevant exploration

Methodologically: have candidate models run the same problem set, recording chain-of-thought length and structure scores per problem as long-term observation metrics. After multiple model iterations, tracking longitudinally will clearly show which models are “strong on hard problems but expensive and unstable” and which are “clean and efficient on mainstream problems.”

4. Engineering Coding Tasks: Focus on “Multi-Level Collaboration,” Not Single-Turn Performance

Real R&D isn’t one-shot generation—it’s continuously running tasks: finding bugs, reviewing code, adding small features. Such tasks are best suited to a “main model + sub-agent” architecture: the main model handles planning, decomposition, and result review; the sub-agent (typically configured as a smaller, faster, cheaper model) handles concrete execution. How well this architecture operates is itself a core evaluation dimension of engineering capability:

1. Task delegation: Does the main model delegate routine work like bug localization and template-based code review to sub-agents as much as possible, saving its own quota and context?
2. Review quality: Does the main model actually check the sub-agent’s results, rather than accepting them wholesale?
3. Context control: Does it avoid repeatedly reading large files and re-summarizing the same information?
4. Failure handling: Do subtasks have a backtracking limit? Does it consume indefinitely in one place?

A reference instruction configuration (pseudocode):

# Task instruction file (placed in project root, fixed template)
role: system_architect_reviewer
rules:
  - Plan before executing: produce a plan, then dispatch tasks
  - Delegate routine work (bug localization, code review, test scaffolding) to sub-agents where possible
  - Keep core design and logic judgment for yourself
  - Each subtask backtracks at most once; if unresolved, continue and summarize risks at the end

During evaluation, focus on whether the model “follows the division-of-labor strategy in the instruction file.” For the same task volume, one model has a concise chain of thought and dispatches efficiently, while another does everything itself—only the former truly saves time and money in production.

5. Quota and Cost: Understand the Limit Mechanism Before Evaluating

This is the most easily overlooked yet most direct factor affecting experience. Many subscription services use two-tier quotas: a short-window limit (hourly) layered with a long-cycle limit (daily or weekly). Before evaluating, you must clarify two things:

1. The conversion relationship between the two tiers. If the long-cycle quota translates to only a few full-load sprints of the short window, then “usable for a long time in one go” is an illusion—the real bottleneck is long-cycle capacity. A typical failure case: the previous cycle’s balance is nearly depleted, a new cycle starts and you plan to run for several days, but after one morning of high-intensity parallel use, the long-cycle quota drains far faster than intuition suggests.
2. The real overhead of high-tier modes. The higher the reasoning tier and the longer the chain of thought, the more output, and the faster quota is deducted. When multiple sessions run continuously at high tiers, even if “it hasn’t been long,” the weekly quota may be mostly consumed on day one.

Actionable recommendations:

• Estimate intensity before running tasks: sessions × reasoning tier × expected duration, then extrapolate weekly consumption;
• Proactively downgrade non-core tasks, reserving quota for core reasoning;
• Reduce concurrency—run long tasks serially or with small parallelism to avoid draining both short and long windows;
• Record quota usage at fixed times to establish your own “consumption baseline,” so anomalies in the next evaluation can be spotted by comparison;
• At the team level, if there are large-scale, stable invocation needs, consider commercial API services to convert a fixed subscription into elastically metered resources, which is typically more controllable than quota-based plans at scale.

6. Making Evaluation Reproducible: Records, Scoring, and Regression

Effective evaluation must be reproducible; otherwise, conclusions have no longitudinal reference value. Three suggestions:

1. Fix environment variables: the same prompt templates, the same context window configuration, the same toolchain versions;
2. Automate the process: batch-run scripts, automatically collect results, and archive scoring tables;
3. After each model version update, rerun the full problem set with the same parameters—doing “evaluation regression” just like software regression testing.

Once the mechanism is established, a complete evaluation report can be produced on the day a new model is released, without starting from scratch each time.

Conclusion

The entire methodology can be compressed into four steps: use open-ended problems to measure the reasoning upper bound; use chain-of-thought structure to measure stability and cost; use multi-level collaborative tasks to measure engineering maturity; and use a quota consumption baseline to measure economic efficiency. With this framework, even as leading models keep raising the bar, you can derive a capability ranking suited to your own scenarios—rather than relying on secondhand experience and marketing talk for technical decisions.