A Practical Guide to LLM Evaluation: From Math Reasoning to Quota Design

1. Why Evaluating a New Model Deserves Serious Attention

Whenever a new large language model (LLM) is released, the community quickly floods with reviews. However, these evaluations are often fragmented: some draw conclusions based on the success or failure of a single prompt, some rely on subjective vibes as ranking criteria, and others mix performance across completely different scenarios. For developers and enterprise users, this approach to evaluation suffers from three main flaws: it is irreproducible, incomparable, and non-cumulative.

A responsible evaluation must answer at least three questions: What tasks are being tested? What is the baseline for comparison? Under what cost constraints will it be used? Using “math reasoning capability” and “real-world coding tasks” as two main threads, this article breaks down an actionable evaluation methodology. Along the way, it also discusses a frequently overlooked topic: the rational design of usage quota systems.

2. Math Reasoning: The Litmus Test for LLM Capabilities

2.1 Why Use Math as a Litmus Test?

Mathematical tasks offer three inherent advantages:

• Verifiability: Answers are clearly right or wrong, requiring no subjective scoring, and results can be audited.
• Continuous Difficulty Gradient: Ranging from basic arithmetic to open-ended conjectures, math tasks can precisely map a model’s capability boundaries.
• Resistance to Memorization: Open-ended problems have a vast space of variations, making it difficult to score high simply by memorizing question banks.

2.2 Three Levels of Capability

The industry generally categorizes a model’s mathematical capability into three levels:

1. Routine Calculation and Derivation: Solving equations, symbolic simplification, and numerical computation. Mainstream flagship models can generally handle these, offering limited differentiation.
2. Competition-Level Problems: These require multi-step construction and techniques. Top-tier models begin to拉开差距 at this level.
3. Open-Ended Mathematical Problems: With no readily available answers, these require proposing conjectures, constructing proofs, or finding counterexamples. This is currently recognized as the hardest level. The vast majority of models can only produce text that “looks like a proof,” while models that can consistently make real progress are extremely rare.

When evaluating, it is recommended to build a custom tiered test set, run different models against the same set of questions, and record both the accuracy and the number of reasoning steps, rather than relying on single-prompt experiences.

3. Chain-of-Thought Quality: A More Informative Window Than the Final Answer

Even with identical accuracy rates, the Chain-of-Thought (CoT) generated by different models can vary significantly, primarily across three dimensions:

• Conciseness: Some models produce short, direct CoTs, reaching the conclusion in a few steps. Others take long detours and repeatedly second-guess themselves. The former implies lower token consumption, lower latency, and reduced costs.
• Structure: Whether the reasoning steps are readable and auditable directly impacts the efficiency of manual review.
• Naturalness of Style: In scenarios requiring explanatory text output, whether the expression feels “human-written” has a tangible impact on final delivery quality.

A practical evaluation technique is the Anchor Positioning Method: instead of striving for an absolute ranking, use a few models you are already familiar with as a reference baseline to determine where a new model falls. For example, “The new model’s math capability falls between Tier A and Tier B, which I commonly use.” This relative positioning is more robust and better aligns with practical selection needs than absolute score rankings. After all, the essence of model selection isn’t finding the “best in the world,” but determining whether a solution is better or worse than your current one.

4. Coding Capabilities: Testing with Real-World Engineering Tasks

4.1 Task Design

Beyond benchmark scores, real-world engineering tasks offer a closer approximation of actual use cases. Typical tests include:

• Locating Hidden Bugs: Given a codebase with legacy technical debt, see if the model can find the root cause rather than just patching surface-level symptoms.
• Code Review: Check for boundary conditions, concurrency issues, and resource leaks—defects that static analysis tools often struggle to cover.
• Minor Feature Development: Add a small feature to an existing system to evaluate how well the model respects existing code styles and architectural constraints.

4.2 Multi-Agent Collaboration Architecture

A highly recommended practice is the “Primary Model + Lightweight Sub-model” division of labor:

# Example Project Collaboration Instruction File
[Division of Labor Strategy]
Primary Model: Task decomposition, complex logic reasoning, cross-file modifications, final integration
Sub-model: Repetitive checks, style reviews, simple bug location, comment completion
[Dispatching Principle]
All template-able checking tasks should be prioritized for the sub-model
The primary model only handles steps requiring deep reasoning

The core value of this architecture lies in cost control. Lightweight sub-models are cheaper and faster. By offloading mechanical work, the quota for the high-end model can be concentrated where it matters most. Tests show that a rational division of labor can significantly slow down the rate of quota consumption while maintaining virtually unchanged delivery quality.

5. Usage Quota Systems: An Underestimated Product Design Problem

5.1 A Typical Quota Dilemma

Many AI coding tools employ a dual-layer quota system: a “short rolling window + weekly limit.” The problem is that the conversion relationship between the two is often opaque to users. Consider this illustrative calculation:

# Conversion trap of dual-layer quotas (illustrative data)
window_hours = 5          # Short window: 5-hour rolling window
weekly_windows = 3        # Weekly limit actually covers only about 3 full windows
daily_use = 8             # High-intensity usage hours per day for heavy users
# The nominal "weekly quota" actually supports only about 1.5 working days of high-intensity usage
effective_days = weekly_windows * window_hours / daily_use

A typical scenario: a user runs a long task overnight and hits the short-window limit by morning. Shortly after resuming, they hit the weekly limit—which, when calculated, is equivalent to only three short windows. This misalignment between “nominal period” and “actual capacity” is the most common trigger for trust crises in quota system design.

5.2 Design Principles

A well-designed quota system should meet the following criteria:

• Transparent and Calculable: Users should be able to estimate when they will hit the cap based on historical usage, rather than being unexpectedly cut off.
• Observable Usage: Provide real-time dashboards and threshold alerts, rather than post-facto notifications.
• Degradation over Hard Cut-offs: When the limit is reached, offer a low-speed tier or a pay-as-you-go channel to ensure long tasks are not interrupted.
• Peak Elasticity: Provide a reasonable buffer for occasional tasks that run continuously for several hours.

When selecting tools, enterprise users should also pay attention to the usage monitoring and alerting capabilities of mainstream cloud platforms (such as Baidu AI Cloud) model services, treating “quota observability” as a hard metric in their procurement evaluation.

6. Actionable Checklist: Establish Your Own Evaluation Process

1. Fixed Baselines: Maintain a proprietary test set covering math, coding, and long-context tasks, and version-control your prompts.
2. Control Variables: Use the same batch of tasks, the same set of prompts, and the same environment to avoid “good mood today” type errors.
3. Record Metrics: Accuracy, CoT length, token consumption, latency, and rework count.
4. Dual-Track Positioning: Use benchmark scores to locate the capability tier, and use engineering tests to verify delivery quality.
5. Cost Accounting: Factor in the rate of quota consumption. When capabilities are similar, prioritize the solution with a more favorable cost structure.
6. Regular Re-testing: Models iterate rapidly. It is recommended to re-run benchmarks quarterly to prevent conclusions from becoming outdated.

7. Conclusion

Evaluating an LLM is essentially answering the question: “On what tasks, at what cost, and achieving what quality?” Math reasoning reveals its reasoning ceiling; CoT quality tells us whether its reasoning process is reliable and readable; engineering tests show whether it can integrate into real workflows; and the quota system determines whether all of this can happen at a sustainable cost. Incorporating these four dimensions into a single evaluation sheet will make your model selection process far more reliable than relying on “a day’s subjective experience.”