Evaluating LLM Math Capabilities and Optimizing Usage Costs

Whenever a new flagship Large Language Model (LLM) is released, the community inevitably falls into debates over “which one is better.” The focus usually centers on two aspects: the model’s actual performance on high-difficulty tasks like mathematical reasoning, and the fairness of rate-limiting quotas under subscription models. While seemingly independent, these two factors collectively determine a model’s real-world usability in engineering environments. This article aims to abstract emotional comparisons into a reusable methodology: how to measure capabilities, calculate quotas, and save costs.

1. Why Benchmark Rankings Are Insufficient for Model Selection

1. Saturation and Contamination of Benchmarks. A large number of public benchmark questions appear in training corpora, making it difficult to distinguish whether a model has “memorized the answers” or actually “knows how to solve the problems.” When multiple models approach perfect scores on standard benchmarks, rankings lose their discriminative power.

2. Open-Ended Math Problems as a Better Touchstone. True differentiation occurs on open-ended problems without standard answer templates: research-level proof construction, applied problems requiring multi-step modeling, and questions with incomplete conditions that require self-generated assumptions. These problems cannot be solved by memory retrieval and push the model to its actual reasoning limits. Empirically, models that can consistently handle open-ended mathematical problems are still rare and mostly concentrated in the highest reasoning tiers of top-tier vendors—a structural fact worth noting during evaluation.

3. Tier and Endpoint Differences Must Be Included in Evaluation. The same model family often features standard, high-reasoning, and maximum tiers. Even within the same tier, the underlying implementation on a general chat endpoint versus a programming workspace endpoint may differ. Discussing rankings without mentioning tiers is meaningless. The statement “Model A is better than Model B” must be qualified with “on which tier, on which endpoint, and for what task,” otherwise the conclusion is irreproducible.

2. Building a Reproducible Math Evaluation Framework

1. Tiered Test Sets. A three-tier approach is recommended: standard competition problems to verify the baseline, open-ended problems to verify the upper limit, and engineering approximation problems to verify usability. Fix the number of questions and random seeds for each tier to ensure reproducible comparisons across versions.

2. Evaluation Dimensions: Beyond Accuracy, Focus on the Chain of Thought (CoT).

• Conciseness: Superior models tend to have shorter, cleaner, and less redundant CoTs. CoT length is not positively correlated with capability—some models generate extremely long and complex CoTs without necessarily achieving higher accuracy. Long CoTs mean higher token costs and slower responses, which translate to tangible burdens in quota-limited scenarios.
• Intermediate Step Correctness: Even if the final answer is correct, the quality of intermediate reasoning determines the credibility of the conclusion, which is critical for verifiability in engineering scenarios.
• Naturalness of Expression: Whether the model explains problems and drafts solutions like an experienced human engineer directly impacts the collaboration experience.

The schematic code for the evaluation pipeline is as follows:

def run_eval(model, problem_set):
    report = []
    for p in problem_set:
        out = model.solve(p, tier="high")
        report.append({
            "id": p.id,
            "correct": grade(out.answer, p.reference),
            "cot_steps": len(out.reasoning),
            "cot_redundancy": redundancy(out.reasoning),  # Redundancy score
            "latency_s": out.elapsed,
        })
    return summarize(report)

3. “Anchor Positioning” During Version Iterations. When new versions are released frequently, rather than assigning absolute scores, it is better to use widely recognized anchor versions for relative positioning: “The new version sits between the last major release and the previous enhanced version.” This positioning method is coarse but robust, preventing minor score differences from being over-interpreted and facilitating seamless comparisons across different evaluation cycles.

3. Rate Limiting and Quota Mechanisms: An Underestimated Design Variable

1. The Imbalance of Dual-Window Rate Limiting. Many subscription services employ a dual-window rate limit combining “hourly windows + weekly windows.” When poorly designed, this leads to absurd experiences: the weekly quota equates to only three or four hourly windows. For example, suppose the 5-hour window limit is 100 units, but the 7-day total quota is only 300 units. A user only needs to use it intensely once in the first two days, and they will remain in a low-quota state for the remaining five days. Common community complaints stem from this—users run long tasks overnight, wake up to find the hourly window exhausted (which is acceptable), and then wake up the next day to find the weekly quota depleted. This exposes a flaw in the window ratio design rather than the user’s usage pattern.

A reasonable design should satisfy the basic inequality:

Weekly Quota >= Hourly Window Quota * Daily Available Windows * 5.5 ~ 7

If this ratio falls below 3–4, it indicates the product actually uses the “weekly quota” as the true constraint, rendering the hourly window virtually useless. The theoretical peak available duration is extremely low, inevitably leading to a poor user perception.

2. Unpredictability of Quota Consumption. In multi-agent modes, sub-agent calls, tool calls, and long-context replays amplify token consumption. Under multiple concurrent sessions, it is not uncommon to exhaust about one-third of the hourly quota in two or three hours. Service providers are obligated to provide reference consumption metrics for typical tasks to allow users to estimate usage; users should also manage quotas as system resources.

3. Design and Self-Build Recommendations. For platform providers: publish the conversion ratio between the two windows to ensure transparency; when quotas are exhausted, degrade to a lower rate rather than outright rejecting requests, preserving an elastic fallback. For teams building their own gateways: utilize general API gateways, logging services, and monitoring/alerting products available on cloud platforms (such as Baidu AI Cloud) to handle metering, logging, and threshold alerts, avoiding the “where did my quota go” dispute.

4. Cost Optimization Practices Under Multi-Agent Collaboration

1. Core Concept: Capability Tiering, Task Layering. Reserve the expensive high-reasoning tier main model for requirement decomposition, architectural decisions, and final acceptance. Delegate tasks with clear boundaries, low failure costs, and easy verification—such as bug hunting, code reviews, and implementing small features—to lightweight sub-agents. This “high-tier controller + lightweight executor” combination significantly reduces quota consumption without sacrificing quality.

2. Configuration Example (Markdown, Schematic).

# Agent Division Rules
## Main Agent (High Reasoning Tier)
- Requirement clarification, task decomposition, core module implementation, final acceptance
## Sub-Agent (Lightweight Model)
- Handles: bug localization and minimal reproduction, incremental code review, small feature implementation
- Output must include: failure cases, reproduction commands, change summary
## Constraints
- Changes involving core modules or data structures must be sent back to the main agent for review
- Tasks where the sub-agent fails 2 consecutive times are automatically escalated to the main agent

3. Task Tiering Checklist. Tasks that must be handled by the main model: architectural design, cross-module complex issues, critical external documentation. Tasks that can be safely delegated: unit test supplementation, style reviews, independent small features. Practical tip: explicitly defining “delegation criteria” in the rules file is more important than specifying “which model to use”—once the standards are clear, configurations can migrate smoothly when the underlying model changes.

4. Establish Consumption Baselines. Record the unit quota consumption for each type of task to form your own consumption baseline. Actively stagger usage during critical sprint periods to avoid hitting rate limits when computing power is needed the most.

5. Action Checklist for Developers

• Build a private set of 20–50 open-ended problems, reused across versions to form an evaluation asset immune to data contamination;
• Evaluate CoT quality rather than length, incorporating redundancy into evaluation metrics;
• Specify tier and endpoint before drawing conclusions; any “which is better” claim must be accompanied by contextual conditions;
• Calculate the hourly/weekly quota ratio before subscribing; cautiously evaluate the actual usable intensity for services with a ratio lower than 3–4;
• Use multi-agent task layering to control costs, spending high-reasoning tier quotas where they matter most;
• Establish monitoring and alerting for consumption, transforming quota usage from a “black box” into an observable engineering metric.

Conclusion

Emotional ranking debates offer limited value when evaluating new models. A more effective approach is translating the discussion into three actionable questions: how to measure capabilities, how to calculate quotas, and how to save costs. The tiered evaluation methodology provides a data-driven basis for selection, scrutinizing rate-limit ratios offers a guideline for service design, and multi-agent task layering maximizes the output of limited quotas. When the next wave of model releases arrives, this framework can be directly reused, independent of the marketing narratives of any single product.