How to Scientifically Evaluate the Mathematical and Coding Capabilities—and Cost—of Large Language Models

    A practical framework for evaluating large language models across mathematical reasoning, software engineering, long-horizon agent tasks, readability, quotas, and cost. It explains why leaderboard scores are insufficient, how to build reproducible multi-dimensional benchmarks, and how to optimize model routing under budget and quota constraints.

    How to Scientifically Evaluate the Mathematical and Coding Capabilities—and Cost—of Large Language Models

    When selecting a large language model, one of the most common mistakes is treating leaderboard position as a direct measure of engineering usefulness. Mathematical reasoning, coding ability, long-horizon task execution, writing quality, and quota cost are separate dimensions. A model that performs well on open-ended mathematics problems may not be equally effective at repository-level code review, bug localization, unit-test repair, or long-running agent tasks. Likewise, a model that provides a strong code-completion experience does not necessarily have strong formal reasoning capabilities.

    A professional evaluation should therefore move beyond single-score comparisons toward a multi-dimensional, reproducible engineering assessment with explicit cost constraints.

    1. Why a Single Leaderboard Cannot Support a Selection Decision

    First, task distributions vary significantly. Open mathematical problems emphasize exploration, verification, and multi-step reasoning. Business code review focuses on context understanding, constraint adherence, and risk detection. Small feature development requires requirement decomposition, interface consistency, and test coverage. Combining all of these tasks into one aggregate score can hide important weaknesses in specific use cases.

    Second, benchmark contamination and overfitting cannot be ignored. Public benchmarks are useful for understanding general capability boundaries, but they should not be the sole basis for model selection. Teams should build private regression sets that include real repositories, real defects, real mathematical modeling problems, and real documentation styles. Inputs, model versions, and scoring criteria should remain fixed to prevent repeated exposure to the same tasks from producing inflated results.

    Third, process quality is often more important than a single final answer. For mathematics, evaluation should consider whether the reasoning can be verified and whether intermediate steps are skipped. For coding tasks, it should measure regressions, adherence to project conventions, and the model’s ability to explain its changes. For agent tasks, it should examine planning quality, tool-call recoverability, and whether the system can degrade gracefully after failure. Looking only at the final output underestimates engineering risk.

    Fourth, quotas and costs affect practical capability. Short-term quotas, long-term quotas, concurrency limits, session counts, and subtask delegation strategies all influence whether a model can support continuous work. Evaluation should include task cost, cost per passing task, and quota utilization in the same report instead of measuring only the quality of an individual response.

    2. Build a Four-Dimensional Capability Model

    2.1 Mathematical Reasoning

    Mathematical capability should be divided into formal computation, informal modeling, open-ended problem exploration, and proof verifiability.

    A tiered task set can be used:

    • Basic computation problems test consistency and stability.
    • Applied problems test modeling ability.
    • Open-ended problems test exploration depth.
    • Proof tasks test logical completeness.

    For open-ended problems, the final answer should not be the only scoring criterion. Review a sample of intermediate steps, assumptions, and awareness of counterexamples. Where possible, use repeated-sampling consistency checks, process validators, and human review to avoid mistaking plausible-looking long reasoning for correct reasoning.

    2.2 Code Understanding and Generation

    Coding capability should cover at least:

    • Repository-level understanding
    • Bug localization
    • Patch generation
    • Test repair
    • Code review
    • Documentation synchronization

    The evaluation set should include small feature additions, edge-case fixes, refactoring, performance investigations, and security-risk detection.

    Scoring should go beyond whether a patch compiles. It should also measure whether the patch introduces regressions, follows existing abstractions, adds appropriate tests, and explains the impact of the change. For review tasks, track false-positive rate, false-negative rate, and the actionability of recommendations rather than simply counting how many issues the model identifies.

    2.3 Long-Horizon Agent Capability

    Long-running tasks evaluate planning, tool use, state maintenance, error recovery, and subtask delegation.

    Task traces should record whether the model:

    • Clarifies the objective before acting
    • Decomposes the task appropriately
    • Retries or degrades after failure
    • Delegates simple subtasks to a lightweight model
    • Requests human confirmation at critical points

    A model may perform very well in a single response but lose context during multi-step tool use. In that case, its engineering value can decline substantially. Agent capability should therefore be measured with process metrics such as task completion rate, average number of turns, number of human interventions, failure recovery rate, and trace readability.

    2.4 Readability and Collaboration Quality

    Readability includes concise reasoning, clear change descriptions, documentation that follows team conventions, and actionable review comments.

    Overly long and complicated reasoning increases review costs, while overly brief explanations may conceal risks. Evaluation should use blind reviews from different roles:

    • Developers assess actionability.
    • Reviewers assess risk coverage.
    • Maintainers assess long-term readability.

    Writing style is not merely a subjective question of whether the output “sounds human.” It is fundamentally a measure of collaboration cost.

    3. A Reproducible Evaluation Plan

    A practical approach is to combine a three-layer task set with dual-track scoring and continuous regression testing.

    Three-Layer Task Set

    • Public benchmark layer: Used to understand general capability boundaries, but not to make the final selection decision.
    • Private regression layer: Built from real repositories, real tickets, and real mathematical modeling problems, with fixed inputs and expected outputs. Run it on a regular schedule.
    • Adversarial and boundary layer: Covers ambiguous requirements, incorrect context, very large files, conflicting constraints, and failure-recovery scenarios.

    Dual-Track Scoring

    Result metrics include:

    • Task pass rate
    • Test pass rate
    • Defect repair rate
    • Mathematical accuracy

    Process metrics include:

    • Average number of turns
    • Token consumption
    • Human review time
    • Failure recovery rate
    • Quota usage

    Every evaluation should record the model tier, prompt template, context length, tool configuration, and random seed to ensure reproducibility. Important conclusions should be supported by blind review and confidence-interval estimates so that small-sample fluctuations are not mistaken for meaningful capability differences.

    4. Model Selection Under Quota and Cost Constraints

    Quota management is not simply a question of whether capacity is sufficient. It is also a task orchestration problem.

    Short-term quotas are useful for handling bursts of activity, while long-term quotas are better suited to sustained development planning. Teams should establish token budgets and define separate limits for mathematical exploration, code generation, code review, and documentation work. Strong reasoning tiers should be reserved for high-value tasks, while lightweight tiers can handle low-risk subtasks. Repeated context should be cached where supported, and failed tasks should have explicit retry limits and fallback paths.

    An illustrative configuration is shown below:

    1. {
    2. "task_routing": {
    3. "open_math_proof": "strong_reasoning",
    4. "repository_review": "strong_reasoning_with_reviewer",
    5. "small_feature_patch": "balanced_coder",
    6. "test_fix": "balanced_coder",
    7. "log_summary": "lightweight"
    8. },
    9. "budget": {
    10. "short_window_ratio": 0.7,
    11. "long_window_ratio": 0.5,
    12. "max_retry": 2,
    13. "fallback": "lightweight"
    14. },
    15. "observability": {
    16. "record_trace": true,
    17. "record_token_cost": true,
    18. "record_human_intervention": true
    19. }
    20. }

    Engineering teams should also implement task routing. Complex mathematics and critical architecture reviews should use a strong reasoning tier. Small features, log summarization, and simple reviews can be assigned to lightweight tiers. Agent configurations should explicitly prioritize delegating simple subtasks, while requiring human review before critical changes are merged.

    On general-purpose cloud platforms such as Baidu AI Cloud, teams can use object storage to retain evaluation sets and task traces, logging services to collect call chains, and monitoring and alerting to track quotas and failure rates. This connects capability evaluation with cost observability and creates a closed feedback loop.

    5. Common Mistakes and Final Recommendations

    Common mistakes include:

    • Replacing batch regression testing with a single demonstration
    • Using an aggregate score instead of scenario-specific results
    • Ignoring the effect of quotas on continuous tasks
    • Equating long reasoning traces with high-quality reasoning
    • Evaluating generation without evaluating review
    • Operating without a failure fallback plan

    A more reliable approach is to build a five-dimensional decision matrix covering quality, cost, latency, quota stability, and observability. Start with a small-scale A/B test, increase traffic gradually, and retain human fallback for high-impact tasks.

    Ultimately, evaluating the mathematical and coding capabilities of large language models requires a focus on real tasks, process metrics, and cost constraints. By decomposing capabilities, making evaluations reproducible, monitoring quotas, and designing degradable routing strategies, teams can make technical decisions that remain grounded in engineering evidence rather than marketing claims.