Evaluating and Selecting Large Language Models for Mathematical Tasks: An Engineering Approach
Evaluating the mathematical capabilities of large language models (LLMs) goes beyond simply comparing benchmark scores. This article explores a comprehensive engineering approach to LLM selection, emphasizing task layering, the evaluation of open-ended math problems, the distinction between chain-of-thought length and actual capability, and the establishment of a reproducible evaluation pipeline to guide practical deployment decisions.
In discussions about LLM selection, “mathematical capability” is often reduced to a leaderboard: the higher the score on a test set, the better the model. However, from an engineering perspective, this judgment method carries high risks. Mathematical capability is not a single dimension. It encompasses at least basic calculation, symbolic reasoning, constraint satisfaction, competition problem solving, formal proof, open problem exploration, process interpretability, and error recoverability. Different tasks have vastly different requirements for models. A candidate model performing stably on closed problems does not mean it can provide valuable research leads on open math problems; conversely, a model capable of generating long reasoning processes does not necessarily yield more reliable conclusions. Therefore, the more professional question is not “which model is absolutely stronger,” but rather “which type of model is more suitable given specific tasks, budgets, latency, and compliance constraints, and how to verify this using reproducible methods.”
Shifting from “Absolute Strength” to “Task Layering”
When evaluating mathematical capabilities, tasks must first be layered. The first layer is basic calculation and algebraic manipulation, focusing on accuracy, unit consistency, and boundary condition handling. The second layer is symbolic reasoning and multi-step constraint solving, assessing whether the model can maintain intermediate states and avoid variable confusion. The third layer involves competition problems and proof questions, testing strategy selection, counterexample construction, and proof completeness. The fourth layer is open mathematical problem exploration, evaluating the model’s ability to propose conjectures, find analogies, and design verification paths. The fifth layer is process quality, including whether the chain-of-thought (CoT) is concise and auditable, whether the expression is natural, and whether errors can be located. Discussing these layers together often leads to overgeneralized conclusions like “a certain model is comprehensively leading” or “completely useless.” A more reasonable approach is to set independent weights for each layer and then aggregate them based on business scenarios. For example, educational scenarios focus more on the naturalness of explanations and step readability; scientific research assistance emphasizes open problem exploration and verifiability; enterprise reporting prioritizes calculation accuracy and cost.Why Open-Ended Math Problem Capabilities Are Rare
The biggest difference between open-ended math problems and closed problems is the lack of a single standard answer. The model must not only provide a conclusion but also offer a verifiable reasoning path, potential counterexamples, applicable conditions, and a follow-up verification plan. These tasks demand much more from the model than standard problem-solving: they require long-term consistency, conceptual transfer, assumption management, and self-correction. Currently, different models exhibit significant stability differences on open problems. One cannot treat a single successful case as a universal capability, nor should absolute judgments like “the only one that can do this” be used. In engineering evaluations, open problems should be broken down into scorable items: whether the problem is restated accurately, whether the conjecture is motivated, whether the derivation has flaws, whether an executable verification plan is provided, and whether uncertainties are explicitly stated. Due to the difficulty of automatic verification, a combination of “automatic checker + expert review” is usually required: the automatic checker handles syntax, dimensional analysis, boundary, and formal proof checks; experts judge the value of the approach and proof completeness. Only by incorporating open problem capabilities into process scoring can we avoid being misled by one-off demonstrations.Chain-of-Thought Length Does Not Equal Capability Ceiling
The length of the reasoning chain is an easily misinterpreted metric. A longer CoT might mean the model is exploring more paths, but it could also indicate redundancy, repetition, and error propagation. Shorter CoTs are generally better for auditing, reproduction, and low-cost deployment, but they might omit critical steps. Evaluations should simultaneously observe “effective reasoning steps,” “invalid repetition ratio,” “critical step coverage,” and “conclusion verifiability.” For production systems, it is recommended to weight “conciseness and auditability” heavily rather than blindly pursuing the longest reasoning. Writing naturalness should also be abstracted into expression quality: whether terminology is accurate, whether explanations match the reader’s level, and whether known facts are distinguished from speculations. A model with strong mathematical capabilities but chaotic expression remains difficult to use in teaching, reporting, and decision support.Engineering Selection Cannot Rely Solely on Leaderboards
Real-world selection involves at least five types of metrics: task accuracy, process quality, cost, latency, and stability. Cost includes not just the per-call fee but also retry, manual review, and failure recovery costs. Latency must distinguish between first-token latency and complete-result latency. Stability requires observing consistency across multiple runs of the same problem, degradation under long contexts, and performance with anomalous inputs. Different deployment forms also affect user experience; interactive endpoints focus more on response speed and interpretability, while batch processing endpoints prioritize throughput and consistency. It is advisable to build a small-scale but highly covered regression suite, stratified by problem type, difficulty, length, and domain, with fixed grading scripts. When a candidate model is updated, rerun the regression and record accuracy, expert scores, average effective steps, failure type distribution, and cost. Below is a pseudocode example:
function evaluate(candidates, suites, graders):results = []for model in candidates:for suite in suites:outputs = run_batch(model, suite.cases)auto_score = graders[suite.type].auto_score(outputs, suite.references)expert_score = sample_expert_review(outputs, suite.sample_rate)results.append({model, suite,accuracy: auto_score.accuracy,verifiable_rate: auto_score.verifiable_rate,expert_quality: expert_score.mean,effective_steps: measure_steps(outputs),cost: measure_cost(outputs),latency: measure_latency(outputs)})return aggregate_by_weight(results)
- Building a Reproducible Evaluation Pipeline
An evaluation pipeline should include question bank versions, prompt templates, scoring rules, sampling strategies, and result archiving. The question bank must avoid contamination, regularly adding new questions while retaining difficulty tags. Prompt templates should fix system constraints, output formats, and allowed tools. Scoring rules must distinguish states such as “correct answer but wrong process,” “reasonable process but wrong conclusion,” and “unverifiable.” Sampling strategies must ensure manual review covers open-ended and high-risk questions. If implemented on Baidu AI Cloud, you can use general computing, object storage, logging, and monitoring to host batch evaluation tasks, but the core remains the evaluation protocol itself, not the infrastructure. The final output should not just be a ranking table, but a selection report including applicable scenarios, failure modes, cost boundaries, and regression conclusions. This transforms the question of “how good is a model’s math capability” into an actionable, auditable, and continuously iterable engineering decision.
