Evaluating LLM Math Reasoning: Chain-of-Thought Quality and Model Selection Practices
For development teams selecting and deploying large language models, this article explains how to evaluate mathematical reasoning, inspect chain-of-thought quality, and control cost and latency in engineering. It covers evaluation protocol design, CoT inspection items, model capability grouping and selection, multi-model routing architecture, and risk controls. All numeric values are example parameters that should be calibrated with business data.
For development teams that need to select and deploy large language models, it is important to evaluate not only final answers but also the reasoning process. This article explains how to assess a model’s mathematical reasoning ability, how to inspect chain-of-thought quality, and how to control cost and latency in engineering. All numeric values in this article are example parameters and should be calibrated against your own business data.
1. What Is an Open-Ended Math Problem?
Open-ended math problems are problems without a fixed solution template. Standard problems usually have a unique answer, and their solutions can directly apply textbook methods. Open-ended problems require the model to choose its own modeling approach, propose intermediate assumptions, and verify whether those assumptions hold.
Common examples include:
- Given constraints, find the optimal bound of a function and provide a construction that attains the bound.
- Design a combinatorial counting scheme that satisfies multiple conditions and prove its completeness.
- Starting from a set of experimental data, choose an appropriate statistical model, derive conclusions, and explain the scope of applicability.
The difficulty is that intermediate steps may have multiple paths, and incorrect assumptions can be amplified in later steps. Evaluation therefore requires a separately designed protocol.
2. Designing the Evaluation Protocol
Before evaluation, clarify three things: how questions are stratified, how answers are scored, and how results are repeated.
Question stratification. We recommend three difficulty tiers, with at least 30 questions per tier, and labels for question type: calculation, proof, construction, or modeling. If the sample size is too small, the randomness of individual questions can hide real differences between models.
Scoring rules. A three-tier scoring scheme can be used: 1 point when both the answer and derivation are correct; 0.5 points when the answer is correct but the derivation has obvious skipped steps; 0 points when the answer is wrong. Numeric questions can be automatically verified by program. For proof questions, two reviewers should score independently; if the disagreement exceeds 1 point, a third person should arbitrate. Equivalent answer forms, such as 1/2 and 0.5, should count as correct but must be noted in the evaluation record.
Repeated runs. LLM outputs are stochastic. The same question should be run multiple times, and the pass rate and standard deviation should be reported. A single run is not sufficient to support model selection conclusions.
3. What to Check in Chain-of-Thought Quality
Chain-of-thought is the reasoning text a model produces before giving an answer. It has two uses: helping judge whether the answer is credible, and helping locate errors. Five items are worth checking:
- Logical closure: Whether each step’s conclusion depends on content already given earlier, and whether any condition is introduced from nowhere.
- Step density: The proportion of effective derivation steps in the total output. More irrelevant repetition and boilerplate lower the density.
- Loop detection: If the same proposition appears multiple times in the chain without new information, it should be judged as circular argument.
- Branch recovery: Whether abandoned assumptions are explicitly marked as invalid, rather than quietly used later.
- Conclusion consistency: Whether the final answer is consistent with the intermediate derivation.
Logical closure and conclusion consistency are hard to judge automatically; they require manual sampling or assistance from another model. Step density, loop detection, and branch recovery can be implemented with rules or simple text statistics.
4. Grouping Model Capabilities and Model Selection
Vendor materials often provide only total scores, without breakdowns. For model selection, compare at least three categories of metrics:
- Math capability: Tiered pass rates on the open-ended question set, and the type distribution of failed questions.
- Reasoning form: Average CoT length, step density, and loop rate.
- General capability: Instruction following, multi-turn context retention, and whether the refusal policy is reasonable.
There are usually trade-offs among these three categories. Models with higher accuracy on difficult open-ended questions often produce longer outputs, with higher latency and per-call cost. Models with shorter reasoning chains are easier to debug, but may miss key branches on questions that require multi-step construction. Models that rely on very long CoT can benefit on some questions, but they also have a higher probability of repeated verification, invalid branches, and hallucinations, which increases manual inspection cost.
Therefore, in high-difficulty math scenarios, some models can be one of the candidate options; whether to adopt them should be determined by the team’s own evaluation set. Decisions should not be based only on a single leaderboard or vendor description.
5. Engineering Architecture Practices
Multi-model routing is a common approach. The parameters below are example values and should be adjusted before launch based on evaluation data and cost budget.
Demand routing. Deploy a lightweight classifier at the application entry point to output a difficulty score between 0 and 1. Requests below 0.4 go to a general model; requests between 0.4 and 0.7 go to a medium model; requests above 0.7 go to a heavy reasoning model. The classifier should reach an F1 value of about 0.85 on a labeled validation set. If it does not, fix the classifier first, then adjust the thresholds.
Selection metrics. For each candidate model, record four data points: open-ended accuracy, average output tokens per question, P95 latency, and average cost per correct answer. Compare the four data points in the same table to avoid looking only at accuracy.
Token budget. Set an output cap for each request type, for example 2000 tokens for general requests and 8000 tokens for heavy math requests. Interrupt generation when approaching the cap, and record
finish_reasonaslength.Latency target. For interactive scenarios, the time to first token should not exceed 2 seconds, and the P95 for a full response should not exceed 30 seconds. After a timeout, fall back to a medium model, or return verified partial results first and tell the user that the result is incomplete.
Log fields. Each request should record at least:
request_id,model_name,difficulty_score,input_tokens,output_tokens,reasoning_tokens,latency_ms,finish_reason,loop_flag, andanswer_verified. Sensitive raw user data should not be stored in logs; de-identify it first.CoT pruning. Deploy middleware in the business backend and trigger interruption when:
- The same proposition appears 3 or more times.
- Five consecutive output segments contain no new symbols, values, or citations.
- The output has exceeded 80% of the budget and still has not produced a conclusion.
After a trigger, the middleware keeps verified intermediate conclusions, asks the model to continue deriving from those conclusions, and forbids repeating completed verification. Retry at most 2 times; if it still fails, route the request to manual review or return “cannot determine.”
Cloud platform container services, logging services, and monitoring and alerting can host the above architecture. For example, relevant components from Baidu AI Cloud can be used for container deployment and log collection; this is only an example. Comparable services from other cloud vendors can implement the same architecture. When choosing a platform, compare pricing, availability zones, compliance requirements, and integration cost with existing systems.
6. Risk Considerations
Long chain-of-thought increases cost and latency and may produce plausible but incorrect derivations. A correct mathematical derivation does not mean the conclusion can be directly used for high-risk decisions. In scenarios such as financial quantitative analysis, research conclusions, and resource allocation, model output should be manually reviewed.
