Evaluating Math Reasoning in LLMs: Accuracy, Chain of Thought, and Cost

    Benchmarks alone cannot tell you how well a large language model handles mathematical reasoning—public leaderboard questions may already be in training data, and scores say little about performance on your own business problems. This article presents a reproducible evaluation method: define clear metrics first, compare models on your own test set, then decide whether longer chains of thought are worth it, with cost and compliance factored into the selection process.

    Public leaderboard scores are a poor proxy for real-world math reasoning ability. Leaderboard questions may already be part of a model’s training data, and even high scores reveal nothing about how a model performs on your own business problems. This article walks through a reproducible evaluation workflow: define metrics first, compare models on a test set you control, then decide whether longer chains of thought are justified—while factoring cost and compliance into the selection decision.

    1. The Bottom Line Up Front

    When evaluating mathematical reasoning, track four dimensions in parallel: accuracy, chain-of-thought efficiency (output length and step count), latency and cost, and output parsability. Keep them separate rather than collapsing them into a single score, because they often involve trade-offs. For example, a model that scores two percentage points higher on accuracy but doubles its output length may or may not be worth it—depending on your specific use case.

    2. Define Your Metrics Before You Evaluate

    Whether evaluation results are trustworthy depends on how clearly the metrics are defined. Here are five commonly used definitions.

    Accuracy: Every question needs an explicit grading standard. Questions with definitive answers can be graded with rules or numerical comparison. Proof problems should be scored independently by two reviewers, with a third arbitrator in case of disagreement. Aim for at least 100 samples per question category—with too few samples, a few percentage points of difference may just be random noise.

    Chain-of-thought length: Track input tokens, output tokens, and separately billed reasoning tokens. Looking at totals alone can hide problems, since long prompts also drive up cost.

    End-to-end latency: Record the time from request submission to complete response, reporting both P50 and P95 percentiles. Run the same batch of questions multiple times at similar times of day and under similar network conditions, so queueing delays aren’t mistaken for the model’s intrinsic speed.

    Logical drift rate: Sample the reasoning process for erroneous steps, internal contradictions, or content irrelevant to the question, and compute their share of the total samples. This metric requires manual annotation, and annotation guidelines should be documented before review begins.

    Parsability: Can the final answer be extracted programmatically by fixed rules? If your prompt requires answers in a fixed format, simply measure the proportion of format-compliant outputs. This metric directly affects the failure rate of automated pipelines.

    During evaluation, fix sampling parameters such as temperature and run each question multiple times to average out randomness, since most models produce stochastic output.

    3. Capability Tiers: A Testable Classification

    It’s common in the industry to sort models into three tiers: top-tier models capable of open-ended proofs and exploration; strong reasoning models that perform consistently on competition problems and engineering calculations; and baseline models that rely mainly on pattern matching. Note that this is a common observation, not a rigorous conclusion—“open-ended” and “consistent” have no universally accepted definitions.

    To use this framework for model selection, convert the tier criteria into measurable conditions, for example:

    • On an unreleased set of open-ended problems, accuracy reaches a threshold you define.
    • On standard calculation problems, accuracy and latency meet business requirements.
    • For proof problems, the manual review pass rate meets requirements.

    Thresholds should be set by you based on your own risk tolerance—don’t simply borrow someone else’s numbers. Also watch for data contamination: if test questions have leaked into training data, your accuracy measurements will be inflated.