Reproducible Math Reasoning Evaluation: Chain-of-Thought Length and Production Monitoring

    This article explains how to establish a verifiable evaluation methodology for large language model math reasoning. It covers standardizing evaluation criteria, comparing the effects of different chain-of-thought (CoT) lengths, estimating operational costs, and implementing post-launch monitoring. It provides actionable templates, metric definitions, and alerting thresholds for engineering teams.

    The mathematical reasoning capability of large language models (LLMs) is often used as a metric to gauge a model’s upper limit. However, public conclusions frequently lack the necessary evaluation context. For the same model, switching the problem set or the scoring rules can significantly alter the results. This article does not provide a fixed model ranking. Instead, it explains how to establish a verifiable evaluation methodology, compare the effects of chain-of-thought (CoT) length, estimate costs, and implement monitoring after deployment. The numerical values provided are suggested starting points and should be calibrated using your own business data.

    1. Standardize the Evaluation Criteria

    Evaluation conclusions are only valid under specific conditions. Before publishing or citing any results, ensure you record at least the information in the table below.

    Dimension Information to Record Recording Requirements
    Model Version Specific version number, deployment region, whether fine-tuned Consistent with runtime logs
    Problem Set Source, quantity, difficulty tier Attach the problem list and SHA-256 hash
    Evaluation Date Run date and server-side version Accurate to the day
    Sampling Parameters temperature, top_p, max output tokens Keep consistent across each run
    Metrics Accuracy, convergence rate, average tokens, P95 latency, loop rate Specify the calculation formula
    Sample Size Repetitions per problem and total number of problems Note if there are repeated runs

    The calculation methods for each metric are as follows:

    • Accuracy: The number of runs with a correct final answer divided by the total number of runs.
    • Convergence Rate: The number of runs that provide a clear conclusion within the step limit divided by the total number of runs.
    • Average Tokens: The sum of output tokens across all runs divided by the total number of runs. Input tokens should be tracked separately.
    • P95 Latency: Sort all end-to-end latencies from smallest to largest, and take the value at the position ceil(0.95 × N), where N is the number of runs.
    • Loop Rate: The number of runs detected as loops divided by the total number of runs. The detection rules for loops are detailed in Section 5.
    • Signal-to-Noise Ratio: The number of steps actually referenced by subsequent derivations divided by the total number of steps. A low ratio indicates many steps are irrelevant to the conclusion.
    • Drift Rate: The number of steps that deviate from the problem constraints divided by the total number of steps.

    The industry often categorizes models into different tiers. Such categorization is only comparable when using the same problem set and the same scoring rules. Without the information in the table above, tier conclusions cannot be verified and should only be treated as a reference.

    2. Problem Types and Failure Mode Recording

    Open-ended math problems do not have fixed solution methods or a single solution path. Common problem types include:

    • Multi-step Deduction: Given premises, deduce a conclusion through numerous intermediate steps.
    • Counterexample Construction: Determine if a proposition holds. Provide a proof if it holds, or a counterexample if it does not.
    • State Space Search: Find a path that meets the conditions among multiple branches, such as combinatorial counting or game analysis.
    • Self-Correction: Discover an earlier error during the reasoning process and correct it.

    During evaluation, record the failure modes rather than just right or wrong. The table below provides judgment examples.

    Failure Mode Judgment Criteria Judgment Example
    Step Skipping Missing key intermediate conclusions or relying on unstated premises Step 3 concludes n is even, but no basis is provided earlier
    Drift Changing the meaning of variables or conditions in the problem The problem defines n as a positive integer, but n is treated as a negative number during derivation
    Loop The same conclusion or step repeats The derivation in Step 7 is identical to Step 4
    Fabricated Basis Citing a non-existent theorem or using an incorrect formula Citing a “combinatorial identity” that does not hold
    Inconsistency The conclusion contradicts earlier derivations The earlier text proves x > 0, but the conclusion sets x = -1

    During manual review, it is recommended to fill out a record for each run with the following fields: Problem ID, Difficulty Tier, Correct/Incorrect, Failure Mode (multiple selections allowed), Problematic Step Number, and Reviewer. Reviewers should evaluate without knowing the model version to reduce bias.

    3. Chain-of-Thought Length Experiment Design

    3.1 Tiering and Step Counting

    Many discussions simply categorize CoT into “concise” and “verbose,” but this classification lacks a unified definition. It is recommended to tier by step count and compare the effects across tiers.

    • Short Chain: No more than 5 steps.
    • Medium Chain: 6 to 15 steps.
    • Long Chain: More than 15 steps.

    Step counting rules are as follows. Each numbered derivation item counts as one step. Each paragraph containing a verifiable deduction also counts as one step. Sentences that purely restate the problem, transitional phrases, and the final conclusion sentence do not count as steps. Counting should be performed by a script, with manual spot-checks.

    3.2 Prompt Template

    Constrain the maximum step count via the prompt so the same model can run under different tiers. The template is below, where {max_steps} is replaced by 5, 15, or an unrestricted value depending on the tier.

    1. You are a math reasoning assistant. Please answer the following question using no more than {max_steps} numbered steps.
    2. Each step should contain only one verifiable deduction. After the deduction, write "Final Answer: " followed by your conclusion on a new line.
    3. If you cannot reach a conclusion within the step limit, write "Not converged" and explain the completed parts.
    4. Question: {question}

    3.3 Sampling Parameters and Repetitions

    • Sampling Parameters: Set temperature to 0.0 and top_p to 1.0. If the service provider does not support 0.0, use the minimum allowed value and note this in the report.
    • Max Output Tokens: Set to 2048 for short chains, 4096 for medium chains, and 8192 for long chains. Runs that reach the limit without finishing are recorded as “length” truncations.
    • Repetitions: Run each problem at least 5 times. When reporting statistics, report the distribution of accuracy rather than just a single result.
    • Manual Review Sampling: For each combination of model and tier, sample no fewer than 20 runs, or 10% of total runs, whichever is greater. Sampling should be stratified to cover each problem type. Two reviewers should evaluate independently; if the agreement rate falls below 90%, a third reviewer will arbitrate, and the judgment criteria should be discussed.

    3.4 Cost Estimation

    The cost of a single request can be estimated using the following formula:

    Single Cost ≈ (Input Token Count ÷ 1,000,000 × Input Unit Price) + (Output Token Count ÷ 1,000,000 × Output Unit Price)

    The unit price is calculated per million tokens. The currency, billing unit, and tax inclusion must be consistent with the service provider’s invoice. If the provider bills by other units, convert them to the price per million tokens before applying the formula.

    Example: A request inputs 2,000 tokens and outputs 3,000 tokens. The input unit price is $2 per million tokens, and the output unit price is $8 per million tokens. The single cost is (2000 ÷ 1,000,000 × 2) + (3000 ÷ 1,000,000 × 8) = 0.004 + 0.024 = $0.028.

    The table below shows a comparison using hypothetical data. Here, the input is fixed at 800 tokens, using the prices from the example above. The output token counts are hypothetical; actual values should be based on measured averages.

    Tier Hypothetical Output Tokens Single Cost ($) Total Cost for 100 Problems × 5 Runs ($)
    Short Chain 400 0.0048 2.40
    Medium Chain 1200 0.0112 5.60
    Long Chain 3000 0.0256 12.80

    Cost conclusions must be evaluated alongside accuracy. If the accuracy of the long chain is only marginally higher than the medium chain, but the cost is more than double, the medium chain is more suitable as the default tier.

    4. Private Evaluation Set Checklist

    Public problems may have entered the model’s training corpus, so teams should maintain a private evaluation set. Before launching an evaluation, check the following items one by one.

    Check Item Verification Method Pass Criteria
    Overlap with Public Corpus Compare 13-character continuous segments of the problem text Overlapping segments should not exceed 5%; replace the problem if exceeded
    Answer Verifiability Each problem has a standard answer or clear judging rule Judging rules are scriptable and unambiguous
    Difficulty Tiering Tier by manual annotation or baseline model accuracy At least 20 problems per tier, with the tiering basis recorded
    Annotation Consistency Two annotators independently label the data Agreement rate not less than 90%
    Version Management Calculate the SHA-256 hash of the problem set Increment version number after any modification and recalculate the hash
    Tuning Isolation Use a separate development set for prompt tuning The test set must not participate in any tuning
    Result Retention Save raw outputs and judging records Retain for at least 6 months for review

    5. Log Fields and Alerting Thresholds

    5.1 Log Fields

    Each request should write a structured log entry with the following fields:

    Field Name Type Description
    request_id String Unique request identifier
    timestamp Timestamp Request start time, UTC
    model_version String Specific model version number
    region String Deployment region
    eval_set_id String Problem set ID and hash value
    item_id String Problem ID
    chain_tier Enum short, medium, long
    temperature Float Sampling temperature
    top_p Float Sampling parameter
    max_output_tokens Integer Maximum output token count
    input_tokens Integer Input token count
    output_tokens Integer Output token count
    latency_ms Integer End-to-end latency in milliseconds
    finish_reason Enum stop, length, error
    step_count Integer Step count calculated per Section 3 rules
    converged Boolean Whether a conclusion was reached within the step limit
    loop_detected Boolean Whether a loop was detected
    answer_correct Boolean Whether the answer is correct, backfilled after manual review
    error_code String Server-side error code, empty if no error

    When logging raw outputs, data masking should be applied. If the problem or output contains personally identifiable information (PII), only retain the hash value.

    5.2 Loop Detection Rules

    Split the output into sentences using periods, line breaks, and semicolons, and trim leading and trailing whitespace. If the same sentence (completely identical after normalization) appears 3 or more times, or if the same paragraph repeats consecutively 2 or more times, it is judged as a loop. Normalization includes removing whitespace and unifying full-width and half-width characters.

    5.3 Alerting Thresholds

    The following thresholds are starting points. Before going live, calibrate using two weeks of baseline data. It is recommended to initially set the latency threshold to 1.5 times the baseline P95 latency.

    Metric Calculation Window Initial Threshold Action
    Loop Rate Rolling 1 hour Exceeds 5% Reduce max output tokens, enable loop truncation, and check recent prompt changes
    P95 Latency Rolling 15 minutes Exceeds 60 seconds Check rate limiting and queueing; switch to a backup deployment region if necessary
    Length Truncation Rate Rolling 1 hour Exceeds 10% Check tier settings; consider increasing max output tokens or lowering step limit
    Convergence Rate Daily Falls below 90% Review prompt template; sample and inspect non-converged runs
    Error Rate Rolling 15 minutes Exceeds 1% Review error code distribution; contact the service provider or roll back the version
    Single Cost Daily average Exceeds 1.5x baseline Check if output tokens grew abnormally; verify billing unit price

    Alerts should notify both the algorithm and platform leads simultaneously. The resolution of each alert should be recorded in an incident log to facilitate future threshold calibration.

    6. Scope of Application and Conclusion

    This methodology is applicable to the following scenarios: internal model selection, version regression testing, comparing before and after prompt or parameter adjustments, and cost budget estimation.

    This methodology is not applicable to the following scenarios: directly citing public benchmark rankings as a basis for selection; comparing results across different problem sets; or situations with insufficient sample sizes (e.g., fewer than 100 runs per tier) where stable conclusions are required. Discrepancies between public benchmarks and private evaluation sets are normal; you should prioritize your own evaluation set.

    Cost conclusions will change based on the following factors: service provider price adjustments, model version upgrades, and changes to CoT tier settings. If any of these occur, recalculate costs and review the comparison results.

    In summary, short chains typically have the lowest cost, but low cost does not mean greater accuracy. Tier selection should be determined by accuracy and convergence rate, not just step count. It is recommended to run the entire process with a small sample first to ensure logs and alerts work properly before scaling up the number of problems and repetitions.