How to Evaluate LLMs' Open-Ended Math Capabilities and Choose the Right Model

    As LLMs become widely used for solving complex problems, open-ended math capability has emerged as a key indicator of a model's overall reasoning ability. Unlike closed problems with a single fixed answer, open-ended math questions require a model to autonomously handle problem understanding, condition abstraction, method selection, derivation and proof, result verification, and explanation. Evaluating and selecting among models therefore calls for a reproducible, quantifiable, and extensible framework built around task distribution, scoring rules, and engineering cost. This article walks through the nature and difficulty of open-ended math problems, capability layering, problem-set construction, scoring dimensions, the combination of automated grading and manual spot checks, the sources of model differences, and scenario-based selection advice, offering a practical guide that teams can put to work.

    How to Evaluate LLMs’ Open-Ended Math Capabilities and Choose the Right Model

    As large models are increasingly used for solving complex problems, open-ended math capability has gradually become an important indicator of a model’s overall reasoning level. Unlike closed problems with a single standard answer and fixed steps, open-ended math problems usually require a model to independently complete problem understanding, condition abstraction, method selection, derivation and proof, result verification, and expression. Because of this, comparing models based on isolated examples or subjective impressions easily leads to bias.

    In engineering practice, what developers and enterprise users truly care about is often not “whether a model looks impressive on a single problem,” but “whether, under a real business distribution, the model can complete tasks reliably, control errors, reduce the cost of review, and deliver acceptable output within reasonable latency and cost.” This calls for a reproducible, quantifiable, and extensible evaluation framework built from an engineering-assessment perspective.

    This article covers the definition of open-ended math problems, evaluation challenges, capability layering, scoring dimensions, problem-set construction, error analysis, and selection recommendations, forming a practical evaluation guide.

    1. The Nature and Challenges of Open-Ended Math Problems

    Open-ended math problems differ from routine exercises that have a single standard answer and fixed steps. They typically have the following characteristics:

    • Conditions may be implicit: The problem does not state all premises directly, and the model must identify hidden assumptions on its own.
    • Solution paths are not unique: Different methods may all reach a result, but they differ in efficiency, rigor, and interpretability.
    • Reasoning chains are long: There are many intermediate derivation steps, and an error in any single step can lead to a wrong final answer.
    • Result forms are complex: An answer may not be a single number, but rather a proof, an interval, an expression, a construction process, or a counterexample.
    • Expression quality matters: Beyond a correct conclusion, the model must clearly explain its reasoning and applicable conditions.

    Together, these characteristics make open-ended math evaluation difficult. The main challenges include:

    1. Answer determination is difficult
      Math answers may contain symbolic expressions, proof processes, case discussions, or natural-language explanations, so they cannot be judged by simple string matching.

    2. The process matters more than the result
      In some cases, a model may give a seemingly correct conclusion while its reasoning contains jumps, circular arguments, or missed boundary conditions. Such output is unacceptable in rigorous scenarios.

    3. Stability varies significantly
      The same problem can produce inconsistent outputs under different sampling parameters, prompts, or context lengths. A single test can easily overestimate or underestimate capability by chance.

    4. Cost and quality must be balanced
      A longer reasoning process may improve performance on hard problems, but it also increases response time, invocation cost, and user wait time.

    Therefore, open-ended math capability evaluation should not rely on a small number of samples. Instead, it should be designed comprehensively around task distribution, scoring rules, and engineering cost.

    2. Evaluation Goals and Capability Layering

    Before starting an evaluation, you need to clarify the goal: does the current business need to assess research-grade math reasoning, engineering-calculation assistance, business data analysis, or math explanations in ordinary Q&A scenarios? Different goals correspond to different evaluation priorities.

    It is recommended to divide math capability into multiple layers so you can build a graduated problem set.

    2.1 Basic Operations and Rule Application

    This layer mainly assesses the model’s grasp of basic math syntax and routine rules, including:

    • Arithmetic;
    • Algebraic simplification;
    • Unit conversion;
    • Applying common formulas;
    • Judging common function properties.

    Although this layer looks simple, it is very important in real systems. Errors in basic operations directly affect a user’s judgment of the system’s reliability.

    2.2 Standard Derivation and Multi-Step Problem Solving

    This layer requires the model to complete several reasoning steps, such as:

    • Solving equations;
    • Analyzing inequalities;
    • Computing probabilities;
    • Judging function monotonicity;
    • Deriving geometric relationships;
    • Modeling common optimization problems.

    These tasks usually have relatively clear evaluation criteria and form an important part of regression testing and automated evaluation.

    2.3 Open Modeling and Strategy Selection

    This layer assesses the model’s ability to abstract a mathematical model from a natural-language problem. For example:

    • Turning a business problem into an optimization model;
    • Identifying the key variables and constraints in a problem;
    • Choosing among multiple feasible methods;
    • Making reasonable assumptions for problems with insufficient conditions;
    • Comparing multiple solutions.

    Such problems are harder to grade automatically, but they better reflect a model’s comprehensive capability.

    2.4 High-Complexity Frontier Problems

    This layer may involve:

    • Rigorous proofs;
    • Complex combinatorial constructions;
    • Cross-domain math reasoning;
    • Long-chain symbolic derivations;
    • Hard problems that require repeated verification.

    These problems are often not suitable as the sole evaluation standard, because samples are few, judgment is difficult, and chance plays a large role. However, they have reference value for assessing a model’s ceiling and complex reasoning ability.

    Different models may perform very differently across layers. Some models are stable in standard derivation but tend to miss constraints in open modeling; some have longer reasoning chains, achieve higher coverage on hard problems, but may be overly verbose on simple ones. Therefore, selection must be based on the real task distribution, not just the results on extreme-difficulty problems.

    3. Building a Problem Set: From Single Impressions to Statistical Evaluation

    The credibility of open-ended math evaluation depends largely on problem-set design. It is recommended to build a layered problem set that covers different difficulty levels, question types, and error patterns.

    3.1 Basic Problems

    Basic problems are used to quickly verify whether a model has fundamental math ability. For example:

    • Solving simple equations;
    • Comparing numerical values;
    • Applying common formulas;
    • Converting units;
    • Computing basic probabilities.

    Although basic problems are low in difficulty, there should be enough of them, because they reflect the model’s stability in high-frequency scenarios.

    3.2 Standard Problems

    Standard problems cover mainstream question types and are suitable for regression testing. Recommended coverage includes:

    • Algebra and functions;
    • Geometry and trigonometry;
    • Probability and statistics;
    • Sequences and limits;
    • Basic optimization;
    • Simple combinatorial counting.

    Standard problems usually have reference answers or clear scoring points and are suitable for building automated evaluation pipelines.

    3.3 Open Problems

    Open problems assess the model’s performance when conditions are incomplete, solutions are non-unique, and modeling and verification are required. For example:

    • Building a cost model from a business description;
    • Proposing a feasible plan under given constraints;
    • Proving a proposition or explaining why it cannot be proven;
    • Providing and explaining a counterexample;
    • Comparing results across multiple scenarios.

    Open problems do not necessarily require a single answer, but they should provide scoring points or an acceptable answer range.

    3.4 Trap Problems

    Trap problems specifically assess whether a model tends to overlook key conditions. Common traps include:

    • Domain restrictions;
    • Inconsistent units;
    • Boundary conditions;
    • Misjudged sample spaces;
    • Independence assumptions for probability events;
    • Conditions for exchanging limits;
    • Parameter value ranges;
    • Feasibility constraints on the optimal solution.

    Trap problems can reveal whether a model has a habit of producing reasoning that “looks reasonable but is actually wrong.”

    3.5 Business Problems

    Business problems are closer to enterprise implementation scenarios, such as:

    • Capacity planning;
    • Cost estimation;
    • Risk-probability analysis;
    • Resource allocation;
    • Data-validation rules;
    • Metric-anomaly detection.

    The value of business problems lies in testing whether a model can turn math capability into usable results, not just complete abstract problems.

    3.6 Suggested Problem-Set Structure

    A fairly complete problem set can have a structure like this:

    1. Problem ID: math-open-001
    2. Type: Proof problem
    3. Difficulty: High
    4. Input: Proof of a proposition under given conditions
    5. Scoring points:
    6. 1. Correctly identify the given conditions
    7. 2. Construct the key intermediate conclusion
    8. 3. Derivation contains no obvious leaps
    9. 4. Conclusion is explicit
    10. 5. Does not cite theorems that were not provided
    11. Automated check: No
    12. Manual review: Yes

    For each problem, try to record its source, difficulty label, applicable scenario, reference answer, scoring points, common errors, and expected output length. This metadata helps with error attribution and model comparison later.

    4. Scoring Dimensions and Metric Design

    Open-ended math evaluation cannot look only at the final answer. It is recommended to adopt a multi-dimensional scoring system that comprehensively evaluates correctness, completeness, rigor, efficiency, verifiability, and expression quality.

    4.1 Correctness

    Correctness is the most basic metric, but it can be broken down further:

    • Whether the final result is correct;
    • Whether the key steps are correct;
    • Whether the proof is closed (forms a complete loop);
    • Whether the counterexample is valid;
    • Whether case discussion is complete.

    For numerical problems, results can be required to fall within an allowed error range; for proof problems, the key reasoning nodes need to be checked.

    4.2 Completeness

    Completeness assesses whether the model covers all necessary cases. For example:

    • Whether it discusses parameter value ranges;
    • Whether it handles boundary values;
    • Whether it explains special cases;
    • Whether it lists all feasible solutions;
    • Whether it omits constraints.

    Many errors are not a wrong final conclusion but solving only part of the problem.

    4.3 Rigor

    Rigor focuses on whether there are logical problems in the reasoning process, including:

    • Skipping steps;
    • Circular arguments;
    • Assuming unproven premises;
    • Misusing theorems;
    • Ignoring applicable conditions;
    • Substituting approximate values for rigorous derivation.

    In high-risk scenarios such as finance, engineering, and medical assistance, rigor usually matters more than surface-level expression.

    4.4 Efficiency

    The efficiency metric measures whether the model completes a task with a reasonable number of steps. Common observation points include:

    • Whether there is repeated computation;
    • Whether there is verbose, irrelevant content;
    • Whether it chooses an inefficient method;
    • Whether it over-reasons on simple problems;
    • Whether it repeatedly loops without converging on hard problems.

    Efficiency does not necessarily determine quality directly, but it affects engineering cost and user experience.

    4.5 Verifiability

    Verifiability refers to whether the model’s output is easy for humans or tools to review. For example:

    • Whether it lists intermediate variables;
    • Whether it gives key formulas;
    • Whether it states assumptions;
    • Whether it gives computable intermediate results;
    • Whether it explains uncertain items.

    Highly verifiable output is easier to bring into an engineering pipeline, because it can be further checked, corrected, and audited.

    4.6 Expression Quality

    Expression quality includes:

    • Whether the conclusion is clear;
    • Whether terminology is accurate;
    • Whether the structure is well organized;
    • Whether it is easy for people without a math background to understand;
    • Whether it avoids ambiguity.

    In scenarios such as math Q&A, customer-service explanations, and business-analysis reports, expression quality significantly affects user trust.

    4.7 Example Composite Scoring Formula

    You can use a weighted scoring approach:

    1. Composite score =
    2. 0.35 × Correctness
    3. + 0.20 × Completeness
    4. + 0.20 × Rigor
    5. + 0.10 × Efficiency
    6. + 0.10 × Verifiability
    7. + 0.05 × Expression quality

    The weights are not fixed and should be adjusted to business goals:

    • High-risk, rigor-focused scenarios: raise the weights of rigor and verifiability;
    • User-experience scenarios: raise the weight of expression quality;
    • High-concurrency services: raise the weights of efficiency and cost;
    • Complex-analysis scenarios: raise the weights of correctness and completeness.

    5. Combining Automated Grading and Manual Spot Checks

    Open-ended math evaluation usually cannot be fully automated and requires a combination of approaches.

    5.1 Automated Grading

    Suitable for problems whose answers can be standardized, such as:

    • Numerical problems;
    • Multiple-choice questions;
    • Fill-in-the-blank questions;
    • Problems that can be converted into standard expressions.

    Automated grading is fast and suitable for large-scale regression testing.

    5.2 Rule-Based Validation

    Rules can be used to check for common errors, such as:

    • Whether units are consistent;
    • Whether a result falls outside a reasonable range;
    • Whether division by zero occurs;
    • Whether boundary conditions are ignored;
    • Whether the output is blank or obviously irrelevant.

    Rule-based validation cannot replace human judgment, but it can filter out a large volume of low-quality outputs early on.

    5.3 Multi-Model Cross-Evaluation

    You can use multiple models to cross-evaluate against scoring points. Note that multi-model cross-evaluation has preference bias and should not be the sole basis for a final conclusion. It is more suitable as an initial filter or auxiliary scoring.

    5.4 Manual Spot Checks

    For proof problems, open problems, disputed problems, and high-risk problems, manual review should be set up. Manual spot checks can record results by error type, which helps with later optimization.

    Suggested sampling strategies include:

    • Random sampling: observe overall performance;
    • Full review of hard problems: control major risks;
    • Focused review of boundary problems: uncover hidden defects;
    • Review of user-reported problems: align with real-scenario issues.

    5.5 Example Evaluation Workflow

    An executable evaluation workflow looks like this:

    1. Input: problem set, model-calling interface, scoring rules
    2. Output: composite score, error distribution, manual-review list
    3. For each problem, execute:
    4. Call the model to generate an answer
    5. Record response time, output length, retry count
    6. Run automated validation
    7. If automated validation can judge, record the score
    8. If automated validation cannot judge, add it to the manual-review queue
    9. Aggregation phase:
    10. Compute accuracy by question type
    11. Compute distribution by error type
    12. Compute the composite score
    13. Output the manual-review priority list

    6. Chain-of-Thought Complexity Is Not the Capability Ceiling

    In practice, a common misconception is that the longer the chain of thought and the more complex the reasoning steps, the stronger the model. In fact, a long chain of thought may bring higher coverage on complex problems, but it may also introduce more accumulated errors.

    When evaluating, focus on the following types of issues:

    • Over-reasoning on simple problems: If basic problems are also drawn out very long, it may increase cost without benefit.
    • Whether hard problems converge stably: Some models produce very long output on hard problems but repeat themselves and fail to reach a reliable conclusion.
    • Whether errors concentrate in a particular stage: For example, symbolic derivation, case discussion, numerical calculation, or theorem usage.
    • Whether acceptable quality is reached within a limited budget: Engineering systems usually cannot extend reasoning time indefinitely.

    Therefore, evaluation metrics should not look only at the single best output, but should also track:

    • Average accuracy;
    • First-pass accuracy;
    • Consistency rate across multiple samples;
    • Average response time;
    • Average output length;
    • Retry count;
    • Manual-review ratio;
    • Cost per invocation;
    • User adoption rate.

    For engineering implementation, model selection is not about choosing the “absolutely strongest,” but about choosing the “best cost-performance under the target distribution.”

    7. Analyzing the Sources of Model Capability Differences

    Differences in how models perform on open-ended math tasks usually come from several factors.

    7.1 Training-Data Coverage

    High-quality math text, proof materials, problem analyses, and structured reasoning data affect a model’s baseline understanding of math tasks.

    7.2 Reasoning Strategy

    Whether a model can decompose problems, backtrack to check, and express itself in a structured way directly affects performance on complex problems. For example:

    • Whether it can break a problem into sub-problems;
    • Whether it can recognize whether the current path is viable;
    • Whether it can adjust its method after an error;
    • Whether it can proactively check boundary conditions.

    7.3 Context Utilization

    The model needs to correctly use the conditions in the problem statement and avoid missing restrictions. Common manifestations include:

    • Ignoring implicit constraints such as “positive integer”;
    • Treating an approximate value as an exact solution;
    • Failing to distinguish conditional probability from joint probability;
    • Incompletely discussing parameter ranges.

    7.4 Tool Collaboration

    In open-ended math problems, some steps can be completed with external validation capabilities, such as:

    • Numerical computation;
    • Expression simplification;
    • Code verification;
    • Retrieving definitions;
    • Checking formula references.

    Systems with tool-collaboration ability usually achieve more stable results in high-reliability scenarios.

    7.5 Output Standardization

    If a model’s output is hard to structure, downstream processing costs will be very high. Therefore, evaluation should also focus on:

    • Whether the conclusion is easy to extract;
    • Whether it is easy to review in segments;
    • Whether it supports structured-field output;
    • Whether it clearly states missing variables when conditions are insufficient.

    8. Enterprise Selection Advice: Matching Models to Scenarios

    Different business scenarios have different math-capability requirements, so selection should start from the task.

    8.1 High-Risk Financial or Engineering Scenarios

    These scenarios place high demands on rigor and verifiability. Recommended practices:

    • Use a hard-problem set for focused testing;
    • Introduce code validation or rule-based validation;
    • Manually review key conclusions;
    • Use multi-model cross-validation;
    • Build a mechanism to intercept anomalous outputs.

    You cannot rely solely on a model’s natural-language output as the basis for final decisions.

    8.2 Cost-Sensitive Q&A Scenarios

    For high-frequency, low-risk Q&A, focus on:

    • Response speed;
    • Cost per invocation;
    • Stability;
    • Baseline accuracy;
    • Cache-hit capability;
    • Rule/template supplementation capability.

    These scenarios do not necessarily need the strongest reasoning model, but they do need a stable, controllable, and scalable solution.

    8.3 Complex-Analysis Scenarios

    Complex-analysis scenarios can tolerate higher latency and focus on assessing:

    • Correctness on complex problems;
    • Stability of long-chain reasoning;
    • Condition-modeling capability;
    • Ability to compare multiple solution methods;
    • Quality of result explanation.

    Such scenarios are suitable for introducing multi-round verification and human review.

    8.4 Content Generation and Explanation Scenarios

    If the model needs to translate math results into business language, focus on evaluating:

    • Clarity of expression;
    • Accuracy of terminology;
    • Structural completeness;
    • Whether it avoids exaggeration;
    • Whether it can flag uncertain items.

    A math explanation must not only be correct, but also let users understand and trust it.

    9. Implementation Practice Workflow

    A reusable evaluation workflow can include the following steps:

    1. Clarify the goal
      Define the math task to be evaluated, the acceptable error, and the risk level.

    2. Build the problem set
      Layer it by difficulty, question type, and business scenario, covering basic problems, standard problems, open problems, and trap problems.

    3. Design the scoring rules
      Determine the automated metrics, manual scoring points, and weights.

    4. Establish a baseline
      Record the performance of the current model, rule-based solution, or manual solution.

    5. Run in batch
      Keep parameters such as prompt, temperature, context length, and maximum output length consistent.

    6. Attribute errors
      Classify errors as arithmetic errors, conceptual errors, logic gaps, missing conditions, unclear expression, tool-call failures, and so on.

    7. Optimize the strategy
      Adjust prompts, introduce validation, add tools, or switch models based on error type.

    8. Regression-test the release
      Repeat testing on the core problem set to prevent capability regression.

    9. Monitor in production
      Sample real requests, and monitor error rates, user feedback, and anomalous inputs.

    10. Prompt Design Essentials

    Prompts significantly affect performance on open-ended math tasks. It is recommended that prompts include a clear reasoning structure.

    10.1 General Math Task Prompt

    1. You are a math reasoning assistant. Please complete the following steps:
    2. 1. List the known conditions and implicit assumptions;
    3. 2. Explain the solving approach;
    4. 3. Derive step by step, avoiding leaps;
    5. 4. Verify the key results;
    6. 5. Give the final conclusion and flag uncertain items.

    10.2 Proof-Problem Prompt

    1. Do not cite theorems that were not given; if you use a theorem, state its applicable conditions.
    2. Each derivation step must follow from the previous step or from the given conditions.
    3. Give an explicit final conclusion: "proof holds / does not hold / insufficient conditions."

    10.3 Prompt for Insufficient Conditions

    1. If the data is insufficient, output a list of missing variables and give an acceptable estimation range.

    10.4 Structured-Output Prompt

    1. Please output in the following format:
    2. - Known conditions
    3. - Key assumptions
    4. - Derivation steps
    5. - Final result
    6. - Verification method
    7. - Uncertainty notes

    Structured prompts help reduce the cost of parsing output and make manual review easier.

    11. Common Evaluation Pitfalls

    Common pitfalls in open-ended math capability evaluation include:

    1. Looking only at cases, not the distribution
      An occasional impressive answer does not represent overall capability.

    2. Evaluating only hard problems, not basic ones
      This can lead to frequent basic errors that hurt the daily experience.

    3. Ignoring version differences
      Changes to the model, prompt, or parameters all affect results, so evaluation conditions must be recorded.

    4. Over-relying on text similarity
      Open-ended math answers cannot be judged by simple string matching.

    5. Not recording error types
      Without error classification, it is hard to tell whether a problem comes from the model, the prompt, or the task design.

    6. Ignoring cost and latency
      Evaluation conclusions must consider engineering feasibility; otherwise they are hard to implement.

    7. Focusing only on a single output
      A high-variance model may perform very well or very poorly in a single test, so multiple samples should be used to measure stability.

    12. Conclusion

    Evaluating a large model’s open-ended math capability cannot stop at subjective impressions or a single leaderboard. A more reasonable approach is to build a layered problem set around real tasks, design multi-dimensional metrics covering correctness, completeness, rigor, efficiency, and verifiability, and combine automated evaluation with manual spot checks.

    For enterprise implementation, you should choose models and strategies that match the business’s risk, cost, and latency requirements. High-risk scenarios emphasize rigor and verification, low-cost scenarios emphasize stability and efficiency, and complex-analysis scenarios emphasize long-chain reasoning and structured expression. Only through continuous regression testing, error attribution, and workflow optimization can you form a stable, reusable, and extensible system for evaluating math reasoning capability.