Cross-Lingual Math Evaluation for LLMs: Uncovering Capability Gaps and Enabling Governance

    When evaluating multilingual model capabilities, a common misconception is to read every wrong answer as "the language is too hard." In practice, when the same model performs steadily on English math problems but its accuracy drops sharply in other languages, the cause is rarely language alone. It is usually the combined effect of training distribution, evaluation design, data contamination, parameter capacity, and the inference pipeline. This article explains why identical problems appear "harder" across languages, outlines the structural blind spots in traditional multilingual math benchmarks, and shows how symbolic templates, contamination isolation, layered metrics, and error attribution can turn cross-lingual gaps into a measurable, explainable, and optimizable engineering problem. Readers will find a practical, closed-loop path from evaluation to governance, including template design, dynamic problem generation, unified pipelines, and post-processing checks.

    Cross-Lingual Math Evaluation: Uncovering Model Capability Gaps and Enabling Governance

    When we discuss multilingual model capabilities, a common misconception is to interpret a wrong answer directly as “the language is too hard.” In reality, when the same model performs reliably on English math problems but its accuracy drops sharply under certain language settings, the problem is often more than the language itself. It is usually the combined result of the model’s training distribution, evaluation design, data contamination, parameter capacity, and the inference pipeline.

    A representative observation is this: for the same elementary-level, multi-step math problem, the model is noticeably more likely to answer correctly when prompted in English than when prompted in another language. For example, in some multilingual benchmarks, accuracy under English may reach a high level, while switching to certain low-resource languages can cause accuracy to fall off significantly. This gap is easy to misread as a “language difference,” but a more reasonable explanation is that a model’s effective capability differs across languages, and an evaluation system that is not strictly aligned will further amplify this difference.

    1. Why Does the Same Problem Become “Harder” in Another Language?

    On the surface, the logical structure of a math problem does not change: given quantities, perform operations, produce a result. But from the model’s processing perspective, switching languages changes the input distribution, token forms, numeric expressions, syntactic structure, and how context dependencies are handled.

    Take a problem like “someone has a number of items, gives away some, and we need the remaining count.” In English, this may appear as a relatively concise sentence where the relationships among numbers, units, actions, and objects are easy to align. In other languages, however, there may be more complex morphological changes, numeral structures, word-order differences, or honorific systems. If the model’s training corpus in that language is insufficient, it may deviate even at the comprehension stage, despite the underlying math logic being simple.

    So-called “language gaps” are, more precisely, capability gaps under different language distributions. Language is only the surface variable; what truly affects performance is whether the model has seen enough, sufficiently diverse, and sufficiently clean target-language samples.

    2. Three Blind Spots in Traditional Multilingual Math Evaluation

    To explain and narrow this gap, we first need a fair measuring stick. However, traditional multilingual math evaluations often suffer from several structural problems.

    2.1 Inconsistent Problems Make Scores Incomparable

    If English users take one set of problems and another language’s users take a different set, the two sets likely differ in difficulty, question type, length, and phrasing. Comparing accuracy in this case is like grading two students with two different exam papers: the score difference does not directly reflect a real difference in capability.

    2.2 Data Contamination Makes High Scores Look Like “Memorization”

    Many public math benchmarks, years after release, have already spread widely into model training corpora. A high score on an evaluation set does not necessarily indicate stable reasoning ability; it may simply reflect that the model has seen the original questions or highly similar ones. For multilingual scenarios, contamination is even more complex: even if the original-language questions did not leak, translated versions, community discussions, and derivative problems can also enter training data indirectly.

    2.3 A Single Accuracy Metric Hides the Real Error Structure

    A model can get a problem wrong for entirely different reasons: some fail at number extraction, some at unit understanding, some at a break in multi-step reasoning, and others at language-generation degradation. If you only look at overall accuracy, you cannot tell whether the problem lies in language understanding, mathematical computation, or output formatting.

    3. Symbolic Templates: Making Every Problem Controllable, Reproducible, and Comparable

    To bypass the “inconsistent problems” and “data contamination” difficulties, an effective approach is to generate problems using symbolic templates. A symbolic template does not literally translate one fixed problem into many languages; instead, it abstracts a problem into a structure with replaceable variables.

    For example, a word problem can be abstracted into the following form:

    1. {person} has {count1} {item}, gives away {count2}, how many are left?

    You can then maintain different template expressions across languages:

    1. Chinese template: {person}有 {count1} 个 {item},送出 {count2} 个,还剩多少个?
    2. English template: {person} has {count1} {item} and gives away {count2}. How many are left?
    3. Other-language templates: rewritten by localization specialists according to grammatical conventions, rather than mechanically translated.

    Then substitute parameters to generate a large number of problems:

    1. function render(template, person, count1, item, count2):
    2. return template.replace("{person}", person)
    3. .replace("{count1}", count1)
    4. .replace("{item}", item)
    5. .replace("{count2}", count2)

    This approach offers three advantages.

    First, difficulty is controllable. Developers can control problem difficulty by adjusting numeric ranges, the number of operation steps, and semantic complexity, rather than relying on a fixed exam paper.

    Second, problems are comparable in parallel. The same logical structure can be instantiated across languages with identical or near-identical difficulty parameters, reducing interference from difficulty differences.

    Third, it can be updated dynamically. Each evaluation can generate fresh problem instances, lowering the risk of a model scoring high by recalling answers.

    However, symbolic templates are not simple string replacement. Different languages have different requirements for quantifiers, noun plurals, verb forms, word order, and honorifics. If a template is only mechanically translated, it may introduce unnatural expressions and distort the evaluation results. Therefore, a high-quality template needs to combine language rules, localization review, and human spot-checks.

    4. Data Contamination and Memory Effects: A Variable That Must Be Isolated

    In multilingual math evaluation, data contamination is a more subtle problem than it appears. It does not necessarily originate from the evaluation set itself; it can also come from problem variants, back-translations, web reposts, example data, or similar structures in training corpora.

    To reduce the impact of contamination, you can take several approaches.

    4.1 Strict Separation of Training and Evaluation Sets

    Evaluation questions should never enter training data in any form. For questions that have been public for many years, it is best to avoid using them directly as core evaluation criteria, or to parameterize and rewrite them.

    4.2 Use Dynamically Generated Problem Pools

    By combining symbolic templates with random numbers, random roles, and random contexts, you can continuously generate new problems. Even if a model has seen a template, it may still be unable to predict the specific answer.

    4.3 Perform Near-Duplicate Detection

    Some problems look different on the surface but are highly similar in logic and phrasing. Text similarity, structural similarity, or semantic similarity checks can filter out large numbers of homogeneous questions in the evaluation set.

    4.4 Build a Private Evaluation Set

    For enterprise-internal or long-term research scenarios, you can keep a pool of questions that is not public, not used for training, and not distributed externally, reserved for final acceptance. This reduces external contamination.

    4.5 Watch for Translation Backflow

    In multilingual evaluation, once an English question is translated into multiple languages, if the model encounters those translated versions in subsequent training, cross-lingual memory leakage can occur. It is therefore important to maintain version records and prevent evaluation questions from flowing into public corpora over the long term.

    5. Parameter Scale: An Amplifier of the Gap, Not the Only Cause

    One point worth expanding from the original discussion is this: the performance gap on certain languages can look like the distance between a “32B model” and a “10B model.” Although metaphorical, this phrasing highlights an important fact: a model’s effective capability is not consistent across languages.

    A model with around 30 billion parameters theoretically has greater capacity and stronger potential expressive power. But if its training corpus in a given language is insufficient, those parameters may not fully translate into effective capability in that language. In other words, a larger model is not equally strong in every language.

    Conversely, a smaller model with clean training data, balanced language coverage, and thorough instruction alignment may also perform stably on certain tasks. Therefore, cross-lingual gaps cannot simply be attributed to “language difficulty,” nor can they be solved solely by scaling up parameters.

    The impact of parameter scale is mainly reflected in three aspects:

    1. Capacity ceiling: Larger models can generally hold more knowledge, but this requires sufficiently high-quality training signals.
    2. Distribution coverage: If a language makes up a very small share of the training data, even a large model may perform poorly on it.
    3. Reasoning stability: Small models are more prone to formatting errors, broken steps, or number confusion on complex multi-step problems, but large models can likewise be affected by prompts and language templates.

    A more reasonable approach is therefore to observe parameter scale, data mix, template quality, and evaluation contamination together, rather than attributing a low score in a given language directly to the language itself.

    6. How to Build an Interpretable Multilingual Math Evaluation?

    A mature cross-lingual math evaluation system should not output only a single overall accuracy. It should provide layered, traceable, and reproducible results.

    6.1 A Unified Evaluation Pipeline

    It is recommended to bring all languages, all models, and all question versions into a single pipeline: problem generation, language validation, model inference, answer parsing, error classification, and result aggregation. This avoids inconsistent standards caused by different teams evaluating independently.

    6.2 Layered Metrics

    Beyond overall accuracy, you should also track:

    • Accuracy per language;
    • Accuracy per difficulty level;
    • The effect of numeric length on accuracy;
    • The stage in the multi-step reasoning chain where errors occur;
    • Stability across different template versions;
    • The range of variation for the same model across multiple samplings.

    6.3 Labeling Error Types

    Structurally attributing errors enables categorization such as:

    Error Type Description
    Number extraction error The model misreads quantities or units
    Operation step error Errors in arithmetic operations or the order of multiple steps
    Semantic understanding error Misinterpreting people, items, actions, or references
    Format parsing error The answer cannot be reliably extracted
    Language-generation degradation Incomplete output, repetition, or deviation from the question
    Contamination-induced misjudgment Output relates to training memory rather than genuine reasoning

    With these error labels, a team can determine whether the problem lies in language ability, math ability, or the engineering pipeline.

    6.4 Establish Confidence Intervals and Repeated Sampling

    A single evaluation run is easily influenced by sampling variation. For key models, you should repeat inference multiple times at a fixed temperature and report the mean, variance, and worst-case performance. Especially for low-resource languages, variation may be more pronounced than for high-resource ones.

    7. From Evaluation to Governance: A Practical Engineering Path

    If an enterprise wants to validate a model’s multilingual math ability in real business scenarios, it can follow this closed loop.

    Step 1: Define the Business Scenario

    First clarify the evaluation goal. Is it elementary math, financial calculation, inventory reasoning, or form understanding? Different scenarios have different requirements for numbers, units, terminology, and multi-step reasoning.

    Step 2: Build a Language Template Library

    Establish templates for each target language and localize them through review. Templates should cover common operation types, such as addition, subtraction, ratios, unit prices, totals, remaining amounts, and average distribution.

    Step 3: Parameterize Problem Generation

    Generate problem instances using random seeds while preserving the generation parameters. This both ensures reproducibility and lowers the probability that problems get memorized.

    Step 4: Connect Model Evaluation

    Run different models and different language templates within a unified framework. This stage can be supported by general-purpose cloud capabilities: for example, using object storage to hold the problem pool, a message queue to schedule batch jobs, a container platform to run inference services, a logging service to record the invocation process, and monitoring and alerting to track failure rates.

    Step 5: Output a Multi-Dimensional Report

    The final report should not contain only a single score, but should include:

    1. Evaluation version number
    2. Problem template version
    3. List of languages
    4. Model version
    5. Parameter scale
    6. Temperature and sampling settings
    7. Overall accuracy
    8. Accuracy per language
    9. Accuracy per difficulty level
    10. Error type distribution
    11. Confidence interval

    Step 6: Form an Improvement Loop

    Take different measures based on error types. For example, if number extraction errors are frequent, optimize the prompt format or produce structured intermediate steps; if errors concentrate in low-resource languages, supplement with high-quality parallel corpora; if format parsing failures are frequent, strengthen output constraints and result validation.

    8. Practical Suggestions for Business Optimization

    During implementation, the following points are also worth noting.

    8.1 Do Not Look Only at the Final Score

    A model may have a close total score yet suffer serious weaknesses in certain languages, question types, or numeric ranges. If a production system serves multilingual users, focus first on the weakest area.

    8.2 Keep Few-Shot Examples Parallel

    If you use examples to prompt the model, the example language should match the test language, and the question structure, difficulty, and expression style should be aligned as closely as possible. Otherwise, the examples themselves may introduce bias.

    8.3 Add Validation for Critical Computational Tasks

    Math tasks do not have to rely entirely on free-form model generation; you can add rule-based validation after the output. For example, check whether the result is a non-negative integer, falls within a reasonable range, or satisfies basic conservation relationships. This post-processing can significantly reduce obvious errors.

    8.4 Measure Language Ability Separately

    To determine whether a model “cannot compute” or “did not understand,” you can design two types of questions: one that tests only language understanding, and one that tests only mathematical reasoning. Comparing them helps locate the source of the gap.

    8.5 Continuously Update the Evaluation Set

    Models keep iterating, and contamination keeps accumulating. The evaluation set needs periodic updates and should retain dynamic generation capability, rather than relying on a fixed set of questions over the long term.

    9. Conclusion: The Gap Lies Not in the Language Surface, but in Capability Distribution and Evaluation Design

    In multilingual math evaluation, the accuracy gap for the same model across languages is not merely a difference in language difficulty. It is the combined result of training coverage, data quality, model capacity, template design, and evaluation contamination. Simply attributing the gap to “language” tends to obscure the real problem; treating high scores directly as capability can also be misled by data contamination.

    A more reliable path is to build a controllable problem pool with symbolic templates, use dynamic generation to reduce contamination risk, ensure comparability with a unified evaluation pipeline, locate problems with layered metrics and error attribution, and then continuously govern through data supplementation, prompt optimization, output validation, and model iteration. In this way, cross-lingual gaps are no longer misread as linguistic destiny, but become a measurable, explainable, and optimizable technical problem.