The Multilingual Math Reasoning Gap: Model Capability or Language Resource Divide

    Many assume that an AI model's mathematical capability remains consistent regardless of the input language. However, real-world testing reveals a significant performance drop when switching from high-resource to low-resource languages. This article explores the root causes of this multilingual math reasoning gap, including data imbalances, tokenization issues, and evaluation contamination. It also provides practical strategies for developers to distinguish language comprehension from reasoning, conduct fair evaluations, and build robust multilingual math applications through structured workflows and language-specific monitoring.

    Many people assume that once a model masters mathematical principles, it should provide similar answers whether asked in English, Chinese, Marathi, or any other language. Reality tells a different story: a model’s “mathematical capability” is often significantly influenced by the input language.

    A research team translated a set of classic elementary school math problems into 15 languages and tested them across nearly 50 mainstream models. Results showed that while the accuracy for the same problem was around 83% in English, it could drop to roughly 30% in Marathi. The problem itself and the model’s parameter size remained unchanged; only the language carrying the question varied.

    This indicates that the gap among multilingual models is not merely a question of “knowing a language.” It is more likely a systematic difference caused by training data scale, mathematical expression habits, tokenization methods, reasoning sample distribution, and evaluation contamination. For development teams serving global users, evaluating a model solely based on its English test scores can easily lead to overestimating its true cross-lingual reasoning capability.

    1. Distinguishing Language Capability from Reasoning Capability

    When tackling a math problem, a model typically executes several consecutive steps:

    1. Understanding entities, quantities, and relationships in the problem.
    2. Converting natural language into an internal structured representation.
    3. Selecting the appropriate calculation method.
    4. Executing addition, subtraction, multiplication, division, or multi-step reasoning.
    5. Reorganizing the result into a language the user can understand.

    A deviation in any part of this chain can lead to an incorrect final answer.

    For instance, an expression like “Xiao Ming has some apples, and then buys three times the original amount” uses completely different word orders, quantity modifiers, and unit expressions across different languages. For high-resource languages, a model has seen massive amounts of similar sentence structures, math Q&As, and reasoning examples during training, making it easier to establish stable mappings. For languages with less training data, the model might not lack calculation skills, but rather deviate during problem comprehension, quantity identification, or intermediate step generation.

    Therefore, errors in test results cannot be simply categorized as “insufficient math capability.” A more accurate statement is: the model lacks the overall capability to complete mathematical reasoning tasks in a specific language.

    2. Why Different Languages Cause Huge Disparities

    1. Highly Imbalanced Training Data Distribution

    Large models are typically trained on massive amounts of public text, but the volume of content across different languages on the internet varies drastically. High-resource languages possess more textbooks, forum Q&As, technical documentation, math explanations, and program code, whereas high-quality digital content for low-resource languages is relatively limited.

    This imbalance directly impacts the capabilities a model learns. A model needs not only to have “seen the language” but also to have seen sufficient math problems, solving processes, and error-correction samples. If a language has abundant general text but scarce math education content, the model might perform naturally in daily conversations yet clearly lose points in mathematical reasoning.

    2. Mathematical Expressions Are Not Perfectly Cross-Lingual

    Mathematics itself has a relatively unified symbol system, but problems are usually expressed through natural language. Different languages may vary in the following aspects:

    • Combination methods of numbers and units differ;
    • Plurals, case changes, or honorific forms increase comprehension complexity;
    • Writing habits for decimals, fractions, and dates are inconsistent;
    • The expression order for multiplication, proportions, and comparative relationships differs;
    • The usage of negative sentences, conditional clauses, and omission structures varies.

    When training data is insufficient, a model might easily misinterpret syntactic structures as quantitative relationships or ignore constraints, leading to a scenario where it seems to “know how to calculate” but actually misunderstands the problem.

    3. Tokenization Amplifies Difficulties for Low-Resource Languages

    Language models do not directly read “words”; they split text into a series of subwords or token units. Certain languages with rich morphological changes, unclear word boundaries, or low frequency in training data might be split into longer token sequences.

    This causes two issues. First, a math problem of the same length may consume different amounts of context space in different languages. Second, when key terms and quantitative expressions are split too finely, it becomes harder for the model to stably recognize their semantics. For multi-step math problems, the extra sequence length can also increase the attention burden during the reasoning process.

    4. Language Bias in Reasoning Samples

    Many math training samples are first generated in a high-resource language and then translated into other languages. While this expands coverage, it does not necessarily preserve natural expression habits. Machine-translated problems might be grammatically correct but do not align with the narrative style commonly used by local users.

    More importantly, the model might learn the mapping of “problem type to solution” in high-resource languages without adequately learning the corresponding reasoning chains in other languages. Consequently, the model can translate a foreign language problem into a familiar language before solving it, but accuracy drops when it must directly analyze and answer in the original language.

    3. Why Traditional Multilingual Evaluation Is Unfair

    Different Questions Make Scores Incomparable

    If different languages use different test questions, the final score is simultaneously affected by both language and problem difficulty. A higher score in one language might simply mean its corresponding questions were easier; a lower score in another language might mean the problem types were more complex.

    A fair approach is to establish a “parallel test set”: the same original problem is professionally translated into multiple languages while maintaining the problem structure, numbers, conditions, and answers. Only then does language become the primary variable, making the test results comparable.

    Translation Quality Affects Test Conclusions

    Word-for-word translation does not equal equivalent translation. A qualified multilingual math evaluation requires at least the following checks:

    • Numbers, units, and operational relationships remain consistent;
    • Conditions, negations, and comparative relationships are not omitted;
    • The local language expression is natural and unambiguous;
    • Translations are reviewed by personnel familiar with math education scenarios;
    • Upon back-translation, the core semantics still align with the original problem.

    If the translation itself changes the problem’s meaning, the test measures translation quality rather than model capability.

    Data Contamination Inflates Model Scores

    Math test sets that have been publicly available for years may have already appeared in the model’s training materials. A correct answer does not necessarily mean the model performed on-the-fly reasoning; it might have memorized the problem, the answer, or a fixed explanation.

    To reduce contamination, several methods can be adopted:

    1. Use recently constructed, unpublished questions;
    2. Programmatically alter characters, numbers, and scenarios in the problems;
    3. Keep the problem structure unchanged but replace entities and values;
    4. Simultaneously evaluate the original problem, rewritten problems, and cross-lingual versions;
    5. Observe whether the model can explain the solving process rather than just looking at the final answer.

    Programmatic rewriting is particularly important. If the model only memorizes templates, slight numerical changes will often cause significant fluctuations. If the model truly understands the operational relationships, its performance on rewritten problems should remain relatively stable.

    4. How to Determine if the Problem is Language or Reasoning

    To locate the source of errors, layered experiments can be designed rather than running a single Q&A test.

    Layer 1: Direct Answering in Original Language

    Let the model understand the problem using the target language and provide the answer in that language. This is the test closest to real user scenarios.

    Layer 2: Answering After Translation

    First, translate the problem into a language where the model performs better, then ask the model to solve it. If this result is significantly higher than direct answering in the original language, it indicates the main bottleneck lies in language comprehension or problem mapping, not entirely in mathematical reasoning.

    Layer 3: Answering with Structured Input

    Organize the numbers, units, relationships, and conditions from the problem into structured fields, then let the model calculate. If performance improves significantly with structured input, it indicates that natural language parsing is the primary error source.

    Layer 4: Calculation Task Only

    Convert the problem into a clear mathematical expression and separately evaluate the model’s calculation capability. If the calculation is correct but the natural language problem is wrong, the issue is concentrated in the semantic comprehension phase.

    These results can be represented as a simple error decomposition:

    Total Error Rate ≈ Language Comprehension Error + Structure Mapping Error + Reasoning Error + Output Format Error

    This is not a strict statistical identity, but it helps engineering teams establish a troubleshooting mindset rather than blaming all issues on model size.

    5. Improvement Strategies for Multilingual Applications

    1. Build Language-Balanced Math Datasets

    Training data should not only pursue total volume but also focus on problem type coverage for each language. The dataset should at least include:

    • Single-step and multi-step arithmetic problems;
    • Proportions, percentages, time, and unit conversions;
    • Conditional judgments and simple logic problems;
    • Different difficulty levels;
    • Localized expressions and real educational scenarios;
    • Correct answers, incorrect answers, and reasons for errors.

    Meanwhile, avoid generating all low-resource language data automatically from the same template. Otherwise, the model might learn fixed sentence patterns rather than general reasoning capabilities.

    2. Introduce Intermediate Representations in Reasoning

    For business systems, the model can first be required to extract structured information before performing calculations. For example:

    1. Problem: A small shop sold a certain number of items in the morning, and sold twice as many in the afternoon. A total of 36 items were sold.
    2. Structured Information:
    3. Morning quantity = x
    4. Afternoon quantity = 2x
    5. Total quantity = 36
    6. Equation: x + 2x = 36

    This approach separates “understanding language” from “executing calculations,” making it easier to check for errors. If the extracted structured information is incorrect, the system can request a re-parse. If the structure is correct, a dedicated calculation module can complete the operation.

    3. Use Cross-Lingual Consistency Checks

    The same problem can be solved by the model in the original language, another high-resource language, or a structured format, and the results compared. If multiple paths yield different answers, the system should flag it as low confidence rather than directly returning one of the results.

    Note that cross-lingual voting cannot be considered absolutely correct. Different language versions might share the same model bias, making it more suitable as a risk detection mechanism rather than definitive proof.

    4. Optimize Prompts, but Don’t Rely on Them Solely

    Explicitly requiring the model to “analyze step-by-step,” “check units,” or “confirm the final answer format” usually helps reduce careless errors. However, prompts cannot compensate for the fundamental lack of training data. For low-resource languages, more effective methods often include:

    • Providing a small number of high-quality examples in that language;
    • Specifying explanations for key terms and units;
    • Requiring the model to restate the problem before calculating;
    • Allowing the model to use structured intermediate results;
    • Adding manual review or programmatic verification for high-risk issues.

    5. Establish Language-Specific Monitoring Metrics

    Post-deployment monitoring should not only track overall accuracy but also statistics by language:

    • Problem comprehension accuracy;
    • Numerical calculation accuracy;
    • Final answer accuracy;
    • Average output length;
    • Rejection and anomaly rates;
    • Manual correction ratio;
    • Performance differences across problem types.

    If only the global average is monitored, the high volume of requests from English users might mask severe issues experienced by users of other languages.

    6. Risks to Consider for Enterprise Deployment

    Insufficient multilingual math capability impacts more than just test scores. In scenarios like educational tutoring, financial Q&A, customer service billing, inventory management, and public services, a seemingly simple numerical error can lead to actual losses.

    Therefore, tasks involving amounts, contracts, medical dosages, taxes, or resource allocation should not rely entirely on language models for direct calculation. A more robust architecture is:

    1. The model is responsible for understanding natural language;
    2. The structured module is responsible for extracting variables and constraints;
    3. The calculation component is responsible for executing deterministic operations;
    4. The verification module is responsible for checking units, ranges, and consistency;
    5. The model is responsible for translating the result into the user’s desired language.

    This division of labor—“language model for understanding, deterministic program for calculation”—is typically more reliable than simply expanding prompts.

    Conclusion: True Multilingual Capability Must Be Built Independently

    The performance gap between English and Marathi cannot be simply understood as one language being “more suitable for AI.” A more accurate explanation is that the model has acquired different scales, qualities, and types of training resources across different languages.

    Therefore, when evaluating multilingual models, one should not only ask “does it know this language” but also:

    • Has it seen sufficient math data in this language?
    • Can it accurately identify quantitative relationships?
    • Has it genuinely completed reasoning rather than memorized the answer?
    • Does the translated problem maintain semantic equivalence?
    • Does the system have cross-lingual verification and fallback mechanisms?

    Only by combining language fairness, data quality, contamination control, error decomposition, and engineering verification can we accurately judge a model’s true capabilities. For developers, the greatest takeaway is not to blindly pursue larger models, but to establish equally rigorous testing, training, and deployment assurance systems for different languages.