Quantifying and Closing the Multilingual Capability Gap in Large Language Models

    A common assumption in multilingual LLM deployment is that strong English performance will transfer to other languages. This article explains why that assumption often fails, how to measure language gaps with isomorphic templates, paired evaluation, and contamination controls, and how to interpret results through retention, token inflation, error taxonomies, and equivalent model scale—then converge the gap with data, representation, training, inference, and regression practices.

    Quantifying and Closing the Multilingual Capability Gap in Large Language Models

    1. The Problem Is Not Only “Whether the Translation Is Accurate”

    A common misjudgment in multilingual large language model deployment is that strong English capability will naturally “come along” for other languages. Reality often works differently. On the same elementary math problem, a model may exceed roughly 80% accuracy when the question is asked in English, then drop to around 30% when the question is switched to Marathi. The change is not only a vocabulary substitution. It is the combined effect of data coverage, tokenization efficiency, instruction understanding, units and numeric expression, how reasoning chains are organized, and answer format across languages.

    A language gap therefore has at least three sources:

    1. Model capability differences: pretraining corpora, instruction-tuning data, and preference data are uneven in how much of each language they contain and how natural that language is.
    2. Interface and representation differences: the same meaning may consume more tokens in another language, and localized forms of numbers, units, and names can interfere with computation.
    3. Evaluation construction differences: if languages use different items, different difficulty, or different translation quality, score gaps cannot be attributed directly to the model.

    To answer where the gap comes from, whether it can be predicted, and how to narrow it, the first step is not more prompt tuning. It is a reproducible, paired, and auditable yardstick.

    2. Four Controls for Fair Evaluation

    1. Semantic control: the same capability must be isomorphic

    Languages should not each receive a “roughly similar” item set. Variable relationships, operation steps, the number of distractors, and answer form must stay identical. For example, “someone has some items; after giving away a stated amount, how many remain” must keep the same quantities and reasoning depth in every language.

    2. Contamination control: isolate by template, not only by item deduplication

    Public static test sets are easily absorbed into training data. A more robust approach is dynamically generated template instances, with both a public sample set and an internal holdout set. The holdout should preserve not only specific items but also template families. Otherwise a model may never have seen the original item and still have memorized the answer pattern of that template.

    3. Statistical control: use paired comparisons, not absolute scores alone

    For each instance, record English and the target language under the same model and the same run configuration. Compute the paired accuracy difference, relative retention, confidence intervals, and sample size. Reporting only “60 for the target language and 80 for English” is not enough, because item-set difficulty may differ. A paired difference is closer to the language factor itself.

    4. Error control: separate language-understanding errors from mathematical reasoning errors

    Error analysis should at least distinguish: misunderstanding the problem, number recognition errors, unit-conversion errors, calculation errors, broken reasoning chains, answer-format mismatches, and ambiguity caused by unnatural wording in the target language. Only after these errors are separated can you decide whether to add data, fix tokenization, improve instructions, or change the evaluation.

    3. Symbolic Templates: Same Skeleton, Reorderable Surface

    A symbolic template is not a fixed item that is simply translated. It abstracts the item into a semantic structure with variables. Variables include person names, objects, quantities, units, relational words, and the final question. A renderer then generates natural language from a language-specific template. The same template can produce thousands of different surface forms in each language while the mathematical structure stays fixed.

    A simplified template definition looks like this:

    1. template_id: give_take_v1
    2. semantic_expr: "A has W of I; after giving D to B, how many remain"
    3. constraints:
    4. W: 5..50
    5. D: 1..W
    6. I: a set of countable entities matched to grade level
    7. languages:
    8. en: "A has W I. After giving D I to B, how many are left?"
    9. target: "rendered by a native reviewer with the same meaning; numeric relations unchanged"
    10. answer_fn: W - D
    11. quality_gate:
    12. - the problem is unambiguous in the native language
    13. - the answer unit is explicit
    14. - no extra cultural-knowledge burden specific to the target language
    15. - no dependence on a particular layout or symbol set

    The point is that the template skeleton is shared while the surface text can be localized. Names, objects, units, numeric ranges, and question wording can change, but variable constraints and the answer function must stay locked. This “same structure, different surface” approach fits especially well for math, logic, unit conversion, and similar capabilities.

    To make evaluation more credible, add reverse perturbations: swap the order of “given” and “remaining,” rename quantities, insert irrelevant distractor sentences, and use localized number forms or different unit expressions. If a model does well on the unperturbed version but drops sharply under light linguistic perturbation, the issue is weak stability, not a pure lack of mathematical ability.

    4. Data Contamination: Absolute Cleanliness Cannot Be Proven, but Risk Can Be Reduced Continuously

    The main risk of static item sets is memorization. Engineering controls include:

    • Split train-visible data and evaluation holdouts by template family, so the holdout is not merely the same problem with different numbers.
    • Dynamic seed generation: each evaluation uses new variable combinations and records the seed and hash.
    • Time-slice audits: check whether a test instance could have appeared before the model’s training data cutoff.
    • Near-duplicate detection: fingerprint item text, answer patterns, and solution steps.
    • Counterfactual tests: change numbers, order, units, and irrelevant descriptions, and see whether the answer follows the mathematical relation.
    • Multi-round sampling: reduce the effect of lucky guesses and record the output distribution.

    Contamination cannot be fully ruled out by surface deduplication alone. An evaluation report should describe the risk-control process rather than claim that the set is absolutely uncontaminated.

    5. Quantifying the Gap: From Score Differences to Equivalent Model Scale

    Basic metrics include:

    • Paired accuracy difference: the accuracy gap between English and the target language on the same instance.
    • Relative retention: target-language accuracy divided by English accuracy.
    • Token inflation: token consumption of the same item in the target language relative to English.
    • Stability: variance across repeated samples, prompt templates, and variable combinations.
    • Error spectrum: the share of errors from language understanding, reasoning, calculation, format, and related types.

    When several model scales share the same test suite, you can go further and estimate an “equivalent model scale.” First fit how English accuracy changes with model scale. Then find a smaller scale whose English performance is close to the current model’s performance in the target language. Conceptually:

    1. Fit English: Acc_en = f(model scale)
    2. For target language l: given base model scale S0, measure Acc_l(S0)
    3. Find S_eq such that f(S_eq) ≈ Acc_l(S0)
    4. Equivalent scale ratio R_l = S_eq / S0

    This ratio translates a language gap into a more intuitive sense of compute or parameter count. Degradation on some languages can feel like dropping the model by one size tier. It depends on the same evaluation set, multiple model scales, and confidence intervals. A single score difference should not be extrapolated into a universal conclusion. Engineering reports should publish an interval, not only an isolated number.

    6. Predicting Language Gaps in Advance

    Language gaps are not entirely unpredictable. A “language capability profile” can estimate high-risk languages at modest cost. Common features include:

    • Distance from high-resource languages in writing system, morphology, and word order;
    • Token inflation of the same item across languages;
    • Coverage and naturalness of that language in training corpora and instruction data;
    • Complexity of localized numbers, units, dates, and mathematical symbols;
    • Semantic ambiguity rate after translation or localization;
    • Auxiliary signals on mixed-language text, cross-lingual retrieval, and terminology consistency;
    • Reasoning depth of the target capability. The more a task depends on multi-step reasoning, the more representation differences can be amplified.

    In practice, a small probe set can train a predictor that maps these features to the expected gap versus English and outputs a confidence interval. Prediction does not replace formal evaluation. It prioritizes native review, data supplementation, and prompt adaptation for high-risk languages.

    7. Five Engineering Paths to Narrow the Gap

    1. Data: add native-language reasoning, not only answers

    Adding target-language question-answer pairs alone does not necessarily improve mathematical reasoning. It is more effective to add step-by-step reasoning, explanations, error correction, and unit-conversion samples in the target language, reviewed by native speakers, so translationese does not create a new distribution shift.

    2. Representation: reduce token inflation and friction in numeric expression

    Normalize numbers, units, mathematical symbols, names, and common terms. Check whether tokenization and detokenization break number boundaries. Test consistency between localized and standard number forms. Higher token use crowds the context window, and multi-step reasoning is more likely to drop conditions midway.

    3. Training: add a cross-lingual consistency objective

    Build parallel semantic samples and require consistent intermediate steps and final answers across languages. Use rejection sampling on incorrect outputs and a verifier to filter reasoning chains. The goal is not that the model “can translate the problem,” but that it executes the same computation in different languages.

    4. Inference: combine language adaptation with verifiable tools

    For high-risk languages, use language detection followed by dedicated prompt templates, glossaries, and tool calls. For verifiable tasks such as math and unit conversion, external calculation or rule checks reduce format errors and arithmetic drift. Tool output still needs to be written back in the target language with a consistent answer format.

    5. Evaluation: put multilingual metrics into continuous regression

    Fix paired difference, relative retention, token inflation, and the error spectrum as regression metrics. After each model or prompt change, run the holdout template set on high-risk languages. Set thresholds so English gains do not hide silent regressions in target languages.

    8. A Pipeline You Can Actually Run

    1. 1. Define a capability matrix: math, logic, unit conversion, format following, and similar skills.
    2. 2. Build a symbolic template library: multiple template families for each capability.
    3. 3. Native review: naturalness, ambiguity, grade fit, and cultural neutrality.
    4. 4. Dynamically generate parallel instances: the same variable combination rendered into multiple languages.
    5. 5. Build evaluation packages: item text, answer, variable hash, template ID, difficulty label.
    6. 6. Paired execution: multiple languages and multiple samples on the same instance.
    7. 7. Statistics: accuracy difference, relative retention, confidence intervals, error types.
    8. 8. Version trace: record model, prompt, template, seed, and decoding configuration.
    9. 9. Feedback loop: by error spectrum, add data, adjust tokenization, change prompts, or add tools.

    One paired evaluation can be summarized as:

    1. for case in paired_holdout_set:
    2. for lang in languages:
    3. prompt = render(case.template, lang, case.vars)
    4. answers = sample(model, prompt, k)
    5. score[lang][case] = mean(verify(answers, case.answer))
    6. delta = paired_mean(score["en"], score["target_lang"])
    7. retention = mean(score["target_lang"] / max(score["en"], eps))
    8. report(delta, retention, confidence_interval, error_taxonomy)

    9. Common Mistakes

    • Mistake 1: direct translation is enough for a fair evaluation. Translation quality, naturalness, and cultural context all change difficulty.
    • Mistake 2: looking only at the overall average. Averages can hide severe drops on specific capabilities in specific languages.
    • Mistake 3: assuming a larger model will automatically erase the gap. Scale may lift many languages at once without shrinking the gap proportionally.
    • Mistake 4: strong English means strong multilingual performance. A high English score shows high-resource language capability, not deployment quality for low-resource languages.
    • Mistake 5: ignoring answer format. Changes in numbers, units, and punctuation in the target language can mark a correct reasoning chain as wrong.

    10. Conclusion

    The gap between English and Marathi looks like a pair of scores. Underneath it is a systematic capability difference across data, representation, instruction, and evaluation. To make that an manageable engineering problem, first build isomorphic item sets with symbolic templates; obtain credible differences with paired statistics and contamination control; explain the sources with equivalent model scale, token inflation, and an error spectrum; then converge the gap item by item through native-language data, representation optimization, cross-lingual consistency, and continuous regression.

    For developers, the important part is not memorizing a particular score. It is a evaluation and governance loop that keeps running as models iterate, languages are added, and systems go to production. Only when a language gap moves from “it feels a bit worse” to metrics that can be measured, attributed, and regressed does multilingual capability become engineering-controllable.