When Will Domestic LLMs Reach the Frontier? Inference Efficiency Is the Key Variable

1. The Real Question: Not “Can It?” but “Is It Affordable and Fast?”

In recent years, domestic large models have steadily closed the gap with international frontier models on public leaderboards. Looking only at single-point success rates, the difference may seem marginal. But shift the lens from a one-dimensional score to a two-dimensional performance-versus-cost plane, and the picture changes immediately: on tasks close to real production—code generation, agentic workflows—many domestic flagship models land in the lower-right region relative to frontier models. Success rates are comparable or slightly lower, but each task consumes noticeably more inference tokens, and when converted to task-level cost, the gap can even invert.

This is the most worthwhile question in the current catch-up phase: how long will it take for domestic models to genuinely match the frontier in the sense of “effectiveness and efficiency both meeting the bar”? To answer it, we first need to understand two different technical paths.

2. Two Paths: Think Longer, or Think Better

Current approaches to improving reasoning model capability can be broadly grouped into two paths:

Path one: long thinking. Through test-time compute scaling, models are allowed to generate longer chains of thought, converting “thinking longer” into “answering better.” This path is relatively easy to make work in engineering terms: reinforcement learning during post-training typically rewards only the correctness of the final answer, and models readily learn the shortcut of “thinking a bit longer”—the marginal gain from extended thinking is often easier for the reward signal to capture than improvements in per-step decision quality.

Path two: efficient reasoning. International frontier flagship models exhibit another distinctive trait: high reasoning token efficiency. That is, under similar or even fewer thinking tokens, they achieve higher problem-solving rates, with greater “effective reasoning density” per token.

On the surface, the difference between the two paths looks like a stylistic divergence; in essence, it reflects a gap in base model quality and post-training refinement. Path one can trade “stacking duration” for scores, at the cost of growing token consumption and latency. Path two demands a solid pretraining foundation, clean data mixtures, and fine-grained reward signals so the model learns to “think right” rather than “think more.” Many domestic models currently follow path one in practice. Under catch-up pressure this is understandable—raising scores first is a reasonable staged strategy—but the cost becomes concentrated when deployments scale.

3. Why Reasoning Token Efficiency Directly Determines Deployment Cost

Task-level cost can be roughly decomposed as:

Task cost     = (input tokens + reasoning tokens + output tokens) × unit price
Daily cost    = task cost × daily task volume

Input and output tokens are determined by the task itself, with limited room for variation. The real biggest variable is reasoning tokens (the thinking process). For models on the long-thinking path, a single complex task can easily consume tens of thousands or even hundreds of thousands of thinking tokens, creating three pressures:

1. Cost pressure: in agentic scenarios where task volumes run into the thousands or tens of thousands, token consumption is amplified by several orders of magnitude;
2. Latency pressure: longer chains of thought mean longer time-to-first-token and overall response times, directly degrading the interactive experience;
3. Stability pressure: in extremely long chains of thought, early deviations are progressively amplified, and the probability of going off track or falling into self-loops rises with chain length.

A simplified estimation script makes the scale effect tangible:

# Example: cost comparison between two models at 5,000 tasks/day (illustrative values only)
models = {
    "Efficient reasoning model": {"avg_reasoning_tokens": 60_000,  "success_rate": 0.62},
    "Long-thinking model":      {"avg_reasoning_tokens": 150_000, "success_rate": 0.58},
}
price_per_1k = 0.02          # price per thousand tokens (illustrative)
tasks_per_day = 5_000
for name, m in models.items():
    tokens = m["avg_reasoning_tokens"] * tasks_per_day
    daily_cost = tokens / 1000 * price_per_1k
    cost_per_success = daily_cost / (m["success_rate"] * tasks_per_day)
    print(f"{name}: daily cost {daily_cost:,.0f}, per successful task {cost_per_success:.2f}")

Note the last line: calculated as cost per successful task, the long-thinking model’s disadvantage is further amplified—it not only spends more but also lacks a success-rate advantage. This also explains why, in some public evaluations dominated by agentic tasks, the average task cost of certain domestic models exceeds that of international frontier models: what falls short is not the score itself but the token bill behind the score.

4. How to Evaluate Reasoning Efficiency Rigorously

Many teams look only at leaderboard scores when selecting models—a classic one-dimensional decision. A more reasonable approach is to build the evaluation as a two-dimensional plane: the horizontal axis is average token consumption per task (or directly converted to cost), and the vertical axis is task success rate (or a composite performance score). The ideal position is the upper left (high performance, low consumption); frontier models currently occupy the upper-left region of the Pareto frontier, while many domestic models fall in the lower right. A minimal viable evaluation workflow looks like this:

def evaluate(model, tasks):
    total_tokens, total_cost, success = 0, 0.0, 0
    for task in tasks:
        trace = model.run(task)      # record token usage and final result
        total_tokens += trace.tokens
        total_cost   += trace.cost
        success      += trace.passed
    n = len(tasks)
    return {
        "success_rate":     success / n,
        "avg_tokens":       total_tokens / n,
        "cost_per_success": total_cost / max(success, 1),  # key metric
    }

Three engineering recommendations:

1. Use your own task distribution: the task composition of public benchmarks differs greatly from real business workloads, and cost structures drift accordingly—conclusions cannot be transferred directly;
2. Track token details separately: account for input, reasoning, and output separately. Many model services distinguish reasoning token tiers in usage metering and billing (a common industry practice; refer to the billing documentation of the service you use), and aggregated statistics mask the true cost driver;
3. Sample multiple times to observe variance: run the same task repeatedly; the stability issues of long-thinking models only become visible in a statistical sense.

5. The Gap Is Structural: Four Things to Fix to Catch Up

The gap in reasoning token efficiency is not a problem that can be “optimized away” at a single point; it is the outward manifestation of a set of structural factors:

1. Pretraining foundation. Reasoning density per token ultimately depends on how efficiently the base model compresses knowledge and reasoning patterns. Data quality, mixture ratios, and training sufficiency determine the model’s “starting level”; post-training can only amplify on top of it, not fill it in from nothing.
2. Reward design. If reinforcement learning rewards only the final answer without constraining or penalizing redundancy, loops, and无效 branches in the chain of thought, the model will systematically learn to “think more” rather than “think right.” Modeling process quality—for example, soft constraints on chain-of-thought length or penalties for redundant steps—is a key lever for improving efficiency.
3. Inference systems engineering. The same model can differ several-fold in throughput and cost across different inference stacks: speculative decoding, KV cache reuse, continuous batching, quantization and sparsification, long-context optimization—all are engineering measures that “cut costs without changing model weights” and require sustained investment.
4. Evaluation and iteration loop. Frontier labs’ internal evaluation systems are far more refined than public leaderboards and can iterate continuously on dimensions such as efficiency. If challengers optimize only against public leaderboards, they risk falling into the trap of “matching scores while falling behind on experience.”

6. How Long: Three Scenarios

Giving a precise timetable is not serious, but an analytical framework is possible. The pace of catch-up depends on the mix of two kinds of problems: engineering of proven paradigms (someone has shown the path works; reproduction cycles are measured in months) and the ability to define paradigms themselves (new methods, new efficiency frontiers; cycles measured in years). On this basis, three rough scenarios:

• Optimistic (1–2 years): base model quality takes a leap, combined with inference engineering gains, achieving parity in both performance and cost on mainstream tasks;
• Neutral (2–4 years): scores reach parity first, efficiency converges slowly, with usability sustained during the period through engineering optimization and price competition;
• Conservative (4+ years): frontier models continue to widen the gap on the efficiency dimension, and catch-up remains a moving target.

It must be emphasized: inference efficiency is a systemic gap, jointly determined by base model quality, reward design, and inference engineering—it is almost impossible to close in one single breakthrough. Therefore “preparing for a long catch-up” is not pessimism; it is the prerequisite for rational resource planning.

7. Action Checklist for Developers and Enterprises

Rather than agonizing over macro timelines, it is better to do the controllable things well:

1. Put “cost per successful task” into selection criteria. Any model review should report three numbers together: success rate, average tokens, and task-level cost—none can be omitted;
2. Build a private evaluation set and regression mechanism. A few dozen tasks close to real business, scored regularly with trends recorded, offer far more decision value than public leaderboards;
3. Design model routing at the architecture level. Route simple tasks to lightweight models and call heavy reasoning models only for tasks that truly require deep reasoning; some inference services provide thinking-budget-style parameters (such as maximum thinking length) that can be used generically to cap costs—refer to the documentation of the service you use;
4. Connect a unified observability system. Use general capabilities such as log services and monitoring/alerting offered by platforms like Baidu AI Cloud to bring token usage, cost, and success rate into a single dashboard, turning “inference efficiency” from a subjective impression into a measurable engineering metric;
5. Set stop-losses for long thinking. Set timeouts and token caps on individual calls within agent loops to prevent models from falling into thinking loops that blow up costs.

Conclusion

The answer to “how long until it catches up” depends largely on how “catch up” is defined. Looking only at leaderboard scores, the gap may be measured in months; requiring simultaneous parity in reasoning token efficiency, task-level cost, and response latency makes it a systemic marathon measured in years. Admitting this is no disgrace—it is precisely the starting point for formulating the right strategy: treat inference efficiency as a first-class metric across model R&D, inference engineering, and business model selection, and only then can the pace of catch-up possibly exceed the pace at which the frontier advances.