Base Models Enter the Era of Token Efficiency Competition
A new generation of foundation models shows that competition is shifting from intelligence per token to cost per task. This article breaks down the architecture behind sparse MoE and sparse attention, translates benchmark scores into token and dollar terms, and provides a practical playbook for task-level cost measurement, prompt caching, tiered deployment, and long-running agent governance.
Over the past month, a leading cloud vendor released a new generation of foundation model, and three numbers in the public disclosures deserve particular attention from developers: total parameters of roughly 2.4T with only about 71B activated per token; average token usage per attempt in a professional coding benchmark dropping from 289K to 151K; and average cost per task falling by nearly 60% (about 60% per official figures, roughly 58% in independent evaluations). Taken together, these point not toward “a bigger model” but toward an emerging industry consensus: the competitive axis for foundation models has shifted from intelligence per token to cost per task.
For enterprise teams building applications, the implications of this shift are direct. Over the past two years, application bills were driven mainly by call volume, and the usual optimizations were switching to smaller models or trimming parts of the pipeline. Now that model vendors are optimizing on a “cost per task” basis, application-side architectural assumptions need to adjust accordingly — model selection, prompt organization, agent loop step limits, and caching strategies all become engineering variables with measurable returns. This article explains the technical logic behind the shift and offers a reusable evaluation and governance methodology.
1. Architecture: Why “Huge Total Parameters” and “Cheap per Token” Can Coexist
The new generation follows a路线 that combines sparse Mixture-of-Experts (MoE) with sparse attention. Public materials describe a novel MoE architecture based on sparse attention, where only about 71B of the 2.4T total parameters are activated at inference, and the activated parameter count per token is roughly 25% lower than the previous generation.
Two easily confused concepts are worth separating here:
- Total parameters determine the capability ceiling. Larger expert capacity means the model can hold richer knowledge and skill parameters across domains, languages, and task distributions.
- Activated parameters determine inference cost. The number of parameters actually involved in the forward pass for each token directly drives compute consumption and latency.
The value of sparsification is decoupling the two: capacity sits in memory and storage, routed on demand, while compute is paid only for the expert subset each token actually needs. Sparse attention (approaches like QSA) solves the other half of the problem — in long-context scenarios, attention compute and memory grow super-linearly with sequence length, making “million-token contexts” nearly unusable in cost terms without sparsification. Combined, these mean “ultra-long context support” and “controllable per-token cost” are no longer mutually exclusive.
This also explains why “performance gains” and “cost reductions” are not contradictory. Performance gains come from higher information density per token; cost reductions come from lower activation volumes and cheaper cache hits. The two curves are driven by different mechanisms and can therefore move in the same direction.
2. Benchmarks: Translating “Scores” into “Tokens and Dollars”
Among the published evaluation results, the most informative is a set of numbers from a terminal-based professional coding benchmark (terminal-class tasks):
| Metric | Previous Generation | New Generation | Change |
|---|---|---|---|
| Average tokens per attempt | ~289K | ~151K | Down ~48% |
| Benchmark score | 37.9 | 48.5 | Up ~28% |
| Average cost per task | Baseline | — | Down ~58% |
This table is worth reading row by row.
The first row shows that fewer tokens produced a higher score. In agent-class tasks, token consumption is roughly equivalent to “number of thinking steps × context length.” A common failure mode of older models is repeated probing, re-reading files, and circling around the same step; once the chain grows long, the context fills with its own historical noise and errors accumulate. Token usage nearly halved while scores rose, indicating that what was reduced was mostly无效 round trips, not effective reasoning. What the industry calls improved “thinking efficiency” refers precisely to this.
The second row shows how hard the score improvement actually is. Terminal coding benchmarks require the model to complete build, debug, and test loops in a real command-line environment, with low fault tolerance and a large state space. Moving from 37.9 to 48.5 on such a benchmark is not an increment achievable through prompt tricks.
The third row converts the first two into financial terms. One key detail here: the measured 58% reduction is notable alongside the official ~60% figure — and the reduction comes largely from a substantial cut in cache-hit pricing, not merely from fewer tokens. In a typical agent trajectory, system prompts, tool definitions, long documents, and codebase retrieval results are repeatedly included in the context, and cache hit rates can reach very high proportions. A lower cache-hit price pushes the marginal cost of this repeated input close to zero.
The engineering implication is very concrete: cache-hit pricing is a direct input to prompt architecture design. If volatile content (timestamps, random IDs, user nicknames) is placed at the front of the prompt, the entire prefix cache is invalidated. The correct approach is to put stable system instructions, tool descriptions, and domain knowledge first — kept byte-level stable — and place session-specific content later.
3. How Enterprises Should Act: A Practical Evaluation and Governance Playbook
1. Measure at the Task Level, Not the Call Level
Most teams’ dashboards stop at the call level: QPS, token counts, per-request latency. But in agent scenarios, a single user request may trigger a dozen or more model calls, and what really determines experience and cost is task-level metrics:
# Cost attribution per task: convert "price per million tokens" into "cost per task"def cost_per_task(trace, price):agg = {"uncached_input": 0, "cached_input": 0, "output": 0}for step in trace: # every step of an agent is one model callu = step.usageagg["uncached_input"] += u.input_tokens - u.cached_tokensagg["cached_input"] += u.cached_tokensagg["output"] += u.output_tokenscost = (agg["uncached_input"] * price.input+ agg["cached_input"] * price.cache_hit # usually a discount off the input price+ agg["output"] * price.output)return cost, agg
With this function, you can compute the four numbers teams actually care about: task success rate, average cost per task, cache hit rate, and P95 end-to-end latency. The cache hit rate is the most easily overlooked yet most easily optimized metric — it is almost entirely determined by how prompts are organized.
2. Controlled A/B Evaluation
When comparing two foundation models, fix the following conditions, or the conclusions are invalid:
- The same toolset and tool descriptions;
- The same maximum loop step count and the same retry-on-failure policy;
- The same temperature and the same reasoning budget cap;
- The same task set and the same grading scripts.
Be especially wary of hidden bias from “the new model scores higher”: if the new model defaults to a longer thinking budget, score gains may come from compute spend rather than efficiency. Only by aligning budgets explicitly can you tell capability improvement from resource brute-forcing.
3. Tiered Deployment: Sensible Division of Labor Between Large and Small Models
Public information indicates that mid-size and small models in this series will be progressively open-sourced. Such tiering has a major impact on cost structure. A common division of labor:
- Cloud-hosted ultra-large models: handle complex planning, code generation, cross-document reasoning, and long-chain agent orchestration;
- Mid-size and small models: handle intent recognition, format conversion, summarization, classification, parameter extraction, and other high-frequency short tasks, with private deployment possible;
- Rules and small models as front-line filters: intercept requests that clearly require no reasoning before they reach the large model.
A pragmatic rule of thumb: if a step’s inputs and outputs can be fully specified, it probably should not be handled by the most expensive model.
4. Long Context: Use It, But Also Govern It
Native vision and million-token context are capabilities — not free ones. Three rules of thumb:
- Long context ≠ effective context. Middle-of-context information is commonly ignored; key constraints should be repeated in both the system prompt and task instructions;
- Context needs lifecycle management. Compress, summarize, and retrieve from historical trajectories rather than appending indefinitely;
- Visual input is cost-sensitive by resolution and page count. Document scenarios should run layout analysis and on-demand cropping first, rather than ingesting an entire manual.
5. Cost Optimization Checklist
| Technique | Main cost item affected | Implementation difficulty |
|---|---|---|
| Stable prefixes, volatile content last, to improve cache hits | Input (cache-hit price) | Low |
| Eliminating wasteful agent round trips, capping step counts | Total tokens | Medium |
| Structured output constraints to reduce retries | Input + output | Low |
| Offloading short tasks to smaller models | All | Medium |
| Context compression and retrieval-based recall | Input | Medium-high |
In practice, these capabilities typically ride on general-purpose cloud infrastructure: model services handle unified access and canary rollouts, message queues carry asynchronous long-running tasks, object storage holds multimodal assets and evaluation datasets, and logging and monitoring handle task-level metric collection and threshold alerts. Baidu AI Cloud offers a complete set of general-purpose product categories for this “model capability + engineering infrastructure” combination, and enterprises can integrate with their existing technology stacks rather than re-architecting everything around a single model.
4. An Extension Worth Watching: Engineering Constraints for Long-Running Autonomy
Public materials also mention an intriguing case: in AI-assisted scientific discovery, the model ran autonomously for 48 consecutive hours, screening around 2.7 million candidate molecules, and through four rounds of “design–synthesize–test” loops actually synthesized a new coolant material, offering a fresh approach to data center cooling challenges posed by high-power chips.
Setting the specific domain aside, the engineering significance of this case is that it pushes agent reliability requirements to a new magnitude. 48 hours of uninterrupted operation means:
- Resumable execution is mandatory: state must be persisted, tasks must resume from any breakpoint, and retries must be idempotent;
- Cost circuit breakers are mandatory: set per-task budget caps and tiered alerts to prevent runaway loops from burning through quotas;
- Human review gates are mandatory: any step involving real-world actions (synthesis, deployment, data deletion) must retain explicit confirmation;
- A full audit trail is mandatory: inputs, outputs, tool calls, and decision rationale for every step must be logged, or results cannot be reproduced or verified.
These constraints apply equally to everyday agent scenarios such as code generation, data processing, and operations. In other words, the bottleneck for agent adoption often lies not in model intelligence, but in the surrounding observability, permission boundaries, and recovery mechanisms.
5. Conclusion: Write “Cost per Task” into Your Technical Selection Criteria
Looking back at this release, what truly belongs in team technical standards is not a few benchmark scores but three judgments:
- Architectural efficiency has become a first-class citizen. The combination of sparse experts and sparse attention allows “larger capacity” and “lower per-token cost” to coexist; selection cannot look only at parameter count or only at unit price.
- Token efficiency equals cost efficiency. Completing the same task with fewer round trips saves money while being faster and more stable. When evaluating models, look at both “task success rate” and “tokens per task.”
- Cache and context governance are the lowest-cost optimizations. Given the significant drop in cache-hit pricing, merely restructuring prompts into “stable prefix + volatile suffix” can yield substantial cost savings.
The recommended order of implementation: first close the gap in task-level instrumentation and cost accounting (no data, no optimization), then govern prompt structure and cache hit rates, then move to tiered deployment and small-model offloading, and only then consider switching or upgrading the foundation model. The benefit of this sequence is that every step’s gains are measurable — and none of them gets wiped out by the next model upgrade.
