How to Evaluate Ultra-Large Open-Source LLMs: A Complete Guide to Architecture, Capability, and Cost

Introduction: The Methodology Behind a Piece of Folk Wisdom

A new batch of ultra-large open-source models has just had its full weights released for download, and community discussion is heating up. One question comes up constantly: are these models actually good, and are they worth adopting? A popular piece of folk wisdom captures the verdict: “You can call it slow, you can call it expensive—but you can hardly call it weak.” The phrasing is casual, yet it points at the correct posture for evaluating large models: capability and efficiency are two independent sets of metrics, and they must be analyzed separately before making any combined trade-off.

This article translates that folk conclusion into a rigorous technical evaluation framework: first, what the architecture design determines; then, what post-training contributes; next, how benchmark scores should be read; and finally, the cost math of self-hosting.

1. The Evaluation Framework: Separate Capability from Efficiency

Evaluating a large model requires at least four dimensions:

1. Capability ceiling: success rates on complex reasoning, code generation, and agent tasks;
2. Serving efficiency: time to first token, generation throughput, and concurrency capacity;
3. Unit cost: beyond the per-token price, what matters more is the total cost per task;
4. Engineering properties: context length, weight format, sharding scheme, inference-engine ecosystem, and deployment barrier.

Why separate them? For reasoning-enhanced models, “slow” and “expensive” are often side effects of “strong”: to gain accuracy, the model generates longer chains of thought, and longer thinking means more tokens consumed and more time per task. If a model leads clearly on accuracy for hard problems, its higher token consumption may be precisely evidence of its capability. Conversely, a model that is fast and cheap but fails often may not save money in a real business—rework is itself a cost.

The right approach: use capability metrics to decide whether the model can do the job at all, then use efficiency and cost metrics to decide whether it is worth it.

2. Architecture: Three Key Design Decisions in an Ultra-Large Model

Taking the recent batch of representative open-source models as a reference—a sparse MoE architecture with roughly 2.8 trillion total parameters, a million-token context window, and natively low-precision floating-point weights (about 1.5 TB in total, split into nearly 100 shards)—three design decisions deserve a closer look.

2.1 Sparse MoE: Decoupling Capacity from Compute

The core idea of sparse mixture-of-experts (MoE) is that each token activates only a small fraction of the expert parameters. The direct benefit: parameter capacity is decoupled from per-token compute—the model’s total knowledge capacity can scale into the trillions while the actual computation per token stays within an acceptable range.

But there is no free lunch, and MoE’s costs concentrate on memory and communication:

• All weights must reside in memory: regardless of how few experts are activated per token, the entire expert set has to stay loaded. At roughly 1.5 TB, this is firmly beyond what any single machine can handle.
• Expert parallelism drives cross-GPU communication: experts are distributed across dozens or even hundreds of GPUs, and routing plus All-to-All communication places extreme demands on interconnect bandwidth. This is why deploying such models depends on “supernode”-class high-bandwidth interconnect architectures rather than ordinary server stacking.

2.2 Hybrid Attention and Multi-Token Prediction: The Memory–Throughput Trade-off at Million-Token Context

Standard softmax attention has two well-known bottlenecks: compute complexity grows quadratically with sequence length, and the KV cache grows linearly with context length. At the million-token scale, the KV cache alone could consume several terabytes of memory, so a pure full-attention design is effectively a dead end.

The recent batch of models widely adopts a “linear attention + sparse attention” hybrid stack:

• Linear attention compresses attention complexity to roughly linear and condenses KV information into a fixed-size state, dramatically reducing memory usage for long contexts;
• Sparse attention preserves precise access to local windows and key positions;
• The two are interleaved across layers: most layers use linear attention to “compress the state,” while a small number of critical layers keep sparse attention to “preserve retrieval,” striking a balance between long-context memory, decode throughput, and long-range recall.

Multi-token prediction (MTP) is another mechanism worth noting: the model predicts several future tokens in a single forward pass. During training, it supplies denser supervision signals and helps the model learn better task planning; at inference, it pairs naturally with speculative decoding—generate multiple candidates at once, then verify them in parallel with the main model, which lifts decode throughput. This also explains an industry pattern: many model teams release an inference-optimized version alongside the standard one, boosting throughput by 2–3x. Slowness is a variable that engineering iteration can keep solving.

2.3 Native Low-Precision Weights: Where the ~1.5 TB Comes From

These models mostly ship weights natively in a low-precision floating-point format (4-bit floats with grouped scaling factors) rather than quantizing after training. The arithmetic is simple: 2.8 trillion parameters × 0.5 bytes per parameter ≈ 1.4 TB; adding the router, embeddings, and scaling factors stored at higher precision brings the total to roughly 1.5 TB across nearly 100 shards, which matches publicly reported measurements.

Compared with integer quantization, the grouped-scaling float format is friendlier to outliers and loses less precision. And “natively trained at low precision” means the weight distribution itself matches the format, which does more to guarantee inference quality.

3. Post-Training and Cross-Domain Reinforcement Learning: The Real Source of the Capability Ceiling

One industry trend deserves attention: as pretraining scale approaches diminishing returns, the gap between models is increasingly determined by post-training. The recent standout models have poured compute into large-scale post-training and cross-domain joint reinforcement learning (RL):

• Cross-domain joint RL: math, code, and tool-use tasks are put through the same RL pipeline and trained jointly, so capabilities transfer across domains—the “step-by-step verification” learned in math reasoning migrates naturally to code debugging and solution planning;
• Full-stack engineering rebuild: training infrastructure, inference engines, and sandboxed evaluation environments are integrated end to end. For agent scenarios in particular, the model must execute commands in a real terminal environment, observe results, and revise its strategy; a reproducible sandbox evaluation system is both the source of training signal and a quality guardrail.

Judging by public evaluation trends, this investment pays off clearly: on agent benchmarks that require completing multi-step engineering tasks in real terminal environments, leading open-source models now trade wins with closed-source flagships, with gaps compressed to single-digit percentage points; on code-agent benchmarks, a single version iteration within the same model family has delivered jumps of several percentage points. The capability gap between open and closed models is being closed rapidly on the new agent frontier.

4. Benchmarks: How to Read the Scores

4.1 Choose the Right Benchmarks: Static Ones Get Contaminated; Agent Ones Are Hard to Game

Traditional static knowledge QA and single-turn code-completion benchmarks have two problems: high risk of training-data contamination, meaning scores can be inflated; and weak alignment with real-world usage. Agent benchmarks, by contrast, require the model to interact over multiple turns in a real terminal environment, call tools, and correct itself based on feedback—nearly impossible to pass by memorization—so their reference value is much higher.

4.2 Per-Task Cost Is the Real Cost

One empirical finding that is easy to overlook: on the same agent tasks, token consumption across different models can differ by an order of magnitude. This means:

Per-task cost = token price × tokens consumed per task

A model with a cheap per-token price but longer chains of thought and more retries can end up more expensive in total; a model with a high per-token price but shorter solution paths and a higher first-pass success rate can end up cheaper. Comparing token price alone is one of the most common mistakes in model selection. By the same logic, long chains of thought also stretch task completion time, so latency also has to be measured on a per-task basis.

4.3 Build Your Own Evaluation Set

Public benchmarks are only good for initial screening. The reliable approach: pull a few hundred representative tasks from your own business (repository refactoring, ticket handling, data analysis, and so on) and run continuous regression testing on three metrics—success rate, average token consumption, and average completion time. That is the evaluation standard genuinely tied to your own business.

5. Self-Hosting: Count the Machines and the Budget First

With the weights available for download, many teams’ first reaction is “can we deploy this ourselves?” The answer: yes—but do the math first.

5.1 The Memory Baseline: Weights Are Just the Entry Ticket

Back-of-the-envelope math with roughly 1.5 TB of weights and high-end inference cards providing about 140 GB of memory each:

import math
weight_gb   = 1.56 * 1024     # total size of the native low-precision weights
gpu_mem_gb  = 140 * 0.9       # usable memory per GPU (10% headroom reserved)
# Tier 1: minimum GPU count just to hold the weights
weight_cards = math.ceil(weight_gb / gpu_mem_gb)   # roughly a dozen or so
# Tier 2: MoE expert parallelism + KV cache + concurrent throughput
# The actual GPU count is often several times the weight-only count;
# finalize it through load testing against target concurrency and context length

Two takeaways: first, the weights alone need a double-digit number of GPUs, and that is merely the bar for being “loadable”; second, to achieve usable throughput you have to add the KV cache, cross-GPU communication from expert parallelism, and concurrent batching, so the real cluster size is often several times the weight-only count. Hybrid attention does dramatically reduce KV overhead for long contexts, but million-token contexts under high concurrency still put non-trivial pressure on memory.

5.2 Cluster Shape and Budget Magnitude

Working from the common “supernode” form factor—dozens of GPUs in a single node, tens of terabytes of aggregate memory, and high-bandwidth interconnect within an integrated rack—running a trillion-parameter sparse MoE model at full capacity typically takes a cluster of multiple supernodes, plus network switching, racks, and redundancy design. At current market prices for flagship inference cards, the total investment is on the order of tens of millions of RMB (several million US dollars), and it moves sharply with GPU prices—figures quoted six months ago may already be off today, so cost estimates need to be refreshed continuously.

5.3 Three Deployment Paths: Match the Scenario, No Need to Jump Straight to the Top End

Deployment path	Best for	Key considerations
Self-hosted cluster	Strict data compliance, very high call volume, deep fine-tuning needs	Data never leaves your environment; but high budget floor and demanding ops requirements
Cloud GPU cluster	Fast validation, elastic scaling, periodic peak loads	Pay-as-you-go, minute-level scaling; GPU compute, container platforms, object storage, and monitoring capabilities from providers such as Baidu AI Cloud can shorten the path from loading weights to serving traffic
Inference API	Small teams getting started, business-validation phase	Zero ops, pay-per-token; validate business value with an API first, then evaluate self-hosting once usage is stable

One more selection note: it is worth watching how quickly major cloud providers adapt—announcing same-day cluster support and out-of-the-box deployment for a major open-source release has become industry routine. Also, the maturity of the software stack (how well inference engines support MoE, hybrid attention, multi-token prediction, and low-precision formats) directly determines whether you get “paper performance” or “production performance.” Validate on a small environment first, then scale up.

6. Summary: A Practical Evaluation Checklist

Condensing everything above into a checklist to run through during selection:

1. Capability: How does it perform on agent benchmarks? What are its failure modes? Is retrieval reliable within a million-token context?
2. Efficiency: Do throughput and latency match your workload? Is there an officially accelerated version that keeps iterating?
3. Cost: How many tokens does a typical task consume? Which wins—API pricing or self-hosted TCO?
4. Engineering: Are the weight format, sharding, and inference-engine ecosystem mature? Is the deployment documentation solid?
5. Risk: License terms, the model’s iteration pace, and your team’s own operations capability.

Finally, back to the opening line: slow and expensive are engineering problems that inference optimization will keep improving; the capability ceiling is the model’s true ceiling. When evaluating an ultra-large open-source model, rather than agonizing over today’s latency and price, look at its architectural headroom and its post-training direction—the former may well change in three months, while the latter determines how far the model can go in three years.