Beyond the Benchmarks: How to Properly Evaluate the New Generation of Open-Source LLMs

1. Conclusion First: Benchmarks Are the Least Important Part

Whenever a major open-source model is released, the community’s first reaction is usually to refresh leaderboards, post screenshots, and rank the contenders. The numbers matter, of course—they shape first impressions—but the right response after seeing them is not to cheer or dismiss, but to ask three more fundamental questions:

1. What hardware was this model run on, and at what cost?
2. Is its inference pipeline locked into a single compute ecosystem?
3. What constraints did the release team quietly acknowledge in the fine print of the technical report?

This article uses the “flagship + lightweight” dual-spec release pattern now common among the latest open-source MoE (Mixture-of-Experts) models as a case study to unpack these three questions.

2. Dual-Spec Release: Product Tiering Logic Under MoE Architectures

Nearly all new open-source models adopt a MoE architecture. The core idea is to “enlarge the total parameter pool while activating only a small subset of experts per token,” balancing model capacity against inference cost:

# MoE sparse activation: each token routes to only a few experts
def moe_forward(x, router, experts, top_k=8):
    logits = router(x)                    # compute expert preference for each token
    weights, idx = topk(logits, k=top_k)  # keep only top-k experts
    out = sum(w * experts[i](x) for w, i in zip(weights, idx))
    return out                            # the vast majority of parameters are not computed

On this foundation, the “flagship + lightweight” dual-spec approach is becoming an industry paradigm:

• Flagship spec: total parameters reaching the trillion scale, with tens of billions activated per pass, targeting complex reasoning and high-difficulty coding tasks—this is where the capability ceiling lives.
• Lightweight spec: total parameters in the tens-of-billions range, with just over ten billion activated, targeting high-concurrency, low-latency, cost-sensitive production scenarios.

Both specs share the same training and alignment methodology: capability floors aligned, cost ceilings spread apart—essentially, one R&D investment yielding two commercialization paths. For users, this means selection no longer requires an all-or-nothing bet on the flagship version; for many business scenarios, the lightweight spec plus engineering optimization is already sufficient.

3. The Fine Print of Technical Reports: Cross-Hardware Validation Is the Real Signal

I have a habit when reading release materials: read the big text in the announcement, hunt for the fine print in the report. After going through this round of technical reports, the most informative items were not any benchmark score, but three very understated statements.

Statement one: The fine-grained expert parallelism (EP) scheme was validated on both general-purpose GPUs and domestic NPUs.

The weight of this point needs to be understood in an engineering context. The bottleneck in MoE inference has never been raw compute—it is data movement. Fine-grained experts slice experts smaller and routing finer, which increases all-to-all communication frequency and scheduling complexity. Running the same EP scheme across two instruction sets, two memory hierarchies, and two sets of communication primitives with markedly different characteristics means the engineering team did something rarely advertised explicitly: decoupling the model runtime from single-hardware dependency. The inference path now has cross-platform adaptation capability.

Statement two (fine print): The core components released as open source are still built primarily on the native toolchain of the mainstream GPU ecosystem.

In other words, “it can run on NPUs” and “everything you need to run it on NPUs is open-sourced” are two different things. The currently open-sourced low-level implementations—grouped matrix multiplication, fused operator libraries, and so on—remain deeply tied to the mainstream GPU ecosystem. To reproduce the inference pipeline on other hardware, users must complete operator adaptation and tuning themselves: validation was done internally by the release team, and the adaptation cost is temporarily left to the community.

Statement three (an operational note): Constrained by high-end compute supply, the flagship spec currently has limited serving throughput; once the new-generation super-node architecture of domestic compute clusters lands at scale, prices will drop substantially.

Read the three statements together and a dual-track strategy becomes clear: on one hand, push optimization to the extreme within the existing GPU ecosystem to squeeze every bit of throughput from current compute; on the other, reserve room for a multi-compute environment, waiting for domestic super-node clusters to ramp up so that scale can amortize costs and price cuts can buy ecosystem position. Benchmark scores are merely a byproduct of this strategic line.

4. Capability Structure: Read All the Numbers, But Don’t Just Look at the Total

Back to the leaderboards themselves. Objectively, this generation of open-source models shows very clear structural characteristics:

• Code: Essentially locked in as the top of the open-source camp, having caught up to or surpassed the international first-tier closed-source models on continuously updated code-generation benchmarks, and approaching top-tier performance on real software engineering tasks (issue fixing, commit-level changes).
• Math: On olympiad-level math benchmarks, very close to the strongest closed-source models, with the gap narrowed to single-digit percentage points.
• Knowledge: This is the most obvious weakness—a significant gap versus international top closed-source models on factual short-answer benchmarks, and high-difficulty comprehensive knowledge benchmarks have not yet been matched.

This structure reveals the training strategy’s priorities: first invest in code and math—capabilities that are “verifiable, synthesizable from data, and amenable to reinforcement learning loops”—then fill in knowledge breadth and factuality. The implication for model selection: if your core scenarios are code generation, code review, or structured reasoning, the open-source camp has entered a range where it can compete head-on; if your business relies heavily on factual Q&A and knowledge breadth, you still need to carefully evaluate hallucination rates and knowledge-update mechanisms rather than accepting it wholesale.

But after reading all the numbers, my first reaction remains: so what? There are still several layers between benchmark scores and engineering usability:

• Benchmark saturation and data contamination create systematic deviation between high scores and real capability.
• Benchmark task distributions almost never overlap with your real workload distribution.
• Leaderboard scores contain no information about throughput, latency, concurrency stability, or long-context degradation—yet these are exactly what determine production experience and cost.

5. An Evaluation Checklist for Deployers

Rather than arguing about “who is strongest,” build your own evaluation pipeline. A directly actionable checklist:

model_eval_checklist:
  capability:            # capability: let your own data speak
    - business replay set: sample real historical tasks, build a private eval set
    - long-context spot checks: large-document retrieval, multi-turn consistency
  engineering:           # engineering: measure against production load
    - throughput: tokens/s at target concurrency
    - latency: P99 first-token latency
    - stability: failure rate and degradation behavior under stress
    - cost: blended cost per million tokens (including VRAM, node count, network overhead)
  compatibility:         # compatibility: keep options open for multi-compute
    - backend: does the inference framework support multiple hardware backends?
    - ops: coverage of key operators (grouped matmul, communication primitives) on target hardware
    - runtime: is there strong coupling to a single hardware toolchain?

At the infrastructure level, unified management of heterogeneous compute will quickly become a necessity: use a container platform for unified scheduling and pooling of GPU/NPU resources, object storage to handle model weight and dataset distribution, and logging plus monitoring alerts to observe the throughput and latency levels of inference services. These general-purpose capabilities offered by cloud providers such as Baidu AI Cloud are a cost-effective starting point for most teams building heterogeneous inference platforms—there is no need to build from scratch.

6. Closing: One Release, Three Things to Look At

To evaluate a new-generation open-source model, I recommend this order:

1. Start with architecture and spec design—the ratio of total to activated parameters in MoE directly determines the cost curve.
2. Then read the fine print of the technical report—cross-hardware validation, open-source boundaries, and throughput constraints hide the release team’s real roadmap.
3. Only then look at benchmark scores—and be sure to recalibrate them with your own private eval set.

Benchmarks determine whether a model’s floor is sufficient, while architectural decoupling and compute adaptation determine its—and your business’s—cost ceiling and supply-chain security over the next two to three years. That is what the fine print is really trying to tell you.