How to Scientifically Evaluate Trillion-Parameter Multimodal LLMs
When Chinese flagship models start closing the gap with the top overseas closed-source models, developers and enterprises face a practical question: how should they evaluate and choose among them? This article lays out a quantifiable, reproducible framework for assessing trillion-parameter, sparse mixture-of-experts (MoE) multimodal models with million-token context. It explains why total parameters differ from per-token activated parameters and what that means for inference cost and VRAM, how to measure effective long context and native multimodal understanding, how to benchmark long-horizon coding, document agents, and multimodal creation, how tiered pricing and prefix caching shape real bills, and the trade-offs between hosted APIs, self-hosted open weights, and cloud model services. It closes with a method for building a private evaluation set to run continuous regression as model versions drift.
How to Scientifically Evaluate Trillion-Parameter Multimodal LLMs
In mid-2026, Moonshot AI released its new flagship model, Kimi K3. The published figures include roughly 2.8 trillion (2.8T) parameters, 896 experts with 16 activated per pass, a 1M-token context, and native image and video understanding; the full weights are planned to open before July 27, 2026. The company also openly acknowledged that its overall level is slightly behind two overseas closed-source flagships, Claude Fable 5 and GPT-5.6 Sol. The independent evaluation group Artificial Analysis gave it an intelligence index of about 57, placing it third overall. In the Web Development category of blind-vote leaderboards, it topped the list with a score of about 1679.
Note: This article borrows the technical details of the case above as background, but only discusses general technical topics—trillion-parameter scale, sparse mixture-of-experts, million-level context, native multimodality, and tiered pricing. All prices shown are illustrative; refer to the official billing documentation for the latest figures.
As more domestic flagships “hug the international ceiling” appear in quick succession, a real question lands on developers and enterprises: how should we evaluate them, and how should we choose? The answer is not a one-line “close to the strongest closed-source model,” but a method that is quantifiable, reproducible, and directly usable for decisions.
1. Evaluation prerequisites: define the boundary first
1. Separate “model capability” from “service form”
Before evaluating any model, distinguish three ways to obtain it, because each maps to a completely different evaluation system:
| Acquisition method | Core focus | Main cost item |
|---|---|---|
| Hosted API | Quality, latency, rate-limit quotas, SLA, data compliance | Per-call fee (tiered by tokens) |
| Open weights, self-hosted | Throughput, VRAM usage, quantization quality loss, fine-tuning headroom | GPU resources and inference-engineering effort |
| Cloud model service (e.g., Baidu AI Cloud’s Qianfan platform) | Unified access, multi-model switching, integration with storage/monitoring/security | Pay-as-you-go plus supporting cloud resources |
Many arguments come from mismatched comparisons: dismissing self-hosting’s “strong control” using a hosted API’s “out-of-the-box convenience,” or the reverse.
2. Define the evaluation scenario
- High-frequency short chats: care about first-token latency and input cost;
- Long-document and code-base analysis: care about effective context and caching gains;
- Long-running agent tasks: care about tool-call stability and long-horizon consistency.
The same model can score several times differently across these scenarios.
2. Architecture: three numbers behind a trillion parameters
2.1 Total parameters do not equal compute per pass
K3’s 2.8T is the total parameter count, representing the upper bound of the model’s knowledge capacity. It uses a fine-grained sparse mixture-of-experts (MoE) architecture in which only 16 of 896 experts are activated per round. Ignoring sparse matrix multiplications where most values are zero, the compute actually involved per token is mainly the attention part plus the feed-forward networks of the 16 experts—and the experts usually make up the vast majority of parameters in this kind of model. From this, a rough estimate puts the activated share at a little over ten percent:
# Estimate activated parameters per token for an MoE model# (approximation that ignores shared layers, attention, and other negligible terms)def estimate_activated_params(total_params: float, # total parameter countn_experts: int, # total number of expertsn_activated: int, # number of experts activated per tokenshared_params_ratio: float, # share of non-expert params (attention, etc.) - empirical value) -> float:shared = total_params * shared_params_ratioper_expert = (total_params - shared) / n_expertsreturn shared + per_experts if False else shared + n_activated * per_expert # corrected belowdef estimate_activated(total, n_experts, n_activated, shared_ratio=0.08):shared = total * shared_ratioper_expert = (total - shared) / n_expertsreturn shared + n_activated * per_expertt = 2.8e12a = estimate_activated(t, 896, 16)print(f"Activated params ~ {a/1e9:.0f}B, {a/t:.1%} of total")# Activated params ~ 336B, 12.0% of total
This yields two key conclusions:
- Inference cost: it grows mainly with activated parameters, not linearly with total parameters. This is the fundamental reason MoE can be both “large” and “relatively fast.”
- VRAM usage: all expert weights must stay resident and cannot be trimmed in proportion to activation. When self-hosting, the VRAM requirement is close to the total parameter count, so quantization, expert parallelism, and KV-cache management are all indispensable.
So evaluation should ask not only “how many parameters,” but also:
- After sparsification, has the common-sense knowledge and instruction-following that a smaller dense model easily masters degraded?
- Is routing stable within a long context (do the activated experts drift widely across repeated samples of the same task)?
These questions can only be answered by self-hosting, where the routing distribution is observable—and that is exactly where open weights deliver real value.
2.2 Million-token context: the window is a ceiling, effective context is capability
One million tokens is roughly the volume of dozens of full-length novels, enough to load a mid-sized code repository at once. Yet the industry broadly observes that models use information in the middle of the context worse than at the beginning and end—the so-called “lost in the middle.” The longer the context, the more retrieval-style accuracy tends to decay.
Evaluation method: layered “needle in a haystack”
- Insert several unrelated “needles” at fixed positions (10%, 30%, 50%, 70%, 90%) in an ultra-long document;
- Have the model find all needles and answer multi-hop questions built around them (e.g., “add the value of needle 2 to the value of needle 5”);
- Measure recall with exact keyword matching, and multi-hop accuracy by comparing against the ground-truth answers.
Engineering notes:
- Million-level context relies on efficient attention, positional-encoding extrapolation, and KV-cache optimization; during evaluation, watch first-token latency under long inputs;
- In engineering, pair it with retrieval-augmented generation (RAG): first narrow candidates with vector retrieval, then use long context for cross-segment reasoning—rather than stuffing all material into the window;
- Use the same tokenizer and token-counting method across all models being compared, otherwise “1 million” holds different amounts of content for each.
2.3 Native multimodal: the focus is “input understanding,” not “content generation”
K3’s understanding of images and video is native, while its output remains text. This is a different capability from text-to-image and text-to-video, and it is evaluated in entirely different ways:
- Images: whether UI screenshots, hand-drawn sketches, and design drafts can be accurately turned into runnable code is the core of “understand the interface, then modify it”;
- Video: whether it can grasp long-range temporal information, such as recognizing the step order in an operation demo and pinpointing when an event happens. Most interfaces currently require the client to extract frames before uploading, so evaluation must consider both the frame-extraction strategy and the ability to preserve inter-frame relationships;
- Self-test method: prepare 50–100 real samples with ground-truth answers (screenshots, report charts, screen recordings of operations), and measure field-level accuracy and manual-correction cost—not just “can it recognize it or not.”
3. Task level: how to test high-value scenarios
3.1 Long-horizon coding: from “completing code” to “completing engineering tasks”
K3 targets a core capability of long-horizon coding with minimal human intervention: working through large repositories, calling terminal tools, and modifying front-ends from screenshots. Official demos cover generating playable mini-games from hand-drawn sketches, long-running low-level kernel optimization, and even Triton-like compiler pipelines; in community testing, its front-end aesthetics and first-pass success rate are often praised.
For evaluation, use public code benchmarks as an initial screen, then build a private evaluation set from your own real repositories:
- End-to-end pass rate: for bug-fix tasks, use “do all unit tests pass” as the criterion;
- First-pass success rate: for front-end tasks, use “visual diff + human scoring”;
- Manual intervention cost: track average step count, the ratio of ineffective tool calls, and the number of rounds needing human takeover.
For agent scenarios, also verify error recovery. After a compile or test failure mid-run, the model should read the error, make changes, and retry until it converges or clearly reports a risk.
# Illustrative: the core loop of a long-horizon coding agentMAX_STEPS = 50messages = [{"role": "system", "content": SYSTEM_PROMPT}, # stable prefix, improves cache hits{"role": "user", "content": "Fix the login timeout issue and make sure all unit tests pass"},]for step in range(MAX_STEPS):resp = client.chat.completions.create(model="your-model-name",messages=messages, tools=TOOLS)msg = resp.choices[0].messagemessages.append(msg)if not msg.tool_calls: # no tool calls means the task is done or needs clarificationbreakfor call in msg.tool_calls: # run tools in order: read, search, run commands, edit files, etc.result = run_tool(call, workspace="/repo")messages.append({"role": "tool", "tool_call_id": call.id, "content": result})else:print("Reached the step limit; task incomplete and needs human takeover")
The code above follows the OpenAI-style tool-calling protocol commonly supported across mainstream model services; refer to the actual documentation for parameter names.
3.2 Document and office agents: structured output decides real-world quality
The difficulty here is not single-turn Q&A but multi-turn, multi-document, long-flow work. You need to verify:
- After multiple tool calls, is the goal still clear, or does it drift off-topic or forget constraints?
- Can the output reliably satisfy a strict schema so downstream systems can consume it automatically?
- Are citations real and traceable, with no “fabricated page numbers”?
Have the model “prove” its answer, then let a program verify it:
# Illustrative: have the model attach a source citation to each conclusionPROMPT = """Answer the question based on the document below.Requirements:1) After each conclusion, mark the source paragraph number as [n];2) If you cannot determine the answer, reply "Not mentioned in the document"; do not speculate.Document:{document}Question: {question}"""resp = client.chat.completions.create(model="your-model-name",messages=[{"role": "user", "content": PROMPT}],response_format={"type": "json_object"}, # require JSON output for easy parsing)print(resp.choices[0].message.content)# Validation checks: whether citation numbers are out of range, whether the JSON parses,# and whether unanswered questions are correctly refused
3.3 Multimodal-assisted creation: from understanding assets to producing content
Scenarios like video editing and design reproduction test the “see–think–do” loop:
- Understand the content, rhythm, and structure of the source material;
- Produce storyboards, subtitles, or code;
- Iterate and refine based on preview or playtest results.
The evaluation focus is not single-frame recognition accuracy, but the number of human interventions across the full workflow and the usability of the final output.
4. Cost: breaking down the bill under tiered pricing
K3’s API uses tiered pricing: cache-hit input at 2 yuan, cache-miss input at 20 yuan, and output at 100 yuan (per million tokens, RMB). There is a 10x gap between hit and miss, so the cost of long-context applications is driven mainly by the cache hit rate.
Pricing formula:
cost per request = cache-hit input tokens x hit unit price+ cache-miss input tokens x miss unit price+ output tokens x output unit price
Two cost-sensitive reminders:
- At the rates above, output per million tokens is 5x the cache-miss input. So “controlling output length” and “compressing input” are equally important; only watching the input price easily distorts the calculation;
- The input unit price of domestic flagships is usually already below or near most overseas flagships, with the price gap showing up more in the output and cache-hit portions. When choosing, estimate the monthly bill from your real traffic structure (input/output ratio, cacheable share) rather than comparing unit prices alone.
Making the cache actually hit:
# Illustrative: place stable content first to maximize the prefix-cache hit ratedef build_messages(repo_index: str, question: str):# Order is fixed: system prompt -> stable project knowledge -> changing question# If the order changes, or the stable part changes, all cache after that point is invalidatedsystem = "You are a senior backend engineer. Always give runnable code with explanations."context = f"[Project notes]\n{repo_index}" # low change frequency, large size, placed firstreturn [{"role": "system", "content": system},{"role": "user", "content": f"{context}\n\n[Current question]\n{question}"},]
In practice, follow three principles:
- Reuse the same prefix as much as possible (a fixed system prompt, fixed tool definitions, and a fixed knowledge-base summary);
- Put content that changes every time—timestamps, random IDs—at the end of the message;
- Periodically check the actual hit rate. Some model services offer prompt caching, where for identical prefixes beyond a certain length, that portion of the input can be reused on subsequent requests (see the official documentation for specific rules).
Latency also matters:
- For real-time interaction, first-token latency usually affects experience more than total elapsed time;
- For long tasks, the number of output tokens per unit time determines how long a full flow takes.
Both should be part of your evaluation metrics.
5. Open weights: the source of control, and the start of operations
Opening the full weights delivers three capabilities: fully in-domain private deployment where data never leaves; full-parameter or parameter-efficient fine-tuning on domain corpora; and free evaluation and research in a controlled environment. The cost is that a self-built cluster needs dedicated VRAM and parallelism optimization, you own throughput and stability yourself, and regression validation after each model upgrade falls on you too.
| Dimension | Hosted API | Self-hosted weights | Cloud model service |
|---|---|---|---|
| Time to start | Fast | Slow, must set up inference serving | Relatively fast |
| Data control | Depends on service terms | Fully self-managed | Depends on deployment form |
| Elastic scaling | Guaranteed by the provider | Must build your own scale-up/down | Can scale via containers and other platforms |
| Fine-tuning / customization | Usually only lightweight fine-tuning | High flexibility | Depends on platform capabilities |
| Suitable scenarios | Quick validation, spiky traffic | Sensitive data, large-scale steady calls | Unified multi-model governance, fast launch |
On cloud platforms such as Baidu AI Cloud, these capabilities can be combined further:
- Use object storage to hold weight files and evaluation datasets;
- Use a container platform to host inference services;
- Use a message queue to smooth peaks and fill valleys for offline batch jobs;
- Use log and monitoring/alerting services to track call volume and cost.
This extends “evaluating a model” into “governing a model service.”
6. Putting it into practice: continuous regression with a self-built evaluation set
Public leaderboards are only an initial screen; enterprise decisions must rely on a private evaluation set that stays close to the business:
- Collect tasks: extract 300–500 real tasks from tickets, requirements, and defect records, covering both high-frequency and difficult cases;
- Define metrics: use automated assertions for programmable tasks (compile, test, schema validation, exact match), and human blind review or “model review + human spot checks” for open-ended tasks;
- Regression comparison: run candidate models, the current production model, and historical versions on the same question set, with fixed random seeds and decoding parameters;
- Continuous monitoring: model services keep iterating, so periodically re-run the evaluation set to catch version drift and roll back or switch in time.
7. Common pitfalls and conclusion
- Leaderboard-only thinking: a composite intelligence index and blind-vote rankings reflect average preference, not your business performance;
- Context worship: a bigger window does not mean it is used well; lost-in-the-middle and first-token latency are the real bottlenecks;
- Only watching input unit price: output and cache hits are often where the cost really sits;
- Ignoring version drift: production quality needs continuous regression, not a one-time acceptance test.
Back to the original question: how do you evaluate a flagship model with trillions of parameters, a million-token context, and native multimodality?
- Architecture: total parameters determine capacity, activated parameters determine cost. Structural improvements like sparse experts and attention residuals must ultimately show up in effective performance on long-text and long-horizon tasks;
- Tasks: use real code bases, real documents, and real assets to measure end-to-end pass rate and the number of human interventions;
- Cost: compute the total bill from tiered unit prices, cache hit rate, and the input/output ratio;
- Control: weigh data compliance, fine-tuning needs, and operations capability when choosing among hosted APIs, self-hosting, and cloud services.
The company openly admits K3 is still slightly behind the strongest overseas closed-source models, but it has compressed the gap between domestic flagships and the international first tier to “half a step”—while being cheaper, self-deployable, and complete in both long context and multimodality. For the vast majority of teams, the more worthwhile question is not “where does it rank,” but “under my tasks and my cost constraints, is it the best value-for-money option right now?” Answering that with a reproducible evaluation set is more reliable than any leaderboard.
