Flagship LLM Evaluation Guide: How to Choose the Right Large Model for Your Workload
A newly released flagship model — roughly 2.8 trillion parameters, a million-token context window, native image and video input, open weights and cache-discounted pricing — has put the question "which model is the best?" back in the spotlight. This guide breaks selection into three layers: what a spec sheet actually implies, how to evaluate capability beyond third-party leaderboards, and how to model real cost. It includes practical tests for long context and multimodality, a reusable Python evaluation script, scoring dimensions for long-horizon and agentic tasks, guidance on choosing between subscriptions and pay-as-you-go APIs, a token cost model built around cache hits, and a decision checklist for selection reviews.
Why evaluating flagship models is hard
A model vendor recently shipped a new flagship model. The published specs look strong:
- roughly 2.8 trillion total parameters;
- about 1 million tokens of context;
- native image and video input;
- tiered pricing that separates cache-miss input tokens from output tokens, with a discounted rate for cache hits;
- a stated plan to release the full weights to the public.
The vendor also admits the model still trails the strongest closed-source models overall. Yet on third-party independent evaluations and on human blind-vote leaderboards aimed at frontend development, it already sits in the top tier — at a lower cost per call.
That makes “how do you evaluate a model” a popular topic again. Most people reach for “which one is the best,” but that question is hard to answer:
- Parameter counts, context length, and leaderboard scores are proxy metrics, and the gap between them and real business performance is not small;
- “Capable” is not the same as “right for you” — cost per completed task is usually the real deciding factor;
- Benchmarks run on standardized questions. Business runs on messy data, large codebases, and vague requirements. How well the two correlate has to be verified first-hand.
A better approach is to break “which is best” into three layers:
- Layer 1, what the specs mean: what parameter count, context length, multimodality, and open weights actually imply;
- Layer 2, how capable it is: how far to trust third-party signals, and how to validate against your own workload;
- Layer 3, whether you can afford it: how to model per-call and monthly cost, and how to choose between a subscription and pay-as-you-go.
The article walks through all three and ends with reusable evaluation scripts and a selection checklist.
1. Reading the spec sheet: translating a parameter table into engineering meaning
1.1 Total vs. activated parameters: a trillion parameters is not a trillion FLOPs
The claimed “roughly 2.8 trillion parameters” refers to total parameters under a sparse mixture-of-experts (MoE) architecture — the sum of all expert subnetworks. A gating network selects a few experts per input, so a single forward pass activates only a small fraction of the weights. Activated parameters determine compute per token; total parameters mostly determine knowledge capacity and the ceiling on expressiveness.
Three consequences follow:
- Parameters and capability are not linearly related. Training data quality, data mixture, and post-training methods (supervised fine-tuning, preference alignment, reinforcement learning) matter just as much.
- Separate weight size from compute consumption. Trillion-scale weights need large memory capacity or multi-GPU tensor parallelism; activation sparsity keeps per-token compute well below a dense model of the same nominal size, which usually means friendlier throughput.
- Be careful with quantization. Moving weights from half precision to 8-bit or even 4-bit can cut memory dramatically, but may cost quality. Verify it empirically.
Practical guidance: when consuming a hosted model service, total parameter count mostly shapes your expectation of knowledge breadth. If you plan to self-host from open weights, estimate memory as total parameters × bytes per parameter, then add headroom for the KV cache and runtime overhead.
1.2 Context window: advertised length is not effective length
One million tokens is enough to hold an entire novella — roughly 600,000–700,000 Chinese characters, or about 750,000 English words. What matters in engineering, though, is “effective context”:
- Length comes with a price. More input tokens means higher prefill compute and higher time-to-first-token. Under attention, the KV cache grows roughly linearly with context, so memory and cost climb with it.
- The middle gets lost. Research has found that models use information at the beginning and the end of a long context reasonably well, while recall of key information placed in the middle drops noticeably — the “lost in the middle” effect.
- A long window is not strong reasoning. A long window makes “it fits” possible; “finds it, uses it, reasons correctly” still needs targeted verification.
How to test: combine the following:
- Plant factual “needles” at different positions in 8K, 32K, 128K, and near-limit inputs, and check localization and recall;
- Ask multi-hop questions that require linking information across distant passages;
- Ask for a structured summary of a long document and check that numbers and clauses are cited correctly;
- Complete a cross-file change inside a large code repository.
If output quality degrades clearly as input length grows, the advertised window cannot be used directly for long-document work.
1.3 Native multimodality: from “looking at images” to “watching video”
“Native image and video input” means images and video are converted into visual tokens and processed together with text tokens before entering the language model. That differs from a bolted-together pipeline that runs OCR first and then has the model read the extracted text:
- Advantages of the native approach: it can understand visual structure — layout, charts, formulas, UI composition, and temporal sequence across frames;
- Scenarios it covers: screenshot Q&A, parsing papers and reports, extracting key points from video, reconstructing frontend pages.
Keep an eye on:
- Token accounting: images are usually converted to visual tokens by resolution or fixed grid tiles, and video is billed by sampled frames. Higher resolution and higher frame rate mean more consumption.
- Detail sensitivity: small type and low-contrast screenshots lose detail. Cropping and upscaling before upload is often more effective than switching models.
- Reproducibility: fix the frame-sampling strategy and the order of visual input, and record them in your version notes.
1.4 Open weights: open source is not the same as unrestricted use
Full weight availability means an organization can deploy privately and fine-tune or build on the model without data leaving its own perimeter. That meaningfully improves data compliance and long-term control, but open weights still come with constraints:
- Review the license first. Commercial restrictions, attribution requirements, naming rules for derivative models, and whether your changes must be open sourced — assess each one item by item.
- Then assess operating cost. Serving trillion-scale weights requires matching resources. GPU or accelerator cost, scale-up and scale-down behavior, and operations headcount all belong in total cost of ownership (TCO).
- Match the deployment path to the workload. Core pipelines touching sensitive data can be self-hosted; elastic peak load can draw on general cloud compute (GPU instances, container platforms, and similar) and scale on demand.
2. Evaluating capability: read third-party signals correctly, then build your own eval set
2.1 How to read three types of third-party signal
| Signal type | Typical form | Strengths | Main limitations |
|---|---|---|---|
| Public benchmarks | Standardized tasks on general knowledge, code generation, competition math | Reproducible, easy to compare side by side | May be contaminated by training data, may be saturating, sensitive to prompting |
| Independent intelligence indices | A ranking produced by aggregating many benchmarks | Fast read on the overall tier | Weighting is subjective; aggregation can hide a weak individual dimension |
| Human blind-vote leaderboards | Pairwise anonymous comparisons, votes converted into a ranking score | Close to real subjective experience | Affected by voter preferences and question mix; code and frontend scenarios are weighted heavily |
Across those signals, the flagship model looks roughly like this:
- close to the top tier on the composite intelligence index, with a small gap to leading closed-source models;
- notably strong in blind votes aimed at frontend development;
- ahead of same-class models in the vendor’s own evaluation, though the vendor also admits it still falls short of the strongest closed-source model overall.
This “near the top, leading in one area, cheaper” combination is exactly what makes evaluation hard: the gaps between first-tier models are often smaller than the gaps between different scenarios. External leaderboards are good for shortlisting, not for making the final call.
2.2 Build your own eval set: five steps to a business benchmark
Step 1: define the task list. Rank scenarios by call volume × business impact, for example:
- code completion, unit test generation: high frequency, automatically verifiable;
- requirements-to-API-documentation, failure root cause analysis: low frequency, high value;
- multimodal document recognition: an explicit accuracy requirement.
Step 2: collect real samples. Sample from de-identified production requests, covering long and short inputs, multiple languages, multi-turn conversations, and outlier cases, so the distribution matches production.
Step 3: define scoring criteria.
- Automatically checkable: unit test pass rate, JSON Schema validation pass rate, field-level accuracy (precision/recall);
- Subjective quality: score against a checklist of key points, combined with model judging and human spot checks;
- Judging bias: cross-judge the same samples across several candidate models so the judge model’s own preferences matter less.
Step 4: control size and repetition count. Repeat each task at least three times and report both pass rate and variance. Even at temperature 0 and other deterministic settings, results can vary; a single run proves nothing.
Step 5: keep regressing. After a model version update, a prompt change, or a switch of default parameters, rerun the eval set automatically and compare the key metrics.
Cases can use a structured format like this:
{"id": "code-0007","task": "Generate unit tests for the function write_config","input": "Prompt containing the function source code and dependency notes","checks": [{"type": "unit_test", "target": "tests/test_write_config.py"},{"type": "contains", "values": ["tmp_path", "raises"]}],"weight": 2}
2.3 A verification script: quality, cost, and latency in one pass
The script below sketches the evaluation flow in Python. model_service is an abstract placeholder — swap in a generic wrapper around whatever service you use, and treat the provider’s documentation as authoritative for actual interfaces.
import jsonimport timefrom dataclasses import dataclassfrom typing import List@dataclassclass Case:id: strtask: strinput: strchecks: List[dict]weight: int = 1def evaluate(cases, model_service, judge_model, repeat=3):# Illustrative unit prices in CNY per million tokens; confirm with your providerPRICE_HIT, PRICE_MISS, PRICE_OUT = 1.0, 4.0, 20.0stats = {"score": 0.0, "weight": 0, "cost": 0.0,"latency": [], "pass": 0, "total": 0}for case in cases:for _ in range(repeat):t0 = time.perf_counter()resp = model_service.chat(messages=[{"role": "user", "content": case.input}])stats["latency"].append(time.perf_counter() - t0)u = resp.usagehit = getattr(u, "cached_tokens", 0) or 0miss = max((u.prompt_tokens or 0) - hit, 0)stats["cost"] += (hit * PRICE_HIT + miss * PRICE_MISS+ (u.completion_tokens or 0) * PRICE_OUT) / 1e6ok = run_checks(case.checks, resp.text) # automated checksif not ok: # fall back to the judge modelok = judge_model.score(case.task, resp.text) >= 0.8stats["pass"] += int(ok)stats["total"] += 1if ok:stats["score"] += case.weightstats["weight"] += case.weightstats["latency"].sort()stats["p95_latency"] = stats["latency"][int(len(stats["latency"]) * 0.95) - 1]stats["pass_rate"] = stats["pass"] / stats["total"]stats["weighted_score"] = stats["score"] / max(stats["weight"], 1)return stats
The script’s weighted_score, pass_rate, p95_latency, and cost are the core inputs to a selection decision. The judge model only supplements what automated checks cannot cover — subjective quality. It is worth turning your internal eval set into a team asset: keep versioned datasets in object storage, and observe token usage, latency, and error rates on the call path over time through logging, monitoring, and alerting.
3. Long-horizon and agentic tasks: the deep end of evaluation
3.1 From “generate a response” to “finish a job”
Long-horizon tasks are completed over dozens of turns and multiple steps, for example:
- code: implement a requirement in a project with hundreds of files and get all tests passing;
- research: search sources, cross-check them, and write a structured report;
- video: segment-wise understanding of a long video, ending in editing decisions.
When evaluating these, “the quality of one response” no longer tells the story. Look at:
- task completion rate: was the goal actually reached — tests passing, documentation complete;
- step efficiency: the number of turns and tokens needed to finish the same task;
- error recovery: after a compile failure or an abnormal tool return, can it diagnose and fix itself;
- consistency: are results stable across runs, or does it fall into useless loops.
3.2 Simulate long-horizon work with tasks that have exactly one right answer
To make long-horizon capability measurable, the industry commonly uses “repository-level repair” task sets:
- A code snapshot of a real open-source project at a specific historical commit;
- A natural-language description of a bug or a new requirement;
- The official patch and the accompanying tests held back;
- The model works in an isolated environment to locate and modify the code, and the tests are run to decide pass or fail.
The answer is unambiguous and grading is automatic, which avoids subjective scoring and separates “can write a code snippet” from “can deliver a correct change.”
3.3 Agent evaluation details
- Tool calls: are the function arguments valid, is the wrong tool being called, does it fail to stop when it should;
- Prompt injection risk: retrieved documents or web content may carry malicious instructions, so include adversarial samples on purpose;
- Observability and traceability: every step’s inputs and outputs should be traceable, so failures can be reviewed after the fact.
3.4 Typical scoring dimensions
| Task type | Core metrics | Suggested measurement |
|---|---|---|
| Frontend page reconstruction | Visual similarity, interaction completeness | Screenshot diff, human rating, automated interaction tests |
| Repository-level code repair | Hidden test pass rate | Run tests automatically in an isolated sandbox |
| Long-document research | Factual accuracy, traceable citations, structural completeness | Key-point checklist plus fact checking |
| Video understanding and editing | Hit rate on key segments, timestamp error | Comparison against human annotations |
| Tool-calling agent | Task success rate, average number of steps, unauthorized call count | Sandbox plus rule validation |
4. Access options: subscription or API calls
4.1 Two billing models
- Membership subscription: paid monthly or yearly in exchange for a fixed allowance or unlimited use of features such as a coding assistant, chat, and file upload. Suits individual developers and small teams whose usage is mainly interactive.
- Pay-as-you-go API: billed per token. Suits scenarios that must be embedded in product flows and orchestrated by code.
4.2 Pick by scenario
| Usage scenario | Recommended option | Why |
|---|---|---|
| Personal everyday coding help | Membership subscription | Steady usage rhythm, predictable cost |
| Team collaboration and consistent coding standards | Subscription or team plan | Unified toolchain and permission management |
| In-product document summaries and Q&A | Pay-as-you-go API | Needs to integrate with the business flow |
| Large batch offline jobs (code scanning, document classification) | Pay-as-you-go API (check batch discounts) | High volume, low real-time requirement |
4.3 Integration points
- Batching and async jobs: many services discount delay-tolerant batch requests, which suits offline work.
- Pair with retrieval: a very long context is not a universal fix. Retrieval-augmented generation (RAG) is usually a better balance of recall precision, cost, and freshness.
- Structured output: when you need the model to emit JSON Schema-conforming output, prefer a service that supports constrained decoding to lower parse failure rates.
- Model routing: send simple requests to a lightweight model and hand complex ones to the flagship. This is usually cheaper than “flagship for everything.”
- Model fallback: give critical paths at least two backup models and a retry policy so one service’s instability does not hit the business.
5. Cost modeling: cache hits are a hidden discount
5.1 Three-tier token pricing
| Line item | What it charges | Characteristics |
|---|---|---|
| Cache-hit input | Input tokens identical to an already cached prefix | Unit price is usually a fraction of the normal input price |
| Cache-miss input | All remaining input tokens | Normal input price |
| Output | Tokens generated by the model | Unit price is usually well above the input price |
The general mechanism: when a request’s prefix is exactly the same as a previous request (system prompt, fixed background knowledge, multi-turn history), the server can reuse the already computed attention state (KV cache) instead of recomputing, and those tokens are billed at the discounted rate. Prompt layout therefore converts directly into cost savings.
5.2 Five design points that decide cache effectiveness
- Keep the stable prefix first. Put the system prompt, tool definitions, and fixed background knowledge at the front, and user input, timestamps, and other dynamic content at the back.
- Do not pollute the prefix with volatile content. Injecting the current time or a random ID at the start of a conversation makes every request’s prefix different and invalidates caching entirely.
- Append incrementally within a session. In multi-turn conversations, only append new content — do not modify or reorder history.
- Manage cache lifetime explicitly. For stable background such as long documents or large code bases, set the cache TTL according to the service’s rules so it does not expire and get billed again.
- Measure before optimizing. Track the cache hit rate first, then restructure prompts.
5.3 Worked example
Assumptions (illustrative, not any provider’s actual pricing):
- Unit prices: CNY 1 for cache-hit input, CNY 4 for cache-miss input, CNY 20 for output, per million tokens;
- Typical interaction: about 4,000 input tokens, of which about 3,000 (system prompt plus fixed background) hit the cache, and about 800 output tokens;
- 10,000 calls per day, 90% cache hit rate.
Cost per call then works out to:
- Cached input: 3000 × 1 ÷ 1,000,000 = 0.003
- Uncached input: 1000 × 4 ÷ 1,000,000 = 0.004
- Output: 800 × 20 ÷ 1,000,000 = 0.016
- Total ≈ 0.023 per call, ≈ 6,900 per month over 30 days
Two things stand out:
- Output accounts for roughly 70% of per-call cost. Controlling generation length — a leaner system prompt, no rambling expansion — is more effective than compressing the input.
- Caching pays off. If caching fails completely, input cost rises from 0.007 to 0.016, per-call cost becomes 0.032, about 39% higher.
5.4 A reusable cost model
monthly cost ≈ daily calls × 30 × [(input tokens × hit rate × cached unit price)+ (input tokens × (1 - hit rate) × miss unit price)+ (output tokens × output unit price)] ÷ 1,000,000
Substitute real token usage from production logs and you can compare models and orchestration strategies on cost side by side.
6. Overall selection: one decision checklist
Turn the evaluation methods above into a checklist and tick it off item by item in the selection review:
| Dimension | Key question | How to verify |
|---|---|---|
| Spec fit | Does the context window cover your longest task? Do you need native multimodality? | Test with your longest real sample |
| Capability | Are weighted score and pass rate on your own eval set good enough? | Repeat three or more times and use statistics |
| Stability | What are P95 latency, error rate, and long-horizon consistency like? | Monitoring and alerts plus sandbox regression |
| Cost | Is the monthly bill within budget? Is the cache hit rate acceptable? | Cost model estimate plus invoice reconciliation |
| Compliance | Do the license, data residency, and log retention policy meet requirements? | Legal and security review |
| Evolvability | Is there a backup model, and can versions be switched smoothly? | Abstract a common call layer so the business is not tied to one specific interface |
Conclusion
Back to the opening question: how do you evaluate a trillion-parameter, million-token, natively multimodal flagship model? Not by a point or two on a leaderboard, but by whether it delivers consistently on your task distribution at a cost per task you can accept.
- Leaderboards position a tier; business data decides whether it stays.
- Specs define the boundaries, evaluation sets the floor, and a cost model guards the budget.
Drop the scripts and checklist from this article straight into your team’s toolchain and let data replace impressions. Model service platforms such as Baidu AI Cloud’s Qianfan ModelBuilder typically offer multiple model sizes alongside tool calling, knowledge base retrieval, and private deployment options, which makes horizontal comparison inside a single evaluation framework easier and lets a team gradually build up its own selection baseline.
