Evaluating Flagship LLMs: Capabilities, Boundaries, and Selection Guide

1. Introduction: Why “Evaluation Criteria” Matter

Whenever a flagship large language model is released, terms like “most powerful,” “topping the charts,” and “first-tier” become standard marketing language. However, for developers and enterprise technical decision-makers, the real question is never whether it is “the strongest.” Instead, it boils down to three specific questions:

1. For which task types is it practically usable, and where is it still just a “demo-level” toy?
2. When official benchmarks and third-party evaluations conflict, who should you trust, and why?
3. Where is the cost-effectiveness inflection point within the same price range?

This article uses a recently released 770B-level MoE flagship model as a sample, analyzing it across five dimensions: architectural parameters, evaluation criteria, multimodal generation, visual understanding, and agent operations. It does not endorse any specific vendor’s product but aims to establish a reusable methodology for evaluating flagship models.

---

2. The Basics: Parameter Scale, Capability Increments, and Cost Structure

2.1 Architecture and Parameter Criteria

The model uses a Mixture of Experts (MoE) architecture with total parameters of 770B and activated parameters of about 49B per inference. Understanding this distinction is critical:

• Total parameters determine the model’s “knowledge capacity limit” and training cost.
• Activated parameters determine the compute overhead and response cost per inference.

The core value of MoE lies in decoupling these two aspects—using massive capacity to store knowledge while using small activations to control costs. The 770B total parameters represent a roughly 2.6x scale increase over the previous generation’s sub-300B series, making it the largest model specification in the vendor’s history.

It is important to note that large MoE parameters do not linearly equate to “strong capabilities.” Routing strategies, expert granularity, training token volume, and data mixture often matter more than total parameter count. Therefore, when seeing “770B,” a rational reaction is to focus on the activation ratio (about 6.4% in this case) and inference cost, rather than directly comparing it with the parameter count of dense models.

2.2 Three Key Capability Increments

Capability Dimension	Specification	Engineering Significance
Context Length	1 million tokens (API)	Supports one-shot input of long documents and large codebases, reducing fragmentation and retrieval orchestration costs
Visual Understanding	First-time introduction	Can process image inputs, chart parsing, and UI screenshot comprehension
Computer Use	First-time support	Can execute cross-application tasks via command line and graphical interfaces

The latter two represent the most significant increments of the official release over the preview version. In engineering terms: The preview version was a pure text reasoner, while the official release features perception channels (vision) and execution channels (operation). This marks an evolution from a “content generation model” to a “task-completion agent foundation.”

However, we must be clear-eyed: visual understanding and Computer Use are still in early stages industry-wide. Stability, error accumulation in long-chain tasks, and the security boundaries of operational permissions are pressing issues for real-world deployment. These two capabilities should be evaluated as “verifiable increments” rather than relied upon as hard dependencies in production environments.

2.3 Cost Structure: Why Unit Price is a Key Variable

The model’s API pricing is set at 6 units of currency per million input tokens, 18 units per million output tokens, and 0.3 units for cache hits, matching the preview version. In contrast, comparable overseas flagship models typically price outputs at around $10 per million tokens—roughly 4 times more expensive when converted.

For developers, the practical significance of this price gap manifests in three scenarios:

• High-frequency invocation scenarios (e.g., online Q&A, code completion): Unit price directly determines gross margin.
• Long-context scenarios (e.g., repository-level code review): A single 1-million-token context call can cost several units of currency, making the cache hit discount (0.3 units/million tokens) a crucial optimization point.
• Agent multi-turn scenarios: Computer Use tasks often require dozens of interactions, each carrying historical context, causing costs to scale almost linearly.

A practical cost estimation model:

# Cost estimation example (unit: local currency)
def estimate_cost(input_tokens, output_tokens, cached_tokens=0,
                  p_in=6/1e6, p_out=18/1e6, p_cache=0.3/1e6):
    fresh_in = input_tokens - cached_tokens
    return fresh_in * p_in + cached_tokens * p_cache + output_tokens * p_out
# Scenario: Repository-level code review, 500k input tokens, 400k cache hits, 3k output tokens
print(round(estimate_cost(500_000, 3_000, 400_000), 4))  # Approximately 0.72 units

As shown, cache hit rate is the primary cost lever in long-context scenarios. Architecturally, you should strive to keep prefixes stable (fixed system prompts, fixed code context order) to maximize cache reuse.

---

3. The Benchmarking Debate: Why Official Scores and Third-Party Leaderboards Clash

A typical phenomenon occurs: the official announcement claims to have “entered the ranks of the strongest open-source models for the first time,” yet a third-party evaluation reports a comprehensive intelligence index 11 points lower than a peer competitor, with hallucination rates flagged as “high.”

This discrepancy is not an isolated incident; it is an industry norm. Understanding its root causes is far more important than memorizing a specific score.

3.1 Four Sources of Discrepancy

1. Benchmark Composition Differences: Vendors typically select benchmarks that align with their training data distribution or highlight their strengths (e.g., math, code, long text). If third-party leaderboards prioritize open-ended reasoning, instruction following, or factual Q&A, conclusions may differ entirely.
2. Prompts and Scoring Strategies: The same model’s score can fluctuate by several percentage points under different prompt templates or answer extraction rules. 0-shot vs. few-shot, the use of chain-of-thought, and tool-calling permissions all alter rankings.
3. Hallucination Measurement Criteria: There is no unified definition of hallucination. Is it “contradicting the given context,” “conflicting with objective facts,” or “unverifiable”? High/low labels can contradict each other under different criteria.
4. Version and Configuration Drift: If the evaluated model version, reasoning intensity level, or decoding parameters (temperature, top_p) are not disclosed, results cannot be reproduced.

3.2 Practical Advice for Developers

• Do not make selection decisions based on a single leaderboard. Cross-reference at least 2 to 3 different sources, and review their evaluation methods and raw samples.
• Build a small-scale golden dataset. Prepare 50 to 200 real samples tailored to your business scenarios, including inputs, expected outputs, and scoring rules. This is the only “leaderboard” that truly belongs to you.
• Distinguish between “capability” and “reliability.” The former looks at the best result; the latter looks at the distribution tail. Production environments should focus on P95 performance and failure modes.
• Treat hallucination rates as engineering risks requiring fallbacks, rather than outright dealbreakers. Retrieval-Augmented Generation (RAG), structured output constraints, and citation tracing are universal mitigation strategies.

A minimal viable custom evaluation script skeleton:

CASES = [{"prompt": "...", "expect_keywords": ["..."], "must_not": ["..."]}]
def score(resp, case):
    s = 0
    s += 1 if all(k in resp for k in case["expect_keywords"]) else 0
    s -= 1 if any(k in resp for k in case["must_not"]) else 0
    return s
# After batch testing, focus on the mean and worst percentiles, not single best runs

---

4. Hands-on Test 1: From a Single Prompt to an Interactive 3D Scene

One test demonstrated generating a low-poly style 3D mini-game from a single prompt, including elements like scenes, characters, winches, and hooks, directly compared against a peer competitor.

From an engineering perspective, this type of task simultaneously tests four capabilities:

1. Structured Code Generation: Must produce a runnable single HTML/JS file containing render loops, input handling, and collision/grasping logic.
2. State Machine Modeling: The extension, retraction, grab detection, and scoring of the hook essentially form a small state machine.
3. Resource-Constrained Trade-offs: Without external assets, it must substitute textures and model files with procedural geometry and inline styles.
4. One-shot Runnability: The output cannot just “look correct”; it must actually run.

4.1 Key Points in Prompt Design

To ensure stable, runnable outputs from a “single prompt,” the prompt must explicitly constrain the tech stack and acceptance criteria:

Generate a single HTML file using vanilla JS and Canvas 2D (no external dependencies).
Requirements:
1) Implement a complete loop of "hook swing - extension - grab detection - retraction and scoring";
2) Adopt a low-poly/flat visual style using procedural geometry;
3) Include start, timer, and score UI;
4) Code must be self-contained and runnable directly in a browser.

Key takeaway: “Runnable” must be explicitly written in the prompt, otherwise the model tends to output fragmented snippets.

4.2 The Right Way to Conduct Comparative Testing

The original text compared the model with a competitor. Here, we must emphasize methodology: comparisons should follow the “three sames” principle—same prompt, same configuration, same scoring rules—and evaluation dimensions must be separated:

Dimension	Evaluation Method
Runnability	Open directly, record number and types of errors
Feature Completeness	Check requirement list item by item
Code Quality	Structure clarity, readability, presence of logical flaws
First-Try Success Rate	Proportion of generations requiring no manual fixes across multiple runs

A single case’s success or failure is insufficient to form an overall judgment of a model. The sample size must cover at least 10 similar tasks to be statistically meaningful.

---

5. Hands-on Test 2: Hand-drawn Sketch to Runnable Webpage

Visual understanding is the core increment of this official release. Converting a hand-drawn wireframe into a webpage is a highly representative validation task because it simultaneously tests:

• Layout Structure Recognition: Can it correctly segment navigation, sidebars, card areas, and forms?
• Text Recognition and Semantic Placement: Annotated text in the sketch should fall into the correct DOM positions.
• Component Semantic Mapping: A box with rounded corners should map to a card; a long rectangle with an underline should map to an input field.
• Responsive Inference: Sketches usually have a single width; the model must independently deduce reasonable breakpoints.

5.1 Acceptance Checklist

Instead of relying on “whether it looks right,” use a structured checklist for acceptance:

[ ] Structure matches the sketch's block count, no omissions or fabrications
[ ] Text content matches sketch annotations (reasonable placeholders allowed)
[ ] Uses semantic HTML tags instead of purely stacking <div>s
[ ] Contains at least one responsive breakpoint
[ ] No external dependencies, or dependencies are controllable
[ ] Styles and structure are separated, readability meets standards

5.2 Known Common Failure Modes

• Over-design: The sketch is low-fidelity, but the model adds gradients, shadows, and animations, deviating from the goal of “faithful reproduction.”
• Text Fabrication: Guessing illegible annotations into seemingly reasonable but business-irrelevant copy.
• Structure Hallucination: Fabricating content for non-existent blocks (a concrete manifestation of “high hallucination rates” in multimodal scenarios).
• Ignoring Accessibility: Missing basic items like alt, label, and contrast ratios.

The engineering response is a two-stage prompt: first, ask the model to output a structured layout description (block tree + text list). After manual or rule-based validation, generate the code. This step significantly reduces structural hallucinations.

Step 1: Output only the JSON layout tree, do not generate code.
{"blocks":[{"type":"header","text":"..."},{"type":"form","fields":[...]}]}
Step 2: (After validation passes) Generate semantic HTML + CSS based on the JSON above.

---

6. Computer Use: Value and Boundaries

The model supports computer operation capabilities for the first time, enabling it to execute cross-application tasks via command lines and graphical interfaces. This is a crucial step from “conversation” to “execution,” but three points must be clarified for deployment:

1. Not suitable for unsupervised high-privilege operations. Actions involving accounts, payments, or data deletion must include a human confirmation step.
2. Error accumulation in long-chain tasks. Assuming a 95% single-step success rate, a 20-step task has an end-to-end success rate of about 36%. Therefore, checkpoints and rollback mechanisms are necessary.
3. Observability first. Screenshots, commands, and return codes for each step must be logged, otherwise failures cannot be traced.

The recommended deployment form is a human-in-the-loop semi-automated process: the model handles planning and routine execution, while critical nodes are handed over for human confirmation.

---

7. Selection Guide: Who is it for?

Combining architecture, cost, and hands-on performance, we can offer the following scenario-based selection insights:

When to prioritize trying it out

• Batch processing tasks requiring ultra-long context, such as repository-level code review or long document structured extraction, where cache hit rates can be boosted via prefix stability.
• High-frequency invocation scenarios sensitive to unit price, where cost advantages are amplified by volume.
• Assistive tasks requiring multimodal input, such as sketch-to-prototype or chart interpretation, backed by a human review process.
• Exploratory AI agent projects looking to low-cost validate the feasibility of Computer Use-style workflows.

When to proceed with caution

• Scenarios with zero tolerance for factual inaccuracies: build custom evaluation sets and pair them with RAG and citation tracing.
• Production pipelines requiring stable reproduction: conduct small-scale canary testing and baseline failure rate measurements first.
• Scenarios requiring strong ecosystem and toolchain support: assess whether the accompanying monitoring, logging, evaluation, and governance capabilities are comprehensive.

General Principles for Decision-Making

1. Run scores on a custom golden dataset first; then look at public leaderboards.
2. Calculate the unit task cost (including retries and manual fixes) before discussing unit price.
3. Validate failure modes before evaluating average capabilities.
4. Treat multimodal and agent capabilities as “incremental gains,” not “baseline dependencies.”

---

8. Conclusion

The competition among flagship models has entered a phase where “parameters are no longer scarce, but evaluation criteria are key.” The 770B scale, 1-million-token context, introduction of visual and operational capabilities, and a unit price significantly lower than peers collectively form a candidate positioned favorably on the cost-capability curve. However, phrases like “strongest open-source model” remain marketing language. What truly matters to developers is a reproducible evaluation process and a golden dataset of their own.

Treat benchmarks as clues, custom evaluations as evidence, and failure modes as design inputs—this is the most prudent stance when facing every generation of flagship models.