LLM Selection in Practice: Evaluating Parameters, Cost, and Multimodal Capabilities

1. Why Do Benchmark Scores Conflict?

Whenever a new large language model is released, the official announcement usually reads “ranks among the strongest open-source models,” yet third-party evaluation results often tell a different story: composite intelligence scores differ by a dozen or more points, hallucination rates are flagged as “high,” and task-specific rankings don’t match the official narrative.

This discrepancy is not accidental. The root causes include:

• Different evaluation sets: Vendors tend to pick benchmarks close to their training data distribution, while third parties focus more on generalization and adversarial samples.
• Different metric definitions: Some leaderboards emphasize knowledge QA, others code generation, and some count “safety refusals” as deductions.
• Inconsistent inference configurations: Whether chain-of-thought is enabled, how reasoning intensity is set, and the sampling temperature all significantly affect results.

For developers, the real question is not “who ranks first,” but whether a model’s cost, latency, and accuracy meet acceptable thresholds in the scenarios you care about. Below, we use a typical flagship MoE model as an example to break down a reusable evaluation framework.

2. Parameters and Architecture: What Does 770B Total Parameters Mean?

The model uses a Mixture-of-Experts (MoE) architecture with 770B total parameters and roughly 49B activated per inference. Understanding the difference between these two numbers is essential:

• Total parameters determine the model’s knowledge capacity ceiling, affecting training cost and storage requirements.
• Activated parameters determine the compute and latency of a single inference, directly affecting API response speed and billing.

For reference, the previous generation in the same series had fewer than 300B total parameters; the jump to roughly 2.6x represents a generational leap. The core advantage of MoE is that it uses a large parameter count for strong knowledge representation while keeping activation small to control inference overhead.

Key accompanying capabilities:

Capability	Description
Context length	1 million tokens (per API specifications), enough to process long documents or large codebases in a single pass
Visual understanding	Image input, chart analysis, and UI screenshot parsing
Computer Use	Executes cross-application tasks by combining the command line with graphical interfaces

Visual understanding and Computer Use are the biggest additions compared with the preview version. An analogy: the preview version was “a top student who could only read text,” while the full version “grows eyes and hands” — it can see interfaces and operate toolchains.

3. Cost Accounting: Does Cheap Mean Cost-Effective?

API pricing is a key variable in model selection. This model uses three pricing tiers (in RMB):

• Input: about ¥6 per million tokens
• Output: about ¥18 per million tokens
• Cache hit: about ¥0.3 per million tokens

Compared with flagship models in the same class, some overseas products charge the equivalent of more than ¥70 per million output tokens — a gap of roughly 4x. But cheap does not automatically mean cost-effective. Consider the following factors:

1. Effective throughput: If a model needs multiple retries to produce a usable result, the actual cost multiplies.
2. Cache hit rate: For long system prompts or fixed knowledge bases, cache hits can push costs down to a very low level.
3. Reasoning intensity: High-intensity thinking significantly increases output token counts, requiring a trade-off between quality and cost.

Here is a practical cost estimation framework in Python:

def estimate_cost(input_tokens, output_tokens, cache_hit_tokens,
                  price_in=6, price_out=18, price_cache=0.3):
    """
    Input, output, and cache-hit tokens are all priced per million tokens.
    """
    cost = (input_tokens / 1_000_000) * price_in \
         + (output_tokens / 1_000_000) * price_out \
         + (cache_hit_tokens / 1_000_000) * price_cache
    return round(cost, 4)
# Example: a single request with 500K input tokens, 20K output tokens, and 300K cache-hit tokens
print(estimate_cost(500_000, 20_000, 300_000))

4. Capability Testing: How to Design Your Own Evaluation Set?

Rather than agonizing over a dozen-point gap on third-party leaderboards, build a small evaluation set around your own business. Two directions are worth prioritizing.

4.1 Complex Generation: From One Sentence to a Runnable Prototype

Ask the model to generate a 3D scene or interactive prototype from a single natural-language description. This tests the combined abilities of spatial reasoning, code organization, and resource scheduling. For example, given the prompt “generate a low-poly cartoon-style mine scene with a miner, a winch, a hook, and rails,” check whether the output:

• Is structurally complete, with all components present
• Runs directly or needs only minor modifications
• Maintains a consistent style with no logical omissions

Tasks like this place high demands on instruction following and long-horizon planning, making them an effective litmus test for separating models that “can chat” from those that “can do real work.”

4.2 Visual Understanding: Hand-Drawn Sketch to Web Page

Upload a hand-drawn UI sketch and ask the model to produce the corresponding HTML/CSS code. Key evaluation points:

• Layout fidelity: Are element positions and hierarchy relationships accurate?
• Semantic understanding: Does it recognize buttons, input fields, navigation bars, and other components?
• Code quality: Does it use semantic tags? Is the output responsive?

Models whose visual understanding is still maturing often expose weaknesses at this step: they can describe an image but struggle to map it precisely into executable code.

5. Scenario Fit: Who Is This Model For?

Based on parameters, cost, and capability boundaries, the suitable scenarios for this class of flagship MoE models can be summarized as follows:

A good fit for:

• Document analysis and summarization involving very long context (contracts, research papers, large codebases)
• Cost-sensitive batch tasks that still require flagship-level generation quality
• Automation prototypes that need visual understanding and interface operation
• Serving as the underlying model for AI coding tools, assisting with code completion and refactoring

Use with caution:

• Factual QA scenarios highly sensitive to hallucination — pair with retrieval-augmented generation (RAG) and human review
• Real-time interactions with strict first-token latency requirements
• Production environments demanding very high stability — run A/B tests and prepare rollback plans first

A model selection checklist:

1. Define the task type: generation, understanding, or operation?
2. Calculate per-call cost and acceptable latency.
3. Build a private evaluation set of 20–50 business samples.
4. Compare 2–3 candidate models across accuracy, cost, and latency.
5. Roll out with a small canary traffic and continuously monitor hallucination rates and user feedback.

6. Conclusion

There is no single “right answer” in LLM selection. The gap between official benchmarks and third-party evaluations ultimately reflects differences in evaluation goals and scenario distributions. Developers should return to their own business: validate capabilities with private data, account for costs with a cost model, and control risk with canary releases. Flagship MoE models have made real progress in parameter scale, context length, and multimodal capabilities, opening new possibilities for complex applications — but whether they deliver in production depends on whether you have built a rigorous evaluation and iteration mechanism.