Evaluating Million-Token LLMs: From API Specs to Real-World Testing

Introduction: Specs Over Press Releases

As large language models settle into a rolling release cadence of “preview” and “general availability” (GA) versions, developers face a recurring question every month: Is a vendor’s newly announced GA version worth migrating to? Will the actual experience deliver a qualitative leap?

A highly recommended habit is to examine the API specification sheets before reading the launch narratives. Taking the public API documentation of a mainstream LLM as an example, comparing the preview and GA specs side-by-side reveals a representative set of data: the context window remains 1024k tokens, maximum input is 960k tokens, and maximum output is 64k tokens for both. The only real difference is that image comprehension—previously unlisted as a supported capability—is now officially supported.

This comparison highlights two things. First, “million-token context” is not a new capability of the GA version; it was already available during the preview stage. Second, the true incremental value of a version upgrade is often hidden in the changes to the “supported capabilities list” rather than the adjectives used in press releases. Following this thread, this article provides a complete evaluation methodology: “Read Specs → Test Empirically → Define Architecture.”

1. Understanding the Three Core Numbers: The Context Window Budget

The three most critical numbers in an API spec sheet are the context window, maximum input, and maximum output. Their engineering implications are straightforward:

• Context Window: The total token budget the model can “see” in a single call, shared between input and output.
• Maximum Input: The upper limit on the prompt side.
• Maximum Output: The upper limit on the generation side.

A common industry design follows simple arithmetic: Context Window ≈ Maximum Input + Maximum Output. In the spec example above, 1024k ≈ 960k + 64k, meaning the output budget only accounts for about one-sixteenth of the entire window. This “input-heavy, output-light” allocation is not arbitrary and is driven by three underlying reasons:

1. Decoding costs are significantly higher than encoding costs. Autoregressive generation is sequential. Every generated token must carry the entire context’s KV cache to participate in attention calculation. Under long inputs, the latency and memory overhead of generating tokens grow linearly—or even faster—with length.
2. Task positioning. The primary battleground for long-context models is “read extensively, answer accurately.” Outputs are mainly summaries, answers, and analyses. 64k tokens is sufficient to cover an in-depth report of tens of thousands of words.
3. Quality and abuse control. Unrestricted long-form generation tends to diverge. Compressing the output budget is a way to constrain uncertainty.

The direct takeaway for developers is: “Inputting an entire book and outputting an entire book” is not realistic under current architectures. If a task genuinely requires massive generation, the correct approach is task decomposition—outline first, expand section by section, and iterate over multiple rounds, rather than expecting a single API call to solve everything.

Another frequent question is token conversion. A rough heuristic is that English requires about 4 characters per token. For Chinese, heavily influenced by the tokenizer, 1 token roughly corresponds to 0.6 to 1.5 characters. Based on this, 1024k tokens can accommodate hundreds of thousands to a million characters, enough for a full-length novel, hundreds of pages of technical documentation, or a medium-sized code repository. However, relying on heuristics in engineering is discouraged. A safer approach is to call the platform’s token counting capability before sending the request:

def check_context_budget(prompt_tokens: int,
                         expected_output_tokens: int,
                         max_input: int = 960_000,
                         max_output: int = 64_000,
                         context_window: int = 1_024_000) -> None:
    """Validate budget before sending request to avoid truncation or errors."""
    if prompt_tokens > max_input:
        raise ValueError(f"Input exceeds limit: {prompt_tokens} > {max_input}")
    if expected_output_tokens > max_output:
        raise ValueError("Expected output exceeds max limit, consider chunked generation")
    if prompt_tokens + expected_output_tokens > context_window:
        raise ValueError("Input + expected output exceeds window, consider compressing input")

2. The Value of the Capabilities List: What “Unlisted” to “Supported” Means

In the preview specs, image comprehension was “unlisted as a supported capability”; in the GA version, it is “supported.” This single line change deserves elaboration:

• Unlisted ≠ definitely unusable. During the preview phase, experimental capabilities might be gradually rolled out. The API might work, but without guarantees.
• Unlisted = no quality commitment. Capabilities can be changed or taken offline at any time, billing rules are unclear, and bugs are not guaranteed to be fixed. Relying on such capabilities in a production system equates to betting your stability on someone else’s experiment.
• Supported = formal declaration of boundaries. It means the API contract, billing, and change notifications enter a formal management process.

The engineering rule is: Capability dependencies in production systems must be based on the “officially supported” list in the documentation.

After image comprehension transitions to official support, new scenarios combined with million-token context are worth exploring: long documents with mixed text and images (papers, financial reports, manuals), batch parsing of scanned documents and screenshots, long-form report analysis with chart values, and design-to-frontend-code conversion. Specific testing methods for these scenarios are expanded upon in Section 4.

3. Version Comparison Methodology: Diffing Two Spec Sheets

Returning to the initial data: why does the fact that “million-token context already existed” matter? It corrects a common misconception: equating “GA release” directly with “capability upgrade.”

The actual differences between preview and GA versions usually concentrate on five dimensions, rather than necessarily the model’s core capabilities:

Dimension	Preview	GA
Stability Commitment	No SLA	Usually provides availability guarantees
Rate Limiting & Quota	Restricted, subject to change	Clear, can apply for increases
Billing	May be free or discounted	Official pricing
Capabilities List	Some “unlisted”	Clear boundaries
Deprecation Policy	Can be taken offline anytime	Changes require advance notice

Therefore, for every “GA release,” it is recommended to adopt a fixed action: place the old and new API spec sheets side-by-side and diff them item by item. Taking the aforementioned case as an example, the diff results are clear:

Spec Item	Preview	GA	Increment Assessment
Context Window	1024k tokens	1024k tokens	No change
Maximum Input	960k tokens	960k tokens	No change
Maximum Output	64k tokens	64k tokens	No change
Image Comprehension	Unlisted	Supported	Key Increment

The conclusion is clear: the true increment of this upgrade is the officialization of multimodal capabilities, not long context. For teams already using million-token context, the real evaluation is whether to add an image comprehension pipeline. For teams not yet integrated, a full integration evaluation is required.

4. Testing Checklist: Subjecting Specs to Business Validation

Spec sheets are vendor promises; empirical tests are your own facts. For million-token context, it is recommended to organize evaluations across four levels.

1. Position-Bucketed Long-Text Recall Testing

A common industry practice is the “Needle in a Haystack” test: insert key facts at different positions within an extremely long text to verify if the model can accurately recall them. The core logic can be simplified as:

def build_haystack(total_tokens: int, position_ratio: float,
                   needle: str) -> str:
    """Insert key fact at position_ratio, fill the rest with noise text"""
    prefix_len = int(total_tokens * position_ratio)
    prefix = noise_text(prefix_len)
    suffix = noise_text(total_tokens - prefix_len - est_tokens(needle))
    return prefix + needle + suffix
# Grid test on "total length x insertion position", track recall rate at each position
for total in (128_000, 256_000, 512_000, 1_000_000):
    for ratio in (0.0, 0.25, 0.5, 0.75, 0.98):
        text = build_haystack(total, ratio, needle=KEY_FACT)
        assert recall(query(text, "Please find the key fact in the text"))

Two reminders: First, this type of test only validates “retrieval,” not “reasoning.” It must be supplemented with cross-chapter multi-hop QA, global summarization, and cross-paragraph consistency tasks. Second, industry research widely observes the “middle dropout” phenomenon—models utilize information in the middle of a document less effectively than at the beginning or end. Therefore, critical information should be positioned at the front or back in engineering, and testing should track statistics by position buckets.

2. Mixed Text-Image Capability Testing

Once image comprehension is officially supported, specialized testing should be conducted using business data: numerical extraction accuracy from table screenshots, chart trend descriptions, cross-referencing between text and images (“What is the data shown in Figure 3?”), and multi-image sorting and comparison. Scoring should combine manual spot-checks with automated metrics to form a regression-ready evaluation set.

3. Performance Degradation Curve

Long input is not free. Fix the output length and measure changes in three metrics as input scales from 8k, 32k, 128k, to 512k: Time To First Token (TTFT), Time Per Output Token (TPOT or throughput), and request success rate. Plotting the results will yield the “true performance profile” of the model under your specific account quota.

4. Worst-Case Cost Estimation

For token-billed models, API costs scale linearly with input length. Always calculate the worst-case cost of a single call based on “maximum input + full output,” then multiply by estimated call volume. Many teams discover during stress testing that million-token context is technically feasible but economically unviable.

5. Architectural Trade-offs: Million-Token Context is Not a Silver Bullet

After passing the evaluation, three decisions remain before deployment.

Long Context or Retrieval-Augmented Generation (RAG)? The industry consensus is that they are complementary, not mutually exclusive. Long context is suited for deep single-document reading, cross-chapter consistency verification, and global refactoring of small-scale codebases. RAG is suited for massive corpora far exceeding the window size, frequent updates, high concurrency, and cost-sensitive scenarios. A more mature approach is a hybrid architecture—first use retrieval to narrow down candidate ranges, then feed the candidate paragraphs into a long-context model for deep reading, balancing recall and cost.

What if Output is Restricted? For long output requirements, use Map-Reduce style segmented summarization, outline-first then section-by-section expansion, and multi-round iterative refinement. The 64k token output limit is enough to accommodate an in-depth report, but you need to proactively plan the generation structure rather than requesting it all at once.

How to Build Supporting Engineering? Productionization requires API logging, distributed tracing, monitoring alerts, and canary rollbacks. These capabilities can rely on general-purpose cloud product combinations: use object storage to cache long document originals and sliced results, use logging services to record token consumption and latency distribution per call, and use monitoring alerts to track costs and error rates. If you choose to deploy on an integrated platform like Baidu AI Cloud, the combined use of model services and the aforementioned infrastructure can significantly reduce self-built operational costs.

6. A Ready-to-Use Evaluation Checklist

• Diff old and new version spec sheets item by item to confirm true increments;
• Verify that all capability dependencies are on the “officially supported” list;
• Validate budget using “Maximum Input + Expected Output ≤ Context Window”;
• Run position-bucketed recall tests using real business documents, focusing on middle-segment performance;
• Supplement with multi-hop QA and global summarization tasks to avoid testing retrieval only;
• Plot TTFT and throughput degradation curves relative to input length;
• Calculate single-call cost based on worst-case scenarios and multiply by estimated call volume;
• Prepare alternative plans for model capability degradation or switching.

Conclusion

Version releases follow a business rhythm; spec sheets and empirical data are technical facts. The term “GA” deserves respect—it signifies formal commitments regarding stability, billing, and capability boundaries. However, where the actual increment lies must always be proven by side-by-side diffs and your own evaluation sets. By institutionalizing “Read Specs → Test Empirically → Define Architecture” as a fixed team process, no matter how LLM version numbers roll, you can instantly determine: what does this upgrade mean for me?