Million-Token Context Windows in Practice: How to Evaluate and Choose a Model
A million-token context window is not automatically a million tokens of reliable understanding. This guide explains how long-context architectures work, how to test information retrieval, repository-scale refactoring, and frontend generation, and how to compare reliability, cost, and operational risks using your own data. All example figures are templates or hypothetical values and should be replaced with measured results.
From “Can Fit” to “Can Use Reliably”
Large language model context windows have expanded from 4K and 32K tokens to 128K, and many models now advertise support for up to one million tokens. However, the advertised window and the practically usable window are not the same thing. An input may fit entirely within the limit without the model reliably retrieving information from every part of it. Many models begin to miss content toward the end of long inputs, with recall declining as the sequence grows.
Model selection should therefore be based on your own test data. Published specifications are only a starting point. This article does not rank specific models or cite unverified benchmark results. All example numbers are hypothetical or intended as test templates; replace them with your own measurements.
1. Why Long Context Has Become Standard
Standard Transformers use full self-attention. The computation required for attention scores grows quadratically with sequence length, while GPU memory usage for the key-value cache grows linearly.
Considering attention-score computation alone, increasing a sequence from 40,000 tokens to 1 million tokens multiplies its length by 25 and the computation by approximately 625.
This cost curve made long-text processing expensive for years. Over the past two years, code assistants, document question answering, and AI agent workflows have created rapidly growing demand for long inputs. Model vendors have responded with architectural changes designed specifically for long-context processing. As a result, long context has shifted from a flagship feature to a standard capability in many new models.
2. Architectural Changes
Recent models mainly introduce changes in two areas: attention mechanisms and training optimizers.
2.1 Attention Mechanisms
The table below summarizes several common attention patterns. Here, n represents sequence length and w represents the sliding-window size.
| Attention type | Computational complexity | Key-value cache | Main advantage | Main cost |
|---|---|---|---|---|
| Full attention | O(n²) | Grows with sequence length | Most complete information retention | High cost for long sequences |
| Sliding-window attention | O(n·w) | Retains only the window contents | More stable memory usage | Information outside the window must be passed through other layers |
| Sparse attention | Depends on selected tokens | Depends on the implementation | Can select important distant positions | Incorrect selection can lose information |
| Compressed attention | Decreases with the compression ratio | Significantly reduced | Preserves a global overview | Details may be averaged away |
| Linear attention | O(n) | Fixed-size state | Lowest resource usage for long sequences | Weaker precise retrieval |
Most new models do not rely on only one attention type. Instead, they interleave different layer types in specific proportions. One common strategy is to use some layers for detailed modeling of nearby content and others to compress distant history while maintaining global semantics. This creates a balance between retrieval quality and computational cost.
The exact layer types, interleaving ratios, and terminology should be verified in the model’s official technical report. Different vendors may use different names for similar structures, so the architecture’s behavior cannot be inferred from its name alone.
2.2 Optimizer Changes
Traditional large-model training commonly uses AdamW. Some newer models use Muon-like optimizers during training. Muon applies approximately orthogonalized updates to two-dimensional weight matrices and is often used for matrix parameters in hidden layers. Embedding layers, output layers, and normalization parameters generally continue to use AdamW.
Some published experiments have reported faster convergence with Muon-like optimizers. However, the result depends on model size, learning rate, and training data, and cannot be generalized to every scenario. Determining whether a model uses this type of optimizer also requires consulting its technical report.
For users, the direct impact of the optimizer is training efficiency. Under the same compute budget, a more thoroughly trained model will often produce better outputs. Whether that benefit exists for a particular model must be confirmed through your own evaluation.
3. How to Test Three Common Scenarios
3.1 Deep Retrieval
Deep-retrieval testing measures whether a model can find information placed at different positions in a long text. A common approach is a “needle-in-a-haystack” test.
- Prepare a long input containing a mixture of real documents, code, and historical records. Set the total length to the target tiers, such as 32K, 128K, 256K, and 1M tokens.
- Insert anchor facts at different depths. Possible positions include 0%, 25%, 50%, 75%, and 95%. Each anchor should use a fictional API name or parameter value that does not appear elsewhere in the source text.
- Repeat every length-and-depth combination at least 20 times and record whether the model answers correctly.
- Define pass criteria. For example, you might require recall of at least 95% at every depth and at least 90% at the worst-performing depth. These are suggested values and should be adjusted to your business’s tolerance for errors.
- Save failure cases and distinguish among three categories: the model completely missed the information, returned incorrect content, or confused one anchor with another. These failures have different causes and may require different fixes.
As an engineering rule of thumb, some teams limit actual input to 70%–80% of the advertised context window to leave room for output and reasoning. This is not a universal constant. The most reliable approach is to run the test above on your own tasks and identify the length at which performance begins to decline.
3.2 Repository-Scale Refactoring
One of the main benefits of long context is the ability to load an entire repository and understand dependencies across files. Traditional chunking can lose cross-file references, while full-input processing may reduce these omissions.
Before testing, record the repository size, including the number of files, total token count, and dependency depth. Then define the task, such as standardizing names, splitting modules, or migrating an interface.
Recommended metrics include:
- Compilation or type-check success rate
- Unit-test success rate
- Number of files requiring manual changes
- Number of missed references
When comparing approaches, require full-input and chunked-input workflows to complete the same task and use the same evaluation scripts.
| Approach | Repository files | Compilation passed | Tests passed | Files requiring manual changes | Missed references |
|---|---|---|---|---|---|
| Full input | |||||
| Chunked input |
The values in this table should be filled in with your own results.
3.3 Frontend Generation
Asking a model to generate a personal blog page from a simple prompt can provide a quick view of its interface-generation capabilities. Subjective impressions can be misleading, so use fixed scoring dimensions:
- Layout: Is the information hierarchy clear, and are the modules aligned?
- Color: Does the contrast between text and background meet the WCAG AA standard, with a minimum ratio of 4.5:1 for normal body text?
- Responsive behavior: Does the page render correctly at widths of 375px, 768px, and 1440px?
- Executability: Can the code be opened and run directly without errors?
Invite three to five reviewers. Have each reviewer score every dimension from 1 to 5, and define an average score of at least 4 as a pass. Because aesthetic judgments are subjective, also report agreement among reviewers; otherwise, the conclusion will be difficult to interpret.
4. Model Selection and Cost Control
4.1 Selection Guidelines
- Test before selecting. Use your own documents and code to run deep-retrieval and refactoring tests before choosing a model.
- Distinguish flagship and lightweight versions. Complex reasoning and long-document analysis are better suited to larger models. Simple, high-concurrency tasks may be better served by lightweight models with fewer active parameters, which generally offer advantages in cost and speed.
- Integrate with coding tools. Many command-line coding agents support custom model interfaces. Using the same tool to switch between models makes task-level comparisons easier.
- Control costs. The expense of long inputs should not be overlooked. Three direct cost-control measures are described below.
4.2 Cost Calculation
The cost of a single request can be estimated with this formula:
Cost = uncached input tokens × input price + cached input tokens × cache price + output tokens × output price
Assume an input price of 10 yuan per million tokens, a cache-hit price of 1 yuan per million tokens, and an output price of 40 yuan per million tokens. Suppose one request contains 500,000 input tokens, of which 450,000 are cache hits, and produces 20,000 output tokens.
- Uncached input: 0.05 million × 10 yuan = 0.50 yuan
- Cached input: 0.45 million × 1 yuan = 0.45 yuan
- Output: 0.02 million × 40 yuan = 0.80 yuan
- Total: 1.75 yuan
These prices are hypothetical. Use the service provider’s official pricing page for actual calculations.
4.3 Ways to Reduce Costs
- Caching strategy: Put stable content, such as system prompts and project instructions, at the beginning of the input. A consistent prefix is required for cache hits.
- Segmented summarization: When the history exceeds a defined length, compress it into a summary before adding new content.
- Cost limits: Set a maximum token budget for each task and configure daily budget alerts. This helps prevent an agent from repeatedly reading a large repository and generating an unexpectedly high bill.
5. Risks to Consider
- False retrieval: Similar passages in a long context may be confused with one another. Verify critical conclusions against the original text.
- Uncontrolled costs: Automated tools may repeatedly read the same files, causing spending to increase faster than expected.
- Privacy and security: Repositories and documents often contain secrets, customer data, and personal information. Redact sensitive content before submission, remove credential files such as
.env, and review the provider’s data-retention and training policies. - Copyright: When using another party’s documentation or code, preserve its source and license information.
6. Conclusion
Long context has evolved from a marketing specification into an engineering capability. Architectural changes have made this progress possible, but only testing can determine whether a model is suitable for a particular business use case.
Start with deep-retrieval tests to identify a reliable context length. Then use repository-scale refactoring and frontend-generation tests to evaluate practical performance. Finally, estimate long-term spending with the cost formula. Preserve the test records so they can be reused when evaluating future models.
