From Parameter Race to Memory Race

Over the past few years, competition among large language models has shifted along two dimensions. The first was parameter scale, growing from hundreds of billions to trillions. The second is context length, expanding from early 4K and 8K windows to hundreds of thousands and now millions of tokens. Recently released flagship models commonly highlight trillion-level parameters and million-token context as core capabilities, and domestic Chinese models have also climbed the rankings on several composite intelligence benchmarks.

For developers, the value of a longer context is easy to grasp:

• A 300,000-word novel translates to roughly 400,000–500,000 tokens;
• The core code of a mid-sized repository often runs into hundreds of thousands of tokens;
• An agent running for weeks can easily accumulate hundreds of thousands of tokens in trajectory and tool-call records.

In other words, a million-token context means a model can, for the first time, read an entire book rather than an excerpt, and see the whole picture rather than guess from fragments. But scaling the nominal context from 8K to 1M is far more than changing a window parameter. It involves a full technical chain spanning algorithms, model architecture, and inference systems. This article follows that chain.

Three Technical Challenges of Million-Token Context

1. Quadratic Complexity of Attention

In autoregressive architectures based on attention, the compute cost of self-attention scales quadratically with sequence length. Expanding sequence length by 100x (from 10,000 to 1,000,000) inflates the attention portion of compute by roughly 10,000x. This shows up directly in two key metrics:

• Time to First Token (TTFT): The prefill phase must read the entire input in one pass. With dense attention over a million tokens, even a large GPU cluster needs substantial time;
• Throughput and cost: Training is constrained by the same quadratic complexity. Compute costs for long-text training rise sharply, and data mixing ratios and training strategies must be redesigned.

2. The KV Cache Memory Wall

During autoregressive generation, the system caches the Key/Value pairs for each token to avoid recomputation. KV cache memory usage can be estimated with a general formula:

KV memory ≈ 2 × layers × KV heads × head dim × sequence length × bytes per element

For a typical configuration with 64 layers, 8 KV heads, 128 dimensions per head, and FP16 precision, each token requires about 0.25 MB of cache. A single million-token request would consume roughly 250 GB of GPU memory—and that is just one concurrent request. The memory wall is the most direct and rigid bottleneck for long-context inference.

3. Position Extrapolation and “Lost in the Middle”

Even with enough compute and memory, two subtler problems remain:

• Position encoding extrapolation: Models are trained on limited position ranges (e.g., 32K). Feeding million-scale position indices at inference time rapidly distorts the attention distribution, producing irrelevant answers or repetitive output;
• Lost in the middle: Multiple studies have observed that information at the beginning and end of a long context is recalled well, while key information buried in the middle is easily overlooked. A nominal 1M context does not equal an effective 1M context.

Key Technical Solutions

1. Extending Position Encoding

Mainstream models generally use rotary position encoding and extend its effective range through two approaches:

• Interpolation: Compressing the positions of very long sequences back into the training length range, similar to recalibrating the markings on a ruler;
• Frequency scaling: Applying differentiated scaling to different frequency components—preserving local precision at high frequencies and extending global range at low frequencies.

On top of this, curriculum-style training—incrementally increasing training length from short to long while mixing in high-quality long-text data (books, code repositories, multi-turn dialogue trajectories)—helps the model expand its window without degrading general capabilities.

2. Sparse and Hierarchical Attention

Dense attention computes pairwise relationships for all tokens, but in practice much of that attention is redundant. Common optimizations include:

• Sliding window plus global anchors: Computing local details with a small window while retaining a few global tokens (system prompts, chapter summaries) to carry long-range information;
• Block-sparse and hierarchical attention: Splitting the sequence into blocks, applying full attention within blocks, and letting blocks interact through summaries or top-level tokens—reducing quadratic complexity to approximately linear;
• Hardware-aware operator optimization: Reducing memory reads and writes through operator fusion and IO-aware scheduling. Even when theoretical complexity is unchanged, actual speed can improve several-fold.

3. KV Cache Reduction and System Management

• Grouped-query attention (MQA/GQA): Multiple query heads share a single set of KV heads, compressing KV memory by an order of magnitude;
• Paged memory management: Managing KV cache in blocks much like an operating system manages memory pages, reducing fragmentation and improving concurrency;
• KV quantization and hot/cold tiering: Applying low-bit quantization to the cache or tiering hot and cold data across levels, trading a small amount of precision for several times the capacity;
• Context caching (prefix caching): Reusing KV computation results for repeated common prefixes (system prompts, knowledge bases, code headers), significantly reducing TTFT and redundant computation—especially critical for multi-turn agent applications.

4. Trillion-Scale Parameters and Mixture-of-Experts

Trillion-level parameter counts usually do not mean dense stacking. Instead, they rely on mixture-of-experts architectures that activate only a subset of parameters per inference. This provides stronger knowledge and capability reserves in total capacity while keeping per-inference compute within acceptable bounds. Parameter scale determines the capability ceiling; context length determines the information boundary. Together they define how complex a problem a model can solve.

5. Long Context and RAG: Complementary, Not Substitutive

It is worth emphasizing that long context is not a replacement for retrieval-augmented generation (RAG):

• Retrieval narrows the scope: Quickly locating relevant passages from massive knowledge bases—low cost, easy to update;
• Long context deepens understanding: Performing cross-paragraph reasoning and global synthesis within sufficient information bandwidth.

A mature approach combines both: retrieval first to coarsely filter highly relevant material, then long-context models for close reading and synthesis—balancing quality and cost.

Interpreting Intelligence-Index Benchmarks

1. Common Dimensions of Composite Evaluation

Intelligence-index leaderboards published by international evaluation organizations typically aggregate scores across reasoning, mathematics, coding, knowledge QA, and instruction following. Some also distinguish “pure model capability” from “tool-augmented capability.” Chinese models have steadily risen in these rankings in recent years, reflecting rapid progress in training methods and data engineering.

2. Long-Context-Specific Evaluation

For long context, commonly used evaluation methods include:

• Needle-in-a-haystack testing: Embedding key “needle” information in an extremely long “haystack” text to test accurate recall, often upgraded to multi-needle and multi-hop versions;
• Long-document QA and summarization: Testing cross-chapter information integration;
• Long-range code understanding: Providing repository-level context to test cross-file dependency analysis.

3. Three Cautions for Reading Leaderboards

• Beware of data contamination: If evaluation questions leak into training data, scores will be inflated;
• Watch variance and prompt sensitivity: The same model may score noticeably differently under different prompt templates;
• The capability-cost-latency triangle: Ranking first does not mean optimal for your business. Production selection must factor in unit price, TTFT, and throughput together.

For developers, the most reliable approach is: use leaderboards to narrow the candidate range, then build a small-scale evaluation set from your own real business data for the final decision.

Typical Application Scenarios and Engineering Practices

1. Long-Document Understanding and Summarization

Contract review, research report analysis, and book summarization all benefit directly. Practical recommendations:

• Organize input with hierarchical structured formats (headings, numbering, tables) to help the model build a sense of table of contents;
• Put key requirements first and explicitly instruct the model to “cite original passages when answering” to reduce hallucination;
• For very long material, adopt a two-stage strategy of “chunked summarization plus global merge.”

Below is a vendor-agnostic illustrative code snippet showing a general “split first, merge later” framework:

# Generic illustration: Map-Reduce framework for very long documents (pseudocode)
def split_text(text, max_tokens=6000, overlap=200):
    """Split by token length, keeping slight overlap between chunks for continuity"""
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + max_tokens])
        start += max_tokens - overlap
    return chunks
def map_summarize(chunk):
    """Map stage: produce a local summary for each chunk"""
    prompt = f"Extract the key facts, data, and conclusions from the following passage:\n{chunk}"
    return llm_generate(prompt)
def reduce_summaries(summaries):
    """Reduce stage: merge local summaries into a global conclusion"""
    joined = "\n\n".join(summaries)
    return llm_generate(f"Based on the following chunk summaries, produce a global analysis report:\n{joined}")
def analyze_long_document(text):
    summaries = [map_summarize(c) for c in split_text(text)]
    return reduce_summaries(summaries)

When the task requires true full-text close reading (e.g., detecting conflicts across chapters), use the long-context model’s whole-document capability directly to avoid information fragmentation caused by chunking.

2. Repository-Level Code Understanding

Placing an entire codebase (or core modules) into context enables cross-file dependency analysis, global refactoring suggestions, and impact assessment for new requirements. Practical tips:

• Prioritize entry files, interface definitions, and type declarations; exclude build artifacts and binary files;
• Include a description of the repository directory structure so the model can quickly build a global map;
• For very large repositories, adopt a hybrid strategy of “retrieval to locate relevant files plus long-context close reading.”

3. Long-Range Memory for Agents

The longer an agent task runs, the more trajectories, tool-call results, and intermediate reflections accumulate. Million-token context makes “full trajectory replay” possible. Combined with prefix caching, multi-turn interaction latency and cost can be kept within engineering-acceptable bounds. For extremely long tasks, “rolling summarization plus key-event archiving” can compress memory.

4. Cost and Latency Optimization Checklist

• Enable and fully leverage context caching so common prefixes are computed only once;
• Slim down inputs: deduplicate, denoise, and remove low-relevance passages—token count is cost;
• For latency-sensitive scenarios, evaluate percentile values of TTFT and generation speed, not averages;
• Constrain output length to avoid meaningless verbose generation.

Selection and Deployment Recommendations

Here is an actionable evaluation checklist:

1. Verify effective context: Build needle-in-a-haystack QA using your own real long documents, and measure recall accuracy at different lengths—rather than relying only on the officially claimed window size;
2. Stress test performance: Measure TTFT, inter-token latency, and throughput at target concurrency to confirm the business SLA can be met;
3. Calculate the cost structure: Pay attention to input/output price differences, cache hit discounts, and whether very long requests incur additional fees;
4. Check engineering support: Are document parsing, vector retrieval, logging and monitoring alerts, canary deployment, and rollback capabilities all in place?
5. Assess migration cost: Are API protocols standard? Is it easy to switch between different models? Avoid being locked into a single implementation.

At the enterprise deployment level, consider leveraging mature cloud platform capabilities to lower the engineering barrier. For example, one-stop LLM service platforms—such as Baidu AI Cloud—typically package model inference, context caching, document parsing, vector retrieval, and monitoring alerts as general-purpose components. Developers can combine them as needed and focus on business logic and prompt design rather than reinventing the inference system.

Conclusion

From trillion-level parameters to million-token context, the main line of LLM evolution is shifting from “knowing more” to “processing a more complete world at once.” For developers, rather than obsessing over fractions of a point in the latest leaderboard ranking, it is better to return to engineering fundamentals: understand the technical trade-offs behind long context, verify effective context rather than nominal context, use the combination of caching and retrieval well, and continuously evaluate and iterate on real business data. Once these fundamentals are solid, long-context models can truly transform from benchmark champions into productivity tools.