Million-token contexts and long-horizon agents: inference engineering practices

    This article examines the inference-engineering requirements of million-token contexts and long-horizon agents. It explains where context limits come from, how positional encoding, attention design, KV-cache management, and input governance make long context practical, and how to turn single-step generation into a resumable agent execution chain. It also covers latency optimization through decoding paths and service scheduling, measurable evaluation, cost control, security boundaries, and an implementation checklist for production teams.

    1. The Starting Point: From Answering One Question to Completing One Task

    Over the past two years, competition in large-model capability has largely focused on the accuracy of single-turn question answering. The more important shift introduced by the latest generation of ultra-large-scale models is that the capability boundary is expanding from “single-generation output” to “long-duration autonomous execution.”

    When a model can continuously call tools, read and write files, run code, and remain aligned with the original goal even after hundreds of steps, the limiting factors are no longer model parameters or training data. Instead, the constraints are three things:

    1. How much context can fit into a single request?
    2. What is the end-to-end latency of a single call?
    3. How can the failure rate of the entire task chain be controlled?

    Understanding these three constraints is more useful for engineering than memorizing a capability list from any product launch.

    2. Million-Token Contexts: Where the Constraints Are and How to Break Them

    2.1 Positional Encoding and Length Extrapolation

    Standard rotary position embeddings often suffer from attention entropy collapse when used beyond the training length. Directly extrapolating them usually leads to degradation after a few thousand tokens.

    Common industry approaches include position interpolation, NTK-aware scaling, and YaRN-like frequency-band adjustments. The core idea is to compress the rotation speed of high-frequency components while preserving the discriminative ability of low-frequency components. This allows the model to maintain a stable relative position sense even when the inference length is far greater than the training length.

    The factor that truly determines success is continued pre-training for long context. In practice, teams often start by annealing at a moderate length, such as tens of thousands of tokens, then gradually extend the window toward the target length. At the same time, the data mixture should be adjusted so that naturally long sequences—such as long documents, code repositories, and multi-turn logs—make up a sufficient proportion.

    2.2 Attention Structures: Trading Sparsity for Capacity

    Dense global attention is not sustainable over extremely long sequences. In engineering practice, three common routes are:

    • Local sliding-window attention for nearby details
    • Global tokens—such as sink tokens, task instructions, and summary anchors—for cross-segment information routing
    • Sparse or linear attention, or hybrid state-space layers, for low-cost approximation of long-range dependencies

    Multi-query attention and grouped-query attention reduce the size of the KV cache by compressing the KV dimension. This is a prerequisite for many subsequent inference optimizations.

    2.3 KV Cache: From a Memory Problem to a Cost Problem

    A common way to estimate the KV cache size for a single request is shown below. This is illustrative; the actual result depends on the model architecture.

    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len,
    2. bytes_per_elem=2, batch=1):
    3. # Cache both K and V for every layer
    4. return 2 * layers * kv_heads * head_dim * seq_len * bytes_per_elem * batch
    5. # Example: 80 layers, 8 KV heads after per-card parallelism, head_dim 128, BF16
    6. per_token = kv_cache_bytes(80, 8, 128, 1)
    7. print(f"KV per token: {per_token / 1024:.1f} KB")
    8. print(f"1M-token context: {per_token * 1_000_000 / 1024**3:.1f} GiB")

    This calculation explains why “supporting a million-token context” and “running a million-token context at low cost” are two different things.

    With a fixed memory budget, there is a trade-off between the available concurrency batch size and the maximum supported context length. Three approaches provide the highest returns:

    1. Paged KV management: split KV into fixed-size blocks, allocate blocks on demand, and use an indirect table for addressing.
    2. Prefix sharing: keep only one physical copy of identical system prompts.
    3. KV quantization: compress the cache to 8 bits or lower.

    Paging introduces addressing overhead and kernel complexity. These costs need to be balanced with fused operators and coarser block sizes.

    2.4 Position Sensitivity: A Long Window Is Not Equally Useful Everywhere

    In practice, models often recall information at the beginning and end of the input better than information in the middle. This is a frequent source of failures in retrieval-augmented scenarios.

    The right response is not to keep increasing the window. Instead, teams should focus on input governance:

    • Move key constraints forward into the system instruction.
    • Split long documents into structured blocks with explicit titles and numbering.
    • Re-rank retrieval results by relevance and attach source markers.
    • Require the model to cite block numbers in the output.

    The context window is a resource, not an unlimited quota that can be freely overspent.

    3. Long-Horizon Agents: Turning One Generation into a Resumable Execution Chain

    3.1 The Main Loop

    A long-horizon agent is essentially a stateful retry loop. The model is responsible for decisions, while deterministic logic enforces the boundaries.

    1. async def run_task(goal, tools, max_steps=60, budget_s=900):
    2. memory = Memory(window=200_000) # Layered memory: summary layer + raw text layer
    3. memory.add(system_prompt(goal))
    4. for step in range(max_steps):
    5. plan = await llm.plan(memory.render(), tools=tools.schemas())
    6. if plan.finish:
    7. return await verify(plan.answer, memory)
    8. result = await tools.call(plan.tool, plan.args, timeout=120)
    9. memory.append(plan.thought, plan.tool, result)
    10. if memory.tokens() > budget_s: # Trigger compaction instead of dropping context
    11. memory.summarize(keep_recent=8, keep_pinned=[goal])
    12. raise StepBudgetExceeded(goal)

    Four design points matter:

    1. Set both a step budget and a wall-clock budget.
    2. Tool calls should have timeouts, idempotency keys, and explicit semantics for partial failures.
    3. When compacting context, keep the goal and the most recent raw turns; summarize everything else.
    4. Persist every step so that the task can resume from any interruption point.

    3.2 Failure Recovery and Observability

    The success rate of long-horizon tasks is determined by tail failures. Therefore, the data to collect is not simply “whether the final answer is correct,” but rather:

    • Input summaries for each step
    • Tool outputs
    • Retry counts
    • Latency per step

    A recommended practice is to propagate a trace_id across the entire chain. Token consumption, cache hit rate, and tool error rate should be treated as first-class metrics in monitoring and alerting systems. Logs and trace data should be stored in a unified logging service and object storage. This supports both post-hoc replay and offline evaluation-set construction.

    Replay capability is part of the infrastructure needed to iterate on long-horizon agents. Without it, it is difficult to attribute the effect of any prompt change.

    3.3 Context Layering

    A three-layer context structure is recommended:

    • Resident layer: goal, constraints, and output format
    • Sliding layer: the most recent N turns of raw interaction
    • Archive layer: structured summaries of earlier content plus retrievable pointers to the original text

    The archive layer should not re-inject full text into every step. It should only expose a query interface, allowing the model to retrieve information actively when needed. This keeps the input size per step under control and avoids information drift caused by repeated summarization.

    4. “Same Intelligence, Several Times the Output Speed”: Where Latency Savings Come From

    Output-speed improvement is not a single trick. It comes from combining two lines of work: the decoding path and service scheduling.

    4.1 Decoding Path

    Speculative decoding—using a small draft model or self-speculation to propose several tokens ahead, followed by parallel verification from the main model—can often increase tokens per second while keeping the output distribution unchanged. Hit rate and draft length need to be tuned for the business corpus.

    Low-bit quantization reduces pressure on weight bandwidth. The benefit is especially direct during decoding, where the stage is limited by memory bandwidth.

    Mixture-of-experts architectures naturally support sparse computation, but teams need to control the number of activated parameters per token. Otherwise, sparsity can turn into scheduling complexity.

    4.2 Service Scheduling

    Continuous batching fills batches at the token level rather than the request level. This is a major source of throughput.

    Chunked prefill splits long-input prefill work into segments and inserts them into decoding batches. This prevents long requests from blocking short requests and helps flatten the tail of time-to-first-token latency.

    Prefill-decode disaggregation allows the two types of workloads to select different compute resources and parallelism strategies. Long-context prefill is compute-intensive, while decoding is bandwidth-intensive. Mixing them in the same deployment inevitably causes mutual interference.

    Prefix caching is highly effective for fixed system prompts and fixed tool definitions. When the cache hits, time to first token can drop significantly.

    4.3 Tiered Routing

    A better engineering approach is to expose latency tiers rather than a single model endpoint. Requests can be routed based on SLO requirements: interactive completion uses a low-latency tier, while offline batch jobs and long-horizon tasks use a full-capability tier.

    Example configuration:

    1. routing:
    2. - name: interactive # Editor completion, CLI question answering
    3. slo: {ttft_ms: 400, tpot_ms: 25}
    4. tier: low_latency
    5. - name: long_horizon # Repo-level changes, long-document analysis
    6. slo: {ttft_ms: 3000, tpot_ms: 60}
    7. tier: full_capability
    8. max_context: 1000000

    TTFT—time to first token—determines interactive responsiveness. TPOT—time per output token—determines the smoothness of typewriter-style streaming output. These two metrics should have separate targets and separate alerts. Looking only at average end-to-end latency can hide tail-latency problems.

    5. Evaluation: Replacing Marketing Claims with Measurable Metrics

    A four-layer evaluation structure is recommended:

    1. Single-step capability layer: code-completion pass rate, valid structured-output rate
    2. Tool layer: tool-selection accuracy, parameter-extraction accuracy
    3. Long-horizon layer: multi-step task success rate, average number of steps, human-intervention rate
    4. System layer: TTFT/TPOT percentiles, cost per million tokens, cache hit rate

    For the long-horizon layer, real task sets should be used rather than synthetic puzzles. Failure clusters should also be recorded.

    Rollouts should use shadow traffic and gradual release. A new configuration first receives real traffic in parallel but does not return results to users. Teams then compare distribution differences before releasing to a small percentage of traffic. Both configuration and prompts should be versioned to support rollback and attribution.

    6. Cost Perspective

    The cost of long-context scenarios is highly nonlinear. Prefill cost is roughly linear or even superlinear with respect to input length, while cache hits can amortize the cost of repeated prefixes to nearly zero.

    As a result, cost optimization priorities are usually:

    1. Hoist fixed prefixes and cache them
    2. Apply input governance and remove redundancy
    3. Cache results and deduplicate work
    4. Use quantization and tiered low-latency routing
    5. Change the model

    Token consumption, cache hit rate, and task success rate should be placed on the same dashboard. Otherwise, hidden losses can occur when cost reduction leads to lower success rates.

    7. Risks and Boundaries

    Ultra-long context expands the attack surface for prompt injection. External content from web pages, repository files, and tool outputs must be treated as untrusted data and strictly separated from instructions.

    High-privilege tool calls should have:

    • Whitelists
    • Secondary confirmation
    • Least-privilege design

    In specialized domains such as financial analysis and 3D asset generation, model output is only a candidate. It must be passed through deterministic validators—such as numerical consistency checks, geometry validity checks, compilation, and tests—and confirmed by a human in the loop.

    For scenarios involving enterprise data, teams also need clear policies for data retention, encryption, and auditability. Isolation and lifecycle management should be enforced on container platforms and object storage. Monitoring and alerting should cover abnormal calls and unauthorized access.

    8. Implementation Checklist

    1. First improve input governance and retrieval re-ranking; only then consider expanding the context window.
    2. Use the capacity formula to calculate memory and concurrency limits, then determine paging size and cache strategy.
    3. Enable chunked prefill and prefix sharing for long requests, and isolate long and short workloads.
    4. Set SLOs based on TTFT/TPOT percentiles and use tiered routing.
    5. Add step budgets, cost budgets, idempotency, and resumability to agents.
    6. Build a four-layer evaluation system and shadow-traffic mechanism.
    7. Put token cost, cache hit rate, and task success rate on the same dashboard.
    8. Set isolation and confirmation mechanisms for external content and high-privilege operations.

    The capability jump of the new generation of models ultimately has to land on these practical engineering details: the proportion of context that can be used effectively, the latency of each call, and the completion rate and unit cost of the whole task chain. The teams that build observable, replayable, and rollback-ready systems around these three metrics are the ones that can turn model capability into business results.