Engineering Agent Applications with Million-Token Contexts: Context Organization, Prompt Caching, and Evaluation

    As models begin to support million-token contexts, agent developers can place entire code repositories, sets of technical specifications, and long documents into a single request. This article explains how to move from merely fitting information into the context window to using it reliably. It covers the limits of long-context capabilities, layered context assembly for agents, prompt caching, output speed and streaming, practical patterns for coding assistance, document processing, and research analysis, and guidance on choosing between long-context input and retrieval-augmented generation. It concludes with a production checklist covering evaluation, cost, cache governance, security, fallback, and infrastructure.

    1. From “Fitting It In” to “Using It Well”

    Improvements in long-context capabilities are changing how developers handle complex tasks. In the past, a model could read only a few thousand characters at a time, so developers had to split code, documents, and conversation history into small chunks and rely on retrieval or manual stitching to fill in the missing information. When the capacity of a single input expands to the million-token scale, an entire code repository, a batch of technical specifications, or hundreds of pages of analysis material can be placed in the same request.

    A token is the basic unit that models use to process text. In Chinese-language scenarios, one Chinese character usually corresponds to one or more tokens, depending on the tokenization method. Capacity should therefore be evaluated in tokens rather than roughly estimated by character count or page count.

    A larger window does not mean the problem is solved automatically. Context capacity determines how much the model “can see,” but it does not determine whether the model “can use that information accurately,” nor does it determine invocation cost and response speed. Engineering teams need to consider three dimensions at the same time—information organization, inference cost, and interaction experience—to turn long-context capabilities into stable business value.

    2. Understanding the Limits of Long-Context Capabilities

    2.1 Context Is Not Memory

    The content a model can read in a single request is not the same as content it can use evenly across every inference. Research and engineering practice commonly observe that the position of information affects recall. Key information at the beginning and end of the input is usually easier to use, while details buried in the middle of a large body of material are more likely to be ignored.

    Therefore, stuffing all materials into the context at once does not guarantee accurate results. A safer approach is to tell the model explicitly which materials are most important and to specify in the prompt the range of paragraphs on which the output should be based.

    2.2 Cost and Latency Grow with Length

    The longer the input, the more tokens each request must process, and first-token latency and total time usually increase accordingly. Input cost also rises linearly. For high-frequency applications, if every request carries the same large block of background material, costs can accumulate quickly.

    This problem can be mitigated in two ways: reduce irrelevant content, and reuse stable prefixes that have already been processed—that is, prompt caching, which is discussed in the next section.

    3. Context Organization in Agent Tasks

    An agent typically calls tools, reads results, and updates state over multiple rounds of reasoning. Context grows as steps increase, so it needs deliberate layered management. Common layers include:

    • System layer: role definition, behavior rules, and output format constraints. The content is stable and changes rarely.
    • Tool layer: descriptions and parameter structures of available tools. It is needed on every call and usually does not change.
    • Task layer: the current user goal and key constraints. It may be updated each round.
    • Material layer: code files, document fragments, retrieval results, and similar content. It is the largest and most frequently changing layer.
    • History layer: summaries of previous steps and key decisions. It should be compressed periodically rather than appended indefinitely.

    The following illustrative context assembly code reflects the order of “stable content first, dynamic content last”:

    1. def build_prompt(system_rules, tool_schemas, materials, history_summary, task):
    2. # Stable parts go first so the prefix cache can be reused
    3. prefix = [
    4. {"role": "system", "content": system_rules},
    5. {"role": "system", "content": tool_schemas},
    6. ]
    7. # Materials are ordered by relevance and labeled with sources for model citation
    8. material_block = "\n\n".join(
    9. f"[Source: {m['path']}]\n{m['content']}" for m in materials
    10. )
    11. # Dynamic parts go at the end
    12. suffix = [
    13. {"role": "user", "content": f"History summary:\n{history_summary}"},
    14. {"role": "user", "content": f"Reference materials:\n{material_block}"},
    15. {"role": "user", "content": f"Current task:\n{task}"},
    16. ]
    17. return prefix + suffix

    This organization has two benefits. First, the stable prefix can be cached and reused. Second, materials are labeled with their sources, making it easier to trace the basis of conclusions in the output.

    4. Prompt Caching: Principles and Engineering Practices

    The core idea of prompt caching is that if the prefixes of multiple requests are exactly the same, the server can skip repeated computation for that part, reducing latency and input cost. Cache hits depend on exact prefix matching, so any single character change in the prefix invalidates the cache.

    In engineering practice, the following principles can be followed:

    1. Keep the prefix fixed: system instructions, tool definitions, and long background materials should remain byte-for-byte identical. Do not embed dynamic fields such as timestamps or request IDs in them.
    2. Place dynamic content later: user questions, real-time retrieval results, and other changing parts should be placed at the end.
    3. Mark cache boundaries explicitly: in APIs that support explicit caching, clearly mark the length of the reusable prefix instead of relying on implicit judgment.
    4. Monitor hit rate: record the number of cached tokens for each request. If the hit rate remains low, check whether the prefix has changed unexpectedly.
    5. Pay attention to expiry: caches usually have a time-to-live and are cleared after a long period without requests. Applications need to tolerate latency fluctuations caused by cold starts.

    Illustrative monitoring code:

    1. def log_cache_stats(usage):
    2. total_input = usage["input_tokens"]
    3. cached = usage.get("cached_tokens", 0)
    4. hit_ratio = cached / total_input if total_input else 0.0
    5. print(f"Input tokens: {total_input}, cached tokens: {cached}, hit rate: {hit_ratio:.1%}")

    5. Output Speed and Interaction Experience

    Output speed is usually measured in output tokens per second (OTPS), calculated as:

    OTPS = total output tokens ÷ generation time in seconds

    This metric directly affects the waiting time users perceive. For example, for an answer of about 2,000 tokens, if the output rate is 50 tokens per second, generation takes about 40 seconds; if the rate increases to 100 tokens per second, it takes only about 20 seconds. In scenarios such as code completion and interactive question answering, shorter waiting times significantly improve the user experience.

    Two related but different metrics should be distinguished:

    • First-token latency: the time from sending a request to receiving the first output token. It is mainly affected by input length and queuing.
    • Output throughput: the sustained rate during the generation phase. It is mainly affected by model size, inference deployment method, and concurrent load.

    For frontend display, streaming output is recommended. Users can see partial results as early as possible, and even if total time remains the same, the subjective sense of waiting drops noticeably. For batch tasks, the focus should be on overall throughput and unit cost rather than excessive optimization of single-request speed.

    6. Common Scenarios in Practice

    6.1 Coding Assistance

    In coding scenarios, long context can hold a larger range of source code, helping the model understand call relationships between modules. In practice, consider the following:

    • Build a dependency graph first and divide materials by module or package instead of indiscriminately feeding the entire repository.
    • Put interface definitions, configuration files, and test cases into the context first, because they directly constrain the implementation.
    • Modification suggestions should include file paths and line numbers for human review.
    • All generated code must go through compilation, unit tests, and static checks. Model output cannot be treated directly as a reliable result.

    6.2 Document Processing

    Common requirements for long-document processing include summarization, clause comparison, and information extraction. If a document far exceeds what a single context can hold, a “chunked extraction, then aggregation” strategy can be used:

    1. Split the document by chapter or fixed token window, and preserve chapter titles.
    2. Perform extraction independently on each chunk and output structured results such as fields, values, and original positions.
    3. Merge the results from each chunk and have the model perform consistency checks and aggregation.
    4. Validate key values with rules, for example, checking whether totals match the sum of line items.

    This approach offers strong controllability: the input and output of each step can be recorded and audited.

    6.3 Research Analysis

    For long materials such as research reports and financial documents, the analysis focus is usually cross-section consistency—for example, whether management statements match data performance. Such tasks benefit from keeping key tables and original paragraphs in the context and requiring the model to provide citation locations, reducing unsupported inferences.

    7. Choosing Between Long Context and Retrieval-Augmented Generation

    Long context and retrieval-augmented generation (RAG) are not mutually exclusive; they apply under different conditions.

    Dimension Direct long-context input Retrieval-augmented generation
    Material scale Suitable for medium-scale materials that require holistic understanding Suitable for massive corpora where only local fragments are needed
    Information relationships Suitable for tasks requiring cross-section reasoning Suitable for question-answering and location-based tasks
    Update frequency Materials must be provided again on each request The index can be updated incrementally
    Cost structure Input tokens grow with materials Mainly retrieval cost and index maintenance cost
    Traceability Depends on model citation labels Retrieval results naturally carry sources

    In real systems, a common pattern is to use retrieval to narrow the scope first, then place the relevant documents as a whole into the long context for deeper analysis.

    8. Production Readiness Checklist

    Before putting long-context capabilities into production, confirm each of the following:

    • Evaluation set: build test samples covering typical scenarios and edge cases, and record accuracy, citation correctness, and latency distribution.
    • Cost monitoring: count input and output tokens by functional module and set budget alerts.
    • Cache governance: confirm prefix stability and check hit rates regularly.
    • Data security: de-identify sensitive information in materials, follow the principle of least privilege for access, and avoid storing business data in plaintext in logs.
    • Fallback strategy: when long requests time out or the service is unavailable, fall back to chunked processing or a smaller context scope.
    • Infrastructure: raw documents can be stored in object storage, and call chains and error information can be collected through logging services and monitoring alerts for easier troubleshooting. If the system is deployed on Baidu AI Cloud, these general cloud capabilities can be used to build the components described above.

    Long-context capabilities expand the task boundaries that agents can handle, but they do not replace good information organization, cache design, and evaluation mechanisms. Teams should start from task requirements, choose an appropriate context scale, and continuously validate results through quantifiable metrics.