Long Context and High Throughput: Engineering Practices for LLMs in AI Agent Applications
AI agent applications require handling large volumes of context while maintaining fast response times. This article explores the engineering practices necessary for balancing long context windows and high throughput in LLM-based agents, covering context layering, cache-friendly prompt design, performance measurement, and security governance.
1. Background: Why AI Agent Tasks Rely on Both Long Context and High Throughput
Early conversational applications typically handled only a few rounds of Q&A with short contexts, where response speed had limited impact on user experience. AI agent applications are different. They need to read code repositories, parse technical documentation, call tools, record intermediate results, and maintain state across multiple steps. Each step resends previous content back to the model, causing the input length to grow rapidly.
Simultaneously, scenarios like coding assistance, document generation, and research analysis require the model to return complete results within a short timeframe. If the output speed is slow, a multi-step task might take several minutes, making it difficult for developers to maintain a continuous workflow. Therefore, long context capability and output speed have become two core dimensions for evaluating AI agent infrastructure.
2. Core Concepts
2.1 Context Window
The context window refers to the maximum number of tokens a model can process in a single inference, including system instructions, conversation history, tool returns, and the content to be generated. A larger window allows the model to consider more information simultaneously, but it does not mean content can be stacked indefinitely. As information increases, it becomes harder for the model to locate key details, and the computational cost per request rises with the input length.
2.2 Output Throughput
Output throughput is typically measured in tokens per second (TPS), determining the time required to generate long text or code. Another related metric is Time To First Token (TTFT), the time between sending a request and receiving the first output token. Interactive scenarios are more sensitive to TTFT, while batch generation scenarios focus more on overall throughput.
2.3 Prompt Caching
Prompt caching refers to the server-side reuse of repeated prefixes in requests. System instructions, tool definitions, and fixed reference documents often remain unchanged across multiple requests. When a cache hit occurs, it reduces redundant computation, thereby lowering latency and, in most billing schemes, reducing input costs. Caching is effective only when the prefix content is exactly identical, so the organization of the prompt directly impacts caching performance.
3. Typical Workload Characteristics in Agent Scenarios
- Input greatly exceeds output: Reading large amounts of code or documents while generating only a few conclusions or patches.
- Multi-turn invocations: The same context is sent repeatedly, resulting in highly repetitive prefixes.
- Unpredictable tool results: The length of content returned by tools fluctuates significantly, easily filling the window.
- Latency sensitivity: Every second of delay disrupts the workflow while developers wait for results.
Given these characteristics, engineering designs should focus on both “capacity” (fitting it in) and “speed” (running it fast).
4. Engineering Practices
4.1 Layered Context Organization
It is recommended to organize the context into three layers:
- Static Layer: System roles, output specifications, and tool definitions, which rarely change.
- Task Layer: Goals, constraints, and key file summaries for the current task.
- Dynamic Layer: Recent interaction history and tool outputs.
When the window is insufficient, prioritize compressing older content in the dynamic layer while retaining the static and task layers. Below is a sample implementation for trimming history based on a token budget:
def build_messages(static_prefix, history, query, budget, count_tokens):"""Retain recent history records based on a token budget, always keeping the fixed prefix and current query."""remaining = budget - sum(count_tokens(m) for m in static_prefix) - count_tokens(query)kept = []for msg in reversed(history):cost = count_tokens(msg)if cost > remaining:breakkept.append(msg)remaining -= costreturn static_prefix + list(reversed(kept)) + [query]
This function assumes that count_tokens is provided by the caller according to the tokenization rules of the model in use. Different models yield different tokenization results, so budget estimation should use the tokenizer corresponding to the target model.
4.2 Designing Cache-Friendly Prompt Structures
Since caching relies on an exactly identical prefix, the following principles should be followed:
- Place fixed content at the very beginning, including system instructions and tool descriptions.
- Position variable fields like timestamps, random IDs, and usernames further back.
- Maintain the order and format of tool definitions within the same session.
- Avoid inserting dynamic content in the middle of the prefix; otherwise, subsequent content cannot reuse the cache.
4.3 Measuring Throughput and Latency
Before deployment, use real workloads to measure key metrics rather than relying on a single test. The following example demonstrates how to calculate TTFT and output speed:
import timedef measure_stream(stream):"""Calculate the TTFT and average output speed of a stream (approximating token count by pieces)."""start = time.perf_counter()first_at = Nonepieces = 0for piece in stream: # Each piece is treated as an approximate token unitif first_at is None:first_at = time.perf_counter()pieces += 1end = time.perf_counter()ttft = first_at - start if first_at else float("nan")tps = pieces / (end - first_at) if first_at and end > first_at else 0.0return {"ttft_s": round(ttft, 3), "approx_tps": round(tps, 2)}
Measurements should cover different input lengths, concurrency levels, and time periods, recording percentiles (such as P50 and P95), and comparing them against acceptable business latency.
5. Security and Governance Essentials
- Credential Management: Store API keys in environment variables or key management services; never hardcode them or log them.
- Data Boundaries: Verify if inputs contain sensitive information, perform desensitization if necessary, and confirm the data processing and retention policies of the service in use.
- Least Privilege: Restrict tool calls to accessible directories, APIs, and operation scopes to prevent the agent from executing actions beyond the task scope.
- Output Validation: Generated code and commands must be tested and manually reviewed before entering the production environment.
- Audit Logging: Retain metadata of key requests to facilitate troubleshooting and compliance audits.
6. Deployment Checklist
- Clarify the typical input and output lengths of the task to select an appropriate context strategy.
- Organize prompts into static, task, and dynamic layers, and verify that prefixes remain consistent.
- Set token budgets and trimming rules for history records.
- Establish monitoring metrics for TTFT and output speed, and configure alert thresholds.
- Complete security and governance reviews before deployment.
Long context capabilities expand the scope of tasks an agent can handle, while prompt structure, budget management, and performance monitoring determine whether these capabilities can be reliably translated into development efficiency. Combining architectural design with measurement ensures that LLM applications can complete complex tasks while maintaining fast enough response times.
