Million-Token Context LLMs for Code and Agents: Engineering Practices
This article explains how to build production systems around large language models with million-token context windows, covering code generation and long-horizon AI agents. It describes how to organize and compress context, structure the plan-execute-observe-revise loop with sandboxing and checkpoints, produce structured outputs for specialized workflows, set factual accuracy gates, apply security controls, and manage costs through model routing and budgets. It also includes a deployment checklist.
Million-token context LLMs are moving long-document understanding and long-horizon task execution into a new stage of engineering. A larger context window does not mean all materials can be fed to the model at once. Stronger code generation and agent capabilities do not remove the need for sandboxing, observability, and rollback. What determines real-world results is a complete engineering system covering context governance, tool calling, output verification, security boundaries, and cost control.
1. A Million-Token Window Is Not a Place to Dump Everything
The value of a million-token context is that it reduces information fragmentation across documents. In a single task, the model can see requirements, interface documentation, historical logs, code snippets, and acceptance criteria together. Longer context also brings more noise, repetition, and conflicting information. Engineering teams should divide context into four layers:
- Persistent system constraints: these stay stable across tasks.
- Immediate task materials: these are trimmed according to priority.
- Retrieved supporting materials: each item carries its source and timestamp.
- Execution feedback: keep only the most recent rounds and summaries of key errors.
A practical approach is to retrieve first and compress second, rather than concatenating everything and hoping the model ignores irrelevant parts. The pseudocode below shows the basic idea:
function build_context(task, docs, history):constraints = load_system_constraints()selected = retrieve_and_rank(docs, task.query, top_k=20)compressed = summarize_with_citations(selected, max_tokens=budget.docs)recent_feedback = keep_recent_and_key_errors(history, max_rounds=6)return assemble(constraints, compressed, recent_feedback)
Citation traceability is also necessary. Each key conclusion should be linked to its source snippet or tool output where possible. Without this link, the model may reorganize long context in a way that loses the origin of the information.
For caching, distinguish between cache hits and cache writes, and account for the write cost across different time windows. Place stable prefixes, tool definitions, and long-document summaries in reusable positions. Keep frequently changing task state in the dynamic section.
2. The Engineering Loop for Code and Long-Horizon Agents
For code generation and long-horizon agents, the core is not a single response. It is a loop of planning, execution, observation, and revision. A workable agent system includes at least a task decomposer, a tool router, a sandbox executor, a result parser, a failure recovery component, and an audit log. Tool calls must be governed by permissions: file reads and writes are limited to approved directories, command execution is limited to an allowlist, network access is enabled only when needed, and sensitive operations require human confirmation.
The pseudocode can be abstracted as follows:
function run_agent(task):plan = decompose(task)for step in plan:tool = route_tool(step)result = sandbox_call(tool, step.args, timeout=limits.timeout)observation = parse_and_validate(result)log(step, tool, observation, cost)if observation.failed:plan = replan(step, observation.error)return finalize_with_tests(plan)
Recoverability matters especially for long-horizon tasks. Save a state snapshot at every step. On failure, resume from the latest checkpoint instead of rerunning from the beginning. For code tasks, include static checks, unit tests, type checks, and runtime logs in automated acceptance. For tasks running in restricted terminals, limit execution time, memory, and output length to prevent runaway loops.
For observability, record each round’s tool name, parameter summary, duration, token usage, error codes, and points of human intervention. These records help determine whether a failure came from planning, the tool, or missing context.
3. Structure Specialized Workflows Instead of Relying on Free-Form Generation
In specialized domains such as parametric modeling, CAD scripting, financial modeling, investment analysis, and industry research, model output should be structured rather than free-form natural language. For example, a modeling task can require a parameter table, a script, expected geometric features, and validation steps. A financial analysis can require an assumptions table, calculation steps, data sources, and sensitivity analysis. The system then completes the loop with parsers, rule-based validators, and human review.
Acceptance criteria for specialized workflows should be as measurable as possible. Check whether the script runs, whether the model is generated, whether report figures reconcile, and whether citations are complete. High-stakes conclusions need manual review and secondary verification. Generated results should not be treated as final deliverables without that check.
4. Factual Accuracy Needs Metrics and Release Gates
Prompting a model to “answer carefully” is not enough to govern factual accuracy. Engineering teams should combine retrieval augmentation, citation requirements, fact verifiers, and release gates:
- Require sources for verifiable facts.
- Cross-check numbers, dates, and entity relationships.
- Downgrade conclusions without sources to recommendations.
- Apply refusal or human-escalation policies in high-risk domains.
Internal evaluation sets should deliberately cover error-prone cases. Track three measures separately: the error rate for individual facts, the proportion of answers that contain at least one factual error, and the rate of missing citations. Error rates from stress tests do not represent overall performance in everyday use. They are useful for version admission and regression comparison. Each time the model or a prompt changes, run the same evaluation set and record the differences.
5. Build Security and Misuse Protection in From the Start
Security governance should cover data, training, deployment, and runtime. On the data side, identify and remove high-risk content related to attacks and defenses. On the deployment side, enforce least privilege, sandbox isolation, command allowlists, and audit logging. At runtime, monitor unusual call frequency, access to sensitive paths, and bulk export behavior.
Outputs should avoid operational details that could be used directly for attacks. For technical questions with dual-use potential, provide abstract explanations and defensive recommendations. Teams should also maintain a red-team evaluation set, but use its results only for internal improvement and do not publish details that could be misused.
6. Use Model Routing and Budgets to Control Cost and Performance
Million-token context and long-horizon agents can significantly increase costs for input, output, cache writes, and tool calls. Common billing dimensions include input tokens, output tokens, cache hits, cache write time windows, and premiums for high-speed channels.
Engineering teams should set up model routing:
- Send simple classification, extraction, and formatting tasks to low-cost channels.
- Send complex code repair and long-horizon planning to more capable channels.
- Use caching for repeated prefixes.
- Run batch tasks asynchronously.
- Set a token budget, a tool call limit, and a timeout for each task.
In general-purpose cloud environments, including Baidu AI Cloud, inference services, sandbox execution, log services, monitoring and alerting, and key management can be deployed as separate components. A unified observability dashboard can then track cost per task and success rate.
7. Deployment Checklist
- Have you defined a context layering and compression strategy, and can key conclusions be traced to their sources?
- Do you enforce a tool allowlist, sandbox isolation, timeouts, and least-privilege permissions?
- Do you have checkpoints, retries, rollback, and human intervention mechanisms?
- Do specialized workflows produce structured outputs with automated validation?
- Have you built a factual accuracy evaluation set and version release gates?
- Do you cover security red-teaming, audit logs, and anomaly monitoring?
- Do you route models by task complexity and monitor token and tool costs?
In short, the engineering value of million-token context LLMs and long-horizon agents lies in turning context governance, execution loops, domain validation, security boundaries, and cost control into reusable infrastructure. Agent systems can enter production only when they move from “can generate” to “can be verified, observed, and rolled back.”
