Engineering Million-Token Context Models for Code Agents

    Million-token context windows make it possible to place large repositories, long documents, multi-turn tool logs, and complex business materials into a single session. This article explains why effective context utilization, not window size alone, determines production value. It covers an end-to-end evaluation framework for code and long-horizon agents, context budgeting and compression, sandboxed execution and auditability, parameterized modeling and industrial scripts, financial workflows, factuality and refusal mechanisms, safety controls, service-tier and cost governance, and a small-traffic canary rollout path.

    Engineering Million-Token Context Models for Code Agents

    1. Background: Long Context Is Only the Starting Point

    The arrival of million-token context windows allows a model to hold large code repositories, long documents, multi-turn tool-call records, and complex business materials in a single session. For developers, this significantly reduces the cost of frequent truncation, repeated concatenation, and manual summarization. For enterprises, it provides a new technical foundation for code agents, long-horizon task automation, parameterized modeling, and financial workflows. But a large context window does not mean high effective context utilization. Whether a model can reliably locate key information in extremely long inputs, preserve task state, call tools correctly, and produce auditable results depends on a full set of engineering capabilities: context organization, retrieval compression, tool protocols, sandbox execution, evaluation systems, and cost control.

    Therefore, when discussing million-token context models, it is not enough to focus only on parameter scale or single-point benchmark scores. Teams should establish an evaluation framework that considers capability, boundaries, cost, and risk together. Public leaderboards can serve as references, but business target distributions, tool environments, data definitions, and security requirements vary widely. What is truly reusable is an internal evaluation set, a canary release pipeline, and an observability system.

    2. Capability Evaluation: From Point Benchmarks to End-to-End Workflows

    For models targeting code and long-horizon agents, evaluation should cover at least the following dimensions:

    • Code generation and modification: Look beyond function-level completion to cross-file refactoring, dependency understanding, test generation, defect repair, and regression verification.
    • Long-horizon task execution: Assess multi-step planning, state preservation, tool invocation, failure recovery, and final deliverable quality.
    • Terminal-restricted tasks: Execute commands, read and write files, and handle permissions and network restrictions in isolated environments, with emphasis on robustness in real environments.
    • Parameterized modeling and script generation: Check whether scripts can execute, whether geometric constraints are satisfied, and whether units and versions are consistent.
    • Financial and office document workflows: Focus on data definitions, traceable formulas, citation accuracy, and human review efficiency.
    • Factual accuracy and refusal capability: Distinguish single-fact errors, the proportion of answers containing errors, and whether the model refuses when it does not know.
    • Security and abuse risk: Evaluate high-risk request identification, tool permission boundaries, and differences between released and unfiltered versions.

    The value of public evaluations is that they provide horizontal references, but enterprises should build internal benchmarks close to their own business. For example, code tasks can use historical defect-fix records from real repositories as test sets; long-horizon tasks can be designed as end-to-end flows containing 10 to 30 tool calls; financial workflows can require the model to output formulas, data sources, and assumptions. Evaluation metrics should include accuracy, first-pass success rate, manual rework rate, average latency, and cost per task.

    3. Long-Context Engineering: Budgeting, Layering, and Compression

    Million-token context needs to be managed like memory. Context should be divided into system instructions, task goals, long-term memory, retrieved evidence, tool results, and temporary scratch content, with budgets and eviction policies. For code repository scenarios, relevant files can first be recalled through symbol indexes and dependency analysis, then long files can be structurally summarized. For long-document scenarios, content can be split by chapter while preserving citation anchors. For multi-turn tool calls, key state should be recorded rather than full logs.

    An illustrative pseudocode example is shown below:

    1. context_budget = {
    2. "system": 0.05,
    3. "task": 0.10,
    4. "memory": 0.15,
    5. "evidence": 0.45,
    6. "tool_results": 0.20,
    7. "scratch": 0.05
    8. }
    9. def build_context(task, memory, evidence, tool_log):
    10. ctx = []
    11. ctx += compress_system_rules()
    12. ctx += task_summary(task)
    13. ctx += retrieve_memory(memory, top_k=8)
    14. ctx += rerank_and_compress(evidence, budget=context_budget["evidence"])
    15. ctx += summarize_tool_results(tool_log, keep_state=True)
    16. return enforce_budget(ctx, context_budget)

    At the same time, caching strategy should distinguish among cache hits, cache writes, and cache retention duration. For frequently reused system prompts, repository indexes, and long-document summaries, cache hits should be prioritized to reduce cost. For one-time long inputs, teams should evaluate whether cache writes are worthwhile. For service tiers, a standard tier is suitable for offline batch processing and cost-sensitive tasks, while a high-speed tier is suitable for interactive code completion and real-time agents. The high-speed tier usually trades a higher unit price for lower latency, and enterprises should route requests to the appropriate tier.

    4. Code Agents and Long-Horizon Tasks: Sandboxing, Auditing, and Recovery

    The core of a code agent is not “generating a large block of code in one shot,” but the closed loop of planning, execution, verification, and repair. Tool calls should be restricted to isolated sandboxes, with whitelists and approvals for file writes, command execution, network access, and dependency installation. Every tool call should record input, output, latency, and exit status to form an auditable log. For long-horizon tasks, checkpoints should be set so that failures can recover from the latest checkpoint rather than restarting from scratch.

    1. plan = model.plan(task)
    2. for step in plan:
    3. action = model.select_tool(step, state)
    4. result = sandbox.execute(action, timeout=..., policy=...)
    5. audit_log.append(action, result)
    6. state = model.update_state(state, result)
    7. if result.failed:
    8. state = model.recover(state, result.error)
    9. if state.retry_count > 2:
    10. escalate_to_human(state)
    11. final = model.verify(task, state, tests=...)

    Terminal-restricted tasks especially require environment isolation and least privilege. When a model executes commands in a real terminal, it may encounter missing paths, insufficient permissions, dependency conflicts, and other issues. Engineering systems should provide clear error-return formats and retry semantics. Evaluation of long-horizon tasks should not only examine the final answer, but also whether intermediate steps are explainable, whether constraints are followed, and whether side effects are produced.

    5. Parameterized Modeling and Industrial Scripts: Outputs Must Be Executable and Verifiable

    In 3D modeling, CAD scripts, and game engine scripting scenarios, the value of model output depends on whether it can execute and whether it satisfies constraints. A closed loop of script generation, sandbox execution, geometry validation, and human review is recommended: first generate a parameterized script, then execute it in an isolated environment, check units, coordinate systems, constraints, and export formats, and finally write key parameters and versions into audit records. For scripting interfaces of mainstream game engines and modeling tools, a general adapter layer should be used so that the model does not directly depend on the internal implementation of a specific tool.

    The validation checklist can include: whether the script syntax is correct, whether dependencies are declared, whether units are consistent, whether parameter ranges are reasonable, whether geometric constraints are satisfied, and whether exported files can be read by downstream systems. For failed results, error information should be returned to the model in a structured format, and the model should be asked to modify only the failed fragment to avoid full rewrites that introduce regressions.

    6. Financial Workflows: End-to-End Delivery and Traceability

    Model applications in finance should not stop at “generating a piece of analysis text.” They should cover end-to-end delivery for financial modeling, investment analysis, industry research reports, and other workflows. Key requirements include consistent data definitions, traceable formulas, explicit assumptions, verifiable citations, and clear human review points. Models can assist in generating three-statement models, sensitivity analysis, and draft research reports, but they must preserve formula chains and original data citations to avoid outputs that look reasonable but cannot be recalculated.

    From an engineering perspective, financial workflows should be divided into data access, definition validation, model generation, formula audit, conclusion review, and version archiving. For content involving investment advice, compliance review and disclaimers should be set. For key numbers, the model should be required to provide calculation steps or citation anchors. Evaluation should not only check whether answers are close to standard answers, but also whether formulas are correct, data is consistent, and citations are traceable.

    7. Factual Accuracy: Layered Evaluation and Refusal Mechanisms

    Factual accuracy evaluation should distinguish between the error rate of single factual statements and the proportion of answers containing at least one factual error. The former measures fine-grained errors, while the latter is closer to user perception. Internal stress tests should deliberately focus on error-prone situations, such as numbers, dates, entity relationships, and cross-paragraph reasoning in long documents, but stress-test error rates should not be directly equated with overall daily usage levels. A three-layer mechanism is recommended: retrieval augmentation and citation anchoring, post-generation self-checking and cross-validation, and refusal or human handoff when confidence is low.

    For code and financial scenarios, factual accuracy also includes executable correctness and recalculable correctness. Code must pass tests, financial formulas must be recalculable, and document conclusions must be traceable to evidence. For information that cannot be confirmed, the model should clearly mark uncertainty rather than fabricate.

    8. Safety and Alignment: Data Filtering, Red Teaming, and Release Gates

    On the safety side, high-risk data should be identified and removed during training and reinforcement learning to reduce abuse risks such as cyber offense. At the same time, residual risk should be managed through red-team testing, tool permission control, and release gates. Differences in capability on offensive tasks between released and unfiltered versions should be continuously monitored, but safety filtering should not come at the expense of normal development capability. Safety evaluation should be integrated into the CI process, approvals should be required for high-risk tool calls, and audit logs and rollback capabilities should be retained for sensitive operations.

    9. Service Tiers and Cost Governance

    Cost governance for million-token context models needs to start from four areas: request routing, caching, batching, and monitoring. Standard and high-speed tiers can be separated by latency sensitivity. Cache hits, cache writes, and cache retention duration should be included in the cost model. Offline tasks can be combined into batches. Online services should set guardrails on context length, tool-call count, and retry count. For deployments on Baidu AI Cloud, general-purpose services such as object storage, container platforms, logging, and monitoring can be combined to build evaluation, canary release, and observability pipelines, but the specific architecture should be designed according to business scale and security boundaries.

    10. Implementation Path: Small-Traffic Canary Releases and Continuous Regression

    A recommended implementation path is to first build an internal evaluation set and sandbox environment, then run small-traffic canary releases for code agents and long-horizon tasks, and subsequently expand to parameterized modeling and financial workflows. Each stage should define metrics for quality, cost, latency, and security, and establish a rollback mechanism. For long-context tasks, continuous regression should cover capabilities such as key information localization, tool-call correctness, failure recovery, and citation traceability. Only by bringing model capabilities into an engineering closed loop can million-token context truly translate into delivery efficiency rather than a new source of complexity.