Engineering Million-Token Context and Long-Horizon Agent Models
A technical breakdown of how to build and evaluate large language models for million-token contexts and long-horizon agent tasks—covering context budgeting, compression and caching, code agent loops, tool sandboxing, factuality governance, per-success-task cost accounting, and evaluation strategies beyond public leaderboards.
From Long Context to Long-Horizon Tasks
Million-token context windows and long-horizon agents are two key directions in the engineering of large language models. The former solves the problem of “can the information fit in”; the latter solves “can the task be carried through correctly over many steps.” Combined, a model can read large code repositories, lengthy documents, multi-turn tool call records, and historical decision states. But a larger context window does not automatically translate into higher task success rates. In real-world engineering, the opposite is often true: more information means more noise, longer sequences make state drift more likely, and more tools raise permission and audit risks.
Evaluating long-context models therefore should not rely on parameter counts or window sizes alone. Instead, focus on four capability boundaries:
- Effective context utilization — can the model reliably locate key evidence in long inputs and ignore distractors?
- State retention — can it maintain plans, hypotheses, completed steps, and pending conclusions across multi-step execution?
- Error recovery — when tools fail, tests break, or evidence conflicts, can it roll back and re-plan?
- Cost control — is the total overhead per successfully completed task acceptable?
The core of long-horizon tasks is not “generating a very long answer once,” but “advancing a complex goal through verifiable intermediate states.”
Long-Context Engineering: Budgets, Compression, and Caching
Long-context applications should start with a context budget mechanism. Split the input into six categories—system constraints, task goals, structured memory, retrieved evidence, tool outputs, and historical summaries—and set a token cap for each. System constraints and task goals get priority retention; conversation history should be compressed into structured state; retrieved evidence must carry sources and timestamps; tool outputs should keep only key fields and error codes. The goal is to ensure the model sees the information it needs for decisions within a limited budget, rather than passively receiving all raw text.
Next, combine retrieval, re-ranking, compression, and citation. For long documents, use chunked recall, similarity re-ranking, deduplication, and conflict detection. For code, retrieve according to repository structure, dependency graphs, and call chains. For multi-turn tasks, maintain summary memory and key-fact tables. Design caching in layers: stable system prompts and tool descriptions suit long-term caching; retrieved evidence suits short-term caching; task state should be isolated per session. Cache hits significantly reduce cost, but you need invalidation policies to prevent outdated documents or stale tool results from polluting decisions.
Illustrative pseudocode:
def build_context(query, memory, docs, budget):candidates = retrieve(query, docs)candidates = deduplicate(candidates)evidence = rerank(query, candidates)state = compress(memory, budget.state_limit)context = fill_by_priority(system=system_policy,goal=query,state=state,evidence=evidence,budget=budget)return context, citations(evidence)
Engineering should also include insufficient-evidence detection: when key fields are missing, sources conflict, or confidence is low, the model should proactively ask follow-up questions, expand retrieval, or decline to answer—rather than filling gaps with fluent language.
Code Agents and Long-Horizon Tasks: Closed Loops Matter More Than Single-Shot Generation
Coding tasks are a typical proving ground for long-horizon capability. Single-shot completion only solves local problems; real development requires requirement parsing, repository search, patch generation, static checks, test execution, failure attribution, and rollback. A deployable code agent should have a minimal closed loop: read the task and repository state, locate relevant files, generate small incremental patches, run checks and tests, fix issues based on errors, and finally output a change description with risk notes. Terminal-constrained tasks additionally require the model to interpret command results, environment differences, permission limits, and timeout handling—not just natural-language reasoning.
For evaluation, track task success rate, average step count, human intervention rate, failure recoverability, cost per successful task, and security incident count. Public leaderboards are useful for horizontal reference, but business teams should build internal long-horizon benchmarks covering repository scale, dependency complexity, anomalous inputs, tool failures, and permission denials. Illustrative logic for a tool-call sandbox:
def run_tool(tool, args, policy, sandbox):if not policy.allow(tool, args):return {"status": "denied", "reason": "policy"}if not policy.within_limits(tool, args):return {"status": "denied", "reason": "quota"}result = sandbox.execute(tool, args, timeout=policy.timeout)audit_log(tool, args, result.status)return result
Key principles: the model handles planning and generation, while deterministic code handles constraints and validation. High-risk tools must be whitelisted, run with least privilege, subject to resource limits, and fully audited. Retry budgets should be set so long-horizon tasks do not fall into ineffective loops.
Tool Calls and Parametric Modeling: Keep the Deterministic Boundary Clear
In mainstream game engines, 3D modeling tools, and CAD software, models can perform parametric modeling, batch generation, scene inspection, and render verification through scripting interfaces. The common thread: natural-language requirements must be converted into structured parameters, those parameters must pass geometric, constraint, unit, and resource validation, and the final results must be replayable and auditable. Models are good at understanding intent, generating scripts, and explaining errors—but they should not be treated as geometry solvers or numerical computation kernels. Critical constraints should be validated by deterministic code, such as dimension ranges, topological relationships, dependency versions, and output formats.
Adopt a generate–validate–fix loop: the model generates parameters and scripts, a sandbox executes them and returns structured errors, the model corrects based on those errors, and the validator checks again. For every execution, log input parameters, script versions, tool versions, and output summaries to ensure reproducibility. Restrict file system, network, and external command access to prevent privilege escalation during long-horizon tasks.
Vertical Workflows: End-to-End Delivery Still Needs Human Review Checkpoints
In scenarios such as financial modeling, investment analysis, and industry research reports, end-to-end workflows typically include data extraction, assumption management, spreadsheet computation, citation verification, and report generation. Models can accelerate drafting and information synthesis, but they cannot replace accountable parties. Engineering-wise, set up intermediate artifacts and checkpoints: data source tables, assumption lists, calculation conventions, citation lists, discrepancy notes, and audit logs. Run consistency checks on sensitive figures, require citation traceability for key conclusions, and enforce dual review for high-risk operations.
Evaluation for such applications should use business-specific datasets rather than generic leaderboards, focusing on field extraction accuracy, citation traceability, numerical consistency, report reviewability, and the cost of human edits. End-to-end does not mean fully autonomous—it means automating repetitive work while keeping judgment at controllable checkpoints.
Factuality and Safety Governance: Metrics Must Be Interpretable
Factual accuracy should distinguish between the “per-claim factual error rate” and the “proportion of answers containing at least one factual error.” The former measures local error density; the latter measures overall usability. Stress tests that deliberately focus on error-prone cases do not reflect overall everyday performance, but they do expose weak points. Governance measures include retrieval augmentation, citation binding, self-checking, conflict detection, confidence calibration, and refusal when necessary. For high-risk Q&A, enforce “no conclusion without a source” and display evidence locations.
On safety, training data filtering, red-team testing, least-privilege access, tool whitelists, and audit logs should form a closed loop. Safety filtering may reduce success rates on some high-risk tasks—an expected trade-off between usability and safety. Teams should tier by business scenario: prioritize efficiency for low-risk tasks; prioritize controllability, auditability, and rollback for high-risk ones. No system should claim absolute safety.
Cost and Billing: Account by “Cost per Successful Task”
The cost of long-context and long-horizon agents should not be judged by per-call unit price alone—calculate cost per successful task instead. Common billing dimensions include input tokens, output tokens, cache hits, cache writes, and priority-processing premiums. Optimization levers include context compression, cache reuse, tiered routing, batching, output length control, and retry budgets. Use priority processing for interactive scenarios and standard channels for batch jobs and offline evaluation; stable prompts and tool descriptions suit caching, while ephemeral evidence should be short-lived.
| Billing Dimension | Engineering Concern | Optimization |
|---|---|---|
| Input tokens | Bloat from long documents and history | Chunked recall, summary memory, deduplication |
| Output tokens | Verbose explanations and repetition | Structured output, length limits |
| Cache hits | Stable-prefix reuse rate | Fixed system prompts, layered caching |
| Cache writes | Short- and long-term cache costs | Per-session isolation, invalidation policies |
| Priority processing | Latency vs. cost balance | Priority for interactive, standard for batch |
On Baidu AI Cloud, for example, you can store evaluation datasets and audit logs in object storage, build call-chain tracing with log services, and set up cost and anomaly dashboards with monitoring and alerting—making the effectiveness, latency, and spend of long-horizon tasks observable and attributable. When selecting a provider, also examine billing transparency, caching rules, rate limiting, and data compliance capabilities.
Evaluation and Selection: Public Leaderboards Are Only a Starting Point
Public leaderboards reflect some general capabilities but cannot replace business validation. A three-tier evaluation is recommended: offline business datasets to verify correctness and citation quality; long-horizon task suites to verify step counts, recovery rates, and human intervention rates; and online controlled experiments to verify latency, cost, and user adoption. A selection checklist should cover context utilization, tool-call stability, error recovery, factual accuracy, safety auditing, billing transparency, and ecosystem compatibility. The ultimate goal is not a single top score, but repeatable, governable, and continuously improvable results under real business constraints.
