Million-Scale Context for Long-Horizon AI Agents: Practical Engineering Lessons
Long context and long-horizon execution are becoming decisive factors in whether AI agents and coding assistants can serve as reliable daily drivers. This article presents an engineering framework for defining production-grade capability, making million-token context usable, designing long-running tasks, evaluating coding performance, controlling cost and reliability, and reducing vendor lock-in through portable assets and open, buildable practices.
Long context and long-horizon tasks are becoming the dividing line between AI agents and coding assistants that can support everyday work and those that are limited to occasional demonstrations. Parameter count, benchmark rankings, or an impressive answer in a single demo are not enough to show that a model can handle the critical path.
The practical questions are more demanding:
- Can the model still identify critical constraints within a million-token context?
- Can it independently move a multi-turn, multi-file, multi-tool task to an acceptance-ready result?
- When caching, rate limits, retries, and human review are all involved, are the total cost and reliability predictable?
This article does not evaluate or recommend any particular model. Instead, it organizes widely accepted capability requirements into practical engineering guidance: how to define a production-grade primary model, make long context genuinely useful, design long-horizon tasks, evaluate coding ability, and reduce lock-in through open and buildable approaches.
1. Define “Primary” Before Discussing Capability
Many teams use one model on the main path and other models as fallbacks or assistants. The primary model is expected to handle requirement clarification, solution drafting, code changes, test completion, and regression explanations by default. Supporting models are used only when the primary path fails, becomes too expensive, or presents excessive risk.
The difference is not a slogan. It is the cost of failure.
A primary-grade model can be evaluated against four criteria:
- Task closure: Given explicit acceptance criteria, the model can produce a runnable and reversible result within a limited number of steps, rather than stopping at high-level suggestions.
- Constraint retention: Project conventions, API contracts, protected modules, and security boundaries remain effective after a long conversation.
- Diagnosable failure: When a tool errors, a test fails, or context is truncated, the model can identify the failure point and narrow the next action instead of repeating the same ineffective change.
- Budgetable cost: Across similar tasks, token usage, latency, and human rework remain within an acceptable range for the team.
If all four conditions are not met, the model should generally remain in a supporting role. Moving an unverified model onto the primary path may reduce the cost of individual calls, but it can create hidden rework.
2. “Million-Token Context Support” Is Not the Same as Usability
A larger context window only means that the interface accepts more input. Usable long context means that key facts remain correctly applied when they appear near the beginning, middle, or end of the window, and when they are referenced across documents. Otherwise, recent conversation turns or frequently repeated templates may drown them out.
In production, it is better to divide context into four layers than to fill the window with the entire repository and complete chat history:
- Stable layer: Coding standards, architecture boundaries, directory conventions, and prohibited actions. These change slowly and are suitable for long-term caching.
- Task layer: The current requirement, acceptance criteria, issue summary, and interfaces that must be preserved.
- Evidence layer: The specifically referenced source files, test output, log excerpts, and API examples. Include only the portions relevant to the current subtask.
- Working layer: Recent reasoning, tool results, and unresolved questions.
A representative budgeting strategy can be expressed as follows:
budget = window_limit - reserve_for_output - reserve_for_toolsstable = load_cached(spec, architecture_rules)task = summarize(requirement, acceptance)evidence = select_files(changed_modules, failing_tests)if tokens(stable + task + evidence) > budget:evidence = shrink_by_relevance(evidence, budget - tokens(stable + task))prompt = assemble(stable, task, evidence, recent_turns)
This logic does not depend on a vendor-specific interface. It emphasizes three principles:
- Reserve space for model output and tool results.
- Separate stable information from frequently changing information.
- Trim evidence by relevance rather than truncating every file equally.
Long context has three common failure modes:
- Middle-of-context forgetting: A critical constraint appears in the middle of a large body of material, while the model pays more attention to the initial system instructions and the latest user request. Mitigate this by repeating hard constraints in the task layer and presenting them as a short list rather than a long narrative.
- Reference misalignment: When several similar files are included, the model modifies the wrong copy. Add stable identifiers to each evidence block, such as relative path, symbol name, and line range, and require the model to repeat those identifiers in its output.
- Silent truncation: A gateway or client removes the tail of an over-limit request without informing the model that information has been lost. The calling layer should record the actual token count after assembly, reject over-limit requests, and switch to staged retrieval.
Therefore, “supports one million tokens” is only an entry requirement. Before deployment, use a fixed probe set to test factual recall at the beginning, middle, and end of the context; cross-file symbol tracing; and truncation behavior near the limit. The probes should come from sanitized fragments of the team’s own repositories rather than public demo problems.
3. Long-Horizon Tasks Measure Independent Completion
A long-horizon task is not simply a conversation that continues for a long time. It has a multi-step objective, requires repeated tool use, persists intermediate state, and can continue after an interruption or failure.
Typical examples include:
- Completing a module according to a specification and getting its tests to pass.
- Tracing a production incident from logs to a patch and regression tests.
- Performing a cross-module refactor without breaking public interfaces.
Independent completion should be measured through process indicators rather than final correctness alone:
- Planning quality: Does the first step divide the task into verifiable subgoals instead of making broad code changes immediately?
- Tool discipline: Does the model read only necessary files, and does it reproduce a failure before editing?
- State continuity: After an interruption, can it continue from the checkpoint using only the task record, change list, and test results, without relying on hidden memory that cannot be exported?
- Stopping conditions: When acceptance criteria are met, information is insufficient, or a security boundary is reached, can it stop clearly and explain what remains unresolved?
A practical task record does not require a complex platform. Plain text is sufficient:
Goal: Add pagination to order queries while preserving existing filter semanticsAcceptance: New tests pass; existing tests show no regressions; public response fields are not removedBoundaries: Do not modify the billing module; do not introduce new external dependenciesProgress: Reproduced the failing case; implementation has not been modifiedNext step: Read only the query entry point and its associated tests
The most common failure in long-horizon agents is turning “the next step” into open-ended exploration. Each step should contain one primary action: read, modify, test, or summarize. Write the result back to the task record. The next iteration should consume the task record and the latest evidence rather than replaying the entire history.
This turns a million-token context window into safety margin instead of the default working set.
4. What Does It Mean for a Model to Actually Write Code?
Coding ability should be evaluated across at least four layers. A model that fails at any one of them is not suitable as the primary system.
- Local correctness: A single-function or single-file change passes the existing tests.
- Contract correctness: Types, error codes, null handling, pagination behavior, idempotency, and compatibility remain intact.
- Repository correctness: Builds, static checks, directory conventions, and dependency policies continue to hold.
- Operational correctness: Changes are reviewable and reversible, and the explanation identifies risks and uncovered paths.
Evaluation should not rely only on the pass rate of one generated patch. A more realistic protocol provides failing tests and relevant files, limits the model to a maximum of N edit-and-test cycles, and records:
- The number of steps before the first successful pass.
- The number of unrelated lines changed.
- Whether protected directories were touched.
Excessive unrelated changes are often more dangerous than occasional syntax errors. They increase review effort and amplify future merge conflicts.
Human review should consistently ask three questions:
- Did the model invent an interface or configuration item that does not exist?
- Did it use silent defaults to conceal an edge case?
- Did it modify the tests to accommodate the implementation?
If any of these occur, the capability should be considered below the required threshold rather than assuming that another prompt will solve the problem.
For non-programming users, the same standards can be applied to document, spreadsheet, and workflow tasks:
- Does the model follow the required template?
- Does it cite the provided material instead of inventing facts?
- Does it clearly mark uncertain items?
A primary model is not one that can do everything. It is one that makes fewer mistakes within its authorized scope and can stop when something goes wrong.
5. Open Source, Buildability, and Access Continuity
If advanced capabilities are available only through a closed entry point that cannot be reproduced, migrated, or relied upon continuously, they are difficult to use as an engineering foundation.
The value of open source is not simply that “the files are public.” Teams should be able to answer the following questions:
- Are the weights, inference code, tokenizer, and license sufficient to reproduce the main behavior in an environment they control?
- Does the license permit internal modification, redistribution, and integration with existing systems? Are its obligations practical to fulfill?
- Can prompts, tool descriptions, evaluation sets, and failure examples be exported so that changing providers does not require rebuilding everything from scratch?
- Are context caching, rate limits, and logs controlled by the caller rather than being completely opaque?
Even when managed inference is used in the short term, key assets should remain on the team’s side: task records, specifications, evaluation probes, tool contracts, and sanitized logs. When the access point changes, the team should lose a call path—not its entire working method.
Openness also creates responsibilities. Long context makes it easier to include secrets, customer data, and details of undisclosed vulnerabilities. Before data enters a model:
- Scan for secrets and minimize sensitive information through sanitization.
- Avoid storing complete prompts in logs by default.
- Grant tools only the permissions required for the task.
- Do not provide an agent with long-lived credentials for an entire development environment.
Open source can reduce lock-in, but it does not reduce data-governance requirements.
6. Caching, Quotas, and Real-World Cost
Context caching allows stable prefixes to be reused across later requests with lower marginal cost and latency. A high cache-hit rate usually indicates that the prefix is stable and request structures are consistent. Large fluctuations often result from rewriting system instructions on every turn, inserting timestamps, or placing large amounts of unrelated code in the cached prefix.
Practical measures include:
- Keep system specifications, tool lists, and repository maps byte-for-byte stable and place them in the cacheable prefix.
- Put timestamps, session identifiers, current diffs, and tool output after the prefix.
- Use the same prefix when retrying a task and replace only the failure evidence.
- Track cached tokens, uncached tokens, output tokens, and tool round trips separately instead of relying on the impression that the system is fast.
Free quotas and subscription quotas are budget signals, not evidence of capability. Actual consumption is often higher than the visible allowance suggests. A long-horizon task includes planning, file reading, testing, and summarization. Cache misses, retries, and parallel agents can further increase usage.
Before deployment, replay ten representative tasks and record P50 and P95 costs rather than making decisions from a single fortunate sample. If measured usage is significantly higher than expected, first check whether evidence is being sent repeatedly or entire logs are being inserted into the context. Only then should you investigate whether metering is abnormal. Until the cause is confirmed, unusually low usage should not be treated as a repeatable production capability.
7. Migrating Existing Agent Assets
Teams often have accumulated conversation practices, skill instructions, tool connections, and checklists. Migration should not aim for one-click equivalence. Instead, extract three portable layers:
- Procedure layer: Coding standards, review checklists, commit-message templates, and conditions that require human input.
- Tool layer: Each tool’s inputs, outputs, permissions, timeouts, and failure semantics. Avoid binding these definitions to a client’s private conversation format.
- Case layer: Sanitized successful and failed traces for regression testing, not for continued training on unauthorized data.
Migration acceptance can remain simple. Run the same ten tasks through the new entry point and compare:
- Pass rate.
- Number of steps.
- Unrelated changes.
- Number of human takeovers.
Migration is complete when the procedure layer works without material changes, the tool layer requires only an adapter, and the case-layer results do not significantly regress. Chat history itself usually has little migration value because it contains outdated conclusions and implicit state that cannot be reproduced.
8. Minimum Pre-Production Checklist
Before placing any long-context model on the primary path for coding or agentic work, complete at least the following checks:
- Use probes from the actual repository to verify fact retention at the beginning, middle, and end of the context. Confirm that over-limit requests fail visibly rather than being silently truncated.
- Prepare a task-record template that requires one primary action per step and clearly states acceptance criteria and protected directories.
- Use failure-test-driven loops to evaluate local correctness, contract correctness, and unrelated changes.
- Separate cacheable prefixes from changing evidence, and establish P50/P95 budgets for tokens and latency.
- Export procedures, tool contracts, and sanitized cases to avoid locking the working method to a single entry point.
- Deny access to secrets, customer data, and production write permissions by default. Grant tools temporary, task-specific authorization.
- Define stopping conditions: insufficient information, a security boundary, or two consecutive failures of the same type must trigger a handoff to a human.
Once these conditions are met, long context moves from “can fit” to “can be used effectively,” and long-horizon tasks move from “can keep talking” to “can independently reach closure.”
A primary model is not a leaderboard title. It is a working method that survives regression testing, auditing, and cost replay. Instead of debating which demo looked most impressive, teams should first run their probes, task records, and budget reviews. Only then has a model earned the right to move from the supporting role to the primary path.
