Million-Token Contexts in Practice: Long-Context Coding Essentials

1. What a Bigger Context Window Actually Changes

Over the past year, the context windows of several industry models have grown from the 200K-token range to the 1M-token scale. The AI coding tools built around them have started claiming “whole-repository understanding.” This is not a quantitative change but a change in how we work:

• Before: you had to slice, retrieve, and summarize a codebase before feeding it to the model. Now: you can load the core files of a module, or even of a mid-sized service, into the context all at once. As a result, cross-file call relationships, type definitions, and naming conventions are all visible to the model at the same time.
• Before: long tasks had to be split by hand into many small segments, and the model only knew “this little piece” on each turn. Now: the model can maintain global consistency over much longer stretches. That reduces “fixed A, forgot B” breaks.

One boundary needs to be established right away: nominal window size ≠ effective usable window. Misjudging this is the most common mistake in adopting long context.

“Effective context” means the amount of information the model can genuinely and reliably use in a given inference. A widely observed pattern is that models reference information much less accurately when two conditions hold:

• The key information sits in the middle of a very long context.
• The context contains many semantically similar distractor snippets.

The longer and more crowded the context, the more pronounced this decline becomes. In other words, loading an entire 800K-token repository into the window does not necessarily work better than precisely providing 30K tokens of relevant code. It is often more expensive and slower, and it is more easily led astray by irrelevant code.

A commonly cited reference point for intuition comes from “book-scale” retrieval experiments:

• Setup: a document the length of a book, or even several books, is placed in the context in full. The model is then asked to retrieve facts that appear only rarely. Examples include a specific proper noun, a particular clause, or a specific string constant.
• Why they matter: their value lies not in the exact scores but in what they reveal:
  1. Long windows do support “finding details across the whole.”
  2. Retrieval-style recall rate is a practical metric for effective context.
  3. When the context contains many repeated structures, the model tends to treat content that “looks like” the real thing as actual content.

Point 3 translates into code as hallucinated interfaces, parameters, or config options that don’t exist.

So the first practical principle of long context is: measure success by effective utilization rate, not by how much you cram in. Every method below is designed to raise that effective utilization.

2. Step One: Set an Explicit Context Budget

Treating the context window like memory that must be managed is the starting point for all long-context practice. Teams should keep a fixed budget table and decide how to allocate it before each session starts. Don’t leave it to the tool to fill the window automatically.

Here is an illustrative budget split (for a 1M-token window, expressed as percentages):

# Context budget (illustrative, not tied to any specific product API)
context_budget:
  system_and_rules:     5%    # System prompt, project standards, coding conventions
  task_definition:      5%    # Acceptance criteria and constraints of the current task
  retrieved_code:      25%    # Retrieved files, type definitions, interface contracts
  prior_reasoning:     15%    # Confirmed designs, decision records, summary of completed changes
  tool_output:         10%    # Trimmed compiler, test, and log output
  history_dialog:      10%    # Conversation history with the user
  reserve:             30%    # Reserved headroom, to avoid a scramble when the compaction threshold hits

The most important part of this table is the last item: leave 20–30% of the window unfilled. The reason is practical. As soon as the compaction threshold is reached, the tool automatically replaces earlier context with a summary, and summaries are lossy. Filling the window to capacity means handing the decision of “when information is dropped, and which information” to an uncontrolled moment.

Teams should also write these hard rules into a project standards file that tools read automatically (something like AGENTS.md or CONTRIBUTING.md):

1. Each session should advance one verifiable subtask, and its description must include acceptance criteria.
2. Relevant code should be brought in on demand through retrieval. Never instruct the model to “read the entire src directory.”
3. When a tool returns more than a set number of lines, write the output to a file first and have the model read a summary of that file.

3. Retrieval Over Dumping: Deciding What to Feed the Model

A useful rule of thumb for deciding what deserves context space is this: feed the model information it must see or it will definitely get things wrong. Don’t feed it information it can infer or look up again whenever needed.

For code projects, the priority, from highest to lowest, is roughly as follows:

• Interface contracts and type definitions: function signatures, type and interface declarations, schemas, API definitions, and database table structures. These determine whether a change is valid, so they must be exact rather than inferred.
• Direct dependencies and direct callers of the code being changed: one hop of depth is usually enough. Beyond that, the benefits drop off quickly.
• Project constraints that carry weight: naming conventions, directory conventions, error-handling patterns, logging conventions, and forbidden dependencies. These belong in the rules file as a permanent reference, not in repeated descriptions.
• Tests and runnable samples: existing test cases are the highest-quality “statement of intent.” They constrain model behavior better than any natural-language description.
• No need to feed: full source of third-party libraries (signatures or docs are enough), generated artifacts, dependency lock files, and old implementations from earlier versions.

Most AI coding tools already come with codebase indexing and semantic retrieval. Rather than replace these capabilities, the team’s job is to verify them. A low-cost way to do this:

1. Randomly pick 5 historical bug-fix commits.
2. Give the tool each bug’s description to search.
3. Check whether it retrieves the core files that each fix touched.

If recall is clearly low for a repository, its index quality or code organization is likely the problem. That is exactly when the temptation to “just load the whole repo” is strongest, yet the results are usually the worst.

4. Reliability Design for Long Tasks: Compaction and Checkpoints

Long context is a double-edged sword. It lets the model work continuously for longer, but it also means failures surface later. By the time you notice, dozens of files may already have changed, and rolling back is expensive.

Automatic context compaction, which summarizes and replaces earlier context as the threshold approaches, is a mitigation, not a solution. The problem is that summaries lose precise detail, and code tasks depend on precise detail. A history compressed into “implemented a user authentication middleware” cannot support correctly modifying a specific branch inside that middleware.

A more effective strategy is to use proactive checkpoints. Rather than letting the tool decide when to forget, you explicitly freeze the state at key points.

Keep a lightweight context snapshot file in the project root as the cross-session source of truth:

# Task Context Snapshot (Example)
## Goal
Switch the order module's export from synchronous to asynchronous, keeping the old interface compatible for 90 days.
## Confirmed Decisions
- Reuse the existing message queue; no new middleware
- Export task state machine: PENDING -> RUNNING -> DONE / FAILED
- Keep the legacy synchronous interface; it delegates to the async implementation internally and polls for results
## Completed Changes
- [x] Added the export task table and migration script
- [x] Extracted ExportService, with an adapter layer for the old entry point
## Next Step (only one)
- [ ] Add 3 unit tests for retry on failure
## Explicitly Out of Scope
- Do not change the billing module
- Do not introduce a new third-party CSV library

All three sections of this file are essential:

• Completed: prevents repeated work.
• Next step (only one): prevents scope creep.
• Out of scope: prevents changes that cross the line.

After each compaction, the model loses the conversation history, but this file stays in context. That is what keeps its behavior consistent.

The companion practice is breaking tasks down into the right granularity: small enough that each one can be finished and independently verified in a single session. The test is simple: if a task needs more than three consecutive “read code → make changes → run tests → fix errors” cycles, keep splitting it.

5. The Underrated Cost: Token Overhead of the Tools Themselves

An earlier discussion made a key point: the larger the context, the less advisable it is to attach external tools with large token overhead. This deserves a closer look, because it is the hidden cost most easily overlooked with long contexts.

AI coding tools reach databases, monitoring systems, documentation systems, issue trackers, and other resources through tool integration mechanisms such as external tool plugins. The tools’ definitions and descriptions occupy context permanently. Each tool includes its name, parameter schema, and purpose description, costing anywhere from a few hundred to a few thousand tokens. Attach 30 tools, and their definitions alone may consume 20K–30K tokens. Worse, you pay that cost again on every inference turn. On top of that, the more tools there are, the more likely the model is to pick the wrong one.

The bigger cost is what the tools return. A log query, a directory traversal, or an API doc can easily push tens of thousands of tokens into the context, and often only a small fraction is relevant to the current task.

The corresponding engineering rules are:

1. Trim the tool set per task: attach database tools only when you need a database, and log tools only when troubleshooting. Don’t “attach everything just in case.”
2. Limit response size: favor calls with filters, pagination, field selection, and row limits. Avoid SELECT * queries and reading whole files.
3. Write large results to a file, then summarize: save the full output to a temporary file and have the model read a summary or retrieve from it on demand, instead of letting it enter the conversation history in full.
4. Prefer exit codes and summary statistics from commands: whether the build passed, how many tests passed, and which ones failed are structured conclusions with a far higher information density than full logs.
5. Audit context usage: most tools show the token breakdown of each request. Teams should periodically review which types of tool output take up the most space, and tighten default parameters accordingly.

6. Long Contexts Don’t Lower Verification Requirements; They Raise Them

Here is a counterintuitive but important conclusion: a model being able to see all the code at once does not mean the code it writes is more reliable. The larger the window, the more the model tends to “imitate whatever it sees.” That includes legacy baggage that should never be copied, such as deprecated patterns, temporary compatibility code, and unfinished logic in TODOs. In a long context, these patterns are all in view at the same time, which makes them significantly more likely to be copied.

So the verification loop can’t be skipped, and it should be automated as much as possible:

• Compilation and static checks are the first gate. Nothing that fails to compile should go on to human review.
• Write new unit tests; don’t just run the old ones. Have the model produce test cases alongside the new logic, explicitly covering normal paths, boundary values, and error branches.
• Check the scope of each change. Use the version control diff to verify that none of the modules marked “out of scope” were touched. This is the fastest way to catch scope creep.
• Actually run UI-related output. You cannot judge the correctness of graphics and UI code by reading it; it has to be rendered. When comparing tools’ capabilities, give them the same description and have them generate renderable graphics code. Comparing the visual results and code cleanliness directly is more revealing than any abstract score.
• Verify callers of cross-file changes. This is where long context shows its value. Explicitly list the affected call sites and confirm each one, rather than changing only the definition.

7. How to Measure Whether a Long Context Is Actually Useful

A larger window does not mean a usable one, and that can only be judged through evaluation, not marketing claims. Teams should build a lightweight evaluation set of their own that covers four task types:

Dimension	How to Evaluate	What to Watch
Global retrieval	Locate low-frequency facts in a long document or large repository	Hit rate, whether it fabricates answers
Cross-file refactoring	Change a feature across multiple files in one pass	Whether all call sites are updated, whether it compiles
Long-range consistency	Tasks that require several rounds of iteration	Whether early constraints are forgotten, whether work is repeated
Instruction following	Tasks with explicit prohibitions and format constraints	Out-of-scope changes, format compliance

The evaluation set should be built from the team’s own real historical commits. Re-run it whenever a model or tool version changes, so results stay comparable across runs. Industry benchmarks are useful references, but they cannot replace an in-house evaluation. Performance on public leaderboards has nothing to do with your codebase’s naming conventions, framework versions, or directory structure.

For the same reason, don’t bet on a single model. A sensible setup keeps multiple models switchable:

• Put model calls behind a unified gateway layer, so business code doesn’t depend on a specific provider.
• Route work by task type. For example, send large-scale refactors to models with strong long-context capabilities, and format conversions or simple generation to more economical models.

If your organization already uses LLM services on Baidu AI Cloud, the same advice applies: keep this abstraction in place so you avoid being locked in to a single vendor.

8. Governance: The Risks That Long Context Amplifies

The longer the context, the more information each request carries, and the larger the risk surface:

• Sensitive information: Once secrets, tokens, production data, or customer personal information are read into the context, they enter the processing pipeline of an external service. They must be blocked at the read stage:
  • On the repository side, use secret scanning and pre-commit checks.
  • In the rules file, explicitly prohibit reading .env files, certificates, and data export files.

• Code asset boundaries: Whether private code can be sent out must be settled before any technology choices are made. Compliance usually follows one of two paths:

  • Use enterprise-grade services that explicitly commit to not using customer data for training.
  • Deploy models within a private network.

  What matters is the written agreement, not verbal assurances.

• Licenses and dependency introduction: Models can hallucinate package names that don’t exist, or introduce dependencies whose licenses are incompatible with the project. These are real types of problems that have occurred in practice. To guard against them:
  • Maintain a dependency allowlist.
  • Require that every package be verified against your internal artifact repository.
  • Add license checks to CI.
• Accountability: AI-generated code must go through human review before it is merged into the main branch, and the reviewer is accountable for it. This should be written into the process, not left to individual diligence.

9. Pre-Launch Checklist

Here is the guidance above distilled into an actionable checklist:

1. The project root has a rules file listing conventions, directory structure, and prohibited actions.
2. A context budget table is defined, with at least 20% of the window left unfilled.
3. A context snapshot file is maintained with the goal, completed work, the single next step, and what is out of scope.
4. Tool plugins are trimmed per task, with limits on returned rows and fields.
5. Compilation, static checks, and unit tests are in the automated pipeline as mandatory gates.
6. After every change, the diff is reviewed to confirm no out-of-scope edits.
7. An in-house evaluation set covers retrieval, cross-file refactoring, long-range consistency, and instruction following.
8. Protections for sensitive files, secret scanning, the dependency allowlist, and license checks are in place.
9. Human review responsibility is clearly defined, and all AI output goes through review before it is merged into the main branch.
10. The model call layer stays switchable, with no lock-in to a single vendor.

Million-token contexts genuinely solve one problem: the scarcity of a global, cross-file view. What they don’t solve is precision, scope control, and the cost of verification. Treat the window as a resource to be managed, not a space to spend freely, and long context becomes real delivery efficiency instead of a marketing parameter.