Million-Token Context LLMs: Engineering Practices for Repository-Scale Code Modification

    Code generation is now ubiquitous, but modifying a repository with tens of thousands of files while staying correct remains a hard problem. This article examines the technical foundations of ultra-long context—hybrid and sparse attention, shared KV cache reuse, and sparse expert activation—and explains why a long window alone is not enough. It then walks through building internal evaluation sets from commit history, layered context assembly, closed-loop repair iteration, deployment and integration options, performance and cost optimization, and a troubleshooting guide, closing with a best-practices checklist for turning models from code-fragment providers into executors that deliver usable patches.

    1. Why “Writing a Function” Is Not the Same as “Modifying a Project”

    Code generation is now ubiquitous, yet the real pain point for engineering teams is not “generating a function.” It is “modifying a repository with tens of thousands of files and still being correct.” Over the past two years, mainstream code models have evolved through roughly three stages: syntax-level completion, single-file generation, and repository-level understanding and modification. The true dividing line is the third stage, which imposes three hard requirements on models.

    Context breadth. A single defect often spans the entry layer, business logic, and the persistence layer, with the call chain passing through a dozen files. If the model can only see a local fragment, it will produce code that “looks reasonable but actually conflicts with the project’s contracts.”

    Computational economics of long sequences. The naive implementation of attention scales quadratically with sequence length. If million-token inputs were computed in full at every layer, both inference latency and memory usage would be unacceptable. This means “supporting long context” cannot rely on simply enlarging the window; it must also subtract and reuse computation in the model’s compute structure.

    Engineering closed-loop capability. Changing code is only an intermediate step. The output must be an applicable patch, tests that pass, and an explainable impact scope. Without an “execute—verify—fix” loop, model output stays at the draft level.

    2. The Technical Foundation of Ultra-Long Context (General Perspective)

    Understanding these four mechanisms is what lets you judge capability boundaries during selection and tuning.

    Sparsified and hybrid attention. The mainstream approach is hybrid attention: most layers use local sliding windows to attend to nearby tokens, while a few layers retain global anchors so that key information can be routed between any layers. Some designs add sparse selection and routing between layers, bringing computation down from full quadratic to near-linear. For coding scenarios, the value is that “sparse but critical” information such as repository indexes and symbol definitions will not be truncated by a local window.

    Shared indexes and KV cache reuse. Coding tasks have a distinct characteristic: the system prompt, repository structure summary, and dependency manifest are highly repetitive across consecutive turns. Through shared indexes and cross-request key-value cache reuse, a repeated prefix only needs to be encoded once, sharply lowering the marginal cost of multi-turn interaction. This is the key step that moves long context from “usable” to “affordable.”

    Sparse parameter activation (mixture of experts). Total parameter count determines model capacity, while the number of parameters actually activated per token determines compute cost. Once these two are decoupled, throughput can be improved without sacrificing breadth of knowledge. The engineering implication: a large parameter count no longer means high inference cost. When selecting a model, focus on activation scale and concurrency capacity rather than total parameters alone.

    Long context is not a panacea. A long window solves “whether it fits,” not “whether it gets read.” Attention decays in the middle of long sequences, so an important constraint buried at the 80% position is easily overlooked. For this reason, retrieval augmentation (dependency graph + symbol index + semantic recall) should be combined with long context, rather than choosing one over the other.

    3. How to Evaluate Whether a Model Fits Your Repository

    Public benchmark leaderboards reflect a general distribution that often differs significantly from your tech stack, coding style, and domain conventions. A more reliable approach is to build an internal evaluation set and construct tasks in reverse from historical commits.

    1. Evaluation set construction (offline, one-time investment, reused long-term):
    2. 1. Filter "fix-type" commits from version history: commit message contains a defect
    3. description, and the change touches 1–5 files;
    4. 2. Take the parent commit as the code snapshot, the commit message as the task
    5. description, and the post-commit diff as the ground-truth answer;
    6. 3. Remove samples strongly coupled with test files to avoid answer leakage;
    7. 4. Sample in layers by repository/module to ensure coverage rather than quantity.

    The metrics worth focusing on are not “how similar the output looks,” but engineering usability:

    Metric Meaning Suggested Target
    Patch applicability rate Share of output diffs that apply cleanly without conflict ≥ 70%
    Regression pass rate Share where existing tests remain all-green after applying ≥ 95%
    First-pass rate Share of issues resolved in a single turn ≥ 40%
    Average interaction rounds Iterations needed to converge ≤ 3
    Cost per effective patch Total cost / number of successfully merged patches Continuously decreasing

    Only by fixing the prompt version, temperature, and retrieval configuration can you obtain comparable longitudinal data.

    4. Context Assembly: The Engineering Step That Sets the Upper Bound

    For the same model, the performance gap between well-assembled context and rough context is usually larger than the gap created by switching to a different model. A layered budget is recommended:

    1. # Illustrative code: layered context assembler
    2. class RepoContextAssembler:
    3. def __init__(self, budget=120_000, client=None):
    4. self.budget = budget # Context token budget
    5. self.client = client # Unified model access gateway
    6. self.index = None # Dependency graph / symbol index
    7. def build(self, task: str, entry_files: list[str], max_depth: int = 2):
    8. core = self.client.render_system(task) # Task + constraints, placed first
    9. graph = self.index.expand(entry_files, depth=max_depth) # Dependency expansion
    10. blocks = [self._render(f) for f in graph.ranked()]
    11. used, kept = len(core), []
    12. for b in blocks: # Greedy fill by relevance
    13. if used + len(b) > self.budget:
    14. break
    15. kept.append(b)
    16. used += len(b)
    17. return core + "\n".join(kept) + self.client.render_output_contract()
    18. def _render(self, path: str) -> str:
    19. return f"### FILE: {path}\n{self.index.read_symbol_slice(path)}\n"

    Four practical points:

    1. Put constraints first. Place the task description, compatibility requirements, and the scope of forbidden changes at the very front, followed by retrieved fragments. Leave the middle region of the long context for expendable background information.
    2. Structured, not flat. Each code block should carry its file path, owning module, and relevant symbol names; this significantly improves the model’s localization accuracy. Plain-text concatenation causes the model to confuse symbols with the same name.
    3. Prefer symbol-level slices over whole files. Give the modified symbol and its direct callers/callees first, and provide the whole file only when coupling is high.
    4. Make the output contract explicit. Stating “output only a unified diff,” “do not modify test files,” or “produce an impact analysis before the patch” can greatly reduce post-processing cost.

    5. Closed-Loop Iteration: Let the Model Fix, Test, and Repair Itself

    Single-shot generation cannot handle complex defects; the value comes from a controlled iteration loop. A minimal viable repair loop has four elements: a restricted tool set (read file, search, write patch, run tests), structured memory (a summary of failure logs plus already-attempted approaches), a clear termination condition, and a mandatory human-review gate.

    1. # Illustrative code: defect-repair iteration loop
    2. def auto_repair(repo_root, issue, client, max_steps=6):
    3. """Iterate on a sandbox copy of the repo until tests pass or the step limit is reached."""
    4. assembler = RepoContextAssembler(client=client)
    5. memory = []
    6. for step in range(max_steps):
    7. ctx = assembler.build(issue.title, issue.entry_files)
    8. ctx += "\n".join(memory[-3:]) # Rolling memory to control length
    9. patch = client.complete(ctx, temperature=0.1) # Low temperature for code generation
    10. ok, log = sandbox.apply_and_test(repo_root, patch)
    11. if ok:
    12. return patch, step + 1
    13. memory.append(f"[step {step}] Failure summary: {summarize(log)}")
    14. return None, max_steps

    The engineering constraints cannot be skipped: the loop must run on a sandbox copy; test commands must be given a timeout; write operations need whitelist validation; and patches must be produced as reviewable diffs rather than merged straight into the main branch. On temperature strategy, defect repair and refactoring should stay low-temperature to pursue determinism, while the solution-exploration phase can raise it moderately to widen the search space.

    6. Choosing a Deployment and Integration Form

    Three typical forms each have their applicable boundaries:

    • Platform API calls: No operations overhead, good elasticity; suitable for quick validation and individual developers. Data handling must be assessed against compliance requirements.
    • Private inference deployment: Code never leaves the intranet; suitable for strong-compliance scenarios such as finance and government/enterprise. It requires a self-built inference cluster and planning for memory and concurrency capacity.
    • Hybrid architecture: Sensitive repositories use private inference, while auxiliary tasks on public code (comments, test skeletons, documentation) use platform APIs, balancing cost and compliance.

    Regardless of the form, it is advisable to abstract a unified model gateway on the business side that handles retry with backoff, concurrency throttling, cost accounting, prompt version management, and backend routing. The value of this layer is that when models iterate or are replaced, the business code needs no changes.

    On the integration surface, capability is typically delivered through three entry points: IDE plugins (interactive completion and chat), code-review bots (automatically providing comments and patch suggestions on merge requests), and pipeline gates (automatically supplementing unit tests and static-check fix suggestions). All three reuse the same gateway and the same context-assembly logic, avoiding fragmented capabilities.

    7. Performance and Cost Optimization

    1. Maximize prefix-cache hits. Keep the system prompt, repository summary, and evaluation criteria stable in both order and content, and avoid inserting volatile fields such as timestamps each turn, which would invalidate the entire cache.
    2. Slim down the context. Remove redundant comments and generated boilerplate; replace whole long files with a summary plus key fragments, and load hit fragments on demand.
    3. Layered routing. Route intent classification, format conversion, and comment generation to lightweight models; call large-parameter models only for architecture mapping and cross-file refactoring. Most teams gain a several-fold cost reduction at this step alone.
    4. Batching and concurrency control. Send offline tasks (batch test supplementation, batch code health checks) through batch APIs with semaphore-based throttling; use streaming output for online requests to improve perceived first-byte latency.
    5. Quantized deployment. In private scenarios, low-bit quantization can significantly improve throughput, but any precision loss must be regression-verified against your internal evaluation set rather than judged from general benchmarks alone.
    6. Observability. Record the input/output token counts, cache hit rate, first-token latency, and iteration rounds of each call, and wire them into logging and monitoring alerts, so that cost optimization has a data basis.

    8. Troubleshooting Common Issues

    Symptom Common Cause Handling
    Output is truncated Generation-length limit is too low Raise the output limit; switch to segmented output—impact analysis first, then the patch
    Diff cannot be applied Context fragments are inconsistent with the real file Provide original line numbers and precise slices; run a patch dry-run after generation and feed errors back
    Fixes the local part but breaks the global Callers and contracts were not provided Expand the dependency graph to direct callers; add interface contracts and data constraints
    Misses constraints in a long document The key constraint sits in the middle of the long context Put constraints first and repeat key items; use retrieval to force recall of the relevant spec sections
    Multi-turn spinning, repeated same-type changes Missing failure feedback or truncated memory Record structured summaries of failure logs; set a step limit and add repeated-approach detection
    Unstable output format No structural constraint applied Agree on an output template, with parse validation and automatic retry on failure
    Throttling and timeouts Concurrency exceeds the quota Exponential backoff, request queueing, and a cache-first strategy to reduce call volume
    Unsafe dependencies or code introduced Missing security gate Enforce dependency-source review and code security scanning in the pipeline

    9. Best-Practices Checklist

    • Small-step commits. Keep each task focused on a single intent with a controlled change scope, minimizing review and rollback cost.
    • Tests first. Add a reproducible failing case before letting the model fix it; success is decided by tests, not by subjective judgment.
    • Define contracts before changing. Once interface signatures, error codes, or data formats change, declare them clearly in the context first, so the model does not “invent” new contracts on its own.
    • Reproducible context. Record the prompt version, the retrieval hit list, and the model parameters so problems can be traced back.
    • Least privilege. Restrict write operations to a directory whitelist, forbid sending source code externally, and keep credentials out of the context.
    • Clear division of labor. Keep architecture decisions, public-interface design, and final code ownership with the team; let the model handle retrieval, localization, boilerplate implementation, and test-case generation—work that is verifiable.

    Overall, as ultra-long context and engineered Agent capabilities mature, the LLM’s role shifts from “provider of code fragments” to “executor that delivers patches.” But whether this shift actually lands depends on whether context engineering, evaluation closed loops, and security gates are in place—the model sets the upper bound of capability, while engineering determines the actual output.