Agentic Coding and Million-Token Context: Moving AI from Q&A to Delivery

Introduction: The Measure of AI Is Shifting from “Accurate Answers” to “Delivered Results”

Over the past two years, the form of large-model applications has roughly gone through three stages.

• Knowledge Q&A: The user asks a question, and the model answers based on its training data—but it knows nothing about what happened after its knowledge cutoff.
• Retrieval-Augmented Generation (RAG): External knowledge such as enterprise documents and codebases is retrieved and stitched into the prompt, so the model can answer questions about “your own” data.
• Agentic Coding: The model doesn’t just provide answers; it can decompose tasks, read and write files, run commands, inspect errors, and iterate until the task passes verification.

Some people compare this shift to upgrading an “intern” into a “digital colleague”: you hand over a complete goal, and it plans, executes, checks, and reports on its own. This metaphor captures a key change—the measure of success moves from “was the single answer correct?” to “was the whole task completed, and at an acceptable cost?”

The stronger the capability, the more constraints come with it. Drawing on common industry experience, making coding agents deliver reliably requires three pieces of infrastructure:

1. A context window large enough to hold the entire project;
2. Controls that adjust reasoning depth by task difficulty;
3. Context management and an engineering loop that sustain long-running tasks.

This article breaks down these three parts in turn and offers design patterns you can adopt directly.

1. What Makes Agentic Different from Chat: From “One-Shot Generation” to “Long-Range Search”

Ordinary conversational interaction can be simplified to: user input → model output → session ends. The task exists only within that single turn, and the model neither needs nor is able to take responsibility for the result.

Agentic interaction is a feedback loop:

Plan: Break the goal into steps and determine their order and dependencies.
Act: Call tools to read and write files, run commands, and retrieve information—not just “talk.”
Observe: Read the real results returned by tools, including compile errors, test failures, and API errors.
Reflect: Judge the gap between the result and expectations, revise the approach, and act again.
Verify & Wrap-up: Run tests and checks, summarize changes, and explain trade-offs and remaining risks.

From a control-theory perspective, this is equivalent to adding a feedback loop to the model. Single-turn generation is “open loop”: once the output is produced, it can’t be corrected based on feedback from the real world. An agent is “closed loop”: the result of every tool call becomes the basis for the next decision. This is the fundamental reason it can actually finish complex tasks, and it also explains why an agent’s effectiveness depends not only on the model itself but heavily on tool quality and feedback signals.

However, this loop runs into three constraints in real tasks:

• Context budget: The more you read, the more complete the information; but no matter how large the window, it’s ultimately finite, and long sessions will eventually overflow.
• Reasoning budget: The deeper you think, the higher the accuracy on hard problems; but “overthinking” simple tasks inflates both latency and cost.
• Error accumulation: Every step has some probability of error; the longer the chain, the higher the risk that one mistake cascades, which must be contained through verification and rollback.

For developers, these constraints aren’t abstractions—they’re concrete questions you have to answer while writing code: How should context be organized? How much reasoning strength should be allocated? How do you recover after an error? The rest of this article revolves around these three questions.

2. Million-Token Context: Fitting It In Is One Thing, Using It Well Is Another

2.1 The Capacity Dividend: From File-Level to Repo-Level

Million-token context windows are gradually becoming common in leading models. A medium-sized codebase’s core modules and key configuration files can usually fit in a single pass (the exact count depends on code density and tokenization). Moving from “seeing a single file” to “seeing the whole repository” directly unlocks tasks such as:

• Cross-file refactoring: For example, when changing a shared interface, the model can simultaneously see the definition, all call sites, and related tests, avoiding “changed A but forgot B.”
• Architecture-level analysis: Mapping module dependencies, identifying circular dependencies, and assessing technical debt all require a global view.
• Multi-document comparison: Contract clause comparison, log clustering analysis, cross-version diffing, and similar tasks can skip the tedious chunking and stitching.
• Simplified RAG pipelines: What used to require a full “chunk—embed—retrieve—rerank” pipeline can now, for small-to-medium knowledge bases, be injected wholesale, significantly reducing engineering complexity.

2.2 Effective Context: A More Important Metric than “How Much Fits”

The advertised window is just an upper limit. What actually determines effectiveness is the effective context—the range within which the model can accurately use information given an input of a certain length. Industry research has repeatedly revealed several typical phenomena:

• Middle information is easily ignored: Content at the beginning and end of a long input is used more heavily, while recall of middle information drops noticeably—a phenomenon commonly called “lost in the middle.”
• Instruction-following decay: The longer the input, the more likely the model is to slip on constraints such as formatting requirements and boundary conditions.
• Attention dilution: The higher the proportion of irrelevant content, the more likely the model is to “fixate on the wrong thing,” and reasoning accuracy declines accordingly.

This means that simply dumping an entire repository into the context doesn’t equal “the model understands the whole repository”—it may actually lower output quality. To make long context pay off, you need to “design context” the way you design a database index.

2.3 Principles for Organizing Long Context

Through extensive practice, the industry has converged on a few general principles:

1. Put key constraints at the start and end: Place the task goal and hard constraints (e.g., “must not modify public interfaces”) at the beginning of the context, and put the latest instruction and acceptance criteria at the end.
2. Sort by relevance: Historical information unrelated to the current task should be compressed, summarized, or even removed, rather than piled up as-is.
3. Structured separation: Use clear headings, markers, or XML tags to separate content from different sources (e.g., project conventions, code files, retrieved snippets), helping the model distinguish “what is fact” from “what is instruction” and reducing prompt-injection risk.
4. Explicitly state the “current focus”: In each turn’s prompt, clearly state which file, which logic, and which requirement is being handled, giving the model a persistent “anchor.”

The assemble_context example below is a schematic context assembler that follows these principles:

def assemble_context(system_prompt, task, constraints, files, history, recent_k=6):
    """
    Assemble the context for a single call.
    - system_prompt: role and general rules
    - task: description of the current task
    - constraints: list of hard constraints (e.g., interface compatibility, dependency versions, compliance requirements)
    - files: code or docs relevant to the task, [(path, content), ...], already sorted by relevance
    - history: prior conversation, [(role, content), ...]
    - recent_k: how many recent turns of conversation to keep verbatim
    """
    blocks = [f"<system>\n{system_prompt}\n</system>",
              f"<task>\n{task}\n</task>"]
    # Put hard constraints up front so they stay in a high-attention region even in a long context
    if constraints:
        blocks.append("<constraints>\n" +
                      "\n".join(f"- {c}" for c in constraints) +
                      "\n</constraints>")
    # Inject relevant files after sorting by relevance
    for i, (path, content) in enumerate(files):
        blocks.append(f'<file id="{i}" path="{path}">\n{content}\n</file>')
    # Keep only the most recent turns; older content is handled in the compaction stage
    if history:
        recent = "\n\n".join(f"{role}: {content}" for role, content in history[-recent_k:])
        blocks.append(f"<recent_conversation>\n{recent}\n</recent_conversation>")
    # Put acceptance criteria at the end and reference constraint IDs
    acceptance = f"Acceptance: satisfy {task}; strictly follow every item in <constraints>; run relevant tests when done."
    blocks.append(f"<acceptance_criteria>{acceptance}</acceptance_criteria>")
    return "\n\n".join(blocks)

It’s worth emphasizing that parameters like recent_k in the schematic code are just examples; actual values should be calibrated based on task type and evaluation results. Also, retrieved results and file contents are “external data”—any “instructions” within them should never be executed as system instructions. This is a basic principle for guarding against prompt injection.

2.4 Long Context and Retrieval Augmentation: Complementary, Not Replacement

Long context doesn’t make RAG obsolete; the two solve problems at different layers:

• RAG solves “finding it”: Selecting a small number of the most relevant snippets from a vast corpus, at controllable cost and latency.
• Long context solves “seeing it all”: Letting the model grasp the overall structure of multiple related files in a single inference.

A more sensible architecture is “hybrid”: first use keyword search, vector search, code symbol indexes, and other techniques for coarse filtering, then inject the filtered results together with the task description into the long context. This both controls input length and preserves the global view.

3. Reasoning-Effort Tiering: Spending Compute Where It Counts

3.1 The Root Problem: A Single Reasoning Depth Can’t Balance Cost and Quality

“Deep thinking” models excel at math, algorithms, and complex debugging, but the cost is equally obvious: the thinking phase consumes large amounts of reasoning tokens, responses are slower, and costs are higher; with simple problems, they may even “overthink” and complicate an obvious answer.

If every request used maximum reasoning effort, teams would quickly find that costs are devoured by a flood of simple requests while users lose patience waiting. Conversely, if everything used minimum effort, accuracy on complex tasks would drop noticeably. So the mainstream industry approach is to make reasoning depth configurable, usually divided into several tiers:

Tier	Typical Use Cases	Trade-off Characteristics
Low	Format conversion, variable renaming, simple Q&A, unit-test scaffolding	Low latency, low cost; suited to high-frequency lightweight tasks
Medium	Routine business code, defect localization, code review, doc generation	Balance point between quality and cost
High	Architecture design, cross-module refactoring, complex defect analysis	Quality and depth first; consumption rises significantly
Max	Tough defect hunting, algorithm optimization, deep security analysis	Quality first; suited to low-frequency, high-value tasks

At the implementation level, many platforms control the thinking budget through a reasoning-effort parameter (exact name and values depend on the platform’s documentation), while “adaptive thinking” lets the model or orchestration layer decide when to engage in deeper reasoning. It should be clear that reasoning effort affects the depth and thoroughness of thinking, not the model’s knowledge or capability ceiling; actual differences across platforms should be judged by official evaluations.

3.2 Adaptive Routing: Let the Orchestration Layer “Match the Dish to the Appetite”

Relying entirely on manual tier selection isn’t realistic. A better approach is to have the application orchestration layer route automatically based on task characteristics, with the core principle: judge first, then set the tier; fall back first, then optimize.

import re
def route_effort(task: str, file_count: int, has_tests: bool) -> str:
    """
    Choose reasoning effort based on task characteristics (example rules; calibrate against real-world measurements).
    - task: task description
    - file_count: expected number of files involved
    - has_tests: whether the project has runnable tests
    """
    text = task.lower()
    # 1) Clearly lightweight tasks go straight to the low tier
    lightweight = ["rename", "format", "add comments", "translate", "fix typo", "generate boilerplate"]
    if any(k in text for k in lightweight) and file_count <= 2:
        return "low"
    # 2) Changes spanning multiple modules, or tasks needing architectural judgment, use the high tier
    if file_count >= 8 or any(k in text for k in ["refactor", "architecture", "migrate", "performance"]):
        return "high"
    # 3) High-risk areas like algorithms, concurrency, and security use the max tier
    high_risk = ["concurrency", "deadlock", "encryption", "authentication", "authorization", "overflow", "injection"]
    if any(k in text for k in high_risk):
        return "max"
    # 4) For hard-to-reproduce defect descriptions, start at medium to localize, then escalate after failure
    if re.search(r"(intermittent|occasional|random|not reproducible|in production)", text):
        return escalate_after_retry("medium")  # pseudo-code: auto-escalate after failure
    # 5) Default to medium effort
    return "medium"

Note: The 4th rule only illustrates the “escalate after failure” idea; in a real implementation, a retry strategy should return the tier, rather than returning a string directly.

A few practical points are worth noting in engineering:

• Default tier + override: Platform-level config provides a default tier, overridable at the project and task level, balancing unified governance with flexible adjustment.
• Escalate after failure: If a task repeatedly fails verification at a given tier, automatically switch to a higher tier and retry, avoiding wasting an attempt on the wrong tier.
• Tell the user explicitly: Showing the reasoning effort currently in use, estimated time, and approximate cost has a big impact on the developer experience.
• Cache-friendly: Cache the results of high-frequency low-tier requests to reduce cost further.

3.3 Calibrate Tiers with Evaluation Data, Not Gut Feeling

Tier boundaries shouldn’t be set arbitrarily; they should come from evaluation. It’s recommended to build an internal benchmark set covering real tasks—including bug fixes, feature implementation, refactoring, code explanation, and so on—with each type labeled for expected difficulty. Then run each tier separately and record three metrics:

• Pass rate: whether the task passes all verification in one shot;
• Average token consumption (input, output, and thinking tracked separately);
• P95 latency.

Plotting the three as curves usually reveals a clear inflection point: below a certain tier, pass rate drops sharply; above a certain tier, pass rate plateaus while cost keeps rising. The inflection point is the recommended tier for that task type, and routing rules should be based on this data and re-calibrated regularly as models and prompts evolve.

4. Context Compaction and Memory Management: The “Endurance” Problem of Long Tasks

4.1 Why Long Tasks Are Bound to “Blow Up the Context”

For an agent session that runs for hours, context growth comes mainly from four sources:

1. Conversation history: back-and-forth exchanges between the user and the model;
2. Tool-call records: file contents, command output, compile logs, and the like, easily running into thousands of tokens;
3. Intermediate reasoning: chains of thought and intermediate conclusions;
4. The code itself: file contents read and written multiple times repeatedly enter the context.

Even though a million-token window looks ample, a large refactoring task can approach the limit after a few dozen tool calls. Without a compaction mechanism, the outcome is usually one of two things: the request errors out outright, or the system crudely truncates early content, causing the model to “forget” its original constraints and decision rationale.

4.2 Compaction Strategy: Sliding Window + Structured Summary

A fairly mature industry approach is “sliding window + summary replacement”: when the context approaches a threshold, hand the earlier history to the model for a structured summary, then replace the original text with the summary. The key is that the summary must retain information still relevant to subsequent tasks and discard details that have already been “used.”

Which information must be retained? Usually the following five categories:

• Task goal and acceptance criteria: the “North Star” of the entire session;
• Key decisions already made: e.g., “use the strategy pattern instead of if-else,” “keep the old interface for caller compatibility”;
• File change list: which files were changed and how, for later verification and rollback;
• Unfinished to-dos and blockers: to prevent the task from “breaking up”;
• Failed attempts and their reasons: to avoid the model repeating the same mistakes.

Conversely, large blocks of raw file content, complete command output, and intermediate approaches superseded by later decisions can all be compressed or dropped in the summary.

def compact_context(context, threshold_ratio=0.8, model_window=1_000_000):
    """
    Context compaction (schematic):
    When token usage exceeds threshold_ratio of the window,
    compress older history (except the system prompt and the most recent turns) into a structured summary.
    """
    if count_tokens(context) < threshold_ratio * model_window:
        return context  # Below threshold; no compaction needed
    system = context["system"]                      # always kept, not subject to compaction
    recent = context["messages"][-6:]               # keep the last few turns verbatim
    older = context["messages"][:-6]
    summary_prompt = f"""
Compress the following earlier conversation and tool-call records into a structured summary. You must retain:
1. The task goal and acceptance criteria;
2. Key confirmed decisions and their rationale;
3. The list of modified files and key points;
4. Unfinished to-dos and current blockers;
5. Approaches tried but failed, and the reasons for failure.
Other details may be omitted. Output as a Markdown list, keeping the total under 1500 tokens.
Records:
{stringify(older)}
"""
    summary = call_model(system_prompt="You are a rigorous conversation-summarization assistant",
                         messages=[{"role": "user", "content": summary_prompt}],
                         effort="low")  # summarization doesn't need high reasoning effort
    # Inject the summary as system-level memory so it stays in a high-attention region of the context
    return {
        "system": system + "\n\n# Session Memory (generated from compressed history)\n" + summary,
        "messages": recent,
    }

This implementation makes several deliberate trade-offs:

• The summarization call uses low reasoning effort to keep the cost of compaction itself down;
• The summary is placed after the system prompt, leveraging the “high attention at start and end” characteristic;
• Each compaction keeps only the most recent turns verbatim, preventing the compaction result itself from growing without bound.

4.3 Externalized Memory: Make the “Repository” the Memory Carrier, Not the “Context”

Compaction solves “in-session memory,” but in real projects, a more robust approach is to externalize memory into the repository itself:

• Design docs and decision records: Write architecture decisions into Markdown files under docs/, readable at the start of a new session;
• Task-list files: Maintain a TODO.md or PLAN.md, checking off items as they’re completed so the model can “glance at progress” anytime;
• Structured memory store: Store historical session summaries and bug-fix records in a database or vector store, retrievable by project and module.

This closely mirrors how human engineers work: people don’t keep every detail in their heads either—they write docs, log issues, and leave comments. Migrating “memory” from the context window to a durable, evolving external carrier is key to keeping long tasks coherent.

4.4 Checkpoints and Rollback: A Second Line of Defense Beyond Compaction

Compaction is lossy. To prevent key information from being lost during compaction and sending the task off track, the agent should create a checkpoint before executing important steps—for example, by tagging in version control or committing on a separate branch. If subsequent verification fails, it can return to the checkpoint and re-plan instead of starting over. Checkpoints also naturally form a task trail, which helps with post-hoc auditing and retrospectives.

5. The Engineering Loop of Agentic Coding: From “It Runs” to “Reliable”

5.1 Plan—Execute—Verify: Break “Do It All at Once” into “Small, Fast Steps”

Letting an agent write an entire feature in one shot often causes it to drift off target partway through. A more reliable approach is to first have it output a short implementation plan—for example: files involved, change steps, verification method, and potential risks. After the developer confirms the plan, let the agent execute step by step. The plan itself is a low-cost, editable “contract.”

The execution phase likewise emphasizes small iterations: run the relevant tests after each small, verifiable change, rather than modifying a dozen files and then verifying all at once. That way, if something goes wrong, the search space is small.

5.2 Tool Design: The Invisible Lever That Determines an Agent’s Ceiling

The same model, equipped with well-designed tools, can perform very differently. A few general principles:

• Appropriate granularity: Tools that are too fine (e.g., “read one line of code”) make the number of calls explode; tools that are too coarse (e.g., “fix all defects”) make the intermediate process unobservable. Granularity like “read file,” “write file,” “run tests,” and “code search” is usually about right.
• Structured return values: Error messages should include the file path, line number, error type, and fix suggestion, rather than a hard-to-parse log dump.
• Confirmation for destructive operations: High-risk operations such as deleting files, force-pushing, and running database migrations should require explicit developer confirmation or be confined to an isolated environment.
• Least privilege: Grant only the directory access and command-execution permissions necessary to complete the task.

5.3 Observability: Fit the Agent with a “Black Box”

Every step an agent takes consumes tokens, calls tools, and produces side effects. It’s recommended to record a complete execution trace, including a summary of each call’s input, the tool and parameters, the return value, elapsed time, token consumption, and the reasoning effort used. Combined with logging services and monitoring alerts, this can answer several key questions:

• Which steps are the most time-consuming and most expensive?
• Where do failures cluster—which task types and which tier?
• Are there tool-call loops?

This data is both the basis for optimizing cost and the foundation for incident retrospectives.

6. Enterprise Adoption: Compliance, Cost, and Collaboration Boundaries

6.1 Data Compliance

Code often involves core intellectual property and may even contain sensitive data such as keys and personal information. Adoption should ensure:

• Redaction up front: Filter sensitive fields such as keys, tokens, phone numbers, and ID numbers before calling the model;
• Regional control: For organizations with data-residency requirements, prioritize platform capabilities that support processing in a specified region;
• Encryption in transit and at rest: Sensitive intermediate artifacts (such as summaries containing business logic) should not be persisted in plaintext for the long term.

6.2 Cost Governance

Both long context and deep reasoning amplify costs. Governance measures include:

• Set a default reasoning effort by task type, and cap the maximum token budget per task;
• Cache the results of high-frequency, repetitive requests;
• Use low-effort tiers for offline tasks that can be batched (such as bulk code review and historical defect archiving);
• Build a usage dashboard that breaks down consumption by team, project, and task type to catch anomalies early.

6.3 Human-Machine Collaboration Boundaries

Agents aren’t suitable for a “hands-off” approach in every scenario. A fairly safe division is:

• Safe to automate: adding unit tests, writing comments, formatting, dependency upgrades, log analysis, and doc generation;
• Needs human review: changes touching core transaction flows, permission models, or data migration—the agent produces a proposal, and the developer makes the decision;
• Must be decided by a human: architecture choices, public-interface changes, and compliance-related logic—the agent only provides analysis material.

6.4 Building in the Cloud: Assembling a Complete Pipeline from General Capabilities

To actually run the above design, a typical cloud pipeline can be assembled from general capabilities—for example: an LLM service (for model inference), object storage (for code snapshots and build artifacts), a container platform (to host the agent runtime and sandboxes), a message queue (to decouple long-task scheduling and execution), and logging with monitoring and alerting (to record traces and anomalies). As one example, developers working with Baidu AI Cloud can build an integrated setup—from model access and agent orchestration to observability and governance—on top of its general product capabilities and the model services and development toolchain offered by the Qianfan model platform, without having to build the underlying infrastructure themselves.

Conclusion: Treat AI as a Colleague, Not a Search Engine

The essence of Agentic Coding isn’t making the model “better at chatting,” but having it take responsibility for a complete piece of work: understanding the goal, planning the path, executing operations, checking results, and being accountable for outcomes. To make this capability land reliably, the developer’s focus needs to shift from “switch to a stronger model” to “build the system well”:

1. Context: not just fitting it in, but keeping key constraints within the model’s attention;
2. Reasoning effort: calibrate tiers with evaluation data, so simple tasks are fast and complex tasks are deep;
3. Memory and compaction: structured summaries within a session, and repo docs and external memory across sessions;
4. Engineering loop: tools, sandboxes, verification, checkpoints, and observability—none can be missing;
5. Governance: design compliance, cost, and human-machine boundaries up front, rather than patching them after the fact.

Model capabilities are still evolving rapidly, but engineering problems like “context management, reasoning-effort scheduling, and task-loop design” won’t automatically disappear with model upgrades. Whoever builds this infrastructure first will be able to turn AI from a “Q&A tool” into a genuinely deliverable source of productivity faster.