Building Coding Agents for Production: Multi-Model Routing and Long-Term Memory

4 hr ago0 views

Building a coding agent that can deliver, rather than merely answer questions, is largely an orchestration problem. This article breaks agent quality into four engineering dimensions and shows that only explainability depends on model capability. The rest are harness-layer concerns. It then presents a five-layer architecture: interaction, orchestration, a model gateway with routing and budget control, memory, and tools. The article covers an explicit multi-model routing policy, with YAML and Python examples, budget and fallback rules, and prefix-cache cost control. It also covers long-term memory stored in the repository (AGENT.md, notes, and ADRs), layered skills and tool authorization, credential and compliance practices, a rollout checklist, and common anti-patterns.

1. The Starting Point: “A Model That Can Chat” Is Not “An Agent That Can Deliver”

The first time many people use code generation, the feeling is that writing a small script is quick. But when a task grows to require changes across multiple modules, passing tests, and maintaining build scripts, the experience drops off a cliff. The reason is not whether the model is “smart enough.” The task itself has changed shape.

A single question-and-answer exchange is a bounded problem: the input is complete, the output is delivered in one shot, and correctness can be judged immediately. Engineering work is an unbounded problem: requirements get clarified as you go, intermediate artifacts need to be written to files, failure points need to be located, and a chosen path may have to be retraced. Whether such a task gets done depends mostly on three things:

Process observability: Does each action leave a traceable record, so that a failure can be traced to a specific step?
Process intervenability: Can the agent be stopped at any step and redirected, instead of having the entire output redone after the fact?
State continuity across sessions: Context windows get compacted and sessions get interrupted. Does the knowledge a task depends on persist outside the session?

If only the capability to generate code exists, the more complex the task, the higher the hidden cost the user carries. This is not a token cost. It is the cognitive cost of “not knowing where the agent went wrong.”

2. Requirements Modeling: Turning Experience into Four Implementable Dimensions

To address these problems systematically, a vague sense of “works well” first has to be translated into concrete engineering targets.

Dimension	What users experience	How it is implemented
Explainability	Every step explains why it was taken	A mandatory planning phase that outputs its plan, plus step-level logs
Interruptibility	Can be stopped and redirected at any time	An interactive session loop and a checkpoint mechanism
State persistence	Still “remembers” the project months later	Project memory and decision records stored as files
Extensibility	A different toolkit for each project	Directory-based orchestration of skills and tool protocols

It is worth emphasizing that, of these four dimensions, only the first depends in part on model capability. The other three are basically engineering problems at the harness (runtime framework) layer. This is also why products built on similar underlying models can still feel very different in practice. The gap mostly comes from how the orchestration layer is designed.

3. Overall Architecture: Gateway, Routing, Memory, Skills, and Observability

A coding agent system built to evolve over the long term can usually be abstracted into five layers:

┌─────────────────────────────────────────┐
│ Interaction layer: terminal / IDE / task queue   │
├─────────────────────────────────────────┤
│ Orchestration layer: plan generation, step       │
│ execution, checkpoints, rollback                 │
├──────────────┬──────────────────────────┤
│ Model gateway │ Routing policy + quota/budget control │
├──────────────┴──────────────────────────┤
│ Memory layer: project memory / cross-project     │
│ memory / decision records                        │
├─────────────────────────────────────────┤
│ Tool layer: files, shell, tests, retrieval,      │
│ external APIs                                    │
└─────────────────────────────────────────┘

The core design principle is this: the model can be swapped; the orchestration and memory cannot. Keep memory and skills in the project repository rather than tied to one particular session. Then, when the model is replaced, those assets remain.

4. Multi-Model Routing: Route by Task Characteristics, Not Personal Preference

Models make different trade-offs in reasoning depth, context length, instruction following, multimodal understanding, and cost per token. The sensible engineering approach is not to “pick the single best one.” It is to define an explicit routing policy.

4.1 What Routing Decisions Should Be Based On

The policy should cover at least four signals:

Input modality: Does the input contain images, audio, or other media, and does the task require visual understanding?
Context size: How many files are involved, what is the estimated token volume, and is an ultra-long context needed?
Task complexity: Are there multiple parallel subtasks or multi-round script execution?
Cost constraints: What budget remains for the day or month, and what cache-hit rate is expected?

4.2 Configuration Example

# router.yaml - logical tiers decoupled from specific vendors; actual endpoints are injected via environment variables
routes:
  cheap:
    purpose: Routine edits, comments, formatting, and small single-file changes
    tier: small-chat
    max_context: 32k
  heavy:
    purpose: Cross-module refactoring, multi-subtask planning, long-chain debugging
    tier: large-reasoning
    max_context: 200k
  vision:
    purpose: Screenshot understanding, UI replication, chart and document parsing
    tier: vision-capable
fallback:
  on_rate_limit: degrade_to_cheap
  on_context_overflow: compact_then_retry
budget:
  daily_token_cap: 20000000
  cache_hit_ratio_target: 0.70
  alert_threshold: 0.85

class ModelRouter:
    """Routes by task characteristics rather than vendor preference; endpoints and credentials are injected externally."""
    def route(self, task: Task) -> str:
        if task.has_image_input:
            return "vision"
        if task.est_tokens > 60_000 or task.planned_subtasks >= 3:
            return "heavy"
        if self.budget.exceeded_today():
            return "cheap"
        return "cheap"

The benefit of this design is a controllable cost curve. The vast majority of routine changes go through the low-cost tier, and only tasks that genuinely need deep reasoning or vision trigger the high-cost tier. In practice, when cache reuse is handled well, routine tasks cost significantly less than with a setup that sends everything to high-end models.

4.3 Cache Hits Are the First Lever of Cost Control

Most routine changes concentrate on a few hot files. Keeping the system prompt, project description, and skill docs in a stable position at the start of each request substantially raises the prefix cache hit rate. One rule matters here: never put dynamic content (timestamps, random IDs, changing counters) in the request prefix. Otherwise, the entire prefix is invalidated.

5. Implementing Long-Term Memory

“A coding agent that remembers my project” sounds like a model feature. In reality, it is the result of designing externalized state.

5.1 Using the Filesystem as the State Store

Keep a conventional directory layout in the repository:

your-project/
├── AGENT.md              # Project charter: tech stack, directory conventions, coding standards, common commands
├── .agent/
│   ├── notes/
│   │   ├── progress.md   # Current progress and next steps
│   │   ├── decisions/    # Architecture Decision Records (ADRs), one file per decision
│   │   └── pitfalls/     # Pitfalls: environment dependencies, version constraints
│   └── skills/
│       ├── data-pipeline/SKILL.md
│       └── ui-review/SKILL.md

5.2 Three Layers of Memory

Project memory: AGENT.md plus .agent/notes/. This describes what the project is, how to run it, and what constraints it has.
Cross-project memory: General conventions extracted from multiple projects, such as team coding standards, commit message format, and how tests are organized. Store these separately and inject them as a shared prefix.
Decision memory: The ADRs in decisions/. This is the layer with the highest long-term value. It records why a choice was made at the time, whereas code can only record what things are like now.

5.3 When to Write to Memory

Do not write to memory at the end of every conversation. Noise will quickly drown out the signal. Instead, trigger a write only at three kinds of moments:

When a task is completed: Update progress.md and commit the code changes.
When a trade-off is made: Add an ADR that records the context, the alternatives, and the conclusion.
After a failure has been reproduced: Record the environment- or version-related pitfall in pitfalls/.

Memory should also have capacity limits and an archiving policy. Move outdated entries to an archive directory so that historical noise does not crowd out the context.

6. Layered Orchestration of Skills and Tools

A skill is essentially “the way of doing a certain class of task” frozen into a reusable set of documents and scripts. Its value lies in this: fewer prompts, more consistent output.

Each skill should contain three parts:

SKILL.md: When it applies, prerequisites, execution steps, failure handling, and how to verify the result.
scripts/: Deterministic steps, expressed in code as much as possible rather than in natural language.
references/: Any required sample data or specification documents.

Tool access is authorized in layers through a standard protocol:

Read-only tools (file reads, retrieval, log queries) are available by default.
Write operations (file writes, command execution, external calls) are governed by an allowlist declared for each project.
High-risk operations (deletion, changes to production environments) require explicit confirmation.

As pipelines take shape, the way you interact with the agent shifts from “issuing instructions step by step” to “naming a task.” This is a clear sign of increasing maturity.

7. Compliance, Account Stability, and Credential Management

Accessing third-party services across regions brings three practical risks:

Account stability risk: Policy changes can cause service interruptions.
Data residency and compliance risk: Code and business data leave the jurisdiction in which they originate.
Uncontrolled billing risk: Costs can grow beyond what was planned.

The corresponding engineering countermeasures are clear:

Centralized credential management: Read all endpoints and keys from environment variables or a secrets management service. Never hardcode them into the repository or skill docs.
Audit logs: For every request, record the routing target, token usage, latency, and status code in a logging service. This makes it easier to attribute costs and troubleshoot failures.
Fallback chain: Define fallback policies at the gateway. If the primary channel is unavailable, switch to a backup channel automatically, and flag the task as “produced by a fallback model; review required.”
Sensitive-data routing: Route tasks that involve core code and business data to a channel that meets your compliance requirements first. Redact sensitive data before sending it across channels.

To centrally manage the capabilities of multiple model providers, you can use a model service platform offered by a cloud provider (for example, the model access services from Baidu AI Cloud). It handles unified authentication, quota allocation, and call monitoring, so you do not have to reimplement gateway logic in every project.

8. Rollout Checklist and Common Anti-Patterns

Checklist

Does the project root have a structured AGENT.md and a memory directory?
Are all model endpoints accessed through the gateway and a secrets management service, with no hardcoded credentials?
Is the routing policy defined by task characteristics rather than personal preference?
Is the request prefix stable, and can the cache hit rate be monitored?
Does every skill have clear verification and failure-handling steps?
Is there a human confirmation step for high-risk operations?

Common anti-patterns

Treating the entire conversation as memory: Information will inevitably be lost after context compaction, so long-term knowledge must be kept outside the conversation.
Setting no budget cap: A complex task can consume a large quota while running unattended.
Writing skill docs as prose: The more the instructions are written in natural language, the more ambiguous they become, so script them wherever possible.
Trusting the output unconditionally: A coding agent can state wrong conclusions with confidence, so critical paths need tests or human review as a safety net.
Ignoring explanation: Accepting results without the reasoning behind them gives up the ability to correct course midway.

9. Conclusion

What makes a multi-model coding agent system competitive is not how powerful any single model is. It is the engineering maturity of the orchestration layer: Is the process observable? Can it be intervened in at any time? Is knowledge stored in the repository? Is cost predictable? Are permissions clearly bounded?

A pragmatic path forward looks like this:

First, consolidate multiple providers’ model capabilities behind a single gateway.
Next, set up the project charter and memory directory so that every session starts with existing context.
Then, turn recurring workflows into skills.
Finally, complete the budget, audit, and fallback mechanisms.

Once you get there, replacing a model becomes just a configuration change rather than a rebuild of your workflow.