Multi-Model Routing, Long-Term Memory, and Interruptible Agent Engineering in Practice
This article presents a practical engineering approach for building AI agents that can handle complex software development tasks across multiple LLM APIs. It covers three core design areas: making agent execution interruptible so failures can be corrected mid-task, implementing structured long-term memory through project notes with access controls and automated secret scanning, and routing tasks across models based on input type, cost, and reasoning difficulty. It also provides a minimal executable checklist for automated pipelines and highlights common pitfalls such as missing retry paths and inconsistent cost measurements.
As large language models improve, AI agents have moved beyond simple chat tools and are now used in requirements analysis, coding, testing, and maintenance within software development. In practice, relying on a single model introduces several problems: platform accounts may be rate-limited or the service may go down, failures in complex tasks are hard to diagnose, some models have weak image understanding, and token costs can fluctuate noticeably with call volume.
The approach described here does not guarantee cost savings or fewer errors in every scenario, but it offers a set of engineering practices that can be validated step by step. The prerequisites are: a version-controlled repository, access to two or more model APIs, and a way to record token usage for each call.
1. How to Determine If an Agent Is Interruptible
Many tools are result-oriented. For simple tasks such as generating a single function or writing a test case, the output is usually satisfactory. But when a task involves multiple sub-agents working in parallel or running several scripts across directories, a result-oriented agent is problematic: if it makes an error early on, the developer often has to wait until the whole run finishes, see the final failure, and then restart from scratch.
To judge whether an agent framework is suitable for complex tasks, check for the following characteristics:
- Each execution step outputs both the planned action and the actual command that was run.
- Every tool call records its input, output, and exit code, and saves these details to a log file.
- Execution can be paused at any step, allowing the injection of corrective instructions or additional context, then resumed from that same step without restarting the whole process.
- After an interruption, the agent can state which steps have been completed and which remain.
A simple validation method: set up a test task with four steps and deliberately make the second step choose the wrong branch. Observe whether the agent stops before the third step and reports the anomaly. If it continues all the way to the end, the framework’s interruptibility is insufficient.
2. Designing Long-Term Memory
Traditional conversations rely on the context window of the current session. Once a session becomes long or is started anew, the model loses context about the overall project structure. Long-term memory fills this gap. A common approach is to have the agent maintain structured notes inside the project.
Storage location: Notes can be placed in a docs/agent-notes/ directory within the project, or in a local directory outside the repository. Notes use Markdown format, are organized per module, and are not committed to a public repository.
Note template: Each module’s note should contain at least: the module’s responsibility, key interfaces, the most recent change (date and commit ID), and unresolved issues. A fixed template ensures that every update follows the same structure, making manual review easier.
Updating and conflict handling: After completing a staged task, the agent updates the corresponding module note. When conflicting facts appear, the content from the latest commit takes precedence. Older records are marked as “deprecated” with the reason preserved, rather than being deleted outright, so the history remains traceable.
Access control: The agent has write permission only to the notes directory. Its permissions to modify source code are configured separately. The notes must never contain keys, tokens, personal data, or customer data. The repository should be private, and note changes go through the same code review as code changes.
Automated commit-time security check: Before each commit, a script scans the notes and the files to be committed, using regular expressions to match common secret formats—for example, strings starting with sk- or the header of a private key. If any match is found, the script aborts the commit and prompts manual handling.
Cleanup policy: Notes are reviewed once per quarter. If a module has been deleted, its note is removed. If a note has not been updated in over 180 days, it is marked as “to be re-checked” by a responsible person, who then decides whether to keep or delete it.
3. Multi-Model Routing
Different models vary significantly in unit price, image understanding, and long-text reasoning. Therefore, tasks can be assigned to models based on task type. Routing rules are stored in a configuration file rather than hardcoded in scripts, so they can be adjusted easily.
| Task Type | Typical Examples | Selection Criteria | Cost Measurement | Failure Fallback |
|---|---|---|---|---|
| Routine generation | Boilerplate code, unit tests, format conversion | Passes project tests; prefer the lower-cost model | Average tokens per task × unit price × number of calls | If tests fail, rerun once with the next higher-tier model |
| Multimodal tasks | Recreating front-end pages from screenshots, recognizing charts | Supports image input and meets acceptance criteria on an internal sample set | Count image tokens separately | If vision model fails consecutively, hand off to a human |
| High-difficulty reasoning | Architecture design, cross-module refactoring, tricky bug diagnosis | Requires long context and multi-step reasoning ability | Effective cost per task based on task completion rate | Pause after two consecutive failures; developer takes over |
Cost comparisons need a unified measurement method, otherwise conclusions are unreliable. Specifically: use the same set of tasks and the same version of prompts, and record cache hit rates. Looking only at model unit prices cannot reveal the actual cost difference, because failed retries and rework also add to the total cost.
Routing decisions can be based on input type (whether images are included) and keywords in the task description. The decision is passed to a dispatcher, which logs the chosen model and the reason for each selection.
4. Minimal Executable Checklist for Automated Pipelines
When building an automated workflow, start by configuring the following items, then consider extensions:
- Parameters: model name, timeout, maximum retry count, and budget limits are all written into a
pipeline.yamlfile, not inside scripts. - Dependencies: record the runtime language version and lock dependency packages to fixed versions using files such as
requirements.txtorpackage-lock.json. - Permissions: each step uses a separate least-privilege token. Steps that involve pushing code, deploying, or deleting data require manual approval.
- Error handling: if a single step fails, retry at most twice with an increasing interval. If it still fails, save logs, input data, and intermediate files, then stop subsequent steps.
- Verification: run test commands after each step; a non-zero exit code means failure.
- Cost monitoring: record the number of calls and token usage for each model on every run. Abort immediately if the budget limit is exceeded.
This checklist corresponds to verifiable checkpoints. Each item can be confirmed through configuration files or logs, without relying on the developer’s memory.
5. Common Mistakes
- Focusing only on the final result and not saving process logs. After a failure, it is impossible to pinpoint which step went wrong.
- Writing notes without permission boundaries, putting secrets or customer information into note files.
- Not recording token usage, making it impossible to judge whether costs were actually saved.
- Routing without a fallback mechanism. If one model becomes unavailable, the entire pipeline stops immediately.
- Assigning all tasks to the same model. Using a high-priced model for everything increases costs; using a low-priced model for everything may cause rework, increasing the total cost.
6. Implementation Suggestions
Start with a small module and run a trial for two to three weeks. During the trial, record the time taken, token usage, number of failures, and manual revision effort for each task. Use this data to decide whether the approach is effective before expanding it to other modules.
