Single-LLM-Engine AI Coding Agent: Lessons Learned from a Ground-Up Design

1. Starting Point: From “It Works” to “I Control It”

There is no shortage of mature, usable AI coding tools, and I use them nearly every day. But the longer I worked with them, the more often I ran into four recurring problems:

1. Capability boundaries I could not control. Tool sets, context assembly, and permission policies are mostly wrapped inside the product, which makes it hard to adapt them to my own engineering conventions.
2. Inflexible infrastructure coupling. Which models you can use, and how you switch between them, depends on the infrastructure the tool vendor provides. That leaves little room either for compliance requirements such as keeping code inside a corporate intranet, or for quickly adopting newly released models.
3. An opaque cost structure. There is no fine-grained data showing how many input and output tokens a task actually consumed, or which steps were paid for repeatedly. Without that, optimization has no starting point.
4. Behavior that is hard to reproduce. The same prompt against the same repository can produce substantially different changes on two runs, which makes post-hoc diagnosis very difficult.

I also read a number of open-source agent frameworks, and they gave me a lot of useful ideas. Most of them, however, follow a “pluggable model” path. In order to accommodate every vendor’s API style, the abstraction layers keep piling up, prompts get reduced to the least common denominator of capabilities, and the number of combinations to cover in behavior regression tests grows quickly.

So I started designing my own AI coding agent. The first attempt supported multiple engines, the way most frameworks do, and maintenance cost spiraled almost immediately. After that, I deliberately narrowed the scope to one explicit boundary: the agent adapts to exactly one model engine — a China-developed open-source large model (referred to below as the “target engine”). It follows the industry-standard “conversation plus tool calling” interaction paradigm and supports Function Calling. A conversational model with tool-calling support is precisely the right foundation for building a coding agent.

The rest of this article reviews the system’s key design decisions in the order motivation → architecture → implementation → governance.

2. Why Cut Multi-Engine Support: Narrowness as an Architectural Advantage

2.1 The Hidden Cost of Multi-Engine Adaptation

“One abstraction layer for N models” sounds elegant, but the real cost concentrates in three places:

• Divergent interface semantics. Vendors differ in how tool calls are organized across streaming chunks, when arguments get completed, how parallel calls are handled, and how reasoning content is returned. The abstraction layer either hides the differences and loses capability, or exposes them and stops being an abstraction at all.
• Prompt degradation. Every model has its own preferences for instruction format and tool usage. To stay compatible with all of them, prompts can only be written to the lowest common denominator, which is optimal for none of them.
• Test matrix explosion. The regression matrix of main loop × tool set × model behavior multiplies, and every release has to be validated one by one across several backends.

2.2 What a Single Engine Buys Back

Once the engine is fixed, the benefits are concrete:

• Deep co-optimization. System prompts, tool descriptions, and output formats can all be tuned against the target engine’s known instruction-following behavior.
• Controllable input cost. Input tokens usually account for the bulk of what a coding agent consumes, and many inference services offer an input caching mechanism: when the same prefix is hit again, it is billed at a discounted rate. Engineering work can deliberately optimize the cache hit rate and push down the cost of repeated context. A single engine also makes it straightforward to point the agent at a private inference service that complies with mainstream chat-style inference API conventions, satisfying the requirement that code never leaves the intranet.
• Reproducibility. The same model version plus fixed parameters (for example, temperature set to 0) yields near-identical behavior for the same input, so problems can be replayed and localized.
• Short iteration loop. Changing prompts, adjusting tool descriptions, and reworking context strategy can all be validated quickly against a single backend.

What ultimately determines how well an agent performs is usually not how many models it supports, but how tightly its prompts, tools, context, and verification mechanisms work together. Better to score 90 on one backend than 70 across N.

3. Overall Architecture: Layered, Single-Responsibility

The system is organized into six layers, from top to bottom:

Layer	Responsibility	Key Design
Interaction	Accept tasks, stream output, approve high-risk operations	Render while generating; approval decoupled from execution
Orchestration	Main loop, state machine, termination conditions, error recovery	Single driving loop; every step interruptible and replayable
Model access	Build and parse conversation and tool-calling requests	Adapt to one engine only; keep the interface thin
Tools	File read/write, search, execution, search-and-replace editing	Clear descriptions; structured, truncatable output
Execution	Command execution, sandbox isolation, resource limits	Tiered authorization; least privilege by default
Support	Logging, metrics, cost accounting, evaluation	End-to-end observability with replay

The core principle behind the layering is this: the model decides “what to do,” not “how to do it.” Which files to read, which commands to run, and when to stop are all executed and validated by deterministic engineering code.

4. The Core Loop: A Finite State Machine

An agent is fundamentally a loop of observe → think → act → observe again. I implemented it as a bounded loop driven by a state machine: each state admits only specific exit conditions, which prevents the model from spinning idly inside the loop.

# Pseudocode: main loop structure only
MAX_STEPS = 50          # max iterations per task
MAX_RETRIES = 3         # retries per single model call
REPEAT_LIMIT = 3        # max consecutive identical tool calls
def run_agent(task, workspace):
    messages = build_system_prompt(task) + [user_message(task)]
    loop_guard = {}                      # tool call signature -> consecutive repeat count
    for step in range(MAX_STEPS):
        # 1. Call the model: force tool-calling mode, temperature 0 for reproducibility
        resp = llm.chat(messages, tools=TOOL_SCHEMAS,
                        temperature=0, stream=False)
        msg = resp.message
        # 2. Plain-text answer: treat as task completion
        if not msg.tool_calls:
            return summarize(task, msg.content)
        # 3. Execute tool calls one by one, and prevent spinning in place
        for call in msg.tool_calls:
            key = (call.name, call.canonical_args())  # compare after normalizing args
            loop_guard[key] = loop_guard.get(key, 0) + 1
            if loop_guard[key] > REPEAT_LIMIT:
                result = (f"This call has been repeated {REPEAT_LIMIT} times "
                          f"consecutively without changing the result. Try a different approach.")
            else:
                result = execute_with_guard(call, workspace)   # permission check + execution
            messages.append(tool_result_message(call.id, truncate(result)))
        # 4. If over the context budget, compact and continue
        if estimate_tokens(messages) > CONTEXT_BUDGET:
            messages = compact(messages, keep_last=6)
    return partial_report(task, messages)

A few of these decisions deserve individual explanation:

• Forced tool-calling mode. The first step of almost any programming task is reading or writing files. Requiring a tool call on every turn compresses the tokens spent on “thinking out loud.”
• Termination decided in engineering code. Whether the task is done is judged by code, not by the model announcing “I’ve finished.” For example, passing the project’s lint and test suites can be a hard gate before wrapping up.
• Loop guard. While fixing bugs, models often fall into an “read the same file → make the same edit” dead loop. Counting normalized call signatures and feeding the count back to the model breaks the stalemate effectively.
• Retry policy for model calls. Retries apply only to recoverable errors such as timeouts, rate limiting, and 5xx responses, using exponential backoff with random jitter so that many concurrent tasks do not trigger a retry storm. Deterministic errors such as context overflow or invalid arguments are not retried; the request is corrected directly.

5. Tool Set Design: Handing “Capability” to Deterministic Code

5.1 The Tool List

What matters about tools is not quantity but clean boundaries and controllable output. A usable minimal set usually includes:

1. List directories and browse the codebase. Produce an overview of the project structure so the model can build global awareness.
2. Read a file. Support reading by line ranges, returning line numbers alongside the content so later references can be precise.
3. Write a file or replace it wholesale. For creating new files or rewriting large sections.
4. Search-and-replace editing. Like applying a Git patch: supply unique context plus the old and new content, and let engineering code locate and perform the replacement. Compared with asking the model to emit full diff format, this saves tokens and is less error-prone.
5. Keyword search. Regex search across the repository returning “file + line number + snippet,” used to locate symbols and references.
6. Execute a command. Classified by risk: read-only commands (such as status inspection) can run directly; write and installation commands require user approval; dangerous operations are prohibited outright.
7. Complete the task. Explicitly declare the task finished, submitting a change summary and verification results along with it.

5.2 Implementation Notes for Search-and-Replace Editing

Whether search-and-replace editing succeeds depends entirely on how strict the replacement process is. I recommend three rules:

• Uniqueness check. Zero matches reports “not found”; multiple matches report “ambiguous” and list the candidate locations. Never silently default to replacing the first occurrence.
• Position and context validation. The starting line number supplied by the model is used only as a cross-check, not as the basis for locating the target; the actual search result is authoritative.
• Fail with feedback. On failure, return the actual content snippet to the model so it can correct itself, rather than skipping the edit silently.

// Parameter schema of the search-and-replace editing tool (illustrative)
{
  "name": "apply_edit",
  "description": "In the given file, replace the unique old content with the new content. old_text must match exactly and uniquely within the file.",
  "parameters": {
    "type": "object",
    "properties": {
      "path":     {"type": "string", "description": "File path relative to the project root"},
      "old_text": {"type": "string", "description": "Original content to replace; must include enough context to be unique"},
      "new_text": {"type": "string", "description": "Replacement content; an empty string means deletion"}
    },
    "required": ["path", "old_text", "new_text"]
  }
}

6. Context Engineering: The Real Dividing Line

Most failures of coding agents are not “the model cannot write code” but “the model did not see the right code.” My approximate per-session token budget is:

Component	Budget Share (illustrative)	Notes
System prompt and tool descriptions	~5%	Keep as lean and stable as possible to maximize cache hits
Project structure overview	~5%	Trim the directory tree to key levels; ignore dependency and build-output directories
Search results	30%–40%	Only snippets relevant to the current task
File read and edit history	25%–35%	Keep only the latest version of a given file
Execution output and model reasoning	~10%	Truncate long output; keep only error-relevant portions

The supporting strategies are:

1. A stable prefix. System prompt, tool descriptions, and the project overview go first and stay byte-for-byte stable. Later messages are append-only, so provider-side caching gets as many hits as possible.
2. Retrieve on demand instead of loading everything. Locate targets via the structure overview and keyword search first, then read only the relevant snippets.
3. Summarize execution output. Test and build logs often run to thousands of lines; feed back only the error summary, failing cases, and related code.
4. Rolling compaction. As the budget ceiling approaches, fold earlier reasoning and tool output into a “decision record” with one small request, while preserving raw file contents, recent edits, and verification results. Compaction must be structured: keep the latest content of every file and discard intermediate versions.

def compact(messages, keep_last=6):
    """Rolling compaction: keep recent messages, fold history into a decision summary."""
    head, recent = messages[:-keep_last], messages[-keep_last:]
    summary = llm.summarize(
        "Compress the following conversation into a decision summary. Keep: task goal, "
        "list of modified files, options already ruled out, and unresolved problems. "
        "Ignore pleasantries and repeated content.",
        head,
    )
    return [system_note(f"Historical decision summary:\n{summary}")] + recent

7. The Verification Loop: Let the Agent Find Its Own Mistakes

A model declaring success as soon as it has written code is the most common failure mode of these tools. My approach makes verification a hard step inside the agent:

1. Infer the project’s conventions. Identify the language, package manager, and existing scripts from configuration files to derive the lint, type-check, build, and test commands, rather than assuming them.
2. A minimal verification set. Run in the order static checks → build → relevant tests, with timeouts and output size caps.
3. Feed failures back. Return structured error information to the model and require it to identify the root cause before editing. Also cap repairs at two rounds per verification category: if it still fails, stop, report the current state and likely causes, and hand the decision to the user instead of looping indefinitely.
4. Closing self-check. Confirm that the change scope has not exceeded the task boundary, that no debugging code is left behind, and produce a change summary.

This turns “done when written” into “done when verified,” and substantially reduces the risk of broken code being merged directly.

8. Cost and Performance Governance

Observability determines whether optimization is even possible. Every task should record:

• Input and output token counts for each model call, including how many tokens hit the cache;
• Call count, latency, and output size for each type of tool call;
• Iteration steps, verification rounds, and total task duration.

From that data you can compute the total cost per task and find hotspots such as “too many file reads,” “repeated edits,” and “overly long test output,” then optimize them specifically. On the performance side, three things matter:

• Streaming responses. Show output as it is generated so the wait is visible.
• Overlap execution with waiting. Run independent operations such as file reads and searches in parallel, and pre-read candidate files while the model is generating.
• State persistence. Interrupted tasks resume from the most recent checkpoint instead of starting over.

9. Observability and Evaluation: Turning “It Feels Better” into Data

The biggest payoff of a single engine is that an evaluation suite can accumulate reliably. I maintain a case set for typical tasks covering new feature development, bug fixing, refactoring, and writing tests, with every case carrying automatically verifiable acceptance conditions, such as whether tests pass and whether the changed files match expectations.

Every time a prompt or tool changes, the full evaluation runs again, comparing task success rate, average iteration count, average token consumption, and average duration. The tool execution layer gets unit tests of its own, for example “an ambiguous match must raise an error” and “a dangerous command must be blocked.”

10. Known Limitations and Where It Goes Next

Frankly, this design has clear limits:

• Bounded by a single model’s capability ceiling. Very complex or cross-module tasks still run into a wall.
• Reasoning quality can suffer. With non-streaming calls, the model may not emit internal reasoning, which affects performance on complex tasks. Returning reasoning content is worth evaluating later, provided the context structure is not broken.
• No long-horizon task memory. Cross-session engineering knowledge such as architecture conventions and past decisions has no persistence mechanism yet.
• Limited multimodal support. Screenshot-driven UI reconstruction and error-screenshot localization are not covered.

Next steps are to introduce project-level knowledge accumulation, writing historical decisions and conventions into a searchable knowledge base, and to firm up the evaluation baseline so every iteration has something to stand on.

Closing: Rebuilding a coding agent around a single model engine is, at its core, an exercise in subtraction. Real competitiveness comes from going deep on the execution loop, tool contracts, context, and verification mechanisms, not from stacking up interface layers. If you are building something similar, start with three questions: are the tool descriptions precise enough? Is the context stable and cacheable? Is every change backed by data? Once those answers are clear, the ceiling of the agent becomes clear too.