Adapting and Selecting Coding Agent Frameworks for LLMs: A Practical Guide

1. Why the Same Model Performs Differently in Different Frameworks

Many developers have experienced a similar puzzle: taking a highly capable coding LLM and integrating it into different AI coding tools results in vastly different experiences. For the same multi-file refactoring task, a model might stably complete the full closed loop of “locate, modify, verify, and fix” within one framework. Switch to another, and it frequently misses steps, fails tool calls, loses context, or requires constant rework.

This does not mean the model itself has changed. Instead, the model’s surrounding execution framework—often called the harness in the industry—is at play. You can think of the LLM as the “engine” and the harness as the vehicle’s chassis, transmission, and electronic control system. No matter how powerful the engine is, if the transmission and control systems are mismatched, the vehicle’s actual performance will suffer. Therefore, “which framework the model is plugged into” is fundamentally a systems engineering problem, not just a simple matter of brand preference.

2. Deconstructing the Coding Agent Framework: Five Critical Components

A mature coding agent framework typically consists of the following five collaborating components. A weakness in any single area will amplify the model’s shortcomings.

1. System Prompts and Task Specifications
The system prompt defines the model’s role, workflow, and output constraints. Mature frameworks explicitly require the model to “read code before making changes,” “run verification after modifications,” and “know how to roll back on failure,” making implicit engineering habits explicit. Under-polished frameworks often provide only simple instructions, making it difficult for a strong model to translate its capabilities into stable execution behaviors.

2. Toolchain and Invocation Protocols
File read/write, global code search, command-line execution, test runs, and version control operations are the “hands and feet” of a coding agent. Whether tool definitions are strict (e.g., clear parameter semantics, standardized return formats, readable error messages) directly determines if the model can succeed on the first invocation. When protocols are poorly designed, the model will engage in frequent trial and error, manifesting as “slow responses, high token consumption, and incorrect results.”

3. Context Engineering
During long tasks, the framework must decide what information to inject into the context: directory structures, relevant files, historical actions, and test outputs. Mature frameworks perform retrieval trimming, phased summarization, and key state retention, ensuring the model always “sees” the most critical information. Crude frameworks either flood the context with irrelevant content, diluting the model’s attention, or discard key clues too early, causing the model to suffer from “amnesia.”

4. Project-Level Memory and Instruction Files
Instruction files in the repository root directory (typically in Markdown format) act as the carrier for “team engineering standards”: tech stack conventions, code styles, prohibited actions, and verification workflows. How the framework handles the timing of reading these files, their priority, and their integration with system prompts significantly impacts the consistency of the model’s behavior.

5. Sub-Agents and Permission Management
Complex tasks need to be decomposed: retrieval-oriented subtasks are assigned to read-only sub-agents, write operations are concentrated in the main workflow, and sensitive commands require human confirmation. A robust permission system not only prevents accidental deletions or modifications but also empowers the model to execute bolder automated operations, knowing that failures can be intercepted and rolled back.

3. Objective Comparison of Two Framework Orientations

Based on public community discussions and practical feedback, common frameworks generally fall into two orientations, each with its own applicable scenarios.

One orientation is the proprietary tooling provided by model vendors. The advantage lies in the synchronized iteration with their own model versions: when a new model is released, prompts and tool protocols are updated simultaneously, minimizing adaptation issues. Billing is often integrated with subscriptions, making overall costs more controllable. The drawback is that the execution framework itself may have had less time to mature, and its orchestration, fault tolerance, and context management capabilities for complex, multi-step tasks might still be rapidly evolving.

The other orientation comprises general-purpose execution frameworks honed by long periods of real-world tasks. Their system prompts, tool protocols, sub-agent mechanisms, and permission systems are relatively mature, creating a significant “amplification effect” for capable models—the stronger the model, the more valuable the execution discipline provided by the framework. The trade-off is potentially higher subscription costs, and adapting third-party models may require extra configuration, meaning they might not work perfectly out-of-the-box.

From this, we can derive a more actionable evaluation framework: instead of asking “which framework is better,” ask “which orientation best matches my current model, my task types, and my cost constraints.” When model capabilities are weaker, the execution discipline of a mature framework provides a safety net; when model capabilities are strong, the framework quality almost entirely determines whether that potential ceiling can be realized.

4. A Practical Framework Selection and Evaluation Method

Rather than relying on anecdotal experiences or online reviews, conducting a small-scale, quantifiable comparative experiment is more effective:

1. Build a task set. Select 10 to 20 representative tasks from real-world work, covering four categories: bug fixes (with clear reproduction paths), small feature development (spanning 2-3 files), cross-module refactoring (involving interface changes), and test completion. Document the acceptance criteria for each task.
2. Control variables. Use the same model, the same codebase snapshot, and the same task set across different frameworks. Execute without human intervention and record the entire process.
3. Quantify metrics. It is recommended to track at least these five metrics:
   • First-pass success rate: The percentage of tasks meeting acceptance criteria without human correction.
   • Average rework rounds: The number of times the model self-corrected or required human intervention.
   • Cost: The expenditure calculated from token consumption per task.
   • Long-task stability: The frequency of “going off track” in tasks exceeding 20 steps.
   • Security: Whether dangerous commands were correctly intercepted or prompted for confirmation.
4. Comprehensive scoring. Assign weights based on team priorities. For instance, if production code tasks dominate, increase the weight of stability; if budget is tight, increase the weight of cost.

This method usually takes only a day or two to complete but transforms subjective feelings like “A feels better than B” into reproducible, data-driven conclusions.

5. Tuning Practices After Selecting a Framework

Selection is just the starting point. The following practices can further unlock the potential of the model and the framework.

1. Write robust project-level instruction files. An example is shown below:

# Project Collaboration Instructions
## Tech Stack
- Backend uses Python 3.11, following internal framework specifications
- Data layer uses a relational database, with migration scripts centrally managed
## Code Standards
- All functions must include type annotations and brief descriptions
- Run the project test suite after each modification; completion requires all tests to pass
## Prohibited Actions
- Do not modify applied historical migration scripts
- Do not hardcode secrets; read uniformly from environment variables
- Do not execute deletion or external exposure commands without confirmation

Instruction files should be concise, executable, and continuously maintained as the project evolves. Avoid piling up descriptions irrelevant to the model’s behavior.

2. Follow the principle of least privilege. For daily development, only enable file read/write and test execution. Commands involving deployment, deletion, or external network access should strictly require human confirmation.

3. Leverage sub-agent decomposition. Assign large-scale code searches to read-only subtasks, allowing the main workflow to focus on modifications and verification. This significantly reduces context pollution.

4. Manage context length. Conclude long conversations promptly. Let the framework generate summaries before starting a new phase to prevent irrelevant history from interfering with the model’s judgment.

6. Scenario-Based Selection Recommendations

• Individual developers, budget-sensitive, primarily routine CRUD tasks: Prioritize proprietary tools from the model vendor. Cost and compatibility are usually the optimal solution.
• Teams handling complex refactoring, cross-module long tasks, with high delivery quality requirements: It is worth paying a premium for the stability of a mature execution framework. Simultaneously, validate the actual ROI using the evaluation method in Section 4.
• Enterprise-level adoption: It is recommended to complete integration and governance on a unified model service platform. For example, platforms like Baidu AI Cloud allow you to combine model services with intelligent development toolchains. Start with a small-scale pilot to validate metrics, then gradually scale up, while synchronously establishing code review and permission audit workflows.

7. Conclusion

The phenomenon that “the same model performs differently in different frameworks” is not mysticism; it is the combined result of system prompts, toolchains, context engineering, project memory, and permission systems. For developers, the truly valuable approach is to understand the structural logic of the harness, replace subjective impressions with quantifiable experiments, and continuously tune through instruction files and permission configurations. By achieving these three points, regardless of the framework combination chosen, you can ensure that LLMs stably deliver their true capabilities in coding scenarios.