Why the Same LLM Performs Differently Across Coding Agents
This article explains why the same large language model can behave differently across coding agents. It breaks end-to-end coding performance into three layers — the model, the execution framework, and the engineering environment — then dissects the five parts of an execution framework: system prompts, tool contracts, context engineering, sub-agent orchestration, and permission management. It compares general mature frameworks with vendors' in-house native frameworks, shows how to build a reproducible evaluation with defined metrics and a unified cost formula, and offers a task-based selection framework plus a practical optimization checklist. The takeaway: the model sets the capability ceiling, while the framework decides how much of it you actually get.
1. The Real Issue: End-to-End Performance Is Not the Same as Model Capability
In coding-agent scenarios, what users perceive as “how well it works” is an end-to-end metric driven by at least three layers:
- Model layer: code comprehension, long-range reasoning, instruction following, and structured output stability;
- Execution framework layer: system prompts, tool definitions, context assembly strategy, sub-agent orchestration, and permissions and sandboxing;
- Engineering environment layer: repository testability, dependency completeness, CI feedback speed, and quality of rule documentation.
A common misjudgment is attributing a defect in the third layer to the first. For example, a task may fail because the repository lacks a reproducible build script; unable to verify anything, the model keeps guessing, and it ends up looking like “the model is bad.” So control the variables first, then talk about selection.
2. What Are the Five Parts of an Execution Framework
1. System Prompts and Behavioral Constraints
This determines the model’s default problem-solving strategy: whether to read code first or start editing, whether to ask or assume when facing ambiguity, and how to bound the scope of changes. Mature frameworks usually build in a “survey first — then locate — minimal change — self-verify” loop, and explicitly require the model to output a change list and a verification method.
2. Tool Contracts (Tool Schema)
How precisely tools are described directly determines the call success rate. Tools with vague parameter naming, missing enum constraints, or missing failure semantics significantly increase the number of wasted call rounds. An illustrative definition:
{"name": "run_tests","description": "Run unit tests under a given path and return a structured failure summary; does not modify any files.","parameters": {"type": "object","properties": {"path": { "type": "string", "description": "Relative path to a test file or directory" },"selector": { "type": "string", "description": "Optional filter expression for test case names" },"timeout": { "type": "integer", "default": 120, "minimum": 5 }},"required": ["path"],"additionalProperties": false}}
Key practices include: separating read-only tools from write tools, returning parseable structured errors on failure, providing default values for high-frequency parameters, and limiting the maximum context size returned per call.
3. Context Engineering
This is the most underestimated part of framework differences, covering retrieval/recall strategy, chunking and summarization of long files, compression of prior turns, and cache friendliness. If every request places large blocks of static content after dynamic content, prefix caching cannot hit, and both cost and time-to-first-token rise. The recommended order is: stable, unchanging rules and tool definitions first, repository summary in the middle, and the current-turn task plus dynamic observations last.
4. Sub-Agent Orchestration
Decoupling “exploratory tasks” from “execution tasks” is the main way to control context bloat. The exploration sub-agent searches, reads, and summarizes in an independent context, returning only conclusions to the main agent; the main agent retains decision and write authority. This both avoids polluting the main context with massive file contents and makes parallel search possible.
5. Permission Management and Security Boundaries
Coding agents can execute commands and write files, so permission design must be tightened by default:
- Tiered authorization: read-only exploration is auto-approved, while writes and network access require explicit confirmation;
- Path and command whitelists; reject unbounded recursive deletion, force push, and overwriting bulk rewrites;
- Sensitive information isolation: credentials are obtained via environment injection or a secrets management service, and must never enter the context or logs;
- Full auditability: each tool call records parameter summaries, duration, and return code.
3. Why “In-House Native Frameworks” and “General Mature Frameworks” Each Have Advantages
The advantage of general mature frameworks lies in the fact that their execution paradigm has been refined through many real tasks: complete toolchains, mature sub-agent mechanisms, detailed permission models, and reusable rule files and project conventions. For models with strong instruction following and strong long-range reasoning, these frameworks can more fully convert “capability” into “completion rate.”
The advantage of model vendors’ in-house native frameworks usually shows in three points. First, they are co-optimized with their own model in tool-calling format, thinking mode, and output templates, reducing retries caused by format mismatch. Second, the chain is shorter — one fewer relay in authentication, metering, and network paths — so latency and failure surface are smaller. Third, under a subscription model, cost is predictable, which suits high-frequency daily use.
The conclusion is not either/or, but layered use by task: exploratory refactoring, cross-file changes, and tasks requiring strict verification go to the side with a more mature framework; high-frequency small changes, completion-style development, and cost-sensitive daily iterations go to the side with a shorter chain.
4. How to Build a Reproducible Evaluation
Subjective impressions cannot support selection decisions. It is recommended to build a small internal benchmark:
# Illustrative evaluation skeleton: same task set, same environment, multiple samplesTASKS = load_tasks("repo_tasks.jsonl") # each entry has a repo snapshot, task description, acceptance scriptdef run_one(task, agent, k=3):results = []for _ in range(k): # multiple samples to eliminate randomnessenv = build_sandbox(task.snapshot)trace = agent.solve(task.prompt, env)passed = run_acceptance(env, task.verify_cmd)results.append({"passed": passed,"turns": trace.tool_calls,"tokens_in": trace.tokens_in,"tokens_out": trace.tokens_out,"human_takeover": trace.needed_human,})return aggregate(results)def aggregate(rs):n = len(rs)return {"pass_rate": sum(r["passed"] for r in rs) / n,"pass@k": any(r["passed"] for r in rs),"avg_turns": sum(r["turns"] for r in rs) / n,"avg_tokens": sum(r["tokens_in"] + r["tokens_out"] for r in rs) / n,"takeover": sum(r["human_takeover"] for r in rs) / n,}
Metrics should cover at least five items: task pass rate, pass@k, average interaction turns, token consumption per task, and human takeover rate. The first two measure capability, the middle two measure efficiency, and the last measures usability. The task set should include a number of real bug fixes, new feature additions, cross-module refactoring, and “intentionally ambiguous” requirement-clarification cases, and should fix the container image and acceptance scripts; otherwise data across rounds is not comparable.
Cost can be estimated with a unified metric, rather than comparing unit prices alone:
cost per task ≈ (input tokens × input unit price × (1 − cache hit rate))+ (output tokens × output unit price)+ environment startup and test execution cost+ human takeover hours × labor unit price
When cache hit rate and human takeover rate differ significantly, the option with a lower unit price is not necessarily the one with a lower total cost.
5. Selection Decision Framework
| Scenario characteristic | Preferred choice | Main reason |
|---|---|---|
| Cross-file refactoring, strict verification needed | The side with a more mature execution framework | Complete toolchain and verification loop |
| High-frequency small changes, daily completion | The side with a short chain and native adaptation | Low latency, predictable cost |
| Strong compliance, data cannot leave the intranet | Model services that support private deployment or dedicated network access | Controllable permissions and auditability |
| Long tasks, context prone to bloat | The side with mature sub-agent capability | Effective context isolation |
| Cost-sensitive, standardized tasks | Low-cost subscription or pay-as-you-go options | Budgetable unit cost |
6. Implementation Optimization Checklist
Rules file: maintain a machine-readable convention file in the repository root, clearly stating build, test, and formatting commands and prohibited actions, so the model can verify without probing.
# Project Conventions- Build: make build Test: make test-unit Single test: make test-one T=<case>- Must pass test-unit before commit; do not modify generated directories or dependency lock files- Do not push directly to the main branch; all changes must be small, revertible commits- If a requirement is ambiguous, first output questions to confirm; do not assume business rules on your own
Context control: truncate rather than fully replay tool returns; read specified line ranges of large files on demand; periodically summarize long conversations; put static prefixes first to hit the cache.
Verification first: enforce a “modify — run tests — read failure logs — modify again” loop, avoiding multiple rounds of guessing without verification.
Observability: persist each session’s tool-call sequence, token consumption, duration, and results to disk, and combine them with logging and monitoring/alerting services to build failure clustering, pinpointing specific tools or prompt segments. This kind of capability can usually be assembled from general cloud categories such as cloud-hosted model services, object storage (for storing repository snapshots and traces), logging services, monitoring and alerting, and secrets management — for example, general cloud products from cloud vendors like Baidu AI Cloud can cover such needs.
7. Conclusion
There is no single answer to “which is better,” only conditional answers. The model sets the ceiling of capability; the execution framework decides how much of that ceiling can be realized. The recommended order is: first build the task set and metrics → then run a two-framework comparison → decide based on total cost and human takeover rate rather than unit price → finally route in tiers by task type. When pass rates cannot improve, first check the acceptance scripts and repository testability; when costs spiral out of control, first check cache hit rate and context bloat; when risk events occur, first tighten permissions and audit, rather than switching models.
