LLM Performance Across Coding Agent Frameworks: A Practical Selection Guide
The same large language model can reliably complete cross-file refactoring in one agent framework yet lose context and loop on the same bug in another. This article explains why an agent scaffold — the harness wrapped around a model — shapes model behavior as much as the model itself. It breaks down the six layers that govern performance, compares mature general-purpose frameworks against vendor-native integrations, and lays out a reproducible evaluation methodology, a cost model that goes beyond unit price, and practical recommendations for least-privilege setup, observability, and layered routing — so teams can pick integration paths with data rather than preference.
1. The Core Issue: The Model Is the Engine, the Framework Is the Transmission
In practice, a common pattern shows up: the same model, wired into Setup A, will reliably complete cross-file refactoring; wired into Setup B, it keeps losing context and repeatedly patches the same error. This does not mean the model’s capability changed. It means the model sets the ceiling of what is possible, while the agent execution framework determines how much of that potential is realized.
An agent execution framework (often called a harness or agent scaffold) is the set of mechanisms wrapped around the model that translate natural-language instructions into executable engineering actions. It comprises at least six layers:
- System prompt layer: role definition, behavioral rules, output-format constraints, and refusal/clarification strategies.
- Tool definition and dispatch layer: which tools are provided, how their parameter schemas are described, how conflicts between tools are resolved, and how failure results are fed back.
- Context management layer: loading project-level instruction files, injecting retrieved snippets, and summarizing or forgetting long conversations.
- Sub-agent orchestration layer: splitting large tasks across multiple context-limited sub-agents that run in parallel, then aggregating their results.
- Permission and approval layer: which commands can run automatically, which require human confirmation, and whether write operations are bounded by a sandbox.
- Observability layer: session logs, token accounting, tool-call sequences, and failure attribution.
The quality of these six layers often produces a bigger difference than moving up a model generation.
2. Five Technical Paths Through Which Frameworks Shape Model Performance
Signal-to-noise ratio in instruction following. A longer system prompt does not necessarily mean better results. When rules conflict with one another or are too fine-grained, the model has to weigh many constraints at every generation step, its attention gets diluted, and it becomes more likely to violate the critical constraints. The value of a mature framework lies not in prompt length but in layered organization: global rules, language/stack rules, task-type rules, and current-file context, each scoped so they are not all dumped in at once.
Stability of tool-call shape. The quality of a model’s tool calls depends heavily on how clearly tools are described and how concise their return values are. If a tool returns a raw full stack trace, the model has to hunt for the key line inside thousands of tokens of noise. If it returns a structured summary (failing file, line number, assertion diff), the model’s fix-hit rate improves markedly. This is a pure engineering optimization, independent of how strong the model is.
The error-recovery loop. The main differentiator among strong models is whether they can self-correct after a failure. If the framework lacks a unified retry policy (retry limits, context rollback points, failure-cause classification), the model falls into unproductive loops that burn tokens and yield half-finished code. A reliable framework turns the “read file → modify → run tests → read failure → fix” loop into a controlled state machine rather than letting the model improvise freely.
Efficiency of long-context usage. A large context window does not mean effective usage. Information in the middle is easily overlooked, and repeated snippets crowd out the budget. Retrieval-based injection (loading only relevant snippets) usually beats dumping the whole repository, and combined with summary compression it can carry longer task chains within the same window.
Consistency of format and contracts. Patch format, code style, comment language, and commit-message conventions should all be enforced by the framework at the output layer rather than relying on the model to “remember” each time. Once these constraints are pushed forward into the tool layer, the variance in model behavior narrows considerably.
3. The Trade-offs of Two Integration Approaches
Mature general-purpose frameworks excel at ecosystem maturity: tool protocols refined across many real tasks, compression and retry strategies tuned repeatedly, support for switching between multiple models for comparison, and reusable project-level instruction files and sub-agent mechanisms. Their costs are equally clear: every request carries a longer system prompt and tool definitions, so input tokens are inherently inflated; sub-agent parallelism introduces a token multiplier; the middleware makes prefix-cache hit rates and billing calculations harder to predict; the prompt strategy is a black box whose upgrade cadence you do not control; and you must also assess whether routing code snippets through the middleware meets internal compliance requirements.
Vendor-native integration offers a short path, lower cost, and better alignment with the behavior patterns the model was trained on — thinking-mode toggling, tool-call format, and special-marker handling all need no secondary adaptation. For high-frequency tasks with relatively fixed patterns (adding tests, editing copy, adding simple endpoints, batch renaming), the native path’s per-unit cost advantage is pronounced, and enterprise needs such as permission systems, audit logs, and data staying within a defined boundary are also easier to satisfy. Its downside is that tool ecosystems and context strategies are usually less polished than the most mature general-purpose frameworks, and cross-model portability is weaker — once the model is updated, your prompts and workflows need to be re-adapted.
The conclusion is not “which is better” but task layering: route complex, cross-repository tasks that need multiple rounds of correction through mature frameworks, and route high-frequency tasks with clear boundaries and cost sensitivity through the native low-overhead path. Real engineering capability is the routing strategy, not picking a side.
4. Building a Quantifiable Evaluation Method
The key to avoiding guesswork in selection is to first produce a reproducible head-to-head evaluation at low cost. The evaluation set should come from your own real repositories, not public benchmarks.
Task layering design (30–60 items recommended):
- L1: Single-file bug fix (with a clear failing test case)
- L2: Small cross-file refactor (touching 2–5 files and updating callers in sync)
- L3: Writing unit tests from scratch (raising coverage)
- L4: Dependency and configuration troubleshooting (reading logs, locating the root cause)
- L5: Comprehension Q&A on legacy code (no write operations)
Core metrics: task pass rate (whether all automated tests pass), input/output tokens per task, wall-clock time per task, average number of retry rounds, manual-intervention rate (share of tasks a developer had to fix by hand), and per-task cost.
Controlled variables: the same model and parameters, the same tool set, the same token-budget cap, the same repository snapshot, and the same test command. Experiments without a budget cap get skewed by a few runaway tasks, so it is best to report both the median and the P90.
Illustrative evaluation-driven code (pseudocode; replace the interface names with those of the model service you actually use):
def evaluate(tasks, client, max_budget_tokens=800_000):report = []for task in tasks:repo = checkout(task.repo_snapshot) # an isolated working copy per taskusage = {"input": 0, "output": 0, "steps": 0}try:with budget_limit(max_budget_tokens):for _ in range(MAX_STEPS): # e.g., 25 stepsaction = client.step(system=PROMPT[task.layer],tools=TOOLS, # read / search / patch / run_testscontext=repo.instruction_file() + retrieve(task.query),)usage["input"] += action.input_tokensusage["output"] += action.output_tokensusage["steps"] += 1if action.type == "finish":breakrepo.apply(action) # constrained by permission policyok = run_tests(repo) == 0 # the only trustworthy pass criterionexcept BudgetExceeded:ok, usage["steps"] = False, MAX_STEPSreport.append({**task.id, "passed": ok, "cost": price(usage),"steps": usage["steps"], **usage})return summarize(report) # pass rate / token median / P90
Running the same script against both integration approaches gives you pass rate, token median, and manual-intervention rate — the objective basis for selection.
5. Cost Accounting: Don’t Compare Only Unit Price
A unit price that is three times cheaper can still produce a higher total cost. The reason is that effective cost = unit price × actual tokens consumed, and actual consumption is significantly amplified by the framework’s structure. The table below is illustrative (the values are for explanation only and do not represent any real product):
| Cost item | Mature general-purpose framework | Native lightweight integration |
|---|---|---|
| Fixed overhead per request (system prompt + tool descriptions) | High | Low |
| Accumulated input over long multi-round tasks | High (history must be resent) | Medium |
| Token multiplier from parallel sub-agents | Present | Usually none |
| Discount from prefix-cache hits | Depends on the middleware implementation | Usually more stable |
| Billing basis | May layer additional middleware rules | Usually the model’s own basis |
| Hidden waste from failed retries | High (complex strategy) | Medium |
One often-overlooked detail: the system prompt and tool definitions are sent again on almost every request, so they are a fixed cost multiplied by the number of requests. When a task involves many tool-call rounds, this overhead can exceed the actual value of the model’s output. Correspondingly, a “subscription plan” model suits interactive, exploratory development where usage is hard to predict, while “pay-as-you-go” suits batched, predictable pipeline tasks. When selecting, match the billing model to the task shape rather than comparing unit price alone.
6. Implementation Recommendations
1) Use a project-level instruction file to carry deterministic constraints. Write only what the model cannot infer from the code: build and test commands, directory responsibilities, paths that must not be touched, code-style essentials, and commit conventions. Keep it concise (generally within one screen), and put it under version control so rule changes can be reviewed and rolled back.
# Project collaboration conventions- Test command: `run_tests --fast` (must be all green before committing)- Do not modify: `generated/`, `migrations/legacy/`- New public interfaces must have unit tests and updated interface docs- Use English for comments and docs; use Chinese for comments
2) Least privilege and sandboxing. Default to read-only, and limit write operations to an isolated branch or a temporary working copy; run commands through an allowlist; require explicit approval for network access and package installation. Any automated output should be revertible with one click through version control — this is the most important security baseline when using AI coding tools.
3) Put observability in place first. At a minimum, record token usage per session, the tool-call sequence, failure-cause classification, and the proportion of human-edited diffs. This data serves both selection and ongoing optimization. On the enterprise side, you can rely on a unified cloud model-service integration, together with logging and monitoring/alerting, to attribute costs by team and by project. General capabilities such as those offered by Baidu AI Cloud — model services, logging and monitoring, object storage, and container platforms — are usually sufficient to support this kind of metering and governance.
4) Layer routing instead of a global switch.
Task entry├── clear, well-bounded, high-frequency → native lightweight path (low overhead, low latency)└── cross-file, needs multiple corrections → mature framework path (strong compression and retry strategy)└── per-task tokens exceed threshold → escalate to a human and record as a regression case
7. Four Common Misconceptions
- “A stronger model needs the framework less.” The opposite is true: more capable models depend more heavily on high-quality tool feedback, and a crude framework wastes their advantage in unproductive loops.
- “Longer context is always better.” Irrelevant content dilutes attention and raises cost; precise retrieval-based injection usually beats full-volume injection.
- “Select based on public benchmark scores.” Public scores reflect a model’s performance on general tasks and are strongly tied to your repository structure, toolchain, and coding conventions — you must test on your own.
- “Put all of the team’s conventions into the instruction file.” Rule overload causes key constraints to be ignored; distinguish “must enforce” from “can infer.”
8. A Decision Checklist
Answer six questions in order when selecting: Is the work mostly cross-repository refactoring or local edits? What is the average number of tool rounds per task? Must code stay within a defined boundary? Are audit and permission approval required? Must the ability to switch between multiple models be preserved? How predictable is monthly token consumption?
A “yes” leaning on the first three questions points toward a mature general-purpose framework; a “yes” leaning on the last three points toward native integration. For most teams, the rational answer is both coexisting, measured continuously with one unified evaluation script — let the data, not preference, decide the routing weights. That is the sustainable engineering practice.
