How Coding Agents Evolve and How to Evaluate Them in Engineering Practice
This article explains why teams should focus LLM investment on coding agents, how to turn subjective “good experience” into measurable capabilities, how to build repo-realistic evals, and how to ship agents with permissions, replayable traces, human-in-the-loop gates, and quality-driven routing between reasoning and non-reasoning paths.
Why capability work should converge instead of spreading thin
Early LLM programs often push chat, retrieval, multimodal, domain fine-tuning, and workflow automation at once, hoping one product line covers every scenario. Coverage looks complete in the short term, but three problems usually appear: eval sets are not comparable, optimization goals pull against each other, and engineering cannot tell which path a new model version should replace.
A more durable path is to converge on a general foundation first, then spend incremental effort on high-leverage work. For R&D organizations, coding and agents are typically that leverage: inputs and outputs can be versioned, correctness can be partly verified with tests and static checks, and failures can be localized in code review. Focus does not mean abandoning general ability. It means putting general understanding, long context, and instruction following in service of closed-loop tasks rather than demo-grade Q&A.
Mature iteration is usually four stacked stages, not a single jump:
- General generation is usable, but tool use is unstable and complex edits drift off constraints.
- The agent role is explicit: it plans, calls tools, and corrects the next step from observations.
- A relative advantage appears on programming work: cross-file edits, test completion, and refactor suggestions.
- Non-reasoning quality approaches what the previous generation needed heavy reasoning for, which changes latency and cost.
If the prior stage only patches edges and samples, the next stage is structural: under the same business prompt, a lightweight path can finish work that once required a long chain of thought. That affects gateway routing, quotas, and how humans collaborate with the system.
Turn “does it feel good?” into testable capabilities
Subjective experience is easily inflated by prompts, example selection, and UI. Engineering evaluation should split experience into independent dimensions so a single conversation is not treated as a conclusion.
Instruction following and constraint holding. Language version, directory boundaries, forbidden modules, and output format must be respected. Breaking a hard constraint is worse than “not elegant,” because it can break the pipeline.
Code correctness. Existing unit tests, integration tests, and type checks should pass. For bug fixes, failing cases should go green without expanding the change surface. Correctness is about regression, not whether a new snippet merely runs.
Repo-level context use. The model should read relevant call chains, config, and test fixtures rather than locally optimizing a single file. A large context window does not mean attention is allocated well. Watch whether it cites real symbols or invents interfaces that do not exist.
Closed-loop tool use. Agent value is “act–observe–correct.” Record whether search, read, edit, and test execution are necessary, and whether hypotheses shrink after failure logs appear instead of blind retries.
Reasoning depth and path choice. Hard algorithms, concurrency, and security boundaries may still need fuller intermediate reasoning. Routine CRUD, boilerplate refactors, and test fill-in should prefer non-reasoning or short-reasoning paths. Score “is quality good enough” separately from “must we use heavy reasoning.”
Cost, latency, and stability. Above a quality bar, compare task time, retry count, and human takeover rate. If a non-reasoning mode already matches the previous generation’s reasoning pass rate, reserve heavy reasoning for failures and architecture-level changes instead of turning it on by default.
None of these dimensions depends on a vendor-specific metric. Track them on an internal question bank over time.
Build a question bank close to real work
Items should come from historical merge requests, defect tickets, and standards in your own repo, not only public algorithm problems. Stratify by difficulty and risk.
- Low risk: comments, renaming, table-driven tests, extracting duplicated branches into functions.
- Medium risk: new endpoints against an interface contract, config parsing changes, error handling and logging.
- High risk: concurrency fixes, authorization, data migration, cross-module refactors.
Each item should fix five fields: task statement, allowed paths, forbidden external behavior, acceptance commands, and scoring rules. Acceptance commands should be tests or checks that already exist in the repo. Avoid inventing scripts that only work in a demo environment.
Score with “gates plus bonuses.” Gates include: tests pass, diff stays in scope, no secrets or internal addresses leak, and the output can be applied as a patch. Bonuses include fewer changed lines, naming consistent with the project, tests for failure paths, and an explanation of trade-offs without stuffing unrelated refactors into the same patch.
A platform-agnostic task skeleton (eval fields only, not a product API):
task_id: repo-change-017goal: Add empty-result and missing-parameter error handling for order status queriesscope:allow:- src/order/query.*- tests/order/query_*deny:- deploy/**- secrets/**acceptance:- run: unit-test --filter order_query- run: lint --path src/orderscore:gate: tests_pass AND diff_within_scopebonus: covers_empty_result AND covers_missing_param
Rotate the bank regularly. If models or prompts are optimized only against the same public samples, pass rates inflate. Fold recent production defects (after redaction) into the next round so the distribution stays real.
Control plane when agents land
As capability grows, risk shifts from “wrong answer” to “wrong action.” Once a coding agent can edit a repo and run commands, you need a control plane, not only “please be careful” in the prompt.
Least privilege. The runtime is read-only on dependencies and source by default; write access is limited to the task branch. Network, secrets, and deploy credentials are unavailable by default. If external data is needed, people provide redacted fixtures.
Replayable traces. Store a plan summary, files read, patches applied, and test output. When something goes wrong, you should be able to say which failure drove which line changes, not only keep the final diff.
Graded human-in-the-loop. Low-risk tasks may open a branch and wait for review. Medium risk must show a plan before critical edits. High risk produces a design and patch draft only, with no direct execution. Put the rules in repo policy, not in a chat log.
Stop on failure. Repeating the same command, diffs out of scope, or attempts to read sensitive paths should halt and return to a human. Retry budgets need a cap, or the agent will spend time producing a patch that looks green while bypassing tests.
Hook into existing engineering. Patches go through the same review, static checks, and CI as human commits. The model does not hold merge rights. Review comments should become the next round of constraints, not verbal confirmation in chat.
Routing reasoning vs non-reasoning
When non-reasoning is already close to the previous generation’s reasoning on routine coding, route on a quality bar instead of always turning on the strongest mode.
A practical pattern is two-stage. First, a lightweight path does retrieval, localization, and a patch draft, then immediately runs gate checks. If it passes, go to review. If it fails for algorithmic complexity, missed state machines, or unclear security boundaries, upgrade to fuller reasoning and attach failure logs and attempted patches so the model does not start from scratch.
Make upgrade conditions explicit: the same test failing repeatedly, changes involving concurrency or permissions, or the model stating it lacks information. Do not trigger heavy reasoning on a vague “this feels hard,” or cost returns to full reasoning and focus loses its efficiency.
Watch the other direction too: a lightweight path passing tests does not mean the design is sound. Keep architecture and security on the review checklist so “tests green, abstraction wrong” is not auto-merged.
How to write prompts, context, and conventions
Stronger models need short, hard rules, not longer role-play. Keep a coding-convention digest in the repo with clauses the model tends to violate: error-handling style, log fields, test naming, forbidden dependency directions, and public-interface compatibility. Full standards are for people; the digest should fit in the task context.
Write task prompts in four parts: goal, constraints, acceptance, known clues. Clues can name entry files and failing tests. Do not state a guessed root cause as fact, or the model may skip investigation.
Inject context structure-first, details later: directory and related symbols, then file fragments as needed. Dumping the whole repo dilutes hard constraints. Retrieval should return verifiable paths and symbols, not unattributed blobs.
An illustrative prompt:
Goal: Fix unhandled exceptions when a query is missing filter parameters.Constraints: Change only the query module and its tests; keep public response fields compatible; add no external dependencies.Acceptance: Related unit tests and static checks pass.Clues: Failure occurs before parameter validation; entry is the query module’s main function.
How teams decide whether a generation is worth replacing
Replace on trends and gates, not one demo. Over at least two consecutive cycles, compare production vs candidate on the same bank: gate pass rate, share of patch lines humans still edit, average time, share upgraded to heavy reasoning, and distribution of review rejections.
If pass rate is flat but heavy-reasoning share drops, and rejections shift from “does not run” to style or design discussion, capability has crossed a step and the lightweight path can become default. If pass-rate gains come mainly from bank leakage or looser constraints, do not replace.
Ship with shadow runs or small-traffic tasks: agent submissions land only on draft branches, compared with human solutions for one to two weeks, then expand low-risk categories. Write rollback conditions in advance: out-of-scope edits, sensitive reads, or takeover rate bouncing back.
Common mistakes
Pushing every capability to the maximum at once blurs evaluation and prevents stable prompts and tool protocols. Comparing only final answers, not tool traces, misses high-risk actions. Measuring only reasoning ceilings, not whether non-reasoning is already enough, locks in cost. Pushing model output straight to main bypasses the existing quality system.
A more reliable loop: the general foundation handles understanding and generation; programming tasks provide a verifiable closed loop; the agent multi-step-corrects inside permission boundaries; reasoning depth upgrades on failure evidence. Iterate that way, and better experience becomes a regressable engineering capability instead of an unreproducible chat impression.
