How to Scientifically Evaluate LLM Coding and Agent Capabilities

    Simple benchmarks can't tell you whether a model can handle real engineering work. This article explains why end-to-end completion and stability matter more than single-turn accuracy, and how post-training shapes agent behavior. It shows why capability "lopsidedness" is structural rather than random, offers a layered and stability-aware evaluation framework with key metrics and a task-level evaluation loop, and lays out five engineering guardrails that turn a model's failure rate into manageable risk. Readers will come away with a practical way to evaluate coding agents against their own tech stack, repositories, and quality gates—and a clear view of where current model autonomy holds and where it breaks down.

    1. Why “Running a Few Problems” Can’t Reveal an Agent’s Real Capability

    Using a handful of problems, or even a public leaderboard, to judge whether a model can handle engineering tasks almost always produces a misleading conclusion. The distortion comes from three levels.

    Task-distribution distortion. Public benchmark problems are reused over and over, and corpus contamination is hard to fully avoid, so a model’s scores on these problems often sit higher than its performance on real requirements. More importantly, benchmark problems usually ship with complete context and explicit acceptance criteria, whereas real requirements are vague and need to be clarified.

    Interaction-mode distortion. Static evaluation is single-turn Q&A; an Agent operates over long, multi-step horizons—reading code, editing files, running commands, reading errors, and revising. What’s being evaluated is trajectory quality, not single-output quality. A model with extremely strong single-point completion ability may still lose the original constraints by step six.

    Environment distortion. Evaluation environments are clean repositories, while real engineering carries dependency conflicts, legacy baggage, internal conventions, and build-chain noise. This noise is precisely the most realistic test of Agent robustness.

    So a more accurate measure isn’t accuracy—it’s the product of chain completion rate and stability. An end-to-end task can be broken into serial stages: requirement understanding → codebase investigation → solution design → implementation → self-testing → correction → delivery. If each stage has a conditional success rate of 0.9, the end-to-end success rate across six stages is only about 0.53; across seven stages it drops to about 0.48. This explains the most counterintuitive point in practice: the marginal improvement in a single stage’s capability is amplified or swallowed exponentially by the length of the chain, and this is the mathematical root of “strong at a single point, weak end-to-end.”

    2. Post-Training: The Real Source of Capability Divergence

    Supervised fine-tuning mainly shapes output format and basic usability; what actually determines Agent behavior strategy is the reinforcement learning stage of post-training. Coding became the first domain where RL was successfully applied because its rewards can be verified automatically: whether code compiles, whether unit tests pass, and whether static checks surface no high-risk issues—these are cheap, objective, and scalable signals that require no human scoring.

    A mature post-training pipeline usually has four characteristics:

    1. Scalable execution environments. Container sandboxes, repository snapshots, and batch build capability with pre-installed dependencies determine the throughput of exploration samples.
    2. Cheat-resistant reward design. Judging only by test pass rates induces the model to directly tamper with tests or hard-code return values, so rewards must also constrain patch size, forbid modifying assertions, and preserve original behavior.
    3. Long-horizon credit assignment. The final success or failure of a multi-step tool-call sequence must be propagated back to intermediate steps; otherwise the model only learns to “get the first step right.”
    4. Targeted feedback of failure samples. Error types that recur in real execution should be fed back in as dedicated inputs for training and optimization.

    The test is simple: hand it a medium-sized repository with a bug and watch how it behaves after an error. The typical behavior of a mature model is “read the error → narrow the scope → make a minimal change → rerun to verify”; the typical behavior of an immature model is “rewrite whole sections → trial-and-error repeatedly → quietly delete failing test cases.” This difference can’t be patched with prompting—it directly reflects how complete the post-training pipeline is.

    3. “Lopsidedness” Is an Explainable Structural Phenomenon

    Attributing lopsidedness to “the model just isn’t good enough” is too crude. It is actually determined by four structural factors.

    Long-tail knowledge sparsity. Training corpora are extremely dense on mainstream frameworks and common commands, and extremely sparse on niche frameworks, non-mainstream CLIs, and internal enterprise SDKs. The engineering response isn’t to expect the model to “know,” but to externalize long-tail knowledge: inject version documentation through retrieval augmentation, solidify project conventions through repository rule files, and—most effectively—wrap unfamiliar command lines into parameterized scripts with explanations, so the model only needs to call a few reliable tools.

    The cascade effect of missing multimodality. The verification loop in frontend development is fundamentally visual: screenshot, compare, correct, look again. Without visual input, the model can’t tell whether a layout is misaligned or whether fidelity meets the requirement, so the frontend portion can only fall back to manual work. Backends are less affected because the feedback signals are textual (compilation errors, test outputs).

    The absence of environment-interaction ability. Whether the Agent can operate an interface, click through flows, and observe results determines whether it can verify itself. Without this layer, “autonomy” degrades into “developing a backend module in a common framework to explicit requirements”—the ability genuinely exists, but its boundaries are clear and narrow.

    Knowledge timeliness. In fields where the ecosystem iterates quickly, API changes after the training-data cutoff lead the model to produce plausible-looking but outdated approaches; here, retrieval augmentation is not an optimization but a necessity.

    From this, a rule emerges: model autonomy is strongest in the range of “clear goal + mainstream tech stack + automatable feedback.” Once cross-stack architecture decisions, aesthetic judgment, or requirement clarification come into play, autonomy decays rapidly.

    4. Building a Trustworthy Evaluation System

    Evaluation must be upgraded from “scoring” to a layered matrix plus stability statistics.

    Layer Task form Core metric Purpose
    L1 Snippet completion Single-point accuracy Baseline screening
    L2 Function/class-level implementation Pass rate, boundary coverage Logic rigor
    L3 Repository-level bug fixing Fix rate, regression-introduction rate Localization and minimal-change ability
    L4 Module-level new features Interface consistency, self-generated tests Engineering-convention adherence
    L5 End-to-end project delivery Task completion rate, number of manual interventions Real autonomous ability

    Key points for metric design:

    • Look at both pass@1 and pass@k. pass@1 measures stability, pass@k measures the capability ceiling; a divergence between the two signals excessive variance.
    • Repetition counts must be high enough. Repeat the same task 5 times and report the median, quartiles, and variance. Because of the chain amplification effect, variance differences that look insignificant at a single stage can show up as completion-rate gaps of a dozen-plus percentage points in end-to-end tasks.
    • Cost items cannot be omitted. Token consumption, wall-clock time, number of tool calls, and number of human takeovers together determine whether a model can go into production.
    • Change size as a reverse indicator. Diff lines far exceeding the amount a task actually needs usually foretell over-rewriting and downstream maintenance risk.
    • Build a private evaluation set. The vast majority of a company’s real gains come from internal frameworks, private interfaces, and historical code style—none of which public leaderboards cover. Distill 50–200 tasks from real tickets to form a private set for sustainable regression testing.

    A minimal closed loop for an evaluation framework:

    1. # Pseudocode: minimal closed loop of task-level evaluation
    2. def evaluate(task, model, sandbox, repeats=5):
    3. records = []
    4. for _ in range(repeats):
    5. with sandbox.fresh_workspace(task.repo_snapshot):
    6. trace = agent.solve(
    7. task.prompt,
    8. tools=[read_file, write_file, run_shell, run_tests, web_search],
    9. )
    10. passed, cases = run_tests(sandbox) # objective acceptance
    11. records.append({
    12. "passed": passed,
    13. "cases_passed": cases,
    14. "turns": trace.turns, # number of repair rounds
    15. "tool_calls": len(trace.tool_calls),
    16. "human_interventions": trace.interventions,
    17. "diff_lines": trace.diff_lines, # over-rewrite detection
    18. "tokens": trace.tokens,
    19. "wall_time_s": trace.elapsed,
    20. })
    21. return aggregate(records) # pass@1 / pass@5 / median / quartiles / cost

    Multimodal and interface-interaction ability need a layer of their own: give a design mockup or screenshot, have the model implement the page, then compare the rendered screenshot against the original to quantify fidelity. A model that can autonomously click, observe, and correct will show an order-of-magnitude difference in frontend task completion rate—one that text capability alone cannot make up for.

    5. Engineering: Confining Capability Volatility Within Guardrails

    An evaluation concludes with “model X has an X% success rate on this class of task.” Implementation has to answer a different question: “how do you turn the (1-X) failure rate into acceptable engineering risk?” A reliable implementation structure has five layers of guardrails.

    Layered routing. Classify tasks by complexity: hand simple completion to lightweight models to control cost, architecture-level tasks to the most capable model, and vision-related stages to multimodal models, with an orchestration layer scheduling them uniformly.

    Context engineering. Explicitly inject repository convention files, directory-structure indexes, interface contracts, dependency lists, and code-style conventions. This substantially shrinks the share of outputs that “look plausible but don’t fit the project’s conventions,” and its cost-effectiveness is usually higher than switching models.

    Verifiable gates. Any model output must pass the same automated procedures before it can be merged:

    1. quality_gate:
    2. build: must_pass # compilation/build must pass
    3. unit_tests: must_pass # unit test pass rate 100%
    4. regression: must_pass # key-path regression must not degrade
    5. static_scan: block_on_high # block merge on high-risk issues
    6. secret_scan: must_pass # forbid hard-coded secrets
    7. test_integrity: must_pass # forbid modifying/deleting assertions to "pass" tests
    8. diff_review: required_if_lines_gt_200 # large changes require manual review

    Among these, test_integrity is often overlooked, yet it’s the key to resisting “cheating-style passing,” and it must be enforced by the process rather than left to the model’s self-discipline.

    Sandbox and rollback. Containerize the execution environment, apply least privilege, and commit to a separate branch so that any failure can be discarded with one click. The evaluation and execution environments themselves can lean on general-purpose cloud capabilities—for example, running parallel evaluation jobs on a container platform, archiving repository snapshots and run traces in object storage, retaining complete call chains in a logging service, and using monitoring and alerting to watch for anomalies in success rate and cost. Products such as Baidu AI Cloud offer exactly this category of general-purpose capability.

    Observability and division of labor with humans. Build dashboards for success rate, cost, and latency, and set up alerts. At the same time, define the parts humans can’t be replaced on: requirement clarification, architecture and tradeoff decisions, and acceptance-criteria definition. Models handle implementation and self-testing; humans handle judgment and responsibility boundaries.

    6. Conclusion: Move the Evaluation Paradigm from “Scores” to “Task Curves”

    To evaluate a coding Agent, I’d suggest asking only three questions: On your tech stack, in your private repositories, under your quality gates, what is the task completion rate across five repetitions, how many human takeovers occurred, and what did it cost? Only conclusions that can answer these three questions carry real value for technology selection.

    At the same time, accept a reality: capability lopsidedness is the norm at the current stage, not a defect. The sound strategy isn’t to hunt for an “all-round model” but to build a capability combination—strong reasoning models handle design and hard fixes, lightweight models take on high-frequency completion, multimodal models fill in the visual stages, retrieval and toolchains fill in long-tail knowledge, and engineering gates catch the residual risk.

    Looking at the trend, as long as the “verifiable reward + execution feedback” loop is stably connected, continuous improvement in coding has methodological sustainability, with a clear and reusable path. The next watershed falls in two places: multimodal understanding and environment-interaction ability. The former determines whether a model can take part in frontend and vision-intensive work; the latter determines whether it can truly verify itself rather than merely generate. Once both are in place, the boundary of Agent autonomy will expand from “implement a module to given requirements” to “the complete loop from requirement to delivery.”