How to Scientifically Evaluate the Coding Ability of Large Language Models

    "Which model writes better code?" is a debate that never seems to end among developers, yet most discussions stall at leaderboard scores or a single hands-on impression, overlooking two critical variables: task type and cost structure. This article argues that coding ability must be evaluated in layers. It explains how sparse attention and KV cache costs shape the economics of long contexts, contrasts long-horizon tasks with routine coding tasks through a hands-on example (autonomously planning and delivering a 3D browser racing game), and lays out a reproducible, tiered evaluation methodology with concrete metrics and cost formulas. It closes with practical guidance for choosing models per task tier instead of relying on any single benchmark.

    In developer communities, “how good is model X at writing code” is one of those questions that never goes away. But many discussions stop at leaderboard scores or a single-use impression, missing two key variables: task type and cost structure. “Writing code” can mean autocompleting a single function or autonomously carrying a full project from research to delivery, and the demands on the model are worlds apart. Starting from the evolution of the attention architecture, this article analyzes the capability gap between long-horizon and routine programming tasks, and offers evaluation and model-selection methods you can put to work directly.

    1. Key takeaways

    • For capability, prioritize long-horizon; for cost-efficiency, prioritize routine. In long-horizon tasks that require autonomous planning, multi-round execution, and self-correction, the gaps between models are clearly amplified. In routine tasks with established workflows, most mainstream models are good enough, and cost becomes the deciding factor in the experience.
    • Cheap per token does not mean lower total cost. How many tokens a task actually consumes is tied directly to the model’s planning efficiency and execution ability. When selecting a model, focus on “cost per successful task,” not the unit price.
    • A long context does not mean it is used well. Whether a long context translates into a capability advantage depends on the model’s ability to retrieve key details and keep advancing the task across a long conversation.
    • Evaluation must be layered. Unit tests, repository-level bug fixing, and long-horizon autonomous tasks should be assessed separately; no single leaderboard is enough to support a decision.

    2. The underlying view: how sparse attention reshapes the long-context cost curve

    2.1 Two costs of long context

    Large language models use the attention mechanism to model relationships between tokens, and both training and inference costs fall mainly into two areas.

    The first is compute. Standard attention computes the relationship between each token and every other token, so the cost grows quadratically with sequence length: double the sequence, and attention compute is roughly four times.

    The second is the GPU memory footprint of the KV cache. To avoid recomputation, inference caches the Key and Value vectors of historical tokens. Its size can be approximated as:

    1. KV Cache bytes ≈ 2 × layers × KV heads × head dim × sequence length × bytes per element

    Take 60 layers, 8 KV heads, head dim 128, and FP8 precision (1 byte per element) as an example: each token needs about 120 KB, and a single session with a million-token context alone consumes about 110 GB of GPU memory. During the token-by-token decode phase, the model typically reads the entire KV cache every time it emits a token, so memory bandwidth often becomes the bottleneck before compute does.

    For exactly this reason, the competitiveness of a long-context model largely depends on whether it can hold down both of these costs while preserving quality.

    2.2 The mainstream approach: low-rank compression + hybrid sparsity

    Industry efforts to optimize attention fall roughly into two categories.

    One is compressing the Key/Value representation. This approach maps the Key and Value at each layer into a lower-dimensional “latent space” to cache, then reconstructs the Key and Value for each attention head when in use — an approach analogous to MLA (Multi-head Latent Attention). Because what is cached is the compressed vector, both the memory footprint and the read bandwidth under long context drop in tandem.

    The other is sparsifying attention, so that not every token has to participate in exact computation. Represented by a new generation of “deep compression + sparse selection” methods, this combines the two, and mainly involves the following three mechanisms:

    • Light compression (compressed sparse attention): keeps relatively complete key-value information at a small compression ratio, suited to recent content rich in detail;
    • Heavy compression (high-compression attention): generates highly condensed key-value summaries at a high compression ratio, covering a very long history at low cost;
    • Indexer + block-level selection: a lightweight scoring module assesses the relevance of each historical block to the current query and selects only the few most relevant blocks for exact attention computation.

    In real models, light-compression layers and heavy-compression layers are usually interleaved: the former holds the floor of “seeing nearby clearly,” while the latter lowers the cost of “seeing far.” The pseudocode below illustrates this:

    1. def sparse_attention_forward(x, kv_cache, indexer, cfg):
    2. # kv_cache holds two kinds of entries:
    3. # compressed : lightly compressed K/V that keeps more detail
    4. # coarse : heavily compressed K/V summaries covering a long history
    5. scores = indexer.score(x, kv_cache.coarse) # lightweight scoring: which historical block is more relevant
    6. top_idx = topk(scores, k=cfg.num_selected) # keep only the most relevant blocks
    7. context = attention(x, kv_cache.compressed) # recent content: light compression
    8. context = context + attention(x, select(kv_cache.coarse, top_idx)) # historical content: block-level exact retrieval
    9. return context

    This design brings two benefits: the amount of cached data shrinks significantly under long context; and the content each token actually contributes to exact attention is bounded to a fixed size, avoiding the full quadratic cost. Compared with related earlier approaches, this kind of architecture typically compresses the per-token inference compute to about one-third of the original at the million-token context level, while KV cache usage drops to roughly one-tenth.

    2.3 The costs to acknowledge

    Compression and sparsification are not without costs:

    • Information may be lost: heavy compression inevitably discards some detail, so error messages, interface signatures, or requirements a user added earlier may be “forgotten” in later turns;
    • Recall depends on scoring quality: if only a few historical blocks are selected and the indexer misjudges, the model may overlook context that is actually critical;
    • Context ≠ effective context: no matter how long the window, if the key clue is not accurately recalled, the model will still take detours and may even repeat mistakes it already made.

    So measuring long-context ability cannot look at window size alone; it must also look at recall accuracy for information and stability over sustained work.

    3. Long-horizon task review: autonomously planning, researching, and delivering a 3D racing game

    To vividly show the difference between the two capability orientations, you can design a classic long-horizon programming task: have the model go online to research material, reference the gameplay of a classic racing game, and develop a 3D racing game that runs in the browser. The model is required to proceed through “autonomous planning → execution → multiple rounds of testing → delivery” and ultimately provide complete, runnable code.

    This kind of task is representative because it simultaneously tests five capabilities:

    1. Requirement decomposition: breaking a vague goal into subtasks such as scene building, vehicle control, item systems, and collision detection;
    2. External research: searching for material with clear questions and extracting usable information from the results;
    3. Tool use: calling terminal and file tools, continuously reading and writing code and running tests;
    4. Long-context recall: still accurately locating the modules it wrote and outstanding issues after many rounds of interaction;
    5. Self-verification: discovering problems through repeated test runs, rather than declaring done once the code is written.

    In actual runs, the two types of models showed a clear divergence.

    One type of model completed the task fairly completely, implementing gameplay elements such as track selection, character selection, items, ramps, and speed boosts; the game had basic playability, and across multiple rounds of iteration it kept fixing the problems it found.

    The other type implemented something relatively crude. During the research and testing phases, it tended to finish the task as quickly as possible, with fewer searches and perfunctory test runs; the final delivery was more of a feature demo and did not reflect the advantages a long context can bring.

    Note: single-task performance can be affected by prompts, temperature parameters, and sampling randomness, so conclusions should be built on multiple runs and multiple tasks.

    Analysis of the reasons behind the difference:

    • Planning and convergence: long-horizon tasks require the model to actively set checkpoints, know “what is still missing,” and return to the main thread before resources run out. Models that converge too early tend to wrap up hastily around a goal of “basically done.”
    • Effective use of long context: a long window only solves “fitting it in”; whether the model can accurately recover early design decisions and error messages after dozens of interaction turns is the real dividing line.
    • Intensity of self-verification: multiple rounds of testing are not a formality of repeated runs but require reading errors, locating causes, and forming fix plans. The depth of iteration directly determines the completeness of the final code.

    This is also consistent with the trend reflected by software-engineering evaluations aimed at real codebases (such as SWE-bench Pro): on complex, long-cycle programming tasks, models with strong planning and execution show a clear advantage.

    4. Routine tasks: small differences, cost-efficiency decides

    Beyond large tasks, developers also use many already-solidified skills (Skills) in daily work — for example, scraping information from multiple websites via a fixed process and organizing it into a structured report. This kind of task has clear inputs and fixed steps, and relies mainly on the model’s understanding of the tool-calling protocol and the accuracy of its parameter filling.

    In tool-calling evaluations (such as Terminal-Bench 2.0), the gap between different models is usually small, and they can all reliably meet basic needs. At this point, the side with lower unit cost is often more cost-efficient.

    But “simple task” does not equal “zero risk”:

    • Format stability: whether the output reliably meets the parsing requirements of downstream programs;
    • Error handling: when a target website changes or returns an anomaly, whether the model can recognize it and give reasonable feedback;
    • Long-tail scenarios: whether the model falls into futile retries when encountering anti-scraping, login restrictions, and other special pages.

    A common practice is to first run regression tests on a small batch of real samples, confirm stability, and then expand the scope of use.

    5. Hands-on: a reproducible tiered evaluation approach

    Rather than relying on other people’s leaderboards, it is better to build a small evaluation set around your own business. The following process is recommended.

    5.1 Task set design

    Tier Typical task Context/tool needs Core assessment point
    L1 unit function implementation, adding unit tests single file, local context code correctness, instruction following
    L2 repository fixing known bugs, cross-file refactoring medium-length context, code retrieval code understanding, locating and verification
    L3 long-horizon requirement decomposition, research, multi-round implementation and testing extra-long context, terminal/browser tools, multi-round planning planning, self-correction, long-context use

    The task set does not need to be large; the key is to stay close to real scenarios, with objective acceptance criteria that can be run repeatably.

    5.2 Metrics system

    Dimension Metric Description
    Correctness pass rate, first-pass rate whether the delivered result truly meets the requirement
    Efficiency average interaction rounds, average time whether there is a lot of futile back-and-forth
    Cost tokens consumed per task, cost per task normalized by “cost per successful task”
    Stability variance of results across multiple runs of the same task whether it can be reliably reproduced
    Maintainability human rating code readability, edge-case handling, extensibility

    Total cost can be estimated by the following formula:

    1. Total cost = Σ(input tokens × input price + output tokens × output price)
    2. Cost per successful task = Total cost / number of successful tasks

    Suppose task A costs 60% less per token than task B, but due to insufficient planning and repeated trial and error, it consumes 3 times the tokens of B to complete the same task. Then:

    • A’s cost for a single task run is about 1.2 times B’s, completely offsetting the per-token price advantage;
    • If A’s success rate is another 20 percentage points lower, the cost-per-successful-task gap widens further.

    This shows precisely that planning and execution ability are themselves a kind of “cost.”

    5.3 Execution environment

    • Isolated execution: run model-generated code and commands in a container or sandbox, restrict network and file-system permissions, and avoid affecting the host environment;
    • Context management: enable automatic context compression or rolling summaries; when testing long-context ability, separately track “cumulative tokens before compression” and “tokens actually billed”;
    • Parameter control: fix temperature, max rounds, and timeout, and repeat each task multiple times to reduce the interference of randomness.

    5.4 Automated evaluation script (illustrative code)

    1. import json, time
    2. TASKS = [
    3. {"id": "L1-01", "level": "L1", "prompt": "Implement the argument-parsing function in utils/parser.py", "verify": run_unit_tests},
    4. {"id": "L2-01", "level": "L2", "prompt": "Fix the bug described in issues/101 and keep tests passing", "verify": run_unit_tests},
    5. {"id": "L3-01", "level": "L3", "prompt": "Research and deliver a runnable 3D racing game", "verify": playthrough_check},
    6. ]
    7. def run_one(task, model, max_rounds=50):
    8. # Launch a coding agent in an isolated sandbox
    9. agent = CodingAgent(model=model, sandbox=Sandbox(image="python:3.11"), max_rounds=max_rounds)
    10. start = time.time()
    11. result = agent.run(task["prompt"]) # the agent autonomously plans and calls terminal, file, search, and other tools
    12. passed = task["verify"](result.workspace)
    13. return {
    14. "id": task["id"], "level": task["level"], "passed": passed,
    15. "rounds": result.rounds, "elapsed": round(time.time() - start, 1),
    16. "input_tokens": result.usage.input_tokens,
    17. "output_tokens": result.usage.output_tokens,
    18. }
    19. def evaluate(models, task_ids, repeat=3):
    20. report = {name: [] for name in models}
    21. for task in TASKS:
    22. if task["id"] not in task_ids:
    23. continue
    24. for name, client in models.items():
    25. for _ in range(repeat):
    26. row = run_one(task, client)
    27. row["cost"] = row["input_tokens"] * client.input_price + row["output_tokens"] * client.output_price
    28. report[name].append(row)
    29. for name, rows in report.items():
    30. passed = [r for r in rows if r["passed"]]
    31. print(json.dumps({
    32. "model": name,
    33. "pass_rate": round(len(passed) / len(rows), 3),
    34. "avg_rounds": round(sum(r["rounds"] for r in rows) / len(rows), 1),
    35. # Key metric: average spend per successfully completed task; the cost of failed tasks counts in the numerator too
    36. "cost_per_success": round(sum(r["cost"] for r in rows) / max(len(passed), 1), 4),
    37. }, ensure_ascii=False))

    After evaluation, compare each model’s pass rate, rounds, and cost per successful task by tier, and you can choose the right model for different scenarios.

    6. Selection strategy: configure by task tier

    Based on the analysis above, the following selection recommendations take shape:

    Task type Key capability requirements Selection recommendation
    Fixed-process routine tasks (skill-based, tool-calling-oriented) accurate tool calling, stable output format prioritize models with low unit cost and stable performance
    Cross-file bug fixing, local refactoring code understanding, context locating and verification balance capability and cost; watch repository-level evaluation performance
    Multi-round autonomous long-horizon tasks (research + planning + delivery) planning, self-correction, long-context use prioritize models with strong overall capability, then assess total spend

    In cost-sensitive scenarios, you can also introduce a model routing mechanism: a lightweight classifier or small model judges task complexity, routing simple tasks to a low-cost model and escalating complex tasks to a more capable model. Illustrative code:

    1. def route(request):
    2. # Select a model based on features such as task complexity, number of files involved, and expected rounds
    3. if request.estimated_files <= 1 and request.expected_rounds <= 3:
    4. return cheap_model # routine task: prioritize cost
    5. return strong_model # complex or long-horizon task: prioritize capability

    Whichever strategy you choose, regression-test with real business samples before going live, and continuously track success rate and cost per task, to avoid the hidden waste that comes from “looking only at unit price.” To move quickly, you can leverage large-model services and development tools from platforms such as Baidu AI Cloud to build a coding-focused evaluation and routing system.

    7. Conclusion

    Evaluating a coding model essentially means answering three questions:

    1. Architecture level: Can it support long context at low cost?
    2. Capability level: How does it perform on long-horizon tasks versus routine tasks?
    3. Cost level: How much does it cost in total to complete the same task?

    Combining hands-on experience with public evaluation results, a relatively clear judgment emerges: on long-horizon programming tasks that require autonomous planning and repeated iteration, models with strong long-context utilization and execution deliver a better experience; on fixed-process daily tasks, the gap between models is usually small, and the one with lower unit cost is more cost-efficient. For a development team, rather than asking “who is stronger,” it is better to evaluate by task tier and let the data tell you which model to use in which scenario.