Evaluating LLM Coding Capabilities: Architecture, Long-Range Tasks, and Cost

    Evaluating the coding capabilities of Large Language Models (LLMs) requires a shift from simple chat metrics to a comprehensive understanding of agent workflows. This article breaks down the evaluation process into three critical dimensions: attention architecture, long-range programming tasks, and cost efficiency. It introduces the concept of effective cost—balancing token price with task success rates—and explores how different attention mechanisms affect inference costs. Furthermore, it provides practical frameworks for testing long-range convergence and routine task stability, alongside engineering best practices for routing, context management, and observability to help teams build cost-effective coding agents.

    Evaluating LLM Coding Capabilities: Architecture, Long-Range Tasks, and Cost

    1. The Real Shape of the Problem: Stop Evaluating Coding Agents by Single-Token Pricing

    The cost structure of a chat scenario is straightforward—one request, one output. However, a coding agent operates entirely differently as a multi-turn loop: read code → retrieve repository → modify file → run compile/test → review errors → modify again. The bill is not determined by the unit price, but by the “total tokens consumed to complete a task × unit price.”

    Therefore, cost must be broken down into three measurable items:

    • Single call cost = input tokens × input unit price + output tokens × output unit price
    • Task total cost = sum of single call costs across all rounds for that task
    • Effective cost = task total cost ÷ number of verified, passed tasks

    The third metric is what enterprises truly care about. A model with a lower unit price but a low success rate often requires more rounds of trial and error or ultimately fails, resulting in a potentially higher effective cost. Conversely, a more expensive model that passes on the first try might be more cost-effective.

    1. def cost_per_success(requests, price_in, price_out):
    2. tokens_in = sum(r["prompt_tokens"] for r in requests)
    3. tokens_out = sum(r["completion_tokens"] for r in requests)
    4. raw_cost = tokens_in * price_in + tokens_out * price_out
    5. tasks = {r["task_id"] for r in requests}
    6. passed = len({r["task_id"] for r in requests if r["verified"]})
    7. return {
    8. "tokens_in": tokens_in,
    9. "tokens_out": tokens_out,
    10. "raw_cost": raw_cost,
    11. "success_rate": passed / max(1, len(tasks)),
    12. # If no tasks pass, the effective cost is infinity, directly exposing capability gaps
    13. "cost_per_success": raw_cost / passed if passed else float("inf"),
    14. }

    Key prerequisite: All comparisons must run on the same set of tasks and the same validation scripts, otherwise the numbers are not comparable.

    2. Attention Architecture: Where Cost Compression Happens

    During the inference phase, memory and compute overhead primarily come from the KV cache and attention computation. When context scales from a few thousand to millions of tokens, this overhead quickly becomes the dominant cost driver for long-context services. Recent optimization approaches can generally be categorized into four types:

    1. Latent Vector Compression (e.g., MLA): Projects K/V into a low-dimensional latent space, caching only the latent vectors and restoring them during decoding, significantly shrinking the KV cache footprint.
    2. Indexer-Driven Sparse Selection (e.g., DSA): Maintains an additional lightweight indexer to select only the most relevant few tokens for each query, reducing O(N) attention to approximately O(k).
    3. Compressed Sparse Attention (e.g., CSA): Applies lossy compression to distant history while retaining details for recent context, balancing long-range recall with local precision.
    4. Heavy Compression Attention (e.g., HCA): Uses more aggressive compression ratios, often interleaved with sparse layers at a fixed ratio—a few heavy compression layers carry global information, while the rest focus on local relationships.

    The motivation behind interleaved layering is worth emphasizing: any single strategy has its shortcomings. Pure sparsity might miss critical distant evidence, while pure compression might lose implementation details. Dividing labor among different layers can further reduce per-token FLOPs and KV cache usage at the same accuracy level. Public data often suggests that under million-level context, per-token inference FLOPs drop to a small fraction of the baseline and KV cache reduces to a tenth, but specific numbers vary greatly depending on implementation, sequence length, and task distribution. It is recommended to rely on your own testing rather than directly citing vendor claims.

    More importantly: these optimizations only affect unit inference cost. The actual experience of a coding agent is determined by three things together: unit cost, task success rate, and the amount of tokens required to achieve success. Connecting the first two sections, architecture dictates the “unit price,” while the model’s planning and self-testing capabilities dictate the “usage.”

    3. Long-Range Programming Tasks: The Focus is “Convergence”, Not Just “Writing Code”

    A task design that truly differentiates models should include a complete closed loop: autonomous planning → research and investigation → multi-turn self-testing → final delivery. Take a cross-stack task like “replicating a cartoon-style 3D racing game” (not referring to any specific title) as an example. The ideal execution chain should be:

    • Requirements breakdown: Output a module list first (rendering pipeline, vehicle physics, track generation, item system, UI/HUD, input mapping) before starting to code;
    • Research sufficiency: Whether retrieval has a clear purpose, whether written conclusions are formed, and whether key mechanisms of similar implementations are covered;
    • Self-test loop: Whether it actively runs, checks console errors, takes screenshots or records, and iterates accordingly, rather than submitting immediately after writing;
    • Delivery quality: Code is runnable, directory structure is clear, dependencies and run instructions are complete;
    • Context utilization: Whether it genuinely performs repo-level retrieval and cross-file reasoning, rather than only looking at the first few files.

    Common failure modes are also diagnostically valuable: drawing conclusions after just one round of research; skipping self-tests and submitting directly; implementation stalling at the demo level lacking item and track differentiation; repeatedly making minor fixes without ever converging, causing both rounds and tokens to spiral out of control.

    Scoring should adopt a tiered rubric with clear weights, for example: Requirement coverage 30%, Functionality completeness 25%, Self-test and iteration quality 20%, Code maintainability 15%, Delivery specifications 10%. “Functionality completeness” is judged by automated scripts, while “Self-test and iteration quality” relies on process logs. In publicly available repo-level bug-fixing benchmarks and terminal automation benchmarks, different models winning or losing is the norm—this also proves that there is no universal best, only the most suitable configuration for a given task distribution.

    4. Routine Tasks: Tool Calling Stability is the Core Metric

    For routine tasks that have solidified into Skills or standard workflows (information gathering, batch rewriting, log inspection, report compilation), evaluation dimensions should shift to:

    • Instruction following: Whether output strictly adheres to the given format and fields;
    • Parameter correctness: Whether required/optional parameters for tool calls are accurate and whether there are redundant calls;
    • Error recovery: Whether it can retry or gracefully degrade when API calls fail, time out, or return empty results, rather than silently returning;
    • Idempotency and side effects: Whether repeated execution produces dirty data, and whether write operations are rollback-able;
    • Latency and cost: P50/P95 latency, average token consumption per task.

    These tasks have short context and fixed steps, making it hard for long-context architectural advantages to shine. Therefore, in routine scenarios, configurations with low latency, controllable unit costs, and stable calling formats are often more cost-effective; whereas tasks requiring heavy planning, cross-document research, and long-range validation are better suited for configurations with strong long-context capabilities and deeper reasoning paths.

    5. Engineering Implementation: Turning Capabilities into ROI with Routing and Validation Loops

    1. Tiered Routing: Route tasks by complexity. Retrieval, formatting, and minor single-file edits go to lightweight, fast configurations; cross-module refactoring, bug localization, and architecture migrations go to strong reasoning configurations.
    2. Context Engineering: Repo-level retrieval + hierarchical summarization + full injection of key files. Utilize prompt caching to reuse stable system prompts and repository summaries, significantly reducing repeated input costs.
    3. Validation First: Set “compile pass + test pass” as hard delivery conditions, forcing the model to run validation commands rather than self-evaluating as passed.
    4. Budget and Circuit Breaking: Set token limits, round limits, and cost limits per task. If exceeded, abort and return partial artifacts to prevent long-tail tasks from devouring the budget.
    5. Observability: Implement unified tracking for token distribution, rounds, time consumed, cost, and validation results for every task. Integrate with logging services and monitoring alerts to form drill-down reports by model/configuration dimension.
    6. Evaluation Set Building: Sample from real tickets and incident reports, fix validation scripts, and perform regression testing during version iterations to prevent silent capability degradation.

    These components do not require proprietary infrastructure. They can be built on standard combinations of general cloud products (such as Baidu AI Cloud’s model services, container platform, object storage, logging, and monitoring), making it easy to solidify evaluation from a one-time comparison into a continuously running pipeline.

    6. Conclusion

    • Unit price depends on architecture, total cost depends on the task: Sparse attention and KV cache compression optimize unit inference cost, while task success rate and token usage determine the final bill. The two cannot be conflated.
    • Long-range tasks depend on planning and self-test loops: Whether it can break down first, research next, validate, and then deliver is the core differentiator of strength; whether it can truly utilize long context for repo-level reasoning is another watershed.
    • Routine tasks depend on stability and ROI: Stable formatting, recoverable errors, and controllable latency are more important than architectural advancement.
    • Don’t ask “which model is the best”, but rather ask “which configuration has the lowest effective cost on my task distribution”—the answer comes from your own evaluation set, not from any public leaderboard.