Evaluating LLM Coding Capabilities: From Hands-On Testing to Scientific Model Selection

1. From “Parameter Speculation” to “Hands-On Testing”: Why Leaderboards Can Be Misleading

Whenever a new generation of Large Language Models (LLMs) is released, the most common debates in the community revolve around whether the parameter count exceeds the previous generation or if it can top the benchmark scores. However, from an engineering perspective, parameter scale and public leaderboards are merely a projection of a model’s capabilities:

• Static question banks are easily subjected to targeted optimization, and their question distribution is far from real-world development tasks.
• Parameter count does not have a linear relationship with task performance. The quality of training data, post-training strategies, and the engineering implementation of the inference framework significantly affect the actual user experience.
• Coding tasks are particularly unique: They are not simple “Q&A” snapshots. Instead, they are long-horizon processes involving planning, tool calling, trial and error, and self-reflection. Any static evaluation struggles to cover these aspects comprehensively.

Therefore, a more valuable approach than “guessing who is stronger” is to establish your own controlled comparison experiments: the same task, the same prompts, and a single attempt, allowing different combinations of “execution framework + model” to compete on the same stage, using data instead of opinions.

2. Designing a Fair Evaluation: Tasks, Constraints, and Controlled Variables

The core of a convincing horizontal evaluation lies in task selection and variable control. Take a typical experimental design as an example: the task chosen is highly representative. The agent is asked to generate a 20-second short video purely by writing code, with the theme “disassembling a claw machine and explaining why it always fails to grab anything.” This task has several characteristics suitable for horizontal evaluation:

1. Clear boundaries: The output is a single, playable video file, making it immediately obvious whether it was completed.
2. Moderately high difficulty: It involves modeling, animation logic, duration control, and the rendering process, requiring the model to perform multi-round planning and trial and error.
3. Attachable constraints: For example, “the output language must match the prompt language,” which tests instruction-following capabilities.
4. Single attempt: No manual intervention or retries are allowed, forcing out the true capability level.

The controlled variables are straightforward: unified prompts, unified execution environment, and unified recording standards. We only compare three things: effectiveness, time consumption, and token consumption.

3. Three Typical Profiles: What the Data Looks Like

Referring to a public record of a hands-on test in the community (specific product information has been anonymized; the data is used solely to illustrate how to interpret metrics), the results presented three highly typical profiles:

Profile	Time Magnitude	Total Processed Tokens	Cache Hit Rate	Output Tokens	Characteristics
Efficient	~8 minutes	~3.5 million	97%	~28,000	Clean process, gets it right in one step
Balanced	17–21 minutes	12M–14M	96%	80k–130k	Stable and mediocre, with trade-offs in quality
Deep-thinking	~46 minutes	~46 million	99%	~280,000	Repeatedly re-reads context; final artifact has noticeable flaws

Three details are worth noting: The “Efficient” profile had the lowest total processed tokens and completed the fastest. The “Deep-thinking” profile “thought the most,” yet its final artifact had quite a few flaws—it even output an English artifact for a Chinese prompt, failing in instruction adherence. Interestingly, because the Deep-thinking profile had the highest cache hit rate, its actual billing might not be the most expensive—cost must be calculated in conjunction with the pricing structure, not just by looking at the total processed tokens.

4. Meaning and Interpretation of Five Core Metrics

1. End-to-End Time Consumption

The time from receiving the prompt to producing the final deliverable consists of three parts: model inference time, tool calling and execution time (writing files, running renders, executing scripts), and the scheduling overhead of the execution framework itself. Long durations might be strategic (more rounds of self-checking, finer task decomposition) or environmental (slow dependency installation, rendering queues). When interpreting, “strategic slowness” and “environmental slowness” must be separated; taking a long time does not equate to poor capability.

2. Total Token Processing Volume

In the multi-turn loop of an agent, each step feeds the historical context back into the model. Therefore, the total processed volume is often an amplifier on the input side. The gap between 3.5 million and 46 million essentially reflects the “width of the exploration path”: some combinations advance precisely at each step, while others bounce back and forth between alternative solutions. This is the quantified form of “overthinking”—thinking yields returns, but with diminishing marginal utility. Beyond a certain threshold, only costs remain.

3. Cache Hit Rate

Mainstream inference services offer cache billing discounts for repeated prefix contexts. A higher hit rate means lower actual input-side costs. However, note: a 99% hit rate paired with 46 million total processed tokens might indicate that the agent is repeatedly re-reading the same constantly expanding context—saving money, but not time, and not necessarily improving quality. It is a cost metric, not an efficiency metric.

4. Output Token Volume

There is a tenfold difference between 28,000 and 280,000, but the artifact’s completion level might not have a tenfold gap. Output volume is more like a “talkativeness degree” and must be read alongside quality: high output + high quality equals deep thinking; high output + low quality equals an out-of-control loop; low output + high quality equals precise execution.

5. Result Quality

It is recommended to split this into two layers: The objective layer can be automated (artifact exists, is playable, duration meets the standard, language matches the prompt). The subjective layer uses a fixed scoring rubric (e.g., 1–5 points) to evaluate aesthetics, narrative logic, and technical correctness, with scoring criteria standardized by the same person.

5. Five Common Pitfalls in Interpreting Results

1. A single run has no statistical significance: Sampling randomness and environmental jitter can cause fluctuations. The same task should be repeated at least 3–5 times to observe the distribution rather than looking at a single data point.
2. Equating time consumption with capability: Time reflects strategic orientation, and the optimal strategy inherently differs across various tasks.
3. Assuming a higher cache hit rate is always better: An excessively high hit rate might expose inefficient re-reading loops.
4. Ignoring task distribution matching: Is your daily work rapid prototyping or large-scale refactoring? The evaluation task should closely align with your real-world workload.
5. Looking only at effectiveness and ignoring cost: Total processed tokens × unit price is the actual expense. Input and output tokens typically have different unit prices and must be calculated separately.

6. Building Your Own Evaluation Workbench: A Minimum Viable Solution

You don’t need complex infrastructure; a script and a spreadsheet are enough to start. The core revolves around four things: a unified adaptation layer (wrapping different model services into a consistent interface), a logger (persisting time and token statistics to disk), a scorer (automated checks + manual scoring), and a re-test summary (taking the median of multiple runs). The example code is as follows:

import time, statistics
def run_once(agent, prompt: str) -> dict:
    t0 = time.time()
    result = agent.run(prompt)              # Unified interface: single call, no retries
    return {
        "agent": agent.name,
        "wall_time_sec": round(time.time() - t0, 1),
        "total_tokens": result.usage.total_tokens,   # Includes cached read portions
        "cached_ratio": result.usage.cached_ratio,   # Cache hit proportion
        "output_tokens": result.usage.output_tokens,
        "artifact": result.artifact_path,            # Artifact path, e.g., video file
    }
def auto_check(artifact_path: str) -> bool:
    # Objective checks: artifact exists, is playable, duration meets standard, language matches prompt
    return exists(artifact_path) and playable(artifact_path) \
        and duration_ok(artifact_path) and language_ok(artifact_path)
def summarize(records: list[dict]) -> dict:
    return {
        "median_time": statistics.median(r["wall_time_sec"] for r in records),
        "median_total_tokens": statistics.median(r["total_tokens"] for r in records),
        "pass_rate": sum(auto_check(r["artifact"]) for r in records) / len(records),
    }

Doing this in the cloud can be even more worry-free: by routing through a unified model service gateway to access multiple models, and utilizing general capabilities like logging services and monitoring alerts to persist evaluation data, you achieve traceability and reproducibility. For example, on cloud platforms like Baidu AI Cloud, these general product capabilities can be directly combined and used, allowing the evaluation scaffolding to focus solely on the task logic itself.

7. From Evaluation to Selection: Task Routing Mindset

The endgame of horizontal evaluation is not to “select the single strongest model,” but rather to determine “which model is best for which task”:

• Rapid prototyping / small scripts: Prioritize efficient combinations with low time consumption and low output volume.
• Complex refactoring / long-context analysis: Allow deep-thinking combinations to spend more time and tokens in exchange for a more comprehensive solution.
• High-frequency automated pipelines: Cost-sensitive; focus heavily on total processing volume and caching strategies.
• Strong instruction-following scenarios (e.g., multilingual output):: Make checks like “language consistency” a hard prerequisite.

It is recommended to re-run the same set of tasks every time a new model is released, continuously maintaining a “Task–Model” mapping table. Models iterate rapidly, and yesterday’s conclusions might be invalid today—this is why a “reproducible evaluation workbench” is far more important than a “one-time conclusion.”

Conclusion

Returning to the original question: “Will the parameters and capabilities of the new generation model surpass its competitors?” Parameters and benchmark scores can be speculated upon, but engineering decisions cannot rely on guessing. Put hands-on horizontal evaluations, the five core metrics, and the task routing mindset into your toolbox. Run them every time a new model is released, and what you get will not be opinions, but evidence.