Beyond Parameter Speculation: A Methodology for Evaluating LLM Coding Capabilities

1. Introduction: Real Testing Over Speculation

Whenever a new generation of large language models is about to be released, the tech community inevitably starts speculating: Will the new model’s parameter scale reach a new milestone? Will its overall performance surpass the current flagship? While these discussions are lively, they offer limited reference value for teams making technical decisions, for three reasons:

• Parameter scale is a necessary but not sufficient condition for capability. With the same parameter count, differences in architecture design, training data quality, post-training, and alignment strategies can lead to significant performance gaps. The popularity of architectures like Mixture of Experts (MoE) further separates “total parameters” from “activated parameters per inference.”
• “Strong capability” only makes sense when applied to specific tasks. A model topping the charts in mathematical reasoning doesn’t necessarily excel at code-driven multimedia generation.
• The actual delivery unit is an “Agent Shell + LLM” combination, not an isolated model. The shell determines how tasks are decomposed, how context is organized, and how tools are invoked. The same model can perform drastically differently with a different shell.

Instead of guessing, it is better to subject the problem to a structured, empirical test. Below is a review of an evaluation practice: five “Coding Agent Shell + LLM” combinations, given the exact same prompt and task, with only a single attempt allowed each. We let the undeniable numbers and replayable deliverables do the talking.

2. Evaluation Plan: Making Benchmarks Scientific

2.1 Choosing a “Small but Comprehensive” Task

The task selected for this evaluation was pure code-based video generation. Participating combinations were required to use code to create a 3D disassembly simulation of “why claw machines always fail to grab anything” and render a 20-second explainer video. A general-purpose “3D disassembly simulation” skill plugin was also provided—which can transform any machine into an interactive, step-by-step dissectible 3D model. Participating combinations could decide for themselves whether to utilize this ready-made framework.

This task seems lightweight but actually covers multiple capability chains:

1. Code engineering: Building the simulation and rendering pipeline from scratch.
2. Spatial and physical modeling: Accurately restoring details like the claw, transmission mechanism, and probability tricks.
3. Multimedia processing: Rendering, encoding, and outputting a video of the specified duration.
4. Information expression: Whether the video clearly explains the principle of “why you can’t grab anything.”

It also possesses three characteristics of an ideal benchmark task: clear deliverables (a playable video), a low validation threshold (ordinary people can judge if it makes sense), and visible failures (any broken link in the chain will be直观 reflected in the final video).

2.2 Controlling Variables

• Same prompt, same runtime environment, same skill plugin.
• Single attempt per group: no retries, no human intervention, no “give it another hint.”

Although the sample size of a single attempt is small, it effectively evaluates “first-impression delivery capability”—which is exactly the most common working mode in production environments. No one gives an agent unlimited retries.

2.3 Metric System: Objective Numbers + Subjective Scoring

Three objective metrics, extracted directly from runtime logs:

Combo Style	End-to-End Duration	Total Tokens Processed (Cache Ratio)	Output Tokens
Combo A - Fast Execution	8 mins	~3.5M (97%)	28K
Combo B - Quality First	17 mins	~12.15M (96%)	130K
Combo C - Balanced	21 mins	~13.94M (96%)	81K
Combo D - Steady	36 mins	~9.5M (98%)	59K
Combo E - Deep Thinking	46 mins	~46.44M (99%)	281K

One subjective metric: Deliverable quality. While video effects are inherently subjective, dimensions like “number of flaws,” “instruction adherence,” and “clarity of expression” can be blindly scored by multiple reviewers. Averaging these scores structurally captures the subjective experience as much as possible.

3. Empirical Observations: Five Distinct Execution Styles

3.1 Fast Execution: Fast, but Thoughtless

Combo A completed the task in 8 minutes with the lowest token consumption. However, reviewing its execution trajectory revealed almost zero exploration or planning. It even discarded the implementation framework provided by the skill plugin and directly generated output based on the literal prompt requirements. While instruction following is a strength, abandoning a framework that could have significantly improved the final video quality exposes a “blind execution, lack of judgment” flaw. This style suits execution tasks with clear goals and paths; its shortcomings become immediately apparent when the task requires trade-offs.

3.2 Deep Thinking: Overthinking is a Cost

Combo E was the opposite extreme: taking 46 minutes to complete, with total token consumption about 13x that of Combo A, and output tokens about 10x. Its trajectory exhibited classic “overthinking” characteristics: repeated planning, frequent self-review, and multiple rounds of starting over, with a significant portion of the thinking iterations providing no substantial gain to the final deliverable.

It must be emphasized: “Overthinking” is not an absolute flaw. In complex reasoning scenarios like architectural design or difficult debugging, deep thinking is often the very source of quality. The problem lies in the mismatch between the task and the thinking depth. For a one-time execution task with a clear goal, excess thinking is pure cost. Additionally, some shells offer mode switches like “Standard/Minimalist,” but the mode label doesn’t necessarily correspond to the actual thinking depth, making this a dimension worth evaluating independently.

3.3 Cache Hit Rate: An Underestimated Key Metric

A highly informative yet often ignored metric is the cache ratio: all five combos fell within the 96%–99% range. This reveals a commonality in agent workflows—high context repetition. Every tool call and iteration requires the model to re-read almost identical system prompts, skill documentation, and history logs, resulting in an extremely high proportion of cached input.

This leads to two deductions:

1. Total token count severely misleads cost intuition. Cached portions are usually billed at a steep discount compared to full-price input (platform rules vary; refer to actual pricing). A combo processing 40 million tokens might not actually cost much more than one processing 10 million.
2. Cost must be calculated using a formula, not guesswork.

# Cost estimation example (prices depend on actual platform pricing)
def estimate_cost(tokens, pricing):
    return (tokens.full_input * pricing.input
            + tokens.cached      * pricing.cache_hit   # Usually far lower than full input price
            + tokens.output      * pricing.output)

3.4 The Quality Champion, the Cost-Performance Player, and the Detail Failure

Combo B took the top spot for deliverable quality: clear disassembly logic, strong visual presentation, and accurate explanation of the “why it fails to grab” principle. It perfectly answered the prompt. Its time and token consumption were in the mid-to-low range, proving that quality and deep thinking are not necessarily bound together.

Combo D delivered good quality and ranked second to last in duration, making its efficiency unremarkable at first glance. However, if its unit price is friendly, combined with its extremely high cache ratio, its overall cost-performance could easily surpass others—another proof that “cost must be calculated by formula.”

The negative example is equally valuable: one combo received a Chinese prompt but produced a video with English text. Minor flaws like language inconsistency are a joke in demo scenarios but an incident in client deliveries. It reminds us that evaluation dimensions must include “whether the deliverable is presentable,” not just “whether the function is completed.”

4. Why the “Shell and Model” Fit Matters

A structural phenomenon was observed during the testing: combinations of models and shells from the same ecosystem often adapt well to each other in context organization, cache utilization, and tool invocation protocols. This manifests as higher cache hit rates and smoother execution flows. Cross-ecosystem combinations tend to waste “unnecessary tokens” due to protocol differences and context re-organization.

This validates the correctness of choosing to evaluate “combinations” rather than “isolated models” and leads to a principle for enterprise practices: Evaluate any model together with the actual shell and orchestration framework you use. Otherwise, the conclusions will have almost no reference value for your production environment.

5. Solidifying the Methodology: A Reusable Evaluation Pipeline

A lively tournament has limited value; solidifying it into a process creates sustained decision-making value.

5.1 Build a Task Library

Filter, desensitize, and solidify tasks from the team’s real-world work, covering four categories:

• Clear Execution: Code generation by spec, batch conversion, template-based output.
• Exploratory Design: Architecture design, tech selection, proposal review.
• Debugging: Reproducing, locating, and fixing difficult bugs.
• Creative Presentation: Visualization, animation, video generation.

Maintain 3–5 reproducible tasks per category to form a stable “capability spectrum.”

5.2 Define Metrics

• Objective: End-to-end duration, three types of tokens (full-price input / cached / output), first-pass success rate, failure mode classification.
• Subjective: Multi-person blind review with fixed scoring dimensions (completeness, accuracy, expression quality, detail presentability). Drop the highest and lowest scores and take the average.

5.3 Control Variables and Pipeline Automation

Same task, same prompt, same environment, single attempt. Solidify the task library, prompts, and runtime environment into repeatable container tasks, and automatically log evaluation records:

# Evaluation pipeline pseudo-code
for combo in agent_combos:               # Various "shell + model" combos
    for task in benchmark_tasks:         # Task library
        result = run_once(combo, task.prompt, retries=0)  # Single attempt
        metrics.record(
            combo=combo, task=task.id,
            duration=result.duration,
            tokens_full=result.tokens.full_input,
            tokens_cached=result.tokens.cached,
            tokens_output=result.tokens.output,
            cost=estimate_cost(result.tokens, combo.pricing),
        )
        artifacts.save(result.output)     # Archive deliverables for blind review replay

5.4 Normalization: Keeping Conclusions Fresh

LLMs iterate on a monthly or even weekly basis, and the shelf life of an evaluation conclusion might only be a few weeks. It is recommended to re-run the task library on a fixed cycle (e.g., bi-weekly) and plot the “Capability-Cost Curve” for each combination. Encourage team members to contribute real difficult problems to the library to keep the evaluation distribution close to production. On cloud platforms like Baidu AI Cloud, general capabilities such as model services, container platforms, logging services, and monitoring dashboards can lower the setup and maintenance costs of this pipeline: evaluation tasks run in containers, tokens and durations are written to logging services, and dashboards automatically update curves—making tech selection meetings data-driven.

6. Task Triage: The Ultimate Output of Evaluation

The end goal of evaluation is not an award ceremony, but a “task triage table”: what task goes to which type of combo, and what the budget and expectations are.

Task Type	Recommended Combo Style	Key Considerations
Clear Execution	Fast Execution	Speed and cost prioritized; quality flaws tolerated
Exploratory Design	Deep Thinking	Pay for thinking quality; accept higher duration
Creative Presentation	Quality First	Deliverable faces end-users; quality is pass/fail
Debugging	Balanced/Deep	Requires evidence chains and hypothesis validation

Mapped to our empirical test: for a clear, one-time generation task, there is no need to pay for deep thinking; for user-facing presentation tasks, the quality champion combo is worth waiting for.

7. Back to the Speculation: Do Parameters Dictate Capability?

We can now directly answer the opening question:

1. Parameter scale correlates with capability but does not dictate it. The separation of total and activated parameters in MoE architectures, training data quality, post-training and alignment strategies, and inference-time thinking mechanisms can all alter the final performance.
2. Capability must be measured on a specific task distribution. Discussing “who is stronger” without context is a pseudo-problem.
3. What you actually use is a system of “Model + Shell + Skill Plugins.” The quality of this system engineering is just as important as the model’s parameters.

So, instead of guessing whether the next model’s parameters will surpass the current one, build your evaluation system now. As soon as a new model is released, run it against your task library. Within 24 hours, you will have your own answer based on your specific task distribution. Data is more honest than any speculation thread.

Conclusion

The biggest takeaway from this five-combo, single-task empirical test is not “who won,” but a reusable evaluation methodology: small but comprehensive tasks, single-attempt discipline, three objective metrics (duration/tokens/cost), subjective blind review as a safety net, and task-triage outputs. Models will constantly evolve, and leaderboards will continuously refresh. But for any team, the most reliable judge is always the set of numbers generated on your own task library. Normalize evaluation, quantify tech selection, and the speculation will naturally find its answer.