Evaluating AI Coding Agents: A Data-Driven Framework Using Time and Token Metrics

1. Background: From Speculation to Empirical Testing

Periodically, the tech community debates whether the next generation of AI models will surpass current flagships in parameter scale and capability. These discussions, fueled by parameter rumors, architecture analyses, and leaked benchmarks, rarely yield definitive conclusions. For developers and enterprise users, a more reliable approach is to translate this question into a reproducible experiment: give an AI coding agent a specific task, apply the same resource constraints, allow only a single attempt, and let execution time and token data answer three critical questions—who is faster, where do they excel, and how much do they cost?

This article provides a ready-to-implement evaluation framework, demonstrated through an end-to-end task such as “generating a machine disassembly video using pure code.”

2. Task Design: A One-Sentence Runnable Benchmark

A good benchmark task should meet three conditions:

1. Minimalist Instruction: Requirements should be clear in a single sentence. For example: “Turn a claw machine into an interactively disassemblable 3D simulator and record a 20-second disassembly video.” This problem inherently contains a core puzzle—the disassembly process must explain “why claw machines always fail to grab” (claw strength, center of gravity offset, probability mechanisms, etc.).
2. Verifiable Output: Whether the video is successfully generated and whether the disassembly logic is sound are immediately apparent and do not rely on complex judges.
3. Broad Coverage: A single task simultaneously covers 3D modeling, mechanical structure comprehension, animation orchestration, rendering pipelines, and automated script writing, effectively stress-testing the agent’s comprehensive capability limits.

An advanced approach is to pre-mount external skill frameworks, such as a “disassembly simulator,” to evaluate the agent’s willingness to reuse existing engineering assets—whether it dutifully uses the framework or abandons it to reinvent the wheel.

Rule Setting: Each participant gets only one attempt. Human intervention and retries are strictly forbidden to prevent the “grinding” effect from diluting the conclusions.

3. Hard Metrics: Time and Tokens Are Undeniable Numbers

Subjective perceptions can be debated, but hard data cannot. It is recommended to record at least four metrics:

Metric	Meaning	Why It Matters
Total Time	From task initiation to product delivery	Directly reflects interactive experience and task throughput
Total Processed Tokens	Cumulative sum of input + output + cache	Coarse-grained measure of workload
Cache Proportion	Ratio of tokens hitting the context cache	Key determinant of the real cost structure
Output Tokens	Tokens actually generated by the model	Highly correlated with the “thinking/production ratio”

A crucial point regarding total volume interpretation: long-session agents repeatedly review the same context during task execution, often resulting in a cache hit rate of over 90%. Cached tokens are typically priced an order of magnitude lower than uncached inputs, while output tokens usually account for the bulk of the billing. Therefore, looking only at “Total Processed Tokens” will systematically overestimate the cost differences between agents. Under tasks of the same scale, it is not uncommon to see execution times differ by multiples and total tokens differ by tenfold. This is precisely the value of quantitative evaluation.

4. Soft Metrics: Engineering Subjective Quality

Video effects are subjective, but they can be decomposed into objective sub-items for scoring:

• Task Completion: Did it produce a playable video meeting the duration requirement?
• Content Accuracy: Does the disassembly genuinely explain the mechanical and probabilistic reasons for the “failure to grab”?
• Language Consistency: Does a prompt in a specific language yield an output in the same language? In tests, it is common to see prompts in one language produce videos with subtitles in another.
• Visual Quality: Resolution, frame rate, lighting, and camera movement.
• Framework Adherence: Did it reuse the provided skill framework rather than starting from scratch?

Each item is scored 1–5, with multiple independent reviewers taking the median, turning “subjective” observations into statistically trackable data.

5. Behavioral Patterns: Three Typical Agent Profiles

Based on multiple rounds of testing, agents generally exhibit three behavioral profiles:

1. Deep Thinker: Repeatedly plans, frequently self-rejects and redoes. Both time and token usage are high. Interestingly, “thinking too much” does not necessarily yield better results; overthinking is a recognized phenomenon in both academia and engineering.
2. Speed Submitter: Completes tasks in minutes, strictly following literal instructions, sometimes even discarding given frameworks to take the shortest path to the goal. Fast, but may sacrifice depth and quality.
3. Balanced: Moderate execution time, stable quality. This is the ideal profile for most production scenarios.

There is no absolute good or bad: prototype validation favors the Speed Submitter, while final delivery leans towards the Balanced or even Deep Thinker profiles.

6. Cost Estimation: Don’t Be Intimidated by Total Tokens

The general estimation formula is as follows:

Cost ≈ Uncached Input Tokens × Price A + Cached Tokens × Price B + Output Tokens × Price C

Example code:

def estimate_cost(input_uncached, input_cached, output,
                  p_uncached=1.0, p_cached=0.1, p_output=3.0):
    """Illustrative estimation based on per-million-token price magnitudes. 
    Actual costs depend on the specific platform's pricing."""
    m = 1_000_000
    return (input_uncached / m * p_uncached
            + input_cached / m * p_cached
            + output / m * p_output)

Three reminders: Pricing rules vary significantly across platforms, so always refer to the official pricing page; the unit prices above are for magnitude illustration only and do not represent any specific product; cost must be evaluated jointly with quality—if it’s cheap but requires two reworks, it isn’t truly cheap.

7. Evaluation Engineering and Selection Advice

• Accumulate Data: Write each evaluation round as a structured record (task, time, tokens, score, model version). Long-term accumulation forms your team’s own benchmark curve.
• Control Variables: Fix prompts, fix randomness parameters, and stick to single attempts. Cross-version comparisons only make sense under these constraints.
• Align with Your Business: Community benchmarks are for reference; your actual codebase and workflows are the final testing ground.
• Leverage Cloud Capabilities: For instance, on Baidu AI Cloud, you can use object storage to retain evaluation artifacts and utilize logging services and monitoring alerts to track task execution time, upgrading a one-time horizontal evaluation into a sustainable engineering process.

8. Conclusion

Returning to the initial question: Will the next generation of models surpass the previous one? The narrative of parameter scale will eventually fatigue; the boundaries of capability can only be approximated through reproducible, empirical testing. Rather than taking sides in speculation, it is better to build your own benchmarks. When enough data is accumulated, the answer will naturally surface.