New Model Canary Testing: How to Scientifically Evaluate Coding Agent Performance
When new coding models enter canary (gray-scale) testing, similar benchmark scores don't always translate into similar real-world experience. This article explains why developers often perceive a new model as slower in practice, and breaks down the metrics that gray-scale evaluations tend to overlook—TTFT, decoding throughput, and prompt-cache hit rate. It examines common patterns such as bash-heavy tool use and excessive round-trips caused by framework overfitting, early-release instability and low-priority service pools, and the tradeoff of spending inference time to gain benchmark score. It then lays out a reproducible evaluation workflow with a metrics system, shadow-traffic rollout, task-trajectory review, and a pitfalls reference—so teams can combine different models by task type instead of chasing a single all-round winner.
Introduction: Similar Scores, Different Feel
Recently, many vendors have been rolling out new generations of coding models and opening them up to canary (gray-scale) testing. Plenty of discussion has centered on whether a new model can join the top tier, but most of it stays at two dimensions: benchmark rank and “how fast it feels.” Back in real development scenarios, things get more complicated. On public code-capability leaderboards, a new model may be noticeably larger than the previous generation yet score only comparably, so the cost-performance advantage isn’t obvious. Developers’ gut reaction is often “slow—and not just slow output”: the session takes a long time to start, thinking runs long, and even a simple change takes many back-and-forth rounds.
This isn’t surprising. Leaderboards mostly evaluate single-turn answers to static problems, judged only on whether the final code is correct. Real coding tasks are a long chain stitched together from many rounds of “model reasoning + tool calls”:
Intent understanding → codebase retrieval & localization → edit code → run tests to verify → fix failures → verify again → commit
This chain can be written as:
T_total ≈ Σ ( T_queue + T_prefill + T_think + T_decode + T_tool )
If any single term is high, or if the number of rounds is large, the end-to-end time is amplified many times over. So the real experience during canary testing depends on two categories of factors:
- Model side: time to first token, decoding throughput, context efficiency, prompt-cache utilization, thinking time;
- Service side: capacity stability early in the release, account service tier, rate-limiting policies.
This article starts from several typical phenomena observed in real testing, highlights the metrics that are easily overlooked in coding-agent canary evaluations, and offers a reusable evaluation process and decision method.
1. Phenomenon 1: Break “Fast” into Three Dimensions to Find the Source of the Lag
“Fast” (or slow) is a subjective feeling and must be broken down into measurable metrics. Every round of a coding agent goes through “wait for first token → stream output → tool execution,” and a single round’s latency multiplied by dozens of rounds produces the final felt experience. Three metrics matter most:
- TTFT (Time To First Token): the time from sending the request to receiving the first token, made up mainly of queueing time and prompt-prefix processing (prefill);
- TPS (Tokens Per Second, decoding throughput): the speed at which the model keeps generating tokens after the first token;
- Prompt-cache hit rate: the share of prompt tokens that can be served from a reusable prefix cache.
All three are necessary:
- If TTFT is low but cache hits are poor, long-context scenarios will still lag noticeably;
- If throughput is high but TTFT is long, users feel “nothing happens for ages, then once it starts it won’t stop”;
- When context only grows in long-context scenarios and the cache can’t be reused, each round must reprocess the entire history, pushing up both latency and cost.
The table below shows several common latency profiles in coding-agent evaluation (values are illustrative, used to explain the structure of the problem):
| Latency profile | Typical behavior | Effect on felt experience | Response direction |
|---|---|---|---|
| Slow first token, normal throughput | Waits to start each round, smooth once going | More rounds means more accumulated waiting | Enable prefix caching, shorten system prompts |
| Fast throughput, slow first token | Slow start, smooth on long responses | Dominated by TTFT in multi-round small-step tasks | Prioritize optimizing queueing and prefix processing |
| Both first token and throughput slow | Waiting throughout | Long tasks nearly unusable | Limit thinking time and rounds, or switch to a fallback model |
| Fast throughput, poor cache | Smooth single round, noticeably slower over rounds | The longer the context, the bigger the gap | Stabilize the prefix, trim history on demand |
| Each dimension acceptable but many rounds | No single slow round, long overall | “Not slow, yet spinning forever” | Optimize tool selection and call strategy (see section 2) |
In agents, “fast throughput” is also easily cancelled out by output length. Some models produce verbose output at high speed but generate more content that isn’t needed, so total time isn’t necessarily shorter. A more reasonable metric is therefore effective per-round latency—the full time of one round from kickoff to tool-result return:
Per-round latency ≈ TTFT + thinking time + output tokens / TPS + tool execution timeEnd-to-end time ≈ per-round latency × number of rounds
The following illustrative (pseudo) code collects these metrics:
import statisticsimport timedef measure_model(client, build_prompt, samples=5):"""Measure a model's latency profile under a fixed prompt for side-by-side comparison.build_prompt() constructs the system prompt, tool definitions, and multi-turnhistory to match a real coding scenario.Different services may expose cache fields in `usage` differently; the fieldnames below are illustrative only."""ttft, tps, cache_ratio = [], [], []for _ in range(samples):start = time.perf_counter()first = Noneout_tokens = 0usage = {}for ev in client.stream(build_prompt()):if first is None:first = time.perf_counter() # record first-token arrival timeout_tokens += 1usage = ev.usage or {}end = time.perf_counter()ttft.append(first - start) # queueing + prefix processingtps.append(out_tokens / max(end - first, 1e-6)) # decoding throughputprompt_tokens = usage.get("prompt_tokens", 0)if prompt_tokens:cache_ratio.append(usage.get("cached_tokens", 0) / prompt_tokens)pct = lambda xs, p: sorted(xs)[min(len(xs) - 1, int(len(xs) * p))]return {"TTFT_P50": pct(ttft, 0.50),"TTFT_P95": pct(ttft, 0.95),"TPS_P50": pct(tps, 0.50),"cache_hit_rate": statistics.mean(cache_ratio) if cache_ratio else 0,}
Three recommendations when collecting these numbers:
- Look only at P95/P99, not averages: coding sessions often run dozens of rounds, and a single request that lands on the long tail is enough to slow the whole task noticeably;
- Fix the prompt and tool definitions: frequently changing prompts invalidate the cache constantly and make comparison meaningless;
- Compare against the previous-generation model and the current mainstream model in the same time window: differences in time window and load can mask the model’s own differences.
2. Phenomenon 2: Bash Preference and Small Steps—The Cost of Agent-Framework Overfitting
Many people observe similar behavior in testing: after calling general commands a few times, a model tends to keep using the command line and stops calling dedicated retrieval and read tools. For example, viewing a file or searching a keyword—which should be handled by dedicated tools—gets cobbled together with text-processing commands on the command line. It also tends to break a task into a series of “small steps”: each step does one tiny thing, returns, then starts the next.
2.1 Cause: Mismatch Between the Training and Deployment Environments
A plausible explanation is that a model’s tool-use habits come from the agent framework used during training or evaluation. Many public agent benchmarks use a minimal framework that “only exposes command-line execution.” After repeated reinforcement in such an environment, the model treats “always go through the command line first” as its default strategy. Deployed into a framework with richer tools, the habit doesn’t disappear:
- Dedicated retrieval and read tools get bypassed;
- Output and return formats don’t match the current framework’s constraints, forcing the model to trial-and-error and re-read repeatedly.
In addition, prompt caching usually matches on an exact prefix. Once the prefix changes (for example, tool definitions or the order of system prompts), the cache is invalidated, further amplifying latency.
2.2 Impact: Double Amplification of Rounds and Context
The harm of small steps goes beyond “looking verbose”; it amplifies cost along two paths:
- Round amplification: each tool call corresponds to at least one full model inference (including thinking). If a round takes 20 seconds and a task uses 40 rounds, total time is over ten minutes;
- Context bloat: every round’s model output, thinking, and tool results stay in the context and must be reprocessed each round. The longer the context, the slower the prefix processing and the higher the cost—and some services also rate-limit or slow down extremely long contexts.
The illustrative logic below measures how well framework and model behavior match:
def profile_tool_usage(trace):"""Profile the call trace of a full session to judge framework-model fit."""total = len(trace)shell_only = [s for s in trace if s.kind == "command"]return {"rounds": total,"command_call_ratio": round(len(shell_only) / total, 2),"repeated_reads": sum(s.kind == "read" and s.path in s.seen_before for s in trace),"peak_context_tokens": max(s.context_tokens for s in trace),}
When the “command call ratio” and “repeated reads” are clearly high, it usually means the model doesn’t fit well within the current framework.
2.3 Response: Align Model Behavior with the Framework
- Define tool-use rules clearly: in the system prompt, spell out “prefer dedicated retrieval and read tools; use the command line only when dedicated tools can’t do the job,” with a few examples;
- Soft guidance, not hard blocking: when the model calls the command line, append a hint to the result suggesting dedicated tools next time. Hard blocking tends to make the model retry repeatedly, which adds rounds instead;
- Govern tool output: cap the length or summarize read and retrieval results (for example, return only matched snippets with line numbers) so large blocks of content don’t crowd out the context;
- Set a thinking budget by task type: small changes with clear localization don’t need long thinking; open it up for complex debugging;
- Add behavioral metrics to regression: treat rounds, command-call ratio, and repeated reads as regular regression items for model versions.
3. Phenomenon 3: Early-Release Instability and Low-Priority Service Pools
3.1 Volatility in a New Model’s Early Days Is Normal
Canary phases of new models often come with clear volatility: some stabilize only after a period, while others still wobble long after launch. Common causes include:
- Compute contention: training the next-stage model consumes inference resources;
- Capacity ramp-up: inference capacity hasn’t scaled enough to match traffic;
- Cold-start effects: newly launched models lack mature caching, scheduling, and rate-limiting experience.
So both “new models are necessarily unstable” and “models necessarily stabilize over time” are oversimplified. Each model and each time window must be observed separately.
3.2 Account Tier Affects Experience
As coding-focused AI tools grow rapidly in user base, free or low-tier accounts may enter low-priority service queues at peak times, showing up as long queueing and large TTFT swings. Testers should record account tier, test time window, and rate-limiting (such as the share of 429 responses); otherwise it’s easy to misjudge “service-tier differences” as “model-capability differences.”
3.3 Set Up a Release Observation Window
It’s recommended to set a 1–2 week observation period for a canary model, sample across time windows, and set clear thresholds:
- Timeout rate ≤ 1%
- TTFT P99 ≤ baseline × 1.5
- Meet targets for 3 consecutive working days before moving to formal evaluation or widening the canary.
4. Phenomenon 4: The Tradeoff Behind “Speed for Score” and Token Efficiency
Models with generally good experience tend to be fairly balanced in TTFT and decoding throughput and fairly efficient in tool calls, rather than standing out on a single metric. Others are kept even when speed has clearly dropped, because they have high token efficiency (fewer tokens to complete the same task) and ample call quota. Conversely, many new models show “long thinking, many rounds, high latency on every dimension”—essentially trading inference time and context consumption for a leaderboard score.
The root cause is that the evaluation objective and the product objective don’t align:
- Leaderboards optimize single-turn correctness, where generating more candidates and doing longer self-verification usually helps;
- Developers want how much deliverable work gets done per unit time.
“High token efficiency” and “fast output” are not equivalent: if completing the same task needs more input/output tokens, even a cheaper per-token price doesn’t guarantee lower total cost. Use the notion of “effective tokens” to measure this:
Effective tokens = tokens that truly contribute to the result (final merged code, tests, logs, etc.)Token efficiency = effective tokens / total tokens consumed (including repeated reads, repeated thinking, and discarded output)
In a canary review, place this metric alongside task success rate and per-task cost, rather than looking only at unit price or leaderboard rank. Also, models have different strengths on different tasks; complex planning, large-scale refactoring, and local patches need not be bound to the same model. Routing by actual task is often more worthwhile than hunting for an “all-round #1.”
5. How to Scientifically Run a Coding-Model Canary Evaluation
5.1 Evaluation Process
- Freeze the agent framework: all candidate models use the same tool set, system prompt, and retry/timeout policy, with version numbers recorded in the report;
- Build the task set: draw representative tasks from real projects (bug fixes, new features, refactoring, dependency upgrades), and after de-identification, fix repo snapshots and acceptance criteria (test commands, expected results);
- Fix sampling conditions: unify temperature, thinking budget, context limits, and retry counts, and run each model under the same conditions in similar time windows;
- Repeat runs multiple times: run each task at least 3–5 times, and report mean, P95, and success rate;
- Manual spot checks: review code quality, security, and maintainability beyond automated acceptance via human sampling;
- Continuous regression: rerun automatically after a model or framework upgrade to prevent regressions.
5.2 Metrics System
| Dimension | Core metrics | Description |
|---|---|---|
| Quality | Task pass rate, test pass rate, human rating | Whether it truly solves the problem |
| Experience | End-to-end time P50/P95, per-round latency, rounds | What developers directly feel |
| Service | TTFT, decoding throughput, cache hit rate, timeout rate, rate-limit rate | Pinpointing latency sources |
| Cost | Total tokens per task, cost per task, effective-token share | Avoiding “low unit price, high total bill” |
| Behavior | Tool-call distribution, command-call ratio, repeated reads | Judging framework fit |
The table below is a sample metrics dashboard; thresholds should be set against your own baseline:
| Metric | Candidate A (illustrative) | Candidate B (illustrative) | Baseline model (illustrative) | Pass threshold (example) |
|---|---|---|---|---|
| Task pass rate | 62% | 68% | 65% | ≥ baseline + 3pp |
| End-to-end time P95 | 9.5 min | 14.2 min | 8.8 min | ≤ baseline × 1.2 |
| TTFT P95 | 6.2 s | 18.5 s | 4.8 s | ≤ baseline × 1.5 |
| Cache hit rate | 71% | 46% | 68% | ≥ baseline − 5pp |
| Command-call ratio | 0.38 | 0.67 | 0.40 | ≤ baseline + 0.10 |
| Cost per task | 1.15× | 1.42× | 1.00× | ≤ baseline × 1.2 |
In this example, Candidate B has the highest task pass rate, but its TTFT, command-call ratio, and cost all exceed the thresholds, making it better suited as a “complex-task specialist” than a wholesale replacement.
5.3 Shadow Traffic and Canary Rollout
An offline task set can only reflect part of the picture. A more robust approach is:
- Feed the new model into shadow traffic, run it in parallel on real requests, and only record without affecting users;
- Split traffic by task complexity, ramping up simple tasks first;
- Set automatic circuit-breaker conditions (timeout rate, user-interruption rate, per-task cost over threshold);
- Keep the ability to roll back to the baseline model with one click.
5.4 Task Trajectory Review
Beyond automated metrics, sampling and reading full task trajectories is just as important. Focus on four kinds of signals:
- Back-and-forth count: whether the same file is read repeatedly, or the same location edited repeatedly;
- Rework causes: misunderstanding, tool results not matching expectations, or framework errors;
- Verification habits: whether it proactively runs tests after edits;
- Failure modes: stuck loops, getting worse with each change, accidentally deleting the user’s edits.
These signals explain the reasons behind the metrics and directly determine a developer’s subjective trust.
5.5 Engineering It: Turn Evaluation into a Reproducible Pipeline
Ad-hoc manual testing is hard to reproduce and can’t keep up with the regression demands once user volume grows, so it’s best to engineer it. On Baidu AI Cloud, you can combine several general-purpose capabilities as needed:
- Object storage: archive code snapshots, model outputs, task trajectories, and evaluation reports;
- Container platform: provide an isolated, reclaimable execution environment for each task;
- Log service: collect per-round calls, tool execution, and timing data;
- Monitoring & alerting: set threshold alerts for latency, timeout rate, and failure rate;
- Model service: unify access to candidate and baseline models.
For specific capabilities, quotas, and billing, refer to the official documentation. A pipeline skeleton looks like this:
def evaluate(model, tasks, concurrency=4):results = []with ThreadPoolExecutor(max_workers=concurrency) as pool:futures = {pool.submit(run_task, t, model): t for t in tasks}for fut in as_completed(futures):t = futures[fut]try:results.append(run_metrics(fut.result(timeout=1800))) # per-task timeout capexcept Exception as e:results.append({"task": t.id, "error": str(e)})alert(f"evaluation task failed: {t.id}") # failed tasks go to alerting and manual reviewarchive("report", results) # archive to object storage for cross-run comparisonreturn summarize(results) # output P50/P95, pass rate, cost, etc.
Key points include:
- Record each round’s calls, tool execution, and timing in structured form in the log service, for aggregation by model, time window, and task type;
- Set monitoring alerts for latency and timeout rate to catch release-time capacity swings promptly;
- Record the framework version, prompt version, and sampling parameters for each evaluation, ensuring results are reproducible and traceable.
6. Common Pitfalls and Summary
6.1 Common Evaluation Pitfalls
| Pitfall | Consequence | Right approach |
|---|---|---|
| Only looking at leaderboard rank | Ignores the real task distribution; big gap between online and felt experience | Center on the real task set; use leaderboards only as reference |
| Evaluating coding ability with single-turn Q&A | Can’t reflect the overhead of multi-round tool calls | Run the full task chain end to end |
| Different frameworks for each model | Results confounded by framework differences | Unify the framework and record the version |
| Only looking at average latency | Hides the long tail | Watch P95/P99 and stability |
| Ignoring cache and account tier | Misjudges service differences as model differences | Record test conditions; fix the prompt |
| Chasing an “all-round #1” | Robbing one dimension to boost another; rising cost | Route by task complexity |
Beyond this, canary reviews should also cover security and compliance: license risk in generated code, sensitive-information leakage risk, and isolation and permission control of the execution environment (least privilege, blocking unnecessary outbound network access).
6.2 Summary
In this recent wave of coding-model canary tests, “similar scores, larger size, slower feel” is not a contradiction. Leaderboards measure single-turn problem-solving ability; what developers feel is the efficiency of the whole task chain:
- TTFT, decoding throughput, and cache hit rate decide the wait in each round;
- small-step calls from framework overfitting decide the number of rounds;
- early-release service stability and account service tier decide the long-tail experience;
- the tradeoff of spending inference time for score decides the real per-task cost.
For a team, the more rational strategy isn’t chasing the “strongest model” but building a reproducible, measurable, rollback-capable evaluation system: use real tasks as the baseline, use P95 latency and per-task cost as constraints, use task pass rate as the floor, and combine different models by task type. That way, however new models iterate, selection decisions stay grounded in evidence.
