Benchmarking LLM Clusters with Real Agent Coding Traces: Methods and Practice

    Agentic coding workloads stress inference clusters in ways that single-request benchmarks miss. This article explains how trace-driven benchmarks, built from captured agent execution traces and replayed on a cluster, measure task-level performance. It shows how long contexts, dense tool calls, and parallel sub-agents turn into memory, bandwidth, and queueing pressure. It then lays out metrics that go from TTFT and inter-token latency to goodput, cache hit rate, task quality, and cost. Practical coverage includes attribution of performance differences, six common pitfalls to avoid, a step-by-step guide to building your own benchmark, and the platform capabilities needed to sustain it.

    1. The Starting Point: Why Single-Request Benchmarks Are No Longer Enough

    In chat-style applications, one request usually consists of a single user turn and a single answer. Both are of limited length, and consecutive requests have almost nothing to do with each other. Inference benchmarks built mainly around input length and output length are enough to characterize that kind of load. However, as AI coding assistants evolve from “completing a snippet of code” to “independently finishing a development task,” the shape of the workload changes fundamentally:

    • Context keeps growing. Every file the model reads, every test it runs, and every error message it inspects is appended to the context. A task of moderate complexity often involves tens of interaction turns.
    • Tool calls are dense. Code retrieval, command execution, test runs, and file edits are interleaved with every inference step, so inference latency turns directly into developer wait time.
    • Sub-agents are spawned in parallel. A primary agent can split work among several sub-agents, producing multiple concurrent inference streams in an instant.
    • Prefixes are highly repetitive. Within a task, many requests share the same ever-growing history; across sub-agents, they may also share the same instructions and repository context.

    Combined, these traits make the workload heavier along several dimensions at once. Average input length rises. The share and variance of long sequences increase sharply. KV-cache occupancy and lifetime stretch out. The number of requests per task multiplies. Arrivals shift from uniform to bursty. As a result, measuring throughput and latency with short prompts and fixed output lengths often produces conclusions that do not extrapolate to real scenarios.

    A class of approaches that has emerged in recent years is worth attention: drive benchmarks from real captured agent coding traces, and measure an inference system by the “tasks completed” it actually serves. In late August 2026, a cloud provider evaluating a next-generation rack-scale AI computing platform used this kind of benchmark across large language models of several mainstream architectures. Setting aside the specific products and models, this article discusses the general evaluation methodology behind such benchmarks, to help readers build a reusable assessment framework.

    2. What These Benchmarks Actually Measure

    Traditional inference benchmarks are essentially synthetic load tests. Input length, output length, and arrival rate are set by hand, and the system’s behavior under different pressure levels is observed. The approach is controllable and reproducible, but it carries a systematic bias relative to real business traffic. Trace-based benchmarks instead use replay-style evaluation, which typically involves four stages:

    1. Trace collection. Run agents in a real or highly realistic coding environment and record each request and response turn by turn. Recorded content includes the complete prompt sent to the model, tool return values, the output of every turn, the timing and duration of each tool call, and sub-agent spawning relationships.
    2. Trace reconstruction. Split a conversation into the smallest request units that the inference service can replay, while preserving three key relationships:
      • prefix dependencies between turns;
      • the spawning and convergence relationships between sub-agents and their parent task;
      • the causal ordering of tool calls.
    3. Load replay. Replay requests on the cluster under test following the original timing (or a proportionally scaled timeline). Keep the dependency waits between requests instead of dumping every request at the system at once.
    4. Performance measurement. Beyond token-level throughput and latency, measure whether tasks finish within their constraints and how many resources it takes to complete one valid task.

    Compared with synthetic benchmarks, the biggest value of preserving real traces is distributional fidelity. The following illustrative collection-and-replay structure shows the basic logic:

    1. # Illustrative code: extracting inference requests and their dependencies from a raw trace
    2. @dataclass
    3. class InferenceStep:
    4. step_id: int
    5. prompt: list # Full conversation messages: system prompt, history, tool results
    6. output: str # Model output (text, tool-call arguments, or the final answer)
    7. tool_result: str # Content returned by the corresponding tool; determines the next step
    8. spawned_by: int # Parent step ID; -1 for the root task
    9. children: list # IDs of the spawned sub-agent steps
    10. tool_latency_ms: float
    11. token_usage: dict # Input/output token counts, whether a prefix cache hit occurred, etc.
    12. def replay(trace: list[InferenceStep], engine, rate_scale: float = 1.0):
    13. """Replay in trace order: steps of the same task must run serially; sub-agents may run in parallel."""
    14. for step in trace:
    15. engine.submit(
    16. messages=step.prompt,
    17. max_tokens=step.max_tokens,
    18. # Key point: submit the history prefix ID too, so the server can reuse the prefix cache
    19. cache_hint=step.prompt_prefix_id,
    20. gate=step.spawns or [], # Release only after the sub-agents finish
    21. )

    This structure looks simple, but it constrains three things at once: serial multi-turn execution, parallel sub-agents, and reusable prefixes. That is exactly the part of real workloads that synthetic requests struggle to reproduce.

    3. How Changes in Workload Shape Translate into System Pressure

    Understanding how the workload maps onto system pressure is a prerequisite for interpreting these benchmarks. It can be broken down on three levels.

    Level 1: Sequence length. Input length for an agent task accumulates as the task advances. Let L_i be the input length of turn i; the cumulative input for a single task is ∑L_i. As i grows, L_i grows roughly linearly, so the cumulative total grows roughly quadratically. A task with tens of interaction turns may therefore represent several times, or even an order of magnitude, more real computation than “just look at the last turn” would suggest. In practice, looking only at the last turn’s length severely underestimates the load. This is also why such benchmarks tend to use “per task” rather than “per request” as the unit of measurement.

    Level 2: Memory access and bandwidth. During decoding, each generated token typically requires reading the full KV cache. In long-context scenarios, memory traffic grows linearly with sequence length while compute barely increases, so decoding slides from being compute-bound toward being memory-bound. At that point, HBM bandwidth, cache hierarchy design, and interconnect bandwidth all show up directly in per-user output speed.

    Level 3: Concurrency and queueing. Sub-agent spawning makes request arrivals clearly bursty. At any given moment the system may hold dozens of inference streams of very different lengths and progress, all contending for the same memory and the same compute units. Three typical problems follow:

    • Memory contention: long-context requests occupy large cache space, which prevents the batch size from growing;
    • Long-tail latency: when long requests sit ahead of short ones, the wait for short requests is stretched considerably;
    • Scheduling thrashing: once preemption or recomputation is triggered, the prefix built up so far has to be rebuilt, which amounts to throwing away completed work.

    Because this pressure comes from the workload itself rather than from parameter settings, running the same synthetic benchmark on a machine with a different configuration can yield conclusions far from real business experience.

    4. Metric Design: From Peak Throughput to Goodput

    Trace-based benchmarks usually organize metrics into four groups. When designing your own evaluation, follow this order:

    (1) Service quality

    • Time to first token (TTFT): the time from request dispatch to the first output token. It determines how long a user waits after pressing Enter to see the first character, which matters especially in coding scenarios. Long prompts require long prefill, so TTFT degrades markedly as context grows.
    • Inter-token latency (ITL/TPOT): the average interval between generated tokens and its jitter (e.g., P99). It determines how smoothly the code scrolls out.
    • End-to-end task latency: total time from task start until all tool calls have completed and the final answer has been produced. It is the most direct reflection of developer experience.

    (2) Throughput and resource efficiency

    • Goodput: the number of valid tasks (or valid tokens) completed per unit time while latency constraints are met. It is the key metric that brings service-level objectives (SLOs) into account, and it is closer to a production conclusion than raw peak throughput.
    • Cache hit rate: the fraction of prefill saved by prefix-cache hits. In real traces this term often contributes enormously, which is why it marks the dividing line between a benchmark score and a real-world gain.
    • Memory and bandwidth utilization: KV-cache occupancy, memory bandwidth utilization, and similar figures, used to locate whether the bottleneck lies in compute or in memory access.

    (3) Task quality

    • Task completion rate: the share of target tasks completed correctly within a given number of turns or token budget.
    • Tool call success rate: the share of tool calls with valid arguments that the environment accepts. It is affected both by the model and by inference configuration, such as structured output constraints.
    • Retry and recovery rate: the share of steps re-executed because a tool returned an error or the environment behaved unexpectedly.

    (4) Economics

    • Cost per task: the compute resources and cost consumed to complete one valid task.
    • Cost per unit of goodput: cost per unit of goodput, used to compare different hardware and deployment options side by side.

    In real reports, put the constraints next to the metric. For example: “With P99 TTFT under X seconds and per-user output no lower than Y tokens/s, N valid tasks can be completed per hour.” Such a statement is far more persuasive than a bare peak number.

    5. What to Attribute: From Symptoms to Root Cause

    The value of an evaluation lies not in a single number but in explaining the reasons behind it. Under real trace workloads, performance differences often trace back to the following factors:

    1. Architecture and parallelism strategy. Different attention implementations (such as grouped-query attention, sliding-window attention, and linear attention) have very different KV-cache memory curves for long contexts. The memory-access pattern of the decoding path decides whether it leans “bandwidth-friendly” or “compute-friendly.”
    2. Load balancing in sparse expert models. In sparse expert architectures, routing distributions drift with task content. Expert activation patterns for coding tasks differ markedly from those for casual chat, and so do communication volume and hotspot distribution under expert parallelism.
    3. Caching strategy and scheduling. The eviction policy for prefix caches, whether cross-request reuse is allowed, and the chunk size of chunked prefill can all significantly change goodput in long-context scenarios.
    4. Communication and interconnect. Tensor parallelism, pipeline parallelism, and expert parallelism across multiple nodes and devices determine how much of the total time goes to cross-node communication. The high-bandwidth domain of a rack-scale form factor typically shrinks that share.
    5. Serving framework and configuration. Batch scheduling policy, memory reservation ratios, quantization schemes (precision choices for weights and KV cache), speculative decoding, and similar settings all shift the balance point between goodput and latency.

    A competent evaluation report should therefore contain two parts, a workload profile and an attribution analysis. The former states what was measured; the latter explains why the result is what it is.

    6. Common Pitfalls and How to Avoid Them

    • Pitfall 1: Using short prompts to stand in for long context. Short prompts measure peak compute; long context measures service capability. The conclusions can be opposite. Avoid this by making trace replay preserve the complete history.
    • Pitfall 2: Ignoring tool call latency. Tool execution itself can take far longer than model inference, and it affects perceived experience just as much. Record tool latency separately during evaluation so it is not misattributed to inference performance.
    • Pitfall 3: Looking only at averages. The long tail sets the floor of user experience. Always report P50/P90/P99, and pay attention to fairness across users.
    • Pitfall 4: Ignoring the “cheating” and “invalidation” effects of caching. If replay order diverges from real business traffic, cache hit rate becomes misleading. Conversely, if cache hit rate collapses because of scheduling policy, performance drops off a cliff. Both situations need coverage.
    • Pitfall 5: Treating one model as representative of all. Models with different architectures react differently to the same workload. A benchmark should cover multiple model sizes and architectures before its conclusions can be generalized.
    • Pitfall 6: Ignoring performance collapse under concurrency. Strong single-request performance does not mean the same holds under high concurrency. Measure within the target concurrency range, and watch for a turning point where throughput falls as concurrency rises.

    7. Implementation: Building Your Own Benchmark

    To apply this method to your own business, proceed as follows:

    1. Define the business scenario and trace source. Make clear whether you are evaluating code completion, repository-level modification, or defect fixing, and collect real or highly realistic traces.
    2. Establish a normalized trace format. Recommended fields: session ID, step number, parent step, full message list, tool name and arguments, tool result summary, token statistics, timestamps. Example structure:
    1. {
    2. "session_id": "task-2f81c0",
    3. "steps": [
    4. {
    5. "step_id": 3,
    6. "spawned_by": 1,
    7. "children": [7, 8],
    8. "messages": ["<system prompt>", "<user task>", "<tool result: grep>"],
    9. "tool": {"name": "run_tests", "latency_ms": 4210},
    10. "tokens": {"input": 61240, "output": 318, "cache_hit": true}
    11. }
    12. ]
    13. }
    1. Set service level objectives. Define TTFT and output-rate thresholds based on user experience, and spell out what counts as a valid task (for example, all tests pass and no regressive edits are introduced).
    2. Choose a replay strategy. Options include strict chronological replay (preserving original intervals), proportionally scaled replay (compressing intervals to simulate higher concurrency), and synthetic amplification replay (overlaying multiple real traces). Cover at least the first two.
    3. Cross-validate across models and configurations. For each candidate platform, run at least three sets: a short-context baseline, a medium-context set, and long context under high concurrency.
    4. Produce comparable reports. Unify metric definitions and measurement windows, and record hardware and software versions, so that results from different batches stay comparable.

    8. Platform-Side Capabilities

    Evaluation conclusions ultimately land on what the workload runs on. Beyond the accelerators and the inference framework themselves, a complete agent inference platform needs several general-purpose capabilities, and these should also be in scope of the evaluation:

    • Object storage: holds trace data, model weights, and evaluation artifacts; assess throughput and cold-start read performance.
    • Message queues: decouple replay scheduling from task orchestration and absorb bursty traffic.
    • Container platform and orchestration: elastic scaling, rolling upgrades, and self-healing for inference service replicas.
    • Logging, monitoring, and alerting: collect metrics such as TTFT, ITL, cache hit rate, and memory occupancy, and set threshold-based alerts.
    • Configuration and canary capabilities: support parallel validation of multiple inference configurations, making it easy to compare the effects of different scheduling policies.

    These components do not determine peak performance on their own, but they determine the sustainability and operability of performance. Under long-context, high-concurrency agent workloads, a poorly configured queue or missing observability is enough to degrade cluster performance sharply at peak hours.

    9. Conclusion

    Moving from single-request benchmarks to real trace benchmarks is, in essence, shifting the object of evaluation from hardware peak to business effectiveness. For developers, this means selection can no longer rely only on the peak compute figures quoted in marketing material. For enterprises, it means capacity planning must be based on a real workload profile. Readers are encouraged to use the metric framework in this article as a skeleton: first collect a batch of traces from your own business, then build a benchmark process that can run continuously, so that every capacity expansion and every model upgrade has evidence behind it.