How to Evaluate Local LLM Inference Speed Beyond Single Speed Anecdotes

    Single speed anecdotes—like "X seconds per token" from one machine—can't support conclusions about local LLM inference, because they omit the hardware, configuration, and metric behind the number. This article builds a reproducible, physics-based way to evaluate local inference speed. It starts from three basic quantities on a single machine (static weight memory, KV-cache growth, and effective memory bandwidth), explains the capacity gate and the bandwidth "memory wall" that set the speed ceiling, and covers special cases such as mixture-of-experts models and the shift from bandwidth-bound to compute-bound decoding under batching. It then offers a reproducibility checklist, a four-layer signal framework for reading the ecosystem from distribution to deployment, and a four-step evaluation process. The takeaway: a performance number that others can reproduce under the same configuration is the only number usable for decisions.

    1. Background: Why a Single Speed Anecdote Can’t Support a Conclusion

    Community discussions about a new generation of open-source LLMs have surfaced claims like these: on an ordinary low-memory machine, generating one token takes over twenty seconds; on a well-provisioned workstation with ample memory, it drops to around five seconds. Both scenarios run on the CPU, so at first glance the “it runs locally” conclusion seems settled. But this anecdote can’t answer three critical questions:

    • On what hardware does it run: The processor model and generation determine per-core compute, memory-controller specs, and instruction-set support, which in turn dictate which low-level optimizations an inference framework can use. Thread count affects parallel efficiency and involves hyper-threading, NUMA topology, and core-pinning strategy.
    • With what configuration: Weight quantization bit-width, activation quantization, KV-cache precision, context length, and parallelism all shift speed significantly. In an inference framework, settings such as thread count, memory mapping, number of inference cores, and batch size often affect results more than differences between models themselves.
    • Which metric is measured: Time-to-first-token (TTFT), inter-token interval, and throughput are different measurements. Changes in output length, concurrency, and input length can move the numbers by several times.

    An even more striking observation: on the same machine, as memory usage grows from a few GB to over a hundred GB, speed can change by nearly an order of magnitude. This looks less like linear scaling and more like a qualitative shift once a capacity constraint is lifted. For CPU inference scenarios, for example, single-concurrency generation speed across different memory-capacity bands often differs by 5–10x, and this is precisely why.

    If you cite such numbers directly for selection, capacity planning, or external conclusions, you can easily reach wrong judgments. A more reliable alternative is an explainable speed model: first use hard constraints to decide “can it run,” then use a bandwidth model to estimate “roughly how fast,” and finally use reproducible experiments to verify “how fast in practice.” We’ll start from the physical constraints of inference on a single machine, then return to interpreting ecosystem signals.

    2. What Determines Speed on a Single Machine

    2.1 Three Basic Quantities

    To evaluate local inference speed, you should first work out at least three quantities:

    1. Static memory footprint. Taking weights as an example:

      1. weight memory (GB) ≈ parameters (B) × bytes per parameter
      2. bytes per parameter = quantization bit-width / 8

      For instance, with 4-bit weights (ignoring metadata such as group zeros and scale factors), models of roughly 200B, 400B, and 700B parameters need about 100GB, 200GB, and 350GB respectively. A single CPU machine’s physical memory typically can’t hold the latter two.

    2. Dynamic memory footprint. The KV cache grows linearly with context length:

      1. KV cache (GB) ≈ 2 × layers × KV heads × dim per head × precision bytes × context length × concurrency / 1e9

      (The two factors of 2 correspond to Key and Value respectively.)

    3. Effective memory bandwidth. This determines the actual rate at which weights are read per token, and is affected jointly by the number and frequency of memory channels, whether it’s multi-socket, and core-pinning strategy.

    These three quantities determine which quadrant a model falls into: “fits / doesn’t fit” and “bandwidth-rich / bandwidth-constrained.”

    2.2 The Capacity Gate: The Cost of Not Fitting

    For CPU inference, capacity is a non-negotiable hard constraint. When weights and the KV cache don’t fit in physical memory, the excess must be swapped out to disk, or loaded on demand by a runtime that supports memory mapping and disk offloading. During single-request, single-batch decoding, every output token requires reading the full set of weights that participate in computation. Once that portion of weights isn’t in memory, decoding gets “locked” to storage-device bandwidth.

    This explains the abnormally slow low-memory scenario in the community anecdote: what actually matters is paging and disk reads, not processor compute. Such configurations aren’t suitable as a “performance conclusion”; they’re better used to illustrate an availability boundary—“is this model usable on an ordinary machine?”

    2.3 The Bandwidth Bottleneck: The Speed Ceiling Once It Fits

    When weights reside entirely in memory and the batch size is 1, the time to generate one token is approximately:

    1. t ≈ (weight bytes × (1 + metadata overhead) + KV-cache read for the current step) / effective bandwidth

    That is, inter-token interval ≈ bytes read per forward pass ÷ effective bandwidth. This is what’s often called the “memory wall.”

    An illustrative calculation follows; substitute parameters for your actual hardware:

    Parameters Quant bits Read per token 10 GB/s 25 GB/s 50 GB/s
    7B 8-bit ≈7.1GB ≈1.4s ≈0.57s ≈0.28s
    70B 8-bit ≈71GB ≈14s ≈5.7s ≈2.8s
    70B 4-bit ≈35.5GB ≈7.1s ≈2.8s ≈1.4s

    This table yields two important conclusions:

    • Quantization reduces time roughly linearly: halve the bit-width, halve the bytes read per token, and time roughly halves too (as long as no other bottleneck kicks in).
    • Scenario differences can reach an order of magnitude: for the same “CPU inference,” machines with 10 GB/s versus 50 GB/s bandwidth can differ by 5x in speed.

    This also shows why treating one machine’s numbers as “the model’s speed” is meaningless.

    1. def est_ttft_decode(weight_gb, kv_read_gb, eff_bw_gbs,
    2. prefill_flops=0, cpu_flops=0):
    3. """
    4. Estimate inter-token interval from the bandwidth bottleneck and
    5. time-to-first-token from compute. Fill in every argument from
    6. measurements or published specs; results are for order-of-magnitude checks.
    7. """
    8. bytes_per_token = (weight_gb + kv_read_gb) * 1e9
    9. tpot_ms = bytes_per_token / (eff_bw_gbs * 1e9) * 1000
    10. ttft_ms = (prefill_flops / cpu_flops) * 1000 if cpu_flops else None
    11. return ttft_ms, tpot_ms

    Note that this model only approximates reality when it’s a “single request, weights resident in memory, batch size near 1.” Once multi-concurrency batching is enabled, the decode phase gradually shifts from bandwidth-bound to compute-bound, and you must validate with real measurements.

    2.3.1 Mixture-of-Experts Models: Distinguish Total from Active Parameters

    The mixture-of-experts (MoE) architecture decouples parameter count from compute: total parameters are large, but each token only activates a subset of experts.

    • Capacity is counted by total parameters: To maintain routing, an inference framework usually needs the selected experts’ weights resident in memory. If expert offloading is used, then when a hit expert isn’t in memory, it must be swapped in and out temporarily over a high-speed interconnect.
    • Speed is estimated by active parameters: Applying the “read all weights per token” formula directly to a sparsely activated model will overestimate the inter-token interval.

    When evaluating, it’s advisable to record “total parameters / active parameters / expert parallelism” together.

    2.4 The Compute Bottleneck: Where Batch Size Comes From

    In single-request decoding, the bottleneck is bandwidth; batching can multiply throughput because a single forward pass turns multiple requests’ matrix–vector products into matrix–matrix products, raising arithmetic intensity. Deciding that hardware “fits and is fast enough” is only the starting point—what really determines the throughput ceiling is concurrency:

    • KV-cache budget: after subtracting weights and runtime overhead from available memory, the remaining space determines the product of concurrency ceiling and usable context length;
    • Compute ceiling: the larger the batch, the closer you get to the upper bound of “tokens processed per second = peak compute ÷ compute per token”;
    • Scheduling strategy: capabilities such as continuous batching, paged caching, and speculative decoding determine whether latency stays controllable under high concurrency.

    2.5 Reproducibility Checklist

    A reproducible speed record should contain at least the following fields; none can be missing:

    1. {
    2. "hardware": {"cpu_model": "", "sockets": 1, "cores": 0, "threads": 0,
    3. "ram_gb": 0, "memory_channels": 0, "memory_speed_mts": 0},
    4. "model": {"params_total": "0B", "params_active": "0B",
    5. "quant": {"weight_bits": 4, "act_bits": 0, "kv_bits": 8, "group_size": 128}},
    6. "runtime": {"engine": "", "version": "", "threads": 0,
    7. "batch_size": 1, "context_len": 4096, "backend": "cpu"},
    8. "bench": {"input_len": 128, "output_len": 128, "runs": 5},
    9. "metrics": {"ttft_ms": 0, "tpot_ms": 0, "throughput_tok_s": 0, "ram_peak_gb": 0}
    10. }

    In practice, you can organize these fields into a “minimum information table” and have every speed post in the community fill it out. Any missing item is marked directly as “not reproducible” and excluded from the evidence pool.

    3. A Framework for Reading Signals: From Distribution to Deployment

    In the context of day-to-day tracking, observation of the model ecosystem can be broken into four categories of signals, each answering a different question:

    Signal layer What you observe Question answered Typical lag
    Infrastructure signal Support status in inference engines and runtimes Can it run efficiently Weeks–months
    Application-integration signal Adaptation in toolchains and dev frameworks Can it be integrated into the business Weeks–months
    User-adoption signal Downloads, active usage, API calls, spend How many people use it Days–weeks
    Production-stability signal Error rate, latency, cutovers and rollbacks Can it carry the business long-term Months–quarters

    3.1 Infrastructure: First Ask “Is the Backend Ready?”

    To observe whether a model can be served efficiently on mainstream inference runtimes, you can grade by status:

    • No backend support: requests fail outright due to device-type mismatch; it can’t run;
    • Only a community fork or plugin: you must compile or patch it yourself, and version upgrades easily conflict;
    • Merged into an official branch but not released: you need to pin a commit or branch, and stability is unknown;
    • Stably released and in the default configuration: it can be deployed directly and has production readiness.

    The key here: support status is more trustworthy than marketing claims. A PR that exists but isn’t merged, or a feature covering only some hardware or parallelism strategies, should be treated as “not ready.” When tracking daily, you can treat “status transitions” as the most informative metric.

    3.2 Application Integration: The Model Pool and Fault Tolerance of Coding Assistants

    Take AI coding assistants as an example: such products often offer a selectable model pool and have capabilities like retry on failure and timeout fallback. To judge whether a new model has truly entered their production path, you can observe:

    1. Whether a dedicated model entry appears: if only the general model pool expands with no dedicated entry, the adaptation is still at the framework level;
    2. Whether a dated adaptation announcement exists: version numbers and dates in the changelog are more reliable than a general “we support model X” statement;
    3. Whether reproducible evaluations exist: for example, pass rate, latency, and cost comparisons on the same set of coding tasks.

    When all three are missing, you can only record it as “potential adaptation,” not “adopted.”

    3.3 User Adoption: Activity, Spend, and Relative Share

    Downloads and call volumes are leading indicators of distribution, but watch for two traps:

    • A rise in absolute value doesn’t mean a rise in share: when the whole market grows, any participant can “climb steadily”; you need relative share or penetration;
    • No baseline comparison is meaningless: if a tool page shows “requires a high-tier plan” but provides no daily-usage data, it can’t serve as adoption evidence.

    3.4 Production Stability: Error Rate and Switching Cost

    Real production adoption shows up in stability metrics:

    • day-to-day fluctuation in server error and timeout rates;
    • how latency changes as context length, concurrency, and input length grow;
    • whether repeated “launch–rollback” cycles occur; the cutover and rollback themselves are also observable signals.

    4. An Actionable Evaluation Process

    Turning the framework above into an executable process can be done in four steps:

    1. Build an evidence pool: every piece of information must carry reproducibility fields; items missing fields are marked “not reproducible” to avoid second-hand accounts contaminating conclusions.
    2. Weight by layer: infrastructure and application-integration status carry more weight than community posts, and weights can be adjusted by the formula below.

      1. WEIGHTS = {"infra_state": 0.35, "integration": 0.2,
      2. "user_signals": 0.3, "stability": 0.15}
    3. Decide by stage:

      | Stage | Characteristics | Typical action |
      |—-|—-|—-|
      | Distribution-spread phase | Downloads rise, support status sporadic, no stable production cases | Run a POC, canary at low traffic |
      | Deployment-accumulation phase | Backend reaches stable release, toolchain adaptation appears, production cases grow but share unchanged | Expand the canary with risk control, prepare a rollback plan |
      | Production-explosion phase | Error rate stable, cost predictable, switching cost falling | Large-scale replacement and capacity planning |

    4. Review weekly: prioritize status transitions over numerical fluctuation, so you don’t get pulled around by single-day noise.

    5. Conclusion and Practical Recommendations

    Back to the community speed anecdote at the start: about 26.5 s/token under 8GB of memory and about 5.6 s/token under 128GB. The most valuable thing about these two numbers isn’t the absolute speed, but the capacity–speed relationship they expose. Lacking the processor model, thread count, and quantization info, they can only show “the low-memory config is unusable and the high-memory config is barely usable”—not enough to support a side-by-side comparison.

    The same logic applies to ecosystem observation. When an inference engine’s backend is still empty, when the tooling side has no dated adaptation announcement, and when hands-on benchmark posts on social platforms lack complete logs, the more reasonable conclusion is: distribution is expanding, deployment is accumulating, and production adoption hasn’t yet crossed the tipping point. To speed up this process, teams can focus on three things:

    • Fill in reproducible experiments: use public hardware specs and configuration fields to produce benchmark reports with complete logs;
    • Drive backend and toolchain adaptation: push “not merged” to “stable release,” and “general pool” to “dedicated entry”;
    • Build monitoring and canary systems: with common cloud capabilities such as container platforms, logging services, and monitoring/alerting, bring error rate, latency, and cost into daily observation, replacing rumor with observability.

    For developers, the most practical judgment criterion is a single sentence: a performance number that others can reproduce under the same configuration is the only number usable for decisions.