Running 27B LLMs Locally on Old Hardware: VRAM and Speed Tuning in Practice
This article is a hands-on guide to running 27B-class large language models on aging hardware. It starts from a reproducible single-machine case, then breaks down the methodology: how prefill and decode stress the system differently, how to build a VRAM budget, how quantization and KV cache quantization affect both memory and speed, how long contexts raise real costs, how speculative decoding works, and how engine choice and tuning can change performance by several times. It also includes a six-step workflow, sample acceptance thresholds, an environment-recording table, and a benchmark script, so you can measure and tune your own local setup rather than trusting paper numbers.
1. Introduction: Running 27B on Old Hardware Is About Tuning, Not Buying New
Can old GPUs, old CPUs, and mechanical hard drives still run 27B-class models locally? Yes—but only if you understand where the bottlenecks actually are. Local inference isn’t fundamentally about whether your GPU is new; it’s about how well you allocate a limited pool of VRAM capacity, VRAM bandwidth, memory bandwidth, and CPU capability to the right places.
What do you do when VRAM isn’t enough? How do you choose between 4-bit and 6-bit? How do you speed up long contexts? Will a mechanical hard drive slow down inference? These questions come up constantly in local deployments. Let’s start with one reproducible case, then unpack the methodology behind it.
In one local deployment, an older machine delivered a usable 27B-class experience:
- Whole system: an older server-grade CPU, a second-hand motherboard, second-hand DDR4 memory, with the model files stored on a second-hand mechanical hard drive;
- GPU: a single second-hand card with a non-standard capacity design, roughly 22GB of VRAM;
- Model: a 4-bit quantized build from the community’s many quantization variants, about 14.2GB in size;
- KV cache: 4-bit quantized, with a context of roughly 240K tokens and about 21GB of VRAM usage;
- Inference config: multi-token prediction (i.e., speculative decoding) enabled with a draft count of 3, holding a steady output speed of about 30 tokens/s; after switching inference engines, generated speed in some scenarios’ logs exceeded 40 tokens/s.
That machine then went through three further changes:
- Platform downgrade: memory was swapped from 40GB DDR4 to 128GB DDR3, and the main model was switched to a Q4 build from a lightweight model family. At 512K context, peak speed was about 21 tokens/s; in coding tasks, thinking speed was about 10–13 tokens/s and output speed about 15–18 tokens/s; the same model held a steady 30–40 tokens/s in long tasks.
- Main-model switch: after cloud lightweight model prices dropped, demand for 27B-class models weakened, so the main model was switched to a 35B-class MoE model with about 3B activated parameters, raising generation speed to roughly 80–100 tokens/s.
- Engine upgrade: after moving to an inference engine with faster cold starts, flash-class models paired with 1M-token context and vision input could also run, and the 27B model was essentially “retired.”
Emphasis: the numbers above are illustrative data from a single case, not a universal performance guarantee. Because the exact second-hand GPU model, driver version, and inference engine version were not fully disclosed, anyone reproducing this should record at least the GPU model, VRAM capacity, driver version, inference engine version, quantization format, KV cache bit width, context length, batch size, temperature, top-p, and max_tokens. Otherwise, the same claim of “it runs” can differ by several times across configurations.
But it reveals one key point: the core of local inference isn’t how old the CPU is or how worn the memory is, but whether VRAM bandwidth, quantization strategy, KV cache management, and engine capability match your workload.
2. Performance Breakdown: Prefill and Decode Are Two Different Animals
A single local inference request typically goes through two phases: first prefill the input prompt, then decode token by token. Their bottlenecks are completely different. If you evaluate inference speed with a single metric, it’s easy to reach the wrong conclusion.
2.1 Prefill: Compute-Bound, Determines Time to First Token
Prefill pushes all input tokens through the model at once. The matrix operations are large and have high arithmetic intensity, so this phase is mainly compute-bound and usually achieves high hardware utilization on the GPU.
- Its time roughly scales with input length;
- Unoptimized attention implementations may grow superlinearly with length;
- It directly determines time to first token (TTFT)—the wait from submitting a question to seeing the first character;
- The 240K context in the reference config could alone cause several seconds or more of waiting on prefill;
- If the model files live on a mechanical hard drive, weight loading during cold start further lengthens the first response.
2.2 Decode: VRAM-Bandwidth-Bound, Determines tokens/s
In the decode phase, only one token is generated per step. Each step must move all the weights and the KV cache for the current context from VRAM into the compute units, while the actual computation is relatively small. Decode speed is therefore mainly limited by VRAM bandwidth. For a single user, you can approximate it as:
theoretical tokens/s ≈ effective VRAM bandwidth ÷ (weight bytes read per token + KV cache bytes read per token)
Take a card with a nominal VRAM bandwidth of about 600GB/s. Actual effective utilization is usually estimated at 60%–70%, i.e., about 360–420GB/s of effective bandwidth.
- Reading 14.2GB of weights gives a pure-weight theoretical ceiling of about 25–30 tokens/s;
- Measured results reaching 30 tokens/s or beyond usually come from two contributions: first, engine optimizations to weight layout, memory access patterns, and attention kernels; second, speculative decoding, which lets a single forward pass verify and produce multiple tokens;
- The longer the context and the larger the KV cache, the lower per-request decode speed typically falls below the pure-weight estimate.
Here it’s important to distinguish single-request latency from total throughput. A higher batch usually raises GPU utilization, boosting total throughput for multiple requests; but for a single request, a higher batch can introduce queuing, scheduling, and batching delays, lowering the per-user tokens/s and lengthening the wait for the first token. For low-latency single-user scenarios, prioritize batch=1 or small concurrency; for multi-user serving, balance throughput against latency.
2.3 The Third Factor: Hidden Overhead
Beyond prefill and decode, there are three kinds of overhead that are easy to overlook:
- Cold start: weight loading, VRAM initialization, engine initialization—all affecting the first response;
- Sampling and post-processing: decoding strategy, temperature sampling, structured-output constraints, tokenization, and incremental decoding all consume CPU time;
- Memory copies: moving weights and KV data between CPU and GPU; some engines may repeatedly copy during generation, becoming a hidden bottleneck in long tasks.
So evaluation can’t look at tokens/s alone—also watch TTFT, long-context degradation, VRAM peak, and cold-start time.
3. VRAM Budget: The First Constraint in Local Deployment
3.1 The “Four Slices” of VRAM
The VRAM on a card is roughly divided among four parts:
total VRAM ≈ model weights + KV cache + runtime buffers + reserved headroom
- Model weights: determined by parameter count and quantization bit width, essentially fixed;
- KV cache: grows with context length, batch size, and concurrent requests;
- Runtime buffers: activations, intermediate attention results, sampling buffers, etc., fluctuating with batch and length;
- Reserved headroom: typically 10%–15% of the total, to absorb peak fluctuations and avoid VRAM overflow that would fail requests.
The combination in the case—“weights ~14.2GB + KV ~5GB + runtime overhead, total ~21GB”—is the result of budgeting, not arbitrary parameter fiddling.
3.2 KV Cache VRAM Overhead Can Be Estimated
Mainstream models commonly use grouped-query attention and similar structures. KV cache usage can be estimated as:
KV bytes = 2 × layers × KV heads × head dim × bytes per element × context length × batch size
The factor of 2 accounts for the two caches: Key and Value. Below is a set of assumed 27B-class parameters for illustration: 44 layers, 4 KV heads, head dimension 128, and 0.5 bytes per element (i.e., 4-bit quantization).
def kv_cache_gb(ctx_len, batch=1, layers=44, kv_heads=4, head_dim=128, bytes_per_elem=0.5):"""Estimate KV cache VRAM usage; bytes_per_elem=0.5 means 4-bit quantization."""per_token = 2 * layers * kv_heads * head_dim * bytes_per_elemreturn per_token * ctx_len * batch / 1024 ** 3print(kv_cache_gb(240_000)) # ≈5.04 GBprint(kv_cache_gb(512_000)) # ≈10.75 GBprint(kv_cache_gb(1_000_000)) # ≈20.98 GB; on top of weights, over budgetprint(kv_cache_gb(32_000, batch=1)) # ≈0.67 GBprint(kv_cache_gb(32_000, batch=4)) # ≈2.69 GB; concurrency increases usage sharply
Compare this with the unquantized FP16 case, at 2 bytes per element. A 240K context would need more than 20GB of KV alone, which already clearly exceeds a 12GB card’s total VRAM, let alone the effective VRAM available for weights. This shows KV quantization isn’t optional—it’s a prerequisite for running 27B-class models with long contexts on consumer or mid-range cards.
3.3 Budget First, Then Act
Use a “budget first, load later” workflow:
- Look up the weight size for each quantization tier;
- Plug into the formula to estimate KV usage at the target context;
- Add 10%–15% headroom and confirm the total stays under VRAM capacity;
- After startup, observe actual usage and then tune context or concurrency back down.
You can fill in this VRAM budget table item by item:
| Item | Estimation method | Current value | Notes |
|---|---|---|---|
| Model weights | Read the actual size of the quantized file | e.g., ~14.2GB for 4-bit | |
| KV cache | Estimate with the KV formula | affected by context, batch | |
| Runtime buffers | Reserve based on engine experience | activations, sampling, attention intermediates | |
| Headroom | 10%–15% of total VRAM | guards against peak overflow | |
| Conclusion | Are weights + KV + runtime + headroom below VRAM? | only load if it fits |
4. Quantization: A Lever for Saving VRAM and a Lever for Speed
Quantization maps weights and activations from high-precision floats to low-bit integers, and reconstructs approximate values using a set of scale factors and zero points. For example, mapping a group’s maximum absolute value to the 4-bit integer ceiling gives the scale factor for that group. Other common techniques include allocating bits by importance and keeping outlier channels at higher precision.
4.1 Bit Width Determines Both VRAM and Speed
Every halving of bit width roughly halves the amount of weights read. So quantization saves VRAM and directly improves decode speed. Picking a file from community Q3, Q4, Q6 builds is essentially searching for the “size, quality, speed” balance point within the same model family.
| Quantization tier | Weight size (27B-class, illustrative) | Pure-weight theoretical speed* | Typical use case |
|---|---|---|---|
| 8-bit | ~27GB | ~13–16 tokens/s | Ample VRAM, prioritize output quality |
| 6-bit | ~20.5GB | ~18–21 tokens/s | Quality-first tier for 24GB-class cards |
| 4-bit (with a few high-precision layers) | ~14.2GB | ~25–30 tokens/s | Mainstream value tier for long context on 22GB cards |
| 3-bit | ~10.7GB | ~34–39 tokens/s | Squeeze out longer context at the same VRAM |
| 2.5-bit | ~9GB | ~40–47 tokens/s | Noticeable quality loss; generally not recommended |
*Estimated at a nominal 600GB/s bandwidth and 60%–70% effective utilization, i.e., ~360–420GB/s; does not account for KV cache reads, speculative decoding gains, or engine differences. Actual speed may exceed this pure-weight ceiling, or drop due to long context, concurrency, CPU offload, or batching scheduling. Engines and quantization formats differ widely in VRAM alignment, grouping granularity, and protection of key layers; the table’s numbers are for order-of-magnitude reference only.
4.2 Lower Bit Width Isn’t Always Better
- Smaller models are harder to quantize: models with fewer parameters have less room to tolerate error, so quality tends to degrade more at the same bit width; models in the tens-of-billions range often hold up better at the same width.
- The hard parts usually concentrate in long-chain reasoning: simple Q&A is insensitive to quantization error, while code generation, multi-step reasoning, and long-context information integration are more sensitive.
- Quality loss should be measured: prefer a task set that matches your business—for example, code-completion pass rate, long-document QA accuracy, instruction-following pass rate—rather than looking at perplexity alone.
- How to choose between 4-bit and 6-bit: if VRAM can only just fit 4-bit, prefer a mixed-precision 4-bit build that keeps key layers at higher precision; if 6-bit still fits the budget and long-chain reasoning quality matters more, 6-bit is usually more stable.
5. Context: The Longer You Stack It, the More Real the Cost
5.1 Context Costs More Than VRAM
At decode time, generating each token theoretically requires reading the KV cache for the current context. The longer the context, the more data must be moved per token, and the slower the speed. Take 4-bit KV, 14.2GB weights, and 360–420GB/s effective bandwidth for a rough estimate:
| Context length | KV usage (4-bit) | Rough theoretical ceiling* |
|---|---|---|
| 10K | ~0.21GB | ~25–29 tokens/s |
| 100K | ~2.10GB | ~22–26 tokens/s |
| 240K | ~5.04GB | ~19–22 tokens/s |
| 512K | ~10.75GB | ~14–17 tokens/s |
*Estimated from 14.2GB weights plus the current context’s KV, not accounting for speculative decoding. Actual speed may exceed these values, or drop due to engine, batching, sampling, post-processing, and CPU offload overhead.
One easily misunderstood point needs explaining: the reference config’s peak of about 21 tokens/s at 512K context doesn’t contradict the same model holding a steady 30–40 tokens/s in long tasks—but the conditions must be met.
About 21 tokens/s at 512K context usually reflects a reference state where the nominal context is nearly full, each step reads a large KV, and there’s no extra cache reuse. A steady 30–40 tokens/s in long tasks usually comes from one or more of the following:
- The actual effective context is far smaller than the nominal context; the system prompt and dialogue history are relatively stable;
- Prefix cache hit rate is high, reducing repeated prefill;
- The engine uses paged management, efficient attention, or sparse attention, lowering the actual amount read;
- The speculative decoding acceptance rate is high, producing multiple valid tokens per forward pass;
- The input prompt doesn’t continuously approach 512K, or the context is retrieved, truncated, or summarized/compressed;
- Single-request scenarios with a small batch and no multi-user queuing.
If each step fully reads the 512K KV with no cache, compression, sparseness, or speculative-decoding gains, the speed can’t magically return to 30–40 tokens/s. So long-context performance can’t be judged by nominal max length alone—effective length, caching strategy, and engine capability matter too.
5.2 TTFT Is Highly Correlated with Input Length
The reference config’s thinking speed of about 10–13 tokens/s and output speed of about 15–18 tokens/s in coding tasks is lower than the 30–40 tokens/s in long tasks. Common reasons:
- Coding-task inputs include long code context, raising KV usage and lowering decode speed;
- The “thinking” phase is often a long output, so the overall experience depends more on steady decode speed;
- In long tasks, the system prompt and dialogue history are relatively stable, making cache hits more likely and speed higher;
- Coding tasks may involve more structured constraints, tool calls, or complex prompts, adding prefill and post-processing overhead.
5.3 Context Management Strategies
- Allocate context by scenario: keep only the last few turns for chit-chat requests; for long-document QA, prefer injecting relevant snippets via retrieval rather than stuffing in the whole document;
- Prefix caching: unchanged system prompts, tool descriptions, and project notes across multiple turns can be cached and reused, significantly reducing repeated prefill overhead;
- Prefix cache invalidation conditions: the prefix cache may become invalid and need recomputation when the system prompt changes, tool definitions change, the role template changes, the tokenizer changes, the model or quantization version switches, KV pages are evicted, the engine restarts, the context is truncated, or a new prefix is inserted;
- KV reuse and paged management: requests sharing the same prefix can reuse KV, and paged management reduces VRAM fragmentation;
- Structural choice: models with fewer KV heads have lower KV overhead at the same layer count, head dimension, and bit width, making them better suited to long context;
- Capability boundaries: nominal context length doesn’t equal effective context; when key information is buried in the middle of an ultra-long context, model recall ability may degrade;
- Multi-user isolation: in multi-tenant scenarios, prefix caching and KV reuse must enforce tenant isolation and permission checks. The minimum checklist includes tenant-ID binding, cache keys that include a tenant dimension, key or token authentication, expiry cleanup, access auditing, and a ban on cross-tenant reuse of sensitive prefixes.
6. Speculative Decoding: A Small Model Drafts, the Big Model Grades
The idea behind speculative decoding is to use a lightweight “draft” source to guess several tokens in a row, then have the target model verify those guesses in a single parallel forward pass and accept the longest correct prefix verbatim. Common implementations include:
- A small model from the same family as the draft model;
- An extra prediction head trained into the model itself, i.e., multi-token prediction;
- Model-free drafting based on rules such as n-grams.
With a correct implementation, if the verification logic is right, speculative decoding usually only affects speed and doesn’t change the final output distribution. But in real engineering, different engines, sampling parameters, draft lengths, acceptance-rate thresholds, and caching strategies can produce different felt experiences. For example, templated, highly repetitive text has a high acceptance rate; code generation, creative writing, and high-temperature sampling usually have lower acceptance rates.
The reference config’s “MTP=3, steady 30 tokens/s” is multi-token prediction with a draft length of 3 at work. The gain from speculative decoding is determined by acceptance rate: the more draft hits, the lower the average cost per token.
Also, the draft model itself occupies VRAM and consumes compute, so it must be factored into the VRAM budget. Practical advice: use your target business’s real prompts, compare end-to-end speed across draft lengths 1–5, and pick the balance point between acceptance rate and overhead.
7. Engines and Tuning: Same Card, Performance Can Differ Several Times
7.1 The Engine Sets the Performance “Foundation”
With the same hardware, same model, and same quantization, switching inference engines can produce a multiple-fold difference in speed. The difference usually comes from these capabilities:
| Capability | Effect | Tuning tip |
|---|---|---|
| Mixed-precision weight format | Higher bits for key layers, lower bits for the rest | Balance speed and quality within the VRAM budget |
| Efficient attention kernels | Fewer VRAM reads/writes, lower long-context KV overhead | Compare kernels’ behavior under long context |
| CPU/GPU hybrid inference | Offload layers that don’t fit in VRAM to memory, computed on CPU | Tune the GPU layer ratio; prioritize keeping key layers |
| Cold-start optimization | Memory-mapped loading, lazy initialization, on-demand KV allocation | Watch cold-start time, not just tokens/s |
| Speculative decoding | Lower the average cost per token | Tune draft length by task type |
| Prefix caching | Reuse unchanged-prefix KV | Isolate and permission-check in multi-user scenarios |
| Quantized KV and paged management | Reduce long-context VRAM pressure | Trade off context length, concurrency, and quality |
| Built-in serving layer | Stream output over HTTP | Easier integration, load testing, and business integration |
7.2 Cold Start: Determines the “First” Experience
Storing the model files on a mechanical hard drive mainly affects the cold-start phase:
- If the engine loads weights via memory mapping, the first load must read the data from disk into memory;
- On subsequent launches, if memory is ample and the file still resides in the OS page cache, load speed improves markedly;
- Some engines preallocate a large block of VRAM or fully load the weights at startup, and cold-start time can differ by several times.
If the inference node also serves online traffic, focus on cold-start time. Common practices are a resident process, warm instances, and weight pre-warming.
7.3 Hybrid Inference: No Simple Formula for the CPU/GPU Ratio
When VRAM is insufficient, you can place only some layers on the GPU. CPU/GPU hybrid inference speed is affected jointly by layer count, memory bandwidth, operator efficiency, and data-transfer overhead, so it can’t be estimated with a simple “CPU layers divided by GPU layers” ratio.
A reference tuning order:
- First confirm with a small context whether the model can load fully into VRAM;
- Then gradually increase context, finding the balance point between speed and VRAM;
- If offloading is necessary, keep key layers—attention layers, embedding layers—on the GPU, and put the relatively independent feed-forward layers on the CPU;
- Confirm the CPU core count and whether memory channels are correctly enabled;
- Watch for frequent CPU-to-GPU data movement, avoiding hidden copies that slow down long tasks.
7.4 Server Parameters: Don’t Let “Concurrency” Drag Down Single-User Speed
Batch inference can significantly boost total throughput, but single-user requests get hit with “batch-stacking” delay, showing up as lower tokens/s and longer waits for the first token. For local single-user scenarios, prioritize low latency; only raise batch size moderately when multiple users share a service.
If your business is code completion, interactive Q&A, or Agent tool calls, TTFT and steady decode speed usually matter more than peak throughput; if it’s offline batch processing, bulk summarization, or batch evaluation, prioritize throughput and accept higher latency.
8. An Actionable Methodology: From Budget to Acceptance
8.1 The Six-Step Method
- Define the goal: pick the main scenario—code completion, long-document QA, Agent tool calls—and set acceptable TTFT, generation speed, and context length;
- Build the VRAM budget: compute “weights + KV + runtime + headroom” item by item, using a script to estimate when needed;
- Choose a quantization tier: start from 4-bit, and when VRAM allows, prefer a mixed-precision format that keeps key layers at higher precision;
- Tune the engine and parameters: compare multiple engines’ cold start, decode speed, and long-context behavior, set the CPU/GPU layer ratio, and enable speculative decoding and cache reuse;
- Load-test and accept: use a unified script to record metrics across context lengths and concurrency levels;
- Route by tier: when a local model can’t handle it, route complex tasks to a cloud large model.
8.2 Sample Acceptance Thresholds
Different businesses have different requirements, but you can start with a reference set of thresholds. These need to be calibrated against GPU bandwidth, quantization tier, concurrency, model architecture, and context strategy. For example, at the same 4-bit quantization, a 6-bit tier is usually slower; at the same 32K context, a model with more KV heads is usually slower; for the same model, thresholds can be more aggressive once speculative decoding and prefix caching are enabled.
| Metric | Reference passing line |
|---|---|
| Cold-start time | Under 60s; shorter with page cache or a resident process |
| TTFT @32K | Under 3s |
| Decode speed @32K | Over 20 tokens/s |
| Decode speed @128K | Over 15 tokens/s |
| VRAM peak | Not exceeding 90% of card VRAM |
| Error rate | Under 1% |
These thresholds aren’t absolute—they help a team establish an acceptance baseline. The closer the business is to real-time interaction, the more it should focus on TTFT; the closer to batch processing, the more it should focus on throughput and VRAM peak.
8.3 Measured-Environment Record Table
To avoid “it runs on my side but not yours,” record at least the following:
| Config item | Example |
|---|---|
| GPU model | Second-hand non-standard-capacity card |
| VRAM capacity | ~22GB |
| Driver version | Record the specific version |
| Inference engine | Record name and version |
| Quantization format | e.g., community 4-bit quantization |
| KV quantization | 4-bit |
| Context length | 240K |
| batch | 1 or 4 |
| temperature | e.g., 0.7 |
| top-p | e.g., 0.9 |
| max_tokens | e.g., 256 |
8.4 Benchmark Script
The script below uses a common streaming HTTP interface for local inference services to separately record TTFT and decode speed. It doesn’t mistake the number of streamed chunks for tokens; it prefers the server’s returned usage field, and if there’s no usage, it counts the generated text with a tokenizer matching the model. The prompt is also built or truncated to the target context via the tokenizer, and the actual token count is printed.
import jsonimport timeimport requestsfrom transformers import AutoTokenizerBASE_URL = "http://127.0.0.1:8080/v1" # Local inference service URL, loopback-onlyMODEL = "local-27b-q4" # Model identifier for the local deploymentTOKENIZER_PATH = "path/to/model" # Tokenizer directory matching the modeltokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH)def count_tokens(text: str) -> int:return len(tokenizer.encode(text, add_special_tokens=False))def build_prompt(target_tokens: int) -> str:prefix = "Read the material below and summarize the topic.\n"unit = "Local inference performance test. "text = prefixwhile count_tokens(text) < target_tokens:text += unit + "\n"ids = tokenizer.encode(text, add_special_tokens=False)[:target_tokens]suffix = "\nSummarize the topic of the material above in no more than 100 words."return tokenizer.decode(ids, skip_special_tokens=True) + suffixdef bench(target_ctx_tokens: int = 32000, max_tokens: int = 256):prompt = build_prompt(target_ctx_tokens)prompt_tokens = count_tokens(prompt)payload = {"model": MODEL,"messages": [{"role": "user", "content": prompt}],"max_tokens": max_tokens,"stream": True,}start = time.perf_counter()ttft = Nonetext_parts = []usage = Nonechunk_count = 0with requests.post(f"{BASE_URL}/chat/completions",json=payload,stream=True,timeout=600) as resp:resp.raise_for_status()for raw in resp.iter_lines():if not raw or not raw.startswith(b"data:"):continuedata = raw[5:].strip()if data == b"[DONE]":breakobj = json.loads(data)if isinstance(obj, dict) and obj.get("usage"):usage = obj["usage"]choices = obj.get("choices", [])if not choices:continuedelta = choices[0].get("delta", {})content = delta.get("content")if content:if ttft is None:ttft = time.perf_counter() - starttext_parts.append(content)chunk_count += 1elapsed = time.perf_counter() - startgenerated_text = "".join(text_parts)if usage and usage.get("completion_tokens") is not None:decode_tokens = usage["completion_tokens"]token_source = "usage"else:decode_tokens = count_tokens(generated_text)token_source = "tokenizer"decode_seconds = elapsed - ttft if ttft is not None else Nonedecode_tps = decode_tokens / decode_seconds if decode_seconds else Nonereturn {"target_ctx_tokens": target_ctx_tokens,"actual_prompt_tokens": prompt_tokens,"ttft_s": round(ttft, 3) if ttft else None,"decode_tps": round(decode_tps, 1) if decode_tps else None,"decode_tokens": decode_tokens,"token_source": token_source,"chunk_count": chunk_count,}if __name__ == "__main__":for ctx in (1_000, 32_000, 128_000):print(f"Target context ~{ctx} tokens:", bench(ctx))
Before running the script, install the dependencies:
pip install requests transformers
And make sure the local inference service is running. The service should listen only on the loopback address and be reachable only from the local machine; if you need to expose it externally, don’t expose a bare port directly. It’s recommended to enable HTTPS, API-key or OAuth2 authentication, rate limiting, an IP allowlist, access auditing, and timeout control via Nginx, Caddy, or another reverse proxy, and to apply basic safety filtering to model inputs.
Run each configuration 3–5 times and record these metrics:
| Config item | Cold start (s) | TTFT@32k (s) | Decode speed @32k | Decode speed @128k | VRAM peak (GB) |
|---|---|---|---|---|---|
| Engine A / 4-bit / no speculation | |||||
| Engine A / 4-bit / speculation | |||||
| Engine B / 4-bit / speculation |
9. Common Misconceptions
- Looking only at tokens/s: ignoring TTFT, long-context degradation, and cold-start time, where actual experience can differ by several times;
- Treating context length as a free lunch: every added segment of context consumes VRAM, increases prefill time, and lowers decode speed at once;
- Believing lower quantization is always faster: the bandwidth saved by a low bit width can be offset by repeated generation and manual rework caused by quality loss;
- Ignoring concurrency and batching: single-user benchmark data looks great, but once multiple users share it online, timeouts become frequent;
- Ignoring CPU and memory bottlenecks: when VRAM can’t hold the model, CPU-side memory bandwidth, channel count, and operator efficiency directly limit overall speed;
- Searching for decode speedups on a mechanical hard drive: storage speed mainly affects cold start, while decode speed depends on VRAM bandwidth, quantization bit width, and engine efficiency;
- Treating a non-standard-VRAM card as an ordinary consumer card for capacity expansion: a second-hand card may ship with non-standard capacity, and ordinary consumer cards can’t simply expand VRAM through modification;
- Counting streamed return packets as tokens: chunking strategies differ across engines, so chunk count doesn’t represent generated token count; you must use usage or tokenizer counting.
10. Summary: The Cost-Performance Formula
Back to the opening question: why can old hardware still hit usable speeds? The reason comes down to a simple cost-performance formula:
cost-performance ≈ (usable context × steady decode speed × output quality) ÷ (hardware cost + power + maintenance)
The lessons from the reference config:
- VRAM and bandwidth are the core resources: with a limited budget, spend money first on a card with more VRAM capacity, and use second-hand parts for the rest;
- Quantization, KV quantization, and speculative decoding are the three main levers: they respectively determine “it fits,” “it runs long,” and “it produces fast”;
- The engine and parameters set the floor and the ceiling: cold start, long-context degradation, and concurrency often decide the real experience more than paper specs;
- Switch the main model by need: when cloud lightweight models get cheaper, or when local hardware can take on heavier tasks, choose an MoE model with fewer activated parameters for higher generation speed at the same bandwidth;
- Long context must be combined with caching, retrieval, and effective-context management: blindly stacking up to the nominal max length often just sends VRAM and latency out of control.
Emphasize once more: the speed, VRAM, and context numbers in this article all come from an illustrative single case and are affected by GPU model, driver version, inference engine version, quantization format, KV bit width, sampling parameters, concurrency, and model architecture; they aren’t a performance guarantee. Measure VRAM, bandwidth, quantization, context, and architecture in the same framework, and any “old machine” can find its own optimal configuration. That’s also what makes local deployment so compelling: performance isn’t simply bought—it’s budgeted, tested, and tuned step by step.
