# Running a 9B LLM on an 8GB Device: 4-bit Quantization, Memory Budgeting, and Operator Optimization

    This article explains how to fit a 9B-parameter model into 8GB of unified memory by treating memory as a hard system-level constraint. It covers a realistic budget that accounts for weights, activations, KV cache, runtime, and buffers; practical 4-bit quantization engineering such as group-wise scaling, outlier handling, on-the-fly dequantization, and mixed precision; a decision process for when to degrade instead of forcing a full-budget run; operator-level optimization through low-level compute APIs, memory layout, batching, and command buffering; separate memory accounting for offline ASR, LLM, and TTS pipelines; a reproducible performance reporting template; an OOM troubleshooting guide; and the compliance boundaries that apply to restricted or unofficial environments. The goal is not a single headline token rate but a sustainable balance between model compression, kernel optimization, scheduling, and evaluation on constrained hardware.

    1. Start With the Total Budget, Not Just Whether 4-bit Weights Fit

    Putting a 9B model on an 8GB unified-memory device is not only a question of whether 4-bit weights fit. What actually determines feasibility is whether model weights, activations, KV cache, the runtime, temporary buffers, and system overhead can all be held inside the available memory at the same time.

    For reference, a 9B model stored in FP16 needs roughly 18 GB of weights, which is clearly out of reach for an 8GB device. 8-bit needs about 9 GB, which is also essentially over the limit. 4-bit comes in at roughly 4.5 GB in theory, and only at that point does the setup become plausible. In engineering terms, however, 4.5 GB is not a final figure: 4-bit weights usually also carry scale factors, zero points, group metadata, alignment padding, and quantization calibration tables. With group-wise quantization, real occupancy can exceed 4.5 GB — commonly by 0.1 GB to 0.5 GB above the theoretical value.

    A more realistic plan keeps total occupancy under 8GB. The following conservative reference works as a starting point:

    Item Budget reference Notes
    4-bit weights 4.5–5.1 GB Includes group scales, zero points, alignment metadata
    OS and inference runtime 0.7–1.0 GB Depends on the device OS, background services, runtime implementation
    Activations and temporary tensors 0.2–0.5 GB Scales with context length, batch size, operator implementation
    KV cache 0.2–0.5 GB Scales with context length, concurrency, whether it is quantized
    Command buffers and synchronization structures 0.1–0.3 GB Depends on low-level submission, queues, synchronization mechanisms

    Sum these up, leave a safety margin, and make “total occupancy must not exceed 8GB” an explicit constraint. If the estimate lands close to or above 8GB, treat the configuration as infeasible, or immediately shorten the context, lower concurrency, disable the voice modules, and shrink temporary buffers. Do not attempt a “run at full budget” configuration on an 8GB device. OOM usually shows up during prefill, long-context generation, or when several workloads are stacked together.

    2. 4-bit Quantization Is Error Engineering, Not Simple Truncation

    Converting FP16 weights to 4-bit is fundamentally about approximating floating-point values with a limited set of integers. The most common approach computes a scale factor that maps float values onto a symmetric integer range:

    1. import torch
    2. def quantize_group(w, bits=4, group_size=128):
    3. # Each group of group_size weights shares one scale and zero point
    4. qmax = 2 ** (bits - 1) - 1 # 7 for 4-bit
    5. scale = w.abs().max() / qmax
    6. scale = torch.where(scale > 0, scale, torch.tensor(1e-6, device=w.device))
    7. q = torch.round(w / scale).clamp(-(qmax + 1), qmax).to(torch.int8)
    8. return q, scale
    9. def dequantize(q, scale):
    10. return q.float() * scale

    Five points matter most when taking this into production.

    2.1 Group-wise quantization is more stable than per-tensor quantization

    Per-tensor quantization is easy to implement but unstable on layers with a wide dynamic range. A better default is one group per 128 weights, each with its own scale. This limits the influence of outliers on the entire layer and reduces visible degradation during long-form generation. Finer groups generally mean more stable accuracy, but they also increase the number of scales and zero points, which costs additional memory.

    2.2 Division-by-zero and outliers must be handled explicitly

    If a weight group is entirely near zero, computing the scale directly can produce division-by-zero or a numerical blow-up. Set a minimum scale, for example 1e-6. For outliers, consider clipping based on a 99.5th or 99.9th percentile rather than using the maximum value directly. Otherwise a single outlier stretches the quantization range for the whole group.

    2.3 Dequantization cost should be front-loaded or fused

    If inference first dequantizes all 4-bit weights into FP16 and then runs the matrix multiplication, memory usage climbs straight back up and can OOM immediately. The better design has operators dequantize on demand during computation: weights are loaded onto the chip, restored to full precision at the moment of use, and discarded immediately afterward. This cuts memory round-trips and peak occupancy.

    2.4 Sensitive layers can use mixed precision

    Attention layers, output projection layers, the projections around layer normalization, and some layers that matter for long context tend to be more sensitive to quantization. If 4-bit produces obvious repetition, semantic drift, or degraded instruction following, keep those layers at 8-bit or FP16. Mixed precision increases memory but substantially improves stability.

    2.5 The KV cache needs quantization too

    With short contexts, 4-bit weights alone may be enough. In long contexts or continuous conversation, however, the KV cache grows quickly. Options include quantizing the KV cache to 8-bit or 4-bit, capping the context window, or using sliding-window or paged cache strategies. Without this, no amount of weight compression will save you — the KV cache will pull the system back into OOM.

    3. The Core Rule: If the Ceiling Is Unreachable, Degrade

    Many projects fail not because the model is unusable, but because the budget was only estimated and never enforced as a hard constraint. Treat 8GB as a ceiling that cannot be crossed, and let every optimization decision revolve around that limit.

    A workable decision flow:

    1. First estimate the actual occupancy of the 4-bit weights, including scales, zero points, and metadata.
    2. Then add the OS, runtime, activations, KV cache, and command buffers.
    3. If total occupancy is below 6.5 GB, some headroom remains.
    4. If it sits between 6.5 and 7.5 GB, the configuration is only suitable for short contexts, low concurrency, and with voice or background services disabled.
    5. If it approaches or exceeds 8GB, treat it as infeasible unless you reduce the model size, the context length, the concurrency level, or the number of enabled modules.

    Common OOM scenarios include:

    • Loading the model, when mmap, cached pages, and runtime warm-up push the memory peak too high;
    • The prefill stage, when a long prompt makes activations spike;
    • The generation stage, when the KV cache keeps growing;
    • Running ASR, LLM, and TTS simultaneously, since the voice modules need extra memory;
    • Low-level operators that allocate a temporary workspace and never release it;
    • Concurrent inference requests whose buffers stack on top of each other.

    For that reason, run offline stress tests before deployment rather than only checking whether the model loads.

    4. Operator Optimization: Prefer Low-Level Compute APIs Over Abandoning the Driver

    The conventional inference path looks like this:

    1. Framework/runtime → Compute API → Graphics driver → GPU hardware

    This path is portable, safe, and easy to maintain, but on older hardware it can add scheduling and submission overhead. The idea often described as “bypassing driver abstraction” is more accurately stated as reducing the extra overhead of high-level graphics APIs and general-purpose frameworks in favor of lower-level, more direct compute interfaces.

    The recommended order is:

    1. Use the vendor’s official low-level compute API. Lighter than a general high-level framework, while still preserving the driver, memory management, and error handling.
    2. Tune the compilation stack. Let the compiler pick better vectorization, loop unrolling, and memory access patterns for the target hardware.
    3. Adjust memory layout. Reduce bandwidth waste and improve cache hit rates.
    4. Batch and pipeline. Merge small requests to amortize fixed submission costs.
    5. Optimize local hotspots. Customize only the most expensive operators — matrix multiplication, attention, normalization — rather than rewriting the entire chain.

    Fully bypassing the driver or runtime is not advisable. Going to a lower level of submission means dealing with the target GPU microarchitecture, instruction sequences, command buffers, resource binding, synchronization, barriers, queue priorities, power limits, and error recovery. Such implementations are expensive to maintain and highly sensitive to hardware model, driver version, and firmware policy.

    One boundary should be explicit: this article does not provide steps for unlocking devices, escalating privileges, or circumventing security mechanisms. For non-unsigned or restricted boot environments, this is background context for research only and is not suitable for production deployment.

    5. Command Buffering: Reduce Stalls Rather Than Chasing Peak Throughput

    Even when an individual operator is fast, the GPU can still sit at low utilization if every instruction is submitted and synchronized frequently. The value of command buffering is turning fragmented scheduling into a continuous supply of work:

    • Lower submission latency: batch multiple commands into one submission to reduce driver round-trips.
    • Higher utilization: keep compute units fed continuously instead of waiting in line.
    • Fewer context switches: reduce the jitter caused by frequent synchronization and queue switching.

    Command buffering is not a silver bullet, though. It makes debugging harder and can amplify synchronization bugs, resource leaks, and power fluctuation. On older hardware, whether instruction supply stays continuous often affects the real experience more than the nominal clock speed. Profile first to confirm the bottleneck really comes from submission or synchronization overhead, and only then decide whether a low-level buffering mechanism is worth it.

    6. Offline Voice Pipelines: Budget ASR, LLM, and TTS Separately

    If the device needs voice interaction, the pipeline becomes:

    1. Microphone audio → ASR → LLM → TTS → Speaker

    LLM memory is no longer the only number that matters; ASR and TTS consume memory as well.

    Component selection options include:

    • ASR: whisper.cpp, on-device Whisper implementations, Vosk.
    • TTS: Piper, Coqui TTS, eSpeak NG.
    • Principle: favor options that support low-resource deployment, streaming or sentence-by-sentence processing, have a clear license, run fully offline, and support the target language.

    Memory reference:

    Component Model size reference Runtime memory reference Notes
    ASR Tens to hundreds of MB 100–500 MB Depends on model size and whether it streams
    LLM (4-bit) 4.5–5.1 GB 5–6 GB Includes activations, KV cache, runtime
    TTS Tens to hundreds of MB 50–300 MB Depends on acoustic model, vocoder, voice

    If an 8GB device runs ASR, LLM, and TTS together, give the LLM the highest memory priority and compress the concurrency window of ASR and TTS. Streaming ASR, sentence-by-sentence chunking during generation, per-sentence TTS, and synthesizing while playback is already underway are preferable to waiting for the complete answer before synthesizing audio.

    Latency can be broken down against targets:

    • ASR first partial result: target 500–1500 ms;
    • LLM first token: target under 1 second;
    • TTS first chunk: target 300–800 ms;
    • End-to-end voice response: keep within roughly 2–4 seconds.

    These are example targets, not commitments. Actual results must be validated separately for a given device, model, quantization scheme, context, and measurement method.

    7. Performance Measurement: Do Not Cite Numbers That Cannot Be Reproduced

    Without a reproducible measurement, avoid quoting a terminal result such as “43 tokens/second” or “22 tokens/second”. Any performance figure should be accompanied by the device, model, quantization, context, concurrency, sampling settings, measurement script, and token definition.

    A reporting template like this keeps claims honest:

    Field Example
    Device model Legacy edge terminal A
    CPU/GPU 8-core ARM + integrated GPU
    Memory 8GB unified memory
    OS/firmware Custom restricted environment, research use only
    Model 9B instruction-tuned model
    Quantization W4A16, group size 128
    Context 2048 tokens
    Concurrency Single request
    Sampling Temperature 0.2, top-p 0.9
    First-token latency Measured value
    Steady-state token throughput Measured value
    Cold/warm start Recorded separately
    Power draw/temperature rise Whether thermal throttling occurred
    Token definition Whether spaces, punctuation, special tokens are included

    When only a conservative statement is defensible, phrase it this way: “On a specific older device, with a short context, low concurrency, and a particular quantization and firmware configuration, the model may reach a speed close to usable for interactive use; results do not transfer directly across platforms.” That is more reliable than an unsupported comparison.

    8. Pre-Deployment Checklist

    8.1 Does it fit in memory?

    Look first at actual 4-bit weight occupancy, then at the KV cache and runtime. If total occupancy exceeds 8GB, do not force it to run.

    8.2 Is the 4-bit quality loss acceptable?

    Perplexity alone is not enough. Combine a business task set, human evaluation of generated outputs, instruction-following tests, and long-form consistency tests.

    8.3 Do you actually need low-level operators?

    Profile first. If the bottleneck is bandwidth, scheduling, or memory layout, optimize memory access and batching. Only consider low-level APIs once you have confirmed that a high-level framework is the actual bottleneck.

    8.4 Is the voice pipeline real-time?

    Measure ASR, LLM, and TTS separately. If end-to-end latency is too high, switch to streaming or sentence-level processing first.

    8.5 How do you diagnose OOM?

    Symptom Likely cause Suggested fix
    OOM during load Weight metadata, runtime warm-up Use mmap, reduce duplicate caching
    OOM during prefill Long prompt, oversized activations Shorten the context, cap input length
    OOM during generation Growing KV cache Quantize the KV cache, cap maximum context
    OOM in voice scenarios Extra ASR/TTS memory Disable non-essential modules, reuse buffers
    Slowdown after running a while Memory fragmentation, unreleased buffers Fixed memory pool, periodic reclamation

    8.6 How do you fall back when a low-level operator fails?

    Keep a fallback path: try the vendor’s low-level compute API first, then fall back to a general inference backend; if that still fails, disable the custom kernel, reduce batch size, shorten the context, or switch to a higher-precision but more stable implementation.

    9. Compliance and Safety Boundaries

    Running large models on older hardware, development boards, or custom terminals may involve non-unsigned or restricted boot environments. This can lead to:

    • Violating the device terms of use or the vendor agreement;
    • Voiding the warranty;
    • Introducing unsigned components, which increases security and supply-chain risk;
    • System instability, data loss, or abnormal power behavior;
    • Crossing compliance lines on managed devices, enterprise terminals, or automotive systems.

    Practices described here are therefore best positioned as experimental research, performance validation, and technical exploration. This article provides no steps for unlocking, privilege escalation, or bypassing security mechanisms. For product use, prefer vendor-supported on-device inference solutions, controlled model formats, and compliant distribution channels.

    10. Conclusion

    With 8GB of memory, a low-performance GPU, and a 9B model, local inference succeeds not through a single clever trick but through system-level tradeoffs: 4-bit quantization and mixed precision to control weight size, KV cache strategies to control long-context risk, low-level operators to reduce execution overhead, clear performance reporting to avoid misjudgment, and compliance boundaries to constrain the deployment scope.

    What is worth borrowing is not the token rate some particular device achieved, but whether you can find a sustainable balance among model compression, operator optimization, system scheduling, and experience evaluation under resource constraints.