Cutting LLM Inference Costs: Mixed FP8/Int4 Quantization and Custom AI Accelerator Deployment

    A practical guide to lowering large language model (LLM) inference costs through mixed-precision quantization and deployment on custom AI accelerators. You'll learn why memory capacity, bandwidth, and KV Cache growth drive the per-token cost of inference, and how to compress it safely. It covers combining Int4 weight storage with FP8 activations and selective high-precision layers, calibrating quantization, exempting sensitive layers, and validating output quality. It then shows how to adapt quantized models to custom AI accelerators by checking operator coverage, converting compute graphs, optimizing kernels, and running accuracy regression. Finally, it walks through a repeatable workflow, from baseline measurement and load testing to observability, that turns hardware savings into measurable throughput gains and lower costs.

    1. The Cost Bottlenecks of LLM Inference

    As parameter counts keep climbing, large language models face a double bind during inference: the memory wall and the compute wall. Unlike training, inference is far more sensitive to per-request latency and concurrent throughput. Model weights must stay resident in device memory, the KV Cache grows linearly with context length, and every generated token requires a full forward pass.

    Under the default half-precision (FP16/BF16) formats, a model with tens of billions of parameters can consume so much memory and memory bandwidth that it becomes the main bottleneck. This directly determines how many concurrent requests each device can handle and how much it costs to serve each token.

    The industry generally follows two complementary paths to lower inference costs:

    • Quantization: reduce the bit-width of weights and activations to cut both memory usage and data movement.
    • Hardware co-design: adapt the compute graph to the underlying AI accelerators so that theoretical compute translates into real throughput.

    This article focuses on the first path, along with how to put it into practice on custom AI accelerators, including China’s domestically developed AI chips.

    2. Mixed-Precision Quantization: Combining FP8 and Int4

    Quantization represents high-precision floating-point values using fewer bits. Intuitively, the lower the bit-width, the greater the savings, but a uniform, one-size-fits-all reduction often hurts model accuracy. That’s why mature quantization approaches use a mixed-precision strategy: different layers and tensors are assigned different bit-widths according to their sensitivity to error.

    • FP8 (8-bit float): retains both exponent and mantissa bits, so it offers a friendly dynamic range. It suits values with sensitive distributions, such as activations and attention-related computations. It compresses these values while maintaining numerical stability reasonably well.
    • Int4 (4-bit integer): offers a lower bit-width and a higher compression ratio. It’s typically used with per-channel scaling factors and suits cases with relatively well-controlled distributions, such as weight quantization.

    A typical mixed strategy is to store the weights of linear layers (which make up the bulk of memory usage) in Int4. Meanwhile, compute-intensive paths that are sensitive to precision remain in FP8 or higher precision.

    The following illustrative quantization configuration shows this per-layer approach. It’s pseudocode that conveys the idea only:

    1. quant_config = {
    2. "weight": {
    3. "default": {"bits": 4, "granularity": "per_channel"}, # most linear layer weights
    4. },
    5. "activation": {
    6. "default": {"bits": 8, "format": "fp8"}, # activations use FP8
    7. },
    8. "keep_high_precision": ["embed_tokens", "lm_head"], # sensitive layers stay in high precision
    9. "calibration": {"method": "percentile", "ratio": 0.999}, # calibration to limit the impact of outliers
    10. }

    3. Keeping Accuracy After Compression

    Lower bit-width isn’t always better: with quantization, the key lies in controlling error. In engineering practice, you can work on three fronts:

    1. Proper calibration: collect activation statistics from representative samples to determine scaling factors and zero points. Use percentile truncation to limit the influence of outliers on the overall range.
    2. Exempting sensitive layers: keep embedding layers, the output projection (lm_head), and some attention projection layers at higher precision. This prevents errors from being amplified along critical paths.
    3. Error feedback and validation: compare output distributions and key metrics (such as perplexity and task accuracy) on the same evaluation set before and after quantization. Keep deviations within an acceptable threshold. If necessary, use quantization-aware training (QAT) instead of relying on post-training quantization (PTQ) alone.

    The goal is to significantly reduce memory usage and per-inference cost while keeping accuracy essentially intact. That’s exactly where mixed quantization stands out from single low-bit schemes.

    4. Deploying Models and Custom AI Accelerators as One Stack

    A quantization scheme only delivers on its promise with support from the underlying hardware. Deploying large models on custom AI accelerators, such as China’s domestically developed AI chips, typically calls for integrated model–accelerator engineering:

    • Operator adaptation: low-precision computations such as FP8 and Int4 require matching operators and instruction support on the chip. Teams need to confirm how well the target operators are covered. For missing ones, you can split the graph and fall back to supported operators, or write custom implementations.
    • Compute graph conversion: export the model from the training framework into an intermediate representation that the accelerator vendor’s toolchain recognizes. Then complete operator mapping, memory planning, and graph optimization.
    • Kernel and scheduling optimization: taking into account the hardware’s memory hierarchy and parallelism, optimize for overlapping data access with computation to improve effective compute utilization.
    • Accuracy regression: re-validate the quantized model’s outputs on the target hardware. Make sure the software-level quantization strategy behaves consistently with the hardware’s numerical behavior.

    The point of this integrated approach is to tune model compression and hardware execution together, rather than treating them in isolation. The operator ecosystems and toolchain maturity of custom AI accelerators are still evolving quickly. In engineering terms, that means planning ahead for operator fallbacks, accuracy safeguards, and version alignment.

    5. End-to-End Engineering Practice and Cost-Benefit Analysis

    Putting all of these pieces together yields a reusable inference cost-reduction workflow:

    1. Baseline measurement: run the evaluation set at the original precision first. Record memory usage, throughput, latency, and accuracy metrics as your baseline.
    2. Quantization configuration: set up a mixed-quantization scheme following the strategies in Sections 2 and 3, prioritizing low-bit compression of weights.
    3. Hardware adaptation: complete graph conversion and operator mapping with the custom accelerator’s toolchain, and handle missing operators.
    4. Accuracy alignment: compare outputs before and after quantization. If needed, keep sensitive layers at higher precision or apply calibration corrections.
    5. Load testing and tuning: load-test concurrency on the target accelerator. Adjust batching strategies and KV Cache management to get close to the hardware’s limits.
    6. Continuous monitoring: use logging and monitoring/alerting services to track online inference latency, memory usage, and error rates. This closes the optimization loop.

    On the deployment side, you can containerize the inference service and run it on a container platform. Pair it with a load balancer to enable elastic scaling, use object storage to manage model weights, and use a logging service to capture call traces for easier troubleshooting. Only then can the memory and compute savings from model compression translate reliably into measurable throughput gains and lower costs.

    6. Wrap-Up

    Cutting LLM inference costs means finding the right engineering balance between “essentially no loss in accuracy” and “substantial resource reduction.” Mixed-precision quantization, represented by FP8 and Int4, offers an approach that balances weight compression with numerical stability. Integrated deployment of models and custom AI accelerators then lets compressed models actually run efficiently.

    For developers, the key is to handle the critical stages well: calibration, exempting sensitive layers, operator adaptation, and end-to-end validation. Doing so gives you a reusable, monitorable, and scalable path to low-cost inference.