Cost Reduction Through Mixed-Precision Inference on Domestic AI Chips
This article presents an integrated model–hardware approach to reducing large language model inference costs on domestic AI chips. By combining FP8 and Int4 quantization according to layer sensitivity, and coordinating model conversion, compiler optimization, memory planning, and service deployment, the approach reduces memory and bandwidth pressure while improving throughput. Under specific workload, concurrency, SLA, and hardware conditions, API unit costs may reach roughly one-seventh of some mainstream commercial model solutions, although production results should always be validated through benchmarking.
Cost Reduction Through Mixed-Precision Inference on Domestic AI Chips
As large language model services move from experimentation to large-scale production, inference cost has become a critical factor in business sustainability. Simply expanding the compute cluster does not solve the problem at its root. A more effective approach is to optimize four areas together: model representation, chip operators, compiler scheduling, and service delivery.
This article discusses an integrated inference solution for domestic AI chips. It uses mixed FP8 and Int4 quantization to reduce memory bandwidth pressure and improve throughput while preserving model quality as much as possible. Under specific input and output lengths, concurrency levels, SLA requirements, and hardware configurations, the resulting API unit cost can be reduced to approximately one-seventh of the price of some mainstream commercial model solutions. These figures are intended only as a reference; production deployments should rely on measured results.
1. Why Mixed Quantization and Hardware–Software Co-Optimization Matter
The main costs of large language model inference come from three areas:
- Compute consumption
- Device memory capacity and bandwidth
- Redundant resources required for service delivery
Traditional FP16 or BF16 inference generally provides stable accuracy, but its weights and KV cache consume significant memory. This limits the concurrency that a single device can support. Weight-only Int4 quantization can substantially reduce memory usage, but an overly aggressive strategy may cause quality degradation in attention layers, output layers, or sensitive MLP layers, affecting business outcomes.
The core idea behind FP8 and Int4 mixed quantization is to assign precision according to numerical sensitivity:
- Use FP8 for computation paths that require a wider dynamic range or are more sensitive to numerical errors.
- Use Int4 for parameter-heavy paths with greater redundancy and higher tolerance for lower precision.
- Retain higher precision for embeddings, output layers, normalization paths, or other sensitive components.
This approach captures the memory capacity and bandwidth benefits of lower-bit representations while avoiding the severe quality degradation that can result from quantizing the entire model to a low bit width.
More importantly, quantization is not an isolated model transformation. The final benefit depends directly on whether the domestic AI chip efficiently supports low-bit operators, whether memory layouts match the hardware, whether the compiler can fuse operators, and how the KV cache is managed. The solution must therefore be designed from an integrated model–chip perspective rather than treating quantization as an offline model-compression task.
2. Overall Architecture: Coordination Across the Model, Compiler, and Service Layers
An integrated inference solution can be divided into three layers.
- Model layer: Performs sensitivity analysis, generates the mixed-precision quantization policy, runs calibration, and validates accuracy. The output includes the quantized model representation, quantization parameters, and fallback rules.
- Compiler and runtime layer: Maps the quantized model to the operator library and memory hierarchy of the domestic AI chip. It handles operator fusion, weight reordering, KV cache layout optimization, and batch scheduling.
- Service layer: Provides stable APIs, dynamic batching, rate limiting, circuit breaking, logging, monitoring, and canary releases. In general-purpose cloud environments such as Baidu AI Cloud, container platforms, logging services, and monitoring and alerting tools can be used for service operations. Resource specifications should be determined through actual load testing.
The key is to establish a closed loop covering accuracy, performance, and cost. The model layer defines acceptable accuracy boundaries, the compiler layer exposes operator support and performance bottlenecks, and the service layer validates unit-token cost against real request distributions. No single optimization can replace end-to-end evaluation.
3. Designing an FP8 and Int4 Mixed-Quantization Strategy
3.1 Assign Bit Widths According to Layer Sensitivity
Applying Int4 uniformly across the entire model is generally not recommended. A more reliable method is to analyze sensitivity at the layer or module level and then generate a mixed-precision policy. Common principles include:
- Attention projections and selected MLP layers can be evaluated with FP8 to balance throughput and dynamic range.
- Linear layers with a large share of the parameters and relatively high redundancy can use Int4 weights together with group quantization and scale/zero-point parameters.
- Embeddings, output heads, normalization layers, and residual-related paths typically retain FP16, BF16, or FP8 to avoid shifts in the output distribution.
- Layers where quantization causes a sudden increase in error should be added to a fallback allowlist.
The following configuration illustrates the approach. The exact fields should be adapted to the capabilities of the target inference framework.
quantization:default_dtype: fp8_e4m3weight_dtype_policy:- pattern: "attention.*"dtype: fp8- pattern: "mlp.*"dtype: int4group_size: 128- pattern: "embedding|lm_head|norm.*"dtype: fp16activation_dtype_policy:- pattern: "attention.*"dtype: fp8- pattern: "mlp.*"dtype: fp8calibration:method: minmaxsample_count: 512dataset_profile: "business-representative corpus"fallback:max_accuracy_drop: "0.5%"
3.2 Keep the Calibration Set Representative of Real Workloads
A calibration set should not consist only of general-purpose text. It should cover the actual distribution of business inputs, including input lengths, domain terminology, multi-turn conversations, and structured outputs.
Insufficient calibration can produce inaccurate scale estimates and cause quality degradation on long-tail requests. Sampling should be stratified according to production traffic, with separate evaluations for long inputs, short outputs, and high-concurrency workloads.
3.3 Make Quantization Error Observable
Mixed quantization is not a one-time operation. The service should record the quantization version, request type, output-quality sampling results, and anomaly rate so that deployments can be versioned and rolled back.
For critical workloads, use a small canary traffic percentage to validate accuracy and latency before expanding the deployment.
4. Model–Chip Integrated Deployment Workflow
4.1 Model Export and Graph Optimization
After exporting the trained model to a general intermediate representation, apply constant folding, operator fusion, and dead-code elimination. At this stage, verify the supported range of FP8, Int4, and mixed-precision operators on the target domestic AI chip.
If certain operators support only specific layouts, perform the required reordering during graph optimization. This avoids repeated runtime conversions.
4.2 Quantization Calibration and Policy Generation
Generate layer-level quantization policies based on sensitivity analysis and calibration data. Apply group quantization to Int4 weights to reduce the impact of outliers. For FP8 activations, select an appropriate exponent-bit configuration to balance dynamic range and accuracy.
The resulting artifacts should include quantization parameters, fallback rules, and version metadata.
4.3 Compilation and Memory Planning
The compilation stage should focus on weight reordering, KV cache partitioning, attention operator fusion, and batch scheduling.
Domestic AI chips may have different on-chip memory capacities and bandwidth characteristics. Tile size, batch size, and concurrency limits therefore need to be tuned together. Otherwise, the system may encounter an imbalance in which memory capacity is sufficient but bandwidth is saturated, or compute resources remain idle while the runtime waits for scheduling.
4.4 Service Delivery and Canary Releases
At the service layer, use dynamic batching and request queuing. Bucket requests by input length to reduce resource waste caused by mixing short and long requests.
The API layer should provide version routing, rate limiting, timeouts, and degradation mechanisms. During canary releases, monitor throughput, P95 and P99 latency, error rate, and sampled output quality at the same time.
5. Accuracy Protection and Evaluation
Maintaining accuracy cannot rely on a single metric. A three-level evaluation process is recommended:
- Offline evaluation: Compare pre- and post-quantization accuracy, F1, BLEU/ROUGE, or human evaluation scores on business-specific test sets. Pay particular attention to long-tail samples.
- Online sampling: Apply rule-based validation, model-based evaluation, or human review to API outputs and monitor changes in anomaly distributions.
- Business metrics: Track end results such as conversion rate, complaint rate, and retry rate. This helps identify cases where offline metrics remain stable but the user experience deteriorates.
If quality degradation is detected, first locate the sensitive layers and adjust the group size, calibration method, or local fallback policy instead of reverting the entire model to higher precision. Combining local fallback with version-based canary deployment can provide a better balance between cost and accuracy.
6. Cost and Performance Benefits
The benefits of mixed quantization mainly come from four sources:
- Lower weight and KV cache usage, allowing each device to support higher concurrency
- Lower memory bandwidth pressure, improving decoding throughput
- Reduced computation overhead from lower-bit operators
- A more compact model that is easier to scale elastically
Together, these improvements can significantly reduce API unit cost. The publicly referenced figure of approximately one-seventh of the price of some mainstream commercial model APIs can be used as an order-of-magnitude reference. Actual costs should be recalculated based on request length, batching efficiency, SLA redundancy, and hardware utilization.
7. Suitable Use Cases and Deployment Recommendations
This approach is suitable for online inference workloads that are cost-sensitive, receive substantial traffic, and require stable output quality. Examples include knowledge-based question answering, content generation, summarization and extraction, and coding assistance.
Recommended deployment practices include:
- Begin with a canary deployment for a non-critical workload.
- Establish version management and rollback mechanisms for quantized models.
- Use production-like traffic for load testing rather than relying only on offline benchmarks.
- Include monitoring, alerting, logging, and container platforms in the operational feedback loop.
- Retain a higher-precision fallback path for critical business workloads.
Through coordinated optimization across the model, chip, compiler, and service layers, mixed-precision inference on domestic AI chips can deliver substantial cost and efficiency improvements while keeping deployment risks under control.
