On-Device AI Software Stack Selection and Precision Regression: A Practical Guide
This guide presents a practical framework for selecting an on-device AI software stack and running precision regression. It covers update verification, layered architecture and hardware adaptation, model asset management, INT8 and BF16 validation, GPU containerization and monitoring, performance and quality regression, and canary release and rollback checklists.
1. Verify the Update’s Verifiable Information First
This article discusses a methodological framework and does not evaluate specific versions. How to select an on-device AI software stack and how to run precision regression depends on the specific information provided by the platform provider. After obtaining the release notes from the developer portal, fill in the checklist below first. If any item is missing, confirm it with the platform provider instead of assuming the support scope.
| Verification Item | What to Record | Recording Requirement |
|---|---|---|
| Software stack name and version | Version numbers of the runtime, compiler, and toolchain | Follow the official release notes |
| New model categories | Supported model architectures, such as LLMs, NLP models, and CNNs | List the specific model architectures |
| INT8 support scope | Supported operators, input shapes, and calibration methods | Follow the official documentation |
| BF16 support scope | Supported hardware categories and operators | Follow the official documentation |
| Tool names | Names of model analysis, performance profiling, and precision comparison tools | Record tool entry points and commands |
| Known limitations | Unsupported operators and known issue IDs | Record issue IDs and status |
After verification, compare this update with the team’s existing environment. If the new version changes operator coverage or precision defaults, the validation steps in Section 4 must be rerun.
2. Layered Architecture and Hardware Adaptation
A maintainable on-device AI software stack is usually divided into five layers: application and business logic, inference service and runtime, model compilation and graph optimization, operators and kernels, and drivers and hardware. An on-device device may contain a CPU, GPU, and NPU at the same time. The software stack needs to perform capability discovery, operator mapping, and fallback configuration.
Maintain a hardware capability description file for each target device. It should include the following fields:
- Device category and model
- Supported compute units: CPU, GPU, NPU
- Precision supported by each compute unit: INT8, BF16, FP16, FP32
- Available memory upper limit and GPU memory upper limit
- Driver and runtime versions
The compilation configuration reads from this description file, and business code does not hardcode hardware branches. In this way, when adding a new device, you only need to add the description file and rerun regression.
When an operator is not supported, the fallback strategy should be explicitly written into the configuration. For example, if an operator does not support INT8 on the target device, switch to the BF16 or FP32 path, and record the fallback operator name and count. If the fallback count keeps growing, it usually indicates that the model architecture does not match the target hardware, and the model should be reevaluated.
Common troubleshooting paths are as follows:
- Unsupported operator: First check the operator coverage report to confirm whether an alternative operator or an available fallback path exists. If coverage is still not achieved, modify the model architecture or change the precision.
- Misdiagnosed memory peak: Do not look only at total process memory. Measure the peaks of model weights, activations, and KV cache separately, then determine the source of the bottleneck.
- Accuracy drift: Compare outputs layer by layer, find the first layer that exceeds the threshold, and then inspect that layer’s operator implementation and quantization parameters.
3. Model Asset Management
Model assets fall into three categories: original models, compilation intermediate artifacts, and deployment packages. Each asset category needs a clear owner, storage location, version identifier, and rollback trigger condition.
| Asset Type | Owner | Storage Location | Version Identifier | Rollback Trigger Condition |
|---|---|---|---|---|
| Original model | Model training owner | Original model directory in object storage | Training job ID plus file content hash | No rollback; used only for traceability and recompilation |
| Compilation intermediate artifact | Compilation engineer | Artifact repository, organized by hardware category | Original model version plus compilation config hash | Operator coverage drops after graph optimization, or precision regression fails |
| Deployment package | Release owner | Deployment directory in the artifact repository | Semantic version number | Canary error rate exceeds the threshold, or business metrics fall below the service-level objective (SLO) |
Different model types also require attention to additional parameters:
- LLMs: context length, batching strategy, KV cache peak.
- NLP models: maximum input length, tokenizer version.
- CNNs: input size, operator fusion result, quantization calibration set version.
4. INT8 and BF16 Validation Steps
Precision configuration cannot be judged only by whether conversion succeeds. The following six steps should be executed in order.
- Prepare a calibration set. Perform stratified sampling according to the business request distribution, covering short and long inputs and boundary categories. The template size is 300 to 1,000 entries and should be adjusted according to model complexity and business requirements.
- Generate golden outputs. Run inference on a BF16 or FP32 baseline, save output tensors and task metrics, and record the environment, random seed, and input hash.
- Perform operator-level alignment. For each low-precision operator, compare output errors using boundary inputs. Common metrics are maximum absolute error and cosine similarity.
- Run model-level regression. Use a fixed evaluation set to compare task metrics, such as classification accuracy, detection recall, or generation quality score.
- Run continuous integration comparison. Automatically run regression on every build, and block release if results do not meet the standard.
- Make operator-level fallback decisions. When an operator’s error exceeds the threshold, switch that operator to a higher precision and record the operator name and reason. When the proportion of fallback operators among all operators exceeds the template upper limit, do not adopt that precision for the entire model.
The validation thresholds are as follows. All values are templates and need to be calibrated by the business.
| Metric | Record Field | Template Threshold | Calibration Method |
|---|---|---|---|
| Operator cosine similarity | operator_name, cosine_sim | Not lower than 0.99 | Take the 1st percentile on the calibration set and leave margin |
| Operator maximum absolute error | operator_name, max_abs_err | Not higher than the template upper limit | Convert based on the numerical range of the operator output |
| Classification accuracy drop | task_metric, delta_acc | Drop no more than 1 percentage point | The business owner confirms the acceptable loss |
| Fallback operator ratio | fallback_count, total_ops | No more than 5% of total operators | The compilation engineer determines it based on performance gain |
5. GPU Containerization and Monitoring
GPU-accelerated environments should be included in containerized builds and continuous integration. The main practices are as follows:
- The base image pins the driver and runtime versions. Deployments reference the image digest instead of a mutable tag.
- Build tasks run in the container platform. Each build outputs the image digest, software stack version, and compilation config hash.
- Model artifacts are stored in object storage or an artifact repository, with hashes and versions recorded.
- Logs and monitoring are collected in a unified way, with fields designed as follows.
Example monitoring metrics are as follows:
| Metric Name | Collection Granularity | Alert Template |
|---|---|---|
| First-token latency (p50, p95) | Per request | Alert when p95 is above the SLO |
| Throughput | Per minute | Alert when the drop from baseline exceeds the template ratio |
| Peak GPU memory | Per instance | Alert when the budget upper limit is exceeded |
| GPU utilization | Per device, every 10 seconds | Check data supply when sustained below the template lower bound |
| Error rate | Per deployment version | Trigger rollback evaluation when the template threshold is exceeded |
| Host-to-device data transfer time | Per batch | Investigate when its share of total time rises abnormally |
Recommended log fields include: request_id, model_version, precision, batch_size, input_len, device_type. Only after fields are unified can issues be compared by version and precision.
6. Performance and Quality Regression
Regression is divided into three layers. The inputs, metrics, pass criteria, and record fields for each layer are as follows:
| Layer | Input | Metrics | Pass Criteria | Record Fields |
|---|---|---|---|---|
| Microbenchmark | A single operator with a fixed shape | Single-run latency, output error | Latency increase relative to baseline does not exceed the template ratio | Operator name, shape, precision, driver version |
| Model benchmark | Fixed evaluation set | End-to-end latency, task metrics | Task metrics are not lower than the golden output threshold | Model version, batch size, input length |
| Business benchmark | Real request replay samples | SLO attainment rate, business quality | Meets the SLO target | Request category, hardware category, environment baseline |
The pass criteria are templates and must be determined by the team before launch.
7. Canary Release and Rollback Checklist
Canary release is executed in the following order, and each step must produce an output.
| Step | Owner | Output | Acceptance Criteria |
|---|---|---|---|
| Version check | Release owner | Version checklist | Software stack, model, and config hash match the records |
| Regression execution | Test owner | Regression report | All three regression layers pass |
| Canary 5% traffic | Release owner | Monitoring snapshot | Error rate and latency are within template thresholds; observation duration is determined by the business |
| Expand canary | Release owner | Scale-out record | No alerts during the observation period at each expansion level |
| Rollback drill | Operations owner | Drill record | At least once per quarter, and rollback duration is recorded |
| Rollback execution | On-call owner | Rollback record | Switch back to the previous deployment package and configuration |
Rollback should be executed when any of the following conditions is met:
- The error rate exceeds the template threshold.
- Precision regression fails.
- Peak GPU memory exceeds the budget upper limit.
- Output values become abnormal, or business metrics fall below the SLO.
Before rollback, back up the current deployment package, configuration, and logs. Keep the old version artifacts until the next canary round is stable; do not delete them.
Follow-up actions are executed in the following order: version check, regression execution, canary observation, and rollback drill. Repeat this process for every update until the team establishes a stable recording habit.
