On-Device AI Software Stack Selection and Precision Regression: A Practical Guide

    This guide presents a practical framework for selecting an on-device AI software stack and running precision regression. It covers update verification, layered architecture and hardware adaptation, model asset management, INT8 and BF16 validation, GPU containerization and monitoring, performance and quality regression, and canary release and rollback checklists.

    1. Verify the Update’s Verifiable Information First

    This article discusses a methodological framework and does not evaluate specific versions. How to select an on-device AI software stack and how to run precision regression depends on the specific information provided by the platform provider. After obtaining the release notes from the developer portal, fill in the checklist below first. If any item is missing, confirm it with the platform provider instead of assuming the support scope.

    Verification Item What to Record Recording Requirement
    Software stack name and version Version numbers of the runtime, compiler, and toolchain Follow the official release notes
    New model categories Supported model architectures, such as LLMs, NLP models, and CNNs List the specific model architectures
    INT8 support scope Supported operators, input shapes, and calibration methods Follow the official documentation
    BF16 support scope Supported hardware categories and operators Follow the official documentation
    Tool names Names of model analysis, performance profiling, and precision comparison tools Record tool entry points and commands
    Known limitations Unsupported operators and known issue IDs Record issue IDs and status

    After verification, compare this update with the team’s existing environment. If the new version changes operator coverage or precision defaults, the validation steps in Section 4 must be rerun.

    2. Layered Architecture and Hardware Adaptation

    A maintainable on-device AI software stack is usually divided into five layers: application and business logic, inference service and runtime, model compilation and graph optimization, operators and kernels, and drivers and hardware. An on-device device may contain a CPU, GPU, and NPU at the same time. The software stack needs to perform capability discovery, operator mapping, and fallback configuration.

    Maintain a hardware capability description file for each target device. It should include the following fields:

    • Device category and model
    • Supported compute units: CPU, GPU, NPU
    • Precision supported by each compute unit: INT8, BF16, FP16, FP32
    • Available memory upper limit and GPU memory upper limit
    • Driver and runtime versions

    The compilation configuration reads from this description file, and business code does not hardcode hardware branches. In this way, when adding a new device, you only need to add the description file and rerun regression.

    When an operator is not supported, the fallback strategy should be explicitly written into the configuration. For example, if an operator does not support INT8 on the target device, switch to the BF16 or FP32 path, and record the fallback operator name and count. If the fallback count keeps growing, it usually indicates that the model architecture does not match the target hardware, and the model should be reevaluated.

    Common troubleshooting paths are as follows:

    • Unsupported operator: First check the operator coverage report to confirm whether an alternative operator or an available fallback path exists. If coverage is still not achieved, modify the model architecture or change the precision.
    • Misdiagnosed memory peak: Do not look only at total process memory. Measure the peaks of model weights, activations, and KV cache separately, then determine the source of the bottleneck.
    • Accuracy drift: Compare outputs layer by layer, find the first layer that exceeds the threshold, and then inspect that layer’s operator implementation and quantization parameters.

    3. Model Asset Management

    Model assets fall into three categories: original models, compilation intermediate artifacts, and deployment packages. Each asset category needs a clear owner, storage location, version identifier, and rollback trigger condition.

    Asset Type Owner Storage Location Version Identifier Rollback Trigger Condition
    Original model Model training owner Original model directory in object storage Training job ID plus file content hash No rollback; used only for traceability and recompilation
    Compilation intermediate artifact Compilation engineer Artifact repository, organized by hardware category Original model version plus compilation config hash Operator coverage drops after graph optimization, or precision regression fails
    Deployment package Release owner Deployment directory in the artifact repository Semantic version number Canary error rate exceeds the threshold, or business metrics fall below the service-level objective (SLO)

    Different model types also require attention to additional parameters:

    • LLMs: context length, batching strategy, KV cache peak.
    • NLP models: maximum input length, tokenizer version.
    • CNNs: input size, operator fusion result, quantization calibration set version.

    4. INT8 and BF16 Validation Steps

    Precision configuration cannot be judged only by whether conversion succeeds. The following six steps should be executed in order.

    1. Prepare a calibration set. Perform stratified sampling according to the business request distribution, covering short and long inputs and boundary categories. The template size is 300 to 1,000 entries and should be adjusted according to model complexity and business requirements.
    2. Generate golden outputs. Run inference on a BF16 or FP32 baseline, save output tensors and task metrics, and record the environment, random seed, and input hash.
    3. Perform operator-level alignment. For each low-precision operator, compare output errors using boundary inputs. Common metrics are maximum absolute error and cosine similarity.
    4. Run model-level regression. Use a fixed evaluation set to compare task metrics, such as classification accuracy, detection recall, or generation quality score.
    5. Run continuous integration comparison. Automatically run regression on every build, and block release if results do not meet the standard.
    6. Make operator-level fallback decisions. When an operator’s error exceeds the threshold, switch that operator to a higher precision and record the operator name and reason. When the proportion of fallback operators among all operators exceeds the template upper limit, do not adopt that precision for the entire model.

    The validation thresholds are as follows. All values are templates and need to be calibrated by the business.

    Metric Record Field Template Threshold Calibration Method
    Operator cosine similarity operator_name, cosine_sim Not lower than 0.99 Take the 1st percentile on the calibration set and leave margin
    Operator maximum absolute error operator_name, max_abs_err Not higher than the template upper limit Convert based on the numerical range of the operator output
    Classification accuracy drop task_metric, delta_acc Drop no more than 1 percentage point The business owner confirms the acceptable loss
    Fallback operator ratio fallback_count, total_ops No more than 5% of total operators The compilation engineer determines it based on performance gain

    5. GPU Containerization and Monitoring

    GPU-accelerated environments should be included in containerized builds and continuous integration. The main practices are as follows:

    • The base image pins the driver and runtime versions. Deployments reference the image digest instead of a mutable tag.
    • Build tasks run in the container platform. Each build outputs the image digest, software stack version, and compilation config hash.
    • Model artifacts are stored in object storage or an artifact repository, with hashes and versions recorded.
    • Logs and monitoring are collected in a unified way, with fields designed as follows.

    Example monitoring metrics are as follows:

    Metric Name Collection Granularity Alert Template
    First-token latency (p50, p95) Per request Alert when p95 is above the SLO
    Throughput Per minute Alert when the drop from baseline exceeds the template ratio
    Peak GPU memory Per instance Alert when the budget upper limit is exceeded
    GPU utilization Per device, every 10 seconds Check data supply when sustained below the template lower bound
    Error rate Per deployment version Trigger rollback evaluation when the template threshold is exceeded
    Host-to-device data transfer time Per batch Investigate when its share of total time rises abnormally

    Recommended log fields include: request_id, model_version, precision, batch_size, input_len, device_type. Only after fields are unified can issues be compared by version and precision.

    6. Performance and Quality Regression

    Regression is divided into three layers. The inputs, metrics, pass criteria, and record fields for each layer are as follows:

    Layer Input Metrics Pass Criteria Record Fields
    Microbenchmark A single operator with a fixed shape Single-run latency, output error Latency increase relative to baseline does not exceed the template ratio Operator name, shape, precision, driver version
    Model benchmark Fixed evaluation set End-to-end latency, task metrics Task metrics are not lower than the golden output threshold Model version, batch size, input length
    Business benchmark Real request replay samples SLO attainment rate, business quality Meets the SLO target Request category, hardware category, environment baseline

    The pass criteria are templates and must be determined by the team before launch.

    7. Canary Release and Rollback Checklist

    Canary release is executed in the following order, and each step must produce an output.

    Step Owner Output Acceptance Criteria
    Version check Release owner Version checklist Software stack, model, and config hash match the records
    Regression execution Test owner Regression report All three regression layers pass
    Canary 5% traffic Release owner Monitoring snapshot Error rate and latency are within template thresholds; observation duration is determined by the business
    Expand canary Release owner Scale-out record No alerts during the observation period at each expansion level
    Rollback drill Operations owner Drill record At least once per quarter, and rollback duration is recorded
    Rollback execution On-call owner Rollback record Switch back to the previous deployment package and configuration

    Rollback should be executed when any of the following conditions is met:

    • The error rate exceeds the template threshold.
    • Precision regression fails.
    • Peak GPU memory exceeds the budget upper limit.
    • Output values become abnormal, or business metrics fall below the SLO.

    Before rollback, back up the current deployment package, configuration, and logs. Keep the old version artifacts until the next canary round is stable; do not delete them.

    Follow-up actions are executed in the following order: version check, regression execution, canary observation, and rollback drill. Repeat this process for every update until the team establishes a stable recording habit.