On-Device AI Development for AI PCs: From Software Stack to Model Deployment

    This article explains why on-device AI development for AI PCs depends heavily on the software stack. It covers the layered runtime and tooling interfaces, INT8 and BF16 deployment trade-offs, validation practices, model coverage requirements, developer tooling, and cloud-edge collaboration for AI PC applications.

    1. Why the Software Stack Is the Hard Part of On-Device AI

    An AI PC is a personal computing device with an integrated neural network acceleration unit (NPU). Local inference offers three main benefits: sensitive data does not need to be uploaded to the cloud, latency fluctuations caused by network round trips are reduced, and workloads can still run when offline or when cloud quotas are limited.

    An NPU does not guarantee that every operator can be accelerated. If a layer is not supported by the NPU, the runtime falls back to the CPU or GPU, and overall performance may actually decrease. As a result, real-world results depend not only on hardware specifications but also on whether the software stack can efficiently compile, quantize, and deploy models to heterogeneous hardware.

    2. Layers and Interfaces of the On-Device AI Software Stack

    A typical on-device AI software stack can be divided into five layers:

    Layer Main responsibilities Developer-visible interfaces
    Hardware abstraction layer Connect to CPU, GPU, and NPU; abstract driver differences Device enumeration and capability queries
    Runtime and scheduling layer Operator scheduling, memory allocation, multitask concurrency Session creation, thread and memory parameters
    Model compilation and quantization layer Convert model formats; generate INT8 or BF16 precision versions Compilation options, quantization configuration, calibration data entry points
    API and tooling layer Unified inference interfaces, model conversion and performance analysis tools Inference interfaces, profiling reports
    Application integration layer Integrate with languages and frameworks commonly used by developers SDKs and sample projects

    The benefit of a layered design is that upper-layer calling code usually does not need to change when switching hardware or precision configurations. A recommended practice is to validate accuracy with BF16 first, then switch to INT8 to compare performance and accuracy. However, numerical results differ across configurations, so validation must be repeated after each switch.

    3. Low-Precision Deployment: Trade-offs Between INT8 and BF16

    INT8 quantization compresses weights and activations into 8-bit integers, reducing memory usage and improving throughput. It is suitable for bandwidth-constrained CNN models and some LLM decoding scenarios. It requires a calibration dataset to determine quantization parameters. A calibration set usually contains 100 to 500 samples, should come from the real business distribution, cover long-tail cases such as long texts and abnormal inputs, and must use exactly the same preprocessing pipeline as inference. In addition, check the vendor’s operator support table to confirm whether the target NPU provides corresponding INT8 kernels. Unsupported operators fall back to other execution units, reducing the benefit.

    BF16 (bfloat16) has a dynamic range close to FP32 and lower accuracy risk, making it suitable for generative tasks that require numerical stability. Its bandwidth benefit is usually smaller than that of INT8, and some NPUs have limited native BF16 support, so this must be confirmed in the device documentation.

    The criteria for accuracy regression should be defined in advance:

    • CNN classification tasks: Use FP32 results as the baseline and compare the drop in Top-1 accuracy. The threshold is determined by the business side; for example, it may be set to no more than 1 percentage point.
    • LLMs: Compare perplexity on a held-out text set, run task-level metrics such as question-answering accuracy and code test pass rate, and manually spot-check generated content.
    • All models: Record end-to-end business metrics, latency, and memory usage, and compare them together with accuracy metrics.

    If INT8 accuracy does not meet the target, keep numerically sensitive layers in BF16 or FP16, which is mixed precision. If accuracy still does not meet the target, fall back to BF16 entirely rather than forcing INT8.

    The following code shows the structure of a three-stage flow. It is for illustration only and cannot be run directly:

    1. # Illustrative code: not runnable as-is; shows the structure of the three-stage flow
    2. # Dependencies: Python 3.10+; the vendor runtime SDK version must match the driver version, subject to the official compatibility table
    3. # Input: input_ids, shape [1, L], dtype=int64, L is dynamic length, upper limit subject to model configuration
    4. # Calibration data: .jsonl files under calib_samples/, one tokenized sample per line, 200 samples recommended
    5. # Target device: NPU; device name subject to runtime enumeration results
    6. from ai_runtime import ModelCompiler, InferenceSession # placeholder module name
    7. compiler = ModelCompiler()
    8. quant_model = compiler.quantize(
    9. model="chat_model.onnx", # must first be exported to ONNX; opset version subject to tool requirements
    10. precision="bf16", # use bf16 first to establish an accuracy baseline, then switch to int8 for comparison
    11. calib_data="calib_samples/"
    12. )
    13. engine = compiler.compile(quant_model, target="npu")
    14. session = InferenceSession(engine)
    15. output = session.run(input_ids) # input_ids: np.ndarray, shape [1, L], dtype=int64

    Common errors and how to handle them:

    • Data type mismatch: The input is int32 while the model expects int64, causing a runtime type error. Check dtype consistently at the entry point.
    • Calibration samples do not match inference input shapes: Different preprocessing causes quantization parameters to shift, leading to accuracy degradation without a clear cause.
    • Operator fallback: Compilation logs list operators that fall back to the CPU. Evaluate each one to determine whether it affects overall latency.

    Replace the placeholder names in the example with the real interfaces from the vendor documentation, then execute according to the input conventions above.

    4. Model Coverage and Support Matrix

    A software stack for AI PCs needs to support three types of models: LLMs, NLP encoders, and CNNs. The value of a unified API is that one set of interfaces covers all three workload types, but their compiled artifacts and optimization priorities are not the same.

    Model type Typical input Key mechanisms Main limitations Validation focus
    LLM Token ID sequence, dynamic length KV Cache management, token-by-token decoding, attention operator optimization KV Cache usage grows with sequence length; dynamic shapes reduce compile-time optimization opportunities Time to first token, output tokens per second, memory usage under long contexts
    NLP encoder Token IDs and attention mask, batched input Batching, sequence length bucketing Variable-length sequences reduce batching efficiency Throughput at different batch sizes, latency distribution at different lengths
    CNN Fixed-size image tensor, e.g. [1, 3, 224, 224] Operator fusion, Winograd convolution, static shape optimization Input size changes require recompilation or bucketing Per-layer time, NPU coverage, classification or detection metrics

    The model support list is subject to the versions published by the vendor. Developers should record the model version, quantization precision, runtime version, and driver version for each validated model to build an internal inventory. Combinations without records should not be treated as directly usable.

    5. Usability Tools: From Environment Checks to Profiling

    Newer software stacks invest heavily in developer experience, mainly in four areas.

    Time-to-First-Inference refers to the time from downloading the software stack to obtaining the first inference result. It measures the barrier to installation and configuration. Time-to-First-Token (TTFT) is the time from submitting input to outputting the first token in LLM inference. It measures response speed for generative tasks. The two metrics have different meanings and should not be conflated in reporting.

    The four areas work as follows:

    1. Environment self-check tools: Check hardware capabilities, driver versions, and runtime dependencies. Environment configuration failures are among the most common problems for beginners, mostly caused by driver version mismatches or missing runtime library paths.
    2. Model zoo and example hub: Provide validated precompiled models and end-to-end examples so developers can start by running examples rather than adapting from scratch.
    3. Performance profiling tools: Output per-layer timing to help determine whether a layer is worth migrating to the NPU. The approach is to measure single-layer latency on both NPU and CPU for the same model, then decide the migration scope.
    4. One-click scripts and interactive documentation: Reduce multi-step installation to a single command and show the expected output for each step in the documentation, making it easier to judge whether execution succeeded.

    6. Cloud-Edge Collaboration: Boundaries and Version Consistency

    On-device AI does not replace cloud AI. A common approach is to run lightweight, privacy-sensitive tasks on the local NPU, while large-model training and batch offline inference run on cloud GPUs. The cloud can handle training, quantization, and compilation, then distribute optimized models to devices for execution.

    This “cloud training, on-device inference” closed loop requires clearly defined boundaries:

    • Version consistency: Model artifacts, runtime versions, and driver versions must be in the compatibility table. Each distribution includes version information.