On-Device AI Development on AI PCs: From Model Quantization to NPU Deployment

On-device AI has rapidly become a new focus for developers. For years, AI inference followed a familiar pattern: applications sent user requests to remote servers, and results were returned after cloud-side computation. With the rise of AI PCs, that model is shifting. New-generation processors increasingly integrate an NPU (Neural Processing Unit) alongside the CPU and GPU, forming a heterogeneous compute foundation that makes it practical to run large language models (LLMs), natural language processing (NLP) models, and convolutional neural networks (CNNs) directly on local devices.

The value of on-device inference can be understood across four dimensions:

• Privacy and compliance: sensitive data never leaves the device, naturally satisfying data localization requirements.
• Low latency: eliminating network round trips enables millisecond-level first-token responses and a noticeably better interactive experience.
• Predictable cost: inference happens locally, avoiding usage-based cloud billing.
• Offline availability: functionality is not degraded without network access, making it suitable for mobile productivity, automotive, and industrial scenarios.

For developers, this introduces a new technical path to master: how to efficiently quantize, compile, and deploy models onto heterogeneous on-device compute.

The Layered Structure of an On-Device AI Software Stack

A mature on-device AI software stack typically consists of five layers, from top to bottom:

1. Model layer: pre-trained models, quantized model artifacts, and the associated tokenizer and pre/post-processing logic.
2. Graph optimization and compilation layer: responsible for operator fusion, constant folding, layout transformation, and compiling the compute graph into efficient instructions for specific compute units.
3. Inference runtime: handles task scheduling, memory management, multithreading, and streaming output interfaces. This is the programming surface applications interact with directly.
4. Driver and hardware abstraction layer: hides differences among CPU, GPU, and NPU, providing a unified interface to upper layers.
5. Toolchain layer: provides development utilities such as environment detection, model conversion, quantization calibration, and performance profiling.

Understanding this layering is critical. When inference performance falls short of expectations, the problem can exist at any layer: the model may contain unsupported operators, compilation settings may be misconfigured, or driver and runtime versions may be mismatched. Isolating issues by layer is the fundamental approach to troubleshooting.

Deployment Differences Across LLMs, NLP Models, and CNNs

Although unified software stacks claim support for multiple model types, the resource characteristics and optimization priorities of these three categories differ considerably.

LLMs (large language models) typically have billions of parameters, making memory capacity and bandwidth the primary bottleneck. Key on-device practices include selecting compact models in the 1B–10B parameter range, compressing model size through quantization, managing KV cache growth as sequence length increases, and using streaming decoding to improve perceived latency.

NLP models are predominantly encoder-based and cover tasks such as text classification, named entity recognition, and sentence embeddings. They have moderate computational density and strict latency requirements. These models benefit significantly from INT8 quantization, and post-training quantization is often sufficient to maintain accuracy.

CNNs (vision models) handle tasks including image classification, object detection, and image segmentation, with an emphasis on throughput. Vision workloads are frequently always-on, such as continuous camera analysis, making them well suited to sustained NPU utilization. Power consumption and thermal constraints require particular attention.

INT8 or BF16: Engineering Trade-offs in Precision Selection

INT8 and BF16 are currently the two most common low-precision formats for on-device deployment. They are not strictly better or worse than each other; they represent trade-offs for different scenarios.

Characteristics of INT8:

• Model size and memory footprint drop significantly, greatly reducing memory bandwidth pressure.
• NPU peak throughput under INT8 is typically the highest among supported precisions.
• The cost is that post-training quantization requires a calibration dataset, and some sensitive layers may experience accuracy degradation, requiring mixed-precision fallbacks.

Characteristics of BF16:

• With an 8-bit exponent and 7-bit mantissa, its dynamic range is close to FP32, offering good numerical stability.
• Training and inference precision are consistent, requiring no calibration data and minimal migration effort.
• The trade-off is that model size and bandwidth requirements remain higher than INT8, and throughput gains are relatively limited.

Practical decision guidance:

• For numerically sensitive workloads such as risk assessment or domain-specific Q&A: prefer BF16 or mixed precision.
• For throughput-oriented scenarios on memory-constrained devices: prefer INT8.
• When calibration data is unavailable or iteration cycles are tight: start with BF16 for faster rollout, then gradually introduce INT8.
• When maximum accuracy is required: consider quantization-aware training (QAT), though it demands additional training resources.

End-to-End Development Workflow: From Model to Application

The following typical pipeline illustrates the complete on-device deployment process.

Step 1: Model acquisition and format unification. After training is complete in the training framework, the model is exported to a common exchange format. The training and conversion stages are computationally intensive and well suited to cloud execution. For example, cloud platforms such as Baidu AI Cloud can handle heavy computation through their model training and inference services, while the device side is responsible only for final inference. This creates a sensible “cloud training, edge inference” division of labor.

Step 2: Quantization and compilation. The following is illustrative pseudocode:

# Pseudocode: post-training quantization and compilation workflow
model = load_model("exported/nlp-small")            # Load the exported model
calib = build_calibration_set(samples=512)          # Build a calibration dataset
q_model = post_training_quantize(model, dtype="int8", calib_data=calib)
compiled = compile(q_model, target="npu")           # Compile for NPU
save(compiled, "artifacts/nlp-small_int8")

Step 3: Application integration. Runtimes typically provide automatic scheduling, letting the system decide whether operators run on the CPU, GPU, or NPU:

# Pseudocode: loading and streaming inference within an application
runtime = create_runtime(device="auto")             # Automatic scheduling across heterogeneous compute
session = runtime.load("artifacts/nlp-small_int8")
for token in session.generate(prompt, stream=True):
    ui.append(token)                                # Streaming output improves perceived latency

Step 4: Validation and tuning. This involves three actions: accuracy alignment against an FP32 baseline using metrics such as accuracy or perplexity; performance profiling to identify the most time-consuming operators and utilization levels across compute units; and fallback validation to confirm that unsupported operators correctly fall back to CPU or GPU execution rather than failing outright.

Usability Tools: A Key Variable in Development Efficiency

Recent iterations of on-device AI software stacks have generally shifted their focus from “can it run” to “is it easy to use.” Tooling capabilities that genuinely improve developer productivity include:

• Environment self-check: one-click detection of whether driver, firmware, and runtime versions are compatible, reducing time spent troubleshooting environment issues.
• Sample model library: ready-to-use example models and benchmark scripts that enable first inference validation within minutes.
• Visual profiling: graphical display of per-layer latency and compute unit utilization for rapid bottleneck identification.
• Compatibility matrix: explicit documentation of supported model architectures, operators, and precision combinations, helping avoid risks during model selection.

When evaluating an on-device AI software stack, these tooling capabilities should be given the same weight as “which models are supported.”

Best Practices and Common Pitfalls

Drawing from real-world deployment experience, the following points are worth keeping in mind:

1. Complete a minimal end-to-end pipeline first. Use the smallest example model to walk through export, quantization, compilation, and inference before swapping in the business model. This avoids getting stuck debugging large models early on.
2. Beware of “fake acceleration.” No matter how powerful the NPU is, overall throughput will not improve if preprocessing steps such as image decoding or tokenization remain blocked on the CPU. Build pipelined parallelism between preprocessing and inference.
3. Estimate peak memory in advance. LLM KV cache grows linearly with context length, so memory must be budgeted for the maximum sequence length. Otherwise, long-context scenarios will trigger out-of-memory errors.
4. Validate accuracy on long-tail samples. Quantization errors often surface on low-frequency or extreme-distribution samples. Testing with only a small number of examples is insufficient.
5. Account for thermal constraints under sustained load. Portable devices under continuous full load will throttle. Performance testing should include extended stability stress tests.
6. Upgrade the stack as a matched set. Drivers, firmware, and runtime have dependency relationships. Keep the entire software stack version-aligned during upgrades to avoid compatibility issues.

Conclusion

On-device AI development is fundamentally a collaborative design effort across models, compilers, and hardware. Developers need to understand the compute characteristics of hardware such as NPUs while also mastering the software methods of quantization, compilation, and scheduling. A pragmatic technical path is to prototype quickly with a small model, establish accuracy and performance baselines, then gradually introduce quantization and heterogeneous tuning, ultimately producing a scalable on-device AI application. As software stacks and toolchains continue to mature, the barrier to entry for on-device AI development is dropping quickly. Now is an excellent time to invest in this direction.