Dual MIPI Cameras for Real-Time AI Inference on a Domestic Operating System

    This article describes a deployable approach for dual MIPI camera access and real-time AI inference on a domestic operating system with a multi-core application processor and integrated NPU. It explains a four-layer architecture and data flow, then covers independent dual-camera capture, bandwidth budgeting, synchronization, resolution tiering, and exception recovery. It also outlines compatibility-layer migration, NPU model deployment with INT8 quantization and asynchronous multi-stream inference, performance tuning, and typical deployment scenarios. The design targets 4K@30FPS capture with 30FPS-level analysis on one stream and about 25FPS-level detection inference on a high-pixel stream, with asynchronous processing or timestamp-aligned fusion.

    1. Background and Design Goals

    During upgrades and replacements in domestic AI vision projects, enterprises usually face three requirements at the same time. First, existing mobile applications and business UIs need to migrate as smoothly as possible to avoid large-scale rewrites. Second, vision algorithms need to move from the cloud or server side to the edge to meet low-latency, low-bandwidth, and privacy-protection requirements. Third, front-end cameras often include more than one stream, so the system must simultaneously connect a high-resolution global view and a high-pixel detail view to cover a larger monitoring area and finer target features.

    Connecting dual MIPI cameras and running object detection, segmentation, or pose estimation models in real time places high demands on the system: sufficient capture bandwidth, efficient memory access, NPU inference scheduling matched to dual data streams, stable operating system and drivers, and application-layer compatibility with the existing ecosystem. This article presents a deployable technical solution based on a combination of a domestic operating system, a multi-core application processor with an integrated NPU, dual independent MIPI CSI DPHY interfaces, and compatibility-layer migration.

    The design goals include dual independent capture, low-latency inference, deployable multiple models, migratable existing applications, and stable long-duration operation. Typical metrics can be set as follows: one stream captures 4K@30FPS and performs 30FPS-level real-time analysis, while the other high-pixel camera achieves about 25FPS-level detection inference. The two streams can be processed asynchronously or aligned by timestamp for fusion.

    2. Overall Architecture and Data Flow

    The overall architecture can be divided into four layers.

    Hardware layer: A general-purpose embedded application processor with an integrated NPU provides dual independent MIPI CSI DPHY interfaces, connecting a high-resolution global camera and a high-pixel detail camera separately. The high-resolution camera handles the 4K-level global view, while the high-pixel camera handles detail close-ups, forming complementary visual sources. The hardware layer also needs attention to ISP capability, memory bandwidth, storage interfaces, display interfaces, and thermal design.

    System layer: A domestic operating system provides camera drivers, ISP tuning, the NPU inference runtime, and network and storage adaptation. A compatibility layer supports existing mobile application packages directly and is compatible with Web front-end applications, reducing migration cost.

    Algorithm layer: Mainstream single-stage object detection models, instance segmentation models, and pose estimation models are deployed. Models are exported through a common intermediate format, optimized at the graph level, calibrated for INT8 quantization, and then accelerated by the local NPU. Preprocessing and postprocessing can be assigned to the CPU, GPU, or dedicated accelerators to avoid bottlenecks.

    Application layer: The application layer handles detection result fusion, alarm reporting, video encoding, storage, and network transmission. For advertising displays, self-service terminals, information publishing terminals, and similar devices, the application layer also needs to balance UI smoothness and background inference stability.

    A typical data flow is: sensor capture → MIPI CSI → ISP processing → memory buffer → preprocessing → NPU inference → postprocessing → dual-stream fusion → business output. In engineering practice, zero-copy, double buffering, and asynchronous inference should be used as much as possible to reduce memory copies and thread waiting.

    3. Key Points for Dual MIPI Camera Access

    The first requirement for dual-stream access is interface independence. Two MIPI CSI DPHY interfaces connect two sensors separately, avoiding bandwidth contention and timing coupling caused by a shared link. For 4K@30FPS and high-pixel full-resolution output, each stream already carries a large data volume. When both run in parallel, bandwidth budgeting must cover MIPI lane configuration, memory write bandwidth, ISP processing capacity, and encoding and display usage.

    The second point is synchronization strategy. If the business requires dual-stream fusion, hardware triggering or a unified timestamp mechanism should be preferred. If the two streams are only analyzed independently, asynchronous capture and asynchronous inference can be used, with queues decoupling the stages. Exposure, gain, and autofocus policies for different sensors should be tuned separately to prevent frequent zooming on one stream from affecting the stability of the other.

    The third point is resolution and frame-rate tiering. The global stream can maintain 4K@30FPS capture, but it can be scaled to an appropriate input size before being fed to the model. The detail stream can perform ROI cropping based on the target area to reduce invalid computation. For detail scenarios such as license plates and faces, the high-pixel original image can be used for snapshots and secondary recognition, while real-time detection uses a downsampled image.

    The fourth point is exception recovery. During long-duration operation, the system needs to monitor camera disconnection, frame timeout, insufficient memory, and NPU queue buildup, and design reconnection, frame-drop policies, and log alarms so that a single-stream failure does not affect the basic functions of the other stream.

    4. Compatibility and Migration on a Domestic Operating System

    The key to deploying a domestic operating system is not simply replacing the OS, but ensuring business continuity. Running existing mobile application packages through a compatibility layer allows the original UI, business logic, and some SDK calls to continue working. Web front-end compatibility allows existing page assets to be reused. The recommended migration steps are as follows:

    1. Inventory application dependencies and distinguish pure business logic, platform-specific calls, and hardware-specific calls.
    2. Replace platform-specific interfaces for camera, storage, network, and display, preferring standard interfaces and compatibility-layer capabilities.
    3. Validate application signing, permissions, file paths, and upgrade mechanisms.
    4. Sink compute-heavy modules to the local NPU, assigning tasks such as detection, segmentation, and pose estimation to local inference.
    5. Run performance regression and long-duration stability tests, watching the CPU and memory overhead introduced by the compatibility layer.

    The compatibility layer is suitable for carrying UI and business orchestration. The real-time inference chain should use the native runtime as much as possible, and high-frequency image processing should be avoided inside the compatibility layer. On the driver side, camera, ISP, NPU, and display链路 adaptation must be completed, along with hardware and software compatibility verification.

    5. NPU Model Deployment Workflow

    Model deployment can be divided into six steps: model selection, export, conversion, quantization, graph optimization, and runtime loading. For detection tasks, mainstream single-stage detectors can be selected. For segmentation tasks, instance segmentation models can be selected. For pose estimation tasks, keypoint models can be selected. During export, a common intermediate format is used. The conversion stage performs operator fusion, memory layout adjustment, and NPU-friendly optimization. The quantization stage uses a calibration set that covers actual scenes, with emphasis on backlight, nighttime, rain and fog, occlusion, and small objects.

    Preprocessing usually includes letterbox, normalization, and color space conversion. Postprocessing includes threshold filtering, non-maximum suppression, coordinate restoration, and class mapping. For dual-stream inference, an asynchronous multi-stream mechanism is recommended so that the two NPU tasks alternate, while the CPU performs postprocessing and business reporting at the same time.

    A pseudocode example is shown below:

    1. func dual_camera_pipeline():
    2. cam0 = open_mipi_camera(phy=0, width=3840, height=2160, fps=30)
    3. cam1 = open_mipi_camera(phy=1, width=4224, height=3136, fps=30)
    4. detector = load_model("./detector.int8")
    5. while true:
    6. f0 = cam0.capture()
    7. f1 = cam1.capture()
    8. input0 = letterbox_normalize(f0, target=640)
    9. input1 = letterbox_normalize(f1, target=640)
    10. out0 = detector.infer_async(input0, stream=0)
    11. out1 = detector.infer_async(input1, stream=1)
    12. det0 = postprocess(out0)
    13. det1 = postprocess(out1)
    14. fuse_and_publish(f0, det0, f1, det1)

    In engineering practice, shared buffers and zero-copy queues should be used to avoid multiple copies per frame. NPU inputs should be memory-aligned. Reasonable affinity should be set for postprocessing threads. Maximum queue lengths and frame-drop policies should be configured for the dual-stream queues.

    6. Performance Results and Tuning Recommendations

    Under typical configurations, the high-pixel camera stream can achieve about 25FPS-level detection inference, and the 4K stream can capture at 30FPS and perform 30FPS-level real-time analysis. To reach this level, tuning should focus on the following areas:

    • Input resolution tiering: Use small input sizes for real-time detection, and use original images for snapshots and secondary recognition.
    • ROI cropping: Run inference only on regions of interest to reduce NPU and memory pressure.
    • Double buffering and asynchronous processing: Pipeline capture, inference, and postprocessing.
    • INT8 quantization: Improve throughput within an acceptable accuracy range, and fall back on sensitive layers when necessary.
    • Thread and frequency strategy: Set NPU and CPU frequencies and thermal policies reasonably to avoid frame-rate jitter caused by downclocking.
    • Bandwidth priority: Coordinate MIPI, memory, display, encoding, and storage usage to prevent contention.
    • Long-duration stability testing: Run continuously for hours to days, and collect latency distribution, frame-drop rate, and temperature statistics.

    If a project requires a cloud training and data closed-loop workflow, general capabilities such as object storage, container platforms, and logging services from Baidu AI Cloud can be used for data archiving, training job orchestration, and runtime monitoring. Real-time edge inference remains on the local NPU, forming a cloud-training, edge-inference collaboration model.

    7. Deployment Scenarios and Conclusion

    This solution is applicable to scenarios such as security surveillance, industrial quality inspection, automotive vision, self-service terminals, advertising displays, and information publishing terminals. Its core value lies in complementary visual sources from independent dual MIPI access, real-time inference guaranteed by the local NPU, lower migration cost through the domestic operating system and compatibility layer, and multi-model deployment covering detection, segmentation, and pose estimation tasks.

    During deployment, the principles should be: stable capture first, bandwidth budgeting first, asynchronous decoupled inference, and a lightweight compatibility layer. Through reasonable dual-stream scheduling, model quantization, and engineering tuning, smooth real-time AI inference with dual cameras can be achieved on a domestic platform, providing a reusable technical path for AI vision project upgrades and replacements.