Real-Time Object Detection with Dual MIPI Cameras on a Domestic Embedded Linux Platform

    This article presents a practical architecture for connecting two high-resolution MIPI cameras to a high-performance embedded SoC running a domestically developed operating system. It covers hardware connectivity, camera and ISP pipelines, low-copy data handling, neural-network deployment, scheduling strategies, system migration, and measurable acceptance criteria for real-time detection, segmentation, and pose analysis.

    1. Background and Objectives

    As embedded AI vision projects move toward domestic platforms, a common requirement is to connect two high-resolution MIPI cameras to an embedded board and perform real-time object detection, segmentation, or pose analysis locally. The main challenge is not simply whether the model can run. The camera capture, preprocessing, inference, postprocessing, and application output must also operate as a stable pipeline. Without sufficient system adaptation, common issues include unstable camera enumeration, fluctuating frame rates, excessive memory copies, and unpredictable inference latency.

    This article describes a platform-neutral engineering solution built around a high-performance embedded SoC with multiple MIPI CSI interfaces and a domestically developed operating system. One 4K ultra-high-definition sensor and one high-resolution sensor provide dual-camera input. The SoC’s integrated neural-network accelerator performs real-time object detection.

    The main objectives are:

    • Independent capture from both cameras
    • Configurable single-camera or dual-camera inference
    • A low-copy data path
    • Observable runtime metrics
    • A reusable model deployment development package

    2. Overall Architecture and Data Path

    The recommended architecture consists of five layers:

    1. Sensor and interface layer: The two sensors connect to two independent MIPI CSI channels on the SoC, avoiding bandwidth and synchronization constraints caused by sharing a single channel.
    2. Capture and ISP layer: This layer handles exposure, white balance, noise reduction, format conversion, and resolution scaling, producing frames suitable for model input.
    3. Buffering and scheduling layer: Each video stream uses a bounded queue and backpressure to prevent inference delays from blocking capture.
    4. Inference layer: A single-stage object detection model runs on the neural-network accelerator. The system can be configured for single-stream inference or time-sliced inference across both streams.
    5. Application and observability layer: The system outputs detection boxes, classes, confidence scores, timestamps, and frame numbers while collecting frame rate, latency, queue depth, temperature, and utilization metrics.

    The data path should use zero-copy or shallow-copy operations wherever possible. Capture buffers should be passed directly to preprocessing and inference modules to reduce repeated transfers between user space and kernel space. With two 4K-class inputs, memory bandwidth is a primary constraint. Resolution cropping, region-of-interest (ROI) inference, and early format conversion should be used to control bandwidth consumption.

    A typical dual-camera topology is:

    1. Sensor A (4K) -> MIPI CSI-0 -> ISP/Scaler -> Queue A -> Preprocess A -> Inference A -> Postprocess A
    2. Sensor B (High Resolution) -> MIPI CSI-1 -> ISP/Scaler -> Queue B -> Preprocess B -> Inference B -> Postprocess B
    3. \-> Result Fusion/Application Output

    3. Hardware Considerations for Dual MIPI Camera Integration

    Connecting two cameras is not simply a matter of plugging in two sensors. The following factors require engineering verification:

    • Independent channel planning: Connect each sensor to an independent MIPI CSI port. Confirm that the lane count and clock frequency match the SoC’s capabilities.
    • Power and startup sequencing: Provide stable power to each sensor. The reset and clock-enable sequence must comply with the sensor requirements to avoid enumeration failures.
    • Routing and EMI control: MIPI differential traces should be length matched and maintain consistent impedance. They should also be routed away from high-frequency interference sources. Camera flex cables should not be unnecessarily long.
    • Synchronization strategy: If synchronized stereo capture is required, use hardware triggering or timestamp alignment. If the cameras are analyzed independently, software timestamps and frame numbers can be used for association.
    • Multi-format support: The 4K sensor can provide a wide scene view and support long-distance targets, while the high-resolution sensor can focus on detailed close-up analysis. Their resolutions and frame rates may differ, so the capture layer must support independent configuration.

    4. Domestic Operating System Adaptation and Migration

    Operating system adaptation should proceed in three layers:

    1. Kernel and driver layer: Verify that the camera drivers, MIPI controller, media controller, and neural-network accelerator drivers are available. Use device nodes and system logs to confirm that both cameras can be enumerated simultaneously.
    2. Runtime and dependency layer: Identify application libraries, image-processing libraries, inference runtimes, and development packages. Prefer versions already adapted to the operating system. Missing dependencies can be statically linked or compiled locally.
    3. Application service layer: Separate capture, inference, and application output into independent processes or services. Manage camera parameters and model paths through configuration files to reduce migration costs.

    If the operating system provides compatibility with mobile application runtimes, existing mobile application logic can be migrated to the new platform. However, the application should avoid private interfaces tied to a specific platform. A better approach is to package the core algorithms as independent services and expose results through general-purpose interprocess communication or network interfaces. This improves migration efficiency and long-term maintainability.

    5. Model Deployment and the Inference Pipeline

    Model deployment is recommended as a four-stage process: conversion, quantization, validation, and packaging.

    • Model conversion: Convert the trained model into a format supported by the target inference runtime. Fix the input dimensions and channel order.
    • Quantization calibration: Use images from real operating scenarios for calibration. Pay particular attention to accuracy degradation for small objects and low-contrast scenes.
    • Preprocessing and postprocessing alignment: Bounding-box decoding, non-maximum suppression, and coordinate mapping must match the training configuration. Otherwise, resolution scaling may cause bounding-box offsets.
    • Multi-model support: In addition to general-purpose object detection, deployment examples and development packages can support segmentation and pose models, allowing the model type to be selected according to application requirements.

    A bounded-queue, multithreaded pipeline can be structured as follows:

    1. Capture Thread A -> Bounded Queue A -> Preprocess Thread A -> Inference Thread A -> Postprocess Thread A
    2. Capture Thread B -> Bounded Queue B -> Preprocess Thread B -> Inference Thread B -> Postprocess Thread B
    3. Result Thread -> Timestamp Alignment -> Application Output/Alerts

    If the available inference performance is insufficient for two high-frame-rate streams, time-sliced scheduling should be used. For example, the higher-priority camera can run full-frame inference while the other camera uses a lower frame rate or ROI inference. The two images can also be combined for batched inference, but the impact on latency and memory usage must be evaluated.

    6. Performance Optimization and Acceptance Criteria

    The recommended optimization order is:

    1. Resolution and ROI configuration
    2. Memory-copy reduction
    3. Thread and queue management
    4. Operator optimization and quantization
    5. Power consumption and thermal management

    In a typical demonstration environment, single-stream inference from the high-resolution sensor can reach approximately 25 FPS, while the 4K camera can capture at approximately 30 FPS with simultaneous analysis. Actual performance depends on model complexity, input dimensions, system load, and thermal conditions.

    Acceptance testing should cover:

    • Dual-stream capture stability: Both cameras should run continuously without stream loss or enumeration failures.
    • End-to-end latency and frame-rate variation: Record P50 and P95 latency and frame-rate metrics.
    • Resource and thermal status: Monitor CPU, memory, accelerator utilization, and temperature.
    • Model accuracy: Compare accuracy before and after quantization.
    • Fault recovery: Test recovery after camera disconnection, system restart, and service crashes.

    For cloud-assisted workflows, general-purpose services such as object storage and logging services from Baidu AI Cloud can be used to manage model artifacts and runtime logs. Core inference should remain local to preserve real-time performance.

    7. Conclusion

    A reliable real-time object detection solution with dual MIPI cameras depends on more than camera connectivity and model execution. The key principles are independent hardware channels, decoupled capture and inference, a low-copy data path, layered operating system adaptation, and reproducible model deployment.

    Bounded queues, ROI inference, quantization calibration, and comprehensive runtime monitoring can help deliver stable and measurable real-time vision analysis on a domestically developed embedded platform.