Homogeneous Fusion Chip Architecture for Edge AI: Unified RISC-V ISA for General Computing and LLM Inference

    Edge deployment of LLMs exposes a growing mismatch between general-purpose workloads and inference demands. The common CPU-plus-discrete-NPU approach brings fragmented software stacks, scheduling difficulty, and data-movement overhead. This article examines a homogeneous fusion architecture that uses a unified RISC-V ISA to integrate a high-performance general-purpose CPU cluster and an AI-first AI-CPU cluster under a single operating system. It covers the design rationale, RVA23 alignment, hardware-software co-design for bandwidth-bound LLM inference, OS scheduling and resource governance, the AI software stack, and remaining challenges. Readers gain a structured view of the architecture's trade-offs and its relevance to edge AI and robotics.

    1. Background: Why Edge AI Needs a New Architecture Paradigm

    As large language models (LLMs) move toward endpoint devices, edge systems face a growing computing conflict. On one hand, scenarios such as robots and intelligent terminals require strong general-purpose computing capability to support operating systems, business logic, and control tasks. On the other hand, LLM inference places requirements on memory bandwidth, matrix compute, and energy efficiency that go far beyond traditional CPUs. The conventional approach of combining a CPU with a discrete NPU has exposed several typical engineering problems:

    1. Fragmented software stacks. The CPU side runs the Linux ecosystem, while the NPU side often depends on vendor-private compilers, runtimes, and operator libraries. Model migration and debugging become costly.
    2. Difficult scheduling. General tasks and AI tasks are split across two hardware sets, making it hard for the operating system to perceive load uniformly. Resource utilization can fluctuate widely.
    3. Data-movement overhead. Frequent memory copies between the CPU and accelerator become a significant source of inference latency.

    To address these pain points, the industry has begun exploring a notable direction: the homogeneous fusion paradigm. On a single edge chip, it adopts a unified open instruction set architecture such as RISC-V, integrates two computing clusters with different roles, and runs them under the same operating system.

    2. Core Design of Homogeneous Fusion: One ISA, Two Cluster Types

    The key to homogeneous fusion is balancing unification with division of labor. In a typical design, the chip contains two clusters:

    • High-performance general-purpose CPU cluster: Handles the operating system, task scheduling, peripheral control, and traditional general-purpose computing, with emphasis on single-core performance, interrupt response, and ecosystem compatibility.
    • AI-first AI-CPU cluster: Essentially a programmable processor, but deeply optimized for AI workloads through microarchitecture and instruction extensions. Examples include strengthened vector/matrix execution units, larger on-chip cache and memory bandwidth, and support for low-precision data types, specifically serving AI computing such as LLM inference.

    This design delivers three direct benefits:

    1. Unified software. Both clusters follow the same ISA specification, and the compiler toolchain, kernel, and driver model remain consistent. The AI cluster can be used as a specially scheduled set of CPU cores rather than a black-box accelerator.
    2. Data zero-copy. Because the clusters share a unified memory subsystem and address space, model weights and activation data do not need to be moved between the CPU and accelerator, significantly reducing inference latency and power consumption.
    3. Elastic resources. The operating system can dynamically decide which cores are assigned to general tasks and which cores form an execution pool for AI inference, enabling fine-grained compute governance.

    3. RVA23 Alignment: An Ecosystem Choice for Edge Chips

    The rapid development of RISC-V has raised concerns about instruction set fragmentation. RVA23, as a standard profile for application processors, defines the instruction sets that hardware must and may implement, providing a stable contract for the software ecosystem.

    For edge AI chips, aligning with RVA23 matters at three levels:

    • Operating systems and compilers can adapt directly. Mainstream toolchains such as the Linux kernel and GCC/LLVM need only be compiled once against the standard profile to cover both the general-purpose cluster and the AI cluster, avoiding custom software branches for each chip.
    • Application portability is better guaranteed. Chips that follow the same profile have a basis for binary compatibility, so developers do not need to recompile the entire software stack when switching to another chip.
    • Extended instructions have a clear basis. The profile allows custom extensions, such as AI-oriented vector extensions, to be layered on top of the standard baseline. This preserves compatibility while retaining room for AI compute optimization.

    A simplified software hierarchy can be illustrated as follows:

    1. Application layer (LLM inference services / robot control / business logic)
    2. │
    3. Runtime layer (unified scheduler + AI inference runtime)
    4. │
    5. Kernel layer (single Linux kernel managing both cluster types)
    6. │
    7. Hardware layer (general-purpose CPU cluster + AI-CPU cluster, unified ISA and memory subsystem)

    4. Hardware-Software Co-Design for LLM Inference

    The core bottleneck for LLM inference at the edge is usually not peak compute, but memory bandwidth and memory capacity. During autoregressive decoding, each generated token requires reading the full model weights, making it a classic bandwidth-bound workload. In this scenario, co-design ideas for homogeneous fusion include:

    1. Compute side: The AI-CPU cluster strengthens vector processing for matrix multiplication and attention computation, and supports low-precision formats such as INT8 and FP8. This moves less data and performs more computation under the same bandwidth.
    2. Memory-access side: Large on-chip cache, sensible data layout such as weight pre-layout to improve cache hit rates, and kernel fusion that combines multiple operators to reduce intermediate activation reads and writes all help lower actual memory traffic.
    3. System side: The inference runtime works with the OS scheduler to guarantee cores and memory bandwidth for the AI cluster during inference, avoiding jitter caused by background general tasks. Memory tiering and page management strategies can also support the largest possible model under limited DRAM capacity.

    This three-layer coordination across operators, memory access, and scheduling is a decisive factor for making edge LLM inference practical. It also reflects the value of integrated hardware-software design: hardware stacking alone cannot solve bandwidth problems; compiler optimization and runtime scheduling must work together.

    5. OS Scheduling and Resource Governance

    Managing two cluster types under a single operating system makes scheduler design especially important. In practice, the following are usually considered:

    • Load classification: Tasks are marked as general-purpose or AI-type. General tasks are preferentially scheduled to the high-performance cluster, while AI tasks are assembled onto a subset of AI cluster cores for execution.
    • Compute quotas: A cgroup-like resource governance mechanism manages CPU time, memory bandwidth, and cache share quotas to prevent inference tasks from starving interactive tasks.
    • Energy-efficiency policies: The two clusters can adjust frequency and voltage independently. Under light load, the AI cluster can be powered down or run at reduced frequency; under heavy inference load, the power budget can be released in the opposite direction. This is especially important in battery-powered robot scenarios.
    • Fault isolation: As long-running intensive tasks, AI computations need to be isolated from critical system paths to prevent a single-point anomaly from affecting overall system stability.

    6. AI Software Stack: End-to-End from Model to Chip

    The AI software stack for homogeneous fusion typically includes, from top to bottom: model formats and quantization tools, graph-level compiler optimization, operator libraries and kernel generators for AI-CPU instructions, a unified runtime, and kernel-level drivers. Compared with a CPU-plus-NPU approach, the advantage is that the entire chain shares the same ISA infrastructure. Operator development can be done with standard compiler infrastructure rather than learning a completely new proprietary toolchain. For developers, this means shorter adaptation cycles and lower maintenance costs, and it also makes integration with open-source community ecosystems such as various inference frameworks more direct.

    7. Challenges and Outlook

    The homogeneous fusion paradigm is still evolving, and several challenges should be acknowledged: the peak compute of AI-CPU clusters still lags dedicated accelerators, and the gap must be narrowed continuously under energy-efficiency constraints; unified schedulers are complex, and policy tuning requires long-term accumulation; and AI instruction sets must continue to expand without breaking ISA compatibility.

    Overall, this architecture of unified ISA + dual clusters + single operating system offers edge AI and robotics a path that is friendly to the software ecosystem and manageable in engineering complexity. It turns AI capability from an external accelerator into programmable, schedulable native compute. As the RISC-V ecosystem and compiler technologies mature, this approach is expected to play an increasingly important role in edge intelligent devices. For engineering teams focused on endpoint LLM deployment, understanding the design trade-offs behind this paradigm provides direct reference value for compute selection and software architecture planning.