Near-Memory Computing and Dual-Model Architecture: A New Compute Paradigm for On-Device Large Models

    On-device AI is hitting the limits of smartphone SoCs: power, thermals, and memory bandwidth can't support large-parameter model inference. This article breaks down a recently demonstrated AI compute terminal prototype built around three ideas—a fast/slow dual-model architecture for dynamic task routing, near-memory computing with wafer-level stacking and hybrid bonding that pushes bandwidth into the TB/s range, and a multi-chip system design with sustained 150W power delivery. It explains how these choices address the memory wall, what they mean for decode throughput and unified memory capacity, and the practical implications for developers building local AI capabilities.

    Near-Memory Computing and Dual-Model Architecture: A New Compute Paradigm for On-Device Large Models

    Background: On-Device AI Is Moving from “It Runs” to “It Works Well”

    Over the past two years, discussions about on-device AI have mostly centered on the NPU compute of phone SoCs—how many TOPS, how large a quantized model can run, and how many milliseconds the first token takes. But the industry has gradually converged on a consensus: constrained by power, thermals, and memory bandwidth, phones struggle to carry true large-parameter model inference.

    A recent AI compute terminal prototype shown by a vendor sparked widespread discussion precisely because it stepped outside the “phone SoC arms race” and addressed a more fundamental question: if we want to run models at the 10-billion-parameter scale or beyond locally on a fixed-form device, how should the compute system be designed?

    The prototype’s technical highlights can be summarized in three points:

    1. Fast/slow dual-model architecture: A large-parameter model (public information points to a 10-billion-parameter scale) and a 3B-class small model are deployed locally, dynamically scheduling tasks of different complexity through “fast/slow system switching”;
    2. Near-memory computing chips: Wafer-level vertical stacking and hybrid bonding place storage and compute logic side by side, using extremely dense copper interconnects to solve the memory wall, with near-memory bandwidth reaching the TB/s range;
    3. Multi-chip system-level collaboration: The main SoC handles general-purpose computing and low-latency tasks, while a dedicated AI accelerator handles heavy inference. The system sustains performance at the 150W level.

    The following sections analyze the architecture, the process technology, and the application implications.

    Fast and Slow Models: A Pragmatic System-Level Scheduling Paradigm

    Why Two Models?

    Large models are capable but slow and power-hungry to run. Small models (such as a 3B Dense model) respond quickly and can easily generate hundreds of tokens per second. In real workloads, the vast majority of user requests are lightweight—checking the weather, reformatting text, simple Q&A—and only a minority require deep reasoning.

    Deploying both models locally and adding a lightweight intent-classification and routing layer creates what is known as a “fast/slow system”:

    1. def handle_request(query):
    2. # The fast model classifies intent first, at very low cost
    3. task_type = fast_model.classify(query)
    4. if task_type in SIMPLE_TASKS:
    5. # Simple task: generate directly with the 3B model, sub-second response
    6. return fast_model.generate(query, max_tokens=256)
    7. else:
    8. # Complex task: switch to the large model for deep reasoning
    9. context = fast_model.summarize(query) # Optional: compress context
    10. return large_model.reasoning(context)

    This paradigm essentially borrows the human cognitive model of “intuition plus deliberation”—a dual-system approach where easy questions take the fast path and hard ones take the slow path. It delivers three clear benefits:

    • Sharply lower average latency: over 90% of everyday requests are handled by the small model within seconds;
    • Controllable power consumption: heavy inference is triggered on demand, avoiding the constant high-power draw of a large model;
    • Complete privacy: no request leaves the device, which is highly attractive for enterprise scenarios such as contract review and internal knowledge Q&A.

    Routing Decisions Are the Real Engineering Challenge

    The dual-model architecture looks simple, but the key to making it work in production is routing accuracy: sending a complex task to the small model causes answer quality to collapse, while sending a simple task to the large model wastes compute. In practice, teams typically adopt a hybrid strategy of “small-model confidence scoring + rule-based fallback + sampled review by the large model,” continuously calibrating routing thresholds with online feedback.

    The Memory Wall and Near-Memory Computing: Where TB/s Bandwidth Comes From

    The Fundamental Bottleneck of LLM Inference

    The autoregressive decode phase of a large model is a classic memory-bound workload: generating each token requires reading the entire set of model weights from memory. Take a 3B model quantized to INT8: the weights are roughly 3GB. With only 100GB/s of memory bandwidth, the theoretical ceiling is about 33 tokens/s—no matter how powerful the NPU is. This is the memory wall: compute capability is growing far faster than the ability to move data.

    From “Fetching Data from Afar” to “Computing Right Next to Memory”

    In traditional architectures, memory connects to the SoC over a bus across a significant physical distance, so both the latency and the energy cost of data movement remain high. Near-memory computing takes the opposite approach: stack memory and compute logic vertically and connect them directly with face-to-face hybrid bonding.

    The process details here are worth unpacking: a bonding pitch on the order of 1.4μm means the two wafer layers are interconnected directly through hundreds of millions of microscopic copper contacts—effectively replacing the “memory bus” with an entire dense array of copper connections. The benefits are threefold:

    • Extremely short distances: signal travel shrinks from centimeters to micrometers, reducing both latency and energy;
    • Extremely wide bandwidth: the number of parallel contacts far exceeds the pin count of a traditional bus, pushing bandwidth into the TB/s range;
    • Improved energy efficiency: data movement energy can drop by more than an order of magnitude.

    A quick back-of-the-envelope calculation: with 1.2TB/s of near-memory bandwidth, running a 3GB INT8 model gives a theoretical decode ceiling of roughly 400 tokens/s; achieving 300+ tokens/s in real engineering is reasonable. This aligns well with the ~330 tokens/s shown in public demos, corroborating the claim that bandwidth determines decode speed.

    The Leverage of Large Unified Memory

    Another advantage of the near-memory architecture is native support for large-capacity unified memory. Public information indicates such designs can scale to around 160GB, enough to host a model in the 200B-parameter range locally (with quantization and sparse activation techniques). This changes not just the performance numbers but the capability boundary of on-device AI—jumping from “runs a 7B” to “can host a near-flagship model locally.”

    Form Factor: Why an “AI Appliance” Rather Than a Phone

    The prototype uses multi-chip collaboration: the main SoC handles a 10-core-class CPU, general-purpose computing, and high-frequency low-latency tasks, while a dedicated AI accelerator focuses on large-model inference. The system sustains 150W of power delivery—a thermal and power budget completely beyond what a phone can handle.

    There is clear logic behind this form-factor choice:

    • Decoupled thermal and power design: a fixed device can use active cooling and independent power, so the chips don’t have to be designed around extreme power constraints;
    • Clear division of chip labor: general-purpose compute and AI inference are each optimized separately, avoiding the area and power compromises of forcing both onto a single chip;
    • Focused use cases: home/enterprise knowledge bases, local agents, and privacy-sensitive inference are natural fits for a desktop AI appliance.

    Challenges remain, of course: yield and cost for wafer-level stacking and hybrid bonding are still high; beyond the vendor’s own models, the level of adaptation and optimization for third-party open-source models will determine the ecosystem ceiling; and the fast/slow scheduling framework needs a mature software stack to support it. These are the key signals to watch in judging whether this class of product can scale.

    What This Means for Developers and the Industry

    For developers, this trend signals several things:

    1. Device-cloud collaboration becomes the default architecture: a fast model on the device handles high-frequency lightweight tasks, while complex tasks call a cloud flagship model through a large-model service platform (such as the LLM services offered by Baidu AI Cloud), forming a layered “fast on-device, deep in-cloud” system;
    2. Inference optimization skills rise in value: quantization, KV Cache management, speculative decoding, and batched scheduling matter far more than training-side skills in bandwidth-constrained on-device environments;
    3. Evaluate hardware by bandwidth, not just compute: when selecting on-device hardware, memory bandwidth, unified memory capacity, and supported quantization formats predict real inference experience far better than TOPS figures.

    Conclusion

    The real value of this prototype lies not in the specs of any single chip, but in the clear technical path it demonstrates: break through the memory wall with near-memory computing, balance capability and latency with a dual-model architecture, and unlock the power budget with a purpose-built system form factor. Competition in on-device AI is shifting from a “single-chip arms race” toward “compute system redesign”—a shift whose impact on the industry may be more profound than any individual chip. For teams planning to build local AI capabilities, now is the ideal time to understand the near-memory computing and fast/slow scheduling paradigms.