A Practical Guide to AI-Driven Co-Design for Custom Chips

Why Bring Models and Chips to the Same Design Table

General-purpose accelerators have reached high maturity in throughput, energy efficiency, and interconnects. However, when model architectures, precision strategies, sparsity modes, and inference batch sizes continuously evolve, relying solely on general hardware often leaves clear performance margins in memory bandwidth, on-chip memory hierarchy, and operator fusion. The core of co-design is not about building a more expensive chip from scratch, but rather feeding model-side computational graph characteristics into hardware constraints as early as possible: which operators are worth hardening, which precision can be pushed down to dedicated data paths, and which communication patterns must be prioritized in the Network-on-Chip (NoC).

For training, the focus is usually on gradient synchronization, activation recomputation, and mixed-precision stability under large-scale parallelism. For inference, it is about tail latency, KV cache layout, and numerical recoverability after quantization. Treating both types of workloads as inputs for chip specifications prevents mismatches, such as “training chips struggling with online inference latency” or “inference chips unable to sustain training throughput.”

What to Define First When Building a Small Chip Design Team

An internal team of a dozen people is not suitable for covering the entire chain from process to packaging. A more pragmatic division of labor includes: Spec and Architecture (compute density, on-chip SRAM capacity, interconnect topology), Microarchitecture and Data Paths (matrix engines, vector units, dedicated decoders), and Toolchain and Verification (RTL generation assistance, formal checks, simulation regression). The model team needs to send engineers familiar with operator implementation and distributed strategies to share the same “workload profile” with the chip team, rather than handing over a model card as an afterthought.

Team goals should be written as verifiable metrics: training step time reduction at equal precision, inference P99 latency, effective compute per watt, and the calendar time from spec freeze to a synthesizable netlist. Without these anchors, AI-assisted design can easily devolve into “generating a bunch of seemingly reasonable RTL drafts.”

How LLMs Integrate into Chip Engineering Workflows

In traditional chip flows, specification documents, microarchitecture tradeoffs, RTL coding, test stimuli, and coverage convergence are serial and labor-intensive. LLMs are better suited for three categories of work:

First, candidate expansion from spec to microarchitecture. Given an operator set, precision, and memory access characteristics, the model can enumerate multiple data path partitions (for example, splitting matrix multiplication in attention and the reduction near softmax into different units) and provide qualitative tradeoffs on area, power, and latency for architects to filter.

Second, repetitive RTL and test scaffolding generation. For highly regular pipeline stages, FIFOs, arbiters, and interface adaptation layers, having models generate the initial draft for engineer review is often faster than starting from scratch. The key is to write coding standards, interface protocols, and prohibitions into the prompt constraints to avoid generating non-synthesizable structures or structures where timing cannot close.

Third, verification loop acceleration. Coverage holes, missing assertions, and insufficient boundary stimuli are schedule killers. The model can fill in directed test cases based on existing test plans and map failed waveform summaries back to potential state machine defects, shortening debugging time.

It must be emphasized: model outputs must enter existing EDA and verification pipelines rather than replacing sign-off. Formal properties, timing constraints, power sign-off, and process rule checks remain the final thresholds.

Key Technical Decisions in Co-Design

Alignment of Computational Graphs and Data Paths. First, perform operator profiling on the target model: matrix multiplication proportion, reduction and transpose frequency, and whether activation functions can be fused. High-proportion, stable-shaped operators are suitable for hardening; parts with variable shapes or low frequency should retain programmable vector paths to prevent the chip from becoming obsolete quickly on next-generation models.

Memory Hierarchy. The training side has large activation and optimizer state volumes, while the inference side is sensitive to cache locality. On-chip storage should distinguish between “software-managed scratchpad” and “hardware auto-prefetch,” and clearly define the ratio to off-chip bandwidth. Quantization and sparsity change effective bandwidth requirements, which must be simulated with real trajectories at the spec stage rather than relying on theoretical peaks.

Precision and Numerical Strategy. Mixed precision, block floating point, and quantization-aware training (QAT) require hardware to provide sufficient format support and overflow protection. Co-design means the model side can proactively choose “hardware-friendly” quantization grids rather than waiting for software compensation after the chip is finalized.

Interconnect and Parallelism. Tensor parallelism and pipeline parallelism have different impacts on NoC latency and congestion modes. If the team serves both training and inference, the interconnect should support configurable topologies or at least multiple virtual channel priorities to prevent one workload from starving another.

Translating “Shortened Design Time” into a Manageable Process

Model assistance alone is not enough; the process must be shifted to short iterations:

1. Freeze a “golden workload” using a representative model subset, and run the same set of simulation metrics for every architectural change.
2. Spec changes must be accompanied by a regression checklist: which RTL modules are affected and which assertions need rewriting.
3. Establish manual review gates: safety-critical state machines, reset and clock domain crossings, and external interface protocols are not allowed to be purely auto-merged.
4. Bind the toolchain (compiler, runtime, operator library) with the RTL version to avoid hardware being half a year ahead of software and unable to go online.

Under advanced process nodes, the industry-common cost for a chip targeting large-scale models from spec to mass production can reach hundreds of millions of dollars. Shortening calendar time is not just about burning less money; it is about aligning the chip window with the model iteration rhythm. Internal teams using models compress the middle phase “from idea to verifiable implementation” rather than skipping tape-out and packaging testing.

Risks and Governance

Generative assistance introduces three types of risks: hallucinated interfaces, non-synthesizable structures, and cell inference mismatched with the process library. Governance measures include: enforcing citations from the internal IP list in prompts; using synthesis and equivalence checks as merge gates; and retaining manual written design documents for critical paths. At the security and supply chain level, chip specifications and model weights are both sensitive assets; access permissions, audit logs, and environment isolation should be as strict as those for training clusters.

Additionally, do not equate “drawing a chip version faster” with a closed business loop. If the compiler cannot map new operators to dedicated units, hardware advantages will remain in simulation reports. Therefore, the success criteria for co-design should simultaneously include: pre-silicon performance models, compiler coverage, and end-to-end latency/power of real models.

Practice Checklist

• Clarify the primary scenarios for training and inference; reject spec bloat for an “all-in-one chip.”
• Drive SRAM, bandwidth, and interconnect using workload profiles rather than defining peak compute first and working backwards.
• Use LLMs for spec expansion, RTL drafting, and verification completion; sign-off must still go through standard toolchains.
• Small teams should focus on architecture, critical data paths, and the software stack; peripheral IPs and process services should be outsourced or purchased as general engineering services.
• Use repeatable golden workloads and review gates to turn assisted generation into controllable acceleration rather than an untraceable black box.

When model architectures are still rapidly evolving, the competitiveness of chip design comes from “constraints entering early enough, verification loops being short enough, and software/hardware versions being aligned enough.” Embedding LLMs into this engineering chain essentially increases the density of spec convergence and implementation iteration, rather than replacing chip science itself with generated text.