“Three-Day Integration”: The Real Engineering Bar for Heterogeneous Adaptation

Announcements like “a frontier model now supports training and inference across multiple accelerator architectures, and integrating one domestic chip took only three days” are easy to read as a simple compatibility demo. But if you treat the adaptation cycle as an engineering metric, what it really measures is infrastructure portability: how many layers of abstraction sit between the model code and the hardware, and how much logic is still hardcoded into a single vendor’s programming model.

For a team already running on a mature, closed-source stack, the cost of migrating one training job varies enormously with the goal:

• Inference bring-up only: usually days to two weeks, with effort concentrated on forward operators, memory layout, and numerical alignment.
• Prefill optimization only: operator fusion and memory access patterns must be retuned for the target hardware, on the order of weeks.
• Full training migration: usually measured in months, because the backward graph, collective communication, optimizer state, and parallel strategy all have to be rebuilt, and convergence has to be re-proven.

So the real question is not “can it run?” but: does the abstraction layer already exist? How are capability gaps discovered and closed? What is the verification bar?

Why Inference Is Easy to Port and Training Is Hard

Workload Characteristics Drive Adaptation Difficulty

Inference requests (especially in the prefill stage) have a fairly fixed computation graph, a convergent operator set, and high tolerance for forward-pass errors, so mistakes are cheap to recompute. More importantly, inference can be validated with live canary traffic, A/B comparisons, and SLA-based rollback, which makes the verification loop fast.

Training has a completely different workload structure:

Dimension	Inference / Prefill	Full Training
Computation graph	Forward-dominated, statically unrollable	Requires backward pass, including custom gradients
State	Stateless, re-computable	Optimizer momentum, second-order statistics, RNG state, data order
Communication	Per-request, fault-tolerant	Gradient synchronization and parameter/optimizer-state sharding, tightly coupled to topology
Numerical requirements	Small errors tolerated	Cross-device consistency directly affects convergence
Verification	Online canary rollout, request-level metrics	Sustained runs, loss-curve comparison, checkpoint recovery
Cost of failure	A single failed request	An interruption wastes large amounts of compute

In one sentence: inference is a stateless, re-computable workload, while training is a stateful, tightly coupled, long-running one. Adapting the latter means rebuilding an entire distributed training system on new hardware and re-proving its correctness.

Typical Breakpoints During Migration

A model originally ran on 1,024 accelerators of a certain type with a specific tensor-, pipeline-, and expert-parallel (TP/PP/EP) configuration. When it moved to a 512-card cluster with a different architecture, “just compiling the operators” was nowhere near enough:

1. Operator gap-filling: custom forward/backward operators were missing and had to be rewritten or replaced with equivalent compositions.
2. Numerical consistency: mixed-precision behavior differs across architectures, so the same weights can produce gradients with larger deviations.
3. Parallel-strategy breakage: with half the cards and a different interconnect bandwidth and topology, the original TP/PP/EP split lost significant performance.
4. Optimizer-state rebuild: momentum and second-order state must either be migrated (limited by memory and formats) or warmed up from scratch.
5. Communication-library differences: collective primitive semantics, bucketing strategies, and how compute-communication overlap is implemented all differ.
6. Convergence verification: only when the loss curve, gradient norms, and validation metrics stay within tolerance of the baseline should production training be switched over.

Layered Architecture of a Heterogeneous Compute Pool

For “write the model once, run it on any accelerator” to hold, you need clear abstraction layers above the hardware, not per-chip patches.

Application & serving layer: gateway, auth, rate limiting, quotas, observability
        ↑
Orchestration & scheduling layer: workload profiling, cost model, routing decisions, elastic scaling
        ↑
Distributed systems layer: parallel strategies, memory management, collectives, checkpointing
        ↑
Graph & compilation layer: graph capture, operator fusion, autodiff, target code generation
        ↑
Device abstraction layer: unified tensor semantics, memory hierarchy, streams & events, device enumeration
        ↑
Hardware layer: general-purpose parallel architectures, tensor-accelerator architectures, various domestic accelerators

What the Device Abstraction Layer Must Abstract

The goal is not to rename interfaces to a common set, but to unify semantics: tensors and their layouts, memory allocation and reuse, streams and events, synchronization primitives, device-to-device copies, and device capability queries (compute throughput, memory, interconnect bandwidth, supported precision formats). Capability queries matter especially because the scheduler must know whether a card supports a given precision format or has a high-speed interconnect before it can make correct decisions.

Illustrative device capability descriptor:

@dataclass
class DeviceCapability:
    dev_id: str
    peak_tflops_bf16: float      # peak compute throughput
    hbm_capacity_gb: int         # memory capacity
    hbm_bw_tbps: float           # memory bandwidth
    interconnect: str            # inter-chip interconnect type
    supported_dtypes: tuple      # supported precision formats
    reliability_index: float     # long-run stability history metric

The Compilation Layer: From Hand-Tuned Kernels to Searchable Generation

Cross-architecture performance does not come from running the same kernel everywhere. It comes from regenerating code for the target hardware. Industry practice follows three common paths:

• Graph-level compilation: lower the model’s compute graph into an intermediate representation, then optimize and generate code against the target’s operator library, fusion rules, and memory hierarchy.
• Kernel description language + autotuning: describe kernels in a hardware-agnostic way, let the compiler map them onto different architectures, and search parameters (tile sizes, register/shared-memory usage, vectorization width) to approach hand-tuned performance on the target device.
• Operator-library routing: dispatch high-frequency operators to vendor-maintained, highly optimized implementations, and fall back to the general generation path for the long tail.

In practice, the pragmatic approach combines all three: high-frequency operators go through vendor libraries for peak performance, while long-tail and novel structures use the general generation path to preserve coverage.

Scheduling and Routing: Turning “Which Card” Into a Computable Problem

The real payoff of multi-chip adaptation is not that a workload “runs,” but routing each request to the hardware with the best cost per unit of work, given its characteristics. Doing this in practice requires three inputs: workload profiling, a cost model, and constraint checks.

Workload Profiling

The gateway layer should record each request’s input length, output length, prefix-cache hit, and SLA tier, and predict output length with online statistics or a lightweight model. Without this profile, routing degenerates into round robin.

Cost Model

Cost per token can be modeled coarsely as “effective compute occupancy + memory/bandwidth occupancy” divided by the hardware’s hourly cost. Precision in the formula matters less than being explainable and regression-testable: every coefficient change can be replayed against historical traffic to confirm that cost and latency do not degrade.

Constraint Checks

Memory capacity, context window, minimum parallel degree, and maximum queue latency are hard constraints and must take precedence over cost minimization.

# Heterogeneous compute pool routing decision (pseudo-code, not a specific product interface)
def estimate_cost(pool: DevicePool, p: RequestProfile) -> float:
    # Prefill is approximately compute-bound; decode is approximately memory-bound
    prefill_flops = 2 * p.in_tokens * POOL_ACTIVE_PARAMS
    decode_bytes = p.out_tokens_est * POOL_ACTIVE_PARAMS * BYTES_PER_PARAM
    prefill_time = prefill_flops / (p.peak_tflops * p.mfu)          # seconds
    decode_time = decode_bytes / (p.hbm_bw * p.mem_bw_eff)          # seconds
    total = prefill_time + decode_time
    if p.sla_ms and total * 1000 > p.sla_ms:
        return float("inf")        # hard constraint violated; eliminate this candidate
    return total / p.hourly_cost   # lower is better: inverse effective throughput per unit cost
def route(profile: RequestProfile, pools: list[DevicePool]) -> DevicePool:
    feasible = [p for p in pools if has_capacity(p, profile)]
    if not feasible:
        return overflow_queue(profile)     # fallback: queue or throttle
    return min(feasible, key=lambda p: estimate_cost(p, profile))

Typical Routing Patterns

• Separate prefill and decode deployments: prefill is compute-intensive and batchable, so it fits pools with a lower cost per unit of compute and more abundant bandwidth. Decode is memory-intensive and latency-sensitive, so it fits pools with better long-run stability and lower latency jitter. This is the most direct place where inference-side heterogeneity pays off.
• Stage-based routing for training jobs: data preprocessing, hyperparameter search, and production training have different hardware needs and can be routed separately.
• Reliability as a scheduling input: use historical failure rates and clock throttling as soft constraints, so long-running jobs stay off unstable nodes.
• Overflow and fallback paths: when a pool is full, queuing, throttling, or falling back to a smaller model gives a much better experience than returning an error.

A Training Migration Engineering Checklist

Treating a cross-architecture training migration as a standard process rather than a one-off project is key to being able to repeat it quickly.

Stage	Check	Pass criteria
Operators	Custom forward/backward operators fully implemented	Numerically aligned with the reference implementation within tolerance
Precision	Mixed-precision strategy, overflow handling	No NaN/Inf in short runs, stable gradient norms
Parallelism	TP/PP/EP/SP re-searched	Effective compute utilization meets the target baseline
Communication	Collective correctness, compute-communication overlap	Results match at small and large scale
State	Optimizer state, RNG, data order	Loss curve continues seamlessly after checkpoint recovery
Long-run stability	Sustained run time, failure rate	No hidden precision or memory-growth issues
Regression	Loss-curve comparison against baseline	Loss deviation at key steps stays within a preset tolerance

Illustrative skeleton for a parallel-strategy search:

def search_plan(model, cluster, target_mfu=0.4):
    best, best_mfu = None, 0.0
    for tp in divisors(model.attn_heads, cluster.devices_per_node):
        for pp in divisors(model.layers, 4):
            for ep in divisors(model.experts, 2):
                if not fits_memory(model, cluster, tp, pp, ep):
                    continue
                mfu = profile_step(model, cluster, tp, pp, ep)  # run one real step to measure
                if mfu > best_mfu:
                    best, best_mfu = (tp, pp, ep), mfu
    return best if best_mfu >= target_mfu else escalate(best_mfu)

This kind of search must be driven by measured step times rather than analytical estimates. Otherwise, estimation error during a cross-architecture migration becomes very large.

Common Pitfalls and Best Practices

Pitfalls

• Declaring adaptation complete after only single-node validation: communication and memory issues in multi-node, multi-card setups are completely different.
• Watching throughput but not latency distribution: P99 latency regressions can break SLAs outright.
• Ignoring long-run stability: memory fragmentation, periodic jitter, and clock throttling often surface only after hundreds of hours.
• Inferring training feasibility from inference results: training migration is several times harder than inference, and the two are not comparable.
• Fragmentation spiraling out of control: the more chips you adapt, the larger the version matrix grows. Without unified observability and a unified release process, this quickly becomes an operations disaster.

Best practices

• Build a cross-architecture benchmark suite with a fixed model, fixed data shards, and a fixed step count. Any new hardware must pass it before optimization work begins.
• Codify adaptation work: version device capability detection, operator substitution, and parallel-strategy configuration to avoid manual patches.
• Build unified observability: normalize utilization, latency, and error codes across different hardware into a single set of monitoring, alerting, and logging standards, or cross-pool issues cannot be diagnosed.
• Use replay-based validation: before a routing policy or cost model goes live, replay historical traffic offline to confirm that cost and SLAs do not degrade.

AI in Kernel Optimization: Where the Capability Boundaries Lie

“AI writing operators that outperform human experts” is a much-discussed direction. Its value is turning kernel optimization from scarce, expert-dependent work into a scalable engineering search process: define a search space and a target hardware, have a model generate candidate implementations, then filter them with benchmarks and numerical tests.

But its limits need to be seen clearly:

• Generation cannot guarantee correctness: every candidate kernel must pass numerical alignment tests, edge cases, and a regression suite, with a human review step kept in place.
• Claims cannot guarantee performance: benchmarks must actually be run to prevent overfitting to the test set while production workloads regress.
• Recursive self-improvement has a higher bar: it requires the environment to provide a verifiable objective function and feedback signal. Without a reliable evaluation set and regression gates, automatic optimization easily degenerates into “looks better, performs worse.”
• The human role remains: defining acceptance criteria, setting safety boundaries, and deciding whether to merge a candidate implementation into the mainline are still engineering judgment calls.

So a more realistic goal is AI dramatically reducing the human cost of adaptation, not AI completing adaptation on its own.

Adoption Roadmap

1. Unify the abstraction: build the device abstraction layer for the inference path first, so any new model requires only configuration changes, not changes to core logic.
2. Multi-pool inference: bring in second and third classes of hardware, separate prefill and decode deployments, add routing, and validate the gains with the cost model and replay.
3. Single-device training validation: complete forward/backward numerical alignment and a performance baseline on the target hardware, on one device and one node.
4. Multi-device training migration: re-search the parallel strategy, adapt collectives, wire up checkpointing and resume, and prove convergence with loss-curve comparison.
5. Routine heterogeneous operations: fold benchmarks, routing policies, cost models, and observability standards into daily workflows, turning “integrating a new chip” from a project into a routine task.

Conclusion

“Full-chip adaptation” looks like a compatibility problem on the surface. In reality, it reflects the combined maturity of four capabilities: the abstraction layer, compilation, scheduling, and verification. Fast inference and prefill bring-up shows the abstraction layer is paying off. Full training migration remains hard because it requires rebuilding an entire distributed system and re-proving convergence. For teams constrained by compute supply and relying on diverse hardware sources, the value of a heterogeneous compute pool goes beyond saving money or sidestepping a single-vendor bottleneck. It turns compute into a schedulable shared resource: choose the most suitable hardware for each workload, and make controlled trade-offs among cost, latency, and stability. That makes it both a worthwhile infrastructure direction today and a prerequisite for higher-order automation later, since more advanced automation and AI-assisted development can only build on top of a stack where hardware differences have been abstracted away.