Frameworks Don't Lie: Reverse-Engineering Training Hardware from Model Weights

    Since 2025, a steady stream of open-weight large models has pushed one question to the foreground: what hardware was each one actually trained on? This article examines a technique that ignores official statements and model cards alike, instead downloading the weights and running numerical analysis across thousands of tensors to infer the architecture of the training accelerators. It explains the underlying principle — that non-associative floating-point arithmetic and mixed-precision training encode hardware-specific execution paths into the low-order bits of weights — then walks through the four sources of these "fingerprints," a four-step hypothesis-testing workflow with reproducible code, how to interpret statistical significance and scale estimates, the conditions under which the method breaks down, and what it all means for release governance.

    1. The Question: Weights Carry a “Hardware Fingerprint”

    Since 2025, several teams in China have released open-weight large models of varying sizes and steadily improving performance. As compute has become scarce, one question has drawn attention: what hardware was each of these models actually trained on? One approach that has emerged in public discussion sidesteps official statements and model-card descriptions entirely. Instead, it downloads the weight files, runs numerical analysis across thousands of tensors, and infers the architecture of the accelerators used for training. The models involved in the analysis all come from publicly or semi-publicly released large-model weights.

    The conclusions are surprising. Six new models released after April 2025 all showed “fingerprints” pointing to the latest generation of data-center accelerators. Two of them matched both a single-die and a dual-die packaging variant, while the other four pointed to a newer multi-die unified-packaging platform. As a control, the previous generation of models from these same teams (including a well-known open foundation model’s classic version) showed fingerprints that consistently pointed to the prior mainstream architecture. Set side by side, the two generations reveal a clear signal of a “generational transition.”

    The most compelling feature of this method is that it relies on no “official claim.” On models with known hardware, it can reconstruct the hardware accurately; on models where the cluster size has been disclosed, its estimates match the public figures; on new models, the statistical evidence is strong enough to be hard to explain as coincidence. For developers, this is more than gossip — it is a technical phenomenon worth understanding. In an era where distributed training and mixed precision are standard, the weight values themselves are becoming a “side channel” that leaks information about the hardware, the framework, and even the training strategy.

    1.1 The Principle in One Sentence: Hardware Is “Encoded” into the Low Bits of Weights

    Start with an analogy to build intuition. Suppose you represent the same decimal with 1, 8, and 53 significant bits:

    1. import numpy as np
    2. x = 1 / 3
    3. print(np.float16(x)) # 0.333251953125
    4. print(np.float32(x)) # 0.3333333432674408
    5. print(np.float64(x)) # 0.3333333333333333

    Fewer bits means a larger deviation from the true value, and the traces left by the rounding mode become more visible. Large-model training typically uses 16-bit floating-point formats (such as half precision and bfloat16) to store and transfer data, but accumulates and updates in 32-bit floating-point precision. As a result, every “downcast write-back” leaves a rounding trace. What that trace looks like depends on who executed the last floating-point operation and in what order — and the order and tiling are in turn determined by the specific hardware instructions and the GPU compute library implementation. The hardware is thus encoded into the low-order bits of the weights.

    2. Where the Fingerprints Come From: Four “Signature” Sources

    The most fundamental — and most easily overlooked — property of floating-point arithmetic is that it is not associative:

    1. (a + b) + c ≠ a + (b + c)

    This equation usually differs only in the last few digits, but once an intermediate result is truncated back to low precision, a tiny last-digit difference can be amplified into a visible numerical deviation. Matrix multiplication is precisely a large superposition of such operations. It splits the result into several partial sums, then accumulates them step by step into registers, and the final rounding pattern is determined jointly by the following factors: accumulation depth, tiling shape, number of registers, and which class of instructions the compute library on a given chip is optimized for. From this, four observable classes of “signature” emerge.

    2.1 Signature 1: Accumulation Order and Tiling Shape

    Matrix-multiply-add instructions on different architectures support different tiling shapes. If the same computation is executed on different architectures, the number and combination order of the partial sums change accordingly, and so does the probability of certain special value patterns appearing once the weights fall back to 16-bit precision.

    • Hardware condition: This involves only compute instructions and execution order, and can occur on GPUs of any architecture.
    • Fingerprint feature: The proportion and positional distribution of patterns such as “symmetric-symmetric bits” or “symmetric uniform values” within the weights.
    • Analysis approach: The features should be rare enough and quantifiable — for example, measuring the fraction of elements where “the weight equals an integer power of two” or where “the symmetric bits equal the remaining bits,” then comparing against each candidate architecture one by one.
    • Enhancement: Measure the regularity with which these elements align to some tiling shape within the tensor — for instance, whether the row/column distribution with a “symmetric plus symmetric” period deviates significantly from a random distribution.

    2.2 Signature 2: Reduction Paths and Swaps

    When a GPU executes matrix multiplication, thread blocks, sub-blocks, and individual threads each take on part of the accumulation, and the results are finally reduced and combined. The reduction method (tree-based, linear, or transferred between registers and shared memory) directly determines the addition-association order. In addition, swap computations often use different synchronization and data-exchange methods across architectures, and differences in the path also show up in the accumulation order. Meanwhile, whether an extra format-conversion step exists along the data path directly affects how error accumulates.

    • Fingerprint feature: Along a given reduction dimension, the low-bit error shows a regular “staircase” or “periodic” pattern whose period matches the grouping granularity of the hardware execution path.
    • Analysis method: Measure the error distribution from three perspectives — rows, columns, and blocks — then apply a periodicity test (such as autocorrelation or spectral peak analysis).

    2.3 Signature 3: Processing Precision and Matrix Instruction Paths

    • Processing precision: Training stores data in 16-bit floating point but performs matrix-multiply-accumulate in 32-bit floating-point precision, and some hardware paths automatically enable this strategy. This intermediate precision is not uniform: some hardware accumulators have a 22-bit mantissa, others use the standard 32-bit; certain matrix instructions that accept 8-bit inputs likewise only accumulate to 22 bits internally. The lower the intermediate precision, the larger the rounding error, and the more pronounced the “staircase” truncation traces left in the weights.
    • Matrix instruction paths: Some matrix units have a “sparse mode” that can treat a tensor as a set of micro-blocks, storing zero values only once and indexing them via metadata. If the weights go through this kind of structured-sparsity processing, the zero-value distribution carries a hardware-specific periodicity. Dense weights that have not been sparsified will not trigger this path — but that, in turn, forms a discriminating feature of “dense versus sparse.”
    • Analysis approach: If the inferred weight-storage format does not match the format actually used at deployment time, the conclusion should be down-weighted rather than accepted at face value.

    2.4 Signature 4: Compilation and Kernel Selection

    Hardware is not the only variable. On the same card, the version of the GPU compute library, the results of auto-tuning, and the operator-fusion strategy can all change the actual computation graph that gets executed. A more rigorous phrasing, therefore, is that the weights leak the fingerprint of the “post-compilation execution path,” not just the fingerprint of the chip model. In the extreme case, the same architecture can present different low-bit statistical features under different software stacks. This is precisely the deeper reason why subsequent cross-validation is indispensable.

    2.5 A Controlled Experiment: The Same Dot Product, Different Last Digits

    The following is a minimal experiment (pseudocode) demonstrating how the same set of inputs produces different results under different accumulation methods:

    1. import numpy as np
    2. rng = np.random.default_rng(0)
    3. a = rng.standard_normal((128,)).astype(np.float32)
    4. b = rng.standard_normal((128,)).astype(np.float32)
    5. def dot_chunked(x, y, chunks):
    6. """Perform FP32 accumulation with different chunking to emulate different hardware execution paths."""
    7. acc = np.float32(0.0)
    8. for idx in np.array_split(np.arange(x.size), chunks):
    9. acc = np.float32(acc + np.sum(x[idx] * y[idx], dtype=np.float32))
    10. return acc
    11. def to_fp16(x):
    12. return np.float16(x)
    13. # Same inputs, three accumulation paths
    14. r_ref = dot_chunked(a, b, chunks=1) # Accumulate all at once
    15. r_tile = dot_chunked(a, b, chunks=16) # Emulate a tiled accumulation
    16. r_deep = dot_chunked(a, b, chunks=128) # Accumulate term by term
    17. # Key point: after writing back to FP16, the three usually agree; a difference only becomes
    18. # amplified and observable when precision is dropped first and accumulation continues afterward,
    19. # or when partial sums are spilled
    20. print(to_fp16(r_ref), to_fp16(r_tile), to_fp16(r_deep))
    21. print(to_fp16(r_tile) == to_fp16(r_deep)) # Whether they agree depends on the hardware spill strategy

    This example shows that the visibility of the fingerprint depends on “when it gets truncated back to low precision.” If accumulation is done in 32-bit throughout and only converted to 16-bit at final output, differences between hardware are usually smoothed away by rounding. A pronounced fingerprint often means the training process involved repeated “accumulate → downcast → accumulate again” cycles, such as master-weight updates in mixed-precision training, gradient accumulation, cross-card reduction, or repeated quantization of activations. Conversely, if the weights are re-quantized or dequantized at inference time, the historical traces may be erased. Understanding this is a prerequisite for judging how reliable a conclusion is.

    3. The Method Framework: From Weight Files to Hardware Conclusions

    Summarized, the analysis in public discussion roughly follows four steps: establish candidate hypotheses, construct fingerprint features, run statistical significance tests, and cross-validate with samples that have known answers. It is essentially a “hypothesis testing plus model selection” problem.

    3.1 Step 1: Establish Candidate Hypotheses

    Enumerate the “possible training hardware” as a finite set, mapping each one to its compute-instruction shapes, matrix-unit capabilities, reduction paths, and common implementations of the matrix-multiply-add library. The candidate set must be as exhaustive as possible and cover combinations of “architecture generation + packaging form,” such as single-die, dual-die, and multi-die unified-packaging platforms. Omit a candidate, and the conclusion may suffer from systematic bias.

    3.2 Step 2: Construct Fingerprint Features

    Under each candidate hypothesis, simulate a pass over a typical operator from training according to that hardware’s computational behavior (or simply run it once on the real hardware), and record the statistical distribution of the weight tensors in their low bits as a “reference fingerprint.” Then compute the same statistics on the target model’s weights to obtain an “observed fingerprint.” Commonly used statistics include:

    • Symmetric-bit distribution (the ratio at which the symmetric bits equal the remaining bits);
    • The proportion of symmetric values (where the symmetric value is exactly an integer power of two);
    • The degree of alignment of weights after being blocked into “symmetric plus symmetric”;
    • The low-bit residual distribution of gradients or activations along a given dimension;
    • If the weights carry tensors related to optimizer state, these can also serve as auxiliary evidence.

    3.3 Step 3: Statistical Testing

    Compare the observed fingerprint against each candidate’s reference fingerprint, typically using two kinds of tests:

    • Goodness-of-fit test: Such as the chi-square test, measuring the degree of deviation between the observed distribution and a given candidate’s distribution.
    • Hypothesis significance: Taking “the weights come from random initialization or from unrelated hardware” as the null hypothesis, compute how many standard deviations the deviation value corresponds to.

    3.4 Step 4: Cross-Validation

    Before the method is put into use, run a retrospective test on models with “known hardware.” This is a form of leave-one-out validation: whenever you judge a model, you must exclude the circular reasoning of “using its own answer.” Only when the method is correct on all known samples does its output on new samples carry any credibility. In addition, you can apply an “internal consistency” check: take a model’s weights already judged to belong to a certain architecture, perform a simulated “transcode” according to another architecture’s behavior, and see whether the algorithm can detect the artificially implanted fingerprint.

    3.5 A Reproducible Analysis Skeleton

    The code below gives a minimal runnable skeleton, using a chi-square statistic to measure the deviation between the “low-bit distribution of weights” and a candidate reference distribution:

    1. import numpy as np
    2. def lowbit_histogram(tensors, shift=8, mask_bits=8):
    3. """Measure the 0/1 distribution of the low bits of FP16 weights after stripping the high bits."""
    4. counts = np.zeros(2, dtype=np.int64)
    5. for w in tensors:
    6. u = np.ascontiguousarray(w, dtype=np.float16).view(np.uint16).ravel()
    7. bit = (u >> shift) & 1
    8. counts += np.bincount(bit, minlength=2)
    9. p = counts / counts.sum()
    10. return p
    11. def chi_square(obs_counts, exp_p):
    12. exp_counts = exp_p * obs_counts.sum()
    13. return float(np.sum((obs_counts - exp_counts) ** 2 / exp_counts))
    14. def z_score(chi2, dof):
    15. """For large degrees of freedom, the chi-square statistic is approximately normal, with mean dof and std sqrt(2*dof)."""
    16. return (chi2 - dof) / np.sqrt(2 * dof)
    17. # Pseudocode: ref_dist is measured from a "known-hardware" model;
    18. # each candidate hardware has a reference distribution, plus a null-hypothesis distribution.
    19. def identify(tensors, candidates, null_dist):
    20. obs = lowbit_histogram(tensors)
    21. scores = {}
    22. for name, ref_dist in candidates.items():
    23. chi2 = chi_square(obs_counts=obs * obs.sum(), exp_p=ref_dist)
    24. scores[name] = {
    25. "chi2": chi2,
    26. "z_vs_candidate": z_score(chi2, dof=1), # Deviation from this candidate distribution
    27. }
    28. # Significance: deviation relative to the "unrelated hardware / random" null hypothesis
    29. chi2_null = chi_square(obs_counts=obs * obs.sum(), exp_p=null_dist)
    30. scores["_null"] = {"z_vs_null": z_score(chi2_null, dof=1)}
    31. return scores
    32. # In practice:
    33. # 1) Sample only some tensors (e.g., take one every N layers) to control compute cost;
    34. # 2) Remove NaN/Inf and all-zero tensors before measuring;
    35. # 3) When cross-validating across multiple models, watch for completeness of the candidate set.

    Engineering implementation also requires attention to three points:

    1. Exclude format interference: BF16 and FP16 have different bit layouts, so they must be measured separately or normalized to a unified representation first;
    2. Remove noise: Strip outliers and all-zero blocks from the weights to avoid a few tensors dominating the conclusion;
    3. Quantify the certainty of the conclusion: Compute separately across different shards of the same model; if the conclusions disagree between shards, the fingerprint may come from a particular layer or training phase rather than from the hardware globally.

    4. How to Interpret Conclusions: Statistical Significance and Estimation Intervals

    4.1 What a Standard Deviation Means

    If the null hypothesis holds, the chi-square statistic should cluster around its degrees of freedom, and the probability of a deviation of dozens of standard deviations is essentially zero. In the public analysis, the strongest evidence for a single model reached dozens to over a thousand standard deviations — a magnitude that can only be interpreted as “the null hypothesis does not hold,” meaning structural traces highly correlated with a certain class of hardware genuinely exist in the weights.

    But significant does not mean “it is exactly this chip.” A more rigorous phrasing is: the model’s weights are highly consistent with “the execution path of a certain class of hardware.” If the new and old architectures of that hardware differ markedly in instruction shapes and reduction paths, then inferring from “consistent execution path” to “consistent architecture” is reasonable; if the candidate architectures are themselves highly similar, then only a probabilistic leaning can be given.

    4.2 Known-Answer Tests and “Scale Estimation”

    What makes this method most trustworthy is that it first passed tests on samples with known answers. For example, a classic open foundation model officially disclosed that training used 2,048 accelerators, while the analysis gave an estimated interval of roughly 2,000 to 2,500 units — essentially a match. This shows the method can not only judge “architecture” but also roughly infer “scale.” The latter mainly relies on the following observation: the shape of the weight-error distribution correlates with how data parallelism / tensor parallelism is partitioned and with the number of gradient-accumulation steps.

    4.3 From Hardware Inference to Cluster Cost

    Pushing the conclusion one step further, you can make an order-of-magnitude estimate. Suppose a single training run requires hundreds to thousands of the latest-generation accelerators, running continuously for two to three months, and referencing the procurement unit-price range from public analysis (roughly $30,000 to $40,000 per accelerator, typically paired with a same-generation host), then the hardware investment for a single training cluster is on the order of $150 million to $200 million. If this spans five or six models across different teams, the total hardware investment easily exceeds $500 million.

    This is only an order-of-magnitude estimate; it does not account for power, networking, operations, or idle depreciation, and the actual figure could differ by several times. But its significance is this: open weights not only make the model’s capabilities public, they also indirectly expose a comparable magnitude of compute investment. For industry observers, this is corroborating evidence for assessing a vendor’s technical roadmap and resource commitment; for enterprise decision-makers, it is a reminder that model selection is not only a matter of quality — behind it lie costs and supply-chain constraints of very different magnitudes.

    5. Boundaries and Limitations: When the Conclusions Do Not Hold

    Even the most rigorous method has boundaries. The following situations require particular caution:

    1. Distillation and fine-tuning: If a model was distilled from another model’s outputs, or continued training on an existing model with a small learning rate, the historical traces in the weights get “overwritten” to varying degrees. In this case the fingerprint may reflect the hardware of the “last significant update,” not the pretraining hardware.
    2. Post-processing of weights: Quantization, dequantization, format conversion, and shard merging all change the low-bit statistics. If the released weights are not the original tensors from training, the conclusion needs to be adjusted accordingly.
    3. Software-stack confusion: The same hardware under different framework versions, different GPU-compute-library versions, or different auto-tuning configurations may follow different execution paths, so the fingerprint changes too.
    4. Mixed-precision details: If training uses 32-bit accumulation throughout and only downcasts at the final step, hardware differences are greatly weakened and the method’s sensitivity drops.
    5. Incomplete candidate set: If the real hardware is not in the candidate list, the algorithm can only return the “closest” candidate, not the “correct” one.
    6. Packaging-form distinction: Differences among single-die, dual-die, and multi-die unified packaging often show up in interconnect bandwidth and reduction paths, requiring finer-grained features to distinguish.

    6. Implications for Developers: Bring the “Side Channel” into Release Governance

    The value of this goes beyond “knowing who used which card.” More importantly, it reveals a class of easily overlooked information-leak channel: the weight values themselves are a side channel. For enterprise users and developers, here are several recommendations:

    1. Run a “fingerprint check-up” before release: Before releasing open weights, use a similar method to self-audit whether the weights leak hardware, cluster size, or training-phase information you would prefer not to disclose. If there is genuine concern, consider a mild quantization or format conversion before release, or adding a low-magnitude perturbation to the weights to weaken the low-bit statistical features — but be mindful of the impact on model quality, and keep reproducible experiment records.
    2. Bring hardware disclosure into the compliance workflow: Training hardware and compute scale are gradually becoming a focus of regulatory and supply-chain compliance. Rather than being passively inferred, it is better to proactively provide a reviewed disclosure statement in the model documentation.
    3. Build reproducible experiment records: Recording framework versions, compute-library versions, precision strategies, and parallelism configurations helps with internal reproduction and also lets you quickly provide an authoritative explanation if an external inference appears.
    4. View third-party inferences rationally: This kind of analysis is statistical inference with assumption dependencies and error ranges. Enterprises should judge based on their own experiment records, and should not treat it directly as fact for external dissemination, nor use it to make unverified accusations against other vendors.

    7. Conclusion

    “Weights leaking hardware” is not a vulnerability but a natural result of three things stacking together: the non-associativity of floating-point arithmetic, mixed-precision training, and hardware-specific compute libraries. It reminds us that in the engineering world there is almost no such thing as a truly “traceless” computation. Understanding this lets developers be more composed in selection, reproduction, and compliance, and lets enterprise users stay clear-eyed when assessing models and compute investment.