Why Ultra-Large Mixture-of-Experts Models Can Shrink to About 200GiB with Nearly Unchanged Scores

    A large mixture-of-experts model can drop from roughly 1.5TB of half-precision weights to about 200GiB without a matching drop in benchmark scores. This article explains what those sizes represent, why the true average bit width is not 1 bit, how expert matrices enable aggressive compression, and what storage, bandwidth, compute, deployment, and evaluation still require.

    1. What 1.5TB and about 200GiB actually mean

    When a very large model is said to shrink from 1.5TB to about 200GiB while its scores stay almost the same, it is easy to assume that the whole model was simply packed to an average of 1 bit and that inference itself did not change. In practice, the result usually combines three kinds of optimization: model architecture, parameter storage format, and the inference path.

    Take a typical large mixture-of-experts model. Its backbone may have about 770B parameters, plus an auxiliary prediction module of about 10B parameters. If each parameter is stored in a 2-byte half-precision floating-point format:

    1. Total parameters ≈ 770B + 10B = 780B
    2. Half-precision weight size ≈ 780B × 2 bytes
    3. ≈ 1.56 TB

    That figure is the “raw weight” size. Real files may also contain tensor metadata, the vocabulary, quantization indexes, scale factors, and alignment padding, so the file size does not have to match the theoretical value exactly.

    About 200GiB usually refers to a low-bit storage build. GB and GiB are not the same unit: 1GiB is 1024³ bytes, while 1GB is commonly counted as 10⁹ bytes. About 213.66GiB is already close to 230GB in decimal capacity. A headline that says “200GB” is therefore more accurately read as “a model file on the order of about 200GiB.”

    2. Why the average width is not 1 bit

    Low-bit quantization is often described in bpw, the average number of bits per parameter. If the whole model averages 2.38 bpw, the theoretical weight size is roughly:

    1. 780B × 2.38 ÷ 8
    2. ≈ 232.05GB

    Once metadata, grouped parameters, and compression details from different file formats are included, landing near about 200GiB is reasonable.

    A “1-bit version” therefore does not necessarily mean that every parameter uses only 1 bit. A more accurate reading is often that one class of expert matrices, which accounts for a very large share of parameters, is encoded at about the 1-bit level, while other layers, shared modules, routing modules, and sensitive tensors still use higher widths. The real average width is total stored bits divided by the total parameter count:

    1. Model-wide average bpw =
    2. sum of stored bits across all tensors ÷ total number of parameters

    Using 1.3 bpw on one region does not make the whole model average 1.3 bpw. If most parameters sit in particular matrices, compressing only those matrices can cut the file size sharply while the model-wide average remains above 1 bit.

    3. Why the mixture-of-experts layout is essential

    These models typically use a mixture-of-experts architecture. Suppose the network has 78 layers: the first uses a dense feed-forward network, and the remaining 77 use experts. Each expert layer might include:

    • 256 routed experts;
    • 1 shared expert;
    • only 8 routed experts activated for each input token;
    • a shared expert that usually participates for every token.

    The model can therefore store a very large parameter count while each token uses far fewer parameters. Two quantities should be kept separate:

    1. Total parameters: everything the model file must store.
    2. Active parameters: the parameters that actually take part in matrix computation for one token during one inference step.

    A model can have about 780B total parameters while activating only on the order of tens of billions per token. Total parameters drive VRAM, disk, and load cost. Active parameters more directly determine compute per token.

    That is why mixture-of-experts models can aim for both large capacity and lower compute: different experts can learn different domains, semantic patterns, or task traits, while the router selects only a few experts for the current token.

    4. Why expert matrices are compressed first

    In an expert layout, routed-expert weights often dominate the parameter count. If three classes of routed-expert matrices together account for 97.7% of all parameters, they are the natural target for aggressive quantization.

    The model can be pictured as a large organization:

    • most people belong to many specialized departments;
    • a smaller group handles shared management, routing, and critical decisions;
    • each request calls only a few specialist departments;
    • every department’s material still has to be stored in the model file.

    If the expert matrices that make up nearly 98% of parameters are stored at low bit width, overall compression remains large even when shared experts, the router, embeddings, and the output layer stay at higher precision.

    This is optimization by parameter share, not one format applied equally to every tensor. The principle is:

    Aggressively compress the largest, more redundant parts that are not extremely sensitive to a single decision; keep higher precision for smaller parts where error is costly.

    That is more practical than forcing the entire model into one quantization format.

    5. Why low-bit quantization does not always move scores much

    Quantization introduces error, but that error does not have to reach the final output in proportion. Several reasons matter.

    1. Experts are not all active at once

    Routing selects a small set of experts. Compressing one expert matrix affects only the token paths that select it, and other experts can still supply complementary information. Multiple paths give the model some tolerance for error.

    2. Shared modules can stabilize core ability

    Shared experts, together with attention, normalization, token embeddings, and output projections, usually carry cross-task and cross-domain representations. Keeping those parts at higher precision reduces the impact of low-bit quantization on overall language ability.

    3. Parameters are statistically redundant

    Not every weight in a very large model is equally important. Some weights barely change the output, and some experts overlap in function. Quantization mainly loses numerical precision; it does not necessarily destroy the higher-level structure those parameters express.

    4. Benchmarks do not cover every behavior

    Common benchmarks focus on a limited set of tasks such as knowledge QA, reasoning, code, and language understanding. Degradation on long-tail cases, very long context, or numerically sensitive work may not show up clearly in an overall average score.

    “Scores stay close” should therefore mean that core ability remains stable on a particular evaluation set and configuration, not that the low-bit model matches the original on every task.

    6. What low-bit quantization actually changes

    Three layers should be separated.

    Storage

    This is the most visible gain. Moving from half-precision weights to low-bit weights cuts disk use and distribution cost, and makes it easier to fit the model into limited VRAM or memory.

    Memory-bandwidth pressure

    Large-model inference is often limited by memory bandwidth. If each weight read moves less data, a single step can be faster, especially at small batch sizes when compute units are not fully saturated.

    Actual compute

    Smaller storage does not mean compute falls by the same ratio. Before a matrix multiply, low-bit weights may need to be decoded, dequantized, or converted into a format the hardware can execute. If the hardware has limited support for that width, some math may still run at higher precision.

    A quantization scheme should therefore be judged by more than file size:

    • time to first token;
    • later token generation speed;
    • peak memory;
    • throughput under concurrency;
    • long-context behavior;
    • quantization decode overhead;
    • compatibility across hardware.

    7. Why a smaller file can still be hard to deploy

    Even at about 200GiB, an ultra-large model is not something an ordinary device can run comfortably. Deployment also has to account for:

    1. Runtime cache: attention cache grows with context length and concurrent requests;
    2. Temporary workspace: matrix math, routing, and dequantization need extra memory;
    3. Expert scheduling: expert weights may be swapped in and out frequently;
    4. Parallel communication: multi-device serving adds traffic for expert sharding and token routing;
    5. Load time: a file of several hundred GiB still takes a long time to read and initialize.

    About 200GiB should be read as a lower barrier to entry, not as “one GPU can run it without pressure.” Real deployments usually need multi-GPU parallelism, tiered storage, weight prefetch, or on-demand expert loading.

    8. How to check whether a compressed model is usable

    Use layered checks rather than a single overall score.

    Layer 1: numerical agreement

    Compare the original and quantized models on fixed inputs, and record:

    • differences in output probability distributions;
    • activation error in key layers;
    • agreement of routing choices;
    • overlap of top-k predictions.

    Layer 2: task ability

    Test general QA, code generation, math reasoning, long-text understanding, and multilingual tasks separately, and look for domain-specific regression.

    Layer 3: engineering metrics

    On the target hardware and under realistic concurrency, measure:

    • model load time;
    • time to first token;
    • average generation speed;
    • peak memory;
    • concurrent throughput;
    • long-context stability.

    Layer 4: edge cases

    Also test very long inputs, rare vocabulary, complex numeric calculation, repeated prompts, and multi-turn dialogue. These cases are more likely to expose the limits of low-bit quantization.

    9. Conclusion: the point is not “compress to 1 bit,” but “compress the right place”

    A large model can move from about the 1.5TB class to about the 200GiB class not because every parameter is uniformly reduced to 1 bit, but because mixture-of-experts models have a particular parameter layout:

    • total parameters are huge, but parameters active on one step are much fewer;
    • most parameters sit in routed-expert matrices;
    • those matrices have redundancy and some tolerance for error;
    • critical modules can keep higher precision;
    • mixed bit widths balance storage against ability.

    “Scores barely change after compression” is the combined result of architectural sparsity, analysis of parameter importance, non-uniform quantization, and the tolerance of the evaluation itself. For engineering work, the useful lesson is not a particular file size. It is the layered approach: find the largest parameter blocks, measure how sensitive different tensors are to the output, and then choose bit widths for the target hardware and application. That is what keeps model size, inference speed, memory use, and task quality in a stable balance.