How to Validate VRAM and Throughput for a 2.8T Open-Weight Model: A Worked Example
As open-weight models scale into the multi-trillion-parameter range, the questions that actually decide whether a deployment is viable are no longer just "how big is it?" but rather: is it dense or MoE, what does the license allow, how much VRAM does it need, and does the vendor's published score hold up in your own workload. Using Qwen/Qwen3.5-Omni-2.8T-A104B as a demo object rather than a verified, deployable release, this article walks through a transferable method for estimating expert activation, per-token FLOPs, weight footprint, KV cache for long context, and the throughput ceiling that separates single-request latency from high-concurrency serving. It also includes a practical pre-launch checklist covering licensing, quant format, sharded download, deployment topology, and post-launch monitoring, so you can decide whether to self-host, use the cloud, or go hybrid.
1. The Example Object and the Scope of Validation
This article centers on a “demo object to be validated”: Qwen/Qwen3.5-Omni-2.8T-A104B. It is worth stating clearly that this article does not treat it as verified, deployable official weights, nor does it treat the model card entry, access date, parameter count, repository size, activated scale, or any other figure as a final conclusion. If the relevant links are inaccessible, if the official figures change, or if the license, weight configuration, and measured records cannot be cross-checked, this article serves only as a method demonstration. Before publishing, the example object must be replaced with officially verifiable links, a license, the weight configuration, and measured records.
For developers, what actually affects production is usually not “how large are the parameters” but four things: whether the weights are a dense model or an MoE; whether the license permits commercial use, redistribution, retraining, or redeployment; how much VRAM is needed, and whether long context consumes a second tranche of memory; and whether the vendor’s published scores are worth re-validating in your own business scenarios. Accordingly, the sections below use MoE, KV cache, VRAM/throughput estimation, and a deployment checklist as the main thread, laying out a transferable way to reason about each.
2. Model Structure: 2.8T Parameters Do Not Mean Every Token Reads Everything
2.1 MoE Architecture: A Large Total Parameter Count Is Not the Same as Computing All of It Per Token
Under the example’s stated configuration, the model uses an MoE architecture: the backbone is roughly 93 layers deep, each layer contains 896 routed experts, each token typically activates 16 of them, and there are 2 additional shared experts that are always active for every token. The sparsity at the expert-routing level can be roughly estimated as:
Expert routing ratio = (16 + 2) / (896 + 2) ≈ 2%
But this 2% only reflects the activation ratio at the expert-routing level. It cannot be equated directly with 2% of the 2.8T total parameters, nor used to derive a 104B activated per token. The “A104B” in the model name indicates roughly 104B activated parameters per token, which may be made up of routed experts, shared experts, the attention structure, multimodal modules, the embedding layer, and other non-expert parts combined. A safer way to state it: per the A104B naming, activated parameters per token are about 104B, and this must be confirmed against the official model card and weight configuration.
Under the “104B activated per token” reading, the forward compute can be roughly estimated as:
FLOPs ≈ 2 × 104e9 = 208 GFLOPs/token
This figure only tells you the compute cost; it says nothing directly about VRAM usage. For MoE models, engineering practice requires separating the concerns: VRAM depends mainly on resident weights, the KV cache, framework buffers, and multimodal modules; compute depends mainly on the activated parameters per token; and throughput additionally depends on expert-parallel communication, routing/scheduling, batching efficiency, and kernel implementation. A common mistake is applying a dense model’s “parameter count → VRAM → compute” table directly to an MoE. The defining trait of MoE is that the total weight count is very large, but a single token only reads the subset it is routed to. That is why the bottlenecks for single-request latency and for high-concurrency serving throughput come from different sources.
2.2 Sparse Attention: Long Context Needs Its Own Calculation
The example configuration notes that some layers may switch from full attention to sparse attention — for instance, 24 layers using DSA. One point must be added here: if only some layers are sparse, the number of layers requiring a KV cache depends on layer type, and you cannot simply assume every layer uses the same cache structure. If 24 layers are sparse and the rest remain full attention, the KV estimate should be computed separately per layer type; if the final weight configuration shows more layers are sparse, you also need to re-split by the number of sparse and full-attention layers.
The benefits of sparse attention typically include: a smaller KV sequence cached per layer, and a lower baseline threshold for enabling million-token context. But it is not “free context.” It usually depends on a retriever or a sparse-selection mechanism, and its effectiveness depends on retrieval quality, long-tail task performance, inference-engine support, and the specific business input. Engineering teams cannot look only at the one-line claim of “supports 1M context”; they must separately calculate the KV cache, visual input, and framework buffers.
2.3 Native Multimodal: The Visual Path Also Consumes VRAM
Native multimodal means visual signals are not bolted on via an adapter at inference time, but are jointly processed with the language backbone during pretraining or model design. This structure is better suited to interleaved image-text, charts, UI screenshots, document understanding, and similar scenarios. But inference serving must uniformly handle multimodal preprocessing, image-text token concatenation, length control, and the VRAM budget. The vision encoder, image tiling, projection layers, and activations can all consume VRAM. So the VRAM estimate cannot count only the language backbone weights; it must fold in the vision modules and their runtime overhead in capacity planning.
2.4 4-Bit Quantization: It Saves VRAM, but It Adds Constraints
The example repository may provide only a quantized version with 4-bit weights, 8-bit activations, and fine-grained scaling factors. You must rely on the model card and weight configuration; do not assume outright that it comes from QAT, and do not assume by default that you can freely convert it to BF16 and run it. Generally speaking: if the weights come from QAT, they are usually more stable than simple PTQ, but accuracy still needs to be measured; if they come from PTQ, you should focus on validating long context, code, math, tool calling, and open-ended generation. Running “as BF16” after changing the format yourself is not recommended — this noticeably increases the weight footprint, but does not necessarily restore accuracy.
A precision comparison should be done before going live. It is advisable to prepare a business evaluation set of 100 to 300 samples covering high-frequency tasks, long-tail tasks, multi-turn dialogue, and tool calling. If the quantized version’s accuracy is insufficient, you can consider: adding VRAM or more cards, shortening context, lowering concurrency, using the engine’s supported KV-cache optimizations, switching to another officially available version, or fine-tuning again within what the license allows. Fine-tuning also needs its own capacity planning, because low-rank adaptation methods may require dequantizing the base weights to a higher precision to participate in computation.
3. VRAM and Throughput: 8 Cards to Start Is Not 8 Cards of Stable Serving
3.1 Working Back to Weight Footprint From the Repository Size
First, estimate bytes per parameter using the public example figures:
Bytes per parameter ≈ 1.56 × 10^12 B / 2.8 × 10^12 ≈ 0.557 B/param
0.557 bytes is roughly 4.46 bits, which broadly matches the description of “4-bit weights + fine-grained scaling factors.” But 1.56 TB is the repository size, not the VRAM footprint. Deployment must also reserve space for framework and communication buffers, the vision encoder and intermediate activations, the KV cache, sharding overhead across tensor/expert/pipeline parallelism, and the inference engine’s own working memory. As a rule of thumb, engineering typically treats 80% to 85% of total VRAM as the usable ceiling:
Theoretical card count N_raw = ceil(weight size / per-card VRAM)Actual card count N_use = ceil(weight size / (per-card VRAM × 0.80~0.85))
When the weights are close to an integer multiple of per-card VRAM, the actual card count often comes out 1 to 2 cards higher than the theoretical value. The reason is not that the model suddenly got bigger, but that framework buffers, communication buffers, and the KV cache eat into usable VRAM.
3.2 The Lower Bound on Card Count and Topology Choice
The following is order-of-magnitude reasoning only and does not represent a specific hardware recommendation. The single-node, dual-node, per-card VRAM, and parallelism strategies in the table are all example assumptions; actual deployment should follow the official deployment notes and target-engine test results.
| Deployment form | Example premise | Total VRAM | Usable at 80%~85% | Can it stably hold the 1560GB weights? | Notes |
|---|---|---|---|---|---|
| Single node, 8 × 196GB | Example assumption, excluding cache and buffers | 1568GB | ~1333GB | No | Raw capacity is close to the weights, but insufficient under the usable figure; it cannot stably load the weights and serve. Even a load-only check may need offloading or host-memory fallback |
| Single node, 8 × 96GB | Example assumption, VRAM clearly too low | 768GB | ~614GB | No | Clearly insufficient VRAM; needs host memory, storage offloading, or more cards |
| 2 nodes × 8 × 141GB | Example assumption, cross-machine interconnect needs measurement | 2256GB | ~1918GB | Yes | Can fit the weights and leave a little room for KV, but cross-machine communication pressure rises |
The point must be stated consistently: 8 × 196GB gives 1568GB of raw capacity, close to the 1560GB weights, but under the 80%~85% usable figure only about 1333GB remains, so it cannot stably load and serve the weights. If you only want to do a load-only check, you may still need offloading, host-memory fallback, or a special configuration. For topology: tensor parallelism is better suited to high-bandwidth interconnect within a node, because intra-layer synchronization is frequent; expert parallelism shards by expert, and each layer usually needs two all-to-all communications, making it more sensitive to cross-machine bandwidth. Within a node, prioritize NVLink or high-bandwidth interconnect; across nodes, pay attention to RDMA, InfiniBand, or equivalent high-speed networks.
A rough approach to estimating cross-machine all-to-all bandwidth:
Communication volume per step ≈ batch × number of tokens × hidden_size × bytes × number of communications
If each layer needs two all-to-all operations, the total communication volume grows with the number of layers. When cross-machine bandwidth is insufficient, the gains from expert parallelism can be offset by communication. Before launch, measure cross-machine throughput, tail latency, and whether expert load is balanced.
3.3 The Second Bill for Long Context: The KV Cache
The KV cache is a cost that is easily overlooked in long-context serving. Assume a compressed-KV design common in sparse attention, caching a 512-dimensional main vector and a 64-dimensional positional vector per token per layer — 576 elements total — stored in 16 bits. Then:
KV per 1M tokens = 93 layers × 576 elements × 2 bytes × 1,048,576 ≈ 112.3 GB
One caveat must be added: this estimate assumes all 93 layers use a 576-dimensional compressed cache; if there are full attention, GQA, MQA, or MLA structures, recompute by number of heads, head dimension, and layer count. If only some layers use the compressed cache, calculate the sparse and full-attention layers separately.
Under identical conditions, a full-attention model could be several times to tens of times higher, depending on KV head count and head dimension. Even at 112 GB, a single million-token context consumes a large amount of VRAM. The common mistake is counting only weight VRAM while ignoring the KV cache, visual input, framework buffers, and sharding overhead. For long documents, code repositories, long-running agent tasks, and similar workloads, context length and concurrency should be treated as central to selection, not just “can it load.”
| Context length | Estimated KV cache | Ratio to 1M KV | Notes |
|---|---|---|---|
| 32,768 tokens | ~3.5GB | ~3.1% | Suited to ordinary dialogue and medium-length documents |
| 131,072 tokens | ~14.0GB | ~12.5% | Suited to long documents, code snippets, and multi-turn tasks |
| 1,048,576 tokens | ~112.3GB | 100% | Significantly reduces the number of concurrent sequences |
This table estimates KV only; it does not include the vision encoder, framework caches, activation buffers, or sharding overhead. Actual concurrency will be lower.
3.4 The Throughput Ceiling: Single Requests See Activated Weights, High Concurrency Sees All Resident Experts
During decoding, a single token does not necessarily read the entire weight set; it reads the activated weights that the token is routed to. Single-request latency is governed mainly by activated-weight bandwidth, KV reads, routing/scheduling, and communication. High-concurrency throughput instead depends on whether the batch is large enough to cover all experts. If routing is sufficiently balanced and the batch is large, the bandwidth for all resident experts and EP communication become the ceiling on serving throughput.
With 104B activated parameters per token, and bytes-per-parameter estimated at 0.557, the activated weights actually read per token are roughly:
A_active = 104e9 × 0.557 ≈ 57.9 GB
Under memory-bandwidth-bound conditions with short context and a single request, a rough estimate is:
TPS_single ≈ (N_gpu × BW_card × η) / (A_active + KV_read_per_token)
Here BW_card is the per-card memory bandwidth and η is an efficiency factor. It must be stated that η=0.2, 8TB/s, batch covering all experts, and so on are all illustrative parameters, and must be calibrated against the target engine’s measured TTFT, TPS, concurrency, VRAM, and long-context behavior; this formula is only for rough estimation of single-request, short-context, memory-bandwidth-bound cases, and does not account for communication, scheduling, batching, KV reads, routing imbalance, or kernel overhead. In production, it cannot be used as a committed figure.
Using illustrative parameters — 8 cards, 8TB/s per-card bandwidth, η=0.2:
TPS_single ≈ (8 × 8e12 × 0.2) / 57.9e9 ≈ 221 token/s
For high-concurrency serving, as batch size B grows and routing is sufficiently balanced, the set of covered experts approaches all experts. At this point, the bytes read per step are no longer just 57.9GB, but approach the full 1.56TB of weights. Averaged per token, the bytes become:
W_full / B
Therefore, the serving throughput ceiling when a large batch covers all experts can be written as:
TPS_batch ≈ (N_gpu × BW_card × η × B) / W_full
This formula likewise applies only to the memory-bandwidth-bound case, ignoring communication and scheduling overhead. A real system must also satisfy compute and communication constraints simultaneously:
TPS_compute ≈ total effective compute / 208 GFLOPsTPS_comm ≈ the token exchange rate the all-to-all can sustain
The final throughput is usually the minimum of memory bandwidth, compute, communication, and scheduling.
| Scenario | Main bottleneck | Order of judgment | What to watch |
|---|---|---|---|
| Single request, short context | Activated-weight bandwidth | First activated bandwidth, then KV reads | Per-token latency, routing efficiency |
| Single request, long context | KV reads and attention | First estimate KV size, then bandwidth | KV cache, context length |
| High concurrency, balanced routing | All-expert weight bandwidth and EP communication | First batch coverage, then all-to-all | Batch coverage, cross-machine communication |
| Cross-machine deployment | Network interconnect | First intra-node, then cross-machine | RDMA, InfiniBand, cross-machine latency |
The core takeaway: do not mistake “all experts resident” for the per-token read volume. Single requests see activated weights; high-concurrency serving is when all-expert bandwidth and communication come into play.
4. Reading the Scores: Both Vendor Leaderboards and Third-Party Evaluations Need Their Conditions
The scores a model card provides are usually vendor-reported. When reading these tables, watch for three traps: sampling settings can change the conclusion; the training corpus may overlap with the evaluation set; and a score is a single snapshot — models iterate quickly, and a leaderboard can be rewritten a few weeks later.
A more reliable reading is to split the table into two columns: what it is strong at, e.g., long context, terminal operation, or multi-step agent tasks; and what it is weak at, e.g., scientific knowledge, complex reasoning, or tool-calling reliability. Then weigh your options against your own workload shape.
If you use third-party evaluations such as LMArena or Artificial Analysis, establish a consistent record template and avoid relying only on screenshots or verbal conclusions. A suggested template:
- Model version and weight source:
- Model card or release page link:
- License and commercial restrictions:
- Inference engine and quant format:
- Hardware and parallelism strategy:
- Sampling parameters: temperature, top_p, reasoning budget, multi-sample strategy;
- Evaluation date and reproduction entry point:
- Business scoring: accuracy, format compliance rate, tool-calling success rate, long-text recall;
- Performance record: TTFT, TPS, concurrency ceiling, VRAM usage, average cost per task.
A more robust approach is to build your own small business evaluation set:
Suggested business eval set of 100~300 samples├─ Coverage: high-frequency tasks, typical long-tail, multi-turn context├─ Scenario: tool calling, file read/write, multi-step tasks with retrieval augmentation└─ Judging method├─ Has a reference answer → automatic scoring└─ Open-ended generation → manual or strong-model cross-review
Beyond the correctness ratio, also record first-token latency, per-token latency, throughput and concurrency ceiling, VRAM usage, and average cost per task. Quality, latency, and cost form a triangle. Only by measuring all three together can “run locally” and “call in the cloud” be compared on equal footing.
5. Pre-Launch Checklist: Confirm Each Item Before Going Live
1. Does the License Permit Commercial Use?
- Confirm the scope of commercial use;
- Confirm distribution and retraining obligations;
- Confirm limits on responsibility for output content;
- Confirm whether attribution or notation of modifications is required;
- Confirm whether specific scenarios or specific deployment methods are restricted.
At this scale, open-weight models often adopt a custom license rather than an OSI-approved open-source license. Code can be rolled back; the cost of a contract breach or license misuse is higher.
2. Can 8 Cards Run It Locally?
- Estimate using the example weight size and 80%~85% usable VRAM;
- Verify against the official deployment notes for different quant formats, parallelism strategies, and inference engines;
- Measure loading, inference, long-context, and concurrency stability on the target hardware.
The conclusion should be limited to: based on the example weight size and the 80%~85% usable VRAM figure, 8 × 196GB gives about 1568GB of raw capacity, close to the 1560GB example weights, but the usable figure of about 1333GB may not be able to stably hold them. Even a load-only check may need offloading, host-memory fallback, or a special configuration. For stable serving, you generally need more cards, higher per-card VRAM, or a more sensible quant and caching strategy. Different quant formats, parallelism strategies, and engines must be confirmed against official deployment notes and measured validation.
3. How Do You Account for 1.56TB of Weight VRAM?
- Weight VRAM requirement ≈ 1.56TB;
- Usable VRAM requirement ≈ weights / (0.80~0.85);
- Deployment VRAM requirement ≈ weights + KV cache + vision modules + framework buffers + parallelism sharding overhead.
Do not judge VRAM by “repository size” alone, and do not ignore the KV cache under long context.
4. What to Watch When Downloading Shards
- Integrity: verify hashes shard by shard, then load the shard index and confirm the shards are complete;
- Source: prefer official channels and verify the published hash manifest; if using a mirror, confirm the source is trustworthy;
- Disk: reserve space for unpacking, caching, and temporary files; for 100GB-class single files, prefer local NVMe over a shared network drive;
- Memory: use memory mapping or shard-by-shard loading when loading, and avoid doubling peak memory.
5. What to Watch With 4-Bit Quantization
- Confirm whether the inference engine natively supports this weight format and activation format;
- Do not blindly convert to BF16 — weight size grows, but accuracy is not necessarily restored;
- Run an accuracy comparison against the business evaluation set before launch;
- Do separate capacity planning for fine-tuning;
- If accuracy is insufficient, prioritize adjusting context, concurrency, hardware, or deployment strategy rather than simply changing the weight format.
6. Local, Cloud, or Hybrid — How to Choose?
| Option | Suitable scenario | Main risk | Mitigation |
|---|---|---|---|
| Fully local or private cluster | Data must stay on the intranet, stable traffic, an ops team in place | Low utilization; version iteration and failures need self-cover | Fill with offline jobs, co-locate multiple models per node, unified monitoring |
| Cloud-hosted or self-built | Highly variable traffic, needs fast launch | Cost under long text and high concurrency | Elastic scaling, reserved-instance fallback, unified gateway routing and rate limiting |
| Hybrid deployment | Sensitive data processed locally, peak bursting to the cloud | Consistency across both sides, complex routing | Unified observability, data tiering, clear routing policy |
If you choose to run in the cloud, pay attention to general capabilities: high-performance RDMA or InfiniBand networking, an elastic container platform, logging, monitoring and alerting, object storage, and object-storage access control. During selection, defer to the cloud provider’s official documentation and measured pricing, and keep the comparison neutral.
7. How to Monitor After Launch
- First-token latency;
- Per-token latency;
- Throughput and concurrency;
- VRAM usage and fragmentation;
- Prefix-cache hit rate;
- Hot-expert distribution;
- Average cost per task.
If the hot-expert distribution is abnormal, consider adding cards, rate limiting, adjusting the routing strategy, switching the parallelism strategy, or lowering concurrency. VRAM and hit rate determine cost; hot-expert distribution determines whether the parallelism strategy needs adjustment.
6. Conclusion: Open Weights Change the Cost Structure
The significance of this kind of open-weight model is not “largest,” but three signals: the ceiling on scale is rising, and MoE may become an important structure for very large models; long context is shifting from a selling point to a threshold, and both sparse attention and the KV cache must be estimated separately; and open weights do not mean zero cost — the license, hardware, ops, and tuning are all hidden expenses.
Before going live, answer three questions: what use does the license allow? Can the target hardware stably hold the workload at the target context length? Between self-hosted and cloud, which has the lower total cost of ownership? Answer those three clearly, then decide whether to run it, where to run it, and how to scale — that is more valuable than watching the leaderboard rank alone.
