Open-Weight Model Roundup: A 2.8T Multimodal MoE and a 65B Dense Model, Compared on Deployability
Each week brings a wave of open-weight releases, but few are worth the time it takes to pull the weights, stand up inference, and run a real business evaluation. This roundup covers two representative releases: a natively multimodal Mixture-of-Experts (MoE) model with 2.8T total parameters and 104B activated per token, and a 65B dense model pretrained on 12T tokens. Rather than just listing specs, it shows how to read an MoE parameter table as two separate budgets, how much VRAM and memory bandwidth each model actually costs, and why a 1M-token context claim does not automatically hold on a single GPU. It also gives concrete methods for capacity and decode-speed estimation, long-context validation beyond "needle in a haystack," building reproducible agent evaluations, routing between dense and MoE models, and checking licenses before you download a single shard.
Open-weight models ship every week, but only a handful are worth the effort of downloading weights, getting inference running, and benchmarking against your own workload. This week brought two representative releases: a natively multimodal MoE model with 2.8T total parameters that activates 104B per token, and a 65B dense model pretrained on 12T tokens.
Teams evaluating these models keep asking the same questions: dense or MoE, and how do you decide? How large a context actually fits on one GPU? Is long context usable in practice, or just a spec sheet number? Can open weights go straight into commercial use?
This roundup answers those questions in that order: first the parameter tables and what each deployment costs, then practical methods for sizing VRAM and bandwidth, validating long context, running evaluations, and staying compliant.
The 2.8T MoE: a wide capability canvas, a higher deployment bar
Read the parameter table in two layers
The headline numbers for this MoE model are:
- 2.8T total parameters, 104B activated per token
- Native multimodality
- 1M-token context window
- Built for agentic tasks
Activated parameters are roughly 3.7% of the total.
Understanding MoE starts with separating two kinds of resources:
- Total parameters mainly determine VRAM capacity and deployment topology.
- Activated parameters mainly determine compute per token and decode speed.
In other words, total parameters decide whether the model can run; activated parameters decide how fast it runs. Every expert’s weights stay resident in memory, but each token only activates a fraction of them for computation.
Native multimodality plus a 1M-token context signals the design intent: not single-turn Q&A, but agentic work — reading an entire repository, looking at screenshots, calling tools, and executing multi-step tasks in sequence.
Inference cost: VRAM is expensive and so is decode bandwidth
Stored at 8-bit, 2.8T parameters need roughly 2.8TB; at 4-bit, roughly 1.4TB. That figure excludes KV cache, framework overhead, and multi-GPU communication. A single node with 8 × 80GB accelerators offers 640GB of VRAM in total, which cannot hold even the 4-bit weights, so multi-node deployment is normally required.
Decoding is a bandwidth-bound workload. For every token generated, MoE reads at least the weights belonging to the activated experts. Assuming roughly 3.35TB/s of VRAM bandwidth per GPU:
- 104B activated parameters at 8-bit means reading about 104GB of weights
- Theoretical decode ceiling: about 32 token/s
- At an assumed 0.6 bandwidth utilization: about 19 token/s
- With 4-bit weights: about 64 token/s theoretical, about 38 token/s at 0.6 utilization
These are single-GPU figures only. They exclude expert parallelism, cross-node communication, load imbalance across experts, and scheduling overhead. Real-world decode speed across multiple machines is typically lower still.
Compared with a dense model, it is too simple to say “MoE is always faster” or “dense is always slower” — you have to match precision and measurement basis. At the same precision, a 65B dense model reads about 65B parameters per token, while 104B activated MoE reads about 104B activated parameters, a ratio of roughly 0.63. Only when the 65B model uses 4-bit and the MoE uses 8-bit does the byte-read ratio drop to about 0.31.
So this class of MoE tends to be “relatively cheap in compute, more expensive in bandwidth and VRAM,” which makes it a better fit for high-concurrency batch workloads. For latency-sensitive interactive use, evaluate carefully before committing.
The 65B dense model: it fits on one GPU, but 1M context is not a single-GPU promise
Why training from scratch matters
The 65B dense model was trained from scratch by a chip design company, using 12T tokens of pretraining data. Training from scratch means designing the architecture, preparing the data, and running the full training pipeline — an investment typically on the order of tens of thousands to hundreds of thousands of accelerator-hours.
For a hardware vendor, an in-house model can serve internal needs such as chip design, verification, and script generation. It can also act as a long-lived model foundation that evolves over time and attracts developers to the vendor’s own hardware and software ecosystem.
12T tokens is about 185× the parameter count. Given sufficient high-quality corpus and thorough cleaning and deduplication, that is a reasonably ample training budget — though final quality still depends on the data mixture, language coverage, and training details.
4-bit weights are about 32.5GB, but KV cache is a separate line item
One selling point of this model is that it “runs on a single 128GB GPU at 4-bit.” That claim needs unpacking:
- 65B parameters at 4-bit is about 32.5GB in decimal, or about 30.3GiB.
- Inference frameworks, activations, temporary buffers, CUDA contexts, and similar all consume additional VRAM.
- The main memory pressure at long context comes from KV cache.
With a typical configuration of 48 layers, 8 KV heads, and head dimension 128, KV cache size is:
- 8-bit KV: about 96KiB per token
- 16-bit KV: about 192KiB per token
Context lengths that correspond to:
- 32K context: about 3GiB at 8-bit, about 6GiB at 16-bit
- 128K context: about 12GiB at 8-bit, about 24GiB at 16-bit
- 1M context: about 96GiB at 8-bit, about 192GiB at 16-bit
So “runs at 4-bit on one GPU” generally applies only to shorter contexts and low concurrency. Any claim of 1M context support has to be stated alongside KV cache precision, total VRAM, and batch size. After loading 32.5GB of weights plus framework overhead, a single 128GB GPU will struggle to reliably hold a 1M-token context with 8-bit KV cache.
VRAM and decode estimation: first answer “can it run,” then “how fast”
Keep the VRAM math consistent with the example you are quoting
A simplified formula can estimate weights and KV cache:
def estimate_gib(params_b, weight_bits, layers, kv_heads, head_dim,kv_bits=8, ctx_len=8192, batch=1, overhead=0.15):weight_gib = params_b * 1e9 * weight_bits / 8 / 2**30kv_gib = (2 * ctx_len * batch * layers * kv_heads* head_dim * kv_bits / 8 / 2**30)total = (weight_gib + kv_gib) * (1 + overhead)return round(weight_gib, 1), round(kv_gib, 1), round(total, 1)print(estimate_gib(65, 4, 48, 8, 128, kv_bits=8, ctx_len=32768))# Outputs roughly: 30.3, 3.0, 38.3
With those example parameters — 65B, 4-bit, 32K context, 8-bit KV — weights come to about 30.3GiB and KV cache to about 3.0GiB, for roughly 38.3GiB with a 15% overhead allowance. Converted to decimal capacity that is about 41GB. Once you also reserve a framework memory pool, communication buffers, activation space, and headroom for long-tail requests, capacity planning typically budgets around 45GB.
For the 2.8T MoE, assuming 60 layers, 8 KV heads, head dimension 128, 1M context, and 8-bit KV: 4-bit weights are about 1304GiB and KV cache about 120GiB, for roughly 1.78TB at 15% overhead — normally requiring multi-node deployment.
Decode throughput estimates must state the utilization assumption
At decode time, you can start with a bandwidth-based ceiling:
def decode_upper(active_params_b, weight_bits, mem_bw_tbs, util=0.6):bytes_per_token = active_params_b * 1e9 * weight_bits / 8return mem_bw_tbs * 1e12 / bytes_per_token * utilprint(round(decode_upper(104, 8, 3.35, 0.6), 1)) # about 19.3print(round(decode_upper(104, 8, 3.35, 1.0), 1)) # about 32.2print(round(decode_upper(104, 4, 3.35, 0.6), 1)) # about 38.6print(round(decode_upper(65, 4, 3.35, 0.6), 1)) # about 61.8
Be explicit about what these numbers are: 19.3, 38.6, and 61.8 token/s are single-GPU estimates at 0.6 utilization, while 32.2 token/s is the theoretical bandwidth ceiling with no utilization loss. Real serving also has to account for tensor parallelism, expert parallelism, cross-node communication, scheduling, and request queuing.
Long context: a 1M claim does not mean you can use 1M well
Both models advertise 1M-token context, but engineering answers two questions: does it fit in memory, and is the distant information actually being used?
Look at the cost structure. KV cache grows linearly with sequence length. Attention computation grows quadratically with sequence length within a single layer, and the overall attention cost accumulates linearly across layers. For a 1M-token context, the prefill cost comes mainly from sequence length, not simply from reading parameters.
Common optimizations include:
- Using GQA or MQA to reduce the number of KV heads
- Quantizing and compressing the KV cache
- Using PagedAttention to reduce memory fragmentation
- Using chunked prefill and chunked attention to lower peak pressure
- Combining retrieval, summarization-based compression, and structured memory to move part of the long-context work into external storage
When validating long context, do not stop at “supports 1M” — check “can actually use 1M”:
- Needle in a haystack: insert key facts at different depths and measure recall.
- Multiple distractor needles: insert several similar facts and check whether the model can tell them apart.
- Aggregation tasks: require cross-passage counting, comparison, or summarization to test global attention.
- Real workload replay: run end-to-end evaluation on your team’s own long documents, codebases, and logs.
- Cross-validation: answer the same question via short-context retrieval and via direct long-context input; if the results diverge significantly, suspect the long-context capability first.
Evaluation and model selection: your own eval set is the decision basis, public leaderboards are only a reference
Vendor-reported numbers need five checks: who ran the evaluation, whether it is reproducible, whether data contamination is possible, whether the metrics match your task, and whether the measurement basis is clearly stated. When a vendor claims it “beats a certain closed-source flagship model on 16 of 37 benchmarks,” you should still ask for evaluation date, model version, the name of the closed-source model, the prompt template, and the metric definitions — otherwise the boundaries of the conclusion are hard to judge.
For in-house evaluation, agentic tasks should be built as reproducible pipelines first:
TASKS = {"fix_unit_test": {"prompt": "Fix the bug in /app/calc.py so that pytest passes in full. Do not modify the test files.","verify": "pytest -q /app/tests","seed": 20250601,"max_steps": 20,"retry": 3,},"generate_report": {"prompt": "Read /data/metrics.csv and write /out/report.md with three sections of statistics: mean, median, and P95.","verify": "python /app/check_report.py /out/report.md","seed": 20250601,"max_steps": 20,"retry": 3,},}
Each evaluation record should contain at minimum the model version, weight hash, prompt template version, acceptance script version, random seed, failure/retry policy, token consumption, and end-to-end latency. Beyond automated acceptance scripts, keep human spot checks that cover edge cases.
On selection, dense and MoE is not an either/or choice. Layered routing is more common:
User request→ Lightweight classification, extraction, rewriting, single-step Q&A: prefer the 65B dense model or smaller→ Long-document reasoning, multi-step tool calls, mixed image-and-text understanding: route to the flagship MoE→ High-concurrency traffic: use batching to raise total throughput, and set P95 latency alerts
Licensing and compliance: open weights are not unconstrained
Read the license before downloading weights. Permissive and custom-restricted licenses differ substantially; a checklist should cover at least:
- Whether commercial use is allowed
- Whether redistribution is allowed
- How ownership of outputs is defined
- Whether derivative models must be open-sourced or attributed
- Whether there are prohibited uses, regions, or industries
- Whether later versions may change the license
In production, pin the weight version and record a snapshot of the license. Verify weight hashes, scan serialized files for suspicious code, and prefer releases with verifiable uploader identity and signatures. Training data, fine-tuning data, and input data must also comply with applicable data security and privacy requirements.
An enterprise adoption path: from offline evaluation to hybrid deployment
A practical sequence:
- Offline evaluation: fix a business sample set and compare dense and MoE models on accuracy, latency, and cost per call.
- Small-scale pilot: use a 65B-class model to power internal tooling such as code assistance, log analysis, and document processing.
- Elastic capacity: request GPU instances on demand from mainstream public clouds or a private cluster, and use a container platform for elastic scaling and canary releases.
- Version management: centrally host weights, datasets, prompt templates, and evaluation artifacts in object storage.
- Observability: monitor latency, throughput, error rate, token consumption, KV cache hit rate, and queue length.
- Hybrid deployment: route normal traffic to the dense model and complex tasks to the flagship MoE, with a unified gateway handling authentication, rate limiting, and cost control.
Conclusion: scale is no longer the only selling point, deployability is back at the center
This week’s two models represent two directions in the open-weight ecosystem. The trillion-parameter MoE keeps pushing the capability ceiling higher, at the cost of multi-node VRAM, expert parallelism, and more complex operations. The 65B dense model pulls the deployment bar back to a single GPU and a permissive license, making it suitable for scale-out production and private deployment.
Three recommendations summarize the selection process: evaluate first, and judge with real business samples rather than leaderboards alone; size capacity first, and estimate VRAM and bandwidth before discussing deployment; check compliance first, and confirm the license and data boundaries before going live.
