Open-Weight Model Weekly: 5 Weights Shipped, 2 Still Promises — A Practical Deployment Guide
This week's open-weight releases accelerated sharply: five models shipped downloadable weights while two remain announcement-only. This guide focuses on what developers actually need to decide a deployment—whether weights can be downloaded, whether they can be used commercially, whether they can go to production, and how much VRAM they require. It covers MoE total-versus-activated parameter math, why million-token context gets expensive and how to size KV Cache, world models for simulation data, base-model plus fine-tuning workflows, lightweight 27B local deployment, and an eight-point pre-launch checklist to turn "launch-day specs" into a real deployment plan.
1. This Week at a Glance: Five Weights Shipped, Two Still Promises
The release pace of open-weight models picked up noticeably this week (Nov 3–9). Five models formally released downloadable weights, while two only issued release announcements without shipping weights yet. For developers, what matters is not just “how big is the model,” but “can I download it, can I use it commercially, can I put it into production, and how much VRAM does it need.” This article gives an actionable selection-and-deployment checklist around those questions.
| Form | Scale & structure | Key features | Typical deployment | License & checkpoints |
|---|---|---|---|---|
| Flagship MoE | 2.8T total params, 104B activated per token | Native multimodal, 1M context, agent capabilities | Multi-node, multi-GPU + weight quantization | Commercial use and redistribution subject to official license text |
| Dense model | 65B, 12T-token pretraining | 1M context, runnable at 4-bit on a single card | Single card with 128GB memory | Permissive license, but still keep an inventory |
| World model | Four sets of simulation model weights | Multi-view video and LiDAR generation for autonomous driving, dual-arm robot simulation | Data production and training | Commercial use subject to official license text |
| Pure base | 280B MoE, 14B activated, pretraining checkpoint only | ERNIE 5.5 Flash Base | Deploy after fine-tuning | Custom license, excludes the EU, UK, and South Korea |
| Community fine-tune | 27B, secondary fine-tuning on a same-tier base | ~16.5GB at 4-bit, runnable on a 32GB GPU | Single card or local | Permissive license, but verify the specific terms |
Another two models only updated their inference services this week, with weights still unreleased: a “small cup” version of a startup’s 594B MoE model, and a 770B production version from an internet vendor that adds vision and computer-operation capabilities for the first time. Both have promised open source, but there is still a time gap between a “promise” and a “download.” For engineering teams, the sensible move for this kind of release is to connect it to a unified service entry point for side-by-side testing first, then evaluate a switch only after the weights are actually available.
2. Selection Question One: How Do MoE Total and Activated Parameters Work?
This week’s flagship model reaches 2.8T total parameters but activates only 104B per token—an activation ratio below 4%. Mainstream open flagship models generally use a Mixture-of-Experts (MoE) architecture: each token is computed by only the few experts selected by the router, so the total parameter count can be very large while the per-token compute stays manageable.
This means a model’s “parameter scale” has at least two different measures:
- Total parameters determine weight storage and loading cost. All expert parameters must stay resident in memory or VRAM, which decides whether the model “fits.”
- Activated parameters determine per-token compute and latency, which decides whether it “runs fast.”
Looking only at total parameters overestimates deployment difficulty; looking only at activated parameters underestimates memory pressure. Capacity planning must therefore estimate the two separately:
# Weight footprint estimation (approximate, for capacity planning)BYTES_PER_PARAM = {"fp16": 2, "int8": 1, "4bit": 0.5}def weight_footprint(params, precision="4bit", overhead=1.15):# overhead covers quantization metadata, framework caches, and other engineering headroomreturn params * BYTES_PER_PARAM[precision] * overhead / 1024**3print(round(weight_footprint(2.8e12), 1)) # 1499.4 GiB, round to ~1500 GiB for planningprint(round(weight_footprint(104e9), 1)) # 55.7 GiBprint(round(weight_footprint(65e9), 1)) # 34.8 GiBprint(round(weight_footprint(27e9), 1)) # 14.5 GiB
The 1.15 here is an empirical margin. Without it, the 2.8T model’s 4-bit weights come to about 1304 GiB; with the margin, about 1499 GiB, which you can plan around as roughly 1500 GiB. Different quantization frameworks will vary, but the order of magnitude will not change.
Three conclusions follow:
- Even at 4-bit, the flagship’s weights alone are about 1500 GiB. A single 8-GPU server with 80GB per card totals roughly 640 GiB of VRAM, so it still takes multi-node, multi-GPU expert parallelism to load. If you use bf16/fp16 for offline inference or training, the weight footprint grows further: switching from 4-bit back to bf16/fp16 increases the VRAM requirement roughly fourfold, and moving from int8 to fp16 roughly doubles it.
- For online scenarios with low concurrency and short outputs, the weights read per token are close to the activated-parameter scale. With 104B activated parameters at 4-bit, that is about 56 GiB. Memory bandwidth often becomes the bottleneck here, and raising concurrency and batch size is the key to letting the sparse structure show its throughput advantage.
- The 65B dense model comes to about 35 GiB at 4-bit, so a single 128GB card can hold it and still leave room for context and batching. That is where the “runs on a single card” claim comes from—though long contexts still require a separate KV Cache estimate.
3. Selection Question Two: Where Does Million-Token Context Get Expensive?
Both the flagship MoE and the 65B dense model this week support 1M-token context. The value of long context is not just “reading more”—it lies in long-document analysis, whole-repository code understanding, long-running agent tasks, and memory retention across many rounds of tool calls.
As context grows, inference VRAM is made up of two parts: weights of fixed size, and a KV Cache that grows linearly with “sequence length × concurrency.” The KV Cache significantly affects the cost of long-context deployment.
def kv_cache_gib(layers, kv_heads, head_dim, tokens, concurrency=1, bytes_per_elem=2):# Each token, each layer keeps one copy each of K and V cacheelems = 2 * layers * kv_heads * head_dim * tokens * concurrencyreturn elems * bytes_per_elem / 1024**3# Example structure: 60 layers, 8 KV heads, head dimension 128print(round(kv_cache_gib(60, 8, 128, 1_000_000), 1)) # ~228.9 GiBprint(round(kv_cache_gib(60, 8, 128, 32_768, 32), 1)) # ~240.0 GiBprint(round(kv_cache_gib(60, 8, 128, 1_000_000, 1, 1), 1)) # ~114.4 GiBprint(round(kv_cache_gib(60, 8, 128, 32_768, 32, 1), 1)) # ~120.0 GiB
From these results, “1M context, single stream” and “32K context, 32-way concurrency” have similar KV Cache overhead—both in the ~240 GiB range. If the KV Cache drops from fp16 to 8-bit, it falls to about 114–120 GiB. Real structures will produce different numbers, but the logic holds: total tokens × concurrency determines the KV Cache size.
Four common techniques address long context:
- Grouped Query Attention (GQA): Multiple query heads share one set of KV heads, sharply reducing KV Cache size. If each query head held its own KV, the cache could balloon several times over.
- Chunked prefill: Split an extremely long input into smaller blocks computed in batches, lowering the VRAM peak and preventing a single very long request from blocking others for a long time.
- Prefix caching: Fixed prefixes such as system prompts, tool definitions, and long documents can be reused across requests, skipping recomputation on a cache hit.
- KV quantization and paged management: Lower the cache precision, or allocate VRAM blocks on demand, to reduce fragmentation and waste.
Before launch, run load tests and record first-token latency, per-token output latency, and throughput under different input lengths and concurrency levels. This data reflects real capacity-planning results better than any parameter sheet.
4. Once World Models Are Open, the Simulation Data Loop Starts to Close
One special class of weights opened this week targets physical-world simulation: multi-view video generation for autonomous driving, LiDAR data generation, and dual-arm robot manipulation simulation, complete with a full post-training recipe.
Traditional simulation relies on hand-built models and rendering pipelines, which gets very expensive for rare scenarios. Generative world models can produce sensor data with geometric information from text or structured conditions; once connected to a simulation engine, data annotation, and training pipelines, they can form a closed loop:
- Scene collection: Mine long-tail scenarios from real road tests or real-robot operation.
- Conditional generation: Use scenario descriptions as conditions to generate multi-view video, LiDAR point clouds, or robot manipulation trajectories.
- Simulation training: Train perception and decision policies in a simulated environment, reducing dependence on real-vehicle and real-robot mileage.
- Real-world validation: Check policies against real data, then feed failure cases back into the scenario library.
Keep in mind that the value of synthetic data depends on its fidelity. When evaluating, look at:
- Whether multiple sensors are aligned in space and time;
- Whether LiDAR point clouds show artifacts such as see-through or smearing;
- Whether physical laws are reasonable—for example collisions, friction, and lighting changes;
- How large the deviation is from the real data distribution.
On compliance, don’t just check “are the weights open.” World-model output may involve sensitive information such as vehicle appearance, pedestrian likenesses, road signs, and sensor trajectories. Before integrating, add checks for: copyright of generated content, privacy de-identification, de-identification of people and vehicle information, sensor-data compliance, and whether commercial model training is permitted. The scope of commercial licensing should follow the official license text, not a launch-day statement alone.
5. When Base and Fine-Tuning Are Separated, How Do Enterprises Build on Top?
This week also saw a pure pretraining base checkpoint go open: Baidu’s ERNIE 5.5 Flash Base, a 280B MoE activating 14B per token, released with pretraining weights only and no dialogue alignment. For developers, “base + fine-tuning” does not mean out-of-the-box readiness; it hands the alignment-strategy choice back to you.
A practical landing plan breaks into four steps:
- Data preparation: Assemble business instructions, Q&A pairs, and tool-calling samples. Volume usually starts in the thousands and grows iteratively based on results.
- Parameter-efficient fine-tuning: Use low-rank adaptation methods like LoRA to train only a small number of extra parameters, with clearly lower memory and storage costs than full-parameter fine-tuning.
- Alignment and evaluation: Add preference data, implement refusal behavior, format constraints, and safety policies, and run regression against an independent evaluation set.
- Compression and deployment: Quantize the fine-tuned model. Fine-tuning changes the weight values, so you must re-validate the effect after quantization.
# Storage cost of parameter-efficient fine-tuning (illustrative)base_params = 280e9 # Total base parameterslora_rank, target_layers = 16, 60hidden_size = 8192 # Illustrative value, only to show the order of magnitude# Each linear layer inserts two low-rank matrices (A: hidden→r, B: r→hidden)trainable = 2 * lora_rank * hidden_size * target_layersprint(f"Trainable params are ~{trainable / base_params:.2e} of the base")# Save the adapter in bf16 (2 bytes per parameter)print(f"Adapter size ~ {trainable * 2 / 1024**2:.1f} MiB")
# Minimal training data format for base → chat model (illustrative){"instruction": "Judge the urgency of the ticket below; output only high/medium/low","input": "Payment callbacks keep failing, affected orders keep growing","output": "high"}
The same workflow applies to open flagship models. Fine-tuning scripts are mostly provided by the vendor or the community, and private data alone is enough to adapt a general model to a specific domain. Note that ERNIE 5.5 Flash Base uses a custom license that excludes usage in regions such as the EU, the UK, and South Korea; cross-border business should confirm the applicable scope in advance during selection.
6. Lightweight Fine-Tuned Models Move Down-Stack: Estimating 27B Local Deployment
This week also brought a 27B fine-tuned model—a secondary fine-tune on a same-tier base—that runs on a 32GB GPU at about 16.5GB after 4-bit quantization. By the general estimate, 27B weights at 4-bit come to about 14.5 GiB; the gap between the stated 16.5GB and the actual 14.5 GiB may come from the quantization implementation, tokenizer, context cache, framework overhead, and runtime memory allocation.
Model forms therefore split into three tiers:
- Flagship MoE: Pursues the capability ceiling, requires cluster deployment;
- Mid-size dense model: Balances capability and cost on a single card;
- Lightweight fine-tuned model: Supports secondary development on consumer- or entry-level GPUs.
With 60 layers, 8 KV heads, dimension 128, 8K context, and fp16, the KV Cache estimate is:
print(round(kv_cache_gib(60, 8, 128, 8192), 1)) # ~1.9 GiB
Add about 14.5 GiB of weights plus runtime overhead, and a 32GB card still has room. But stretch the context to 32K or 64K, or raise concurrency, and the KV Cache climbs quickly—you must recompute with the actual KV head count and precision.
The value of local deployment shows up in three areas:
- Sensitive data such as code, contracts, and medical records never leaves the intranet;
- Fast tuning iteration, with no per-token billing;
- It can be embedded in development environments to provide code completion, log analysis, and document Q&A.
Emphasize that a fine-tuned model’s capability is closely tied to its base and training data. Before adopting, run a small-sample comparison on your own tasks rather than relying only on model-card scores.
7. Eight-Point Pre-Launch Checklist: From “Launch-Day Specs” to a “Deployment List”
A parameter sheet answers “how big is the model”; a deployment list answers “can it go live.” Work through these eight items.
1. VRAM and capacity planning
Compute peak usage as “weights + KV Cache + batch headroom.” For MoE, balance total and activated parameters: total parameters set the loading floor, activated parameters set the inference overhead. Before quantizing, compare output quality on a small batch, and watch perplexity, task accuracy, format compliance, and long-context recall to confirm there is no clear regression before switching fully.
2. Quantization strategy
- Weight-only dequantization: Weights are restored to higher precision during compute, so compute stays the same and the gains come mainly from VRAM and bandwidth.
- Weight-and-activation quantization: Both weights and activations are computed at low precision, enabling further speedup but making precision and numerical stability more sensitive.
When VRAM is the constraint, prefer weight-only dequantization; when pursuing throughput, evaluate the latter, provided quality still meets requirements after quantization.
3. Long-context strategy
Assign different context budgets to different workloads: prioritize chunking and summarization for retrieval-augmented tasks, prioritize evaluating long-context capability for code and log analysis, and let measured latency and throughput decide.
4. Licensing and data compliance
# Model license compliance check (pseudocode)def can_use(license_policy, usage_class, region, scale):return (usage_class in license_policy.allowed_usageand license_policy.covers_region(region)and license_policy.within_scale(scale))# The same model can yield different results across marketsprint(can_use(policy, "commercial_prod", "APAC", 1)) # Trueprint(can_use(policy, "commercial_prod", "EU", 1)) # False, may trigger regional exclusionprint(can_use(policy, "commercial_prod", "North America", 10000)) # False, may exceed the scale cap
Build a “license ledger” for each candidate model that records at least: license name, license version, license link, permitted uses, regional restrictions, scale caps, attribution requirements, whether redistribution is allowed, and whether outputs may be used to train derivative models. Open weights only mean “downloadable”—whether commercial use, redistribution, or region-specific restrictions apply must each be checked line by line.
5. Evaluation and validation
Vendor-reported scores are an “upper-bound reference.” One flagship claims to beat the strongest current closed model on 16 of 37 benchmarks—valuable, but benchmarks may overlap with training data, and prompts and evaluation methods differ, so this cannot be equated directly with business performance. Build three layers of evaluation:
# Offline evaluation scheduling (pseudocode)suites = ["heldout_bench", "business_eval"] # held-out benchmark + business private setfor model in candidates:report = {}for suite in suites:records = []for case in load_cases(suite):out = model.generate(case.prompt)records.append({"id": case.id, "output": out, "expect": case.expect})# heldout_bench is for initial screening only, forbidden for training and launch decisionsreport[suite] = score(records, exclude_from_training=suite == "heldout_bench")promote_if(report["business_eval"] >= baseline(report["business_eval"]))
- Layer 1: Held-out benchmark. Use questions beyond the vendor’s public set or an independent evaluation set, for initial screening only; forbid using it for training or launch decisions.
- Layer 2: Business private set. Sample from real tickets, code reviews, and support conversations, covering typical, boundary, and adversarial cases. Keep it strictly isolated and exclude it from any fine-tuning.
- Layer 3: Canary and monitoring. After a small-traffic launch, track answer quality, call success rate, first-token latency, and cost per request, and periodically feed production failures back into the evaluation set.
6. Canary rollout and operations
Adopt a “dual-track parallel” strategy: run old and new models simultaneously, split traffic by user or request features at the entry point, and roll back quickly if problems appear. Combine logging with monitoring and alerting, setting thresholds for latency, error rate, and output anomalies. For multimodal and world-model services, also add manual spot-checks of generated samples.
7. Cloud deployment path
On cloud platforms such as Baidu AI Cloud, these steps can reuse mature infrastructure:
- Use elastic GPU compute or a container platform to host model services, scaling with traffic;
- Use object storage for weights and training data;
- Use Baidu’s Qianfan model platform for model hosting, fine-tuning, and effectiveness evaluation.
8. Choose a form by team profile
| Team profile | Recommended form | Deployment | Core concerns |
|---|---|---|---|
| Small/mid teams, on-premise | Mid-size dense or lightweight fine-tuned model | Single-card 4-bit deployment | License scope, output stability |
| Teams with GPU clusters | Flagship MoE | Multi-node, multi-GPU + quantization + prefix caching | Throughput, concurrency, cost |
| Autonomous driving, robotics | Multimodal world model | Data production pipeline | Generation fidelity, post-training recipe |
| Data-sensitive enterprises | Pure base + self-built fine-tuning | Private training and inference | Data isolation, fine-tuning cost |
| Individual developers | Lightweight fine-tuned model | Local or workstation | VRAM headroom, ease of use |
8. How to Track Models That Are Only Promises
There is often a time lag between “released” and “downloadable.” For models that only promise open source with no weights yet, keep a tracking list:
- License terms: Whether commercial use is allowed, and any regional or scale restrictions;
- Weight completeness: Whether it includes full parameters plus the tokenizer and inference code;
- Evaluation reproducibility: Whether evaluation scripts and settings are provided;
- Docs and maintenance: Whether the model card and usage examples are complete, and whether updates will continue.
A safer approach is to connect such models to a unified service entry point first and run side-by-side tests through a standard interface. Once weights open, switch to a self-hosted version, avoiding repeated architecture changes tied to a single release.
9. Conclusion: Select on Structure, Land on Validation
This week’s releases were dense, but so was the noise: total and activated parameters get conflated, self-reported scores diverge from real experience, and license terms carry their own conditions. For developers, a more reliable approach holds to three lines:
- Use the VRAM formulas to convert “model size” into “deployment cost”;
- Use the license ledger to convert “downloadable” into “usable”;
- Use a private evaluation set to convert “benchmark scores” into “business results.”
Meanwhile, keep watching releases that are only promises with no weights yet, and turn “release announcements” into “verifiable deliveries.” If you have already done hands-on work with any of these forms, share your measured VRAM data and evaluation experience in the comments.
