Open-Weight Security LLM Compression: Quantization and Expert Pruning in Practice
Open-weight models appeal to security teams because sensitive source code stays in-house and weights can be audited and fine-tuned. The catch is size: a deployable checkpoint for a representative security model can reach 1.51TB. This article explains how quantization and expert pruning each reduce that footprint, how the two combine, and what to measure before trusting the result. Applying both techniques took a representative security model from about 1.51TB to roughly 328GB (about 78% smaller) while keeping about 92% of the original vulnerability coverage. It also covers post-training quantization vs. quantization-aware training, accuracy controls such as per-channel and activation-aware quantization, how to measure expert importance and prune MoE routing layers, the evaluation dimensions that make up a regression suite, and the deployment practices (batching, canary rollout, observability, elastic scaling) needed to run a compressed model in production.
1. Background: Large Models Are Becoming a New Paradigm for Vulnerability Detection
Traditional vulnerability detection leans heavily on rule engines and static signature matching. Against code defects that are semantically complex and appear in many variants, that approach tends to produce incomplete rule coverage and high maintenance costs. As large language models have matured in code comprehension, cross-file reasoning, and semantic analysis, applying them to vulnerability discovery and code security auditing has become a direction the industry is actively exploring.
Among the available routes, open-weight models are attractive to security teams for concrete engineering reasons:
- Data stays in-house. Security audits usually involve core source code, so self-hosted deployment avoids leaking sensitive code.
- Auditable and fine-tunable. Open weights let a team adapt the model to its own data.
- No external dependency risk. The inference path is fully under the team’s control, with no hard dependency on third-party services.
The difficulty is that stronger models tend to be larger. For one representative security use case, the deployable weights of the base model reach up to 1.51TB, which strains storage, GPU memory, and inference throughput alike, and directly limits whether the model can be deployed inside an enterprise network at all.
2. The Core Tension: Aligning Detection Capability With Deployment Cost
Security models need substantial reasoning depth. They have to understand control flow, data flow, and cross-function call relationships, and that kind of task naturally favors a larger parameter count. Enterprise deployment, meanwhile, faces three constraints:
- Storage and GPU memory cost. Hundreds of billions of parameters do not load comfortably on a standard inference node.
- Inference latency. Vulnerability scanning is typically a pipeline workload, so throughput determines usability.
- Compliance requirements. Data has to stay within a controlled boundary.
The core engineering problem this article addresses can be stated as follows: how to shrink a model to a size that can be privately deployed without materially sacrificing vulnerability detection capability. The standard industry answer is a combination of quantization and expert pruning.
3. Technique One: Quantization
3.1 How It Works
Quantization represents weights and activations at lower numerical precision. Starting from 16-bit floating point (FP16/BF16), values are mapped to 8-bit or even 4-bit integers (INT8/INT4), which reduces storage and bandwidth requirements by a multiple while barely changing the numerical distribution.
The mechanism is to learn a scale factor and a zero-point that linearly map the continuous floating-point range onto a discrete integer range:
# Map high-precision weights to a low-bit integer representation# (framework-agnostic illustration)def quantize_weight(w, bits=8):qmax = 2 ** (bits - 1) - 1 # largest integer the range can representscale = (w.max() - w.min()) / (2 * qmax)zero_point = round(-w.min() / scale)q_w = round(w / scale + zero_point) # quantize: float -> integerreturn q_w, scale, zero_pointdef dequantize_weight(q_w, scale, zero_point):return scale * (q_w - zero_point) # dequantize: integer -> float
3.2 Two Main Paths
- Post-training quantization (PTQ). After training, a small calibration set is used to gather activation statistics and fix the quantization parameters. It is cheap and fast to adopt, which makes it the first choice for compressing large models.
- Quantization-aware training (QAT). Quantization error is introduced during training or fine-tuning so the model adapts to low precision. Accuracy is higher, but training cost is significant.
3.3 Controlling Accuracy Loss
A single uniform quantization scheme applied across the board tends to damage important weights. In practice, several ideas are used to reduce the error:
- Per-channel quantization. Each output channel gets its own scale factor, which limits the impact of outliers.
- Activation-aware weight quantization. Calibration activations indicate which channels matter, and significant channels get more protection.
- Mixed precision. Sensitive layers, such as the first and last, keep higher precision while the rest are compressed aggressively.
4. Technique Two: Expert Pruning
4.1 Mixture-of-Experts Architecture and Routing
Many large models use a mixture-of-experts (MoE) architecture: the model contains multiple expert subnetworks, and a router activates only a subset of them for each token. This sparse activation makes total parameter count much larger than the compute actually spent per token, but it also means redundant experts exist, since some experts are rarely activated under real task distributions.
4.2 Measuring Importance and Pruning
The goal of expert pruning is to identify and remove experts that contribute little. A common metric is average gate activation probability: over a representative dataset, count how often each expert is selected by the router, and treat low frequency as a sign of redundancy.
# Measure importance as each expert's average activation probabilitydef expert_importance(router_scores):probs = softmax(router_scores, axis=-1) # routing probability per token per expertreturn probs.mean(axis=0) # average activation probability per expertdef select_experts_to_keep(importance, keep_ratio=0.85):k = int(len(importance) * keep_ratio) # how many experts to keepkeep_idx = argsort(importance)[-k:] # keep the k most important expertsreturn sorted(keep_idx)
After pruning, the router layer dimensions must be adjusted to match, and an optional light fine-tune on the retained experts helps correct output drift caused by the missing experts.
5. Combining Both Techniques and Reviewing the Results
Quantization and expert pruning act on different dimensions and can be stacked. Quantization compresses the bit width used to represent each expert; pruning reduces the number of experts. Combined, the effects multiply.
For a representative security model, the combination reduced storage from about 1.51TB to roughly 328GB, a reduction of about 78%, while retaining about 92% of the original model’s vulnerability coverage. The outcome breaks down as follows:
- Storage gains are substantial. The footprint drops from the terabyte range to the hundreds-of-gigabytes range, which a standard inference node can carry.
- Capability loss is manageable. An roughly 8% drop in coverage can often be offset in engineering practice with additional detection rules or a second-pass check.
- There is a clear trade-off. The more aggressive the compression, the faster capability degrades, so the setting has to be chosen against the business’s sensitivity to detection rate versus cost.
6. Evaluating and Validating Vulnerability Detection Capability
Compression should not be judged on size alone; it has to be backed by evaluation. Recommended dimensions for a regression suite:
- Detection rate / coverage. Compare detection rates before and after compression on a standard vulnerability discovery benchmark. A representative base model reaches about 84.5% on commonly used industry benchmarks for vulnerability discovery, and that figure should serve as the baseline when checking the post-compression drop.
- False positive rate. Confirm compression has not introduced noise that floods users with false alarms.
- Performance by category. Evaluate separately across categories such as injection, out-of-bounds, and logic flaws, to avoid a situation where the aggregate looks acceptable but one category collapses.
- Regression consistency. Fix the input samples and compare the stability and explainability of outputs before and after compression.
The critical point: the evaluation set must cover the real distribution of business code, otherwise offline metrics look good while production behavior fails.
7. Engineering and Deployment Considerations
- Serve the model behind an inference service. Package the quantized model as a standard inference service and use batching to raise throughput.
- Canary and roll back. Release a new compressed version to a small share of traffic first, run it alongside the previous version for comparison, and roll back quickly if something looks wrong.
- Observability. Use logging and monitoring with alerting to track latency, throughput, and detection hits continuously.
- Elastic resource scheduling. Container platforms can scale with scanning load to control cost.
Supporting this work can rely on general-purpose cloud capabilities, for example object storage for model weights, a container platform to run the inference service, and monitoring with alerting to keep it stable. Baidu AI Cloud and similar platforms offer product categories of this kind as optional infrastructure; teams can select based on their own compliance and cost requirements.
8. Best Practices and Risks
- Keep the calibration set close to the real distribution. Quantization accuracy depends heavily on calibration data quality.
- Prune first, quantize second, or the reverse. The usual order is to prune experts first to reduce their number, then quantize the retained set, which makes error control easier.
- Protect critical capability first. For vulnerability classes central to security coverage, exempt them from aggressive compression where necessary.
- Maintain a continuous evaluation loop. Model iterations and base model upgrades should each trigger a fresh before/after comparison, so capability does not quietly regress.
- Handle compliance first. Security work involves core source code, so ensure both the weights and the full inference path stay inside a controlled boundary.
Summary
Quantization and expert pruning are the two keys that make open-weight security models affordable and deployable. Quantization relieves pressure on representation precision; pruning relieves pressure on model structure. Stacked together, they can cut deployment size by an order of magnitude while keeping roughly 90% of vulnerability coverage. What ultimately determines success is not how aggressively you compress, but whether you build an evaluation loop grounded in the real distribution, backed by canary releases, monitoring, and a sound compliance posture, all within an explicit business trade-off.
