Lightweight Security LLMs: Combining Expert Pruning and Quantization
This article explains how to compress large security-focused language models into deployable sizes without sacrificing core capabilities. Using the Altar security model as a reference case—which was reduced from 1.51TB to 328GB while retaining about 92% of its vulnerability-detection ability—the article walks through a complete workflow: selecting a strong open-weight base, pruning inactive MoE experts, applying low-bit quantization, evaluating capability retention, and deploying privately. It also covers base-model selection criteria, building internal evaluation sets, mainstream quantization algorithms (RTN, GPTQ, AWQ, KV-cache quantization), the ordering of compression steps, resource planning, and a checklist of common pitfalls.
1. Why Security LLMs Need to Slim Down
Code-focused large models keep improving at semantic understanding, pattern recognition, and logical reasoning. Vulnerability discovery is shifting from being “mostly rule-based” toward a collaboration between rules and models. Security teams want models that can read cross-file call chains, recognize how dangerous functions are used, determine whether tainted data actually reaches a dangerous sink, and produce actionable fix recommendations.
But open-weight models released by the community tend to be large, often weighing in at the TB scale once written to disk. Using them directly for security work runs into three practical constraints:
- High deployment cost: Distributing TB-scale weights to multiple inference nodes creates heavy storage and bandwidth overhead, and cold starts are slow.
- Memory pressure: Security scanning often requires long context and high-concurrency batching. Insufficient GPU memory directly suppresses throughput.
- Compliance difficulty: Source code, vulnerability intelligence, and binary samples are all highly sensitive. They should not be processed through external inference APIs, which makes private deployment almost mandatory.
The key engineering challenge, then, is to “shrink the model to a deployable size while keeping capability loss acceptable.” Common approaches so far have been to simply pick a smaller-parameter model, or to run instruction tuning on top of a dialogue-tuned dataset built on an open base. The former starts from a low baseline of general code ability; the latter only aligns surface-level behavior, leaving the base model’s vulnerability-analysis potential largely untapped.
A more efficient path is to take an open-weight model that already excels at code and security, apply structured compression first and then low-bit compression, and obtain a dedicated security model at relatively low cost.
1.1 Reference Case: From 1.51TB to 328GB
Recently, a European cybersecurity company released a security-specific model called Altar. During a security evaluation round, their team found that GLM-5.3, a flagship model in Zhipu AI’s open-source GLM series, ranked among the top performers. On the CyberGym vulnerability-discovery benchmark, GLM-5.3 scored 84.5%, and Zhipu built two vulnerability-discovery variants, GLM-5.2 and GLM-5.3, on that foundation. The team chose it as the base for Altar and used quantization and expert pruning to reduce the size from 1.51TB to 328GB while retaining 92% of the original model’s vulnerability-detection ability.
- 328 ÷ 1510 ≈ 21.7%, so the size is a bit more than one-fifth of the original;
- 1510 ÷ 328 ≈ 4.6, roughly a 4.6x compression.
At a time when leading open-source models commonly reach TB-scale, this ratio shows that a “base-to-derivative” relationship can hold between two models with different capability focuses. For security teams, this means there is no need to train from scratch, nor to accept underpowered lightweight models. Instead, you can “subtract” from a strong base to obtain a dedicated model with controlled size and usable capability.
1.2 Overall Approach: Two Compression Lines—Structure and Bits
Large-model compression generally falls into two categories:
- Structured compression: Reduces the number of parameters, such as pruning, distillation, and expert trimming for Mixture-of-Experts (MoE) architectures.
- Unstructured compression: Reduces the representation cost of each parameter, with quantization being the typical example.
Altar used both methods, with a clear division of labor:
- Expert pruning handles “structural redundancy”: it removes experts that are rarely activated by routing after training, directly reducing the total parameter count.
- Quantization handles “representational redundancy”: it lowers the retained weights from half precision to fewer bits, continuing to shrink the size.
Note that expert pruning is a form of structured compression specific to MoE architectures, and most open-weight code bases use exactly that architecture. Quantization alone typically shrinks the size only to about 1/4–1/2 of the original; only by combining both lines can a TB-scale model be pressed into the hundreds-of-GB range. Whether it can actually be deployed ultimately depends on a rigorous evaluation of capability retention. The sections below follow the order: base selection → expert pruning → quantization → capability retention → deployment.
2. Step One: Choose the Right Base
2.1 Compression Inherits the Base’s Weaknesses
Compression and fine-tuning can amplify a model’s strengths, but they cannot conjure capabilities that were never learned during training—and they tend to damage already-weak capabilities first. If the base performs only moderately on a given language or a class of logic vulnerabilities, the compressed model will be worse. So the capability ceiling of a security-specific model is essentially set by its base.
2.2 Seven Base-Selection Criteria
- Security-specific capability: Scores on vulnerability-detection benchmarks, including detection rate, coverage of vulnerability types, and cross-language and cross-repository performance. Coding and reasoning benchmarks can corroborate general ability but cannot replace security-specific evaluation.
- Open weights and licensing: Whether the license permits commercial use, modification, and redistribution. If you plan to embed the model in your own product or offer inference services, this item needs confirmation with legal.
- Architectural compressibility: MoE architectures, with their many low-frequency experts, are well suited to expert pruning. Dense models can also be compressed, but only via quantization or structured pruning that requires retraining—the latter carries higher risk and cost.
- Context window and vocabulary: Code scanning often requires fitting multiple functions or an entire file at once, so context length directly determines the usable scenarios. How efficiently the vocabulary encodes code symbols (identifiers, operators, indentation) also affects how much code fits in the same context.
- Output controllability: Whether the model can reliably emit structured results (such as JSON) determines whether it can plug directly into a scanning pipeline.
- Inference-ecosystem compatibility: Whether the target hardware and inference frameworks maturely support the model’s architecture and low-bit formats. Otherwise, compression results won’t translate into throughput.
- Iteration cadence: A base with continuous updates, thorough documentation, and strong community support makes it easier to keep pace with later security-capability upgrades.
2.3 Build an Internal Evaluation Set, Don’t Be Led by Leaderboards
Public leaderboards carry two risks: inflated scores from training-data contamination, and bias from a mismatch between benchmark tasks and your company’s real code distribution. It’s recommended to build an internal evaluation set covering these four aspects:
- Temporal split: Mainly use vulnerabilities publicly disclosed after the base’s training-data cutoff, reducing the chance that “the answer was already seen during training.”
- Balanced positive and negative samples: Sample positive examples stratified by vulnerability type, then draw negative examples from code that has passed security review, and track false positives separately.
- Difficulty gradient: Cover single-function defects, cross-function taint propagation, and cross-file call chains at different difficulty levels.
- Multi-language coverage: Allocate proportions according to your team’s actual tech stack.
The evaluation set typically needs only hundreds to a few thousand entries; what matters most is labeling quality and distributional representativeness.
2.4 Establish a Full-Precision Baseline
Before any formal compression, run the full-precision base in the target inference environment and record metrics across dimensions along with latency and throughput. This serves as the reference baseline for all subsequent compressed versions, and you compute the capability retention rate:
Capability retention rate = Compressed model metric / Base full-precision metric × 100%
Altar’s 92% vulnerability-detection retention corresponds to detecting about ninety-two percent as many issues as the original model on the same evaluation set.
3. Step Two: Expert Pruning
3.1 MoE Architecture and Uneven Expert Load
In an MoE model, each expert is essentially an independent feed-forward sublayer, and a router picks a few experts for each token to participate in computation. As a result, the total parameter count can be very large, while only a small fraction is activated in any single inference pass.
But expert utilization after training is uneven: some experts handle the routing for most tokens, while others are rarely selected across the entire corpus. Those rarely-used experts still occupy memory and storage, yet contribute little at the margin to overall performance.
3.2 How Pruning Works
Deletion at the security level: Removed weights do not participate in forward computation and are no longer loaded into memory, so they cannot be used to reconstruct the original parameters. This goes a step beyond relying on access control alone to prevent weight leakage.
Low risk at the capability level: Because these experts are rarely activated to begin with, the model’s output is almost unaffected for the vast majority of inputs. This is fundamentally different from dense-model pruning: in a dense model every parameter may participate in the computation for any input, so pruning must be compensated by retraining.
3.3 Implementation Steps
- Prepare a calibration set: Samples should match the real scanning distribution, cover multiple programming languages and multiple vulnerability types, and include a certain proportion of normal code; a few thousand entries is enough.
- Profile expert load: Run forward inference with the calibration set, recording per layer how often each expert is selected by the router or how many tokens it handles, then compute each expert’s selection ratio.
- Decide the pruning strategy:
- Keep the most-used experts in each expert layer, but keep at least one when fewer than one would remain;
- Set a minimum-ratio threshold (e.g., 1%) and also remove experts below it, avoiding a situation where proportional retention “fails to keep even one”;
- Never touch shared components (such as attention layers and shared experts);
- Update the expert count in the model configuration accordingly.
- Sample evaluation: Run sampled evaluations at different pruning ratios to find the sweet spot where “size drops noticeably while metrics stay roughly flat.”
- Output the pruned model: Save the new weights and configuration, and compute the parameter reduction from the structured pruning.
The code below shows a minimal working implementation (illustrative):
from collections import defaultdictdef profile_expert_load(model, layers, router, tokens):"""Count how many tokens each MoE layer's experts are routed"""counts = [defaultdict(int) for _ in layers]for token in tokens:# router returns the list of expert IDs hit by each tokenfor i, hit_ids in enumerate(router(model, token)):for eid in hit_ids:counts[i][eid] += 1return countsdef prune_experts(layers, counts, min_ratio=0.01):"""Remove experts below the usage threshold and update layer config"""for i, layer in enumerate(layers):total = sum(counts[i].values()) or 1kept = [eid for eid, c in counts[i].items()if c / total >= min_ratio]if not kept:# Fallback: keep the single highest-load expertkept = [max(counts[i], key=counts[i].get)]layer.drop_experts([e for e in range(layer.num_experts)if e not in kept])layer.num_experts = len(kept)
3.4 Recovery Fine-Tuning After Pruning (Optional)
If the capability retention rate after pruning is not ideal, you can run a round of parameter-efficient fine-tuning (such as low-rank adaptation) with a small amount of high-quality code and security samples. The goal here is not to inject new knowledge but to let the model re-adapt to the changed routing distribution—i.e., “recovery” rather than “learning.” The fine-tuning data must also be kept separate from the evaluation set to avoid overlap.
3.5 Effect and Cost
In this case, expert pruning was not the largest contributor to size reduction—quantization was. But the value of pruning extends beyond saving a few dozen GB:
- Pruned parameters no longer participate in quantization, reducing subsequent workload;
- With fewer active parameters, fewer weights need to reside in memory;
- It acts on a different redundancy dimension than quantization, so the two effects stack rather than replace each other.
4. Step Three: Low-Bit Quantization
4.1 How Quantization Works
Quantization maps weights originally represented in 16 bits (BF16/FP16) to low-bit integers (commonly INT8, INT4), attaching a scale factor to a group of weights. During inference, they are converted back to floating point for computation. Taking weights as an example:
- INT8 gives a theoretical compression ratio of about 2x;
- INT4 gives a theoretical compression ratio of about 4x;
- Accounting for extra overhead such as scale factors, the actual ratio is roughly 3–4x.
The mainstream approach today is “weight-only” quantization: weights are stored in low bits, while activations are still computed in floating point. It’s simple to implement, keeps accuracy loss controllable, and the reduced weight-read bandwidth directly speeds up decoding.
4.2 Mainstream Quantization Algorithms
- Naive round-to-nearest quantization (RTN): Directly partitions the range by min/max values; simple to implement. When the weight distribution has long tails, the error is large, so it’s usually only used as a reference baseline.
- Layer-wise quantization based on second-order information (GPTQ-style): Quantizes weights block by block column-wise, using approximate Hessian information to compensate for the error that already-quantized columns introduce to the remaining columns. Accuracy is good, but it’s sensitive to calibration data.
- Activation-aware scaling quantization (AWQ-style): Based on the observation that a few salient weight channels correspond to larger activation magnitudes, it applies equivalent scaling to those channels before quantization according to activation statistics, thereby protecting critical weights. It has low requirements on the calibration set and generalizes more robustly.
- KV-cache quantization: In long-context scenarios, the memory footprint of the key-value cache (KV Cache) grows rapidly with context length and concurrency, often becoming a throughput bottleneck. Quantizing the KV cache to INT8 or fewer bits as well can significantly reduce this overhead. This is activation-side quantization, complementary to weight quantization.
These methods have become standard capabilities in the open-source community and in major inference frameworks, so you can select what you need in engineering practice.
4.3 Sensitive Parts That Must Be Protected
Not all weights are suited to low-bit representation. The following usually should retain higher precision:
- The embedding layer and output projection layer (large vocabulary, directly determining token-representation precision);
- The few layers at the very start and end of the network;
- The attention projections that most affect the output distribution.
In addition, the calibration set should cover real code and vulnerability samples in the target languages. Calibrating with general dialogue corpora often fails to achieve ideal accuracy on code tasks.
5. Coordination: Order, Size Accounting, and Capability Retention
5.1 Prune First, Quantize Second
For the compression stage, the recommended order is “prune first, quantize later.” In practices like Altar’s, quantization carries the main size reduction while pruning plays a supporting role. After pruning, the parameter distribution changes, so pruning first lets quantization always face the parameters in their final form. Conversely, if you quantize first and then prune, the quantizer is based on the old distribution, and error compensation after pruning will be off. By the same logic, any recovery fine-tuning after pruning should also come before quantization.
5.2 Get the “Size Accounting” Right
The number often cited is the on-disk weight size of the full-precision model, for example 1.51TB. But actual deployment also involves:
- Memory and GPU memory: weights + KV cache + runtime overhead;
- Quantization-format differences: different quantized versions of the same model (such as an officially released FP8 weight versus community quantizations) can differ several-fold in size. When doing baseline comparisons, always confirm you’re comparing the same precision format;
- Number of deployment units: the more compact the weights, the easier to deploy on a single machine with a single GPU, avoiding the communication overhead and deployment complexity of multi-GPU tensor parallelism.
5.3 Quantization vs. Pruning Compared
| Compression method | Dimension acted on | Main benefit | Main risk |
|---|---|---|---|
| Expert pruning | Total parameter count (structure) | Reduces weight size and memory residency | Routing distribution shifts; low-frequency capabilities may drop |
| Weight quantization | Bits per parameter | Size drops by multiples; faster inference when bandwidth-bound | Accuracy loss at low bits, especially in sensitive layers |
| KV-cache quantization | Runtime activations | Saves memory in long-context, high-concurrency scenarios | Accuracy may drop on long text; needs separate evaluation |
5.4 Capability-Retention Evaluation
The 92% figure must be interpreted together with the evaluation methodology. It’s recommended to build a multi-dimensional evaluation matrix:
- Evaluation dimensions: vulnerability-detection rate (the core metric), false-positive rate, detection rate across different vulnerability types, and the usability of fix recommendations;
- Comparison targets: the full-precision base model, the low-bit base model, and the compressed model;
- Layered reporting: broken down by language, vulnerability type, and difficulty, to avoid an aggregate score masking local regressions;
- Additional focus: whether the model can reliably emit structured results, its behavior under extreme context lengths, and its inference latency and throughput.
If recall, false positives, and structured output all stay above ninety percent of the base, the model can be considered production-ready.
5.5 Security Properties Are Themselves Evaluation Items
In the security domain, the model’s own robustness also needs verification:
- Prompt-injection resistance: Comments or strings in scanned code may contain instructions that induce the model to output “no risk.” A malicious developer could plant such content in code to fool a review, so both pre- and post-compression models should be tested for their resistance;
- Supply-chain verification: The weight hash of the compressed artifact should match the value published by the provider, guarding against tampering in the download chain;
- Output trustworthiness: Model conclusions should come with supporting snippets and retain a human-review step; they should not serve as the sole basis for blocking a release.
6. Private Deployment and Implementation Advice
6.1 Resource Planning
Estimate first, then validate with a small-scale stress test:
GPU memory need ≈ weight size (in the quantized format)+ KV cache (proportional to context length and concurrent requests)+ runtime and activation overhead
Example: Suppose the compressed weights are about 120GB, which a single machine with a single GPU cannot hold. But 8 accelerator cards with 80GB each provide 640GB total, theoretically enough. However, the KV cache and runtime overhead consume a fair amount of that headroom, so be sure to reserve 20%–30% redundancy and confirm through stress testing the actual supported context length and concurrency.
6.2 Service Integration
It’s best to plug the compressed model into the development lifecycle in the form of a “security LLM inference service”:
- Trigger methods: on merge-request submission, scheduled full scans daily, and pre-release gates;
- Result format: uniformly output “file, line number, vulnerability type, dangerous data-flow path, and fix recommendation”;
- Trust annotations: Model conclusions should clearly state confidence or provenance to help developers prioritize, and to help the security team continuously accumulate real false-positive and false-negative samples that feed the next iteration.
6.3 Continuous Operation
Going live is not the finish line. Vulnerability patterns evolve, and tech stacks change, so it’s recommended to do three things regularly:
- Add newly disclosed vulnerabilities to the evaluation set;
- Monitor online false-positive rates and human-review pass rates;
- Decide, based on monitoring results, whether a new round of pruning, quantization, or fine-tuning is needed.
7. Common Pitfalls and a Checklist to Avoid Them
- Looking only at size, not capability retention: A compressed model must still pass the same evaluation set as the base. Size cannot be the sole measure of success.
- Estimating GPU memory from disk size: Quantization only affects weights; the KV cache and runtime overhead do not shrink proportionally, especially in long-context, high-concurrency scenarios.
- Using general corpora for quantization calibration: Calibration data should match the target scenario’s distribution; prioritize real code and vulnerability samples.
- Blindly pushing the compression ratio: Low-bit quantization loses noticeably at high compression ratios, and too high a pruning ratio also harms low-frequency but critical capabilities (such as rare languages and complex logic vulnerabilities). Find the cost-performance sweet spot rather than chasing the smallest size.
- Ignoring prompt-injection risk: Once the model is wired into CI/CD, malicious comments in scanned code may induce wrong conclusions, so defensive design and a human-review mechanism are needed.
- Overlooking licensing and provenance verification: Confirm the base’s license before commercial use; verify the hash when distributing compressed artifacts and note the derivative relationship.
8. Conclusion
Altar’s practice offers security engineering teams three takeaways:
- At a time when leading open-source models routinely reach TB scale, a strong base and lightweight deployment are not mutually exclusive;
- Expert pruning and quantization handle structural and representational redundancy respectively, and stacking them can shrink the size to about one-fifth of the original;
- Even so, more than ninety percent of the vulnerability-detection ability is retained.
For implementation, proceed along the path of “choose the right base → prune redundant experts → apply low-bit quantization → run multi-dimensional capability evaluation → deploy as a private service,” and insist on “same-methodology comparison before and after compression” at every step. This lets you avoid both extremes—being unable to afford deployment and lacking capability—and truly integrate the security LLM into everyday development workflows. For most companies, rather than waiting for a giant model that “does everything,” it’s better to polish a dedicated model that is “good enough, controllable, and deployable” on your own code and threat distribution.
