Bidirectional Encoder Classification and Retrieval: Why the Mask Ratio, Not the Architecture, Is the Real Trade-off
Bidirectional encoders have long seemed forced to choose between strong classification and strong retrieval. This article argues that the trade-off comes not from model architecture but from the training recipe—specifically the masked-language-modeling (MLM) mask ratio, which has been left at its default of 15% for years. Building on the idea that the optimal mask ratio drifts with training budget, the article explains why classification and retrieval place different demands on the vector space, why larger compute shifts the optimal ratio upward, and how to run a controlled mask-ratio ablation that scores both classification and retrieval. It also covers how to calibrate the ratio on smaller budgets, complementary retrieval-side objectives, and the caveats around data, scale, and evaluation. Readers will come away with a practical case for treating the mask ratio as a first-class hyperparameter and plotting their own score-versus-ratio curve.
1. Why Encoders Still Matter
Generative models dominate almost every technical discussion in the current LLM wave. Yet in search, recommendation, risk control, and knowledge-base question answering, the workhorses are often the much smaller bidirectional encoders: models that compress an entire passage into a fixed-dimension vector for scoring, ranking, deduplication, or clustering.
A typical case is the retrieval layer of a retrieval-augmented generation (RAG) system. A large generative model handles the final answer, but “finding the few dozen most relevant documents out of millions” is extremely latency-sensitive. It usually has to complete in tens of milliseconds while absorbing massive online traffic, so calling a generative model per candidate is impractical on both cost and speed. Encoders bring three advantages here:
- Low cost: a single forward pass costs far less than a generative model, and it can be batched and run continuously;
- Controllable latency: inference does not depend on autoregressive decoding, so latency stays stable;
- Clean output structure: a fixed-dimension vector or a class probability is easy to index, threshold-filter, and wire into a pipeline.
The same needs show up across many pipelines:
- using a classifier to quickly filter low-quality training data;
- using a cross-encoder or reward model to score generative outputs and provide a signal for preference alignment;
- semantically merging product titles and routing support tickets by intent.
Quality demands keep rising on these tasks. Once an encoder is “accurate for classification but weak for retrieval,” retrieval quality directly caps the ceiling of the whole pipeline.
2. The “Pick One” Trap
Around 2019, the NLP community’s consensus was that bidirectional encoders were better suited to understanding and classification, while their semantic representations were relatively weak, so retrieval usually required a separate approach. Over the following years, encoders kept absorbing innovations from the decoder ecosystem: new positional encodings such as rotary position embeddings (RoPE) improved length extrapolation, and efficient attention implementations lowered the cost of long-context training. Newer models climbed steadily on classification leaderboards.
Yet one pattern kept recurring: encoders that excel at classification still fall short on retrieval tests, while models optimized for semantic retrieval perform unremarkably on classification. The explanation is often pinned on “capacity allocation” or “architectural trade-offs.” A university research team was skeptical of these claims and proposed a simpler hypothesis: the problem may not be the architecture, but the training recipe.
The reasoning is direct. Architectural innovations have been widely adopted, but the MLM pretraining objective has barely changed and still follows the practice from the classic bidirectional-encoder era. Its most important parameter is the mask ratio—how large a fraction of tokens are hidden during training. This value has long defaulted to 15% and is treated almost as common knowledge. But the team noticed that experiments on the “optimal mask ratio” were almost all conducted under training budgets of roughly 10 to 100 billion tokens. Whether that conclusion still holds once the budget rises by one or even two orders of magnitude had never been seriously checked.
3. Masked Language Modeling and the Mask Ratio
Masked language modeling works like this: randomly hide a portion of the input tokens, usually replacing them with a special [MASK] token, then have the model recover them from the remaining context. A common practice splits the selected tokens into three buckets—replaced with [MASK], with a random token, or left as the original word—so the model does not learn only to recognize the placeholder.
To understand the mask ratio’s effect, start with what it changes:
- How incomplete the context is: fewer masks mean the model sees more complete information when predicting a position, making recovery “easier”;
- Supervision density: fewer positions truly need predicting per batch, so each unit of compute yields a sparser training signal;
- Information-integration distance: with denser masking, the model can no longer rely on a few neighboring characters to “guess” a word and must integrate longer-range, higher-level semantics.
Put another way, the regularities a pretrained model distills from data fall into roughly three layers:
- Statistical: common collocations and local co-occurrence;
- Knowledge: entity attributes, world facts, and other factual associations;
- Capability: disambiguation, coreference, reasoning, long-range dependencies, and other abilities that require actively integrating context.
The masking task is like an exam for the model, and the mask ratio sets its difficulty. Too easy, and the student can “guess” the answer without learning deeper abilities; too hard, and the student cannot get started, so the training signal becomes thin.
4. The Deeper Reason for “Pick One”: The Two Tasks Demand Different Representations
Classification and retrieval both look like “understanding text,” but they place very different demands on the vector space:
| Dimension | Text classification | Semantic retrieval |
|---|---|---|
| Core dependency | Discriminative local features | Overall semantics + comparability between query and document |
| Typical approach | Pool to a vector, add a classification head, fine-tune with supervision | Encoder outputs a vector directly; compute cosine similarity |
| Decision style | Learn a decision boundary within a single task | Compare across queries and documents; depends heavily on the geometry of the space |
| Common failure mode | Over-relies on keywords and topic words | Semantically close items fail to rank; semantically unrelated items rank high |
Take sentiment analysis: for a short sentence containing “although … but …”, the model can judge polarity just by catching the evaluative word after the turn. It can safely ignore the sentence’s overall structure. For retrieval, though, queries are often short, colloquial, and keyword-sparse, while documents are long and diverse in phrasing. Retrieval is accurate only when the overall semantics are compressed into the same space, and when vectors with similar meaning sit close together while vectors with different meaning stay far apart.
Under a limited training budget, gradients naturally flow toward patterns that are easier to learn and pay off faster. A low mask ratio makes the training objective “cheap,” and the model quickly settles into a “keyword-matching” style of understanding: the gains on classification data are immediate, but the geometric quality of the vector space is sacrificed.
This also explains the “reverse bias”: a model trained for contrastive retrieval optimizes similarity directly but rarely touches labeled supervision, so it struggles on classification tasks that need fine-grained decision boundaries.
A point worth emphasizing: since the root cause is the optimization objective’s over-reliance on local features, and the mask ratio directly controls “how much the model is forced to integrate context,” treating the mask ratio as the tuning knob between classification and retrieval is logically sound. That is precisely where the research team started.
5. Why Large Compute Changes the Optimal Mask Ratio
The team’s core claim: the optimal mask ratio is not a constant but a quantity that varies with training budget. This can be argued from a few angles.
1. Conclusions have boundaries of applicability
Optimal hyperparameters in large-scale pretraining often drift with scale—the relationship between learning rate and batch size is a classic example. If early ablations were constrained by compute and could only sweep parameters between roughly 10 and 100 billion tokens, then the optimum on the curve may have already moved at the trillion-token scale.
2. There is a “difficulty match” between the mask ratio and the training budget
- Limited budget: the model has not fully absorbed the basic knowledge in the data, so too high a mask ratio raises task difficulty and dilutes the supervision signal at the same time—a losing trade.
- Sufficient budget: the model has seen enough corpus and already masters the basic statistical regularities, so recovery under a low mask ratio is too easy for it, and the remaining compute goes more into memorizing local co-occurrence than into developing global semantic-integration ability.
From this angle, the mask ratio should rise as compute increases; it must match the data scale.
3. Gains in representation ability transfer across tasks
Classification and retrieval share the same encoder backbone. Improving representation quality will not benefit only one of them: once the model is forced to rely on global semantics rather than keywords, classification and retrieval accuracy can improve together. This is the key to “not having to pick one.” It aligns with the general design principle of “sharing low-level capability and adapting high-level tasks separately.”
6. Validation: A Proper Mask-Ratio Ablation
For such a conclusion to be convincing, the experimental design must hold up to scrutiny.
1. Variable control
Fix model size, data mixture, context length, tokenization, learning-rate schedule, and total training volume, and change only the mask ratio.
2. Treat training volume as a first-class independent variable
Repeat the same set of ablations under two or more training budgets. Only by observing the optimum shift systematically with the budget can the claim “the optimal ratio drifts with compute” be supported.
3. Multi-granularity evaluation
Do not look only at downstream fine-tuning scores. You can directly compare how representations learned under different mask ratios differ on metrics such as isotropy, alignment, and uniformity, explaining the phenomenon at the mechanism level. Teams sensitive to inference efficiency should also record the downstream gain bought per unit of training compute.
4. A reproducible experimental scaffold
The following code is illustrative only, showing how to parameterize the mask ratio and sweep it in batches:
# Illustrative code: make the mask ratio a configurable variabledef mask_tokens(token_ids, ratio, vocab_size,mask_id, pad_id=0,strategy=(0.8, 0.1, 0.1), seed=0):"""Generate masked training samples at the given ratio.Returns:input_ids: model input, with some positions replaced by [MASK] or a random tokenlabels: original token kept only at selected positions, ignored elsewhere"""rng = random.Random(seed)valid = [i for i, t in enumerate(token_ids) if t != pad_id]n_mask = int(len(valid) * ratio)chosen = set(rng.sample(valid, n_mask)) if n_mask > 0 else set()input_ids, labels = list(token_ids), [-100] * len(token_ids)for i in chosen:labels[i] = token_ids[i]p = rng.random()if p < strategy[0]:input_ids[i] = mask_idelif p < strategy[0] + strategy[1]:input_ids[i] = rng.randrange(vocab_size)# Otherwise keep the original tokenreturn input_ids, labels# Sweep different mask ratios at each budget tier, keeping other settings fixedBUDGETS = [{"tokens": 1_000_000_000, "steps": ...},{"tokens": 10_000_000_000, "steps": ...},]RATIOS = [0.15, 0.30, 0.40, 0.50]for budget in BUDGETS:for ratio in RATIOS:train(max_steps=budget["steps"],mask_ratio=ratio,config=fixed_config(), # keep model size, LR schedule, etc. unchanged)evaluate(run_name=f"budget{budget['tokens']}_ratio{ratio}",suites=["classification", "retrieval"],)
One common pitfall is worth noting: if you average the loss over the total number of selected tokens in a batch, the effective amount of signal differs across mask ratios, and the comparison gets contaminated by an irrelevant variable. A cleaner approach is to normalize per sample and then average over samples, so every sample contributes with equal weight.
5. Additional training objectives on the retrieval side
The mask ratio mainly improves the backbone representation and cannot replace the specialized optimization retrieval needs; the two should work together:
- a contrastive learning objective that pulls queries and positive documents together while pushing hard negatives apart;
- in-batch negative sampling, which uses other samples in the same batch as negatives—low cost, effective;
- adding a denoising autoencoder–style objective on top of the original masking target, i.e., shuffling token order first and then recovering it, forcing the model to capture word-order information; this is especially helpful for fine-grained semantic matching.
7. How to Calibrate the Mask Ratio in Your Own Scenario
If you do not yet have a trillion-scale budget, there is no need to copy some “best number.” You can run a small-scale calibration following these steps:
- Align the data distribution: make the domain mix of the calibration corpus close to your actual workload. Corpora that mix domains, languages, code, and tables should be re-proportioned toward the target scenario.
- Small-model proxy experiments: use a smaller model with fewer layers to sweep a set of mask ratios (e.g., 15%, 30%, 40%, 50%) and plot a “downstream score–mask ratio” curve.
- Extrapolate, don’t copy: check whether the curve has passed its peak and whether the peak position shifts right as the budget grows, then set the value for full training accordingly, and validate with one ratio tier during full training.
- Combine masking strategies: simply raising the ratio is not necessarily optimal. Whole-word masking forces the model to recover full semantics from subword combinations, and span masking (hiding a whole phrase at once) forces it to integrate longer-range context; both control the same “task difficulty” dimension.
- Watch long-tail domains: terminology-dense fields such as healthcare, law, and finance should be evaluated separately for how changes in the mask ratio affect the representation of rare entities.
- Configuration-ize the recipe: write the mask ratio, masking strategy, pooling method, training objective, and loss normalization into the training config and put them under version control. The recipe is at least as important as the architecture, yet in practice it is often scattered across scripts and almost never systematically reviewed.
- Leverage general cloud capabilities: in large-scale pretraining, corpora usually need to be cleaned, deduplicated, and desensitized first, then read dynamically as training progresses. Here, general cloud capabilities such as object storage, data pipelines, containerized training platforms, logging services, and monitoring/alerting are common choices. For example, Baidu AI Cloud offers object storage BOS for storing and managing corpora, container engines and other general-purpose capabilities can host distributed training jobs, and cloud monitoring BCT can track training metrics and resource costs. Availability and coverage vary by region, so refer to the official documentation.
8. Boundaries and Caveats
This conclusion has a scope of applicability, and you should keep the following in mind before copying any number:
- Depends on data and scale: the optimal value varies with corpus domain, model size, context length, and total training volume. The optimum for a small, single-domain setting may not be close to that of a large, general-purpose setting.
- Benefits vary by scenario: tokenization granularity, domain drift, and the share of long documents all affect the actual payoff. Scenarios with large domain datasets and many long documents generally have more room to gain from adjusting the mask ratio.
- Retrieval quality depends on the whole chain: the mask ratio changes only the pretraining stage. Hard-negative construction, batch size, temperature coefficient, and how queries and documents are represented all shape the final result.
- Evaluation must be unified: use the same evaluation protocol across different mask ratios, with the same pooling method and normalization strategy; otherwise the conclusions cannot be compared.
9. Conclusion
The “pick one” between classification and retrieval for encoders is, rather than a fate dictated by architecture, a mark left by a long-standing training hyperparameter under specific compute conditions.
- The mask ratio sets the difficulty: it controls how much the model is forced to integrate context, and in turn shapes the properties of the vector space;
- Scale moves the optimum: as training budget, model size, and data volume change, the optimal ratio moves with them;
- The recipe needs a matching upgrade: now that architectural innovations have been widely absorbed, upgrading the training recipe is a lower-investment path with no small potential.
If you are training or fine-tuning an encoder that must serve both classification and retrieval, try treating the mask ratio as a first-class hyperparameter first and plot your own “score–ratio” curve. The result probably will not match everyone else’s, but the judgment “does it move with the budget” alone is enough to guide your next training run.
