Encoder Training Recipes: Balancing Classification and Retrieval
This article explains why bidirectional encoders often struggle to serve both classification and retrieval tasks at the same time, and provides a practical training recipe that balances both objectives without sacrificing either. It covers loss combination, batch construction with hard negatives, pooling and normalization choices, and evaluation protocols that track both metrics simultaneously. The goal is to help engineers build a single encoder that performs well for semantic matching, classification, and retrieval in production environments.
Bidirectional encoders compress an entire text into a vector for classification, scoring, and retrieval, rather than generating tokens one by one. They remain practical in latency-sensitive and cost-sensitive scenarios such as retrieval recall, training data filtering, and reward scoring: a smaller encoder is often more economical than repeatedly calling a large generative model, and its results are not necessarily worse.
For a long time, the industry has observed a recurring pattern: for the same family of encoders, when classification metrics improve, retrieval performance tends to drop; when contrastive learning is pushed to the limit and retrieval becomes stronger, classification lags behind. The analogy of a sprinter versus a marathon runner is vivid, but attributing this tension to an “inherent limitation of encoder architecture” is not rigorous. In most cases, the training recipe still targets a single objective, a single negative sampling distribution, and a single evaluation habit. Once the representation space is shaped by one task, the other task naturally suffers.
The Bottleneck Is the Objective, Not the Bidirectional Architecture
The value of bidirectional encoding lies in the ability of each position to see both left and right context, making sentence or token vectors suitable for matching and discrimination. Generative decoders predict the next token from left to right, optimizing a continuation distribution rather than whether two texts are related or belong to the same class. The loss functions served by the two model types are different, and generation quality cannot directly substitute for encoder metrics.
When an encoder gets stuck, it is usually a combination of the following issues.
First, the pretraining objective is too narrow. If only masked language modeling is used, the model becomes good at completing local contexts, but the geometric structure of sentence-level “relatedness” is unstable. If only large-batch contrastive learning is used, vectors are pulled toward “distinguishable neighbors” — class boundaries become sharper, but fine-grained classes, multi-label settings, and calibrated classification probabilities degrade.
Second, batches and negatives do not represent real retrieval scenarios. Negative classes in classification data are usually other labels in the annotation system; negatives in retrieval are hard neighbors from a massive candidate pool. A space trained with random in-batch negatives may classify small categories well, but it collapses when applied to a million-scale corpus. Conversely, mining only extremely hard negatives pushes the decision boundary to fit a few adversarial samples, squeezing out margin for ordinary categories and hurting classification accuracy and calibration.
Third, pooling and normalization are treated as implementation details. Taking the first token, doing mean pooling, or applying weighted averaging over hidden states changes the isotropy of the vector space. Retrieval usually uses cosine similarity, which implies unit norm. Classification often uses a linear layer plus cross-entropy, where the norm itself can carry “confidence” information. If normalization is forced throughout, the amplitude information usable by the classification head is erased. If no normalization is applied, inner-product retrieval is dominated by vector magnitude, and semantic neighbors are replaced by texts that are longer or have larger activations.
Fourth, evaluation only looks at one score. Classification is measured by accuracy or macro-F1, retrieval by recall or ranking. Optimizing only one of them causes the other capability to degrade invisibly. A recipe is only effective if both types of metrics are evaluated at the same checkpoint, not measured as an afterthought.
A Training Recipe That Serves Both Tasks
The following is a model-agnostic approach, organized as “first stabilize the representation, then adapt the heads separately.” Do not expect a single loss weight to magically solve all data problems; weights are merely knobs to turn after you have defined batches, negatives, and pooling clearly.
1. Shape the Same Backbone with Two Objectives Instead of Choosing One
Masked language modeling preserves token-level semantics and domain vocabulary; sentence-level contrastive or supervised matching preserves the geometry of “related / not related.” In practice, you can use staged training or combine both objectives in the same step with a small weight.
Staged training is more stable: first use the reconstruction objective to fill in domain words and long-range context, then use the matching objective to separate sentence vectors. When adding both objectives in the same step, the numerical scales of reconstruction loss and contrastive loss often differ by several orders of magnitude. You must apply sliding normalization or fixed coefficients separately; otherwise, one will dominate the gradient. There is no universal optimal coefficient — the target is that both classification and retrieval validation metrics do not drop, rather than copying a set of “numbers from a paper.”
When supervised data exists, classification cross-entropy should be applied to an independent linear head, not replaced by contrastive loss. Contrastive loss optimizes relative distances, while cross-entropy optimizes class posteriors. Sharing the backbone and using separate heads is the lowest-cost structure for balancing both.
Illustrative loss combination (only the form of the loss, not tied to a specific framework):
loss = a * mlm_loss(hidden, mask_labels)+ b * contrastive_loss(normalize(pool(hidden)))+ c * ce_loss(cls_head(pool(hidden)), labels)
Here a, b, and c are tuned based on validation curves; pool and whether to normalize must be recorded in experiment logs. Normalizing the contrastive branch while leaving the classification branch unnormalized is a common and reasonable division: retrieval uses angular distance, classification retains magnitude.
2. Batch Construction Should Follow Retrieval Difficulty, Not Just Label Balance
Classification batches aim for class balance to avoid long-tail classes disappearing within one epoch. Retrieval batches aim for informative negatives: random negatives enforce basic semantic separation, while hard negatives separate near neighbors. Using only hard negatives makes the model overfit to a small set of adversarial samples, hurting classification generalization; using only random negatives caps retrieval performance.
A practical mixed approach: for each positive pair, include a portion of random in-batch negatives and a smaller portion of hard negatives obtained by approximate retrieval using the current encoder, with a limit on the hard-negative ratio. Hard-negative mining can use the previous checkpoint or a momentum encoder; you do not need a complex system. Mining frequency should not scan the full corpus every step, otherwise training is slowed by the retrieval process itself, and the negative distribution fluctuation destabilizes the classification head.
Label leakage must be checked separately. If hard negatives are actually synonymous with the positive, come from the same document split, or are annotation noise, contrastive learning will push away truly related samples, breaking both retrieval and classification. Removing same-source texts with simple rules before adding hard negatives is more important than increasing the learning rate.
3. Manage Pooling, Temperature, and Norm Separately
For sentence vectors, mean pooling with masking is a sensible default, avoiding padding positions. If first-token pooling has been proven more stable in your domain, fix it and run comparisons; do not change pooling per experiment without recording it.
The temperature in contrastive learning controls how sharp the distribution is. If temperature is too low, gradients concentrate on the hardest one or two pairs, destroying the smooth inter-class structure needed by classification. If temperature is too high, negatives provide little discriminative signal, and retrieval does not improve. Temperature is coupled with batch size: the larger the batch, the more negatives are available, and typically the temperature needs to be slightly higher; otherwise, the distribution becomes overly sharp. Do not tune to a “magic temperature” without considering batch size.
When using unnormalized representations for the classification head, apply layer normalization or reasonable dropout to the pooled output to reduce magnitude explosion. For retrieval indexing, use a separate normalized vector. Fix the normalization method before building the index, and online queries must use the same normalization. Exporting two vectors (one for classification, one for retrieval) from the same checkpoint is cleaner than forcing one vector to serve both purposes.
4. Define the Evaluation Protocol First, Then Discuss Recipe Quality
At minimum, track simultaneously:
- Classification: accuracy, macro-F1, and probability calibration (e.g., reliability after binning). Looking at accuracy alone misses cases where “retrieval improves but confidence is distorted.”
- Retrieval: recall and ranking under a fixed candidate size, with separate results on random-negative sets and hard-negative sets. Improvement only on a small candidate set does not prove usability on a million-scale corpus.
- Stability: variance of both metric types across different data splits. If a recipe improves on only one split, it is likely leakage or accidental hard negatives.
Early stopping should not rely on classification loss alone. If classification loss decreases but retrieval recall drops, the contrastive branch is being overwhelmed or hard negatives are no longer effective. In that case, reduce the classification coefficient or restore negative quality, rather than increasing the learning rate.
How to Avoid Choosing Again at Deployment
After training, classification and retrieval often go through different service paths, and the advantage of the recipe is most likely to be lost here.
Retrieval path: vector normalization and similarity (cosine or inner product) must match training. Query and document sides use the same backbone and the same pooling. Document chunk lengths should be on the same order as the maximum training length; overly long chunks will dilute key sentences in mean pooling. After index updates, run regression on a fixed query set to prevent “retraining the classification head” from inadvertently changing the backbone weights.
Classification path: save the linear head separately from the backbone. If you only need to add new labels, prioritize freezing the backbone or fine-tuning it with a small learning rate, to avoid pulling the retrieval space away for a few new classes. When both must be updated, include a retrieval validation set in the early stopping criteria of every fine-tuning step.
Filtering and scoring: when using an encoder for data-quality filtering or reward scoring, clarify whether the score comes from a classification head probability or from similarity to a reference text. Do not mix thresholds for the two score types. Use calibration curves for probability thresholds and recall–precision curves for similarity thresholds.
In terms of resources, encoders are suitable for batch vectorization followed by repeated querying. Generative models are suitable for tasks requiring natural-language explanations. Do not force an encoder to do open-ended generation, and do not default to calling a large generative model in a pure recall pipeline. Storing vectors and original text in object storage, using a message queue for batch encoding, running inference replicas on a container platform, and using logs and monitoring alerts for latency, empty recall rates, and classification confidence drift — these are general engineering capabilities, not tied to any specific vendor.
Recipe Checklist
Acceptance should be based on a checklist, not slogans:
- Is the backbone simultaneously constrained by both token-level reconstruction and sentence-level matching, with aligned loss scales?
- Is the classification head independent, and are retrieval vectors separately normalized?
- Are negatives a mix of random and hard examples, with same-source leakage removed and hard-negative ratio limited?
- Are pooling, temperature, max length, and normalization consistent across training, indexing, and querying?
- Whenever the backbone is changed, are both classification and fixed-corpus retrieval regressed simultaneously?
- Are thresholds separated by score type, rather than using one number for two tasks?
The tension between classification and retrieval is real, but it mainly comes from optimization objectives and data distributions, not from a death sentence of bidirectional encoding itself. By separating reconstruction, contrastive, and classification heads, controlling negative difficulty and vector norms, and constraining every update with dual metrics, the same encoder can serve both discrimination and recall. When the recipe becomes outdated, modify the training and evaluation first, then consider a larger model.
