Encoder Training Beyond the Trade-Off: Combining Text Classification and Semantic Retrieval

    Bidirectional encoders are often trained separately for classification and semantic retrieval, but this split is not inevitable. This article examines how pretraining choices—especially masking strategy, supervision density, contrastive objectives, and multi-task weighting—affect whether one encoder can support both clear decision boundaries and vector-space retrieval. It also outlines a four-stage training and evaluation path and key deployment considerations.

    In natural language processing systems, text classification and semantic retrieval are often treated as two different problems.

    Classification tasks require a model to determine which category a text belongs to, such as topic, intent, risk level, or content type. Semantic retrieval requires a model to map questions, documents, or candidate answers into a vector space and then find the semantically closest results based on vector similarity. The former emphasizes boundary decisions, while the latter emphasizes distance relationships. Because the training signals differ, engineering practice often requires separate models for the two tasks.

    However, this is not an immutable rule. A growing body of research suggests that whether an encoder can perform both classification and retrieval well depends not only on network depth, hidden dimension, or attention structure, but also on whether pretraining has learned stable, transferable language representations. In other words, the problem may not be that the model is not large enough, but that the model has not been trained in the right way.

    1. Why Bidirectional Encoders Still Matter

    A bidirectional encoder can use context from both before and after a token, compress an entire text into a single vector, or generate context-dependent representations for each position in the text. Unlike decoder-style models used mainly for step-by-step text generation, encoders are better suited to understanding tasks that require high concurrency and low latency.

    In real systems, encoders are commonly used for:

    • Text classification and intent recognition;
    • FAQ, knowledge base, and document retrieval;
    • Reranking and candidate filtering;
    • Data deduplication and similar-content detection;
    • Training data quality assessment;
    • Content moderation and risk scoring;
    • Text matching in recommendation systems;
    • The recall stage in retrieval-augmented generation systems.

    These tasks usually do not require the model to generate long text. Instead, they require fast representation, comparison, or scoring. Therefore, an encoder with a moderate parameter count and stable inference speed is often better suited for deployment in online services, data processing pipelines, and large-scale recall systems.

    However, traditional encoder pretraining methods have long relied on early experience, most notably masked language modeling.

    2. The Problem with Masked Language Modeling Is Not Just “How Many Tokens to Mask”

    The basic process of masked language modeling is to select some tokens from the input text, replace, perturb, or hide them, and then require the model to recover the original content based on the remaining context.

    For example, the original sentence is:

    1. The user wants to quickly find a technical document.

    After masking, it might become:

    1. The user wants to quickly find a [MASK] document.

    The model needs to predict the content at the hidden position. Through large amounts of similar training, the encoder gradually learns dependencies between words, syntactic structure, and some semantic patterns.

    One important hyperparameter is the masking ratio, which is the proportion of tokens hidden in the input relative to all tokens. Early approaches usually used a low fixed ratio, which was reasonable given the data scale and compute budget at the time. However, as training data expands from billions of examples to larger scales, the original ratio may no longer be appropriate.

    The reason is that the masking ratio determines how much supervision signal each training sample can produce.

    If the masking ratio is too low, the model sees input text that is too complete, the task is too easy, and each sample provides only a limited number of effective prediction positions. The model may learn more about local word co-occurrence than deeper sentence-level understanding.

    If the masking ratio is too high, the input context is damaged too severely. The model lacks sufficient clues and can only rely on shallow statistical patterns to guess, making the training objective noisier and potentially affecting representation stability.

    Therefore, a reasonable masking ratio is not a permanently fixed constant. It should be determined together with the following factors:

    1. The scale and quality of training data;
    2. The average length of individual texts;
    3. Model parameter count and context window;
    4. Training steps and total compute budget;
    5. The specific form of the pretraining objective;
    6. The downstream task’s requirements for sentence-level semantics.

    3. Why Larger Training Budgets Change the Optimal Strategy

    Under a smaller training budget, the model needs to make full use of each sample. A lower masking ratio preserves more context and makes convergence easier.

    However, when training data and training steps increase substantially, the model already sees a sufficiently rich variety of text. At that point, if each sample still predicts only a small number of positions, a problem of insufficient supervision signal density may arise. Although the model processes a large amount of text, it does not obtain enough effective learning targets from each sample.

    Increasing the masking ratio can bring three changes.

    3.1 Increase Per-Sample Supervision Signal

    The same text segment can produce more prediction positions, and the model encounters more training targets in a single forward pass. This helps improve compute utilization, especially for large-scale pretraining.

    3.2 Strengthen Contextual Reasoning

    When some key words are hidden, the model cannot rely only on adjacent words. It must combine syntax, topic, and context relationships to make judgments. For classification and retrieval tasks that require understanding the whole sentence, this training pressure is often more valuable.

    3.3 Reduce Reliance on Local Clues

    A low masking ratio can make it easy for the model to complete the task through simple neighboring-word prediction. For example, if a word is often preceded or followed by a fixed collocation, the model can give the correct answer without understanding the meaning of the whole sentence. More challenging masking forces the model to build a more complete semantic representation.

    Of course, a higher masking ratio does not mean the higher the better. In actual training, different strategies should be compared on a validation set, and transfer performance should be observed beyond language modeling loss, especially classification accuracy, retrieval recall, and vector clustering quality.

    4. Why Classification and Retrieval Tend to Conflict

    Classification models usually use cross-entropy loss. The model only needs to make the correct category score higher than other categories; the exact distances between categories are not necessarily important.

    Semantic retrieval, by contrast, focuses more on vector space structure. Semantically similar texts should be close to each other, and unrelated texts should remain sufficiently far apart. Even if two texts ultimately belong to the same category, a retrieval model may still want to preserve differences in the details they express.

    This leads to different optimization priorities for the two tasks:

    • Classification focuses on decision boundaries;
    • Retrieval focuses on relative distances;
    • Classification can rely on a small number of highly discriminative features;
    • Retrieval needs to preserve richer semantic information;
    • Classification outputs are usually discrete labels;
    • Retrieval outputs are usually continuous vectors.

    If a model receives only classification supervision, it may over-compress information so that all texts in the same category cluster together, reducing fine-grained retrieval ability. Conversely, if a model receives only similarity supervision, it may struggle to form clear category boundaries and perform unstably on tasks with explicit labels.

    Therefore, a general-purpose encoder needs to address these two requirements separately during pretraining and task adaptation.

    5. A Training Path for Building a General-Purpose Encoder

    A relatively robust training process can be divided into four stages.

    Stage 1: High-Quality Text Pretraining

    First, use text with sufficient scale, low duplication, and reasonable domain coverage for masked language modeling. Data cleaning is more important than blindly increasing data volume. Particular attention should be paid to garbled text, templated content, duplicate documents, and low-information-density text.

    The masking strategy can be dynamic: each time the same text is read, different positions are randomly selected for masking. This allows the model to obtain multiple training views from the same piece of data and reduces overfitting caused by fixed masking.

    Different strategies can also be used according to text structure:

    • Use finer-grained token masking for short texts;
    • Appropriately increase continuous span masking for long texts;
    • Preserve different sampling weights for titles and body text;
    • Avoid breaking key formats in structured fields;
    • Mask technical terms at the whole-term or span level.

    Stage 2: Sentence-Level Representation Learning

    After basic pretraining, the model needs to further understand semantic relationships between sentences. A common method is to construct positive and negative samples.

    Positive samples can come from:

    • Different truncations of the same text;
    • Paraphrases of the same semantic content;
    • Questions and their corresponding answers;
    • Titles and body text;
    • User queries and relevant documents.

    Negative samples can be selected from random texts, other samples in the same batch, or candidate texts that are easily confused after an initial retrieval step.

    The training objective usually requires positive sample vectors to be closer and negative sample vectors to be farther apart. Compared with a pure classification loss, this contrastive training directly optimizes vector space structure, which is especially important for semantic retrieval.

    Stage 3: Multi-Task Joint Optimization

    To prevent the model from favoring one type of task, classification and matching objectives can be added at the same time:

    1. Total loss = classification loss
    2. + α × contrastive learning loss
    3. + β × masked language modeling loss

    Here, α and β control the contribution of different objectives. They should not be understood as fixed standards; they should be adjusted according to data scale, task difficulty, and validation results.

    During joint training, attention must be paid to the data sampling ratio. If there are too many classification samples, the model may again favor category boundaries. If there are too many retrieval samples, the model may sacrifice category discrimination ability. In engineering practice, staged training, dynamic sampling, or task-difficulty-based weighting can be used.

    Stage 4: Evaluation for Real-World Scenarios

    It is not enough to look at a single metric. Evaluation should cover at least the following dimensions:

    • Classification accuracy, macro-average metrics, and long-tail category performance;
    • Retrieval recall, ranking quality, and hard-negative identification ability;
    • Stability across different text lengths;
    • Cross-domain transfer ability;
    • Vector inference speed and memory usage;
    • Performance degradation after changes in data distribution.

    Particular attention should be paid to cases where “the average score is good, but critical scenarios fail.” For example, a model may perform well on ordinary queries but fail to distinguish texts that are semantically similar yet have different business meanings. Such problems often require additional domain-specific negative samples and fine-grained evaluation sets.

    6. Engineering Considerations During Deployment

    After model training is complete, deployment efficiency is equally important. Semantic retrieval usually requires offline computation of document vectors, online computation of query vectors, and approximate search in a vector index. Therefore, encoder output dimension, batching strategy, and vector normalization all affect system cost.

    In actual deployment, consider the following:

    1. Use batch inference to improve hardware utilization;
    2. Encode long documents in segments and then aggregate;
    3. Cache high-frequency queries;
    4. Recall candidate documents first and rerank them afterward;
    5. Choose different precision inference options based on latency requirements;
    6. Regularly monitor whether the vector distribution has drifted;
    7. Continuously evaluate new domain data.

    It is important to emphasize that a general-purpose encoder does not mean one model can unconditionally replace all specialized models. Its value lies in providing more balanced foundational capabilities and reducing the cost of building separate classification, matching, and retrieval systems. For extremely specialized domains, strongly rule-based scenarios, or high-risk tasks, domain-specific adaptation with domain data is still necessary.

    7. Conclusion: The Real Breakthrough May Come from the Training Recipe

    The development of encoders is not only about adding more layers, increasing parameter count, or extending context length. Many performance bottlenecks may be hidden in seemingly ordinary training details, such as masking ratio, negative sample construction, data duplication rate, task sampling, and loss weights.

    When training budgets, data scale, and application goals change, default settings inherited from the past should also be revalidated. A more reasonable direction is to use masked language modeling to learn robust language structure, use contrastive learning to shape a retrievable vector space, and then use classification objectives to establish clear decision boundaries.

    An encoder trained in this way has the opportunity to serve classification, retrieval, matching, ranking, quality assessment, and other tasks at the same time. Classification and retrieval are not inherently opposed. What often needs to change is not the choice between them, but the opportunity for the model to acquire both capabilities during training.