Knowledge Updating for Large Language Models After Pre-training: RAG, Continued Pre-training, Fine-tuning, and Knowledge Editing

    This article explains why pre-trained large language models still need knowledge updates after training, compares two main approaches—external knowledge injection and internal parameter updates—and covers RAG, continued pre-training or mid-training, full-parameter supervised fine-tuning, parameter-efficient fine-tuning, and knowledge editing. It also provides selection guidance and engineering practices for balancing cost, freshness, and stability.

    1. Background: Why Knowledge Updating Is Needed After Pre-training

    During pre-training, most capabilities and knowledge of a large language model are embedded into billions or even hundreds of billions of parameters. Once pre-training finishes, the parameter weights are fixed, and the model’s knowledge coverage stops at the training corpus cutoff date. This is commonly called the “knowledge cutoff”.

    This creates two practical pain points:

    • Stale knowledge: The training corpus has a cutoff date, while the world, industry standards, code repositories, and API documentation keep changing. The model may know nothing about recent events.
    • Domain gaps: General-purpose corpora often fail to cover specific industries, enterprises, or highly specialized private knowledge. As a result, the model may be broad but not sufficiently precise or professional.

    Therefore, one of the core challenges in LLM engineering is how to enable an already trained model to keep absorbing new knowledge without retraining from scratch. Leading LLM providers often release new versions every few months and refresh the knowledge cutoff date. This article discusses the technical practices behind that process.

    2. Two Main Routes: External Knowledge vs Internal Knowledge

    From the perspective of whether model parameters are modified, knowledge updating can be divided into two main routes:

    1. External knowledge injection (no parameter changes): Knowledge is stored outside the model and dynamically injected at inference time, usually through retrieval. The representative approach is retrieval-augmented generation (RAG).
    2. Internal knowledge injection (parameter changes): Model weights are adjusted directly so that new knowledge is incorporated into the parameters. This includes continued pre-training (including the mid-training stage), supervised fine-tuning, parameter-efficient fine-tuning, and knowledge editing.

    These two routes are not mutually exclusive and can be combined. The key factors in selection are the scale of the knowledge, update frequency, freshness requirements, and affordable compute cost.

    3. External Approach: Retrieval-Augmented Generation (RAG)

    The core idea of RAG is simple: no model parameter needs to be changed. When a user asks a question, the system first retrieves relevant content from an external knowledge base and then passes both the question and retrieved content to the model, so the model can generate an answer based on the retrieved results.

    Typical flow:

    1. User question
    2. → Vector / keyword retrieval
    3. → Retrieve Top-K relevant chunks from the knowledge base
    4. → Combine as [retrieved context + user question]
    5. → Send to the LLM to generate the final response

    Advantages:

    • Lowest cost and most direct: No training is required; connecting to a knowledge base can take effect immediately.
    • Knowledge can be updated in real time: If one entry in the knowledge base changes, the model’s answer can change immediately, providing strong freshness.
    • Effective hallucination mitigation: Research shows that making the model answer based on given, traceable materials can significantly reduce the probability of fabricating facts.
    • Traceability: Answers can include sources, which helps verification and compliance review.

    Limitations:

    • It depends on retrieval quality; inaccurate recall directly affects the answer.
    • The context length is limited, so a huge amount of knowledge cannot be stuffed into the context.
    • Inference latency and cost increase as retrieval and combined content increase.

    Because its cost is controllable and it can be deployed quickly, RAG has spawned many variants, such as query rewriting, multi-round retrieval, and reranking. It has become a mainstream paradigm for enterprise knowledge-base question answering.

    4. Internalized Approaches: From Fine-tuning to Continued Pre-training

    Internalized methods truly change model weights, allowing knowledge to grow into the parameters. Depending on the stage at which they are applied and the degree of modification, they can be divided into several categories.

    1. Continued Pre-training / Mid-training

    There is a mid-training stage after pre-training and before post-training alignment. During this stage, engineers can supplement large-scale corpus data for the target domain, allowing the model to continue learning and fill in missing capabilities or update background knowledge.

    A clear characteristic of this approach is: the model structure remains unchanged, while the training data and training date are replaced or supplemented. When many vendors release new versions, you may see that the network architecture remains the same and only the knowledge cutoff date or version number changes. In essence, this is the path of “mid-training data supplementation + retraining”. It is particularly efficient for large-scale background knowledge updates and helps maintain comparability between versions.

    2. Full-parameter Supervised Fine-tuning and Its Two Major Risks

    Full-parameter supervised fine-tuning updates all model weights and is suitable for injecting a relatively large amount of new knowledge. However, because it directly rewrites parameters that have already been pre-trained, it inevitably brings two serious problems:

    • Catastrophic forgetting: Newly learned knowledge overrides or dilutes old knowledge, causing the model to regress on existing capabilities. The model may learn new knowledge while forgetting old knowledge.
    • Fine-tuning hallucination (knowledge conflict): The model’s existing knowledge conflicts with newly injected knowledge, or boundaries between similar knowledge become unclear, making it difficult for the model to provide accurate and definite answers. This phenomenon is especially serious for numeric knowledge (such as specific parameters, prices, or version numbers), because small numerical differences are easily smoothed out.

    3. Parameter-Efficient Fine-tuning (PEFT / Low-Rank Adaptation)

    To alleviate the high cost and forgetting problems of full-parameter fine-tuning, the industry widely uses parameter-efficient fine-tuning. The idea is to freeze the backbone parameters and only train a small number of newly added low-rank adaptation modules, thereby injecting new capabilities while hardly touching the original knowledge.

    Its value lies in:

    • Small number of trainable parameters and low compute overhead;
    • The backbone weights remain unchanged, which naturally mitigates catastrophic forgetting;
    • Independent adaptation modules can be trained for different tasks or domains and loaded on demand.

    However, PEFT is better at injecting “capabilities/styles/task patterns”. For pure factual knowledge, especially numeric knowledge, the injection effect is still limited and needs to be combined with other methods.

    4. Knowledge Editing

    Knowledge editing targets a more fine-grained goal: modifying one specific fact in the model while trying not to affect other knowledge. It is like performing “surgical” precise updates on model parameters and is suitable for correcting a small number of high-frequency or critical erroneous facts (such as the default value of a certain API or the definition of a proper noun). The challenge lies in the generalization and side-effect control of editing: the change must correctly propagate to related questions while avoiding affecting unrelated knowledge.

    5. Method Comparison and Selection Guidance

    Method Changes parameters? Update cost Freshness Suitable scenarios
    Retrieval-augmented generation (RAG) No Low Strong Frequently changing, traceable private or real-time knowledge
    Continued pre-training / Mid-training Yes High Medium Large-scale domain background knowledge updates and version iteration
    Full-parameter fine-tuning Yes High Weak A large amount of internalized knowledge is needed and forgetting risk can be accepted
    Parameter-efficient fine-tuning (PEFT) Yes (limited) Medium Medium Injecting capabilities/styles while controlling cost and forgetting
    Knowledge editing Yes (targeted) Low to medium Strong Correcting a small number of key facts

    A common combination in practice is: use RAG for high-frequency, traceable, real-time knowledge; use mid-training/continued training to periodically update background knowledge; use parameter-efficient fine-tuning to inject capabilities; and use knowledge editing to fix individual critical errors. This layered strategy can achieve a balance among cost, freshness, and stability.

    6. Engineering Practices

    To make knowledge updating sustainable and measurable, the following points deserve attention:

    • Data governance first: Whether using mid-training or fine-tuning, the quality, deduplication, and mixing ratio of the corpus directly determine the updating effect. In particular, it is important to balance new corpus with general corpus and avoid sacrificing one for the other.
    • Establish an evaluation loop: For knowledge updating, dedicated evaluation sets should be designed, focusing on three types of indicators: numeric knowledge, time-sensitive question answering, and “old knowledge not forgotten”, to quantify the gains and regressions of each iteration.
    • Control iteration cadence: Continued pre-training updates are relatively costly and are suitable for version-based cadence; RAG and knowledge editing can be updated on demand to achieve “hot updates”.
    • Leverage platform capabilities: In enterprise deployment, teams can rely on general capabilities such as large-model service platforms, object storage, vector retrieval, logging, and monitoring/alerting to build a complete pipeline from corpus management, retrieval recall, training evaluation to online monitoring. For example, relevant general capabilities from Baidu AI Cloud can help teams make these stages engineered and observable and reduce self-build and maintenance costs.

    7. Summary

    The completion of pre-training is only the starting point of the LLM lifecycle. For knowledge updating, the industry has formed two main lines: “external RAG” and “internal parameter updating”, with a clear hierarchy of tools including mid-training, supervised fine-tuning, parameter-efficient fine-tuning, and knowledge editing. There is no universal method, only combinations suited to specific scenarios: retrieval for high-frequency changes, continued training for background knowledge, efficient fine-tuning for capability injection, and knowledge editing for individual corrections. Understanding the boundaries and costs of each approach, and establishing data governance and evaluation loops, is key to ensuring that LLM knowledge evolves continuously and reliably.