LLM Training Method Differences: From Alignment to Engineering Practice

1. Clarifying the Premise: An Early Start Does Not Mean a Capability Gap

Discussions about which large language model launched first, or which model is decisively ahead, often conflate product release timing with model capability level. Clarifying this distinction helps shift attention back to the engineering questions that matter: what different training methods solve, what they cost, and how to choose and evaluate them in specific scenarios.

1.1 Early attention comes more from productization and ecosystem than from an insurmountable technical barrier

One conversational LLM product received attention earlier because of three combined strengths:

• Relatively mature engineering capabilities: stability of inference services, concurrency handling, and cost control.
• A relatively complete product form: conversational interface, context management, plugins, and tool calling.
• A user and developer ecosystem: accumulated feedback data can create a positive feedback loop.

These are advantages at the engineering and operations level, not evidence of a single irreproducible algorithm. Academic papers, open-source models, and public technical reports continuously disseminate methods. Late entrants can often progress quickly in data, compute, and engineering optimization. Front-end models are updated frequently, and a lead at one point in time can easily be matched or reversed.

1.2 “Decisive lead” must be limited to specific dimensions

“Conversational and coding capability” can be broken down into at least the following dimensions:

Capability dimension	Typical evaluation points	Main determining factors
Instruction following	Format constraints, multi-condition constraints, avoiding irrelevant elaboration	Supervised fine-tuning data quality, preference optimization
General conversation	Naturalness, helpfulness, harmlessness, multi-turn consistency	Preference data sources, alignment principles
Long-text processing	Information extraction, cross-paragraph reasoning, positional sensitivity	Pretraining corpus mixture, long-context training, positional encoding extension
Code generation	Syntactic correctness, functional correctness, edge-case handling	Code data scale, execution-feedback training
Repository-level code understanding	Cross-file dependencies, impact scope of changes, consistency with existing style	Repository-level corpus, retrieval, tool calling
Reasoning capability	Mathematics, logic, multi-step planning	Reasoning-chain training, reinforcement learning, test-time compute
Agent capability	Multi-step tool calling, task planning, error correction	Tool-use data, execution-feedback loop

On public comparison leaderboards, top models are often separated by only a few percentage points. Statistically, they are frequently in the same tier, and their relative rankings differ across tasks such as code, mathematics, and multi-turn conversation.

1.3 Subjective experience is affected by multiple factors

Differences in subjective experience often come from the following factors:

• Prompt style: whether examples are provided, or whether the model is asked to analyze before answering.
• Feature switches: whether retrieval, code execution, or longer thinking time is enabled.
• Usage habits: after becoming accustomed to one answer style, users may find other styles noticeably less comfortable.
• Version updates: online models are continuously updated, so early impressions may be outdated.
• Task distribution: daily experience often concentrates on a few high-frequency tasks, which means small sample sizes and high noise.

A more meaningful question is therefore not “which model is decisively ahead,” but “under a given task, budget, latency, and compliance requirement, which training technology route is more suitable?”

2. Where LLM Capabilities Come From: A Five-Stage Technical Pipeline

Most mainstream large models today follow a similar capability-building process.

Large-scale corpus
   |  1. Pretraining: learn language patterns and world knowledge
   v
Base model (can continue text, but does not necessarily “follow instructions”)
   |  2. Supervised fine-tuning: use high-quality instruction data to teach the model to answer according to instructions
   v
Instruction model
   |  3. Preference optimization: RLHF calibrates with human preferences; RLAIF calibrates at scale with AI feedback
   v
Aligned model
   |  4. Executable-feedback reinforcement: reinforcement learning for code execution, unit tests, math verification,
   |     and other verifiable tasks
   v
Capability-enhanced model
   |  5. Inference-time strategies: chain-of-thought, retrieval augmentation, tool calling, test-time compute
   v
User-facing model service

2.1 Pretraining: the “floor” of capability

At the pretraining stage, a model predicts the next token in a self-supervised manner over large-scale text, as well as specialized corpora such as code and mathematics. Through this process, the model learns grammar, common sense, facts, and basic reasoning patterns.

• Scaling laws suggest that coordinated growth in parameters, data, and compute can produce predictable capability improvements. However, the improvement is smooth and continuous; it does not create a sudden “cliff” at a single point.
• Corpus composition directly determines the capability distribution. Models with a higher proportion of code or mathematical corpora usually start with an advantage on related tasks.
• Pretraining is extremely expensive and has long retraining cycles. Post-training can significantly change how an existing base model answers, but it is difficult to create knowledge that is completely absent from the base model. This is one reason retrieval augmentation is often used to compensate for knowledge gaps.

2.2 Supervised fine-tuning: from “continuing text” to “following instructions”

Supervised fine-tuning uses “instruction–response” sample pairs for next-token prediction training. Data quality matters far more than quantity.

• A common workflow is: start with a small set of seed instructions, use a model to expand them into a large number of instructions, generate answers, and then filter or rewrite them using humans or models.
• The model expansion and filtering steps are synthetic-data methods. Under the constraint of high-quality seed data, they are widely used and effective in industry.
• Typical problems include rigid formatting, templated expressions, excessive verbosity, and factual errors being “learned” into the model.

2.3 Preference optimization: making answers better match human expectations

Supervised fine-tuning only tells the model what a standard answer looks like. It is difficult to express differences of degree, such as “answer A is better than answer B.” Preference optimization trains the model by comparing the relative quality of different answers.

• RLHF: Human annotators rank multiple answers to the same question. A reward model is trained from these rankings, and reinforcement learning is then used to encourage the model to produce outputs with higher rewards.
• RLAIF: A model generates preference judgments according to an explicit set of principles, such as a constitution-style list of principles. This can replace some human annotation at lower cost and larger scale.
• Lightweight alternatives, such as DPO, can skip an explicit reward model and directly perform contrastive learning on preference pairs, resulting in a simpler engineering pipeline.

2.4 Executable-feedback reinforcement: a key driver of rapid progress in coding

For tasks whose answers can be automatically judged correct or incorrect, such as “whether code passes unit tests,” “whether a math answer is correct,” or “whether the format is valid,” the environment itself is a free and reliable scorer. This makes it possible to conduct large-scale reinforcement learning with programmatic rewards. This is one of the important technical reasons behind the clear progress in coding and mathematics capabilities in recent years.

2.5 Inference-time strategies: improving results without changing parameters

Chain-of-thought prompting, self-consistency voting, retrieval augmentation, tool calling, and strategies that “trade more reasoning computation for higher accuracy” all belong to inference-time optimization. With retrieval and code-execution tools, the same base model can show noticeable changes in coding tasks.

Summary: Early movers often engineered and productized the post-training pipeline earlier. Later entrants may have taken differentiated alignment routes, such as relying more on AI feedback for harmlessness alignment, or introducing executable feedback into reinforcement learning earlier. Specific technical combinations are usually described in public papers and technical reports. This article does not speculate about unproven “secret training methods.”

3. Differences in Alignment Methods: Human Feedback vs. AI Feedback

The “style” differences in conversational experience are largely determined by the feedback data used in the post-training stage. RLHF and RLAIF are two typical routes.

3.1 RLHF: driven by human preferences

# Illustrative code: three-step RLHF flow (pseudocode; engineering details omitted)
# Step 1: Collect human preference pairs where annotators judge which answer is better
preference_pairs = [
    {"prompt": "Explain database indexes in three sentences",
     "chosen":   "...",
     "rejected": "..."},
    # ...
]
# Step 2: Train a reward model using the Bradley-Terry preference model
#         P(chosen preferred over rejected) = sigmoid(r(chosen) - r(rejected))
reward_model = train_reward_model(preference_pairs)
# Step 3: Use reinforcement learning to optimize the policy model
#         Objective = maximize reward - beta * KL divergence from reference model
#         The KL penalty keeps the policy close to the original model and helps prevent reward hacking
policy = ppo_train(base_model, reward_model, kl_coef=beta)

Problems with this route include:

• High cost: Expert-level preference annotation is expensive and slow, making rapid iteration difficult.
• High noise: Differences in annotators’ value judgments can be directly written into the reward model.
• Susceptible to exploitation: The model may learn to produce verbose, flattering, confident but incorrect answers to obtain higher rewards.
• Difficult to scale: The scale of preference data and response speed can become bottlenecks for iteration.

3.2 RLAIF: driven by AI feedback

In RLAIF, a model evaluates and revises answers according to explicit principles, and these AI preference data are then used for optimization. Taking harmlessness as an example:

# Illustrative code: principle-based AI feedback (pseudocode)
PRINCIPLES = [
    "Do not provide instructions that could cause serious physical harm",
    "Do not disparage any group based on race, gender, region, or similar attributes",
    "Answer the question directly and avoid irrelevant moralizing",
]
def critique(prompt, response):
    return llm([
        {"role": "system", "content": "You are a rigorous reviewer. Evaluate the answer only according to the given principles."},
        {"role": "user", "content": f"Principles:\n{PRINCIPLES}\n\n"
                                    f"Question: {prompt}\n\nAnswer: {response}\n\n"
                                    "Point out any violations of the principles and provide specific revision suggestions."},
    ])
def revise(response, comments):
    return llm([
        {"role": "system", "content": "Rewrite the answer according to the revision suggestions while preserving facts and style."},
        {"role": "user", "content": f"Original answer: {response}\n\nRevision suggestions: {comments}"},
    ])
critique_and_revise = lambda p, r: revise(r, critique(p, r))   # critique -> revise

In practice, the two approaches are often combined. For example:

• Rely more on human feedback for helpfulness, ensuring that answers truly meet user needs.
• Rely more on AI feedback for harmlessness, enabling fast and large-scale constraints on dangerous outputs.

3.3 Comparison of the two routes

Comparison dimension	Human-feedback-based (RLHF)	AI-feedback-based (RLAIF)
Main signal source	Human ranking, scoring, written feedback	Model judgments generated according to a principle list
Human dependency	High	Lower; humans mainly define principles and conduct spot checks
Cost	High	Lower
Scaling speed	Limited by annotation capacity	Can be generated in batches; faster iteration
Advantages	Closer to real user preferences	Transparent, auditable, and easy-to-adjust principles
Main risks	Annotation bias, annotator fatigue, reward hacking	Model biases may be amplified; principles may not cover all cases
Suitable stages	General conversation quality, complex preference judgments	Safety boundaries, format specifications, large-scale filtering

It should be emphasized that RLAIF reduces the amount of manual annotation, but it does not eliminate humans. Principle design, sampling review, and pre-launch evaluation still require human involvement.

3.4 Two common side effects

• Sycophancy: When the reward mainly comes from “whether the user likes the answer,” the model may tend to agree with the user’s views rather than point out mistakes. In scenarios such as code review and architecture decisions, this can create substantive risk.
• Over-refusal: When safety alignment is too strong, normal requests containing sensitive words may be incorrectly refused. Proportionality and calibratable safety policies are therefore especially important.

4. The Overlooked Foundation: Data Annotation and Corpus Compliance

4.1 Corpus acquisition and copyright controversy

High-quality corpora are scarce resources. Public web data is gradually being exhausted. Vendors expand their corpora through web crawling, purchasing datasets, and buying second-hand books for scanning. Large-scale copying of copyrighted books for training has triggered lawsuits and regulatory review in multiple countries and regions. The boundary of “fair use” is still being clarified.

4.2 The reality of the annotation supply chain

• Media investigations have reported that some vendors use outsourcing companies to hire large numbers of annotators in developing countries. Wages are often low, and employment is unstable.
• Some reports mention that annotators have fallen into distress due to psychological trauma and related issues, and some have even lost housing.
• Some annotation work involves classifying sensitive content such as crime scenes and graphic images, used to train content moderation models.

This means that the other side of “a safer model” may be that someone else has already been exposed to this content on the user’s behalf.

4.3 How annotation bias propagates to models

Subtle biases introduced during annotation may be amplified in model behavior. For example:

• The language habits of the annotator population may cause the model to show preferences in certain phrasings.
• Whether annotators prefer concise or detailed answers directly shapes the model’s answer length.
• Differences in understanding accents, dialects, and cultural contexts can affect how friendly the model is toward users from minority groups.

Of course, a single signal cannot prove causation. Many of the above observations are hypotheses from media and community discussions and require controlled experiments for verification. Some vendors adjust such style characteristics in later versions.

4.4 Implications for enterprises

• Move data governance forward: Clearly define the sources and licensing scope of training and fine-tuning data. Prioritize proprietary data, authorized data, and open-source data with clear licenses.
• Make annotation specifications auditable: Annotation guidelines should be version-controlled. Use multi-person cross-annotation and measure agreement, such as with a Kappa coefficient. Conduct periodic sampling and quality checks.
• Avoid outsourcing sensitive content to annotation processes lacking support: Content moderation tasks should be handled by professional teams with psychological support mechanisms and compliance safeguards.
• Watch for data risks in outsourcing: When using external services, confirm the model provider’s data usage policies to avoid sensitive business data being used for training.

5. Why Coding Capabilities Improve Rapidly: Executable Feedback and Engineering Loops

5.1 Why code tasks are especially suitable for reinforcement learning

Code tasks have three natural advantages:

• Answers are executable: Whether unit tests pass is an objective signal that requires almost no manual scoring.
• Feedback is immediate: Code can be run locally without waiting for human review.
• Tasks are diverse: They range from completing a single line of code to fixing defects and refactoring modules, with a continuous difficulty gradient.

5.2 Training and inference loops with executable feedback

# Illustrative code: training/evaluation loop with unit-test pass rate as reward signal (pseudocode)
def reward_for(prompt, code, hidden_tests):
    result = run_in_sandbox(code, hidden_tests)   # Run in an isolated environment to prevent malicious code
    if result.timeout:
        return -0.5                              # Timeout penalty
    if result.syntax_error:
        return -0.2                              # Syntax error penalty
    return result.passed / result.total          # Pass rate as reward, range 0 to 1
# Training side: sample multiple candidate solutions per task, estimate advantage from pass-rate differences, update policy
# Inference side: sample multiple candidates for the same task and prefer versions that pass tests and have cleaner code

5.3 From single-file tasks to repository-level tasks

Real programming is not just writing a function. It also includes:

• Understanding cross-file call relationships.
• Following the project’s existing coding conventions.
• Keeping the impact scope of changes controllable.

For this reason, engineering practice often combines retrieval augmentation and tool calling, allowing the model to retrieve relevant files before answering, or to read before modifying. A multi-turn agent loop of “generate -> run tests -> read errors -> fix” is also an effective way to improve success rates.

5.4 Why conversation and coding capabilities are often considered to be on different levels

• Objective evaluation: Code can be automatically verified with compilers, tests, and static checks, producing clear optimization signals. Conversation quality depends more on subjective preferences.
• Sufficient data: Public code repositories are numerous and have natural structural constraints, making it easier to construct training data.
• Clear optimization paths: Error types can be enumerated, such as syntax errors, type errors, boundary errors, and performance issues, making targeted training and repair easier.

6. How to Evaluate Rationally: From Subjective Experience to Reproducible Experiments

Evaluation should follow the principle of “self-built primary, public auxiliary, blind-test validation.”

6.1 A three-layer evaluation system

• Public benchmarks: They have broad coverage, but carry data-contamination risk because test questions may already appear in training corpora. They can only serve as references.
• Self-built task sets: These are closest to the business. It is recommended to extract 100 to 500 representative tasks from real requirements, covering typical scenarios and edge cases, and to update them continuously as the business evolves.
• Human blind comparison: Hide model identities, have business personnel score answers according to preset criteria, and record evaluator agreement.

6.2 Example of automated evaluation for coding tasks

# Illustrative code: pass@k evaluation based on hidden unit tests (pseudocode)
def evaluate_code_model(model, tasks, k=10, temperature=0.8):
    """tasks: [{"prompt": requirement description, "tests": hidden unit tests, ...}, ...]"""
    scores = []
    for task in tasks:
        passed = 0
        for _ in range(k):                       # Sample k candidates for each task
            code = model(task["prompt"], temperature=temperature)
            if run_in_sandbox(code, task["tests"]).all_passed:
                passed += 1
        scores.append(passed / k)
    return sum(scores) / len(scores)             # Approximate pass@k

For conversation and general tasks, pairwise comparison can be used, and win rates and confidence intervals can be calculated:

# Illustrative code: pairwise blind win-rate statistics (pseudocode)
def win_rate(model_a, model_b, tasks, judges):
    wins = ties = 0
    for task in tasks:
        # Randomly swap A/B presentation order to remove position bias
        if random.random() < 0.5:
            answer_a, answer_b = model_a(task), model_b(task)
        else:
            answer_a, answer_b = model_b(task), model_a(task)
        votes = [judge(task, answer_a, answer_b) for judge in judges]
        # Majority vote determines the winner; if votes are close, record as a tie
        if majority(votes) == "A":
            wins += 1
        elif majority(votes) == "tie":
            ties += 1
    return wins / len(tasks), ties / len(tasks)   # Return win rate and tie rate

6.3 Evaluation design considerations

• Control variables: Fix temperature, maximum output length, and system prompts to avoid prompt-template differences masking model differences.
• Use multiple samples: Single-run results are very noisy, so each task should be run multiple times.
• Prevent contamination: Keep self-built questions private and rotate them regularly.
• Record cost: Record per-call cost, average latency, and context consumption. Capability must be considered together with cost.
• Ensure statistical significance: When the sample size is small, differences may not be statistically meaningful, so confidence intervals should be reported.

7. Enterprise Adoption Recommendations

7.1 Use abstraction layers to isolate model differences

Wrap different model services with a unified interface so that business code depends only on the abstract interface. This makes horizontal evaluation and seamless replacement easier.

# Illustrative code: unified model invocation abstraction layer (pseudocode)
class ChatBackend:
    def complete(self, messages, **kwargs): ...     # Chat completion
    def embed(self, texts): ...                     # Embedding
class ModelRouter:
    """Selects a backend based on task complexity, cost, and data sensitivity"""
    def __init__(self, backends):
        self.backends = backends                    # {"light": ..., "strong": ...}
    def complete(self, messages, task="general", **kwargs):
        backend = self.backends[self._pick(task, messages, kwargs)]
        return backend.complete(messages, **kwargs)
    def _pick(self, task, messages, kwargs):
        # Illustrative rules: actual policies should be based on offline evaluation results and online monitoring data
        if task in ("code_review", "architecture", "hard_reasoning"):
            return "strong"                         # Use higher-capability models for complex tasks
        if len(messages) > 50_000:                  # Use a long-context model for very long contexts
            return "long_context"
        return "light"                              # Use a low-cost model for simple tasks

7.2 Task tiering and cost control

• Assign simple tasks, such as classification, summarization, and format conversion, to lightweight models or traditional rule engines.
• Use higher-capability models for complex reasoning and code generation.
• Cache results for high-frequency requests with fixed patterns.
• Use message queues to smooth peak loads for batch offline tasks, processing them during business low periods.
• For tasks with stable output quality and high call volume, use a large model to generate and review high-quality data, then transfer it to a small model through parameter-efficient fine-tuning.

7.3 Engineering combination strategy

Requirement type	Recommended approach	Explanation
Supplementing private knowledge and real-time data	Retrieval augmentation	Updating knowledge only requires updating the knowledge base, without retraining
Fixed output formats and business phrasing	Prompt engineering + structured output constraints	Low cost and fast to deploy
Stable style and domain habits	Parameter-efficient fine-tuning	Relatively low data requirements
Highly specialized domains with scarce public corpora	Continued pretraining	Requires substantial domain corpus and compute
Long and complex workflows	Agent framework + tool calling	Combined with sandbox execution and human review

7.4 Observability and security governance

• Observability: Record request traces and latency distributions, and connect to logging, monitoring, and alerting services. Focus on timeout rate, error rate, per-call cost, and token consumption.
• Output review: Apply static checks and sandbox tests to generated code, and sensitive-content filtering to text output.
• Data isolation: Do not upload unredacted sensitive data. For sensitive scenarios, choose deployment methods that support data isolation and compliance auditing. For example, Baidu AI Cloud offers multiple online invocation and private deployment options, allowing organizations to choose according to compliance requirements.
• Continuous evaluation: Build a regression evaluation set and run it automatically after every model or prompt version change to prevent performance regressions.

8. Conclusion

The “lead” among large models is dynamic, multidimensional, and scenario-dependent. There is no permanent “decisive gap.” Differences in training methods are essentially different trade-offs made by different teams among cost, controllability, scaling speed, and safety boundaries:

• Pretraining determines the “floor” of capability.
• Supervised fine-tuning determines whether the model follows instructions.
• Human feedback and AI feedback determine what style the answers have and what boundaries they follow.
• Executable feedback determines whether the model can continuously improve on verifiable tasks.
• Inference-time engineering determines the model’s upper limit in real business scenarios.

For enterprises, instead of asking “which model is decisively ahead,” it is better to build their own evaluation sets and abstraction layers. Turning model selection into a measurable, reproducible, and switchable engineering decision is the most reliable way to cope with the rapid iteration of large models.