Why Flagship LLMs Are Not Publicly Released: Synthetic Data and Reward Modeling Practices

1. Why Flagship Models Stay Hidden: Three Rational Motives

Large language models are commonly divided into three tiers: a flagship tier for complex reasoning, a mid-tier for general chat and business workloads, and a lightweight tier for edge deployment and high-frequency calls. The industry has repeatedly seen flagship models delayed and ultimately retained for internal use.

This is usually not a marketing failure. It is a rational choice driven by cost structure, and it can generally be grouped into three motivations:

1. Unit inference cost is too high: The parameter scale and activation strategy of the flagship tier can make the compute cost of a single long-context request an order of magnitude higher than that of a mid-tier model. Exposing such a model to high-frequency external calls can quickly become unsustainable at the gross-margin level.

2. Marginal capability gains are insufficient: If the flagship tier mainly improves performance on a small number of hard tasks compared with the previous flagship generation, the public price and compute cost may not be justified by that incremental gain.

3. Its internal value as a “teacher” is greater: The same set of weights can generate training data, construct preference pairs, and train reward models. In those internal roles, its marginal value is often much higher than selling inference services by token.

The key judgment is this: whether a model is publicly released is a resource allocation problem, not a capability problem. Keeping the flagship tier internal and using it as a data factory and evaluator can shift the entire model lineup’s value curve upward.

2. Internal Track 1: Synthetic Data Generation and Distillation

Distillation is not simply copying answers to a student model. Its essence is to align smaller models’ output distributions with the teacher model’s output distribution on the same input distribution.

An effective synthetic-data pipeline usually contains four gates:

• Task distribution sampling: Prompts should be sampled from real business logs and clustered by intent, ensuring that synthetic data covers real long-tail cases rather than only questions that are easy to generate.
• Difficulty stratification: Difficulty labels can be produced using the teacher model’s multi-sample consistency or a lightweight discriminator. Mid- and low-difficulty samples are preferred for distillation, while hard samples are left for human annotation or reinforcement learning.
• Quality control and deduplication: Use rule-based checks, semantic deduplication through vector-similarity clustering, and training-set leakage detection.
• Traceability and compliance: Each synthetic sample should retain its source, teacher version, generation time, and review status for audit and rollback.

A reusable quality-gate skeleton is shown below:

from dataclasses import dataclass, field
import re
@dataclass
class Sample:
    prompt: str
    response: str
    teacher_version: str = ""
    source: str = ""
    difficulty: float = 0.0
    flags: list = field(default_factory=list)
REFUSAL_PAT = re.compile(
    r"^(As an? (?:AI|LLM) model|I(?:'m| am) sorry[,.][^.]*\bcan(?:not|'t)\b)",
    re.M
)
def quality_gate(s: Sample) -> bool:
    """Four gates: empty values, boilerplate refusals, length anomalies, and obvious repetition."""
    if not s.prompt.strip() or not s.response.strip():
        s.flags.append("empty")
        return False
    if len(s.response) < 20 or len(s.response) > 8000:
        s.flags.append("len_outlier")
        return False
    if REFUSAL_PAT.search(s.response):
        s.flags.append("boilerplate_refusal")
        return False
    # Simple repetition check: a very low trigram diversity ratio is treated as degenerate output.
    trigrams = [s.response[i:i+3] for i in range(len(s.response) - 2)]
    if trigrams and len(set(trigrams)) / len(trigrams) < 0.15:
        s.flags.append("degenerate")
        return False
    s.flags.append("ok")
    return True

One particularly important risk is model collapse, also called data self-devouring. If synthetic data is generated from the same teacher for a long time and lacks human fallback, the model distribution can converge generation by generation, while diversity and factuality both decline.

Mitigation measures include injecting real human data at a fixed ratio, mixing samples from multiple teacher versions, and periodically using external benchmarks for distribution-drift detection.

3. Internal Track 2: Preference Data and Reward Modeling

Reinforcement Learning from Human Feedback, or RLHF, usually has three stages: supervised fine-tuning, reward-model training, and policy optimization. When a flagship model is moved to internal use, it replaces the most expensive stage in this pipeline: human preference annotation.

• Preference-pair construction: Sample multiple candidate responses for the same prompt. The flagship model scores and ranks them according to predefined criteria, producing (chosen, rejected) pairs.
• Principle-driven self-critique: Safety and helpfulness criteria can be written as an explicit checklist. The model first critiques a response and then revises it. This constitution-style critique is often more stable and auditable than simple scoring alone.

A prompt skeleton for principle-based self-review is shown below:

You will act as a reviewer. Evaluate and revise each candidate answer according to the following principles:
P1 Do not provide concrete operational steps that can be directly used for illegal activities.
P2 Do not fabricate nonexistent APIs, parameters, or data.
P3 When uncertainty exists, state it explicitly; do not fabricate facts.
P4 Remain neutral; do not promote a specific commercial entity.
Output format:
[Violated principle] P? / None
[Problem location] <original excerpt>
[Revised answer] <full rewrite>

When training the reward model, the following practical points are often more important than choosing a loss function:

1. Prioritize annotation consistency: Use blind cross-scoring to estimate annotator agreement. Dimensions with low consistency should not be fed into the reward model, otherwise noise becomes fixed into the policy.

2. Prevent reward hacking: Reward models are highly sensitive to length, format, and polite filler. Length normalization should be introduced into the features, and some anti-style samples should be retained to balance the reward signal.

3. Use an offline preference loss as a baseline:

import torch
import torch.nn.functional as F
def pairwise_loss(rw, x_win, x_lose, margin: float = 0.0):
    """Bradley-Terry-style preference loss; margin > 0 strengthens strong preference pairs."""
    diff = rw(x_win) - rw(x_lose) - margin
    return -F.logsigmoid(diff).mean()

1. Add a KL constraint: During policy optimization, the policy must remain constrained by KL distance from the reference model. Otherwise, reward scores may rise while general capabilities degrade significantly.

4. Folding Long-Context and Code Review Capabilities into Mid-Tier Models

One important value of flagship models lies in long-text understanding and code review. To transfer this capability to mid-tier models, it is necessary to distinguish what can be distilled from what should be engineered.

• Can be distilled: intent recognition, summary style, selection of review perspective, and tendencies in defect-type judgment.
• Should be engineered: precise cross-section reference localization, line-level issue localization, and dependency analysis. These tasks rely on deterministic parsers such as syntax trees, indexes, and rules. They should not depend on model memorization.

A practical approach for long documents is hierarchical retrieval-based processing:

1) Chunk the document by semantic or heading boundaries, preserving the global table of contents and page anchors.
2) Generate local summaries and key entity or numeric cards for each chunk.
3) Aggregate across chunks to produce intermediate conclusions with anchors.
4) For a specific question, perform localization-based verification by returning to the original snippets for secondary confirmation.

This “chunk, aggregate, return-to-anchor verification” structure allows mid-tier models to reliably reproduce most long-text understanding tasks while localizing hallucinations to checkable positions in the original text.

5. Real Boundaries of Multimodal Capabilities and Engineering Responses

Image-understanding models can already reliably handle several common tasks: subject and scene recognition, layout region division, and reading trends and relative relationships in charts.

However, well-known capability boundaries remain:

• Precise numeric extraction from dense small text or low-resolution receipts.
• Precise reading and coordinate conversion when multiple chart series overlap.
• Comparative questions that require multi-hop visual reasoning, such as “the maximum value in the second row of the left image minus the average value in the right image.”

For this reason, a vision model should not be treated as the sole extractor. A division-of-labor closed loop is recommended: the vision model handles “what it is” and “what the trend is,” while a specialized layout and text parser handles exact values. The two results should then be cross-checked:

def reconcile(vlm_answer: dict, parser_answer: dict, tol: float = 0.02):
    """When visual readings and parser results disagree, downgrade to human review instead of using the result directly."""
    issues = []
    for key, v_parse in parser_answer.items():
        v_vlm = vlm_answer.get(key)
        if v_vlm is None:
            issues.append(f"{key}: missing visual value")
        elif isinstance(v_parse, (int, float)) and abs(v_vlm - v_parse) > tol * max(1, abs(v_parse)):
            issues.append(f"{key}: inconsistent reading {v_vlm} vs {v_parse}")
    return {
        "status": "ok" if not issues else "need_human_review",
        "issues": issues
    }

The conclusion must be explicit: any number that enters accounting, reports, or key contract terms must have a verifiable source. It cannot rely only on model output.

6. Reference Architecture: Put the Teacher Model in the Data Flywheel

A deployable layered architecture can be separated by responsibility:

Layer	Responsibility	Common General-Purpose Capability Categories
Collection	Business logs, conversations, and document ingestion	Object storage, logging services
Orchestration	Synthetic-task scheduling, retries, and rate limiting	Message queues, task scheduling, container platforms
Generation	Batch inference with the internal-only teacher model	GPU container clusters, elastic scaling
Annotation	Preference-pair generation, principle self-review, and human review	Data warehouses, annotation platforms
Training	Distillation, reward modeling, and policy optimization	Distributed training frameworks, model registries
Evaluation	Layered evaluation sets, regression baselines, and blind testing	Monitoring, alerting, and metric dashboards
Serving	Mid-tier models exposed to external capabilities	Gateways, rate limiting, and quotas

On Baidu AI Cloud, a common implementation maps these responsibilities to general-purpose cloud capabilities such as object storage for raw corpora and dataset versions, message queues for peak-shaving asynchronous synthesis tasks, container platforms for elastic scheduling of inference and training workloads, logging services for generation tracing and teacher-version retention, and monitoring and alerting for cost, failure rate, and data-quality drift. This enables end-to-end construction without depending on a specific proprietary interface.

Two engineering pitfalls are worth calling out:

1. Uncontrolled teacher-inference cost: Batching, result caching, and sampling-rate control are essential.
2. Version chaos: Data versions, teacher versions, reward-model versions, and policy versions must be alignable as a four-tuple. Otherwise, production issues become nearly impossible to attribute.

7. Evaluation and Acceptance: Do Not Use a Single Yardstick

• Use layered evaluation sets: Split tasks into L1, L2, and L3 by difficulty and business value. The gains of the flagship tier often appear only at L3. Using it to evaluate a mid-tier model can lead to the wrong conclusion.
• Prevent leakage: Evaluation sets should be constructed by humans or split by time using future data. Contamination detection should also be included to prevent synthetic data from indirectly leaking answers.
• Use a combined metric set: Task success rate, format compliance rate, factual error rate, average token cost, P95 latency, and refusal rate. An abnormally high refusal rate often signals over-aggressive safety training.
• Prevent regression: Maintain a golden set and run a fixed regression before every release. If any core metric drops beyond a threshold, release should be blocked.

8. Risk Checklist and Release Decision

Risk	Symptom	Mitigation
Distribution collapse	Homogeneous answers and declining diversity	Inject real data and mix samples across versions
Reward hacking	Verbose politeness and formulaic formatting	Length normalization and style-balancing samples
Data contamination	Inflated offline scores and poor online performance	Contamination detection and time-split evaluation sets
Compliance risk	Unknown source, missing anonymization, or inability to audit	End-to-end tracing, anonymization, and retention management
Cost overrun	Teacher-inference cost exceeds budget	Caching, sampling, and pre-distillation filtering

For the release decision, a simplified decision matrix can be used: if external demand size × unit price is clearly lower than the overall model gain generated by teacher use, internal retention is usually the better choice.

Conversely, if capabilities such as long-context understanding or code review have independent commercial value and a clear quota plan, controlled release can be considered. A reasonable position is to preserve the possibility of future release rather than permanently closing the door.

9. Summary

Moving a flagship model to internal use is essentially replacing “selling inference” with “building capability.” For a research and engineering team, the real takeaways are three things:

1. Use a cost-benefit matrix, rather than a leaderboard ranking, to decide how a model should be distributed.
2. Master the complete gate design for synthetic data and preference modeling, especially anti-collapse and anti-leakage controls.
3. Clarify the boundaries of vision and long-text capabilities, and use deterministic parsing and consistency checks to hold reliability in place.

Building this data flywheel into an observable, auditable, and rollback-capable system is more valuable in the long run than chasing the leaderboard position of any single model.