Flagship Large Language Models Are Often Not Released Publicly: Engineering Practices for Synthetic Data and Reward Modeling

1. The Starting Point: Why the Most Capable Model Is Often Not Released Publicly

In the productization of large models, a recurring pattern appears: a vendor plans a flagship-scale model, announces a public timeline, but ultimately does not list the model as available to end users. The reason is usually not technical failure. More often, three factors change at the same time.

Inference-cost structure determines the economic ceiling for a single call. The parameter count and activation scale of a flagship model determine the cost of a single forward pass, and this cost grows approximately linearly with output length. When the use case involves text-intensive tasks such as long-document understanding or cross-file code review, where inputs can easily reach hundreds of thousands of tokens, the compute cost per request is further amplified by input length. Offering such capabilities externally leaves two difficult options: price them at a level users cannot accept, or subsidize usage at a loss to gain market share. Neither is sustainable.

The marginal return of capability begins to diminish. When upgrading from a small or medium model to a flagship model, score improvements on overall capability benchmarks are often concentrated in a small number of difficult task types. However, most online requests are medium-difficulty tasks. The extra benefit delivered by the flagship model, when spread across the full request distribution, becomes quite limited. The difference users perceive is often much smaller than the gap shown on leaderboards.

The same compute creates more value when kept internal. If a flagship model serves only external requests, its value can be approximated as “quality improvement per call × request volume.” If it is converted into an internal role, it can become a generator of training data and a judge of output quality. Its value then becomes “quality ceiling for every downstream sample × training set size.” The latter is not constrained by online traffic. It can be run offline in batches, and its output is reusable and can continuously raise the quality of the entire product line.

Therefore, “keep it internal and do not release it publicly” is not merely a temporary patch to a product decision. It is a resource-allocation choice. At the same time, official statements usually do not completely rule out future public availability. If inference costs fall, or if the model is required for an irreplaceable public-facing scenario, the conclusion may reverse.

2. Three Typical Roles for an Internal Flagship Model

When a flagship model is used as an “internal teacher,” common industry usage can be summarized in three categories. Their input-output contracts are different, and their engineering implementations should not be mixed together.

Role	Input	Output	Key constraints
Data generator	Task instructions, context material, difficulty labels	High-quality candidate samples: answers, reasoning processes, before/after rewrite pairs	Diversity, deduplication, decontamination
Quality judge	One or more candidate answers, scoring rubric	Scores, preference rankings, critiques	Scoring stability, bias resistance
Self-improver	Flawed output, principle set	Revised output, defect explanation	Revision must be verifiable and must not introduce new errors

It is important to emphasize that the teacher model does not “teach” knowledge to the student model. It teaches the shape of the output distribution. What the student model learns is format, tone, organization, common error-avoidance patterns, and what kind of answer is likely to be considered better. This also means that factual errors made by the teacher model can be distilled into the student model. Therefore, data-side validation cannot be omitted.

3. A Synthetic Data Production Line: From Task Sampling to Dataset Delivery

A reproducible synthetic-data pipeline should contain at least six stages. Each stage should have an independent output and acceptance criteria.

1. Task sampling and difficulty stratification. Do not sample uniformly. In practice, it is more valuable to stratify by capability dimension: factual question answering, multi-step reasoning, code completion and review, structured information extraction, long-document summarization, refusal and safety boundaries. Set a target sample size for each layer, so that a large volume of easy tasks—especially chitchat, which is easy to generate—does not drown out genuinely scarce difficult samples.

2. Seed and context preparation. The biggest risk in synthetic data is “questions imagined by the model.” For answer-type tasks, use verifiable seeds: real code snippets paired with unit tests, real tables paired with numerical validation, real document passages paired with traceable source locations. Open-ended tasks without seeds are suitable only for style alignment, not for capability alignment.

3. Decouple teacher generation from diversity. When repeatedly sampling the same task, explicitly perturb prompt templates, output formats, and role perspectives. Otherwise, generated results will be highly homogeneous, and the number of valid samples after deduplication will drop sharply.

for task in sampled_tasks:                     # tasks are stratified by capability dimension
    for k in range(samples_per_task):
        prompt = render(task.instruction,
                        template=random.choice(TEMPLATES),
                        style=random.choice(STYLES),
                        seed=task.seed_context)      # bind to verifiable source material
        raw = teacher.generate(prompt,
                               temperature=0.7,
                               max_output_tokens=task.budget)
        record = validate(task, prompt, raw)         # rule / execution / numerical validation
        if record.passed:
            store(record.with_tags(task.layer, k))

4. Multi-layer quality filtering. Filtering should be layered, with cheaper checks placed earlier. First use a rule layer to remove obvious noise: abnormal length, repeated fragments, invalid format, placeholder text. Then use an execution layer: run unit tests for code, run validation queries for SQL, compare numerical results against reference answers. Finally, use a model layer: have the teacher model or an independent review model score semantic quality according to a rubric. Placing the most expensive model judgment last is key to cost control.

5. Decontamination and memorization checks. If seed material comes from public benchmarks, approximate-match detection must be performed to prevent evaluation-set content from being imported into training data for the student model, which would inflate later evaluation results. This step should be enforced before dataset delivery, not traced back only after evaluation anomalies appear.

6. Delivery format and version management. It is recommended to deliver structured samples with metadata. The metadata is responsible for traceability:

{"id":"sft-000871","task_type":"code_review","difficulty":"hard",
 "context_ref":"repo://snippet#1421","instruction":"...",
 "response":"...","critique":null,"revised_from":null,
 "teacher_model_version":"internal-teacher-2024-11",
 "filter":{"rule":true,"execution":true,"judge":0.86},
 "dedup_cluster":"c-3391"}

4. Reward Modeling: Using the Teacher Model for Part of Human Feedback

The classic alignment process for reinforcement learning is “sample — score — fit a reward model — optimize the policy.” Human annotation usually accounts for the largest share of effort and the longest cycle time. Therefore, replacing part of human feedback with model-based scoring is one of the most valuable uses of an internal flagship model.

Pairwise comparison is usually better than absolute scoring. Asking the model “which answer is better, A or B?” generally produces better consistency and trainability than asking it to assign an absolute score from 0 to 10. Absolute scores drift easily: with the same rubric, a sample may receive a 7 one day and an 8 the next day, which directly pollutes the regression targets of the reward model.

The scoring rubric must be fixed and discriminative. An effective review rubric should contain observable criteria rather than subjective phrases such as “whether the answer is clear.”

rubric = [
  "Fact consistency: Are all verifiable statements supported by at least one context snippet (yes/no)?",
  "Completeness: Are all N required points covered? List missing items (0~N).",
  "Executability: For code answers, can the code run directly? List the first failing assertion (yes/no).",
  "Conciseness: Is there task-irrelevant restatement or disclaimer (yes/no)?",
  "Refusal appropriateness: Does the answer provide substantive content where it should refuse (yes/no)?"
]

Once the criteria and output structure of each item are fixed, scoring becomes reproducible. In addition, critique text should be retained, not only scores. Critique text can serve as input for rejection sampling, self-improvement, and human review, and it contains much higher information density than a single scalar.

Reward model fitting and common pitfalls. Training data is primarily preference pairs. In form, this is a ranking objective: for two answers to the same prompt, the model outputs scalar scores, and the loss penalizes cases where the preferred answer receives a lower score than the non-preferred answer. The risks in this step are concentrated in four areas:

• Position bias: If swapping A and B flips the conclusion, the model is judging order rather than quality. A common mitigation is to evaluate both orders and average the results.
• Length bias: The model may systematically prefer longer answers, pushing policy optimization toward verbosity. Mitigations include explicitly setting a conciseness criterion in the rubric or performing length normalization in the loss.
• Self-preference: The teacher model may score content in its own style more highly. Mitigations include using different configurations for the judge and the generator, or introducing a third-party review model for cross-validation.
• Reward hacking: The policy model learns to generate outputs that “look compliant with the rubric but are incorrect.” A common mitigation is to retain a verifier that cannot be optimized by the policy—such as unit tests, numerical validation, or fact retrieval—as a hard constraint, while the reward model handles only soft preferences.

5. Principle-Driven Safety Alignment: Turning Rules into Trainable Signals

There is a low-cost route to safety alignment: instead of relying on large-scale human annotation, write safety requirements as an explicit set of principles, have the model self-critique and revise its own outputs, and then use “before/after revision pairs” as training data. The engineering focus lies in the principle set itself.

Principles should satisfy three conditions: decidable (it must be possible to answer clearly whether a violation occurred), layered (covering categories such as harmlessness, honesty, harmful assistance, and privacy), and versionable (each revision should leave a trace, making it easier to trace which principle change introduced a behavioral regression). In practice, a common problem is that principles are written too abstractly, so the model cannot locate specific defects. As a result, “revision” degrades into rewriting the same content in different words, and the safety benefit is close to zero.

A corresponding tiered response policy is also needed. For requests in high-risk categories, the correct alignment behavior is to refuse or downgrade to a safe alternative, rather than force an answer. Such samples must maintain a sufficient proportion in the training set. Otherwise, after general capability alignment, the model will tend to “answer whenever possible.”

6. Capability Boundaries: Long Context, Code Review, and Multimodal Performance in Practice

The gap between advertised metrics and engineering experience is especially visible in three capability areas. Countermeasures should be designed in advance.

Long context: capacity and effective utilization are different things. Supporting an ultra-long context means information can enter the model, but it does not mean the model can use information from all positions equally. The empirical rule is that recall is noticeably higher for information at the beginning and end of the context, while information in the middle is more easily ignored. Engineering countermeasures include: placing key constraints at both the beginning and the end (a front declaration plus a closing restatement), performing segment-level summarization before global reasoning over very long documents, and using retrieval-based assembly instead of stuffing everything into the context when precise citation is required.

Code review: strong at pattern recognition, weak at cross-file reasoning. The model is relatively stable at identifying local issues such as null-pointer risk, swallowed exceptions, inconsistent naming, and missing boundary conditions. However, for defects involving cross-file call chains, concurrency timing, and implicit global state—cases that require global context—the miss rate rises significantly. A reasonable positioning is “first-layer screening plus an explicit checklist,” treating results as prompts for human confirmation rather than as an automatically mergeable gate.

Multimodal: can recognize objects and trends, but struggles with precise extraction. The model is usually correct about subjects, layouts, and upward or downward trends in charts. However, error rates are high and unpredictable for precise numerical values in dense tables, small-font text, and fields aligned across pages in multi-page documents. Therefore, workflows involving numerical values and fields must use a combination of structured extraction, rule validation, and, when necessary, a dedicated extraction component as a fallback. Precise numerical values should not be handed directly to a general-purpose understanding model.

7. Platform Deployment: Data, Compute, and Observability

Offline synthesis and offline review are typically high-throughput, delay-tolerant workloads. Their resource profile is completely different from online inference. It is recommended to split them into two independent pipelines when deploying.

• Object storage holds raw material, intermediate candidates, and final datasets. Data should be partitioned by batch and retain lineage information, so that any training sample can be traced back to its source material and filtering records.
• Message queues handle peak shaving and retries for generation tasks. Task failures and result callbacks should be decoupled, avoiding mutual blocking among long-running batch jobs.
• Container platforms host heterogeneous jobs. Generation, execution validation, and model review have very different compute requirements. Once containerized, they can be scaled separately by type, preventing small jobs from occupying high-specification resources.
• Model serving and inference acceleration are used for batch calls to the teacher model and review models. Throughput optimization for batch inference—such as continuous batching and request aggregation—usually lowers the cost per sample more effectively than model-side optimization alone.
• Logging and monitoring alerts must cover data-quality metrics, not only system metrics. These include candidate pass rate, attrition rate at each filtering layer, score distribution drift, deduplication cluster size, and cost per valid sample. A sudden drop in pass rate often appears before a regression in model capability and is the earliest warning signal.

When deploying on Baidu AI Cloud, the same general combination—object storage, message queues, container platforms, and logging/monitoring—can be used. Offline data pipelines and online services should be managed separately according to resource profiles, making batch-level cost accounting and elastic scaling easier.

8. Evaluation: How to Prove That Distillation Does Not Lose Capability

The biggest risk of the internal-teacher route is “metrics go up, experience goes down.” Evaluation must therefore be layered, and a fixed set that does not change with training should be retained.

1. Offline capability evaluation by dimension. Organize independent evaluation sets according to the capability stratification described in Section 3, strictly isolated from training data. Track the student model’s retention rate relative to the teacher model for each dimension.
2. Hard-constraint pass rate. Unit-test pass rate, numerical-validation pass rate, and format-validity rate. These metrics cannot be “gamed” by optimizing writing style, so they are highly credible.
3. Human blind sampling. Sample from the online distribution, hide the model source, and perform double-blind comparisons. Focus on differences in failure modes rather than average scores.
4. Small-traffic online experiments. Use task success rate, human takeover rate, user retry rate, and cost per request as primary metrics. Start with a small traffic share before scaling up.
5. Regression alerts. After any change to the data recipe or training configuration, automatically run the fixed evaluation set. If a metric drops beyond a threshold, block release.

9. Common Anti-Patterns

• Using the flagship model to generate everything: This leads to homogeneous data, uncontrolled cost, and amplification of the model’s own biases. Generation resources should be allocated according to task difficulty.
• Keeping only scores and not critiques: This loses auditability. Humans cannot locate problems, and improvements to the review rubric cannot accumulate.
• No seed binding: Synthetic data from open-ended generation may look abundant, but after training the model may learn to “confidently fabricate,” and factuality metrics may actually decline.
• Using evaluation-set material as seeds: Inflated evaluation scores are quickly corrected by real traffic, and contamination is difficult to remove completely.
• Using exactly the same prompt style for teacher and student: The student learns only a single template, limiting generalization.
• Ignoring safety-sample proportions: After capability alignment, the model tends to answer everything, and the safety boundary moves backward.

10. Conclusion

Whether a flagship model is released publicly is essentially a compute-allocation problem rather than a technical problem. When its inference cost exceeds external revenue, when its capability improvement has not crossed the cost-performance inflection point, and when it can continuously raise the data-quality ceiling of the entire product line internally, keeping it internal is the more rational choice.

For engineering teams, the truly reusable gains are three things: a synthetic-data pipeline with seed binding and layered filtering; a model-review and reward-modeling method based on pairwise comparison and explicit rubrics; and a set of layered evaluation gates covering offline evaluation, hard constraints, and online experiments. If these three things are implemented solidly, even without relying on any single strongest model, it is possible to steadily improve the real-world performance and unit-cost efficiency of small and medium models.