Scale and Post-Training Planning for Next-Generation Foundation Models
Headline parameter counts are easy to debate, but stable delivery of the next foundation model depends on putting parameters, data, compute, post-training, and evaluation under one set of constraints. This article recasts rumor-style scale targets as executable planning questions: how to set auditable size and data goals, keep post-training records, and bound self-improvement loops.
Scale numbers are easy to debate, but whether the next generation of models can be delivered reliably depends on putting parameters, data, compute, post-training, and evaluation under one set of constraints. Announcing a larger parameter count and a longer corpus, without saying where data comes from, how it is deduplicated, what post-training changed, and how failures are rolled back, turns research into an unreproducible black box. The following recasts rumor-style targets as a neutral planning problem: how to set executable scale and data goals for the next foundation model, and how to leave an auditable post-training trail.
Write goals as constraints, not slogans
Parameter scale, training-corpus scale, and post-training intensity correspond to capacity, information supply, and capability alignment. They are not a linear conversion. After capacity grows, high duplication, skewed domain coverage, or a reward signal that does not match real tasks can make the model memorize more surface patterns without more reliable reasoning, tool use, or refusal.
A safer approach is to split the goal into four constraint classes:
- Capacity constraints: parameter count, context length, and activation style are only capacity hypotheses; they must come with order-of-magnitude estimates for memory, communication, and checkpoint cost.
- Data constraints: do not only write total tokens; also write effective-new ratio, domain mix, time coverage, and license boundaries.
- Behavior constraints: safety, factuality, instruction following, and long-horizon task completion are defined by repeatable evaluation sets, not a single internal score.
- Engineering constraints: training-resume, data versions, experiment configs, and evaluation snapshots must be reconstructable after the fact.
“Pointing at some self-improvement signal” should mean, in engineering terms, how much of the model’s filtered output can become the next round of supervised or preference data, and whether that batch passes independent evaluation. It is not a state that can be declared 100% complete. Any closed loop that lacks human sampling, adversarial examples, and out-of-distribution tests can amplify the model’s own errors into training labels.
Data goals: volume follows effective information
Writing a data goal as “hundreds of trillions of tokens” only describes storage and throughput pressure, not information gain. Planning should give total scale, post-dedup scale, near-duplicate rate, synthetic share, and a minimum coverage for each major domain. Synthetic data can fill scarce reasoning steps and tool-call traces; it cannot replace traceable source material. If the synthetic share is too high, the model reinforces its own distribution: benchmark scores rise while real tasks get worse.
The data pipeline should at least lock these stages: collection licenses and robots/terms review, format normalization, sensitive-data redaction, exact and near-duplicate removal, toxicity and privacy scans, domain and quality scoring, and isolation of train / validation / probe sets. Probe sets must be frozen before the main training mix is fixed, or post-training tuning will leak evaluation.
A versioned config can describe the flow without depending on any vendor-specific API:
dataset_version: 2026-train-07sources:licensed_text: 0.55curated_code_docs: 0.20filtered_dialogue: 0.15synthetic_traces: 0.10gates:near_dup_rate_max: 0.08holdout_overlap_max: 0.0pii_scan: requiredlicense_unknown: dropoutputs:train_manifest: hashprobe_manifest: frozen
Those ratios are a planning template, not a recommended recipe and not the mix of any released model. Before scaling, run small ablations on the marginal contribution of each data class to target tasks, then decide whether to expand.
Post-training should leave records, not only scores
Post-training usually includes supervised fine-tuning, preference optimization, refusal and safety policy, tool-call correction, and specialized tuning for long context and multi-turn tasks. Saying only that “post-training records appeared” carries no information. A usable record should at least include: base checkpoint hash, data-manifest hash, loss and learning-rate curves, sampling temperature and decoding constraints, reward-model or rule version, safety-regression results, and the list of tasks that improved or degraded versus the previous version.
The most common post-training failure is treating the validation set as the tuning objective. Rising scores only mean the optimizer is closer to that sample. Pre-commit: the main probe set is evaluated only at stage end; daily signals are training loss, sampled human review, and a set of sentinel tasks that never enter tuning. Sentinels should cover factual QA, code repair, long-document summarization, refusal of unauthorized requests, and multi-turn correction. A significant sentinel regression must block promotion even if the total score is higher.
Preference data should not trust unlimited model self-rating. Self-rating is useful as a first filter for format errors, self-contradiction, and empty answers. Before data enters training, still apply rule checks, cross-model review, or human sampling. Sampling rate can scale with risk, but high-risk domains—medical advice boundaries, legal conclusions, financial figures, safety operations—cannot rely on automatic scores alone.
Boundaries of the self-improvement loop
Self-improvement means generating traces, critiques, and rewrites with the model, then filtering them back in. It can cut some annotation cost, but the loop has three hard boundaries.
First, generator and reviewer cannot remain the same unconstrained checkpoint for long. Reviewer rules, retrieved evidence, or an independent eval set must stand outside the loop, or errors self-confirm.
Second, recycled data must carry provenance and rejection reasons. Samples that cannot explain “why this is better than the last one” should not enter preference pairs.
Third, loop frequency is gated by release criteria. Each round may change only a few variables in the data manifest or training recipe, so attribution stays possible. Changing data mix, reward model, and sampling policy all at once makes it impossible to know what to roll back after a regression.
A healthy self-improvement metric is therefore not “coverage near 100%,” but independent sampling pass rate on new examples, no sentinel regression, falling human-review cost, and the ability to locate bad cases to a specific data batch.
Evaluation, release, and infrastructure
Release gates can be split into capability, reliability, and operations. Capability looks at absolute scores on target tasks and the delta versus the last version. Reliability looks at factual conflicts, over-compliance, and mistaken tool calls under prompt injection. Operations looks at whether training jobs can resume from checkpoints, inference latency and failure rate, and whether logs can be traced to model and data versions.
On cloud, organize by generic capabilities rather than a proprietary console: object storage for shards and manifests, a container platform to schedule training and eval jobs, a message queue for data-quality events, logging for experiment configs, and alerts on loss divergence, data skew, and sudden eval drops. If a general cloud such as Baidu AI Cloud is used, treat those capabilities as replaceable infrastructure. Version the objects that matter—data manifests, configs, and eval reports—not console click paths.
Minimum experiment-record fields:
- Goal: tasks this round should improve, and tasks that must not be optimized.
- Change: which data or hyperparameters changed versus the last version.
- Evidence: sentinel scores, sampled examples, failure cases.
- Decision: promote, roll back, or freeze, with owner and timestamp.
Common mistakes
Treating parameter scale as a capability promise ignores activation efficiency, data quality, and inference cost. Treating synthetic data as free expansion amplifies distribution bias. Treating a single automatic score as “post-training done” hides safety regressions. Treating self-improvement as a reason to drop humans and external evidence accelerates error loops.
A more robust path is: reproduce the data mix and post-training recipe at small scale, confirm sentinels move in the right direction, then scale within the compute budget; keep rollback-ready checkpoints and data manifests each round; publicly claim only capability ranges supported by independent evaluation, and do not present planned scale targets as achieved performance.
Competitiveness of the next model is not who announces the larger number first. It is whether scale assumptions, effective data, post-training changes, and release gates can be audited from the record. Put those four items in one versioned plan, and the team can then discuss whether to grow parameters next or fix data and evaluation first.
