LLM Reinforcement Learning Post-Training: Cost Transparency and Version Governance
As pre-training returns diminish, reinforcement learning (RL) has become the main arena for open-weight large language model (LLM) competition. Using a real-world case study, this article breaks down the four core engineering challenges of large-scale RL post-training: driving RL with verifiable tasks, monitoring the training process in real time while measuring cost accurately, delivering high-performance inference, and completing smooth model version transitions. It covers the RL training loop and its key quantitative patterns, verifiable reward RL (RLVR), why the generation side is the real bottleneck, cost modeling and total cost of ownership (TCO), real-time training dashboards, inference acceleration techniques, and model version governance. Readers will gain practical guidance on budgeting RL runs, designing reward functions, building observability, and managing version migration to keep production systems stable.
1. Introduction: Post-training becomes the main battlefield for open-model competition
Since 2026, the marginal returns of LLM pre-training have continued to shrink, and the industry has broadly shifted its focus on improving complex-task capability toward post-training. Supervised fine-tuning answers “can the model produce output in the expected format,” while reinforcement learning (RL) answers “can the model grow stronger on its own through open-ended exploration.” For open-weight models, this shift is especially pronounced: when the baseline capability gap between vendors narrows to a few percentage points, whoever can complete a high-quality round of RL training at lower cost, in a shorter cycle, and through a more transparent process will iterate faster.
A representative case is a four-step approach taken by a model team:
- Focus: After open-sourcing the previous generation under a permissive license in April, the team released no new model for nearly six months, concentrating instead on pushing the boundaries of RL capability.
- Publicize: On September 17, the team lead first disclosed progress on RL training for the next generation on social media and launched a real-time training page covering two product lines: a flagship omni-modal version and an efficient inference version. By that afternoon, the combined cost of both versions had already exceeded $1.35 million, roughly $31,000 per hour.
- Open-source: On September 22, the team released and open-sourced the new generation, including a flagship omni-modal model and a reasoning model; it also launched an “ultra-fast” mode with output speed up to 20× the base model, along with a stable release of the desktop client. The series was subsequently connected to a third-party composite intelligence index and made available on the official open platform. According to the team, this round of RL training finished in under six days, with 30 steps per version and roughly 750,000 trajectories in total, at costs of approximately $2.62 million and $850,000 respectively.
- Iterate: On October 9, the team announced the retirement of the previous generation, and on October 14, traffic was automatically switched to the new version.
This timeline captures the four core engineering problems of current LLM post-training:
- How to drive large-scale RL with verifiable tasks;
- How to observe the training process in real time and measure cost accurately;
- How to deliver high-performance inference capability;
- How to complete model version transitions smoothly.
This article takes each of them apart in turn and offers reusable practical recommendations.
2. Technical foundations: How RL post-training “scales”
2.1 The basic loop and its core data
At its core, RL post-training is policy optimization. The model samples multiple responses to the same prompt (i.e., trajectories) and adjusts its output tendencies according to how high the rewards are. A typical loop is:
- Sample: Draw tasks from the task pool and have the model generate multiple complete responses;
- Score: Evaluate the quality of each response with a reward function;
- Estimate advantages: Determine how much “better” a response is relative to others in the same group;
- Update: Update the model parameters based on the advantages, and sync them to the inference engine.
Breaking down this round’s training data yields several key quantitative characteristics:
- Trajectories per step: 60 steps correspond to roughly 750,000 trajectories, or about 12,500 per step;
- Time per step: about six days correspond to 30 steps, averaging under five hours per step;
- Cost per trajectory: $3.47 million ÷ 750,000 ≈ $4.6.
Ten-thousand-scale trajectories per step strongly suggest the algorithm adopted group-relative advantage estimation: sampling multiple responses to the same task and using the relative level of in-group rewards as the learning signal. This design substantially reduces gradient variance, but it also means the generation side carries the bulk of the compute.
2.2 Verifiable rewards: the prerequisite for scaling
The prerequisite for RL to scale to “ten-thousand-scale trajectories per step” is that the reward signal can be produced quickly and objectively by machines. Verifiable Reward Reinforcement Learning (RLVR) is a representative embodiment of this idea. It hands “scoring” over to rules and sandbox execution environments: code tasks run unit tests, math tasks compare against the final answer, and tool-calling tasks validate the call chain and return format. Compared with approaches that rely on a reward model, it has three advantages:
- It introduces no additional model preference bias;
- Scoring can be executed in parallel and naturally fits large-scale sampling;
- The scoring criteria are clear, making it easy to locate training anomalies.
But RLVR also has clear shortcomings:
- It applies only to tasks whose results can be judged;
- Rules that are too loose invite reward hacking, such as hard-coding answers to test cases; rules that are too strict lead to sparse rewards, so the model receives positive feedback only rarely over long stretches.
Common countermeasures in practice fall into three categories:
- Layered rewards: check output format, correctness, efficiency, and readability in sequence;
- Difficulty-stratified sampling: when tasks are too easy or too hard, in-group rewards are nearly identical and the advantage is zero, so tasks with a moderate pass rate should be sampled preferentially;
- Holdout monitoring: track true capability with a holdout set that does not participate in reward computation, preventing the model from “gaming rewards” instead of “gaining capability.”
def rl_post_training_step(model, task_pool, group_size=8):# 1. Difficulty-stratified sampling: prioritize tasks with a moderate recent pass rateprompts = task_pool.sample_by_pass_rate(low=0.2, high=0.8)# 2. Parallel generation: sample group_size complete trajectories per tasktrajectories = rollout_engine.generate(model, prompts, n=group_size)# 3. Verifiable reward: format score + sandbox execution score (unit test pass rate)rewards = verifier.score(trajectories)# 4. Group normalization to compute advantages; drop all-correct or all-wrong groups to avoid zero-advantage idle stepsadvantages = group_normalize(rewards, group_size)# 5. Policy updateloss = policy_loss(model, trajectories, advantages)optimizer.step()# 6. Weight sync: push the latest parameters to the generation enginerollout_engine.update_weights(model)
2.3 Engineering challenges: the generation side is the bottleneck
In RL post-training, the training-side computation graph is relatively fixed, while the generation side faces three kinds of pressure:
- Long-tail lengths: reasoning tasks often produce outputs of thousands of tokens, and a very small number of extremely long samples slow down the entire batch;
- Frequent weight changes: after each update step, the new weights must be synced to the inference engine, and the cost of parameter transfer and swapping memory in and out is considerable;
- Cluster heterogeneity: the training side pursues high compute density and large memory, while the generation side pursues high concurrency and low latency, so the two have different hardware demands.
Synchronous training requires that all trajectories finish generating before an update, so the “weakest link” effect is pronounced. In engineering, asynchronous sampling is often introduced: the generation engine keeps producing trajectories while the training side updates on whatever data is available. This can substantially improve utilization, but it introduces a data-staleness problem that must be controlled by limiting sample “age” or applying importance-sampling corrections.
If training and inference are deployed separately, the training side can scale large-model training through tensor parallelism, pipeline parallelism, ZeRO-style memory partitioning, and activation recomputation; the generation side relies on continuous batching, paged memory management, and prefix caching to improve throughput.
2.4 Extended thinking: how far are we from recursive self-improvement?
Some teams view this round of training as a key step in exploring the path of Recursive Self-Improvement (RSI): on a foundation of verifiable complex tasks, they scale up RL compute so the model keeps expanding its capability boundaries through continuous exploration and feedback.
For engineering teams, this goal imposes three requirements on the platform:
- Automatic task generation: use execution results as the supervision signal to continuously produce tasks with ground-truth answers;
- Immediate feedback: sandboxes, test frameworks, and validation services must offer high concurrency and strong isolation;
- Automated closed-loop iteration: the training, evaluation, selection, and release workflow must be highly automated.
This also means sandbox security must be designed in from the start, including restricting network access and filesystem permissions, setting CPU and memory quotas, and setting execution timeouts.
3. The cost perspective: how to understand a single RL training expense
3.1 Per-task cost breakdown
From the public data, the cost ratio of the flagship version to the inference version is about 3:1 ($2.62 million to $850,000), consistent with their positioning around omni-modal, multi-step reasoning: the longer the output and the more samples, the more expensive each trajectory becomes. The table below gives an estimate (the per-step costs are estimated values):
| Metric | Flagship omni-modal | Efficient inference | Total |
|---|---|---|---|
| RL post-training cost | ~$2.62M | ~$850K | ~$3.47M |
| Training steps | 30 | 30 | 60 |
| Avg. cost per step (est.) | ~$87K | ~$28K | ~$58K |
| Total trajectories | ~750K (combined) | — | ~750K |
| Avg. cost per trajectory | ~$4.6 | — | — |
It should be noted that “750,000 trajectories” is a combined figure, with no disclosed breakdown by version. The flagship version’s actual per-trajectory cost should be higher than the average in the table, while the inference version’s should be lower.
Breaking it down further by “compute-hours × unit price,” GPU or NPU cluster cost can be estimated with the following formula:
Total compute cost = effective compute scale × training duration × price per compute-hour ÷ compute utilization
Among the four factors, compute utilization is the most easily underestimated. Mixed-precision training, communication optimization, and data prefetching can keep utilization at a high level, while hidden costs such as trajectory-length long tails, weight-sync waiting, and sandbox-grading queues often drag actual utilization down considerably. Extrapolating from the roughly $31,000 per hour spent in the case, even at $2 per accelerator-hour, it amounts to more than ten thousand cards running at full load for several days, which gives a sense of the scale.
3.2 Cost accounting: post-training is only part of the TCO
The public data specifically emphasizes that the roughly $3.47 million covers only RL post-training, not earlier R&D or inference costs. This is exactly the point most easily misread when enterprises evaluate whether to self-train or call a large model. It is recommended to account for total cost of ownership (TCO) across three layers:
- One-time investment: data construction and cleaning, reward and sandbox development, training infrastructure setup, and labor costs;
- Training cost: number of RL rounds × per-round cost. Behind one successful training run, there are often multiple rounds of failed experiments stacked together;
- Ongoing operating cost: inference cost × call volume × lifetime, which typically scales linearly as users grow.
def estimate_rl_cost(gpu_hours, cards, price_per_card_hour, utilization):"""Estimate the cost of a single RL training run based on effective compute"""if not 0 < utilization <= 1:raise ValueError("Compute utilization must be between 0 and 1")return gpu_hours * cards * price_per_card_hour / utilization# Example: effective compute scale × duration × unit price ÷ utilizationcost = estimate_rl_cost(gpu_hours=120, cards=8000, price_per_card_hour=2.0, utilization=0.6)print(f"Estimated single-run cost: ${cost:,.0f}")
3.3 Real-time cost conversion: give the team a “spending clock”
The public data also provides two cross-verifiable conversion bases:
- By time: the real-time page shows about $31,000 per hour, so linearly extrapolating, the cumulative $3.47 million corresponds to roughly 112 hours of training duration;
- By task: about 750,000 trajectories in total, which works out to roughly $4.6 per trajectory.
The two bases corroborate each other, indicating that the cost figures on the real-time page largely accumulate linearly with actual compute consumption. For internal teams, this kind of “spending clock” is even more valuable: converting the budget into trainable steps or hours lets anyone judge at any time “how many more rounds we can try.” The calculation formula is as follows:
# Budget ÷ (cost per step × hours per step) = number of executable stepsmax_steps = budget / (cost_per_step * hours_per_step)
3.4 Paths to lower cost
- Trajectory reuse: one update corresponds to multiple small policy-gradient steps, spreading out generation cost;
- Length and difficulty control: set a reasonable maximum generation length and filter out samples with no advantage;
- Cost-first: in the exploration phase, let lightweight models do the work, and upgrade to the flagship model only at key stages;
- Hardware and deployment optimization: mix accelerator cards of different price points, and reduce generation overhead through quantization and disaggregated deployment;
- Budget guardrails: set budget thresholds, step limits, and automatic circuit breakers to prevent runaway consumption from anomalous tasks.
4. Process transparency: the value and design points of a real-time training page
Publishing RL training progress before the model’s official release is a practice in training transparency. A real-time page typically shows training progress, token consumption, training cost, sample counts, and several internal metrics. Its value is mainly threefold:
- Credible: cost and progress can be verified externally on a linear, time-based basis, reducing doubts about “inflated data”;
- Educational: peers can directly see the true scale of large-scale RL and avoid underestimating post-training cost;
- Constrained: once the budget is public, resource scheduling, anomalous tasks, and wasted compute all face external scrutiny.
At the same time, it is necessary to clarify the display boundaries of a real-time page: it presents the state during this round of training, not the evaluation results of the final model. Mid-training metrics fluctuate with task difficulty, sampling strategy, and grading rules, and cannot be simply extrapolated into the model’s capability at release time.
When enterprises build their own training observability system, it is recommended to cover at least the following metrics:
- Progress: completed steps / total steps, cumulative samples and tokens;
- Cost: cumulative cost, hourly spend, average cost per step;
- Quality: reward mean and variance, task pass rate, output-length distribution;
- Health: time per step, generation throughput, sandbox failure rate, share of anomalous tasks.
def build_training_dashboard(metrics_store):"""Aggregate real-time training dashboard data from the metrics store"""return {"progress": {"step": metrics_store.latest("step"), "total": metrics_store.latest("total_steps")},"cost": {"usd_total": metrics_store.sum("cost_usd"),"usd_per_hour": metrics_store.avg("cost_usd", window="1h"),},"quality": {"reward_mean": metrics_store.avg("reward", window="50p"),"pass_rate": metrics_store.avg("task_pass_rate", window="50p"),},"health": {"seconds_per_step": metrics_store.avg("step_seconds"),"sandbox_fail_rate": metrics_store.avg("sandbox_fail", window="1h"),},}
Metric collection should follow three principles:
- Asynchronous and low-overhead: metrics are first written to a local queue, then reported in batches, avoiding slowing down the main training loop;
- Long-term retention: raw metrics are written to object storage while aggregated data goes into a time-series database, making later retrospectives easier;
- Unified alerting: when time per step or failure rate exceeds a threshold, monitoring alerts notify on-call staff in a timely manner.
5. Release and delivery: from benchmark scores to diverse inference forms
5.1 How to view composite benchmarks rationally
The flagship model scored 46 on a third-party composite intelligence index, becoming the highest-ranked open-weight model on that index. For those making technology selections, three points are worth noting:
- Look at subscores, not the total: a composite index aggregates multiple capabilities, so it should be read alongside subscores for coding, math, tool calling, long context, and multimodality to judge whether it matches your own scenarios;
- Watch for contamination risk: once an evaluation set enters public corpora, scores may be inflated, so it should be re-checked against a self-built business evaluation set;
- Distinguish open-source levels: open model weights, open training code and data, and whether the license permits commercial use are three different levels. Opening weights only is not the same as being fully open source, and each item should be confirmed before adoption.
5.2 The general techniques behind high-speed inference mode
The “ultra-fast” mode launched alongside the release claims an output speed 20× the base model. From a general-technology standpoint, output speedup usually comes from a combination of the following approaches:
- Speculative decoding: a small draft model proposes multiple candidate tokens at once, and the large model verifies them in parallel, reducing the number of serial decoding steps;
- Quantization and sparsification: reducing weight-memory access and computation;
- Serving optimization: continuous batching, paged memory management, and prefix caching improve concurrent throughput;
- Distillation: using the flagship model to guide a lightweight model, achieving higher speed while retaining most of its capability.
It should be emphasized that output speed depends heavily on input and output length, concurrency, and the measurement method (end-to-end latency vs. pure decoding speed). When comparing across approaches, load-testing conditions should be unified, and time-to-first-token (TTFT), per-request output rate, and total system throughput should be tracked separately.
5.3 Clients and cloud services: building elastic inference infrastructure
Desktop clients and open platforms serve two kinds of needs: the former focuses on local deployment, model-cache updates, and offline availability; the latter focuses on elastic scaling and pay-as-you-go billing. Whichever form, the foundation relies on several general cloud capabilities:
- Elastic compute: to handle traffic peaks and troughs;
- Container platform: to host inference-service orchestration;
- Object storage: to store model weights and training trajectories;
- Message queues: to smooth out traffic and support asynchronous tasks;
- Gateway: to handle authentication, rate limiting, and quota management;
- Logging and monitoring/alerting: to locate latency spikes and sudden error-rate increases.
The corresponding capabilities can be found in the general product categories of cloud providers such as Baidu AI Cloud. Typical load-test metrics for an elastic inference service can be defined like this:
def evaluate_serving(latencies_ms, throughput_rps, slo_ms=2000, target_qps=100):"""Evaluate whether an inference service meets latency and throughput targets"""latencies_ms.sort()p99 = latencies_ms[int(len(latencies_ms) * 0.99) - 1]return {"p99_ms": p99,"slo_met": p99 <= slo_ms,"capacity_met": throughput_rps >= target_qps,}
6. Version governance: smooth migration before a model is retired
The generational switch in the case is very typical: a retirement notice on October 9 and an automatic switch on October 14, leaving users a five-day transition period. It looks simple, but it has a large impact on the stability of business systems. A model update is equivalent to “replacing a runtime dependency”: even if the interface stays the same, output style, format-following ability, refusal boundaries, and inference latency may all change. It is recommended to establish governance mechanisms across five areas:
- Explicit version management: adopt a two-part version number of “generation.capability version,” such as 3.1 and 3.2. Capability upgrades do not trigger automatic switching; only generational upgrades go through the formal migration process, and pinning a specific version is supported;
- Gradual rollout and rollback capability: ramp up traffic gradually by proportion while keeping old and new versions running in parallel for a short period, with one-click rollback when necessary;
- Automatic regression: use a fixed prompt set to cover key capabilities such as JSON output, tool calling, and refusal of sensitive content, and ramp up traffic only after old and new versions pass comparison;
- Observability: track success rate, P95/P99 latency, average token consumption, and cost separately by version;
- Change notification: announce the retirement time, replacement version, and migration recommendations in advance through multiple channels such as official documentation, announcements, and the console.
def route_model(request, traffic_state, policy):"""Select the model by migration policy: gradual rollout, rollback on anomalies"""version = request.model_version or policy["target"]if traffic_state.error_rate(version) > policy["error_threshold"]:version = policy["fallback"] # New version is anomalous, fall back to the stable versionelif version == policy["target"] and random.random() < traffic_state.ratio(version):version = policy["target"] # Gradually roll out to the new version by proportionreturn inference_client.chat(model=version, messages=request.messages)
Accompanying edge cases also need to be validated, including streaming output, long context, multi-turn dialogue, function calling, and content safety and compliance requirements (such as labeling generated content).
7. Practical recommendations
- Validate at small scale first, then scale up: get the reward, grading, and sync pipeline working at a scale of thousands of trajectories, and confirm there is no reward hacking or memory bottleneck before scaling to ten-thousand-scale trajectories per step;
- Design rewards before investing compute: verifiable tasks are the prerequisite for RL to scale, and the grading environment should be treated as core infrastructure rather than an auxiliary tool;
- Establish budget guardrails and a “spending clock”: set budget, step, and duration caps for each training round, and cut losses promptly when returns plateau;
- Decide from a TCO perspective: when selecting, evaluate training cost, inference cost, and migration cost together, rather than looking only at a single benchmark score;
- Make evaluation and load testing a pipeline: run business evaluations and performance load tests automatically on each model iteration, to avoid production incidents triggered by version switches.
8. Conclusion
The figures from the case—“under six days, 750,000 trajectories, about $3.47 million”—visibly demonstrate the true cost and efficiency of large-scale RL post-training; the publicly released real-time training page and the open-source release, meanwhile, represent the trend toward training-process transparency. For developers and enterprises, more important than any single-point score are three things: clearly understanding the task, reward, and compute structure of RL post-training; accounting for cost from a TCO perspective; and using version governance to safeguard online stability throughout model iteration.
