LLM Post-Training with Reinforcement Learning: Engineering Practices, Observability, and Cost Governance
As the marginal returns from pre-training gradually diminish, reinforcement learning (RL) post-training has become an important path for advancing large language model (LLM) capabilities. This article examines how RL post-training lands in practice, how it is observed, how costs are controlled, and how models are delivered. Drawing on the engineering practices and cost governance methods behind a recent wave of open-weight model releases, it helps readers turn frontier techniques into reusable engineering experience. The key takeaways center on four pillars: verifiable-task-driven reinforcement learning, real-time training dashboards, disciplined cost accounting, and smooth delivery pipelines.
Technical Context: From Imitation-Based Alignment to Exploration-Based Improvement
How capability focus has shifted across generations
Piecing together public information, the evolution of a recent open-source model series shows a clear shift in where capabilities are emphasized:
- First generation: Excels at efficient inference, laying the groundwork for mobile and on-device experiences.
- Second generation: Introduces full-modal capabilities, extending multimodal input from text and images to video and audio, while integrating 3D spatial understanding.
- Third generation: Focuses on reinforcement learning. A flagship version and a lightweight version are trained in parallel—the former emphasizes complex reasoning and holistic judgment, while the latter emphasizes inference efficiency and cost.
One noteworthy detail: the second-generation series was taken offline at the end of June 2026. Its two variants were only notified of retirement shortly before the third generation’s release, and calls were automatically routed to the new series on October 14. This suggests that seamless takeover by a new generation has become a routine part of open-source model iteration.
The essential difference between supervised fine-tuning and reinforcement learning
Supervised fine-tuning (SFT) solves “learning to imitate”: the model learns “what a good answer looks like” from labeled examples. Reinforcement learning (RL) solves “learning to explore”: the model generates its own candidate solutions, judges their quality based on a reward signal, and progressively reinforces effective strategies.
This distinction has two direct implications:
- Different reward sources. SFT relies on human-labeled reference answers, whereas RL needs a “judge” that can provide feedback automatically.
- Different modes of capability growth. SFT mainly improves fit quality within the existing distribution, while RL—through large-scale exploration—can teach the model solution paths it previously rarely used, such as self-verification, step-by-step reasoning, and backtracking correction.
Practical opportunities from open weights
Open-weight models are typically released as downloadable weights, often under permissive licenses that allow commercial use, modification, and redistribution. For enterprises and developers, this means post-training experiments can be conducted freely on the same base model, without being tied to a single vendor’s API. It is also why the rankings of open-weight models on public benchmark suites often serve as an intuitive reference for gauging their capabilities.
Core Practice: Reinforcement Learning Driven by Verifiable Tasks
What RSI is and how it relates to reinforcement learning
RSI (recursive self-improvement) refers to a model participating in its own next round of improvement, so that capability gains form a self-sustaining loop. RL post-training built on verifiable tasks is regarded as a key step in exploring this path: using complex tasks whose answers can be automatically judged as a starting point, RL compute is scaled up so the model can continually expand its capability boundary through sustained exploration and feedback.
“Verifiable” is the keyword of this paradigm, for three reasons:
- Reliable rewards: Correctness is determined by rules, programs, or exact comparison, free from the subjectivity of model self-evaluation.
- Strong scalability: Tasks can be generated in bulk by programs, with scale no longer bounded by human annotation capacity.
- Dense signals: Whether a solution is correct is itself clear feedback, well suited to RL optimization.
Common implementations today concentrate in the following areas:
| Task type | Typical tasks | Verification method |
|---|---|---|
| Mathematical reasoning | Competition problems, word problems, multi-step computation | Numeric or expression comparison |
| Code generation | Algorithm problems, unit-test-driven repair | Sandboxed compilation and test execution |
| Logic and constraint satisfaction | Planning, scheduling, puzzle solving | Constraint checkers |
| Tool calling | Multi-step function calls, structured output | Schema validation and result comparison |
| Scientific computing | Numerical simulation, data analysis | Tolerance comparison against reference answers |
Reward design: verifiable outcomes plus rule constraints
At the implementation level, reward functions should be deterministic, reproducible, and unambiguous, while balancing the following principles:
- Combine outcome and process rewards: Judging only the final answer makes it easy to “get lucky” or “take shortcuts,” so rule-based rewards for format compliance and step completeness can help.
- Apply moderate penalties: Give small penalties for format errors and out-of-bounds output, but no negative reward for wrong answers, to avoid the model over-avoiding exploration.
- Stratify difficulty: Tasks that are too easy yield many samples with no gradient contribution, while tasks that are too hard leave the model without positive feedback for long stretches.
As an example for math problems, a sketch of a rule-based verifier looks like this:
import refrom decimal import Decimal, InvalidOperationdef verify_math_answer(generated_text: str, reference_answer: str) -> float:"""Sketch: a rule-based answer verifier for math problems"""# Extract a numeric answer from the generated text, handling common# formats such as "the answer is:" and "\boxed{}"match = re.search(r"-?\d+(?:\.\d+)?(?:/\d+)?", generated_text)if match is None:return 0.0 # No parseable answer providedtry:predicted = Decimal(match.group(0))expected = Decimal(reference_answer)except InvalidOperation:return 0.0# Keep a tiny tolerance for numeric comparison to avoid# floating-point and formatting errorsreturn 1.0 if abs(predicted - expected) < Decimal("1e-6") else 0.0
The core loop: sampling, verification, and updating
The minimal closed loop of RL post-training has four steps: sample a batch of verifiable tasks from the task pool; have the current policy generate multiple candidate trajectories for each task; score them with the verifier; and then update the model parameters with a policy-optimization algorithm. The code below shows the main loop and a cost-accounting framework. In the pseudocode, functions such as generate and train_step must be replaced with actual inference and training implementations:
from dataclasses import dataclass, field@dataclassclass RLVRConfig:batch_size: int # Number of tasks per training stepgroup_size: int # Number of trajectories sampled per taskmax_tokens: int # Maximum generation length per trajectorytotal_steps: int # Number of training stepsprice_per_mtok: tuple # (input, output) price per million tokens, e.g. (0.15, 1.50) USD@dataclassclass RolloutStats:prompt_tokens: int = 0completion_tokens: int = 0rewards: list = field(default_factory=list)def run_rlvr_loop(policy, task_pool, verifier, generate, train_step, cfg: RLVRConfig):"""Sketch: main loop for reinforcement learning with verifiable rewards(RLVR), including per-trajectory cost and reward statistics"""total_cost, total_trajectories = 0.0, 0for step in range(1, cfg.total_steps + 1):tasks = sample_tasks(task_pool, cfg.batch_size)rollouts = generate(policy, tasks, group_size=cfg.group_size, max_tokens=cfg.max_tokens)# Aggregate token usage and rewards for logging, dashboards,# and cost estimationstats = aggregate_rollouts(rollouts, verifier)prompt_price, completion_price = cfg.price_per_mtokstep_cost = (stats.prompt_tokens * prompt_price+ stats.completion_tokens * completion_price) / 1_000_000total_cost += step_costtotal_trajectories += len(tasks) * cfg.group_size# Core update: compute advantages from per-task reward signals# and update the policymetrics = train_step(policy, rollouts, stats.rewards)# Emit structured metrics for real-time dashboard ingestionprint({"step": step,"mean_reward": mean(stats.rewards),"step_cost_usd": round(step_cost, 2),"cumulative_cost_usd": round(total_cost, 2),"cumulative_trajectories": total_trajectories,**metrics,})
An often-overlooked engineering detail: if all group_size trajectories for a given task are correct or all are wrong, the rewards within the group show no difference, the relative advantage is zero, and these trajectories produce no effective gradient for the parameter update. During training, teams should therefore continuously monitor the “effective sample ratio” and dynamically adjust the difficulty distribution accordingly. This is precisely why many public training dashboards highlight internal metric changes.
Parallel specifications and the “flagship + lightweight” productization
Training flagship and lightweight models in parallel is a common practice: the two share the same task system and verifiers but differ in compute budget and resource positioning. According to publicly disclosed figures, the flagship version costs roughly 3 times as much to train as the lightweight version, while also achieving a higher overall score on third-party composite intelligence evaluations. This confirms the positive correlation between post-training investment and capability gains, and it shows that models of different sizes can plan their training budgets against separate cost targets.
This idea maps directly onto productization:
- Flagship model: Aimed at high-value scenarios such as complex reasoning, code generation, and long-document analysis.
- Lightweight model: Aimed at high-concurrency, low-latency, low-cost scenarios such as intelligent customer service, content moderation, and real-time summarization.
- Interface abstraction: Model differences are hidden from upstream business logic, and a unified gateway routes requests by task complexity, making future replacement and upgrades easier.
Process Transparency: Designing and Operating a Real-Time Training Dashboard
The value and limits of public training pages
Practices that make the training process public show that exposing a real-time training page can directly present training progress, token consumption, training cost, sample counts, and internal metric changes. For external developers, its value lies in:
- Providing a cost reference to compare against, making it easier to estimate one’s own training budget.
- Letting outsiders understand the real relationship between “steps—trajectories—cost.”
- Demonstrating the credibility of the team’s engineering capability.
At the same time, its limits must be clear:
- Point-in-time data: The page shows cumulative data as of a specific moment and keeps changing as training proceeds.
- Intermediate, not final: The page reflects an intermediate state of a given training run and does not represent the model’s ultimate capability.
- Differences in accounting scope: The page shows the cost of that reinforcement learning process and typically excludes earlier R&D and inference-deployment spending.
Many readers mistakenly read “the metrics on a live page” as “the model’s final benchmark score,” a misjudgment that enterprises especially need to avoid when making technology choices.
Layering metrics in an enterprise training dashboard
Internal training teams also need real-time dashboards, and metrics are best organized into three layers:
# Example: layered training dashboard metric configurationtraining:throughput: # Throughput layer: reflects hardware utilization and parallelism efficiency- tokens_per_second- gpu_utilization- rollout_latency_p95optimization: # Optimization layer: reflects whether the policy is actually improving- mean_reward- group_pass_rate # Pass rate within each task's group- effective_sample_ratio # Effective sample ratio (rewards within a group are not all equal)- response_length # Generation length; guards against reward hacking causing abnormal growthgovernance: # Governance layer: reflects cost and stability- cumulative_cost_usd- trajectories_total- reward_per_musd # Reward gained per unit of cost- crash_and_resume_count # Number of interruptions and resume events
Among these, reward_per_musd (average reward gain per million USD of cost) is the core metric for cost governance. It helps teams judge whether “continued training is worthwhile” and, when budgets are constrained, promptly adjust task difficulty or sampling strategies.
A framework for accounting costs and benefits
Post-training cost is mainly made up of the following components:
| Cost type | Description | Optimization direction |
|---|---|---|
| Inference sampling cost | Input/output token consumption for generating trajectories; usually the largest share | Control generation length, batch inference, improve inference throughput |
| Training compute cost | Forward and backward passes and parameter synchronization | Mixed precision, gradient checkpointing, sensible parallelism partitioning |
| Storage and data cost | Task pools, trajectory replay, checkpoints | Tiered storage, hot/cold separation, periodic cleanup |
| Ops and labor cost | Cluster scheduling, monitoring, annotation and cleaning | Automated pipelines, standardized templates |
A simplified estimation formula is:
Total cost ≈ Σ(trajectory input tokens × input unit price + trajectory output tokens × output unit price)+ training GPU hours × unit price+ storage and ops overhead
As an example from publicly disclosed data, a given post-training run accumulated roughly 750,000 trajectories in under 6 days, averaging more than 120,000 per day. This indicates that under large-scale parallel sampling, sampling and training can be overlapped efficiently. Meanwhile, the characteristic of “few steps, many trajectories, high cost” shows that each training round consumes sampling compute at high frequency, and cost is highly sensitive to sampling strategy and generation length.
How cloud resources fit in
In enterprise practice, training jobs are usually orchestrated by a container platform; datasets and trajectory data can be stored in object storage with lifecycle policies configured; and training metrics can be collected and alerted in real time through log services and monitoring/alerting. Baidu AI Cloud likewise offers these general-purpose cloud capabilities to support enterprises in resource scheduling, data management, and runtime monitoring during RL post-training.
Capability Delivery: Inference Acceleration and On-Device Deployment
The technical logic behind high-speed output mode
Launching a high-speed output mode typically means introducing one or more of the following optimizations during inference:
- Speculative decoding: Use a small draft model to quickly generate candidates, then have the large model verify them in parallel.
- Quantization and low-precision inference: Reduce the numerical precision of weights and activations.
- Continuous batching: Dynamically batch requests to improve throughput.
- Operator and kernel optimization: Provide custom implementations for hotspots such as attention and normalization.
According to public descriptions, this mode can reach output speeds up to 20 times that of the base model. The significance of the speed gain lies not only in user experience but also in lowering the inference cost per token, which in turn expands the business scale that can be served.
Multimodality and 3D spatial understanding
Supporting multimodal input across text, images, video, and audio, and integrating 3D spatial understanding, means the model can handle at the same time:
- Mixed text-and-image documents, chart screenshots, and meeting recordings;
- Voice commands and audio summaries;
- Spatial data such as indoor scans and 3D model screenshots.
This has direct value for scenarios like enterprise knowledge-base Q&A, video content understanding, and smart-hardware interaction. However, multimodal input significantly increases context length, and compute grows nonlinearly with sequence length, so fine-grained trade-offs between quality and cost are needed.
Open platforms and desktop clients
Alongside a model’s launch, there is usually an accompanying open platform and desktop client. The open platform provides standardized APIs, while the desktop client lowers the barrier to entry for individual users. For enterprises, the value of the open platform lies in quickly embedding model capabilities into existing business systems; for developers, the desktop client is useful for local debugging and prototype validation.
Version Governance: Model Retirement and Smooth Switching
Open-source models iterate quickly, and retiring old versions requires a disciplined process. The following steps are recommended:
- Advance notice: Provide a clear retirement date and a mapping to replacement versions.
- Dual-track parallelism: Keep old and new versions available for a period, allowing gradual migration.
- Automatic switching: For calls that do not explicitly specify a version number, route to the new version by default.
- Regression validation: Compare results before and after switching to ensure key metrics do not degrade.
- Rollback plan: Retain old-version weights and configurations so a rapid rollback is possible if anomalies arise.
For example, based on a public notice, the two variants of the second-generation series will automatically switch to the new series on October 14. Such “automatic switching” strategies are friendly to callers, but they also require callers to perform compatibility testing around prompts, context length, output format, and similar aspects.
Risk Assessment: A Staged View of the RSI Path
Although RL post-training demonstrates significant potential, its limits should be viewed rationally:
- Reward hacking: The model may find loopholes in the rules and “extract” rewards through abnormally long outputs or repeated content.
- Distribution shift: Long training runs may cause the model’s capabilities to degrade on unseen tasks.
- Cost runaway: Scaling up sampling can cause costs to grow nonlinearly.
- Evaluation bias: A high public benchmark score does not guarantee equally strong performance in real business scenarios.
For these reasons, recursive self-improvement is still at an exploratory stage. It is better suited as an engineering approach for improving capabilities on specific verifiable tasks than as an “auto-evolution switch” for general intelligence. When adopting it, enterprises should adhere to the following principles:
- Anchor on business metrics rather than chasing benchmark rankings alone.
- Set training budget caps and automatic circuit breakers.
- Retain human spot checks and red-team testing.
- Establish content-safety and compliance review processes for model outputs.
Conclusion
From supervised fine-tuning to reinforcement learning, from text-only input to full-modal understanding, and from basic inference to high-speed output, the evolution of LLM capabilities is shifting from “stacking parameters” to “innovating training paradigms.” Engineering practices that are built on verifiable tasks, guided by real-time dashboards, safeguarded by cost governance, and aimed at smooth delivery are becoming a core competency for LLM teams.
For enterprises and developers, the key to understanding this paradigm is not chasing any specific version but building a reusable, observable, and cost-controllable RL post-training system. This is both a current priority in LLM engineering and a required course for constructing the foundation of intelligent applications in the future.
