A Practical Guide to RL Post-Training and Capability Evaluation for Large Language Models

    This guide covers the practical engineering behind reinforcement learning (RL) post-training for large language models and how to evaluate the capabilities that result. It explains the three-stage post-training paradigm (supervised fine-tuning, verifiable-reward RL, and preference alignment), why reward design and training stability determine whether the approach succeeds, and how to guard against reward hacking. It then walks through capability evaluation across code, visual aesthetics, and composite benchmarks, including pass@k estimation and pairwise blind testing, and explains why a self-built evaluation set matters most. Finally, it breaks down the cost structure behind falling API prices, how to route tasks across models of different sizes, and an engineering checklist covering gateway design, fallback, observability, versioning, security, and cost governance.

    1. Why Post-Training Became the Main Battleground for Model Capabilities

    Over the past two to three years, the way large-model capabilities grow has shifted noticeably. In the early phase, which relied mainly on scaling parameters and data volume, marginal returns have gradually diminished. Attention has moved to the “post-training” stage that comes after pre-training — supervised fine-tuning (SFT), reinforcement learning (RL), and preference alignment — which is now the primary way to improve complex capabilities such as reasoning, coding, and multimodality.

    A useful way to think about the relationship is “knowledge” versus “behavior”: pre-training teaches the model world knowledge and language patterns from massive text, while post-training shapes “how it thinks, how it acts, and by what standards it acts.” The industry often calls the heavy RL phase of post-training “retraining” or “re-refinement,” and its goal is to make the model form more stable behavioral strategies in reasoning chains, tool use, code generation, and aesthetic judgment — without overturning existing knowledge.

    For developers and enterprises, this shift raises two immediate questions:

    • How do you tell whether a new model’s capability gain is real, rather than a leaderboard effect or an evaluation trick?
    • How do you turn “capability gains” into quantifiable business value while keeping call costs within an acceptable range?

    This article follows those two threads: first the engineering principles and stability concerns of RL post-training, then evaluation methods across capability dimensions such as code and visual aesthetics, and finally cost structure, selection strategy, and implementation practices.

    2. RL Post-Training: Principles, Workflow, and Common Pitfalls

    2.1 The Three-Stage Paradigm: From Imitation to Exploration

    The mainstream post-training workflow breaks into roughly three steps:

    1. Supervised fine-tuning (SFT): Train the model on high-quality “instruction–response” demonstrations so it first learns the basic format and approach for tasks, providing a cold start for reinforcement learning.
    2. Reinforcement learning with verifiable rewards: On tasks with definite answers — math, code, logical reasoning — use rule checks, unit-test execution, or formal verification to produce reward signals automatically. The model explores better solutions on its own through large numbers of “attempt–feedback” cycles. This step is the key to rapid gains in reasoning ability.
    3. Preference alignment: On open-ended tasks such as writing, conversational style, and visual aesthetics, train a reward model on human preference data or apply preference optimization directly, so outputs better match human expectations.

    The division of labor is clear: SFT solves “can it do it at all,” verifiable RL solves “is its thinking correct,” and preference alignment solves “is it good and does it feel right.”

    2.2 Reward Design Is the First Productivity

    The effectiveness of reinforcement learning depends heavily on the quality of the reward function. Common reward signals in practice include:

    • Result correctness: Whether all unit tests pass, whether the answer matches the reference answer;
    • Process soundness: Whether solution steps match expectations, whether the reasoning chain is self-consistent;
    • Format and constraints: Whether the output is parseable, whether it meets structural requirements;
    • Efficiency metrics: Whether code passes performance tests, whether it is overly verbose.

    What deserves particular caution is “reward hacking.” When the reward has loopholes, the model finds shortcuts that do not actually solve the problem, for example:

    • Hard-coding answers that only target the public test cases;
    • Truncating output or stuffing keywords to game the format reward;
    • Producing code that looks reasonable but is not actually executable.

    Reward design should therefore follow three principles:

    • Rewards must be verifiable: Prefer execution results and automated validators over letting the model grade itself;
    • Rewards must be multi-dimensional: Combine correctness, compliance, and efficiency so a single metric is not over-optimized;
    • Rewards need anti-cheating constraints: Use hidden test sets, randomized inputs, and cross-validation to reduce the chance of being gamed.

    Process rewards and result rewards each have trade-offs: result rewards are simple and reliable but give sparse feedback; process rewards provide denser learning signals but more easily introduce labeling noise and the risk of being led astray. In engineering, the two are usually combined, and rewards are normalized and clipped to prevent a few extreme samples from dominating the gradient.

    2.3 Training Stability: Easily Overlooked Engineering Details

    RL post-training is very sensitive to engineering details. Common problems include:

    • Drifting too far from the reference model: Overly large policy updates degrade language ability, usually controlled with a KL-divergence constraint that keeps the new policy close to the reference model;
    • Excessive variance in advantage estimation: Use methods such as group-relative advantage estimation to reduce variance, combined with reward clipping;
    • Training–inference mismatch: Inconsistent sampling parameters, numerical precision, or context concatenation between the two pipelines directly distorts the learning signal, so inference configuration must be unified;
    • Sample selection for long-context and multimodal tasks: Not all data is suitable for reinforcement learning, so it needs to be sampled in layers by task type.

    2.4 Sample Code: Verifiable Rewards and Sampling

    Below is an abstracted implementation of “verifiable rewards for code tasks,” meant to illustrate how reward signals are constructed. The sample code does not include real API calls.

    1. from dataclasses import dataclass
    2. from typing import Callable
    3. @dataclass
    4. class VerifiableTask:
    5. prompt: str
    6. tests: list[str] # public tests + hidden tests
    7. time_limit_ms: int = 2000
    8. def code_reward(
    9. task: VerifiableTask,
    10. completion: str,
    11. run_tests: Callable[[str, list[str], int], float],
    12. ) -> float:
    13. """
    14. Returns a reward in [0, 1]:
    15. 1.0 all tests pass
    16. partial pass ratio * 0.8 (keep the learning signal for "close to correct")
    17. 0.0 execution failure / timeout / suspected cheating
    18. """
    19. if not looks_like_code(completion):
    20. return 0.0
    21. passed_ratio = run_tests(completion, task.tests, task.time_limit_ms)
    22. reward = passed_ratio * 0.8 if passed_ratio < 1.0 else 1.0
    23. # Anti-cheating: down-weight results that look too good but are tightly coupled to public tests
    24. if passed_ratio == 1.0 and overfits_public_tests(completion, task.tests):
    25. reward *= 0.5
    26. return max(0.0, min(1.0, reward))

    The key point here is that run_tests must execute inside an isolated sandbox that restricts CPU, memory, network, and filesystem permissions, because model-generated code is itself untrusted input.

    3. Capability Evaluation: How to Tell Whether “Stronger” Is Actually Stronger

    3.1 How to Read Composite “Intelligence Index” Scores

    The industry often weights sub-scores across multiple disciplines — knowledge, math reasoning, coding, instruction following, multimodal understanding — and aggregates them into a composite index for horizontal comparison. When reading such leaderboards, note three things:

    • The weighting is subjective: Different organizations set different weights, and rankings shift accordingly;
    • Test-set contamination: If evaluation questions have entered the training corpus in some form, scores are inflated;
    • Version and configuration differences: Sampling temperature, prompt templates, and whether tools are available all significantly affect results.

    So a composite index is suitable as a rough filter but cannot replace evaluation tailored to your own business.

    3.2 Code Capability: Execution Environment Matters More Than Scores

    Code-capability evaluation typically includes function-level generation, algorithm-problem solving, and repository-level bug fixing. Compared with looking only at pass rates, more attention should go to:

    • Whether it actually runs: Dependency conflicts, version mismatches, missed edge cases;
    • Maintainability: Whether naming, structure, and comments meet team conventions;
    • Security: Whether it introduces injection risks or hard-codes credentials.

    The industry commonly uses the pass@k metric to measure the probability of “passing at least once across k samples.” Its unbiased estimator is:

    1. from math import comb
    2. def pass_at_k(n: int, c: int, k: int) -> float:
    3. """
    4. n: number of samples per problem; c: number of passes; k: evaluation window
    5. Returns the unbiased estimate of passing at least once within k samples
    6. """
    7. if n - c < k:
    8. return 1.0
    9. return 1.0 - comb(n - c, k) / comb(n, k)

    pass@1 reflects the stability of “getting it right on the first try,” while pass@k reflects the upper bound of “achievable across multiple attempts.” In real business settings, if call cost allows, a strategy of “multiple samples + automated validation + best-of selection” is often more cost-effective than simply switching to a larger model.

    3.3 Visual Aesthetics: Objectifying Subjective Preferences

    Visual aesthetics is one of the most easily perceived yet hardest to quantify capabilities improved during post-training. It shows up in the aesthetic consistency of generated images, the reasonableness of image-text layout, and the readability of interface design suggestions. When evaluating, it is recommended to:

    • Use pairwise blind tests: Have users make a two-choice preference judgment on two outputs, then compute win rates and confidence intervals;
    • Break into multiple dimensions: Score composition, color, sharpness, and consistency with the prompt separately, rather than only “does it look good”;
    • Use automated metrics as an aid: Use image-text matching and aesthetic-scoring models for a rough filter, but always acknowledge that they deviate from human preference;
    • Fix random seeds and sampling parameters: Otherwise evaluation results are not reproducible.

    A simplified pairwise-comparison statistic looks like this:

    1. def win_rate(pairs: list[tuple[str, str, str]]) -> dict:
    2. """pairs: (output A, output B, user choice)"""
    3. wins = {"A": 0, "B": 0, "tie": 0}
    4. for _, _, choice in pairs:
    5. wins[choice if choice in wins else "tie"] += 1
    6. total = len(pairs)
    7. return {k: round(v / total, 4) for k, v in wins.items()}

    When judging whether a win-rate difference is significant, you should also report a confidence interval to avoid being misled by small-sample fluctuation.

    3.4 Building Your Own Evaluation Set: The Most Reliable Basis for Judgment

    No public leaderboard can replace your own evaluation set. It is recommended to build it in layers across these dimensions:

    • Task type: Q&A, summarization, extraction, code, structured output, multimodal understanding;
    • Difficulty gradient: Easy / medium / hard, avoiding an evaluation set so easy that it causes a ceiling effect;
    • Business distribution: Sample by real traffic proportions so results extrapolate to production;
    • Adversarial samples: Include edge cases such as prompt injection, unauthorized requests, and ambiguous instructions.

    The evaluation set should be version-controlled and continuously expanded with production failure cases, forming a closed loop of “discover problems in production → feed them back into the evaluation set → validate through regression.”

    4. Cost Structure: Why Unit Call Costs Keep Falling

    Falling “API prices” are not purely a market strategy; behind them is systemic improvement in inference efficiency. The main engineering levers include:

    • Model compression: Quantization, pruning, and knowledge distillation reduce computation per token;
    • Sparse architectures: Mixture-of-experts structures activate only part of the parameters on each forward pass, lowering compute while preserving capacity;
    • Inference optimization: Paged attention caching, continuous batching, speculative decoding, and operator fusion significantly improve throughput;
    • Hardware and scheduling: Better memory-bandwidth utilization, elastic cluster scheduling, and off-peak shifting raise overall utilization.

    Once these optimizations stack up, the service cost per token keeps trending down, and vendors have more room to pass the savings on to callers. But note:

    • Input and output are priced separately: Generated output usually consumes several times the tokens of the input, so cost models should be output-centric;
    • Context length amplifies cost: Long-document Q&A and multi-turn conversations significantly raise input-side overhead;
    • Caching and batching offer discounts: Prefix-cache hits and asynchronous batch calls usually get better unit prices;
    • Failed retries are also a cost: Timeout and rate-limit retries are billed again, so they must be factored into the budget model.

    5. Selection and Hybrid Deployment: Putting Capability Where It Counts

    Different tasks differ enormously in their capability needs, and a single model “doing it all” is neither economical nor necessary. Recommended task layering:

    • Simple tasks (classification, extraction, format conversion, short summaries): small-parameter or fine-tuned small models, aiming for low latency and low cost;
    • Complex tasks (multi-step reasoning, code generation, long-document analysis): large-parameter models, with test-time compute scaling (multiple samples + self-verification) when needed;
    • Aesthetic and creative tasks: models with strong preference alignment, verified continuously through pairwise blind tests.

    An abstract layered-routing example:

    1. def route(task) -> str:
    2. if task.difficulty == "simple":
    3. return "small_model"
    4. if task.requires_code_execution:
    5. return "code_model"
    6. if task.modality == "multimodal":
    7. return "multimodal_model"
    8. return "flagship_model"

    When deploying on the cloud, a common combination of general-purpose capabilities is: object storage for training and evaluation data, a container platform for hosting inference services, a message queue for smoothing request peaks, and logging plus monitoring/alerting for observability and cost accounting. As one example, Baidu AI Cloud offers general-purpose building blocks — model services, object storage, a container platform, logging, and monitoring/alerting — that can be combined into a complete pipeline for model integration and governance.

    6. Engineering Implementation Checklist

    • Unified gateway layer: Abstract multiple model interfaces and unify authentication, rate limiting, retries, and timeout policies, so business code is not tightly coupled to any specific model;
    • Fallback and safety nets: Automatically switch to a backup model or rule-based policy when the primary model is unavailable, ensuring business continuity;
    • Observability: Track call volume, token consumption, latency percentiles, and failure rates across three dimensions — model, scenario, and version;
    • Version pinning and regression: Before upgrading a model version, run regression on your own evaluation set first, then roll out gradually once it passes;
    • Security and compliance: Bidirectional input/output review, prompt-injection protection, sensitive-data masking, and centralized key management;
    • Cost governance: Set budget alerts, allocate costs by business line, and periodically review high-consumption scenarios.

    In terms of technical evolution, competition in the post-training stage is shifting from “chasing a single metric on leaderboards” toward “composite capability plus engineering usability.” Directions worth watching include the deeper reasoning enabled by test-time compute scaling, layered coordination between reasoning-specialized models and general models, the deep integration of multimodality and aesthetic alignment, and the agent form that moves from single-turn Q&A toward multi-step task execution.

    Risks exist as well:

    • Evaluation distortion: Leaderboard scores decoupled from real business performance lead to selection mistakes;
    • Cost overruns: Long context and multiple sampling rapidly inflate spending without governance;
    • Capability dependency: Over-reliance on external model services, with a lack of your own evaluation and fallback capabilities.

    The way to respond is to establish a “three-piece set” of a self-built evaluation set + layered routing + cost observability, turning model selection from experience-based judgment into a data-driven, continuous process.

    8. Conclusion

    RL post-training is pushing large-model capability growth from “stacking parameters” toward “training behavior,” and the practical meaning of this shift for developers is that evaluation must be more rigorous, cost structure must be finer-grained, and architecture design must be more decoupled. Getting the four things right — reward design, evaluation methods, the cost model, and engineering governance — is what lets you keep earning reproducible business value in an environment where models iterate quickly.