Small Models + Reinforcement Learning for Low-Cost Vertical Domain Deployment

    This article explores how teams can choose and deploy small models with reinforcement learning for vertical-domain tasks at lower cost. It compares general-purpose LLMs with task-focused models, outlines when to prefer supervised fine-tuning, LoRA, reinforcement learning, or inference-time augmentation, and explains how reward signals can improve task quality. It also introduces a restrained example setup for elementary math word problem solving, describes why production-grade RL training frameworks separate inference and training, outlines a four-layer architecture and end-to-end data flow, and explains how GRPO can reduce GPU memory pressure by removing the critic model. The article includes simplified code snippets for GRPO loss and math answer extraction, followed by practical guidance on reward design, sampling, memory budgeting, data quality, and resource scheduling.

    1. How to Choose a Model for Vertical Domains: Do Not Focus Only on Parameter Size

    When teams discuss how to choose a large model for a vertical domain, the first question is often whether they must use a larger general-purpose model. The answer depends on the task boundary, the cost budget, and the acceptable latency.

    General-purpose LLMs have broad common sense and strong generalization. However, for tasks such as mathematical reasoning, code generation, domain question answering, and industrial quality inspection, they may be “adequate but not precise.” Vertical-domain models, by contrast, focus on a single scenario. Their evaluation metrics, data loops, and reward functions are designed around business goals, which often makes it easier to optimize a specific task deeply.

    The real comparison is not simply parameter size. It is the difference in training objectives. General-purpose models aim for “all-round competence on average,” while vertical-domain models aim for “reliability on specific tasks.” This is why “small models + reinforcement learning” has become a common engineering path: reuse an existing base model, apply reinforcement learning for targeted fine-tuning, and avoid training a large model from scratch.

    To make practical selection decisions faster, teams can ask a few questions:

    • If the task relies heavily on broad open-domain question answering, evaluate general-purpose LLMs first.
    • If the task has clear rules, standard answers, or quantifiable rewards, evaluate small models with reinforcement learning first.
    • If data is limited and labeling is expensive, start with supervised fine-tuning, then use a small amount of high-quality data for policy optimization.
    • If the task requires external fact-checking, consider retrieval-augmented generation, tool calling, or inference-time augmentation.

    2. Why Small Models + Reinforcement Learning Can Improve Vertical Task Performance

    In the past, the main method for improving domain capability was supervised fine-tuning. However, supervised fine-tuning is essentially imitation of labeled answers. The model learns to align with the sample distribution, but it lacks direct feedback about whether its reasoning process is correct.

    Reinforcement learning offers another path. It does not require a single correct answer for every prompt. Instead, it uses a reward signal to evaluate whether the model output is “good” and adjusts the policy accordingly.

    This means that even a model with a small parameter count can improve reasoning quality significantly if it is trained with reinforcement learning on a vertical task and the reward function accurately captures domain preferences. Reinforcement learning often optimizes signals that are close to the actual objective, such as whether the final answer is correct, whether the format is compliant, and whether the reasoning steps satisfy business constraints, rather than surface-level text similarity.

    For selection purposes, the following comparison can help:

    • Supervised fine-tuning: Suitable for tasks with standard answers where teams want to align formatting quickly. It is simple, but feedback on reasoning quality is limited.
    • Parameter-efficient fine-tuning such as LoRA: Suitable for GPU-memory-constrained scenarios where low-cost fine-tuning is preferred. It is often used as a baseline approach.
    • Reinforcement learning fine-tuning: Suitable for tasks with clear reward signals where the goal is to improve final answer quality or decision policy. It requires more stable sampling and reward design.
    • Inference-time augmentation: Examples include multi-candidate voting, external tools, and retrieval-augmented generation. It is suitable for tasks with frequent knowledge updates or a need for external fact-checking.

    In real projects, these approaches are not mutually exclusive. A common practice is to first use supervised fine-tuning or LoRA to establish basic capability, then use reinforcement learning to improve key metrics further.

    3. A Restrained and Reproducible Illustrative Case: Look at Leverage, Not Absolute Numbers

    To illustrate the potential of “small models + reinforcement learning,” we can construct a simplified illustrative case. It is not a performance guarantee. Results will vary depending on data quality, reward design, hyperparameters, and evaluation protocols.

    Assume the experiment is configured as follows:

    • Base model: an open-source small model with approximately 1.5B parameters.
    • Task: elementary school math word problem reasoning. The model reads the problem, outputs step-by-step reasoning, and separates the final answer with ####.
    • Data split: training set, validation set, and test set are separated. The test set does not participate in reward design.
    • Reward function: final answer correctness is the main reward, and format compliance is an auxiliary reward.
    • Training method: GRPO. Multiple responses are generated for each question, and the average reward within a group is used as the baseline.
    • Hardware and hyperparameters: single GPU or a small number of GPUs, group size of 8, sampling temperature of about 0.8, training for several hundred steps, and periodic evaluation on the validation set.
    • Evaluation protocol: fixed random seed and decoding temperature, with exact matching between output answers and reference answers.

    In such an illustrative experiment, a small model with only its base capability may answer only a portion of the questions correctly. After introducing reinforcement learning, because the reward directly aligns with answer correctness and format compliance, accuracy may improve significantly.

    The more important point is the methodology: small models are not necessarily unsuitable for vertical tasks. The key factors are whether the reward is precise, whether sampling is sufficient, and whether evaluation is stable.

    If readers want to reproduce a similar path, it is recommended not to start with a complex model-based reward. Instead, use a rule-based reward such as answer correctness to complete the closed loop first, then gradually add constraints for steps, format, and readability.

    4. Why Production-Grade Reinforcement Learning Training Frameworks Are Needed

    Building a large-model reinforcement learning training pipeline from scratch usually leads to four types of problems:

    1. Slow inference: Reinforcement learning requires many “sampling and scoring” iterations. If inference efficiency is insufficient, the entire training cadence will suffer.
    2. Tight GPU memory: Model parameters, optimizer states, activations, and sampling overhead accumulate quickly, causing a single GPU to reach its limit fast.
    3. Complex scheduling: Training nodes and inference nodes play different roles and may have uneven workloads. Without unified scheduling, resource utilization will be low.
    4. Difficult reward design: The reward function directly determines the model’s optimization direction. Poor reward design can cause training to drift or become unstable.

    A production-grade reinforcement learning framework for large models usually addresses these issues systematically. Training uses mainstream deep learning frameworks, inference is handled by high-performance inference acceleration engines, and a distributed scheduling layer coordinates everything. The core improvement is decoupling training and inference.

    In traditional setups, inference often accounts for a large portion of end-to-end time. After decoupling, dedicated inference engines handle high-throughput generation, and overall throughput may increase by several times. However, the magnitude of improvement depends on model size, sequence length, concurrency, number of samples, inference engine optimizations, and the proportion of time spent on inference. Therefore, specific gains should be measured according to the team’s own hardware, model, and data configuration. Numbers from other projects should not be applied directly.

    5. Four-Layer Architecture and End-to-End Data Flow

    A production-grade framework usually adopts a clear four-layer architecture:

    1. Data engine: Responsible for dataset loading, format conversion, and batch organization. It often uses a columnar storage format to improve read efficiency.
    2. Inference engine: A high-performance, high-throughput inference acceleration component responsible for quickly generating model responses.
    3. Training engine: A mainstream deep learning training framework responsible for backpropagation and parameter updates.
    4. Distributed coordination layer: Responsible for unified resource scheduling, node role orchestration, data flow management, and state synchronization.

    The end-to-end data flow can be summarized as:

    Read data → inference nodes generate responses in batches → reward rules or reward models score responses → training nodes update policy parameters → validation evaluation and checkpoint saving.

    To improve practicality, here is a minimal configuration example:

    • Dataset fields: prompt, reference_answer, split, difficulty.
    • Inference batch size: start with 4 or 8 depending on GPU memory, and observe throughput and peak memory usage.
    • Number of samples: sample 4 to 8 responses for each prompt to ensure diversity within a group.
    • Training steps: first use a small validation set to observe learning curves and avoid ineffective long training runs.
    • Reward function interface: input prompt, generated response, and reference answer; output a scalar reward and optional feedback information.
    • Checkpoint strategy: save the best and latest checkpoints based on validation metrics, and preserve the ability to roll back.

    This structure, where generation and training are separated and coordinated by a scheduling layer, helps ensure both sampling throughput and training stability.

    6. How GRPO Reduces GPU Memory Pressure

    Traditional reinforcement learning from human feedback often uses PPO-style methods. These methods require maintaining multiple components at the same time, including Actor, Critic, Reward, and a reference model. The Critic is used for value estimation and occupies a significant amount of GPU memory.

    The core idea of GRPO is to generate multiple responses for the same question and use the average reward within a group as the baseline, replacing the value estimation performed by the Critic. This reduces Critic-related models and usually lowers Critic-related GPU memory usage. The exact savings depend on GPU model, batch size, sequence length, whether parameters are shared, mixed precision, activation recomputation, and other configurations. For GPU-memory-constrained scenarios, a critic-free approach is often easier to start.

    A single GRPO training step can be broken down as:

    Rollout sampling and generation → Reward scoring → Advantage calculation for relative advantage → Policy update → Validation evaluation.

    It should be emphasized that GRPO is not simply about removing the Critic and finishing the job. Real training still needs to handle group advantage normalization, response masks, length bias, numerical stability, and reference model constraints.

    7. Key Code Illustrations: Understanding Principles, Not Direct Production Use

    The following two snippets illustrate core logic. They are not complete training entry points and should not replace production training code.

    1. Core GRPO Loss Calculation

    1. import torch
    2. def compute_grpo_loss(
    3. new_logprobs: torch.Tensor,
    4. old_logprobs: torch.Tensor,
    5. advantages: torch.Tensor,
    6. ):
    7. """Principle illustration only; do not use directly as a training entry point.
    8. All inputs are PyTorch tensors and must match the corresponding token or response mask shapes.
    9. """
    10. if not all(isinstance(x, torch.Tensor) for x in (new_logprobs, old_logprobs, advantages)):
    11. raise TypeError("new_logprobs, old_logprobs, advantages must be torch.Tensor")
    12. old_logprobs = old_logprobs.detach()
    13. advantages = advantages.detach()
    14. ratio = torch.exp(new_logprobs - old_logprobs)
    15. return (-ratio * advantages).mean()

    In real training, the following are usually also needed:

    • Ratio clipping, to prevent importance sampling weights from becoming too large.
    • KL divergence or reference model penalties, to prevent the policy from drifting too far from the reference model.
    • Group advantage normalization, to improve stability across different questions.
    • Response masks, to avoid computing loss on padding positions.

    2. Mathematical Answer Extraction and Tests

    1. import re
    2. _NUMBER_PATTERN = re.compile(
    3. r"####\s*([-+]?(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?(?:[eE][-+]?\d+)?)"
    4. )
    5. def extract_solution(text: str):
    6. """Extract the final numerical answer from reasoning text containing ####."""
    7. if not isinstance(text, str):
    8. return None
    9. m = _NUMBER_PATTERN.search(text)
    10. if not m:
    11. return None
    12. raw = m.group(1).replace(",", "")
    13. try:
    14. if any(c in raw for c in (".", "e", "E")):
    15. return float(raw)
    16. return int(raw)
    17. except ValueError:
    18. return None
    19. # Test examples
    20. print(extract_solution("#### 1,234.5")) # 1234.5
    21. print(extract_solution("#### 1.0e-3")) # 0.001
    22. print(extract_solution("#### -12")) # -12
    23. print(extract_solution("no answer")) # None

    In actual training, rewards can be provided by rules, such as answer matching and format compliance. They can also be provided by reward models. In practice, the two are often combined, but reward caps and human spot checks should be introduced to avoid the model gaming the reward while sacrificing real quality.

    8. Pitfall Guide and Common Questions

    1. Reward design comes before model complexity

    First use a simple and reliable rule-based reward to complete the closed loop, then gradually add constraints such as format and step compliance. Starting with a complex model-based reward too early can easily make training unstable.

    2. Watch out for reward hacking

    For example, optimizing only for format may cause answer quality to degrade. Or the model may learn to output many steps but still fail to provide a correct conclusion. It is recommended to set reward caps, regularly roll back checkpoints, perform human spot checks on the validation set, and keep a small number of human-preference samples for calibration.

    3. Control sampling temperature and number of samples

    Responses sampled for the same question should have diversity. If the temperature is too low, the “group average” becomes meaningless and the advantage signal may approach zero. If the temperature is too high, noise is introduced. It should be adjusted using the validation set.

    4. Plan the GPU memory budget in advance

    Prioritize a critic-free approach to reduce baseline memory overhead. Then combine techniques such as sharded data parallelism, activation recomputation, and mixed precision to further compress memory usage. Before going live, run stability tests with a small batch size and short sequence length.

    5. Data quality determines the upper bound

    A small and precise high-quality dataset can often drive more stable improvements with reinforcement learning than a large and noisy dataset. For vertical domains, standard answers, evaluation sets, and boundary samples should be carefully curated.

    6. Separate training and inference resources

    Letting the inference engine and training engine handle their respective responsibilities, with dynamic resource allocation from the scheduling layer, is key to improving throughput and stability.

    7. Use cloud platform capabilities to reduce operational burden

    Teams can use container platforms to run training and inference workloads, object storage to manage datasets and checkpoints, and logging and monitoring services with alerting to observe training curves and resource utilization. This allows teams to focus on reward design and the algorithm itself.

    9. Conclusion

    The tradeoff between vertical-domain models and general-purpose models should not be reduced to the intuition that “bigger parameters are always better.” When the goal is a well-defined professional task, a “smaller base model + targeted reinforcement learning training” can often achieve better task performance at lower cost.

    By understanding the layered architecture of production-grade reinforcement learning training frameworks, the GPU memory optimization principles of GRPO, reward design methods, and resource scheduling practices, teams can make this approach more reliable.

    For developers who want to focus on vertical domains at low cost, this may be one of the most promising technical paths to evaluate today.