Vertical vs. General LLMs: How Small Models Win with Reinforcement Learning (RL)

    When teams bring LLMs into domain-specific applications, they face a core decision: call a general-purpose model or train a vertical one. This article explains why supervised fine-tuning (SFT) alone cannot break the reasoning ceiling on math, multi-step logic, and code tasks, and how reinforcement learning (RL)—specifically Group Relative Policy Optimization (GRPO)—lets small models, such as a 1.5B model, outperform larger general models on vertical tasks. It covers a production-grade training framework built on training/inference decoupling, walks through GRPO's loss and reward functions with code, and shares practical guidance on verifying reward signals, protecting data quality, and scaling cost-efficiently.

    Vertical Domain LLMs vs. General LLMs: Should You Build or Buy?

    When bringing LLMs into real-world, domain-specific applications, teams typically face a route decision early on: call a general-purpose model with a massive parameter count and broad capabilities, or train a vertical model focused on a specific scenario using your own data?

    General-purpose models are appealing because they cover a wide range of tasks and work out of the box. But inference compute and API costs can run high, and they aren’t always precise enough on specialized vertical tasks. Vertical models look “small and elegant,” but if the approach is wrong, they can easily end up as a general model with a different label. So the key to vertical domain modeling isn’t model size—it’s whether you use the right training paradigm.

    Why SFT Alone Hits a Reasoning Ceiling

    A common mistake is treating “building a vertical model” as simply “running supervised fine-tuning (SFT) on industry data.” The problem is that SFT’s training objective is to imitate a given distribution of answers. It’s good at teaching a model how to speak—matching tone and format, aligning surface patterns—but it struggles to change the model’s underlying reasoning ability.

    For tasks that depend heavily on reasoning, such as math, multi-step logical deduction, and code generation, SFT alone hits a clear bottleneck:

    • It imitates “what an answer looks like” rather than “the reasoning process that produces the answer”;
    • Annotated data provides only positive examples, so the model can’t learn from mistakes;
    • When it encounters out-of-distribution combinations, it lacks a mechanism for self-correction.

    The result: the model answers more and more plausibly, but its accuracy ceiling stays locked in place.

    Reinforcement Learning: How “Small Models” Turn the Tables

    The way out is to introduce reinforcement learning (RL) for targeted training. Unlike SFT, which only learns from positive examples, RL closes a “generate–score–optimize” loop that gives the model feedback from reward signals: correct answers earn positive rewards, wrong answers earn negative ones. This lets the model genuinely learn how to reason step by step toward the correct answer.

    The gains from this paradigm are significant:

    1. Higher sample efficiency: no need for massive human annotation—rewards can be judged automatically by rules;
    2. Capability breakthroughs: the model finds better strategies through exploration instead of just imitating;
    3. Lower cost: a very small model (around 1.5B parameters), after sufficient RL training, can even outperform a general-purpose model on vertical tasks—while consuming only a fraction of the compute.

    A frequently cited real-world result: on a single high-end training GPU, combined with Group Relative Policy Optimization (GRPO), a 1.5B small model raised its accuracy on an elementary-school math word-problem reasoning benchmark from roughly 49% to roughly 80% (actual results vary with data quality, hyperparameter settings, and training steps). This data clearly shows the huge potential of “small model + RL” in vertical domains.

    A Production-Grade RL Training Framework: Architecture and Data Flow

    To run this pipeline stably and efficiently, you need a production-grade training framework. The traditional approach puts both training and inference on the same deep learning framework, which leads to two major problems: first, inference can consume a large share of the time (up to 70% in practice) yet still isn’t fast enough, slowing overall iteration; second, GPU memory scheduling and distributed coordination are stitched together manually, adding complexity and error risk.

    The proven solution used by mature frameworks is to decouple training from inference and organize capabilities in layers:

    • Data engine: handles sample loading, batching, and format conversion;
    • Inference engine (high throughput): runs the “generation” step that dominates RL, using a high-throughput inference backend to speed up overall training by several times;
    • Training engine: handles forward/backward propagation and optimization of policy parameters, often using strategies like Fully Sharded Data Parallel (FSDP) to relieve memory pressure;
    • Distributed coordination layer: orchestrates task scheduling, resource allocation, and communication across nodes.

    The corresponding data flow can be simplified into five steps:

    1. Read training data (often in a columnar storage format for efficient batch reading);
    2. Inference nodes batch-generate multiple candidate answers per question (accelerated by a high-performance inference engine);
    3. A reward function (rule-based or model-based) scores each answer;
    4. Training nodes update the policy parameters based on the rewards;
    5. Periodically run validation evaluation and save checkpoints.

    This “generate–score–update” loop is the backbone of RL training.

    How GRPO Saves GPU Memory

    One of the biggest pain points in RL training is GPU memory overhead. Take the classic policy gradient method (PPO) as an example: it needs an extra Critic (value network) to estimate state values for computing the advantage function—meaning an additional model must live in memory, typically consuming an extra 30%–40%.

    Group Relative Policy Optimization (GRPO) takes a clever approach: for the same question, it samples multiple answers at once, then normalizes using the “group mean/standard deviation” of that set of rewards to compute each answer’s relative advantage—completely eliminating the Critic network. One fewer model means less memory pressure and a simpler training pipeline.

    Its five training steps are: Rollout (generate multiple answers) → Reward (score each one) → Advantage (compute relative advantages from group statistics) → Policy Update (update the policy) → Validation (run evaluation).

    Code Walkthrough: GRPO Loss and Reward

    GRPO can be viewed as a simplified form of PPO. Its core loss function can be expressed concisely as the illustrative code below:

    1. import torch
    2. def compute_grpo_loss(new_logprobs, old_logprobs, advantages):
    3. # Compute the probability ratio before and after the policy update
    4. ratio = torch.exp(new_logprobs - old_logprobs)
    5. # Objective weighted by relative advantage (negated to form the loss to minimize)
    6. return (-ratio * advantages).mean()

    For math reasoning tasks, the key to the reward function is to “extract the final number and check whether it’s correct.” Benchmarks like this typically use #### to separate the final answer:

    1. import re
    2. def extract_solution(text):
    3. # Match the final number after ####
    4. m = re.search(r"#### (\-?[0-9\.\,]+)", text)
    5. return m.group(1).replace(",", "") if m else None

    The role to understand above all: the Actor (policy model) is the only model updated during training. It learns how to answer better; the reward and advantage calculations simply provide its learning signal. Clarifying “who updates and who scores” is the first line of defense against non-convergent training.

    A Practical Guide to Common Pitfalls

    Based on engineering experience, here’s what to watch for when going deep into a vertical domain at low cost:

    1. Validate the reward signal before scaling up: a wrong reward function (e.g., a regex that doesn’t match, or flawed scoring logic) can directly cause “the longer you train, the worse it gets.” Always verify that rewards are trustworthy on a small sample first;
    2. Data quality over quantity: training and validation data must be strictly deduplicated and isolated to avoid data leakage that inflates evaluation scores;
    3. Separate inference and training: handing the generation step to a high-throughput inference backend is a key lever for both speed and memory savings;
    4. Start small: first get the end-to-end loop working with a small model and a small dataset, then scale up gradually—avoid committing huge compute upfront only to hit pitfalls early;
    5. Elastic compute: this kind of training has concentrated GPU demand. On-demand cloud GPU instances (such as Baidu AI Cloud’s elastic computing) let you scale up during training peaks and release resources when idle, avoiding the idle costs of a self-built cluster.

    Conclusion: So, Which Should You Choose?

    Vertical and general-purpose aren’t an either/or choice—they’re a continuous spectrum:

    • If you want a fast launch, broad coverage, and can tolerate API costs, just use a general-purpose model;
    • If the task is highly vertical, sensitive to cost and data privacy, and centered on reasoning ability, then “small model + reinforcement learning” is an extremely cost-effective path.

    The core takeaway: a vertical model’s ceiling isn’t determined by parameter count—it’s determined by the training paradigm. With the right RL, even a 1.5B small model can outperform general-purpose models in its own lane.