Reinforcement Learning Post-Training for Large Language Models: From a 2.8T-Parameter Base to Low-Cost AI Coding

1. Introduction: A New Paradigm — From Parameter Scaling to Compute Refinement

Over the past two years, the race in large language models has largely been an arms race of parameter counts. Models with tens of billions or even trillions of parameters keep resetting leaderboard records, but “bigger” does not always mean “better.” The key driver of real-world capability gains is shifting from pretraining data scale to reinforcement learning (RL) refinement in the post-training stage.

Recently, a leading Chinese LLM team released a post-training upgrade to its open-source flagship model. Built on the original very-large-parameter base, the new version went through nearly 100 days of RL retraining, achieved a significant leap in comprehensive intelligence evaluations, and saw API inference prices cut substantially. The case offers a valuable template for the industry: rather than blindly stacking parameters, it delivers cross-generation capability improvements on an existing skeleton through high-quality post-training.

This article systematically breaks down this “large base + deep RL post-training” technical route from three perspectives—technical principles, capability outcomes, and engineering practice—and explores what it means for everyday developer workflows and enterprise adoption.

2. Inside the Technical Path: What 99 Days of RL Retraining Involved

2.1 The Base Model: A Full-Strength 2.8T-Parameter Skeleton

Understanding what post-training achieves starts with the capability boundaries of the base model itself. The base version has roughly 2.8 trillion parameters and already shows strong foundational ability in deeply technical domains such as operator optimization and compiler theory. However, on benchmarks that test “hands-on” ability—real command-line operations, complex visual understanding—the first-generation version still lagged noticeably; for example, it scored only about 13% on a terminal operation benchmark.

The takeaway: massive parameters provide knowledge capacity and reasoning potential, but converting that potential into stable, controllable output requires deep post-training intervention.

2.2 The Core Logic of RL Post-Training

Reinforcement learning post-training (commonly known as RLHF, or its advanced variants such as RLHF++, PPO, and GRPO) rests on a simple idea: build a reward model, let the model learn from massive amounts of generated samples “what a good answer looks like,” and use policy optimization to gradually move toward high-reward regions.

A retraining run of nearly 100 days typically involves several key stages:

1. Multi-domain data construction. For hundreds of fine-grained domains—code generation, visual understanding, mathematical reasoning, terminal operations—high-quality training pairs (prompt + high-quality answer vs. low-quality answer) are built so that reward signals cover the space comprehensively.
2. Reward model iteration. The reward model itself keeps updating, moving from initial human annotation to model-assisted annotation, improving the granularity and consistency of reward signals.
3. Policy optimization with KL divergence constraints. While pursuing high reward, a KL divergence penalty prevents the model from drifting too far from the base distribution, avoiding “reward hacking”—where the model exploits loopholes in the reward function instead of genuinely improving.
4. Multi-stage curriculum learning. Difficulty ramps up from simple tasks to complex, long-horizon tasks, raising the model’s capability threshold while keeping training stable.

The overall pipeline:

Base model (2.8T) → SFT (supervised fine-tuning) → Reward model training → PPO/GRPO policy optimization → Multiple iterations → Final model

2.3 Why Not Simply Build a “Bigger Next Generation”?

A natural question: if the team can build a 2.8T model, why not go straight to a 4T or 5T next-generation base? The answer lies in balancing engineering efficiency against marginal returns.

• Training cost. Doubling parameters roughly doubles training compute and time, with costs growing superlinearly.
• Inference cost. A larger model means higher per-inference cost, a burden for both API users and enterprise deployments.
• Shifting bottlenecks. Once parameters reach a certain scale, the bottleneck is often no longer “not enough knowledge” but “not knowing how to use it”—a lack of fine-grained control on specific tasks. That is precisely where RL post-training adds value.

3. Where the Capability Gains Show Up: Vision, Coding, and Terminal Operations

3.1 3D and Front-End Visual Generation: From “Can Draw” to “Looks Good”

In visual generation, the model shows a striking sense of aesthetics. Compared with closed-source flagship models of the same class, its visual quality on tasks such as front-end UI reproduction and 3D scene construction stands out.

The technical underpinnings include:

• Enhanced multimodal understanding. By introducing large amounts of UI/UX design samples during post-training, the model learns front-end design semantics such as layout constraints, color pairing, and component hierarchy.
• Reinforced spatial reasoning. For 3D generation, RL reward signals specifically target geometric consistency, plausible lighting, and material detail, pushing output from “roughly similar” to “professional grade.”
• Joint code-visual training. Generated visual results are paired with the corresponding front-end code (HTML/CSS/SVG) during training, forming a closed loop of “what you see is what you get.”

For front-end developers, this means describing requirements in natural language and getting not just code but also a high-fidelity visual preview, dramatically shortening the design-development iteration cycle.

3.2 Coding Benchmarks: Not Just “Writes Code” but “Runs Code”

Coding ability is one of the most important productivity metrics for LLMs. The post-trained version significantly outperforms its predecessor on multiple mainstream coding benchmarks, especially in:

• Long-horizon code generation. Beyond completing a single function, the model understands project-level requirements and produces complete solutions spanning multiple files and modules.
• Terminal operation understanding. On tests of real command-line ability such as Terminal-Bench, scores jumped from about 13% in the first generation—evidence that the model has learned to perform real operations in a sandbox rather than merely simulating them in text.
• Debugging and bug fixing. The RL stage incorporated many “broken code → fixed code” training pairs, so the model can pinpoint errors and suggest precise fixes when given error messages.

A typical coding-assistance scenario:

// Developer provides requirements in natural language:
// "Build a file upload component that supports drag-and-drop upload,
//  chunked transfer, and resumable uploads. Use React + TypeScript,
//  with a progress bar and error retry logic."
// Model output (simplified illustration)
import React, { useState, useCallback, useRef } from 'react';
interface ChunkUploadProps {
  file: File;
  chunkSize?: number;
  onProgress: (percent: number) => void;
  onComplete: (url: string) => void;
  onError: (err: Error) => void;
}
export const ChunkUploader: React.FC<ChunkUploadProps> = ({
  file,
  chunkSize = 5 * 1024 * 1024,
  onProgress,
  onComplete,
  onError,
}) => {
  const [uploading, setUploading] = useState(false);
  const abortRef = useRef<AbortController | null>(null);
  const uploadChunk = useCallback(async (start: number, end: number) => {
    const chunk = file.slice(start, end);
    const formData = new FormData();
    formData.append('file', chunk);
    formData.append('index', String(start / chunkSize));
    try {
      const res = await fetch('/api/upload/chunk', {
        method: 'POST',
        body: formData,
        signal: abortRef.current?.signal,
      });
      if (!res.ok) throw new Error(`Chunk upload failed: ${res.status}`);
      onProgress(Math.min(100, ((end / file.size) * 100)));
    } catch (e) {
      if ((e as Error).name === 'AbortError') return;
      onError(e as Error);
    }
  }, [file, onProgress, onError]);
  // ... chunked upload logic, resumable transfer, and retry mechanism omitted
};

This snippet shows that when facing complex requirements, the model can produce structurally complete, engineering-grade code that accounts for edge cases such as interruption and retry, rather than a simple demo fragment.

3.3 Cost Advantage: Flagship Capability at One-Fifth the Effective Price

Another highlight is the sharp drop in inference cost. According to public information, API pricing is about one-third lower than the previous generation; combined with the capability gains, the effective “cost per unit of capability” falls to roughly one-fifth of before.

For enterprise users, the implications are concrete:

• High-volume workloads. For frequent tasks such as code review, automated test generation, and batch documentation, lower cost translates directly into scalability.
• Broader access for smaller teams. Flagship-grade model capability that once only large companies could afford is now within routine reach of small and mid-sized development teams.
• Edge and private deployment. Combined with the model’s open-weight release strategy, enterprises can deploy on their own compute and further control long-term costs.

4. Engineering Practices: Turning Model Capability into Real Productivity

4.1 Building a Customized AI Coding Workflow

With a model of this capability level, developers can build a workflow like the following:

1. Requirement description → code skeleton generation. Describe functionality in natural language; the model produces an initial code structure.
2. Code review → automated optimization suggestions. Feed existing code to the model for advice on performance, security vulnerabilities, and coding standards.
3. Test case generation → automated verification. Generate unit and integration tests automatically from business logic.
4. Documentation sync → automatic API docs/README generation. Extract information from code and comments to produce structured documentation.

4.2 Terminal and Sandbox Integration: Let the Model Act, Not Just Talk

Plain text generation no longer meets the demands of complex tasks. Modern AI coding tools increasingly embed the model in environments with sandboxed execution:

• Sandbox isolation. Commands generated by the model run in an isolated environment, preventing accidental damage.
• Visual feedback loop. Execution results are returned to the model as screenshots or structured logs, forming a generate-execute-observe-correct cycle.
• Long-horizon task management. For multi-step tasks such as deployment, environment setup, and data migration, the model needs plan decomposition and state memory.

4.3 Cost Optimization Strategies

• Tiered invocation. Use smaller or quantized versions for simple tasks and reserve the flagship version for complex ones, balancing cost and quality.
• Caching and reuse. Build a caching layer for high-frequency, repeated request patterns to avoid redundant inference.
• Batching and asynchrony. Run non-real-time tasks as asynchronous batch jobs, using off-peak compute to reduce peak costs.

5. Summary and Outlook

From a “large-parameter base” to “RL retraining,” this evolution reveals a key trend in LLM development: the bottleneck is shifting from scale to precision. Through nearly 100 days of deep post-training, the model achieved cross-generation gains in visual aesthetics, hands-on coding, and terminal operations while pushing inference costs down to a highly competitive level.

For developers, flagship-grade AI coding assistance is no longer a distant luxury but productivity infrastructure that can be woven into daily workflows. For decision-makers, understanding this “base + post-training” paradigm helps inform smarter choices in model selection, deployment strategy, and cost planning.

Looking ahead, as more open-source LLMs keep investing in post-training, “stronger capability at lower cost” will become the industry norm. The real competitive moat will shift to who can best fuse model capability with specific business scenarios—the last mile in taking LLM technology from the lab to production.