Post-Training Reinforcement for LLMs: Engineering Practice in Finance and 3D Modeling

    This article examines a shift in large language model development: keeping the base model fixed while investing heavily in post-training. It explains the main post-training methods, including supervised fine-tuning, reinforcement learning, and tool-use trajectory training, and shows how they apply to financial analysis and 3D modeling. It also covers the engineering risks, hallucination management, and selection criteria that enterprises should consider when adopting these models.

    Over the past two years, competition among large language models has centered on pretraining: larger parameter counts, longer context windows, and better pretraining data mixes. Recently, however, a clear trend has emerged. With the base model architecture and weights unchanged, heavy investment in post-training can still deliver substantial capability gains.

    Consider a recent iteration of a Chinese-developed large model. The new version did not replace its base model. Instead, the developer shifted a large share of its compute budget to post-training. Through joint reinforcement learning across hundreds of downstream task domains, the model approaches the level of leading overseas frontier models on several professional benchmarks, including programming, agent evaluation, financial research, and 3D modeling. It also ranks among the global leaders on some public leaderboards, and its invocation cost is only a fraction of that of comparable products.

    This trend has three implications for developers and enterprise technology decision makers:

    1. Base model capabilities are converging, and differentiation is moving to post-training quality. When several vendors’ base models perform similarly, the vendor that invests more in alignment and reinforcement for specialized scenarios can deliver better results in concrete business use cases.
    2. Cost structures are beginning to favor large-scale deployment. A price drop of one order of magnitude in inference makes many workflows that were previously too expensive economically viable, such as batch document verification and full-scale code review.
    3. Model selection should shift from general benchmark scores to scenario-based testing. The correlation between general leaderboard scores and real business performance is weakening, so enterprises need to build their own domain evaluation sets.

    2. Breaking Down Post-Training Techniques

    Post-training is not a single technique. It is a combination of methods. Understanding these components helps assess how much a model’s enhancement actually matters.

    2.1 Supervised Fine-Tuning (SFT): Injecting Domain Knowledge and Output Formats

    The first stage is usually instruction tuning and domain SFT. In financial scenarios, the training data covers tasks such as financial statement parsing, regulatory document Q&A, and research memo writing. This teaches the model to:

    • Identify account hierarchies and reconciliation relationships in financial statements;
    • Organize outputs according to industry conventions, such as stating the conclusion first, then the supporting evidence, then the risk disclosures;
    • Explicitly state uncertainty instead of forcing an answer when it is not sure.

    2.2 Reinforcement Learning (RLHF/RLVR): Aligning with Professional Preferences

    The second stage is reinforcement learning. Reinforcement Learning from Verifiable Rewards (RLVR) can automatically construct reward signals for tasks with standard answers, such as code execution results, mathematical derivations, and numerical checks in tables. Reinforcement Learning from Human Feedback (RLHF), which relies on human or model preferences, is used for subjective dimensions such as writing quality and depth of analysis.

    The “joint reinforcement learning across hundreds of downstream domains” described above is essentially large-scale multi-task RL. Reward models and verifiers from different domains are scheduled together, so the model keeps improving on the distribution of professional tasks rather than only aligning with general conversational preferences. The engineering challenges include:

    • Sparse rewards and reward gaming: The model may learn to exploit surface features of the reward function (reward hacking). This requires regularization and adversarial sample constraints.
    • Multi-task interference: Optimization directions in different domains may conflict. This calls for a sound task sampling ratio and gradient isolation strategies.
    • Capability regression control: Post-training should not come at the cost of general capabilities. Teams typically mix in general-purpose data and set up regression monitoring.

    2.3 Tool Use and Agent Trajectory Training

    Models built for workflows also need to learn to call tools for retrieval, computation, code execution, and operating professional software. These abilities are acquired through behavior cloning and reinforcement learning on synthetic or real agent trajectories. They are one of the largest consumers of post-training compute.

    3. Finance: From Conversation to Task Execution

    Finance is one of the key focus areas of this round of post-training. The core requirement in finance is not fluent text generation. It is accuracy, verifiability, and traceability.

    3.1 Three Key Capabilities

    Cross-document verification: Real financial analysis often requires comparing multiple documents, including annual reports, quarterly reports, announcements, and audit opinions. The model must locate how the same metric is described across different files and identify differences in definitions and data contradictions. This requires long-context capability and structured evidence chains, where every conclusion includes a source location.

    Financial modeling: Given historical financial data and business assumptions, the model generates a linked three-statement forecast. The main challenge is numerical consistency. For example, depreciation assumptions in the income statement must flow through to the cash flow statement and balance sheet, and any local change must keep the statements balanced. This type of task suits a hybrid pipeline of model generation followed by programmatic validation:

    ```python
    def validatemodel(statements: dict) -> list[str]:
    “””Check the three financial statements in the model output for consistency and return any inconsistencies.”””
    errors = []
    bs, is
    , cf = statements[“balance”], statements[“income”], statements[“cashflow”]

    1. # Assets = liab