LLM Post-Training and Reinforcement Learning: Practical Deployment in Finance and 3D Scenarios

    As parameter scaling flattens out, post-training — instruction tuning, preference alignment, domain tuning, and reinforcement learning — is what decides whether a general-purpose base model becomes usable inside a professional workflow. This article explains how joint reinforcement learning trains hundreds of downstream tasks inside a single optimization framework, how post-training is applied to financial workflows that demand accuracy, verification, and traceability, how 3D and spatial models can call professional tools without handing over write access, and how to reason about cost, evaluation, and hallucination rates. It includes working code samples for joint RL, a verifiable reward function, and a modeling-plan schema, along with the limits of the approach, a failure-rollback flow, a deployment checklist, and a decision table for prioritizing investment.

    1. Why Post-Training Decides Whether Specialized Scenarios Are Usable

    For the past few years, model capability growth came mainly from parameter expansion and data accumulation during pre-training. Once scale enters a plateau and the marginal return on stacking parameters decays, post-training gradually becomes the stage that determines whether a model can enter real business use.

    Post-training is everything applied on top of the base model: instruction tuning, preference alignment, domain tuning, and reinforcement learning. It moves a model from “able to talk” to “able to get work done.” The base model keeps the general knowledge and reasoning substrate; post-training targets specific downstream tasks with directed enhancement.

    The limits of the effect should be stated plainly. With the base model held fixed and only post-training strengthened, a meaningful improvement on a specific professional task usually comes at a lower total cost. Whether that improvement approaches a task-specific solution depends on three things: whether the evaluation set can be scored automatically, whether business data can be verified, and whether annotation quality is stable. If those three conditions are not met, post-training gains shrink sharply and can even turn into negative transfer. So the selection question is shifting from “whose base model is bigger” to “whose post-training is closer to my business scenario.”

    2. Joint Reinforcement Learning: Hundreds of Tasks in One Optimization Framework

    The traditional approach — one fine-tuning run per task — fragments capability, raises maintenance cost, and invites catastrophic forgetting during single-task optimization. Joint reinforcement learning (joint RL) puts hundreds of downstream tasks into one unified optimization framework and trains them together. The core logic has three parts:

    • Shared representations across tasks. Different tasks share the same underlying representations, and gradients are balanced across multiple objectives, which reduces overfitting on any single task.
    • Difficulty-stratified sampling. Sampling ratios are adjusted dynamically based on task feedback, keeping the model on a learning gradient between “solvable” and “genuinely hard.”
    • Reward shaping. A combination of verifiable rewards, process rewards, and outcome rewards keeps the model on a correct path through long-chain tasks.
    1. # Illustration: task sampling and gradient balancing for multi-domain joint RL
    2. # Prerequisite: every task in task_pool carries an auto-gradable gold answer and a difficulty score
    3. def joint_rl_step(model, task_pool, num_tasks_per_step):
    4. # sample_by_difficulty(task_pool, k) -> list[Task]
    5. # Input: the task pool and the number of tasks to draw
    6. # Output: k tasks after difficulty-stratified weighting
    7. sampled = sample_by_difficulty(task_pool, k=num_tasks_per_step)
    8. losses, rewards = [], []
    9. for task in sampled:
    10. rollout = model.generate(task.prompt) # sample a trajectory
    11. reward = compute_reward(rollout, task.gold) # outcome / process reward
    12. loss = policy_gradient_loss(rollout, reward)
    13. losses.append(loss)
    14. rewards.append(reward)
    15. # balance_gradients(losses, rewards) -> scalar loss
    16. # Input: per-task losses and rewards
    17. # Output: a domain-normalized weighted loss that keeps long-tail tasks from being drowned out
    18. model.step(balance_gradients(losses, rewards))
    19. return aggregate_metrics(rewards)

    The two functions matter in different ways. sample_by_difficulty determines the training distribution: difficulty scores set too low make the model repeat problems it has already mastered, while scores set too high produce sparse rewards and amplified variance. balance_gradients determines the weighting between tasks — typically reward normalization per domain or gradient-norm clipping, so that a single high-reward domain does not dominate parameter updates. Together they let coding, structured reasoning, domain research, and spatial geometry benchmarks rise as a whole, instead of trading off against each other.

    3. Financial Workflows: From Answering to Verifying and Tracing

    Financial scenarios demand accuracy, traceability, and verifiability above all. Post-training usually reinforces three capabilities:

    1. Cross-document verification. Annual reports, announcements, and research notes disagree on definitions and reporting bases. Building “multi-document — conflicting point — verification conclusion” samples improves the reliability of cross-checking.
    2. Financial modeling. Strengthening subject-to-subject reconciliation reasoning, such as the linkage between revenue, gross margin, expenses, and net profit.
    3. Traceable research. Conclusions come with the reasoning behind them and source citations, so output can be traced and re-checked.
    1. analysis_prompt = """
    2. Task: Assess the company's profitability trend over the past three years.
    3. Reporting-basis requirements:
    4. - Amount unit: {unit}; currency: {currency}; FX basis: {fx_basis}
    5. - Time basis: {period_basis} (calendar year / fiscal year; whether restated)
    6. - Approved data sources: {doc_index}
    7. Output requirements:
    8. 1. Cite each figure with its source (annual report, page X / announcement, item Y);
    9. 2. Where documents disagree on reporting basis, list the difference, the affected amount, and the basis for how it was handled;
    10. 3. Give the formulas and key assumptions used for gross margin and net margin;
    11. 4. If the material is insufficient, return insufficient=true and list the information gaps; do not speculate.
    12. Structured fields: value / unit / currency / period / cites[] / assumptions[] / insufficient
    13. Financial materials: {financial_documents}
    14. """

    Hallucination suppression is the dividing line for real deployment. Given verifiable rewards and high-quality annotations, the share of unsupported conclusions in professional scenarios can come down, which directly reduces manual review cost. But the specific hallucination rate depends heavily on data distribution, evaluation criteria, decision thresholds, and model version. Three things must be fixed before any effect claim: the annotation standard and scoring criteria, a frozen evaluation-set version number, and the sampling method for human review — and the result must be verified independently on your own business data rather than taken from vendor marketing numbers.

    4. 3D and Spatial Understanding: Letting a Model Call Professional Tools Safely

    This direction tests three capabilities at once: spatial reasoning (proportions, orientation, relative position), geometry and procedural modeling (turning design intent into executable logic), and professional tool invocation (function calling).

    1. modeling_plan = {
    2. "schema_version": "modeling-plan/1.2",
    3. "source": {"type": "text+image", "ref": "brief_v3.pdf#p7", "trust": "verified"},
    4. "units": "mm",
    5. "coordinate_system": {"handedness": "right", "up_axis": "+Z", "origin": "ground_center"},
    6. "primitive": "cylinder",
    7. "parameters": {"radius": 2500, "height": 8000, "segments": 32},
    8. "transform": {"translate": [0, 0, 4000], "rotate": [0, 0, 0]},
    9. "constraint": {"sit_on": "ground_plane", "clearance_min": 0}
    10. }
    11. # Validate parameters: units and coordinate system must match the host project; values must stay in allowed ranges
    12. validate_plan(modeling_plan, unit_policy="mm", max_segments=256)
    13. execute_modeling_plan(modeling_plan) # execute inside a sandbox

    Engineering practice requires explicitly declaring units, coordinate system, schema version, and input source. Without them, the same geometry silently misaligns between millimeters and centimeters, or between left-handed and right-handed systems. Exception handling covers three cases: out-of-range parameters are rejected and a validation error is returned; a tool timeout degrades to emitting a script for manual execution; and an execution failure rolls the scene back to the pre-operation snapshot.

    The security warning cannot be dropped. Modeling instructions are executed by an external tool, which means write access is effectively delegated to model output. The defaults should therefore be:

    • Sandbox isolation — a container or a separate process.
    • Least privilege — write access limited to a designated working directory, with arbitrary paths, network access, and plugin loading disabled.
    • Input validation — an enumeration whitelist plus numeric ranges plus length caps.
    • High-risk instruction interception — overwriting production project files, bulk deletion, outbound network calls, and loading unsigned scripts are blocked. High-risk operations always require human confirmation.

    5. Cost, Evaluation, and a Deployment Checklist

    The advantage of post-training is cost-efficiency: when the base model is held fixed and only directed enhancement is done, the compute per unit of result is typically lower than training a larger model from scratch. The precondition is that training and evaluation data are reusable.

    A minimal reward function using only automatically gradable signals:

    1. def verifiable_reward(pred, gold, cfg):
    2. score, detail = 0.0, {}
    3. # Outcome reward: values match within tolerance and reporting basis
    4. ok_num = close_enough(pred["value"], gold["value"],
    5. tol=cfg["numeric_tol"], unit=cfg["unit"], currency=cfg["currency"])
    6. # Citation reward: every citation falls inside the approved source list
    7. ok_cite = all(c in cfg["doc_index"] for c in pred.get("cites", []))
    8. # Process reward: reporting-basis gaps are declared explicitly
    9. score += 0.5 * ok_num + 0.2 * ok_cite + 0.2 * bool(pred.get("gap_declared"))
    10. # Abstention penalty: when material is insufficient, the model should abstain rather than guess
    11. if gold["insufficient"] and not pred.get("abstain"):
    12. score -= 0.5
    13. detail["breakdown"] = {"numeric": ok_num, "citation": ok_cite}
    14. return max(score, 0.0), detail # clamp at zero so a few negative samples cannot dominate the gradient

    Four core evaluation metrics:

    • Task pass rate (pass@1) — the completion rate on tasks that can be scored automatically.
    • Citation accuracy — correct citations divided by total citations, tracked separately from citation coverage.
    • Tool invocation success rate — parameter validation passed and execution completed without exceptions.
    • Geometry compliance rate — units, coordinate system, and constraints such as “sits on the ground” are satisfied.

    A general-capability regression set should run alongside them, so that specialized optimization cannot hide overall degradation.

    Failure-rollback flow: take a scene snapshot before every modeling or file-writing operation → generate an idempotency key → on validation failure or exception, roll back to the snapshot by that key → record the failed sample into a negative-example pool for training.

    Deployment checklist:

    1. Can the scenario be scored automatically? If not, build the evaluation first and discuss training later.
    2. Does the reward use only verifiable signals, and is the share of subjective annotation quantified?
    3. Is there a fixed annotation standard, a frozen evaluation version, and independent review sampling for hallucination measurement?
    4. Do the three evaluation layers — general capability, domain, and tool invocation — each have a regression baseline?
    5. Is tool execution sandboxed, least-privilege, and equipped with snapshots and rollback?
    6. Does the high-risk instruction blocklist cover overwrite-write, deletion, network access, and plugin loading?

    Common questions and pitfalls:

    • How do I choose a scenario? Start from the intersection of high value, auto-gradability, and obtainable data — for example financial-report analysis or parametric modeling. Get the loop closed on one scenario before expanding horizontally.
    • What mistakes are most common? Optimizing only the final answer and not the supporting evidence; drawing the training set and the evaluation set from the same source, which inflates metrics; rewarding without penalizing, which leaves the model unable to abstain; skipping cross-domain weight normalization, so small domains get drowned out.
    • How should evaluation be run? Layer the tasks, keep the metrics separate, freeze versions, sample for human review, and retain the general regression set.
    • What about the toolchain? On general-purpose cloud infrastructure, data management, training scheduling, logging, and monitoring and alerting can support observability across the full post-training pipeline. Platform selection should be assessed against your own compliance and cost constraints, and brand references in external material should follow the relevant guidelines.

    6. Conclusion and Decision Table

    The base model sets the ceiling; post-training determines usability. Use the table below to judge where to invest first:

    Criterion Worth doing first Better deferred
    Task can be scored automatically Yes No
    High-quality training data is obtainable Yes No
    Error cost is controllable / rollbackable Yes No
    General-capability regression baseline exists Yes No

    Whoever polishes the data, reward, and evaluation systems behind post-training most thoroughly is the one most likely to take the lead in deploying into professional scenarios.