Building Legal AI: Transitioning from General APIs to Post-Training

A notable industry signal recently emerged when a multi-billion-dollar legal tech company announced the launch of its first proprietary model, post-trained on an open-source base. The company explicitly stated that its core inference workloads would gradually migrate from general-purpose LLM APIs to this in-house model. The discussion this sparked among developers was not simply about “another company building its own model,” but rather the trend it revealed: in vertical professional domains, the marginal returns of general-purpose LLMs are diminishing, while the engineering feasibility of the post-training route is rapidly increasing.

Over the past two years, most vertical applications opted for a “thin wrapper + general API” approach: integrating a sufficiently powerful closed-source model, injecting domain knowledge via prompt engineering and RAG, and launching quickly. This path was arguably the optimal solution during the product-market fit validation phase. However, as operations enter the scaling phase, problems inevitably surface. This article systematically breaks down the true motivations and engineering practices behind this strategic shift from a technical selection perspective.

1. Four Hard Requirements for Models in Legal Scenarios

Legal AI requires more than just conversational capability. Its core tasks fall into four categories, each demanding vastly different model capabilities:

1. Long-document understanding and structured extraction. Contracts, judicial rulings, and regulatory provisions often span dozens of pages. The model must locate key clauses within extensive contexts, identify cross-references, and extract obligated parties and timeframes. This tests the effective utilization rate of long contexts, rather than just raw window length.

2. Precise retrieval-augmented generation for statutes and precedents. Legal conclusions must be traceable. Model outputs must bind to specific statute numbers or paragraphs from judicial documents. Any content that “looks reasonable but has incorrect sourcing” is unacceptable. This requires strong instruction-following capabilities, ensuring the model answers strictly based on retrieved results.

3. Multi-step legal reasoning. For instance, determining breach of contract liability requires sequential steps: fact extraction, element comparison, defense review, and liability apportionment. The chain is long with a low fault tolerance; an error in any intermediate step leads to an incorrect conclusion.

4. Strict constraints on format and style. Legal documents follow strict structural conventions. The structure of a complaint, legal opinion, or due diligence report cannot be left to creative generation. This demands far higher output controllability than general creative tasks.

Mapping these four requirements to model capabilities reveals a commonality: they rely heavily on domain-specific instruction fine-tuning and preference alignment, rather than the breadth of general pre-training knowledge. The advantages of general LLMs in open-domain Q&A translate quite poorly into legal scenarios.

2. Three Structural Bottlenecks of the General API Route

General APIs are a rational early-stage choice, but they encounter three unavoidable problems at scale.

Bottleneck 1: Linear scaling of unit inference costs. Legal tasks are characterized by extremely long input contexts and long outputs, resulting in token consumption far exceeding that of typical conversational scenarios. When daily requests scale from hundreds to hundreds of thousands, API bills become a significant component of Cost of Goods Sold (COGS). Crucially, this cost does not decrease with scale—there is no amortization effect.

Bottleneck 2: Uncontrollable capability boundaries. General models are trained to achieve balanced performance across a wide range of tasks, and their alignment strategies may not fit the strictness required by legal scenarios. You cannot modify its behavioral tendencies; you can only apply “patches” externally via prompts. When business logic requires the model to consistently reject out-of-bounds requests or strictly follow a specific reasoning paradigm, the marginal effectiveness of prompt engineering rapidly decays.

Bottleneck 3: Data compliance and inference locality. Legal operations handle sensitive client information, including undisclosed commercial terms, personal privacy, and litigation strategies. Whether data leaves the local environment and whether it is used for the provider’s model improvement are inevitable questions from legal and compliance teams. In regulated industries, this factor often carries veto power.

It should be noted that these three points do not negate the value of general APIs. During the product validation phase, their low startup costs, zero-maintenance nature, and comprehensive capabilities are irreplaceable. The issue only arises when the business crosses a certain scale threshold, causing the weight of these three factors to reverse.

3. The Technical Logic of Post-Training: Why Open-Source Bases Are Now Viable

Choosing an open-source base for post-training essentially shifts “capability acquisition” from “calling an external service” to an “internal production process.” This path previously had a very high barrier to entry, but its feasibility has significantly improved for several reasons:

Elevated baseline capabilities. The baselines of current mainstream open-source models in instruction following, long context, and structured output are sufficient to serve as a starting point for post-training. There is no need to train from scratch; you only need to perform domain alignment on existing capabilities. The required compute drops from “pre-training” to “fine-tuning” levels, making it affordable for small to mid-sized teams.

Mature toolchains. Stages like parameter-efficient fine-tuning, instruction data construction, preference alignment, and inference acceleration have relatively mature, universal solutions. Engineering teams can focus their effort on data quality and evaluation metrics rather than reinventing training frameworks.

Self-built evaluation. Evaluation sets for legal scenarios can be constructed internally—historical cases, expert annotations, and rule validations all serve as objective signal sources. This means teams can independently determine if post-training actually brought improvements, rather than relying on external leaderboards.

A typical post-training workflow can be abstracted into the following structure:

# Abstract representation of a post-training pipeline (not a specific product API)
def post_training_pipeline(base_model, domain_data):
    # 1. Data governance: deduplication, anonymization, quality filtering
    cleaned = clean_and_anonymize(domain_data)
    # 2. Instruction data construction: task templates + expert annotation + rule validation
    sft_data = build_instruction_set(cleaned)
    # 3. Supervised fine-tuning: primarily parameter-efficient fine-tuning
    model = supervised_finetune(base_model, sft_data)
    # 4. Preference alignment: based on expert ranking or rule feedback
    model = preference_align(model, build_preference_pairs(cleaned))
    # 5. Domain evaluation: statute traceability accuracy, format compliance rate, reasoning chain completeness
    report = evaluate(model, legal_eval_suite)
    return model, report

This pseudocode highlights a key point: the core difficulty of post-training lies not in the training itself, but in Steps 1 and 5. Data governance determines what the model learns, and the evaluation system determines whether you can detect if the model is learning the wrong thing. Many post-training projects fail because they train without evaluating, or their evaluation sets are disconnected from real business distributions.

4. Route Comparison and Hybrid Strategy

Placing the two routes in the same table clarifies the trade-offs:

Dimension	General API Route	Open-Source Post-Training Route
Startup Cost	Low	Medium-High (Data + Compute + HR)
Unit Inference Cost	High, scales linearly with volume	Low, significant scale amortization
Capability Controllability	Weak, prompt-level only	Strong, modifiable behavior and paradigms
Data Compliance	Dependent on provider policies	Fully internal closed-loop capable
Iteration Speed	Fast, follows provider upgrades	Slow, requires self-built iteration process
Ops Burden	None	Requires inference serving and monitoring

In practice, mature teams rarely make an “all-or-nothing” choice; instead, they adopt a layered hybrid strategy:

• High-frequency, domain-specific, fixed-format tasks (clause extraction, document generation, compliance checks) route to the in-house model, optimizing for cost and controllability.
• Low-frequency, open-domain, exploratory tasks (legal knowledge Q&A, preliminary case analysis) continue using general APIs, optimizing for capability breadth.
• The routing layer classifies tasks, dynamically selecting the inference backend based on task type, context length, and sensitivity level.

The key to this architecture is an observable routing layer: logging which path each request takes, along with latency, cost, and quality feedback, to form closed-loop data that guides future model iterations.

5. Four Key Actions for Engineering Implementation

If a team decides to initiate post-training, the following four steps should be executed sequentially:

First, build evaluation before training. Before fine-tuning, establish a baseline using 200 to 500 high-quality evaluation samples covering core tasks. Without a baseline, all subsequent training decisions are blind.

Second, prioritize data governance over data scale. Acquiring high-quality instruction data in the legal domain is expensive. It is better to use a few thousand meticulously annotated samples than tens of thousands of noisy ones. Anonymization and deduplication must be completed before training, with auditable records.

Third, start with a small task to validate the loop. Choose a sub-task with clear boundaries and objective evaluation metrics (e.g., extraction of a specific clause type). Run through the complete cycle of “data—training—evaluation—deployment—feedback” before gradually expanding the task scope.

Fourth, build inference services to production standards. Once the in-house model goes live, you are responsible for the availability, latency, auto-scaling, and monitoring of the inference service. This can be built on general container platforms, logging services, and monitoring infrastructure, but the responsibility shifts from the provider to your team. Capacity planning must be done in advance.

6. Conclusion: Three Signals for Model Selection

Returning to the original question: when should you transition from general APIs to in-house post-training? Observe these three signals:

1. The proportion of inference costs to revenue continues to rise, with no path to reduction in the short term;
2. The marginal returns of prompt engineering are noticeably decaying, and the business’s requirement for output stability exceeds what prompts can control;
3. Compliance teams impose explicit constraints on data flow, making external API calls fail to meet audit requirements.

If any two of these signals appear, you should seriously evaluate the post-training route. However, evaluation does not mean immediate switching—validate the actual benefits of the in-house model on sub-tasks using a hybrid architecture first. Let the data speak, then decide the migration pace. Building proprietary models in vertical domains is fundamentally a long-term investment in data assets and engineering capabilities. Model selection decisions should serve the business rhythm, not chase technology trends.