Domain Adaptation and Evaluation of Legal-Domain Large Language Models
General-purpose LLMs struggle in legal work not because language understanding is hard, but because the cost of errors is high and outputs must be verifiable. This article walks through a practical engineering approach to building legal-domain models: domain continued pre-training, instruction tuning, preference alignment, and a clear division of labor between parametric knowledge and retrieval augmentation. It then presents a multi-dimensional evaluation system built on all-or-nothing task pass rates, field-level accuracy, citation locatability, and appropriate refusal, together with an engineering pipeline that writes review conclusions back into documents as trackable revisions and comments. It closes with governance, risk control, and a phased rollout roadmap.
1. Why general-purpose LLMs fall short in legal scenarios
The difficulty in legal work is not language understanding itself, but the cost of errors and the verifiability of outputs.
- High terminology density and strong context dependence. The same expression can carry different meanings across contracts, litigation documents, and regulatory rules, and this kind of text makes up a very small share of a general model’s training distribution.
- Complex long-document structure. In a merger agreement, schedules, definition clauses, and cross-references are often nested within one another. Simply stuffing the text into the context window does not guarantee that the model has actually performed cross-section reasoning.
- Conclusions must be traceable. When practitioners judge that “this clause carries risk,” they must also provide the clause location, the supporting source, and the reasoning path; otherwise the conclusion cannot be reviewed.
- Very small margin for error. A missed clause or an incorrect citation can directly cause business loss, so evaluation metrics cannot look only at whether the answer “sounds right”—they must look at whether the task passes as a whole.
So the core engineering problem of a legal-domain model can be summed up in one sentence: under controlled cost and latency constraints, harden domain knowledge, task capability, and auditability into the model and the system at the same time.
2. Technical roadmap: from a lightweight base to a domain model
The mainstream construction paths divide into four stages, progressively tightening the capability boundaries.
2.1 Domain continued pre-training (mid-training)
Starting from a lightweight general-purpose base model and injecting domain corpora, the goal is to fill gaps in vocabulary distribution and knowledge structure, not to change the base model’s general language ability.
- Corpus composition: public laws, regulations, and judicial interpretations; public court judgments; contract templates and clause libraries; academic literature and professional textbooks; and internal corporate policies and process documents (subject to authorization and de-identification).
- Mixing strategy: train on a mixture of domain corpora and high-quality general corpora to avoid catastrophic forgetting; typically use a curriculum-style mix, with a higher domain share early on and a gradual rebalancing toward general data later.
- Data processing: split at the clause level rather than fixed-length chunks, keeping the chapter hierarchy as structured metadata; perform entity recognition and replacement for personal information and trade secrets; and apply near-duplicate deduplication on repeated templates to prevent the model from memorizing templates.
2.2 Instruction tuning (supervised fine-tuning)
Turn “knowledge” into “task capability.” The key is that the task system must stay close to the real workflow, rather than building data around single-turn Q&A.
| Task family | Typical instruction form | Output contract |
|---|---|---|
| Contract clause extraction | Locate definitions, obligations, and termination clauses in a long document | Structured fields + source location |
| Contract risk review | Identify missing/unfavorable clauses against a rule library | Risk level + basis + revision suggestion |
| Evidence analysis | Element extraction and timeline alignment over the evidence chain | Element table + list of contradictions |
| Compliance planning | Map regulatory requirements to internal policies | Gap list + remediation priority |
| Document drafting | Generate paragraphs from a template | Text + placeholders to confirm |
On the engineering side, unify the output contract: enforce JSON Schema constraints and require fields to include evidence_span (the source snippet) and source_ref (section/clause number), so downstream systems can validate the output rather than relying on natural-language self-description.
2.3 Preference alignment and reinforcement learning post-training
Supervised fine-tuning is good at “teaching format”; preference alignment is good at “teaching trade-offs.” The focus when constructing preference pairs for the legal scenario lies in two boundaries:
- Balancing conservatism and usability: when there is no basis, the model should clearly refuse or mark “needs human confirmation,” but it should not over-refuse on every request and render the tool unusable. Preference pairs need to include both “over-refusal” and “unsourced speculation” as negative samples.
- Citation quality first: when the same answer can be phrased as “correct conclusion but no citation” or “slightly more conservative conclusion but complete citation,” prefer the latter, because the former cannot be reviewed in practice.
You can adopt reward modeling that mixes rule-based scoring with human annotation; rule items cover format compliance, field completeness, citation locatability, and a prohibition on absolute statements without basis (such as “will certainly be valid” or “absolutely not liable”).
2.4 Division of labor between parametric knowledge and retrieval augmentation
Establish one principle: volatile norms and case-specific facts should not be written into parameters; only stable discourse patterns and reasoning paradigms are suited to be hardened into the model.
- Law updates and changes in industry regulatory interpretations → retrieval augmentation, with an external knowledge base maintaining timeliness and versioning.
- Clause review logic, document structure, and writing paradigms → internalized by the model, ensuring consistency and low latency.
3. Evaluation system: measuring whether it can really be used
A single accuracy metric is misleading. It is recommended to build a multi-dimensional evaluation matrix:
- Task-level all-or-nothing pass rate (pass@k / all-or-nothing): a task contains multiple sub-fields (risk level, clause location, revision suggestion, basis), and it counts as passed only when all of them are correct. This metric is closest to real-world judgment and is the key to distinguishing “seems able to answer” from “able to deliver.”
- Field-level accuracy: pinpoint at which layer the problem lies—for example, the clause location is correct but the risk level is judged wrong.
- Citation locatability rate: whether the given basis can be precisely retrieved in the source text.
- Appropriate refusal rate: whether the model refuses when it should and does not falsely refuse when it shouldn’t.
- Cost and latency: per-document processing cost and P95 latency in long-document scenarios often become the go-live bottleneck earlier than model capability does.
# Example: multi-dimensional evaluation main loop (pseudocode)TASK_FIELDS = {"clause_review": ["risk_level", "clause_ref", "suggestion", "evidence_span"],"evidence_analysis": ["elements", "timeline", "conflict_points"],}def evaluate(model, dataset, temperature=0.0):stat = {t: {"samples": 0, "passed": 0, "field_hit": 0, "field_total": 0}for t in TASK_FIELDS}for sample in dataset:task = sample["task"]pred = parse_json(model.generate(sample["prompt"], temperature=temperature))gold = sample["gold"]passed = Truefor f in TASK_FIELDS[task]:stat[task]["field_total"] += 1if field_equal(pred.get(f), gold[f]):stat[task]["field_hit"] += 1else:passed = Falsestat[task]["samples"] += 1stat[task]["passed"] += int(passed)for t, s in stat.items():print(t,"pass_rate=%.4f" % (s["passed"] / s["samples"]),"field_acc=%.4f" % (s["field_hit"] / s["field_total"]))
Building the evaluation set is equally important: sample from real, de-identified business data, covering different document types and different risk densities, and set up a holdout set and contamination detection (to prevent inflated metrics caused by overlap between training data and the evaluation set).
4. Engineering implementation: review pipeline and in-document edits
4.1 Layered pipeline
Document parsing → Structure reconstruction (clause tree) → Retrieval (rule library / template library) → Model inference (per clause)→ Rule validation (output contract / citation locatability) → Risk aggregation → Human review
The key point is segmented inference rather than direct whole-document output: call the model node by node along the clause tree, which both reduces context pressure and makes each conclusion naturally carry location information, facilitating aggregation and traceability.
4.2 Writing conclusions back to the document: revisions and comments
Model output should not stop at textual suggestions; it should land on acceptable/rejectable revision records in the document. Mainstream office document formats are based on open document standards, where revisions and comments have clear structured representations: insertions and deletions each use paired revision markup elements, and a comment consists of start and end markers for the comment range plus a separate comment content part. An example:
<!-- Revision: change "bears no liability whatsoever" to "bears legal liability in cases of intent or gross negligence" --><w:del><w:r><w:delText>bears no liability whatsoever</w:delText></w:r></w:del><w:ins><w:r><w:t>bears legal liability in cases of intent or gross negligence</w:t></w:r></w:ins><!-- Comment range and comment content --><w:commentRangeStart w:id="1"/><w:commentRangeEnd w:id="1"/><w:r><w:commentReference w:id="1"/></w:r>
This yields three direct benefits: reviewers can accept or reject each change individually; each change carries a risk explanation as a comment; and the original text is fully preserved, meeting traceability and audit requirements.
4.3 Integration with smart office capabilities
Package model capability as a review service within office tools, exposing “document parsing — risk scanning — suggestion generation — revision write-back” through a unified interface, with object storage to save the source and results, message queues for asynchronous processing of long documents, and logging, monitoring, and alerting to track call quality. On the deployment side, use a container platform for elastic scaling and control per-unit cost through quantization and batching.
5. Governance and risk control
- Clear accountability: model output is positioned as an auxiliary opinion; the final conclusion is made and recorded by a person holding professional qualifications.
- Data boundaries: sensitive case materials should preferentially go through private or dedicated deployment; keep training and inference data separate, and prohibit unauthorized corpora from entering the training set.
- Version traceability: record the model version, prompt template version, rule library version, and retrieval snapshot, ensuring the same document can reproduce the same conclusion at different times.
- Continuous regression: after each update to the model or rule library, a fixed regression set must be run to prevent capability regression.
6. Recommended rollout roadmap
- Phase 1 (2–4 weeks): no training; first do retrieval augmentation + a strong output contract to validate business value and the real sample distribution.
- Phase 2 (1–2 months): perform domain continued pre-training and instruction tuning on a lightweight base to form a usable domain-model baseline.
- Phase 3: add preference alignment and the evaluation loop, focusing on refining refusal boundaries and citation quality.
- Phase 4: integrate into the office review pipeline, implement revision and comment write-back, and establish audit and regression mechanisms.
Overall, the value of a legal-specific model does not come from leading on a single metric, but from the coordination of four things: “domain knowledge + task contract + verifiable output + controllable cost.” Defining the evaluation criteria and output contract clearly first, then deciding the scale of training investment, is the order that is easiest to yield results and most easily overlooked in this kind of project.
