Vertical LLM Self-Development: From API Calls to Open-Source Foundation Models
This article explains why vertical AI companies are increasingly shifting from external closed-source API calls to building their own models on open-source foundation models. It covers the main business and technical reasons, including inference cost control, domain customization, vendor lock-in, and data compliance. It then describes the common post-training path—continued pre-training, supervised fine-tuning, preference alignment, and parameter-efficient fine-tuning—along with key challenges such as data quality, catastrophic forgetting, hallucination, evaluation, engineering infrastructure, inference optimization, and deployment. Finally, it provides practical guidance for teams considering this approach.
1. A High-Profile Switch Signals a Broader Industry Shift
A legal-technology AI company with a valuation of around ten billion dollars and a large customer base of leading law firms recently released its first in-house model, built by post-training an open-source large model. Because the company’s early product prototypes relied heavily on a leading model vendor’s closed-source model, and the two parties had even collaborated on legal corpora for specific jurisdictions, its decision to move to an open-source foundation drew significant attention.
This path is not an isolated case. Similar strategies have appeared in areas such as AI-assisted programming and local services, where companies pursue self-developed capabilities or cost optimization. The more important signal is not one company’s choice, but the broader trend it reflects: vertical AI is moving from “calling general capabilities” to “building its own model capabilities.” Understanding the underlying logic of this shift is more valuable than discussing a single event.
2. Why Vertical AI Wants to Build Its Own Models
Before looking at technical details, it is worth clarifying a basic question: why build a model in-house instead of continuing to call an external closed-source API?
- Controllable unit inference cost: Vertical products often have massive, high-frequency inference demand. Long-term usage-based pricing can grow linearly with scale. Self-development and inference optimization can significantly reduce marginal cost.
- Deep domain customization: General-purpose models often have limited understanding of specialized areas such as legal provisions, precedent reasoning, and industry terminology. Only by controlling the model itself can deeper domain adaptation be achieved.
- Reduced vendor lock-in: Binding core capabilities to a single external provider means that the product roadmap, pricing, and availability are constrained by that provider, creating clear business risk.
- Data sovereignty and compliance: Scenarios involving sensitive customer data often have strict requirements around data flow, deployment location, and auditability. Self-hosted or private deployment can make it easier to meet these requirements.
3. Four Decisive Reasons for Choosing an Open-Source Foundation
If a company decides to build its own model, why choose an open-source foundation instead of pre-training from scratch or continuing to use a closed-source model? The choice usually comes down to four practical trade-offs.
1. Better cost structure
Pre-training a foundation model from scratch requires compute and data resources that most companies cannot afford. An open-source foundation externalizes the most expensive part—learning general capabilities—so companies can focus on relatively controllable post-training costs while still obtaining domain capabilities.
2. Modifiability enabled by open weights
Closed-source models can only be accessed through APIs, and their internal weights and intermediate representations cannot be touched. Open-source models allow deeper modification, such as continued pre-training, full-parameter fine-tuning, and parameter-efficient fine-tuning. This is a prerequisite for achieving a meaningful leap in domain capability.
3. Deployment flexibility and clear data boundaries
Open-source models support private deployment, allowing data to remain within the organization’s boundary. This is especially relevant for industries with strict compliance requirements and also gives companies greater control over the performance and stability of the inference pipeline.
4. Ecosystem maturity and iteration pace
A mature open-source foundation usually comes with a well-developed toolchain and community feedback. This can shorten the path from selection to production deployment while leaving room to benefit from upstream model upgrades.
4. From a Base Model to a Domain-Owned Model: Key Technical Paths
After obtaining a general-purpose open-source foundation, how does a team turn it into a model that understands a specific domain? The industry commonly uses a phased, layer-by-layer alignment approach rather than trying to solve everything at once.
Stage 1: Continued pre-training (CPT)
The model continues next-token prediction on unlabeled domain corpora, such as legal documents and professional literature, so that it can first absorb the language distribution and knowledge of the domain. Key challenges include learning-rate control and corpus mixing. An imbalanced ratio can easily cause catastrophic forgetting—the model learns the new domain but loses general capabilities.
Stage 2: Supervised fine-tuning (SFT)
High-quality instruction-response pairs are constructed to teach the model to produce outputs that meet business requirements. This stage is highly sensitive to data quality: more data is not always better. The accuracy, consistency, and diversity of samples matter more than raw scale.
Stage 3: Preference alignment (RLHF / DPO-like methods)
Building on SFT, human preferences or rule-based reward signals are used to align model behavior, making it more suitable for business needs in helpfulness, accuracy, and refusal boundaries. For domains such as law, where it is often better to refuse than to answer incorrectly, the value of this alignment stage is especially clear.
Parameter-efficient fine-tuning (LoRA / QLoRA-like methods)
When compute resources are limited, low-rank adaptation can train only a small number of newly added parameters, significantly reducing memory and training costs. The following is an illustrative configuration that shows the role of key hyperparameters:
# Illustrative configuration: key hyperparameters for LoRA domain fine-tuninglora_config = {"rank": 16, # Low-rank dimension; controls the number of added trainable parameters"alpha": 32, # Scaling factor; affects the magnitude of weight updates"dropout": 0.05, # Regularization; helps mitigate overfitting"target_layers": [ # Usually applied to attention projection layers"query_proj", "key_proj", "value_proj", "output_proj"],"trainable_ratio": "approximately 0.1%-1% of total parameters", # Share of trainable parameters}
5. Core Challenges That Vertical Domains Cannot Avoid
The technical path may look clear, but practical difficulties remain, especially in fields with high accuracy requirements.
- Data quality and scarcity of domain corpora: High-quality, well-annotated domain data is hard to obtain, and cleaning, deduplication, and anonymization can be costly. Data quality often matters more than data volume when determining the model’s upper limit.
- Suppressing catastrophic forgetting: After post-training, a model may regress in general capabilities. Techniques such as corpus mixing, parameter-efficient fine-tuning, and regularization are needed to maintain balance.
- Hallucination and traceability: Specialized domains demand extremely high factual accuracy, and the cost of model fabrication can be unacceptable. In practice, fine-tuning is often combined with retrieval-augmented generation (RAG) so that generation is grounded in verifiable external knowledge.
- Establishing evaluation systems: Domain capability cannot be measured by general leaderboards alone. Teams need evaluation sets tailored to the business, covering dimensions such as accuracy, reasonable refusal, and citability, forming a sustainable regression-evaluation loop.
6. Engineering and Deployment: Making Models Usable in Production
Training a model well is only the first step. Stable service capability is equally important.
- Training infrastructure: Distributed training requires reasonable parallel strategies, such as data parallelism, tensor parallelism, and pipeline parallelism, as well as fault tolerance. In large-scale training, checkpoint resumption and memory management are routine challenges.
- Inference optimization: Quantization, batching, KV Cache management, and model parallelism can reduce latency and throughput costs. For high-frequency scenarios, the cost benefits of inference optimization often exceed those of training itself.
- Operational control: With general cloud capabilities—such as container platforms, object storage, logging services, and monitoring alerts—teams can achieve observability, elastic scaling, and rapid fault localization across training and inference pipelines. For example, Baidu AI Cloud provides general cloud product capabilities that can support model training, deployment, and operations, allowing teams to focus more on models and business themselves.
7. Practical Advice for Technical Selection
Overall, “building vertical models on open-source foundations” is not about chasing a trend. It is a rational trade-off based on cost, controllability, and compliance. Teams considering this path should keep the following points in mind:
- Calculate the economics first: Compare long-term API usage costs with the total cost of self-developed training and operations, and identify the break-even point before deciding.
- Prioritize data over algorithms: Focus most of the effort on building domain data and controlling its quality, because this is the foundation of model differentiation.
- Combine fine-tuning with retrieval: For accuracy-sensitive scenarios, use RAG to support factual grounding and fine-tuning to align expression.
- Build an evaluation loop: Set up a domain evaluation set from day one and let data drive each iteration decision.
- Preserve upgrade flexibility: When selecting a foundation, pay attention to its modifiability and ecosystem activity to avoid being locked into a single technology stack too early.
The core of in-house vertical model development is not simply whether open source is used. It is whether a team has the engineering capability to turn general capabilities into domain competitiveness. Thinking through this point is more important than chasing any single event.
