Open-Source Large Language Models: Technical Architecture and Engineering Deployment Practices

    This article examines open-source large language models from an engineering perspective. It explains how to assess openness by model weights, code, training data, and training recipes; outlines mainstream Transformer-based architectures, including decoder-only designs and mixture-of-experts models; describes the training pipeline from pre-training through supervised fine-tuning and preference alignment; reviews inference memory planning and service optimizations; and provides practical guidance for model selection, parameter-efficient fine-tuning, deployment, observability, risk management, and compliance. It also outlines emerging trends in open-weight models and cloud-hosted inference.

    1. First Clarify the Concept: Openness Levels Matter More Than Labels

    When discussing open-source large language models, the first step is to correct a common misconception: being able to download weight files does not necessarily mean a model is open source. The openness of a model system is usually evaluated across four dimensions:

    Openness dimension Details Engineering value
    Model weights Parameter files can be downloaded and loaded locally for inference Data can remain within controlled environments; offline deployment is possible
    Code assets Forward/backward propagation, training scripts, and inference scripts are publicly available Supports reproducibility and further modification
    Training data Corpus sources, mixture ratios, cleaning rules, and deduplication rules Supports auditing and control of bias and copyright risks
    Training recipe Hyperparameters, learning-rate schedules, data ordering, and compute budgets Supports reproducibility and sustainable iteration

    Based on these dimensions, model openness can be grouped into three levels:

    • Open weights: Parameters and inference code are released, while data and training recipes are not. This is the most common form in engineering practice and is sufficient for private deployment and fine-tuning.
    • Open code and weights: Training scripts are released together with weights, enabling reproducibility, although the corpus still needs to be organized independently.
    • Fully open: Corpus, training recipe, and evaluation details are all released. This provides the strongest auditability, but also the highest engineering difficulty and cost.

    For enterprise decision-makers, the real questions are not “Does it claim to be open source?” but “Does the license permit my use case?”, “Can the weights run in my data center or dedicated instances?”, and “Is the layer I need to modify open?”. Answering these questions completes most of the selection process.

    2. Technical Architecture: Three Routes Built on Transformer

    The mainstream open-source large models are built on Transformer backbones, but their architectures have evolved through several generations. Understanding these structural choices directly affects inference cost and deployment density.

    2.1 Three Architecture Routes

    • Encoder-only route: Well suited to understanding tasks such as classification, retrieval, and extraction, with relatively low inference cost and stable latency.
    • Encoder-decoder route: Balances understanding and generation and remains valuable in sequence-to-sequence scenarios such as translation and summarization.
    • Decoder-only route: Uses autoregressive modeling to unify various tasks, has the most mature ecosystem and toolchain, and is currently the dominant form of open-source models.

    2.2 Key Structural Evolution

    1. Transformer Block (Decoder-Only, modern variant)
    2. ├── Pre-Norm normalization # Improves training stability and supports larger scale
    3. │ └── RMSNorm-style implementation # Removes mean centering and reduces operator overhead
    4. ├── Self-attention sublayer
    5. │ ├── RoPE rotary position encoding # Supports long-context extrapolation
    6. │ ├── GQA grouped-query attention # Compresses the KV cache and significantly improves long-sequence throughput
    7. │ └── FlashAttention-style kernels # Uses chunked computation to reduce peak memory usage
    8. ├── Feed-forward sublayer
    9. │ └── SwiGLU-style gated activation # Generally more expressive than a traditional ReLU FFN
    10. └── Sparsification options
    11. └── MoE mixture of experts # Large total parameters but small activated parameters; decouples compute from capacity

    Among these, mixture of experts (MoE) is one of the most important structural shifts in recent years. The total parameter count of a model can be expanded to the trillions, but each token activates only a small number of experts. As a result, the compute required for a single forward pass remains manageable. This explains why “parameter count” and “inference cost” are not linearly related at the current stage.

    2.3 Model Families: One Large Model Is Not a Single Model

    Modern open-source systems are usually not a single set of weights. Instead, they are families of models divided by parameter scale: small versions for edge scenarios, medium-sized general-purpose versions, flagship versions, and lightweight variants produced through quantization and distillation. In engineering selection, they should be treated as a capability gradient rather than a single product, and matched by scenario layer by layer.

    3. Training Pipeline: Four Stages from Pre-training to Preference Alignment

    1. [Stage 1] Large-scale pre-training
    2. Goal: learn world knowledge and language patterns
    3. Key: corpus cleaning, deduplication, data mixture, and contamination control (detecting evaluation-set leakage)
    4. ↓
    5. [Stage 2] Supervised fine-tuning (SFT)
    6. Goal: align with instruction formats and learn to follow instructions
    7. Key: diversity and quality of instruction data matter more than quantity
    8. ↓
    9. [Stage 3] Preference alignment (RLHF / direct preference optimization methods)
    10. Goal: align with human preferences and suppress harmful or redundant outputs
    11. Key: reward model quality and noise control in preference pairs
    12. ↓
    13. [Stage 4] Capability enhancement (retrieval augmentation / tool calling / reasoning distillation)
    14. Goal: go beyond parametric knowledge boundaries and improve complex reasoning

    It is worth emphasizing that Stage 1 determines the upper limit of a model’s capability, while Stages 2 and 3 determine how well it performs in your business. Most enterprise teams should focus their practical investment on Stages 2 and 4 — this is also where parameter-efficient fine-tuning delivers the greatest value.

    4. Development Timeline: From the Foundational Period to the Trillion-Parameter Era

    Viewed as a timeline, the evolution of open-source large models can be summarized in four stages:

    1. Foundational period (2018–2019): As the Transformer paper spread widely, understanding-oriented and generation-oriented routes moved toward openness separately, establishing the basic paradigm of “pre-training + fine-tuning” and making “downloadable weights” normal for the first time in the research community.
    2. Expansion period (2020–2022): The decoder-only route gradually became mainstream and parameter counts rose quickly. However, open weights for high-performance models were still scarce, and the capability gap between open and closed models was obvious.
    3. Open-weight wave (2023 onward): A group of models with performance close to contemporaneous closed-source models was released as open weights, often with relatively permissive commercial licenses. Open-source and closed-source models began to compete substantively for the first time.
    4. Scale and ecosystem development (2024 to present): Sparse architectures, ultra-long context, multimodal integration, and model-family matrices became standard configurations, and open-source models became common in production environments.

    Several landmark changes in 2026 are worth noting:

    • Several major vendors in China released new open-source models covering languages, code, multimodal capabilities, and reasoning-focused directions;
    • Open-weight models with parameter counts reaching the trillions appeared, validating the engineering feasibility of large-scale sparse models;
    • Several international technology companies and research institutions jointly issued an open letter calling for broader open-weight model release under the premise of manageable security and misuse risks;
    • National research institutions released new open-source model families, promoting the development of Chinese-language corpora and localized evaluation systems;
    • Managed model hosting services on major cloud platforms in China began onboarding locally developed open-source model families at scale, enabling dual-channel delivery through open weights and managed services.

    The essence of this stage is: open source is no longer merely a supplement to closed source, but an independent supply system that operates in parallel with closed source. Within the Chinese cloud ecosystem, major platforms such as Baidu AI Cloud’s model development and serving platform also commonly provide hosting, fine-tuning, and inference acceleration for open-source models, reducing the distance between “obtaining weights” and “running services” for enterprises.

    5. Inference Engineering: Calculate GPU Memory Before Calculating Cost

    The first threshold for deploying an open-source model is GPU memory. A practical estimation method is as follows:

    1. # Inference memory estimation (illustrative; units: GB)
    2. def estimate_inference_memory(
    3. num_params_b: float, # Number of parameters in billions (B)
    4. bytes_per_param: float, # Bytes per parameter
    5. num_layers: int,
    6. num_kv_heads: int,
    7. head_dim: int,
    8. seq_len: int,
    9. batch_size: int,
    10. ) -> float:
    11. """Return an approximate per-GPU memory requirement for capacity planning."""
    12. # 1) Weight memory
    13. weights = num_params_b * 1e9 * bytes_per_param
    14. # 2) KV cache memory (often the dominant cost for long contexts)
    15. kv_cache = (
    16. 2 # K and V
    17. * num_layers
    18. * num_kv_heads
    19. * head_dim
    20. * seq_len
    21. * batch_size
    22. * bytes_per_param
    23. )
    24. # 3) Runtime buffers, activations, and framework overhead
    25. overhead = weights * 0.10
    26. return round((weights + kv_cache + overhead) / 1e9, 2)
    27. # Example: 0.5 bytes per parameter represents 4-bit quantized weights
    28. print(estimate_inference_memory(
    29. num_params_b=7, bytes_per_param=0.5,
    30. num_layers=32, num_kv_heads=8, head_dim=128,
    31. seq_len=8192, batch_size=16,
    32. ))

    From this formula, two main optimization paths can be seen: reduce bytes per parameter (FP16 → BF16 → INT8 → INT4 quantization) and compress the KV cache (GQA/MLA-style structures, paging management, and prefix reuse).

    Further service-level optimization techniques include:

    • Continuous batching: dynamically batches requests at the token level to significantly improve GPU utilization;
    • Prefix caching: computes common prefixes such as system prompts and few-shot examples only once;
    • Speculative decoding: uses a small model to draft and a large model to verify, reducing decoding latency;
    • Parallel strategy combinations: tensor parallelism addresses cases where a single GPU cannot hold the model, pipeline parallelism improves cluster efficiency, and expert parallelism is specifically used for MoE;
    • Combining speculation and quantization: requires attention to potential accuracy regression caused by quantization, and regression evaluation must be performed before production.

    6. Enterprise Implementation Methodology

    6.1 Selection Evaluation Matrix

    Evaluation dimension Key question Recommended action
    License Does it permit commercial use? Are there additional usage restrictions? Involve legal teams early and document an obligations checklist
    Capability benchmark How does it perform on your own business data? Build an evaluation set with 200–500 business samples
    Context window Does it cover long-document and long-conversation scenarios? Stress-test with the longest real samples
    GPU memory and throughput Can it run on a single GPU? What is the peak QPS? Run small-scale load tests before finalizing hardware choices
    Fine-tuning friendliness Does it support parameter-efficient fine-tuning? Verify toolchain completeness
    Ecosystem maturity Documentation, community support, and release cadence Prefer model families with active iteration

    Key principle: Do not use public leaderboard scores as the basis for selection decisions. Always perform secondary validation with your own business evaluation set. Leaderboards carry risks of data contamination and task mismatch, while the business evaluation set is the only reliable judge.

    6.2 Parameter-Efficient Fine-Tuning Configuration Example

    Most enterprise scenarios do not require full-parameter fine-tuning. Low-rank adaptation methods can often complete domain adaptation on a single GPU or a small number of GPUs:

    1. # Parameter-efficient fine-tuning configuration (illustrative)
    2. model:
    3. load_in_4bit: true # Load base weights in 4-bit to reduce memory usage to 1/4
    4. gradient_checkpointing: true # Trade computation for memory to support longer sequences
    5. adapter:
    6. type: lora
    7. rank: 16 # Higher rank increases capacity but also memory usage
    8. alpha: 32
    9. dropout: 0.05
    10. target_modules: # Usually injected into linear layers in attention and FFN modules
    11. - q_proj
    12. - k_proj
    13. - v_proj
    14. - o_proj
    15. training:
    16. epochs: 2 # With high-quality data, fewer epochs are often better than more
    17. learning_rate: 1.0e-4
    18. batch_size: 4
    19. gradient_accumulation_steps: 8
    20. max_seq_len: 4096
    21. dataset:
    22. format: instruction # Instruction-input-output three-part format
    23. min_samples: 1000 # Below this level, expand the data first
    24. dedup: true # Deduplication is often the most cost-effective step
    25. evaluation:
    26. holdout_ratio: 0.1
    27. metrics:
    28. - exact_match
    29. - llm_as_judge_score # Use a fixed judge model to avoid evaluation drift
    30. - latency_p95

    Supporting data governance requirements include: strict separation between training and evaluation sets, desensitization of sensitive information, and one-to-one correspondence between data versions and model versions. Without these three points, fine-tuning results will be difficult to reproduce, and attribution will be difficult when problems occur.

    6.3 Deployment Options

    • Private deployment: Data stays within controlled environments, offering the strongest compliance, but with the highest operational cost;
    • Dedicated inference instances: Resource isolation and on-demand elasticity, balancing cost and compliance;
    • Elastic serverless inference: Handles traffic peaks and valleys, but cold-start latency must be considered;
    • Hybrid architecture (recommended): Small models handle high-concurrency simple requests, large models handle complex requests, and a routing layer distributes traffic by difficulty. This is currently the most effective approach for cost reduction.

    6.4 Evaluation and Observability

    Production systems must establish a complete metrics dashboard:

    1. Performance metrics: TTFT (first-token latency), TPOT (per-token latency), throughput (tokens/s), failure rate
    2. Quality metrics: human sampling pass rate, hallucination rate, refusal rate, format compliance rate
    3. Cost metrics: cost per 1K tokens, average cost per request, GPU utilization

    At the same time, enable full-link request logging and tracing, retain sampled records of prompts and outputs (with desensitization), and set up automatic alerts. A large-model service without observability is effectively launched as a black box.

    7. Risks and Compliance: Four Pitfalls to Address Early

    1. License differences. Permissive licenses and models with additional usage restrictions differ significantly in terms of commercial use, distribution, and attribution obligations. For derivative models trained based on open weights, the scope of open-source obligations must be confirmed in advance.
    2. Transfer of safety-alignment responsibility. Using open weights means that some responsibility for safety alignment shifts to the user. Input filtering, output review, and jailbreak protection must be implemented independently; one cannot assume that the base model is sufficiently safe.
    3. Weight integrity verification. When obtaining weights from public channels, always verify hash values and pin version sources to prevent supply-chain contamination.
    4. Content compliance and auditability. Public-facing services need to retain traceable generation records and meet the generative content compliance requirements of the region in which they operate.

    8. Trend Predictions

    Based on current technical directions, four predictions can be made:

    • Sparsification will become standard, further decoupling total parameter count from single-inference cost. Trillion-scale open models will move from exceptions to regular cases;
    • Openness will move to lower layers: more teams will release not only inference code, but also training recipes and even synthetic data pipelines. Reproducibility will become a new dimension of competition;
    • Small models will gain value: distillation and quantization will allow edge and device-side models to handle more real traffic, and cloud-edge collaborative inference architectures will become mainstream;
    • Managed hosting and open weights will converge: enterprises can either deploy models themselves or quickly access them through managed hosting on cloud platforms. The two paths will coexist rather than exclude each other.

    For engineering teams, the real threshold for open-source large models has never been “whether weights can be obtained,” but whether a complete engineering loop can be built across selection, fine-tuning, deployment, evaluation, and compliance. Whoever completes this loop first will maintain cost advantages and iteration speed as models continue to evolve rapidly.