General-Purpose LLM Architecture Evolution and Production Practice

    This article explains how general-purpose language models are positioned, how pretraining objectives and architectures evolved, and how to select, deploy, evaluate, and govern them in production. It covers GLM-, GPT-, and T5-style training routes, decoder and MoE designs, scaling and routing choices, five-layer application architecture with fallbacks, and a business-specific evaluation template.

    1. Scope of General-Purpose Language Models

    General-purpose language models are pretrained on large text corpora to acquire language understanding and generation. Unlike models trained for a single task, they transfer well: with a few prompts, examples, or instructions they can handle Q&A, summarization, translation, code generation, information extraction, and rewriting.

    This article focuses on three questions: how underlying architectures evolved, how to choose a model given tasks and data, and how to integrate models into systems that can be evaluated and governed. It does not rank specific versions, because rankings shift with benchmarks and release dates.

    2. Three Pretraining Objectives

    The pretraining objective determines what the model learns and what it is better at. Three common routes are:

    Autoregressive blank filling, used by the GLM family. Segments of text are replaced with blank tokens, and the model generates the masked content in order while using context on both sides. It combines bidirectional understanding with autoregressive generation, and fits completion, rewriting, and structured editing.

    Decoder next-token prediction, used by the GPT family and most chat models. The model predicts the next token from left context only. Training and inference share the same form. This approach has extensive practice in open-ended generation, multi-turn dialogue, and instruction following, and underpins most public general chat models.

    Span corruption and reconstruction, used by T5 and BART. Spans in the input are corrupted or masked, and the model reconstructs the original text. It is mature for conditional generation such as summarization and translation, but usually needs an encoder–decoder pair, so deployment differs from decoder-only models.

    Route Representative models Training objective Main strength Typical use
    Autoregressive blank filling GLM family Recover masked spans Strong use of bidirectional context Completion, rewriting, structured editing
    Decoder next-token prediction GPT family and most chat models Predict the next token from the left Unified train/infer; mature generation and dialogue Open Q&A, multi-turn chat, general generation
    Span corruption and reconstruction T5, BART Reconstruct original text from corrupted input Stable conditional generation Summarization, translation, controlled generation

    The objective alone does not determine quality. Data quality, instruction tuning, alignment, and inference settings also matter. Do not choose a route because it “sounds more advanced”; match the task shape and deployment constraints.

    3. Four Architecture Directions

    From pretraining to products, architecture work has concentrated on four directions.

    Decoder-only stacks as a common base for general generation. They are simple to implement, share one compute path for training and inference, and scale to large parameter counts and multi-task instruction training.

    Mixture of Experts (MoE) to cut compute per inference. Multiple expert subnetworks exist; each token activates only a few. Total parameters can be large while activated parameters stay smaller. Routing adds training complexity, and unbalanced expert load can hurt efficiency.

    Attention and position encoding to reduce memory and latency. Grouped-query attention (GQA) shares key/value heads across query heads and shrinks the KV cache. Rotary position embeddings (RoPE) encode relative order and help later length extrapolation.

    Long-context extension to increase input length. Common methods include position interpolation, continued pretraining, and instruction tuning on long documents. Longer context does not automatically yield stable long-document skill; test cross-paragraph association on long-document sets.

    These changes often appear together. When reading an architecture note, identify which bottleneck it targets—training cost, latency, memory, or context length—and whether that bottleneck matters for your workload.

    4. Scale and Model Selection

    Small models fit local, edge, and high-concurrency settings. Mid-size models balance cost and capability. High-capacity models suit complex reasoning, long-document analysis, and high-quality writing.

    Choose scale by task difficulty. Real-time support, batch classification, and simple extraction often work with smaller models (faster, cheaper). Code generation, hard Q&A, and technical-document analysis usually need mid or large models. Multi-step reasoning and cross-document synthesis need extra checks on reasoning, long-context stability, and output consistency.

    A practical sequence:

    1. Split business tasks into simple, medium, and complex.
    2. Run the same sample set on candidates; record accuracy, latency, and cost per call.
    3. Pick the smallest model that meets the pass line for each class.
    4. Route requests by difficulty at the orchestration layer.

    5. Typical Application Architecture and Fallback

    A production LLM app usually has five layers. Each should have a minimal implementation and a failure fallback.

    Interaction layer takes input, shows output, and manages sessions. Minimum: validate length and format; drop clearly invalid requests. Fallback: on timeout, return a preset message and keep the user input for retry.

    Orchestration layer splits work into classify, retrieve, generate, validate, and format. Minimum: a fixed pipeline with logged I/O per step. Fallback: skip optional steps; if retrieval is empty, ask the user for more information instead of letting the model invent an answer.

    Model serving layer unifies calls, routing, timeouts, retries, and concurrency. Minimum: timeout plus at most two retries per model, plus caching of identical requests. Fallback: switch to a backup model and log the switch.

    Knowledge and data layer covers document stores, business DBs, and retrieval indexes. Minimum: chunk documents, build a vector index, and attach source snippets in the prompt. Fallback: if retrieval is down, use cached FAQs and tell the user the answer may be incomplete.

    Safety and governance layer covers auth, access control, content review, sensitive-data handling, and audit. Minimum: restrict data sources by role and log retrieval and generation. Fallback: if review is unavailable, pause external output and keep internal debug only.

    6. Deployment Modes and Checklist

    Three common modes: online API, private deployment, and hybrid.

    Dimension Online API Private deployment Hybrid
    Data isolation Data processed by an external service Data stays internal Split by sensitivity
    Elasticity Scales with traffic You plan capacity Elastic for ordinary tasks, fixed for sensitive ones
    Ops cost Low; no inference stack to run High; you own serving and upgrades Medium; two paths to maintain
    Compliance Review the vendor’s data terms Fits stricter internal rules Audit the routing rules
    Fit Fast validation, general Q&A, low-sensitivity work Confidential data, strong isolation Mix of general and sensitive tasks

    Checklist:

    • Must sensitive data stay on the internal network? If yes, use private deployment.
    • Does ordinary traffic fluctuate a lot? If yes, prefer online APIs.
    • Do hard tasks need a stronger model? If yes, route them at orchestration.
    • Can you operate inference and version upgrades? If not, avoid full private deployment.
    • Is cost measured per task? If not, add billing monitoring before you scale.

    7. Evaluation Template and Test-Set Design

    Do not rely only on leaderboards or a few examples. Build a business test set and log results with a fixed template.

    Suggested fields per case:

    • Task type: Q&A, summary, extraction, code generation, etc.
    • Input: full user request and needed context.
    • Expected output: gold answer, required facts, or format.
    • Scoring rules: fact correctness, missing requirements, format compliance.
    • Pass line: e.g. key-fact accuracy ≥ 90%, or a minimum score per item.
    • Human sample rate: e.g. 15–20% of auto-scored items for review.

    Build the set in four parts:

    1. Normal samples from real traffic, covering main task types.
    2. Boundary samples: very long input, missing information, vague wording, conflicting constraints.
    3. Adversarial samples: prompt injection, unauthorized queries, attempts to leak internal data.
    4. Historical failures: tickets or logs that previously failed or drew complaints.

    Rerun the full set after every model, prompt, or retrieval change, and compare pass rates by slice. Automatic metrics scale; human review finds quality, business sense, and hidden risk.

    8. Common Issues and Pre-Launch Checks

    Three frequent problems:

    First, models produce plausible but false information. Policy, contract, medical, financial, and production-config content must be checked against authoritative sources, with a human review step.

    Second, ever-longer prompts do not fix every problem. Hard tasks are better solved with retrieval, stepwise reasoning, tool use, and output validation.

    Third, long chat history buries key facts and slows responses. Compress history regularly and store preferences, task state, and confirmed facts separately.

    Pre-launch checklist:

    • Access: each user class reaches only authorized data sources.
    • Audit: retrieval, generation, and human edits are traceable.
    • Fallback: model, retrieval, and review outages have defined substitutes.
    • Human takeover: abnormal or high-risk requests can go to people.
    • Sensitive data: logs and prompts must not store secrets or personal identifiers.
    • Cost: dashboards for volume, latency, failure rate, and cost per task.

    9. Conclusions

    Three takeaways:

    First, choose architecture by task shape. Completion and rewriting can use blank filling; open dialogue and general generation fit decoder-only models; summarization and conditional generation can use span-reconstruction models.

    Second, choose deployment by data sensitivity. Keep sensitive data internal, use online services for ordinary work, and route hard requests to stronger models.

    Third, evaluate on a business test set. Cover normal, boundary, adversarial, and historical cases, and revalidate after every change. That process turns model capability into maintainable application value.