General-Purpose Language Models: Architecture Evolution and Engineering Practice

    This article examines the evolution of general-purpose language models and the engineering practices required to deploy them in enterprise environments. It reviews the shift from foundational pretraining to instruction-following and agentic systems, explains the principles of autoregressive blank-filling architectures, and discusses model-scale tradeoffs, instruction tuning, domain adaptation, long-context processing, retrieval-augmented generation, multimodal and tool-using systems, deployment and inference optimization, adaptation to domestically developed compute hardware, evaluation, security governance, and layered enterprise implementation. Readers gain a structured view of how model capabilities, data services, application orchestration, and governance must be combined to move language models from laboratory performance into sustainable production use.

    General-Purpose Language Models: Architecture Evolution and Engineering Practice

    1. Background of General-Purpose Language Models

    General-purpose language models are AI models that are pretrained on large-scale text, code, and other structured data to acquire capabilities in language understanding, content generation, knowledge organization, and logical reasoning. Unlike earlier classifiers and matching models trained for a single task, general-purpose language models typically use a unified pretraining objective to learn language patterns from large-scale data and parameters. They are then adapted to specific business tasks through instruction tuning, preference optimization, or domain adaptation.

    The development of these models has broadly passed through three stages.

    The first stage is foundational pretraining. Research focused on model architecture, training objectives, data scale, and distributed training methods. The goal was to enable models to learn lexical, syntactic, and semantic patterns as well as a certain amount of world knowledge.

    The second stage is dialogue and task generalization. Models began to be trained on instruction data so that they could understand natural-language instructions and perform tasks in forms such as question answering, summarization, translation, and code generation. At this point, usability depended not only on parameter scale but also closely on data quality, alignment methods, and context management.

    The third stage is the agent and multimodal stage. Models are no longer limited to processing plain text. They are gradually gaining capabilities in image understanding, document parsing, tool calling, task planning, and multi-step execution, allowing them to support information analysis and decision assistance in complex business processes.

    2. Core Idea of Autoregressive Blank-Filling Architecture

    Traditional autoregressive models usually predict the next token from left to right. Blank-filling pretraining instead randomly hides one or more spans in a text and asks the model to recover the masked content from context. Compared with unidirectional prediction, this method can use information both before and after the target position, so it adapts well to tasks such as text completion, paragraph rewriting, information extraction, and long-text understanding.

    Formally, the training process can be represented as:

    1. Input: The weather today is [blank], suitable for outdoor activities.
    2. Target: sunny

    The model must combine the preceding text, “The weather today is,” with the following text, “suitable for outdoor activities,” to infer the blank. In actual training, the length, position, and number of blank spans can vary according to data characteristics, improving the model’s ability to model different text structures.

    The advantages of this architecture are mainly reflected in three areas:

    1. More effective context utilization
      The model can refer both to information before the target span and to semantic cues after it. This bidirectional conditional information is valuable for text generation tasks that require semantic coherence.

    2. Better support for text-editing tasks
      In practical applications, users often need to modify part of a sentence rather than generate an entire paragraph from scratch. Blank-filling training enables models to perform continuation, rewriting, error correction, and local completion more naturally.

    3. Unified understanding and generation
      The same model can perform question answering, summarization, classification, code completion, and knowledge extraction through different input templates, reducing the cost of designing separate models for each task type.

    3. Model Scale and Capability Boundaries

    Model parameter scale usually affects knowledge capacity, expressive ability, and complex task handling, but parameter count is not the only measure of model quality. Data quality, training stability, context length, inference efficiency, and alignment quality are equally important.

    Lightweight models offer low deployment cost, fast response, and controllable data handling. They are suitable for edge devices, internal systems, and high-concurrency scenarios. Medium-scale models strike a better balance between capability and cost and are suitable for enterprise knowledge question answering, content moderation, coding assistance, and office automation. Large-scale models are better suited to complex reasoning, long-document analysis, and multi-step task planning, but they usually require higher investment in compute, storage, and operations.

    When selecting a model, the following dimensions can be used for evaluation:

    • Task complexity: Simple classification does not necessarily require a large model, while complex reasoning requires stronger model capabilities.
    • Context length: Legal documents, technical manuals, and meeting records often require a longer input window.
    • Response latency: Online interactive scenarios should focus on time to first token and overall generation speed.
    • Data security: When sensitive data is involved, private deployment or isolated runtime environments should be prioritized.
    • Unit cost: Inference resources, storage, network, and operations costs should be calculated together.
    • Controllability: Enterprise applications need support for output constraints, content filtering, auditing, and version management.

    4. Instruction Tuning and Domain Adaptation

    Foundation models pretrained on general data usually have strong general capabilities, but they may not directly meet enterprise business requirements. Instruction tuning uses samples consisting of a task instruction, an input, and a target output to help the model learn a specific interaction pattern. For example, in a customer-service scenario, training data can require the model to identify user intent, extract key information, and generate a standardized reply in a fixed format.

    Domain adaptation generally includes the following steps:

    1. Define task boundaries
      First determine the problem the model needs to solve, such as knowledge question answering, contract summarization, ticket classification, or code review. Avoid making the training objective too broad.

    2. Build a high-quality dataset
      Data should cover common questions, edge cases, and abnormal inputs. Rather than blindly increasing the number of samples, priority should be given to handling duplicate, incorrect, outdated, and inconsistently formatted data.

    3. Standardize output specifications
      For results that need to enter business systems, fields, formats, lengths, and refusal conditions should be specified to reduce downstream parsing difficulty.

    4. Evaluate and iterate
      Use offline test sets, manual review, and online feedback to continuously identify problems, with emphasis on factual accuracy, format stability, refusal appropriateness, and long-text consistency.

    5. Long-Text Processing and Retrieval-Augmented Generation

    Long-text capability is not merely about expanding the context window. Models also need to address information localization, cross-paragraph association, key-point extraction, and factual consistency. When facing large numbers of enterprise documents, directly feeding all content into a model often leads to high cost, slow response, and dilution of key information.

    Retrieval-augmented generation is a common approach. Its basic workflow includes:

    1. Document collection
    2. ↓
    3. Cleaning and segmentation
    4. ↓
    5. Vectorization or keyword indexing
    6. ↓
    7. Recall of relevant chunks
    8. ↓
    9. Reranking and context assembly
    10. ↓
    11. Model answer generation
    12. ↓
    13. Citation validation and result output

    During implementation, segmentation strategy requires attention. Segments that are too short can break semantic integrity, while segments that are too long can reduce retrieval precision. For technical documents, segmentation can follow chapters, headings, and code blocks. For policy documents, clause numbers, scope of application, and exception conditions should be preserved as much as possible.

    In addition, retrieval results should not all be passed directly to the model. They should be deduplicated, ranked, and filtered for relevance. For high-risk business scenarios, the model should also be required to provide supporting snippets and to explicitly refuse when no reliable basis can be found, avoiding the presentation of speculation as fact.

    6. Multimodal and Agent Capabilities

    As application scenarios expand, model inputs are no longer limited to text. They may also include images, tables, scanned documents, flowcharts, and speech transcription results. A multimodal system usually consists of three parts: data parsing, unified representation, and task reasoning.

    For example, in receipt or invoice processing, the system first needs to perform image recognition, then extract dates, amounts, and entity information, and finally validate the results against business rules. This involves perception problems as well as structured understanding and business reasoning. It cannot simply rely on a single text-generation step.

    Agent systems further introduce tool calling and task orchestration. A relatively complete execution flow usually includes:

    • Understanding the user’s goal;
    • Breaking it down into multiple subtasks;
    • Determining whether retrieval, computation, or business systems need to be called;
    • Executing tools and checking returned results;
    • Adjusting subsequent plans based on intermediate results;
    • Aggregating results and outputting auditable process information.

    Tool calling must be subject to permission control. A model should not directly have unrestricted system access. Risk should be limited through whitelists, parameter validation, approval steps, and operation logs.

    7. Deployment, Inference, and Adaptation to Domestically Developed Compute Hardware

    Putting a model into production is not simply a matter of downloading weights and starting a service. It also involves engineering issues such as model format conversion, memory planning, concurrency control, fault recovery, and security isolation.

    Common optimization methods include quantization, batching, cache reuse, and distributed inference. Quantization can reduce storage and computation costs, but it may affect complex reasoning, code generation, and long-text tasks to some extent, so it needs to be validated against actual data. Batching is suitable for offline tasks with high request volume, while interactive applications place more emphasis on dynamic batching and low-latency scheduling.

    In environments based on domestically developed compute hardware, attention should focus on operator compatibility, compiler toolchains, driver versions, memory bandwidth, and multi-card communication efficiency. Before model deployment, a complete validation process should be established, including model loading, basic inference, long-text inference, concurrency stress testing, exception recovery, and result consistency testing. Adaptation should not be judged complete based on a single successful run.

    8. Evaluation Systems and Security Governance

    Model evaluation should cover capability, efficiency, security, and stability. General capability evaluation can focus on knowledge understanding, logical reasoning, mathematical calculation, code generation, and text summarization. Business evaluation should use real or de-identified samples to verify whether the model meets specific process requirements.

    Security governance should include at least the following:

    • Identify and redact sensitive information in inputs;
    • Perform content safety checks on outputs;
    • Set refusal or manual review rules for high-risk requests;
    • Retain model versions, prompts, retrieval content, and tool-calling logs;
    • Provide sources or evidence for key conclusions;
    • Establish mechanisms for issue feedback, rollback, and version tracking.

    In scenarios such as finance, healthcare, government services, and industrial control in particular, models should be positioned as assistive systems rather than unreviewed final decision-makers. For operations that may cause actual loss, human confirmation must be retained.

    9. Implementation Recommendations for Enterprises

    When building general-purpose language model applications, enterprises can adopt a layered architecture consisting of a model capability layer, a data service layer, a business orchestration layer, and a governance and operations layer.

    The model capability layer handles text generation, understanding, vector representation, and multimodal parsing. The data service layer handles document management, retrieval, permissions, and data updates. The business orchestration layer handles process control, tool calling, and result validation. The governance and operations layer handles monitoring, auditing, evaluation, cost accounting, and security management.

    A recommended approach is to proceed through small-scope pilots, metric validation, and gradual expansion. First select scenarios with clear boundaries and easily quantifiable benefits, such as internal knowledge retrieval, meeting-minutes summarization, and repetitive document generation. Then use metrics such as accuracy, response time, labor savings, and user satisfaction to decide whether to expand the scope.

    Overall, competition in general-purpose language models has gradually shifted from simply pursuing parameter scale toward data quality, inference efficiency, tool collaboration, deployment stability, and governance capability. Only by combining models, data, compute, application processes, and security mechanisms can language capabilities in the laboratory be transformed into sustainable operational productivity.