Operating System-Level Pluggable AI Model Architecture: Design and Implementation

1. Background: From a Single Built-In Model to Pluggable Models

Early operating systems commonly integrated intelligent capabilities through a “single vendor, deeply coupled” model. Features such as voice assistants, text generation, and image recognition depended on fixed backend models, resulting in tight coupling between the system layer and the model layer. This approach could deliver capabilities quickly in the early stage, but it had three clear limitations:

• The capability ceiling was constrained by a single model, making it difficult to evolve rapidly with technological progress.
• Vendor lock-in reduced flexibility and negotiating leverage.
• Differences in compliance requirements and network conditions across regions and scenarios were difficult to accommodate.

In recent years, operating system vendors have generally shifted toward a platform-based distribution approach. Instead of developing every large model capability in-house, the system provides a compatible, secure, and controllable AI service distribution platform. Within an authorized scope, end users can connect to and switch between mainstream third-party models to power system-level features. The essence of this shift is to move “model competition” from inside the system to the ecosystem layer, using ecosystem collaboration to compensate for temporary gaps in general-purpose large model development.

2. Core Design: Layered Abstraction and a Unified Interface

To allow users to freely select and switch models, the first requirement is a stable abstraction layer that decouples system features from specific models. A reasonable layered structure is as follows:

+-------------------------------------------------------------+
|                  System-Level Feature Callers               |
| Voice assistant, writing assistance, image understanding,   |
| search, and other system-level features                     |
+-------------------------------------------------------------+
|              Unified Capability Abstraction Layer           |
| Text generation / Speech recognition /                      |
| Multimodal understanding / ...                              |
+-------------------------------------------------------------+
|                   Model Adapter Layer                       |
| Provider A      Provider B      Provider C      ...         |
+-------------------------------------------------------------+
|             Model Routing and Configuration Layer           |
| Capability negotiation / Fallback / Privacy policy /        |
| Billing                                                     |
+-------------------------------------------------------------+

The key is to define a set of capability interfaces that are independent of any specific vendor. System features only declare what capability they need—for example, “a text completion” or “a speech transcription”—without caring which model provides it. The adapter layer normalizes request and response formats from different vendors into a unified interface.

The following is an illustrative interface definition:

// Capability abstraction independent of specific model providers
protocol TextCompletionProvider {
    var id: String { get }
    var supportedFeatures: FeatureSet { get }
    func complete(
        prompt: String,
        options: GenerationOptions,
        handler: @escaping (Result<String, ModelError>) -> Void
    )
}
// Normalized options that abstract away provider-specific parameter differences
struct GenerationOptions {
    var maxTokens: Int?
    var temperature: Double?
    var locale: Locale
    var privacyLevel: PrivacyLevel   // local-first / anonymized cloud / authorized cloud
}

The privacyLevel field is particularly important. It elevates privacy policy to a first-class concern, enabling the routing layer to select on-device inference or cloud-based invocation according to user authorization and compliance requirements.

3. Model Routing and Switching Mechanisms

With a unified abstraction in place, the next step is to implement two actions: selection and switching. The routing layer needs to address the following problems:

1. Capability negotiation: When a user selects a model as the default driver, the system queries its supportedFeatures. If the selected model does not support a certain feature—for example, speech—the system should be able to fall back to another model with complete capabilities or clearly notify the user.

2. Priority and fallback: Define the order of the primary model and backup models. When the primary model becomes unavailable—due to network interruption, quota exhaustion, or response timeout—the system should automatically degrade to a backup model to maintain continuity of system features.

3. Immediate effect of switching: When a user switches models, the new configuration should take effect at the next feature invocation without requiring an application or service restart.

An example configuration is as follows:

{
  "defaultModel": "provider_a_v2",
  "fallbackChain": ["provider_b_v1", "provider_c_v1"],
  "routingRules": [
    { "capability": "vision", "preferred": "provider_c_v1" },
    { "capability": "tts",    "preferred": "provider_b_v1" }
  ],
  "privacyPolicy": "prefer_local"
}

By combining a default model, a fallback chain, and capability-level routing rules, the system can satisfy user customization requirements while maintaining robustness in complex network environments and multi-model ecosystems.

4. Security and Controllability: The Baseline for Platformization

Connecting third-party models to system-level features naturally expands the security boundary. The platform needs to establish protections at several levels:

• Data minimization: Send only the minimum context required to complete a task to an external model. Sensitive information should be desensitized locally before transmission.

• Local-first policy: For scenarios with high privacy sensitivity, such as those involving contacts or message content, prioritize on-device models or locally cached results. When necessary, explicit user authorization should be obtained.

• Capability isolation and permission control: Calls from system features to models should go through a unified gateway that records the call source, frequency, and data flow, facilitating auditing and rate limiting.

• Content safety filtering: Validate both input and output for compliance to prevent unauthorized data leakage and inappropriate content generation.

The following is a deployable pseudocode example for an invocation gateway:

def invoke_model(feature, payload, user_consent):
    # 1. Compliance and privacy validation
    payload = desensitize(payload, level=user_consent.privacy_level)
    assert user_consent.allows(feature.data_category)
    # 2. Select a model based on routing rules and the fallback chain
    provider = router.select(feature.capability, privacy=user_consent.privacy_level)
    # 3. Record an audit log entry (metadata only, no plaintext)
    audit.log(feature=feature.id, provider=provider.id, ts=now())
    # 4. Invoke the model with unified exception handling
    try:
        return provider.complete(payload)
    except ModelUnavailable:
        return router.fallback(feature, payload)

5. Engineering Strategy: Build a Strong Foundation Before Adding Intelligence

A commonly overlooked but critical lesson is that introducing AI features should not come at the cost of degrading the fundamental quality of the operating system. A mature approach is to follow an “optimize first, then extend” release strategy: prioritize underlying performance work such as system stability, memory and power management, and startup and response speed, and then steadily add AI capabilities on top of that foundation.

The reason is straightforward: AI features often introduce additional compute usage, memory overhead, and network requests. If the foundation itself is inefficient, adding AI capabilities can easily lead to overheating, stuttering, and excessive power consumption. A reasonable progression includes:

• Performance budgeting: Define clear CPU, memory, power, and latency budgets for AI features. Models or features that exceed these budgets should not be enabled by default.

• On-demand loading: Load model capabilities on demand and avoid keeping them resident in memory for long periods, reducing their impact on the base user experience.

• Device-cloud collaboration: Handle simple tasks with small on-device models and reserve large cloud models for complex tasks, balancing response speed and model quality.

6. Practical Recommendations for Developers

• Design a stable abstraction layer first: Once the interface is finalized, adding a new model should only require adding an adapter. System feature code should not need modification, significantly reducing long-term maintenance costs.

• Move privacy and compliance earlier: Introduce dimensions such as privacy level, authorization status, and data classification at the beginning of architecture design, avoiding reactive fixes later.

• Establish observability: Record metrics for each model invocation, such as latency, success rate, and fallback trigger count, to provide data support for model selection and capacity planning.

• Keep models replaceable: Avoid hard-coding proprietary features of any particular model into business logic, ensuring that the system retains the ability to switch models as the ecosystem evolves.

Conclusion

Behind the goal of “allowing users to freely select and switch models” is a paradigm shift in operating system-level AI architecture: from deeply coupled single capabilities to an open, compatible, and controllable model distribution platform. For developers and platform owners, real competitiveness does not lie in developing a single model, but in building an integration framework with stable abstraction, flexible routing, strong security, and predictable performance. Such a framework enables efficient collaboration between a complex model ecosystem and a large device base. This is not only a technology selection issue, but also a reflection of long-term engineering governance capability.