Refactoring Baidu AI Cloud's Large Model Scheduling Gateway: A Technical Path for Dynamic Model Access

    This article analyzes the engineering rationale and technical design behind Baidu AI Cloud's refactoring of a large-model scheduling entry. It covers a five-layer scheduling architecture, dynamic routing, metering and billing, reliability design, security and compliance, developer experience, and a phased implementation approach.

    Refactoring Baidu AI Cloud’s Large Model Scheduling Gateway: A Technical Path for Dynamic Model Access

    As large-model applications enter the scale stage, model supply is shifting from “single capability invocation” to a comprehensive service platform covering “multiple models, multiple versions, multiple specifications, and multiple billing methods”. Taking model marketplaces as a typical form, platforms need to simultaneously support text generation, understanding, Q&A, summarization, code assistance, multimodal understanding, video generation, and other capabilities. At the same time, they must handle differentiated versions, prices, and service quality provided by different suppliers for the same capability.

    Baidu AI Cloud’s refactoring of the large-model scheduling entry reflects this broader transition. The entry point of large-model services is no longer just a simple list of model names, nor should it be a static configuration that forwards requests to a fixed endpoint. What truly needs to be refactored is a “scheduling plane” for large-model invocation. This plane should integrate model discovery, capability selection, routing decisions, traffic governance, metering and billing, runtime monitoring, and compliance auditing, allowing upper-layer applications to use underlying model capabilities in a unified, stable, and predictable way.

    1. From Model Lists to Scheduling Entries: Why the Problem Emerged

    Traditional software service invocation usually revolves around fixed service endpoints, explicit API protocols, and relatively stable capacity planning. Large-model services, however, have several distinctive characteristics.

    First, capability boundaries are semantic rather than structural. A model may support text generation, instruction following, summarization, role-based dialogue, and other tasks, but there is no naturally clear API boundary between these tasks. Platforms cannot rely only on simple URLs or static parameters to determine model capabilities; they need finer-grained capability tags.

    Second, model versions and supply instances are highly dynamic. The same type of capability often has multiple versions, parameter scales, suppliers, and available regional instances. Different instances may differ in price, latency, stability, content-safety policies, and compliance requirements. If the entry layer still uses hard-coded configurations, callers cannot flexibly choose services according to business goals.

    Third, the cost of large-model invocation cannot simply be equated with traditional request counts. Text models are usually measured by input tokens and output tokens. Scenarios such as video generation, image generation, and long-context processing may introduce additional billing dimensions. The entry layer must accurately identify invocation types, resource consumption, and billing rules; otherwise, it is difficult for a platform to build a transparent operations system.

    Fourth, large-model invocation chains are longer and have more complex states. Streaming output, long-running tasks, asynchronous generation, callback notifications, failure retries, content moderation, and security interception are often interwoven. This places higher demands on the governance capabilities of the entry layer. A simple proxy is usually insufficient to solve authentication, rate limiting, routing, degradation, auditing, and billing at the same time.

    Therefore, the core goal of refactoring a large-model scheduling entry is to upgrade “model display” to “model scheduling” and “static entries” to a “dynamic service plane”.

    2. Refactoring Goals: Building a Unified Large Model Scheduling Plane

    From an engineering perspective, refactoring a large-model scheduling entry should achieve at least five forms of unification.

    1. Unified Access

    When different applications call large models, their input/output formats, authentication methods, timeout policies, error-code handling, and streaming protocols may differ. If callers must individually adapt to every model, both platform operating costs and integration costs will rise significantly.

    The value of a unified access layer is to let callers make requests according to one stable specification, while the platform internally handles model selection, parameter adaptation, protocol conversion, and result encapsulation. This reduces developer complexity and helps the platform maintain a stable external contract even when underlying models change.

    2. Unified Model Catalog

    A model catalog is not merely a product list; it is the data foundation for scheduling decisions. Platforms need standardized metadata for each model, including task type, input modality, output modality, context length, applicable scenarios, version status, service level, billing unit, and compliance scope.

    The key to a model catalog is abstraction. Callers do not need to know which specific supply instance serves a request. They only need to declare task goals and constraints, while the platform selects an appropriate target from available instances. In this way, model capability changes from a “fixed resource” to an “orchestrated resource”.

    3. Unified Routing

    The routing layer answers the question: “Which model instance should handle this request?” Routing should not rely only on model names; it should consider latency, cost, success rate, capacity, compliance requirements, tenant policies, real-time load, and other dimensions.

    When the same capability is provided by multiple supply instances, the routing layer also needs to support multi-target matching, priority ordering, failover switching, canary release, and circuit-breaker isolation. Only then can the platform quickly switch away from an unavailable or overly expensive instance while reducing the impact on upper-layer applications.

    4. Unified Metering and Billing

    The sustainability of a large-model platform depends on clear metering. The entry layer needs to identify metering dimensions such as input tokens, output tokens, invocation count, task duration, and generation specifications, then map them to unified billing units.

    In multi-supplier scenarios, the same logical model may have multiple supply instances with different pricing policies. The scheduling entry should support cost estimation before routing, actual billing after invocation, and visibility into resource consumption across instances, business lines, and tenants.

    5. Unified Observability and Governance

    The runtime quality of large-model services cannot depend only on invocation results; it requires end-to-end observability. The entry layer should record key metrics such as request IDs, model versions, routing results, latency, failure reasons, content-safety status, and billing results.

    Observability is not only for troubleshooting; it is also the basis for operational optimization. For example, a platform can adjust routing weights based on latency and success rates, optimize instance selection based on cost changes, and drive model or gateway improvements based on error distributions.

    3. Overall Architecture: A Five-Layer Scheduling Model

    A relatively complete large-model scheduling entry can be abstracted into five layers: access layer, catalog layer, decision layer, execution layer, and observability layer.

    1. Access Layer

    The access layer receives application requests and handles identity authentication, tenant identification, quota checks, basic rate limiting, request-parameter validation, and request-ID generation. This layer should remain lightweight but must be stable enough because it determines the quality of the entire invocation chain.

    Typical capabilities include:

    • Application identity recognition and permission verification;
    • Tenant-level quota checks;
    • Basic rate limiting and burst-traffic protection;
    • Preliminary security inspection of request content;
    • Unified error-code responses;
    • Request IDs that propagate through the entire chain.

    2. Catalog Layer

    The catalog layer manages all schedulable model assets. Each logical model capability should correspond to a catalog record and be associated with one or more supply instances.

    An illustrative structure can be as follows:

    1. {
    2. "model_name": "general_text_model",
    3. "capability": "text_generation",
    4. "modality": {
    5. "input": ["text"],
    6. "output": ["text"]
    7. },
    8. "versions": [
    9. {
    10. "version": "v2",
    11. "status": "active",
    12. "instances": [
    13. {
    14. "instance_id": "instance-a",
    15. "region": "region-a",
    16. "price": {
    17. "input_token": 0.001,
    18. "output_token": 0.002
    19. },
    20. "sla": "high"
    21. },
    22. {
    23. "instance_id": "instance-b",
    24. "region": "region-b",
    25. "price": {
    26. "input_token": 0.0008,
    27. "output_token": 0.0018
    28. },
    29. "sla": "standard"
    30. }
    31. ]
    32. }
    33. ]
    34. }

    The benefit of this design is that an upper-layer request only needs to specify a capability such as general_text_model, while the scheduling layer decides which specific instance to use according to policy.

    3. Decision Layer

    The decision layer is the core of the scheduling entry. It receives request goals and constraints, then selects the most suitable one or more targets from candidate instances.

    Common constraints include:

    • Maximum acceptable latency;
    • Cost budget;
    • Compliance-region requirements;
    • Context-length requirements;
    • Content-safety level;
    • Tenant-specific instance preferences;
    • Whether streaming output is required;
    • Whether fallback to a backup instance is allowed.

    The decision result is not necessarily a single target. For high-reliability scenarios, the decision layer can also output a primary target, a backup target, and a degraded target, allowing the execution layer to invoke them in order.

    4. Execution Layer

    The execution layer converts an abstract request into a call format that a specific supply instance can understand. It also handles request transmission, streaming reception, timeout control, retries, circuit breaking, and result normalization.

    The execution layer must pay special attention to streaming scenarios. For generation results returned in segments, it needs to handle connection persistence, fragment aggregation, interruption recovery, and final-state confirmation. For long-running tasks such as video generation, it also needs to support asynchronous tasks, progress queries, and result delivery.

    5. Observability Layer

    The observability layer runs through the entire request lifecycle. It records routing results, latency, success rates, failure types, cost estimates, and final billing for each invocation. Observability data serves not only operations but also routing-policy optimization and model-catalog improvements.

    For example, when an instance’s error rate rises or latency increases significantly over a period of time, the scheduling layer can reduce its weight. When an instance’s cost changes significantly, operators can re-evaluate pricing and priority settings in the model catalog.

    4. Dynamic Routing: Scheduling Strategies for Multiple Instances of the Same Model

    The original material notes that the same model may have different prices across suppliers. This is typical in a platformized model environment. A scheduling entry cannot simply choose the “cheapest” or “fastest” option; it should establish a multi-dimensional scoring mechanism.

    A general approach is to calculate a composite score for candidate instances:

    1. def score_instance(instance, request, metrics):
    2. latency_score = normalize(metrics[instance.id]["p95_latency"])
    3. price_score = normalize(instance.price.total)
    4. reliability_score = metrics[instance.id]["success_rate"]
    5. compliance_score = compliance_match(instance, request)
    6. return (
    7. 0.30 * latency_score +
    8. 0.25 * price_score +
    9. 0.25 * reliability_score +
    10. 0.20 * compliance_score
    11. )

    Based on this, platforms can choose different strategies for different scenarios.

    1. Cost-First Strategy

    This strategy is suitable for non-real-time tasks that can tolerate longer waiting times and are sensitive to budgets. Examples include batch summarization, offline text processing, and content-archive analysis. Such scenarios can prefer instances with lower prices and stable success rates, and switch to backup targets when necessary.

    2. Low-Latency Strategy

    This strategy is suitable for real-time dialogue, online assistance, search augmentation, and similar scenarios. These scenarios care more about time-to-first-token and overall response speed. Price weight can be reduced accordingly, but compliance and availability must still be guaranteed.

    3. High-Availability Strategy

    This strategy is suitable for critical business paths. It usually does not rely on a single instance. Instead, it prepares primary and backup instances and sets circuit-breaker, timeout, retry, and degradation rules. When the primary instance fails, the system can automatically switch to the backup instance to reduce user impact.

    4. Compliance-First Strategy

    When a business has explicit requirements on data region, content safety, or invocation scope, the scheduling layer must first filter out instances that do not meet compliance requirements. It can then optimize performance and cost within the remaining candidate set.

    5. Canary Release Strategy

    When a new model version or new supply instance goes live, the platform can first route a small percentage of traffic for validation. Based on error rates, latency, cost, user feedback, and other metrics, it can then gradually increase traffic. This reduces the business risk of model replacement.

    5. Metering and Billing: Making Scheduling Results Explainable and Billable

    Refactoring a large-model platform entry should not focus only on successful invocations. It must also make every invocation measurable, explainable, and traceable.

    1. Unified Metering Units

    Different models may be measured by tokens, invocation count, generation duration, image count, video duration, or other units. The platform needs to map these raw measurements to unified billing units at the entry layer. For example:

    1. Unified cost = input token count × input unit price
    2. + output token count × output unit price
    3. + fixed invocation surcharge
    4. + asynchronous task processing fee

    For scenarios where output length cannot be known in advance, the platform can provide a cost estimate before routing and generate the final bill after the invocation ends.

    2. Support for Multi-Supplier Price Mapping

    When the same logical model corresponds to multiple supply instances, each instance may have a different price. The entry layer needs to maintain the mapping between “logical model pricing” and “actual settlement pricing” to avoid exposing underlying price differences directly to upper-layer applications.

    1. def estimate_cost(request, selected_instance):
    2. input_tokens = estimate_input_tokens(request.prompt)
    3. output_tokens = estimate_output_tokens(request.max_output_tokens)
    4. return (
    5. input_tokens * selected_instance.price.input_token +
    6. output_tokens * selected_instance.price.output_token
    7. )

    3. Support for Budget Control and Quota Management

    Enterprise platforms often need to set budget limits by tenant, application, project, or user. When an invocation may exceed the limit, the scheduling entry should be able to intercept, degrade, or switch to a lower-cost instance in advance.

    Budget control is not only a financial issue; it is also a stability issue. Without budget constraints, sudden traffic surges can cause uncontrolled resource consumption and affect the platform’s overall service capacity.

    6. Reliability Design: Streaming, Long-Running Tasks, and Failure Degradation

    Large-model invocation is not always “one request, one response”. Many scenarios require streaming returns, many generation tasks take a long time, and some tasks need asynchronous execution. The scheduling entry must design corresponding strategies for different invocation forms.

    1. Streaming Invocation Governance

    Streaming output can improve interaction quality, but it also increases connection-management complexity. The entry layer needs to handle connection establishment, heartbeat maintenance, fragment transmission, abnormal interruption, and result merging. If a streaming connection to an instance breaks, the scheduling layer can decide whether to automatically switch to a backup instance, avoiding a complete failure from the user’s perspective.

    2. Long-Running and Asynchronous Tasks

    Tasks such as video generation, complex document parsing, and batch content generation may not complete within a short period. In this case, the entry layer should support task submission, task IDs, status queries, result retrieval, and retry mechanisms instead of forcing callers to hold synchronous connections for a long time.

    3. Retry and Idempotency

    Large-model invocation often involves relatively high costs, so blind retries are inappropriate. The entry layer needs to distinguish retryable errors from non-retryable errors. For example, network timeouts can be retried, while errors such as content-safety interception, invalid parameters, or insufficient quota usually should not be retried directly.

    For requests that support idempotency identifiers, the platform can use the request ID to avoid duplicate billing or duplicate generation.

    4. Circuit Breaking and Degradation

    When an instance continuously fails or its latency rises abnormally, the scheduling layer should automatically reduce its weight and trigger a circuit breaker. At the same time, degradation paths can be prepared, such as switching to an instance with similar capabilities but lower cost, or returning cached results, template results, and prompt messages.

    Degradation is not a simple failure; it is a safeguard for business continuity.

    7. Security and Compliance: A Required Boundary at the Entry Layer

    A distinctive characteristic of large-model services is that both inputs and outputs may be unstructured content. They involve data security, content safety, intellectual property, business compliance, and other aspects. The scheduling entry should act as a unified security boundary rather than leaving all security issues to underlying models.

    Key measures include:

    • Performing necessary security checks on input content;
    • Conducting compliance review and risk labeling for output content;
    • Masking sensitive data or minimizing exposure;
    • Enforcing permission isolation for tenants, applications, and users;
    • Retaining auditable logs across the invocation chain;
    • Setting risk-control policies for abnormal invocation behavior;
    • Retaining necessary traceable identifiers for model outputs.

    Security governance should be coordinated with routing policies. For example, some business scenarios may only be allowed to call instances that satisfy specific security policies, and data from some regions may only be routed to instances within a designated compliance scope. The scheduling layer should perform these judgments during candidate filtering rather than after request execution.

    8. Developer Experience: Unified Interfaces Reduce Integration Costs

    The success of a scheduling-entry refactor is ultimately reflected in developer experience. The platform should allow application developers to avoid being aware of underlying model differences whenever possible. They should be able to submit tasks according to a unified specification.

    A simplified invocation example is shown below:

    1. request = {
    2. "request_id": "req-20250112-001",
    3. "task": "text_generation",
    4. "prompt": "Please summarize the key points of the following document.",
    5. "constraints": {
    6. "max_latency_ms": 1000,
    7. "cost_budget": 0.02,
    8. "compliance_region": "allowed"
    9. }
    10. }
    11. result = model_gateway.invoke(request)

    In this call, developers care about task type, constraints, and returned results, rather than which instance, version, or region provides the service.

    Platforms can further provide operational interfaces such as capability queries, budget queries, invocation records, cost analysis, and routing-policy configuration. These interfaces allow platform managers to continuously optimize model catalogs and scheduling rules.

    9. Implementation Path: From a Static Model Marketplace to a Dynamic Scheduling Plane

    For platform providers, refactoring usually cannot be completed in one step. It is advisable to proceed in phases.

    Phase 1: Standardizing the Model Catalog

    First, organize existing model capabilities and unify task classification, metadata fields, version status, and billing rules. The focus of this phase is not scheduling algorithms but building a trustworthy data foundation.

    Phase 2: Unified Access and Observability

    Build a unified gateway that handles authentication, rate limiting, logging, metrics, and standardized error codes. This phase can significantly improve the visibility and controllability of the invocation chain.

    Phase 3: Multi-Instance Routing

    After catalog standardization and unified access, introduce multi-instance selection and dynamic routing. A platform can first adopt rule-based strategies and then gradually introduce weights, scoring, and automatic degradation.

    Phase 4: Billing and Budget Governance

    Associate invocation results with actual pricing, tenant budgets, and resource consumption to form a cost system that is queryable, controllable, and auditable.

    Phase 5: Intelligent Scheduling

    After enough data has accumulated, more complex scheduling models can be introduced, such as comprehensive optimization based on historical latency prediction, cost prediction, tenant profiling, and business objectives.

    10. Conclusion

    Refactoring a large-model scheduling entry is essentially not about adding another model page. It is about turning model capabilities into service infrastructure that is discoverable, selectable, routable, measurable, and governable. In the face of multi-model supply and price differences, a platform needs to establish clear boundaries among the abstraction layer, routing layer, and operations layer: the catalog describes capabilities, scheduling selects targets, the execution layer ensures stable invocation, and observability provides feedback for optimization.

    When these capabilities are gradually improved, large-model services can move from “models available” to “services controllable”, and from “single invocations” to “platform operations”. For developers and enterprise users, this means lower integration costs, more stable service experiences, more transparent cost management, and more flexible business innovation.