Multi-Source LLM Gateway for Enterprise Platforms: Architecture, Invocation, and Billing
This article examines the engineering practices behind a unified multi-source LLM gateway for enterprise platforms. It covers model selection and heterogeneous API challenges, an adapter-based gateway architecture with unified authentication and failover, a prepaid points and asynchronous token metering model, and a simplified Python adapter example. The goal is to show how platforms can abstract vendor differences, route requests flexibly, and control AI costs.
With the rapid development of large language model (LLM) technology, enterprise office platforms are undergoing a deep evolution from traditional SaaS to intelligent SaaS. Recently, an intelligent office platform launched a third-party LLM access service for enterprise customers. Team subscription and enterprise customers can directly invoke multiple mainstream LLMs without separately procuring from or applying to model vendors, and existing points are deducted based on actual usage. This business move reflects a core technical requirement for enterprise AI adoption: how to build a unified, efficient, and billable multi-source LLM access gateway. This article discusses the engineering practices of multi-source LLM access from three dimensions: architecture design, invocation mechanisms, and billing models.
1. Business Requirements and Technical Challenges of Multi-Source LLM Access
In real enterprise scenarios, a single LLM often cannot cover all requirements. For example, some models perform well in long-text logical reasoning, while others have advantages in code generation or multilingual translation. Therefore, enterprise platforms need the ability to invoke multiple models on demand.
1.1 Model Selection Challenges Caused by Diverse Scenarios
Different business lines have significantly different model requirements. The marketing department may need high-quality copy generation, while the R&D department places more emphasis on code completion accuracy. If business teams connect directly to the APIs of each underlying model provider, they face substantial learning and integration costs.
1.2 Complexity of Unified Authentication and Invocation
Different model vendors have different API specifications, including request body formats, authentication mechanisms such as API Key and OAuth, and streaming output protocols such as SSE and WebSocket. How to hide these heterogeneous interfaces at the lower layer and provide a standardized invocation specification to the business layer is the primary challenge for developers.
1.3 Cost Control and Flexible Billing
LLM invocation costs are usually based on token counts. When a platform aggregates multiple models, each vendor may have different token billing ratios and billing granularity. How to implement unified billing logic within one platform so that enterprise customers can consume through prepaid points or a unified quota without handling vendor-specific billing is key to commercialization.
2. Unified Model Gateway Architecture Design
To address these challenges, enterprise platforms typically need to build a unified model gateway. As an intermediate layer between business teams and LLMs, the gateway handles request routing, protocol conversion, security authentication, and logging and monitoring.
2.1 Abstraction Layer and Adapter Pattern
The core design idea of the gateway is the adapter pattern. First, a unified standard API protocol is defined within the platform, and then a corresponding adapter is developed for each third-party LLM that is integrated. The adapter converts requests in the unified format into the format required by a specific model and maps the model response back to the unified format. In this way, business teams only need to program against the standard interface, and replacement or addition of underlying models is completely transparent to the business side.
2.2 Unified Identity Authentication and Permission Management
In enterprise scenarios, permission control is critical. The gateway needs to connect to the enterprise identity authentication system, such as SSO. When an enterprise customer initiates an invocation, the gateway first verifies its subscription status and remaining points. For team subscription customers, role-based access control (RBAC) can be further implemented to restrict specific departments to specific types of models.
2.3 Load Balancing and Failover Mechanisms
LLM API invocations are characterized by high latency and long connections. The gateway needs intelligent routing capabilities to dynamically distribute requests based on the current response latency and availability of each model service provider. When a third-party model service is rate-limited or unavailable, the gateway should automatically trigger failover and redirect the request to a functionally similar backup model to ensure business continuity. At this layer, combining general API gateway and monitoring and alerting products from cloud providers such as Baidu AI Cloud can significantly improve the high availability and observability of the gateway.
3. Usage-Based Unified Billing and Points Management
The original announcement mentioned that subsequent invocations will deduct existing points based on actual usage. This is a mainstream platform-level billing model for AI services. Its technical implementation involves a complex asynchronous metering system.
3.1 Advantages of the Prepaid Points Model
The points mechanism is essentially a virtual currency within the platform. Enterprise customers purchase points in advance, and the platform sets different points/token exchange ratios based on the commercial pricing of different models. The advantages of this model are twofold: on one hand, the platform can obtain margin through bulk procurement of underlying model APIs; on the other hand, enterprise customers avoid the complexity of managing bills from multiple cloud vendors and achieve unified financial management.
3.2 Asynchronous Token Metering and Reconciliation Mechanism
LLMs usually support streaming output, which makes accurate token metering complex. A common approach is to intercept streaming response data chunks at the gateway layer, buffer them, and parse them. When streaming ends, or when the model returns the finish_reason and complete usage field, the gateway sends billing events asynchronously through a message queue such as Kafka or RabbitMQ to an independent billing service. The billing service asynchronously updates the remaining points of the enterprise account and records detailed invocation logs through a logging service, so that enterprise customers can perform subsequent reconciliation and cost analysis.
4. Code Example: Simplified Implementation of a Multi-Model Adapter
The following is a simplified model adapter pattern implemented in Python. It shows how to smooth out differences among multiple models through interface-oriented programming:
from abc import ABC, abstractmethodfrom typing import Dict, Anyclass BaseModelAdapter(ABC):"""Abstract base class for LLM adapters"""@abstractmethoddef invoke(self, prompt: str, params: Dict[str, Any]) -> Dict[str, Any]:passclass ModelAAdapter(BaseModelAdapter):"""Adapter for Model A (e.g., a model strong in text creation)"""def invoke(self, prompt: str, params: Dict[str, Any]) -> Dict[str, Any]:# Simulate converting the standard request into Model A's API format and making an HTTP requestprint(f"Calling Model A with prompt: {prompt}")# Simulate returning a standardized responsereturn {"content": "Response from Model A", "tokens_used": 120}class ModelBAdapter(BaseModelAdapter):"""Adapter for Model B (e.g., a model strong in code generation)"""def invoke(self, prompt: str, params: Dict[str, Any]) -> Dict[str, Any]:print(f"Calling Model B with prompt: {prompt}")return {"content": "Response from Model B", "tokens_used": 95}class ModelGateway:"""Unified model gateway"""def __init__(self):self.adapters = {"text_creation": ModelAAdapter(),"code_generation": ModelBAdapter()}self.user_points = {"enterprise_user_1": 10000}def route_and_call(self, user_id: str, task_type: str, prompt: str, params: Dict[str, Any]):if user_id not in self.user_points or self.user_points[user_id] <= 0:raise Exception("Insufficient points or unauthorized access.")adapter = self.adapters.get(task_type)if not adapter:raise Exception("Unsupported task type.")# Initiate the invocationresponse = adapter.invoke(prompt, params)# Asynchronously deduct points (synchronous simulation here)cost = response.get("tokens_used", 0)self.user_points[user_id] -= costreturn response# Business-side invocation examplegateway = ModelGateway()res = gateway.route_and_call("enterprise_user_1", "code_generation", "Write a python hello world", {})print(res)
5. Conclusion and Outlook
An intelligent office platform integrating multiple third-party LLMs and providing unified invocation and billing services indicates that enterprise AI infrastructure is moving toward platformization and aggregation. By building a unified model gateway, enterprises can effectively shield underlying model differences and achieve flexible routing and scheduling as well as refined cost control.
For developers, when designing and implementing such systems, in addition to interface adaptation and high-availability architecture, they should also pay attention to data privacy and security compliance. In the future, by combining large model development platforms, container platforms, and security components from mature cloud vendors such as Baidu AI Cloud, enterprises will be able to build AI middle platforms with high scalability and fault tolerance more efficiently, further unlocking the productivity value of LLMs in business scenarios.
