LLM Open Weights, Paid Services, and API Engineering: A Practical Serving Architecture

    Large language models are advancing along two parallel tracks: the continuous iteration of open-weight releases and increasingly tiered commercial services. This article examines how both tracks ultimately rest on a shared technical foundation—one that can schedule open model weights efficiently while exposing services in a measurable, controllable, and auditable way. It covers the engineering implications of open weights, how free, subscription, and pay-per-use API models can coexist behind a unified gateway, and how to keep services stable, costs controlled, and data compliant during rapid model iteration. The guidance spans inference optimization, long-context management, low-latency streaming, gray release, cost and unit economics, observability, and compliance, along with actionable recommendations for developers. The goal is a practical, production-ready paradigm for LLM services.

    Introduction: The Two Main Tracks of LLM Development

    The LLM landscape has been advancing along two parallel tracks. The first is the continuous iteration of open weights. Multiple organizations have released increasingly capable open models in rapid succession, forming a loop of “open release → ecosystem adaptation → feedback → re-iteration.” At the same time, vendors commonly reserve their most competitive versions for cloud-hosted services, producing a product matrix in which open and closed flagship models run side by side. The second track is a tiered commercial service system: basic capabilities are offered for free to grow the user base; subscriptions target high-frequency individual users; and metered APIs serve developers and enterprises. In capital markets, dense funding rounds and steadily rising valuations reinforce confidence in this technical direction.

    From an engineering perspective, both tracks ultimately converge on the same foundation: one that can efficiently schedule open model weights while delivering services in a measurable, controllable, and auditable way. A mature system must answer three questions:

    • What engineering constraints do open weights introduce?
    • How can free, subscription, and pay-per-use models coexist?
    • When models iterate rapidly, how do we ensure service stability, controlled costs, and data compliance?

    This article addresses these three questions and lays out a practical architecture paradigm for LLM services.

    1. What Open Weights Mean for Engineering

    1.1 Open Weights Do Not Mean a Lower Deployment Barrier

    Open models dramatically lower the barrier to secondary development, but turning weights into a stable service still requires solid work in several areas:

    • Hardware and GPU memory: Model size, context length, and precision together determine memory demand. Before deployment, run capacity estimates for the weights, KV cache, and batch concurrency to avoid mass request failures caused by out-of-memory errors after launch.
    • Quantization: Low-precision formats such as INT8 and INT4 can significantly reduce memory footprint and improve throughput, but the accuracy loss must be evaluated. For sensitive tasks like math and code, build before-and-after comparative evaluations.
    • Inference acceleration: Techniques such as PagedAttention and continuous batching improve throughput, while prefix caching and speculative decoding effectively reduce time-to-first-token (TTFT) and decoding overhead.
    • Tool calling and structured output: Capabilities like function calling and JSON-mode output are the key interfaces for embedding a model into business workflows.

    1.2 Version Compatibility Governance

    Rapid iteration inevitably introduces compatibility problems. The code below abstracts model capabilities into metadata and parses them uniformly at the access layer, serving as a prerequisite for routing and billing:

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class ModelSpec:
    4. key: str # Public model identifier
    5. context_window: int # Context window (tokens)
    6. max_output: int # Maximum output per call (tokens)
    7. default_temperature: float = 0.7
    8. supports_vision: bool = False
    9. supports_tools: bool = True
    10. pricing_tier: str = "standard"
    11. # Capability registry: adding a new version only adds an entry, keeping old versions available
    12. REGISTRY = {
    13. "flagship-pro": ModelSpec("flagship-pro", 128_000, 8192,
    14. supports_vision=True, pricing_tier="premium"),
    15. "flagship-lite": ModelSpec("flagship-lite", 32_000, 4096, pricing_tier="standard"),
    16. }
    17. def resolve_model(model_key: str, plan: str) -> ModelSpec:
    18. spec = REGISTRY.get(model_key)
    19. if spec is None:
    20. raise ValueError(f"unknown model: {model_key}")
    21. # Permission check: different plans can access different model capabilities
    22. if plan == "free" and spec.pricing_tier == "premium":
    23. raise PermissionError(f"plan '{plan}' cannot access '{model_key}'")
    24. return spec

    Compatibility governance should follow these principles:

    1. Semantic versioning: Version the model identifier, default parameters, and context window, and surface behavioral changes through the version number.
    2. Capability matrix: Use a “version × capability” matrix to define each version’s context length, multimodal support, and tool-calling support, and share it with downstream consumers.
    3. Smooth deprecation: Keep each old version available for at least one full transition cycle, announce the deprecation date in advance, and prompt callers to migrate through response headers or logs.
    4. Prompt compatibility: After an upgrade, expect shifts in answer style or instruction-following behavior. Cover key scenarios with a regression evaluation set before rolling out to full traffic.

    2. Free, Subscription, and Pay-Per-Use: Unified Support for Tiered Services

    2.1 Three Product Tiers

    Dimension Free Tier Subscription Tier Pay-Per-Use API
    Target users New and low-frequency individuals High-frequency individuals and small teams Developers and enterprises
    Billing Free Monthly or annual Billed by actual usage
    Resource quota Low, strictly rate-limited Medium to high, possibly with priority guarantees Elastic, adjusted by account level
    Model capabilities Mainly lightweight models Access to flagship models Wide range of model options
    Service level Best effort, may queue at peak Higher availability guarantee Per commercial agreement

    The three tiers may look very different, but internally they share the same model runtime and gateway capabilities. The differences show up in only four dimensions: identity, permissions, quota, and metering.

    2.2 Unified Gateway and Capability Validation

    Every request passes through a unified gateway that performs authentication, permission checks, quota checks, request rewriting, and metering reports in sequence. The core flow is:

    1. Authentication: API keys, session tokens, or OAuth tokens are all mapped to an internal principal identifier.
    2. Plan resolution: Look up the principal’s current plan (free, a subscription tier, or an API level) and its limits.
    3. Capability validation: Confirm that the requested model, context length, and output length are available under the current plan.
    4. Parameter rewriting: Fill in default parameters by plan and model, and block or restrict high-risk parameters.
    5. Quota reservation: Pre-deduct quota based on estimated usage (see Section 2.3).
    6. Inference execution: Route to a suitable model instance and return results as a stream.
    7. Metering and settlement: Settle based on actual usage and return any remaining reserved quota.

    Example: gateway request-handling skeleton (pseudocode)

    1. def handle_chat(request, principal):
    2. spec = resolve_model(request.model, principal.plan)
    3. quota = quota_service.load(principal.id, principal.plan) # Load plan quota
    4. # 1. Rule engine validation: plan, permissions, and parameter ranges
    5. policy = policy_engine.evaluate(principal, spec, request)
    6. if not policy.allowed:
    7. raise PolicyDenied(policy.reason)
    8. # 2. Quota reservation: token dimension + request count dimension
    9. ticket = quota_service.reserve(
    10. principal.id, spec,
    11. tokens=estimate_tokens(request), requests=1,
    12. )
    13. try:
    14. result = inference_engine.run(request, spec) # Run inference and stream results
    15. except Exception:
    16. quota_service.release(ticket) # Roll back on failure
    17. raise
    18. # 3. Settlement: based on actual usage, release remaining reservation
    19. quota_service.settle(ticket, usage=result.usage)
    20. metering.report(principal, spec, request.id, result.usage)
    21. return result

    Unified metering event structure:

    1. def build_usage_event(principal, request_id, spec, usage):
    2. # event_id serves as the idempotency key, composed of the principal ID and request ID
    3. return {
    4. "event_id": f"{principal.id}:{request_id}",
    5. "principal_id": principal.id,
    6. "plan": principal.plan,
    7. "model": spec.key,
    8. "prompt_tokens": usage.prompt_tokens,
    9. "completion_tokens": usage.completion_tokens,
    10. "cached_prompt_tokens": usage.cached_tokens, # Cache-hit input tokens counted separately
    11. "unit_price_version": spec.price_version, # Price version, for tracing historical bills
    12. "occurred_at": usage.finished_at,
    13. }

    2.3 Quota, Rate Limiting, and Concurrency Control

    The differences between plans ultimately manifest as three kinds of limits, each handled separately:

    • Quota: A cumulative allowance over a period, such as the total monthly tokens available under a subscription, or the daily request cap on the free tier.
    • Rate limit: A rate cap within a short time window, such as requests or tokens per minute. It protects against instantaneous traffic bursts and does not replace quota.
    • Concurrency control: The cap on simultaneously in-flight requests. Streaming generation takes time, so limiting only the rate while ignoring concurrency easily causes requests to pile up at peak and drives up queueing latency.

    There are two approaches to token-based rate limiting: one is to reserve the difference between the request and completion sides, and the other is to dynamically correct based on historical statistics. The following simplified token bucket illustrates the “reserve-then-settle-by-actual-usage” logic:

    1. import time
    2. class TokenBucket:
    3. """Token-based rate limiter using a token bucket (simplified illustration, not a full production implementation)"""
    4. def __init__(self, capacity: int, refill_rate: float):
    5. self.capacity = capacity # Bucket capacity: max available tokens per window
    6. self.refill_rate = refill_rate # Tokens refilled per second
    7. self.tokens = float(capacity)
    8. self.updated = time.monotonic()
    9. def _refill(self):
    10. now = time.monotonic()
    11. self.tokens = min(
    12. self.capacity,
    13. self.tokens + (now - self.updated) * self.refill_rate,
    14. )
    15. self.updated = now
    16. def try_acquire(self, cost: int) -> bool:
    17. self._refill()
    18. if self.tokens >= cost:
    19. self.tokens -= cost
    20. return True
    21. return False
    22. def refund(self, amount: int):
    23. self._refill()
    24. self.tokens = min(self.capacity, self.tokens + amount)
    25. # Reserve 1000, complete 800, so the 200 difference is refunded
    26. def on_request_finished(bucket, reserved: int, actual: int):
    27. if actual < reserved:
    28. bucket.refund(reserved - actual)

    Several points deserve special attention in practice:

    • Overshoot: When output exceeds the reserved amount (for example, a stream that runs past max_tokens), it should be truncated, or a safety margin should be left during the reservation stage.
    • Quota-exhaustion prompts: Return standardized error codes and structured information to clients, such as retry_after, the current plan, and the quota type, so they can implement backoff retries or prompt an upgrade.
    • Multi-instance consistency: Rate-limiting state needs centralized storage or sharding by consistent hashing; otherwise, the effective limit gets multiplied across a multi-instance deployment.
    • Response header passthrough: Exposing remaining quota and quota reset time in response headers lets developers proactively manage their call pacing.

    2.4 Metering, Billing, and Reconciliation

    In a hybrid model of subscriptions and pay-per-use, metering accuracy directly determines both user experience and financial credibility. Key design points include:

    1. Dual metering perspectives: Track both token counts (billing) and request counts (risk control). A streaming response should be counted once, using the request identifier established when the first packet is returned.
    2. Idempotent reporting: Metering events are reported asynchronously through a message queue and deduplicated by event identifier, avoiding double billing caused by network retries.
    3. Latency and out-of-order delivery: Metering for streaming requests is produced on completion, so billing should tolerate some delay (minutes to hours) and settle periodically. Requests that span billing cycles are attributed by their actual occurrence time.
    4. Price versioning: Model pricing changes with versions, so each metering event should record the price version used for billing, ensuring historical bills remain traceable and reproducible.
    5. Reconciliation and dispute handling: Periodically reconcile metering records against billing summaries, and retain original request identifiers so users can query the usage details of a single call.
    6. Plan switching: On upgrade or downgrade, remaining quota is prorated, carried over, or cleared per agreement. This rule should be implemented explicitly in the metering layer rather than left for the billing system to infer implicitly.

    2.5 Multi-Tenancy Isolation and Security

    An open API serves a large number of external developers, so security boundaries must be enforced at the architecture level:

    • Identity and authentication: Rotate API keys regularly, and support key separation by environment (test, production) and by project; use short-lived tokens for management interfaces.
    • Tenant isolation: Isolate request queues across tenants at the inference layer to prevent a single large customer from crowding out shared resources; partition caches per tenant for sensitive scenarios.
    • Prompt injection and data leakage: Enforce a boundary between system prompts and user input to prevent users from extracting system prompts or other tenants’ data through crafted input.
    • Content safety: Apply compliance review on both input and output, blocking violations and recording audit logs.
    • Network and call protection: Set source and call-frequency policies for API callers, and automatically degrade or temporarily block abnormal traffic.

    3. Inference Serving: Performance, Context, and Stability

    Whether tiered services can stand ultimately depends on whether the underlying inference service can balance performance and cost. For example, when hosting such services on Baidu AI Cloud, general-purpose cloud products—object storage, message queues, container platforms, log services, and monitoring and alerting—can respectively handle model files and data storage, asynchronous task decoupling, elastic compute scheduling, and runtime observability, letting the team focus on the inference engine itself.

    3.1 Batching and Scheduling

    • Continuous batching: Traditional static batching must wait for the longest request in a batch to finish, and a long tail can drag down the whole batch. Continuous batching dynamically regroups the batch as sequences complete and new requests arrive, markedly improving GPU utilization.
    • Priority scheduling: Requests from subscription users can enter a priority queue, while free-tier requests are degraded or queued at peak, delivering tiered service at the scheduling level.
    • Preemption and fairness: Set a maximum execution time for long requests. When a high-priority request arrives, it can preempt lower-priority requests, save their KV cache state, and resume generation afterward.

    3.2 Long-Context Management

    Long context is a hallmark capability of flagship models, but it places heavy demands on memory and compute. The KV cache grows linearly with context length and concurrency and easily becomes a throughput bottleneck. Common approaches include:

    • Tiered storage: Offload the KV cache from GPU memory to host memory and even slower storage, swapping data in and out on demand to trade capacity for concurrency.
    • Prefix cache reuse: Share the KV cache of common prefixes—system prompts, knowledge base documents—across multiple requests to significantly reduce repeated computation, and separately mark cache-hit input tokens during metering.
    • Context trimming: For overlong conversations, retain a sliding window or apply summarization compression, balancing quality against cost.
    • Retrieval-Augmented Generation (RAG): Retrieve external knowledge on demand and inject it into the prompt instead of stuffing all material into the context, used together with a vector database and a document-parsing pipeline.

    3.3 Low Latency and Streaming Output

    Conversational products are highly sensitive to time-to-first-token (TTFT), while the generation phase cares more about the interval between output tokens (TPOT). Common optimizations include:

    • Speculative decoding: Have a small model draft multiple tokens, then verify them in parallel with the large model, reducing decoding steps while preserving the output distribution.
    • Quantized inference: Lower the cost of reading weights while keeping accuracy acceptable.
    • Streaming responses: Use a streaming protocol such as SSE to push results token by token or sentence by sentence, markedly improving perceived latency.
    • Edge proximity: Use edge access points and proximity routing to shorten network round-trip latency.

    3.4 Fault Isolation and Degradation

    • Version isolation: Deploy different model versions in separate instance groups so that a single-version failure does not affect others.
    • Circuit breaking and degradation: When a downstream model service is unhealthy, automatically switch to a backup lightweight model or return a clear degradation notice.
    • Capacity guardrails: Set overload protection for burst traffic from a single tenant to prevent a global cascade.
    • Graceful shutdown: When releasing a new version, first drain traffic, wait for in-flight requests to complete, and then terminate instances to avoid forcibly interrupting streaming requests.

    4. Gray Release and Quality Assurance for Model Iteration

    Model iteration is fast, but production cannot follow a “release to full traffic” model. The recommended release process is:

    1. Offline evaluation: Compare old and new versions on an evaluation set covering general Q&A, code, math, long text, and multi-turn dialogue, focusing on accuracy, instruction-following, and safety metrics.
    2. Shadow traffic: Replicate real online requests to the new version for execution, but return only the old version’s results, to observe latency, error rate, and output differences.
    3. Small-traffic gray release: Bucket by hashed user identifier, and ramp up in steps of 1%, 5%, 20%, 50%, and 100% while watching key metrics.
    4. Session consistency: Route the same session to the same version to avoid context discontinuity. This can be implemented with a deterministic hash over “session ID + version number.”
    5. One-click rollback: The model registry retains historical version weights, so an anomaly can be reverted in minutes without redeploying code.
    1. def select_version(model_key: str, session_id: str, rollout: dict) -> str:
    2. """rollout: {version: rollout ratio}, the sum of ratios must not exceed 1"""
    3. bucket = stable_hash(f"{model_key}:{session_id}") % 1000
    4. acc = 0
    5. for version, ratio in rollout.items():
    6. acc += int(ratio * 1000)
    7. if bucket < acc:
    8. return version
    9. return "stable" # Fall back to the stable version by default

    5. Cost Structure and Unit Economics

    Whether free and subscription models can be sustained long term depends on unit economics—the ratio of average revenue per user (ARPU) to average cost per user. The cost structure of LLMs can be roughly divided into four categories:

    • Inference compute: Directly tied to total tokens, context length, and model size, and usually the largest share.
    • Storage: Model weights, vector indexes, conversation history, and logs. The cost is relatively manageable but requires lifecycle management.
    • Network and bandwidth: The overhead of streaming transfer and cross-region calls.
    • R&D and operations: The human effort for model iteration, evaluation, on-call duty, and capacity planning.

    Ways to improve unit economics include:

    • Separating short and long requests: Route simple requests to lightweight models and reserve flagship models for complex ones, forming tiered routing.
    • Raising cache hit rates: Optimize system-prompt structure and place the invariant parts first to reuse the prefix cache.
    • Off-peak usage and elasticity: Use idle resources for training and batch inference to reduce peak-time resource redundancy.
    • Output-length governance: Set reasonable stop conditions on the generation side to avoid meaningless verbose output.
    • Tiered quota design: Set free and subscription limits constrained by marginal cost to prevent heavy users from eroding overall gross margin.

    6. Observability, Compliance, and Audit

    LLM services are inherently uncertain, so their observability system needs to be richer than that of traditional web services:

    • Metrics: Request volume, success rate, TTFT, TPOT, tokens per request, cache hit rate, call share by model version, and quota-hit counts.
    • Logs: Record request identifiers, model version, plan type, token usage, and latency breakdowns. Whether to log conversation content must follow privacy-compliance requirements, with masking and retention periods in place.
    • Distributed tracing: End-to-end tracing from gateway to inference engine to metering system, to help locate latency bottlenecks.
    • Evaluation and regression: Sample live traffic for automated quality evaluation, combine it with user feedback data, and form a continuous iteration loop.
    • Audit: Sensitive operations such as plan changes, price adjustments, and quota modifications must all leave a trail to meet audit and dispute-tracing needs.

    On the compliance front, focus on the following:

    • Collect the minimum data necessary, clearly disclose data usage, and support user deletion requests;
    • For cross-border deployment, evaluate data residency and cross-border transfer requirements;
    • Ensure output complies with applicable laws and regulations and platform content-safety standards;
    • In API documentation, clearly state the boundaries and limitations of model capabilities to avoid setting overly high expectations for outputs.

    7. Practical Recommendations for Developers

    1. Treat models as managed dependencies: Pin model versions, record prompt versions, and bring them into the code-review and regression-testing process.
    2. Abstract a unified gateway: Route all model calls through a single entry point to make authentication, metering, rate limiting, and degradation uniform.
    3. Make idempotency a first-class concern throughout: Request identifiers and metering event identifiers must be globally unique, so retries never cause double billing or double quota deduction.
    4. Quantify before selecting: Use quantization, caching, and routing to lower unit costs first, then design plan pricing and free allowances.
    5. Drive iteration with evaluation: Combine offline evaluation, shadow traffic, and gray rollout so that every model upgrade is verifiable and rollback-ready.
    6. Make observability a first-class citizen: Define key metrics and alert thresholds before launch, not after a failure.

    Conclusion

    The continuous iteration of open models and tiered commercial services are two sides of the same coin: the former determines the ceiling of capability, while the latter determines how that capability is delivered at scale and sustainably. For a technical team, real competitiveness lies not merely in the ability to deploy a model, but in the ability to build a service system that supports multiple versions, sustains multiple plans, and remains measurable, gray-release-capable, observable, and compliance-controlled. By strengthening model-version governance, a unified gateway, quota and metering, inference optimization, gray release, and cost operations, a team can maintain stable engineering delivery even amid a fast-changing model ecosystem.