Layered LLM Serving Architecture and Metering Operations in Practice

    This article walks through the engineering governance model behind large language model (LLM) serving platforms: why per-request pricing cannot be determined statically at the gateway, and how mature platforms split the problem into four decoupled subsystems — metering, entitlement, quota and admission, and capacity with graceful degradation. It covers capability tiering and logical model routing, version snapshots versus aliases, shadow traffic and canary gating, idempotent usage events, a unified entitlement model for free tiers, subscriptions, and pay-as-you-go, token-bucket budget control alongside concurrency and rate limits, unit economics and observability by layer, multi-tenant security and licensing concerns, plus a four-phase rollout roadmap and a pre-launch self-check list. Code and schema examples are included throughout.

    1. Problem Background: Why LLM Serving Needs Its Own Engineering Governance

    The biggest difference between an LLM serving platform and a conventional API service is not the protocol layer — it is the shape of cost.

    The price of a single request cannot be determined statically at the ingress. Input tokens, output tokens, whether a prefix cache was hit, whether a tool call was triggered, and whether the request went through a “reasoning” channel all contribute to how much compute the request ultimately consumes. This means:

    1. At the ingress you cannot do exact per-request pricing, only budget estimation plus post-hoc settlement.
    2. Per-request cost variance is extreme — a long-form generation request can cost tens of times more than an ordinary question-answering request.
    3. Elasticity is poor. Scaling an accelerator pool takes far longer than scaling containers for a stateless service, so traffic spikes must be absorbed by queuing, degradation, and rejection rather than by “spinning up another batch of instances.”

    The common commercial shape in the industry is “free allowance + subscription plan + pay-as-you-go” all running side by side. In engineering terms this is equivalent to four subsystems that must be decoupled:

    Subsystem Responsibility Key Design Points
    Metering Reconstruct an inference call as a structured usage event Complete dimensions, idempotency, replayability
    Entitlement Decide what this caller can use right now and at which capability tier Unified modeling of free allowance, subscriptions, and pay-as-you-go accounts
    Quota & Admission Intercept and prioritize before a request reaches the inference engine Budget token buckets, concurrency semaphores, priority queues
    Capacity & Degradation Preserve availability instead of cascading failure when resources run short Tiered degradation, feature-flagged operational actions

    The rest of this article follows that chain, from the capability supply side to the traffic governance side.


    2. Model Tiering and Intelligent Routing: Turning Model Capability into an Orchestrated Resource

    2.1 Three-Tier Capability Layering

    In practice, available models are usually abstracted into three or four capability tiers:

    • Lite: low latency, high throughput; handles classification, extraction, format conversion, simple Q&A, and batch summarization.
    • Standard: the workhorse for general conversation and content generation.
    • Reasoning: multi-step reasoning, code generation, complex math, and long tool-call chains; latency and cost are significantly higher.
    • Long Context: very long document understanding and cross-file retrieval Q&A.

    The value of tiering is not “more models is better” but making cost a first-class citizen in scheduling decisions.

    2.2 Routing Decisions: From “Users Pick a Model” to “The System Picks a Model”

    The most naive approach is to let the caller specify a model name in the request. That pushes cost responsibility onto the user and creates two problems: users have no way to judge task difficulty, and model retirement or upgrades break hardcoded references.

    A more robust approach introduces a logical model alias together with a routing layer:

    • The caller declares only the capability level and constraints (must tools be called, maximum latency tolerance, maximum cost budget).
    • The routing layer combines tenant plan, remaining budget, and live load to pick a concrete physical model.
    • Physical model names stay invisible to callers, so they can be swapped smoothly.
    1. from dataclasses import dataclass
    2. from typing import Optional
    3. @dataclass(frozen=True)
    4. class ModelTier:
    5. name: str # logical model name
    6. cost_per_mtok_in: float # relative cost coefficient per million input tokens
    7. cost_per_mtok_out: float
    8. p50_latency_ms: int # estimate used for cost/latency trade-offs
    9. LITE = ModelTier("lite", 0.5, 1.5, 350)
    10. STANDARD = ModelTier("standard", 3.0, 15.0, 900)
    11. REASONING = ModelTier("reasoning", 15.0, 60.0, 4200)
    12. def estimate_cost(tier: ModelTier, in_tok: int, out_tok: int) -> float:
    13. """Conservative estimate based on output length: generation length is
    14. unpredictable, so taking the upper bound is better for quota control."""
    15. return (in_tok * tier.cost_per_mtok_in + out_tok * tier.cost_per_mtok_out) / 1e6
    16. def route(tier_hint: str,
    17. tenant_plan: str,
    18. remaining_budget: float,
    19. must_use_tools: bool,
    20. latency_slo_ms: int) -> ModelTier:
    21. """Pick the actual tier from task constraints, tenant plan, and remaining budget."""
    22. tier = {"lite": LITE, "standard": STANDARD, "reasoning": REASONING}[tier_hint]
    23. # Budget fallback: when the balance is exhausted, drop to a usable tier
    24. # instead of making the whole account unavailable.
    25. if remaining_budget <= 0 and tenant_plan != "enterprise":
    26. return LITE
    27. # Latency SLO outranks quality: timing out hurts the experience more
    28. # than an imperfect answer does.
    29. if tier.p50_latency_ms > latency_slo_ms:
    30. tier = STANDARD if STANDARD.p50_latency_ms <= latency_slo_ms else LITE
    31. # Workloads that require tool calls have stricter instruction-following
    32. # needs, so they may not be downgraded to the lite tier.
    33. if must_use_tools and tier is LITE:
    34. tier = STANDARD
    35. return tier

    The numbers above are illustrative only and do not represent the pricing or performance of any real product.

    2.2.1 Open-Weight Models vs. Hosted APIs

    If a team decides to self-host open-weight models (several high-quality open-weight models are already available in the industry), three kinds of work come with it: throughput optimization of the inference engine (continuous batching, paged KV cache management, prefix cache reuse), capacity planning (provisioning accelerators for peak concurrency rather than average), and operational complexity (version upgrades, quantized-precision rollback, failure-domain isolation).

    The value of a hosted API is elasticity plus offloaded operations; the price is unit cost and control. The practice at most mature platforms is hybrid: high-frequency, short-output, standardization-tolerant workloads run on self-hosted lighter models, while complex long-chain tasks use hosted capabilities. This also explains why a routing layer is a must-have rather than an optimization.


    3. Model Versioning and Canary Rollouts: Making Capability Upgrades Reversible

    Model capability keeps iterating, but interfaces must not keep breaking. Three engineering conventions are non-negotiable here.

    3.1 Version Snapshots vs. Logical Aliases

    • Logical aliases (such as chat, chat-lite): caller-facing, meaning “the currently recommended capability.”
    • Snapshot versions (a specific weight/configuration with a date suffix): internal-facing, meaning “reproducible, deterministic behavior.”

    Callers must always be allowed to request a specific snapshot version explicitly for deterministic output — a hard requirement for evaluation regression, legal traceability, and reproducing production issues.

    3.2 Shadow Traffic and Staged Canary Rollouts

    The correct order for switching models is: offline evaluation set → shadow traffic (real requests mirrored to the new model, responses discarded, metrics collected) → small-percentage canary → full rollout. The canary dimensions should support at least a three-way intersection of tenant ID hash, plan tier, and capability type, so you do not discover a regression on some class of very-long-context requests only after a full rollout.

    1. import hashlib
    2. ALIASES = {
    3. "chat": {"primary": "chat-20260301-snap", "canary": "chat-20260915-snap"},
    4. }
    5. def resolve_snapshot(alias: str, tenant_id: str, canary_ratio: float) -> str:
    6. """Stable hashing on tenant ID splits canary traffic so the same tenant
    7. gets a consistent experience."""
    8. cfg = ALIASES[alias]
    9. if cfg["canary"] is None or canary_ratio <= 0:
    10. return cfg["primary"]
    11. bucket = int(hashlib.sha256(tenant_id.encode()).hexdigest()[:8], 16) % 10000
    12. return cfg["canary"] if bucket < canary_ratio * 10000 else cfg["primary"]

    3.3 Release Gates

    Before a new version enters a larger canary window, it should at minimum clear: task accuracy or win rate (human or model judging), refusal rate, average output length (to catch “getting more verbose” and the resulting cost drift), TTFT and end-to-end P99 latency, and tool-call success rate. Drift in average output length is the easiest to overlook, yet it directly inflates the bill.


    4. Metering: Reconstructing Inference as Structured Usage Events

    4.1 Metering Dimensions

    A complete usage event should contain at least:

    • Tenant and account identity (for billing attribution);
    • Request idempotency key (to prevent double charging on retries);
    • Physical model and logical tier;
    • Input token count, including the tokens that hit the prefix cache;
    • Output token count, plus the reasoning token count for reasoning models (usually billed as output, so it must be listed separately);
    • Number of tool/function calls and external retrievals;
    • Request status (success / parameter error / throttled rejection / server-side failure).

    Key point: the billing policy for failed requests must be defined explicitly. The common industry practice is not to charge for parameter errors and throttling rejections, and to charge partially (or not at all) for server-side timeouts based on cost already incurred — but whichever you choose, encode it in the data model rather than scattering it across code branches.

    4.2 The Event Table and Idempotency

    1. CREATE TABLE usage_event (
    2. id BIGINT NOT NULL AUTO_INCREMENT,
    3. idem_key CHAR(32) NOT NULL, -- sha256(tenant_id + request_id)
    4. tenant_id VARCHAR(64) NOT NULL,
    5. account_id VARCHAR(64) NOT NULL,
    6. period CHAR(7) NOT NULL, -- billing period YYYY-MM
    7. model_snapshot VARCHAR(64) NOT NULL,
    8. model_tier VARCHAR(16) NOT NULL,
    9. input_tokens INT NOT NULL,
    10. cached_tokens INT NOT NULL DEFAULT 0,
    11. output_tokens INT NOT NULL,
    12. reason_tokens INT NOT NULL DEFAULT 0,
    13. tool_calls INT NOT NULL DEFAULT 0,
    14. status TINYINT NOT NULL, -- 0=ok 1=reject 2=error
    15. created_at DATETIME NOT NULL,
    16. PRIMARY KEY (id),
    17. UNIQUE KEY uk_idem (idem_key),
    18. KEY idx_tenant_period (tenant_id, period)
    19. );

    UNIQUE KEY uk_idem is the cornerstone of the whole chain: gateway retries, message-queue redeliveries, and re-run batch jobs will never cause double billing. After events are produced they are written to a message queue and consumed by a separate settlement service that aggregates them, keeping the inference path and the billing path in separate failure domains — an outage in the billing system should not make live inference unavailable.


    5. Entitlement and Quota: Expressing Free, Subscription, and Pay-As-You-Go in One Model

    5.1 Abstracting the Entitlement Model

    The three commercial shapes look different, but once abstracted they share one structure: a per-period quota pack + a set of concurrency caps + an overage policy.

    Shape Quota Source Concurrency Cap Overage Policy
    Free / trial Fixed token allowance, reset per period Low Hard reject (429)
    Subscription Token allowance by tier + optional model scope Medium Downgrade to lite, or roll over to pay-as-you-go
    Pay-as-you-go No preallocated quota; constrained by balance and credit By account level Reject when balance is insufficient (402/429)
    1. CREATE TABLE entitlement (
    2. tenant_id VARCHAR(64) NOT NULL,
    3. period CHAR(7) NOT NULL,
    4. plan VARCHAR(32) NOT NULL, -- free / basic / pro / payg
    5. quota_tokens BIGINT NOT NULL,
    6. used_tokens BIGINT NOT NULL DEFAULT 0,
    7. max_tiers VARCHAR(128) NOT NULL, -- set of allowed tiers
    8. concurrency_cap INT NOT NULL,
    9. overage_policy VARCHAR(16) NOT NULL, -- deny / downgrade / payg
    10. status TINYINT NOT NULL, -- 1=active 0=frozen
    11. PRIMARY KEY (tenant_id, period)
    12. );

    The max_tiers and overage_policy columns turn “plan differences” into configuration rather than code branches. Adding a new plan usually means inserting a single entitlement row.

    5.2 Concurrency Quota vs. Budget Quota

    This is the most commonly confused point in practice:

    • Concurrency quota (in-flight): caps the number of requests simultaneously in inference. It protects GPU memory and KV cache capacity and has no direct relation to token counts.
    • Budget quota (TPM, tokens per minute): caps total tokens consumed per unit of time. It protects throughput and cost.
    • Rate quota (RPM): caps request count, used to fend off scraping and abuse.

    All three must exist at once. Capping concurrency alone lets a single very long request hold a slot indefinitely; capping token rate alone lets a flood of short requests fill the scheduling queue. Both belong in the admission layer.


    6. Admission Control and Graceful Degradation: The First Line of Defense Under Traffic Spikes

    1. import asyncio
    2. import time
    3. class TokenBucket:
    4. """Budget token bucket: decremented by estimated token cost, not by request count."""
    5. def __init__(self, tokens_per_sec: float, capacity: float):
    6. self.rate = tokens_per_sec
    7. self.capacity = capacity
    8. self.tokens = capacity
    9. self.updated = time.monotonic()
    10. self.lock = asyncio.Lock()
    11. async def try_acquire(self, cost: float) -> bool:
    12. async with self.lock:
    13. now = time.monotonic()
    14. self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
    15. self.updated = now
    16. if self.tokens < cost:
    17. return False
    18. self.tokens -= cost
    19. return True
    20. PRIORITY = {"free": 0, "basic": 1, "pro": 2, "payg": 1, "enterprise": 3}
    21. async def admit(req, tenant_state, buckets: dict, inflight: asyncio.Semaphore):
    22. """Unified admission: priority -> budget -> concurrency; any gate that
    23. fails returns an explicit error code."""
    24. tier = route(req.tier_hint, tenant_state.plan,
    25. tenant_state.remaining_budget, req.must_use_tools,
    26. req.latency_slo_ms)
    27. # 1) Budget gate: estimate cost rather than measuring it, err on the
    28. # conservative side.
    29. est = estimate_cost(tier, req.estimated_input_tokens, req.max_output_tokens)
    30. if not await buckets[tenant_state.plan].try_acquire(est):
    31. # Subscribers may degrade and continue; free users are rejected.
    32. if tenant_state.overage_policy == "downgrade":
    33. tier = LITE
    34. est = estimate_cost(tier, req.estimated_input_tokens, req.max_output_tokens)
    35. if not await buckets["free"].try_acquire(est):
    36. return {"code": 429, "msg": "quota exceeded", "retry_after": 5}
    37. else:
    38. return {"code": 429, "msg": "rate limited", "retry_after": 2}
    39. # 2) Concurrency gate: non-blocking try, so low-priority requests fail fast
    40. # instead of queueing and dragging out latency.
    41. priority = PRIORITY.get(tenant_state.plan, 0)
    42. if inflight.locked() and priority < 2:
    43. return {"code": 503, "msg": "service busy, try lite model", "fallback": LITE.name}
    44. async with inflight:
    45. return await call_model(tier, req)

    6.1 The Degradation Ladder

    When capacity gets tight, degrade step by step in a fixed order, with every step behind a configuration switch that can be rolled back independently:

    1. Turn off non-core capabilities: long context, web retrieval, and multimodal input first.
    2. Cap output length: dynamically lower max_tokens, the most direct lever on cost.
    3. Push down tiers: standard tier → lite tier.
    4. Queue and prioritize: paid tenants first; free tenants queue or fail fast.
    5. Suspend new entitlement grants: applies only to new accounts and new subscriptions; existing users are unaffected.

    Step 5 deserves separate discussion. “Pausing new subscriptions” is essentially an operational action, and in engineering terms it should be a configuration change to an entitlement-granting switch, not a code release. That requires: entitlement granting and entitlement consumption to be fully decoupled; the switch to support canary scope (by channel, region, or registration date); expiry and renewal policies for existing entitlements to stay unchanged during the pause and be announced in advance; and the change to be captured in audit logs. If this action requires a release to execute, it will never be fast enough under real business pressure.

    6.2 Queue and Timeout Discipline

    • Queues must be bounded. An unbounded queue only drives latency toward infinity under overload and ends in a collapse.
    • The remaining budget of a queued request decays over time (deadline propagation); a request whose deadline has passed is dropped on dequeue instead of being pushed through inference anyway.
    • When clients receive 429/503 they must get a Retry-After, and the server side should add backoff jitter to retries to avoid retry storms.

    7. Observability and Unit Economics

    7.1 Metrics by Layer

    • Experience layer: TTFT (time to first token), TPOT (inter-token generation interval), end-to-end P50/P95/P99, success rate, degradation trigger rate.
    • Capacity layer: accelerator utilization, in-flight concurrent requests, KV cache occupancy, queue depth and queueing time, actual batch size.
    • Cost layer: cost per thousand requests, cost per million tokens, cache hit rate, average output length, share of free traffic.
    • Business layer: plan conversion rate, overage rate, gross margin per tenant.

    7.2 Dashboarding Unit Gross Margin

    Drilling revenue - inference cost - storage and bandwidth cost down by tenant, tier, and capability type is the only reliable way to tell which features are losing money. A common finding is that free batch long-text jobs and repeated system prompts that miss the cache contribute a disproportionate share of cost. The corresponding optimizations are equally clear — enforce prefix caching, cap the maximum output length for free-tier models, and structure reuse for high-frequency repeated prompts.

    7.3 Attribution Requirements

    All billing-path data must trace back to a single request. Retain at least one full billing period of usage detail, and perform dual-track reconciliation on monthly rollups (aggregated event stream vs. debited amount in the business database). Numbers that stay inconsistent over time usually mean there is an unmetered side path for calls.


    8. Where Security and Compliance Land in Engineering

    1. Multi-tenant isolation: vector stores, session storage, and log systems must all filter by tenant dimension, and the filter should be injected mandatorily at the data access layer rather than relying on business code to remember it.
    2. Content safety: prompt injection and sensitive-content detection on the input side; policy validation and streaming-fragment detection on the output side. Streaming scenarios need “review while generating” — on a policy hit, interrupt the stream and return an explicit reason.
    3. Data minimization and masking: never log raw prompts and completions; use hashing or masking. An opt-out from training use should be the default, not an option.
    4. Audit: who used which model snapshot and when, how much usage was generated, and which entitlement changes were made — all traceable.
    5. Open-weight license compliance: if you self-host inference, verify the weight license terms regarding commercial use, redistribution, and derivative models, and bake the conclusion into the release checklist.

    9. Rollout Roadmap and Checklist

    Phase 1 (within 1 month): measurable. Build a unified gateway and connect the usage event table, idempotency keys, and offline aggregation jobs. The goal is singular: the bill can be computed correctly.

    Phase 2 (2–3 months): governable. Introduce logical model names and a routing layer, add the budget token bucket and concurrency semaphore, and complete unified modeling for the free, subscription, and pay-as-you-go tiers.

    Phase 3 (3–6 months): optimizable. Prefix cache reuse, quantization and batching tuning, per-tenant unit-margin dashboards, and feature-flagged degradation ladders with drills.

    Phase 4 (ongoing): evolvable. Shadow traffic and canary rollouts become routine, model versions iterate on a quarterly cadence, and interface snapshots stay backward compatible for at least two version cycles.

    Pre-Launch Self-Check

    • Do retries and queue redeliveries avoid double billing?
    • Are the billing policies for server-side failures, parameter errors, and throttling rejections explicitly defined?
    • Are the inference path and the billing path isolated into separate failure domains?
    • Do concurrency quota and token budget quota both exist?
    • Does every step of the degradation ladder have its own switch, and has a failure drill been run?
    • Can “suspend new entitlements” be executed through a configuration change without affecting existing users?
    • Can single-tenant gross margin be drilled down by tier?
    • Are logs masked, and is tenant filtering injected at the data access layer?

    Conclusion

    The engineering difficulty of LLM serving can be summed up in one sentence: deliver a firm commitment to quality and experience while cost cannot be determined statically at the ingress.

    The answer is not a stronger model. It is turning cost into meterable events, capability into routable resources, quota into configurable policy, and degradation into flaggable actions. Once those four things are independently reversible, the pace of model iteration, the shock of traffic spikes, and adjustments to commercial strategy no longer have to be paid for with production stability.