AI Marketplace Platform Architecture: Model Distribution and Inference Serving in Practice

1. First, Translate “Intermediary” Into Technical Terms

An “AI intermediary” is not just another model vendor. It is an engineering middle layer that connects distributed model supply (model authors, open weights, adapter files, evaluation sets, prompt templates) with distributed demand (developers, SMBs, industry applications). What justifies the high valuations investors assign to these platforms is not how many weight files they hold, but that they have productized three genuinely hard problems: hosting and governing model assets, serving inference across heterogeneous compute, and running a commercial pipeline for usage-based billing and revenue sharing.

Mapped onto architecture, this system naturally splits into four layers:

Layer	Core Responsibility	Key Engineering Challenges
Asset	Registration, versioning, and lineage of models/adapters/datasets/templates	Large-file storage, immutable versions, deduplication, license tracking
Runtime	Multi-model, multi-tenant inference, queuing, batching, preemption	VRAM isolation, long-tail latency, cold starts, GPU utilization
Abstraction	Normalizing heterogeneous backends into a stable contract, routing, failover	Semantic alignment, capability detection, fault-domain isolation
Metering & Billing	Usage collection, quotas and rate limiting, invoicing, revenue-share reconciliation	Billing accuracy, idempotency, dispute traceability

If any single layer is missing, the platform degrades into a “reskinned proxy” — it can neither sustain scale nor settle accounts correctly.

2. Asset Layer: Model Governance Is the Platform’s Foundational Capability

2.1 Separating Large Files From Metadata

Model weights easily run from a few GB to tens of GB — an entirely different problem class from ordinary business data. Engineering-wise, you must manage binary large objects and metadata separately:

• Store large objects in object storage using multipart upload, resumable transfer, and concurrent part merging; verify the digest once the upload completes.
• Store metadata (name, task type, input/output spec, license, source, evaluation metrics, supported hardware, recommended batch size) in a relational database for search and filtering.
• Use content-addressed deduplication on identical digests to avoid writing the same weights to disk repeatedly across tenants and replicas.

Object storage layout (illustrative)
bucket/models/<publisher>/<model>/<version>/
    ├── manifest.json        # format, quantization, parameter count, required file list
    ├── weights/*.safetensors
    ├── adapter/*.safetensors # adapters (lightweight incremental weights)
    └── checksums.txt

2.2 Immutable Versions and Lineage

The model catalog must follow an “immutable version“ principle: once published, neither the path nor the content changes, and fixes are shipped as new versions. The reason is straightforward — callers’ caches, audit records, and regression evaluations all depend on a stable identifier; mutable versions make production incidents impossible to reproduce.

On top of that, build lineage: a derived model (base model + adapter + training set + fine-tuning script) should trace back to all of its upstreams. This is both a licensing-compliance requirement and the foundation for quality attribution and incident investigation.

2.3 Distribution Pipeline

Models need to be “pushed as close to the compute as possible.” The common pattern is tiered caching: central object storage as the single source of truth, with local high-performance cache volumes mounted on training/inference nodes. Nodes pre-fetch and verify per the manifest at startup, then load read-only from the local directory service on cache hits. This both avoids thousands of Pods hammering the origin simultaneously and brings cold-start time down from “minutes” to an acceptable range.

3. Runtime Layer: Multi-Tenant Serving on Expensive, Scarce GPUs

The biggest difference between GPU inference and CPU services is that GPU resource granularity is coarse, exclusivity is strong, and unit cost is high. This dictates the core goal of runtime design: maximize reuse of VRAM and compute without sacrificing tenant isolation.

3.1 Tier Requests — Don’t Serve All Traffic With One Model Pool

At minimum, separate three traffic classes into different scheduling pools:

• Online/interactive: latency-first, on dedicated reserved instances, with preemption disabled.
• Near-line batch: tolerates second-to-minute latency, runs on a shared pool with continuous batching, allows queuing and preemption.
• Offline jobs: image/video generation, bulk embedding, etc., fully delegated to an async task queue and run to fill off-peak troughs.

Mixing long-running jobs into the online pool is the most common root cause of out-of-control long-tail latency.

3.2 VRAM and Batching Strategy

• Use VRAM partitioning or time-slicing to host multiple small models on the same physical card and improve fragment utilization.
• For text inference, enable dynamic batching and continuous batching so new requests can join an in-flight batch instead of waiting for the whole batch to finish.
• For image inference, cap per-request resolution and step counts to prevent a single request from holding a GPU indefinitely.
• VRAM is a hard constraint, so admission control must estimate requirements before enqueueing; if the estimate fails, reject immediately with a clear error code rather than letting the job fail mid-run on an out-of-memory error — the latter wastes the most resources.

3.3 Scale on “Queue Metrics,” Not CPU

CPU utilization on GPU nodes is often persistently low, so it is meaningless as a scaling signal. More effective signals are: queue length, estimated queue wait time, GPU VRAM utilization, and request failure/timeout rate. Combined with node-level warmup (a new node loads high-frequency models before its ready flag is set), this suppresses the jitter introduced by scaling events.

3.4 Cost Governance

Reserving the online pool at peak and filling idle compute with the offline pool is the baseline for improving overall utilization. Beyond that, three things are needed: per-invocation compute cost attribution (millisecond-level GPU instrumentation), per-tenant quotas and budget caps, and a degradation path for low-value requests (switch to a smaller model, lower resolution, return a cached result). If cost is invisible, gross margin is uncontrollable.

4. Abstraction Layer: Consolidate Heterogeneous Backends Behind a Unified Contract

In reality, a platform is hybrid: part self-hosted inference clusters, part managed model services, part third-party open-weight deployments. If every caller had to be aware of backend differences, the platform would lose its reason to exist. The approach is to define a task-semantic unified contract and confine backend differences to adapters.

from abc import ABC, abstractmethod
from dataclasses import dataclass
@dataclass
class InferenceRequest:
    task: str                    # canonical task semantics: text_generate / image / embedding / rerank
    payload: dict
    deadline_ms: int = 2000
    cost_budget: float = 0.0
@dataclass
class InferenceResult:
    backend: str
    output: bytes | str
    in_units: int = 0            # input metering unit (tokens / images / rows)
    out_units: int = 0
    gpu_ms: float = 0.0          # smallest unit for billing and observability
class Backend(ABC):
    name: str
    @abstractmethod
    def health(self) -> bool: ...
    @abstractmethod
    def capabilities(self) -> set[str]: ...
    @abstractmethod
    def infer(self, req: InferenceRequest) -> InferenceResult: ...
class Router:
    """Policy-based routing: three tiers — latency / cost / quality; auto-failover when a backend is unhealthy."""
    def __init__(self, backends: list[Backend], strategy: str = "balanced"):
        self.backends = backends
        self.strategy = strategy
    def pick(self, req: InferenceRequest) -> list[Backend]:
        cands = [b for b in self.backends
                 if b.health() and req.task in b.capabilities()]
        key = {"latency": lambda b: b.name,      # populated from live probe data
               "cost":    lambda b: b.name,
               "quality": lambda b: b.name}[self.strategy]
        return sorted(cands, key=key)            # return ranked candidates to support failover

Three details must land in the design:

1. Capability detection: backends differ in supported resolution, context length, and quantization format, so routing must filter on these first — otherwise you get “the request succeeded but the result is non-compliant.”
2. Fault-domain isolation: a single tenant’s traffic must not all route to one backend pool; shard by tenant or by request hashing so a single point of failure doesn’t escalate into a global outage.
3. Semantic compatibility layer: unifying interface signatures doesn’t mean output semantics are unified. You need a post-processing layer that aligns return structures, units, and precision across backends, and raises an explicit error on alignment failure rather than silently degrading.

5. Long-Running Task Orchestration: Image/Video Generation Is Not a Single HTTP Request

Generation tasks take anywhere from a few seconds to several minutes. Serving them via synchronous requests inevitably leads to timeouts and idle resources. The standard approach is a three-part submit → queue → callback/poll flow:

STAGES = [
    {"id": "validate",     "impl": "validate_params",   "retry": 1},
    {"id": "moderate_in",  "impl": "moderate_input",    "retry": 2},
    {"id": "infer",        "impl": "run_model",         "retry": 2, "timeout_s": 120},
    {"id": "post",         "impl": "upscale_and_store", "retry": 3},
    {"id": "moderate_out", "impl": "moderate_output",   "retry": 2},
]
# Convention: each stage's I/O is stored by reference (URI + digest) in object storage;
# stage implementations must be idempotent; failures are retried at stage granularity, reusing already-completed nodes.

Key points include: DAG nodes must be idempotent (re-entering the same task_id + stage produces no side effects); timeout and retry policies differ per stage (for content-moderation stages, retrying is pointless, so fail fast); artifacts are uniformly stored in object storage and returned as signed accessible URLs rather than inlined as binaries; and both Webhook callbacks and polling are supported to accommodate callers with different capabilities.

6. Multi-Tenant Isolation and Security: Not Optional

A platform hosts other people’s models and other people’s (or end users’) inputs. Poor isolation is a security incident.

• Resource isolation: namespace + quota + dedicated service accounts; on the GPU side, prevent any single tenant from consuming all resources through three limits — VRAM cap, per-task timeout, and concurrency limit.
• Execution isolation: user-defined processing nodes, scripts, and workflows must run inside a controlled sandbox that restricts network egress, filesystem write scope, available syscalls, and CPU time slices.
• Content safety: audit both input and output; generated content needs auditable records (request ID, model version, policy version, disposition) with strictness configurable per tenant.
• Data boundaries: define clearly whether data is used to improve models, how long it is retained, and whether it crosses tenants. Cross-tenant cache reuse may only rest on “byte-identical content with a digest hit,” and must always respect a tenant’s no-cache declaration.
• Permission model: a unified auth gateway handles identity verification, tenant identification, quota pre-deduction, and usage tagging, so business services no longer parse credentials themselves.

7. Metering and Billing: The Real Technical Moat

Billing accuracy directly determines whether a platform can operate long-term. There are a few hard constraints in the design:

USAGE_EVENT = {
    "event_id": "uuid",          # idempotency key; duplicate deliveries are not billed twice
    "tenant_id": "...",
    "model_id": "...", "model_version": "...",   # version participates in pricing
    "task_type": "image",
    "unit": "image", "qty": 1,
    "gpu_ms": 1230.4, "wall_ms": 4100,
    "status": "succeeded",       # whether failures are billed must be defined and written into the contract
    "started_at": ..., "finished_at": ...,
}
# Pipeline: client instrumentation → message queue (at-least-once) → streaming aggregation & dedup → usage detail store
# Invoicing: aggregate per billing period → reconcile against gateway-side pre-deduction records → alert when variance exceeds threshold

• Usage events must be idempotent: the “at-least-once” semantics of a message queue mean duplicate deliveries are inevitable, and deduplication by event_id is the only reliable solution.
• Metering units must align with business pricing: text by token, images by count, video by seconds — mixing units makes accounts impossible to explain.
• Close the pre-deduct/actual-charge loop: the gateway pre-deducts a quota against an upper bound, then reconciles the difference by actual usage after the task finishes, so a long task doesn’t discover it’s over budget only after completion.
• The failure-billing policy must be written into the contract: whether to charge in full, partially, or not at all when VRAM is occupied but generation fails is a commercial term — the technical side only needs to guarantee accurate, traceable tagging.
• Creator revenue shares need reconcilable detail: per-model call volume, reference counts, and revenue attribution should all be recomputable from raw usage events; otherwise settlement disputes are unresolvable.

8. Observability and SLO Tiering

GPU service observability has one more layer than a typical web service: you must watch both request metrics and hardware metrics.

• Request side: QPS, success/failure/timeout rate, queue wait time, end-to-end P50/P95/P99.
• Compute side: GPU utilization, VRAM usage and fragmentation, card temperature and throttling, node readiness rate.
• Cost side: GPU-millisecond cost per request, each tenant’s cost share, idle-compute ratio.
• Pipeline side: stitch gateway → scheduling → inference → post-processing → storage together by request ID, so long-tail issues can be traced to a specific stage.

SLOs are committed per traffic tier: interactive traffic is measured primarily on latency, while batch traffic is committed in terms of completion time — mixing both into a single SLA hurts both sides. Alerts are tiered by “user-perceptible impact”; hardware-metric anomalies serve only as diagnostic information, not as a direct page.

For general-purpose capability selection, these needs can be met by composing common cloud service categories: object storage for models and artifacts, a container platform for the inference runtime, a message queue for async tasks and usage events, log and monitoring/alerting services for observability, and serverless form factors for low-frequency or bursty tasks. Baidu AI Cloud, like other cloud providers, generally offers these as common capability categories; for specific product capability boundaries and specs, refer to the official documentation.

9. Rollout Roadmap and Common Anti-Patterns

A three-phase approach is recommended:

• Phase 1 (close the loop): asset registration + inference serving for a single task type + basic auth and usage instrumentation. The goal is “can host, can call, can meter.”
• Phase 2 (scale): multi-backend adapters and policy routing, request tiering and autoscaling, async task orchestration, content safety, SLO and cost dashboards.
• Phase 3 (ecosystem): creator workbench and revenue-share settlement, derived-model lineage and license compliance, private/dedicated deployment, and evaluation- and leaderboard-driven quality governance.

Finally, a few recurring anti-patterns worth checking line by line during review:

1. Treating “proxying someone else’s API” as a platform, with no asset governance or metering capability.
2. Mutable model versions, making production incidents impossible to reproduce and evaluation conclusions untraceable.
3. Long-running jobs sharing a scheduling pool with interactive requests, leaving long-tail latency unsolvable.
4. Validating resources only after enqueueing, so VRAM overflows waste large amounts of compute.
5. Non-idempotent usage events, so bills and gateway records never reconcile.
6. Cross-tenant caching without tenant-declaration constraints, creating a data-boundary risk.
7. Scaling on GPU utilization alone while ignoring queue wait time, so scaling is always a half-step late.

In the end, an “intermediary’s” value comes down to two things: extracting higher efficiency from scarce compute, and packaging complex model serving into stable, dependable contracts. The former relies on the engineering details of scheduling and isolation; the latter on the rigorous design of the abstraction and metering systems. Only when both are done does the valuation have a matching technical foundation to stand on.