Tiered LLM Model Matrices and Multi-Agent Collaboration: Engineering Practice

1. From a Single Flagship to a Capability Matrix: A Shift in Product Form

For the past two years, the industry has often understood foundation models through a single curve: larger parameter counts and greater training investment lead to stronger capability, and all users rely on the same strongest model. However, the latest iteration cycle has made a clear signal visible: the most capable model does not necessarily need to be delivered to everyone in the same form.

This observation is worth unpacking. It effectively challenges three assumptions:

• One-size-fits-all is not optimal: different tasks vary widely in their requirements for capability, latency, and cost. Using a single capability curve for all scenarios inevitably leads to resource waste.
• Capability and safety do not have to be bundled: the same base model capability can be protected by different safety measures for different deployment scenarios. There is no need to separately train a weakened version just to satisfy higher safety requirements.
• A version bump does not automatically mean proportional gain: the value of a new version is usually concentrated in improvements to specific task dimensions and lower unit cost, rather than across-the-board dominance.

For enterprise engineering teams, this shifts the architectural concern from “how to connect the strongest model” to “how to route requests correctly across models of different tiers.” This is primarily a routing and governance problem, not just a model problem.

2. Three Dimensions of a Capability Matrix

A mature tiered model matrix is usually sliced along three dimensions at the same time, rather than sorted by a single notion of “size”:

Dimension	Tiers	Typical Tasks	Key Constraints
Capability depth	Lightweight tier	Intent classification, field extraction, format conversion, batch data cleaning, simple Q&A	Throughput and unit price
Capability depth	Balanced tier	Document drafting, slide and table generation, scoped bug localization and repair, single-file refactoring	Quality stability
Capability depth	Deep tier	Cross-module architecture changes, complex decisions requiring careful judgment, review of high-risk changes	Latency and call cost
Risk boundary	Internal / Controlled / Public	Internal development tools / regulated business workflows / public-facing services	Refusal boundaries, audit granularity, data retention period
Delivery form	Synchronous interaction / asynchronous batch processing / persistent agents	Conversational assistants / offline processing / long-running pipelines	SLA, timeout, and retry strategy

The key point is that the same task may sit at the intersection of multiple tiers. For example, “a public-facing code repair request” may need the capability of the balanced tier, the safety boundary of the public tier, and the long timeout behavior expected by asynchronous batch processing. When designing the matrix, enterprises should first enumerate business tasks and then map them to intersections, rather than choosing a model first and forcing scenarios to fit it.

3. Shared Base Model plus Differentiated Safety Policies: Configuration, Not Retraining

From a capability reuse perspective, the most economical approach is to let multiple externally facing forms share the same base model capability and apply different safety constraints at the deployment layer. This approach has a clear advantage: when capability is updated once, all delivery forms benefit together. In contrast, repeatedly training “safety-weakened” variants introduces capability drift and the cost of maintaining parallel evaluation tracks.

To do this correctly, safety policies must be treated as configurable, versioned, and regression-testable parameters, not informal rules scattered inside prompts. A reasonable layered structure is shown below:

# Security policy example: applied by deployment profile
security_policy:
  version: "2026.09"          # Version policies independently from models
  profile: "public_facing"    # internal | controlled | public_facing
  input_guard:
    pii_detection: strict     # lenient | standard | strict
    injection_scan: enabled   # Detect prompt injection and unauthorized instructions
  output_guard:
    refuse_on_sensitive_topics: true
    require_citation_for_facts: true
    max_output_tokens: 4096
  tool_permissions:
    allow_filesystem_write: false
    allow_shell_exec: false
    network_egress: deny_list  # Allow only explicitly whitelisted destinations
  retention:
    prompt_log_days: 30
    audit_trace: full          # Record input summary, policy hits, and final action

Four disciplines are important in practice:

1. Policies must be bound to deployment form, not to the model version. The same model version may allow filesystem writes inside an internal development tool, but must disable them in a public-facing service. Putting permissions into configuration makes them visible to code review.
2. Policy hits must be observable. Every refusal, redaction, or instruction block should be written into the audit trail. Otherwise, policies drift silently, and after a few months no one knows where the boundary actually is.
3. Policy changes must be rolled out gradually. Tightening a policy can directly affect business success rates. Rollouts should proceed by traffic percentage, with a one-click rollback switch retained.
4. Policy regression testing must be separate from model evaluation. Model evaluation cares about accuracy. Policy evaluation cares about whether things that should be refused are refused and whether legitimate requests are not mistakenly blocked. These metrics are different and should not be mixed into the same test suite.

4. Long Context: Real Benefits, Real Costs

Expanding the context window from tens of thousands of tokens to the million-token scale changes task patterns, rather than simply making a model “remember more.”

Scenarios that become genuinely viable:

• Cross-repository code understanding: loading multiple modules, type definitions, and configuration files at once makes it possible to understand call chains without manual summarization.
• Structured processing of long documents: full-text comparison and review of contracts, annual reports, or specification documents reduces information fragmentation caused by chunking.
• Long-session consistency: multi-turn iterations can preserve design decisions, naming conventions, and previously rejected alternatives.

Costs that must be taken seriously:

• Effective attention decay: it is widely observed in the industry that when context becomes extremely long, models use “middle-of-the-context information” less effectively. Loading 800,000 tokens into the prompt does not mean the model truly understands all 800,000 tokens.
• Linear increases in cost and latency: input tokens are a major billing driver. Long contexts can push the cost of a single call up by an order of magnitude and significantly increase time-to-first-token.
• Expanded attack surface: the more external content that can be loaded, the more entry points exist for prompt injection. Injection detection recall often degrades in long texts compared with short ones.
• Signal-to-noise issues: irrelevant content consumes attention budget and can actually reduce accuracy.

The correct engineering posture is load-on-demand, not dump-everything:

def build_context(task, kb, budget_tokens=120_000):
    """
    Context budget management: allocate fixed quotas to different components so that
    retrieval results do not crowd out task instructions and tool definitions.
    """
    reserve = {
        "system_instruction": 2_000,   # Role, constraints, and output format
        "tool_schemas":       6_000,   # Definitions of callable tools
        "retrieved_evidence": int(budget_tokens * 0.55),
        "scratch_space":      int(budget_tokens * 0.15),  # Intermediate reasoning and artifacts
    }
    used = sum(reserve.values())
    reserve["retrieved_evidence"] += budget_tokens - used
    evidence = kb.search(task.query, limit=200)
    packed, tokens = [], 0
    for chunk in evidence:                     # Place high-scoring chunks first
        if tokens + chunk.size > reserve["retrieved_evidence"]:
            break
        packed.append(chunk.render_with_citation())
        tokens += chunk.size
    return {
        "system": load("instruction/" + task.profile),
        "tools": load_tool_schemas(task.allowed_tools),
        "context": packed,
        "budget_left": budget_tokens - tokens,
    }

Four practical rules follow:

• Retrieve and structure first, then hand the result to the model.
• Include only cited snippets, and require citations in the output.
• Use rolling summarization for long-running tasks, preserving decisions and conclusions while discarding process noise.
• Split extremely long tasks into multi-stage pipelines, instead of relying on one extremely long input.

5. Multi-Agent Collaboration: From One-Shot Generation to Team-Style Pipelines

Transforming single-model one-shot generation into a pipeline where multiple roles collaborate is a mainstream approach for improving success rates on complex tasks. It works because failures on complex tasks often do not stem from insufficient capability, but from context confusion: requirements, implementation details, and test feedback are all forced into a single generation pass, making it easy for the model to lose track of priorities.

An effective multi-agent design needs to solve four problems:

1. Role boundaries must be clear and non-overlapping. A commonly effective division is: requirement clarification and planning, implementation, independent review, test execution, and integration. The reviewer must maintain an independent perspective. If the implementer reviews their own work, they will almost inevitably confirm their own mistakes.

2. Hand off shared artifacts, not chat history. What passes between roles should be files, patches, test reports, and structured task cards, not increasingly long conversation logs. Artifacts can be verified, rolled back, and inspected by humans. Conversation history only leads to context bloat and accumulated misunderstanding.

3. The judge must be deterministic. Whether one implementation is better than another should not be decided by models scoring each other. It should be decided by deterministic signals such as compilation, unit tests, static analysis, and type checking. Entrusting pass/fail decisions to executable verification is a dividing line for multi-agent system reliability.

4. Define serial and parallel boundaries clearly. Read-only analysis can run in parallel. Tasks that write to the same files must run serially, with a dedicated integration role merging the changes. Otherwise, conflict resolution costs may exceed the benefits of parallelism.

def run_task_pipeline(task, client, max_rounds=3):
    plan = client("planner").run(task.spec)                 # Produces a structured task card
    history = []                                            # Store only artifact references
    for round_no in range(max_rounds):
        patch = client("implementer").run(plan, context=history)
        review = client("reviewer").run(patch, perspective="independent")
        report = run_tests(patch)                            # Deterministic judge
        if report.passed and review.blocking == 0:
            return merge(patch, artifacts=history, rounds=round_no + 1)
        plan = client("planner").revise(
            plan, failures=report.failures, review=review.notes
        )
        history.append(artifact_ref(patch, report, review))
    escalate_to_human(task, last_artifacts=history)          # Failures must be handed off to a human
    return None

One thing must be recognized clearly: multi-agent systems are not a free lunch. They turn a single call into more than ten calls, multiplying cost and end-to-end latency. They also introduce the risk of error amplification: incorrect conclusions produced by an earlier role may be treated as established facts by later roles. Therefore, multi-agent pipelines are suitable only for genuinely complex tasks that can be automatically verified. For well-scoped routine tasks, a single call to a balanced-tier model is often faster and cheaper.

6. General Engineering Techniques for Speed and Cost Reduction

Beyond raw capability, “faster and cheaper” is often the hardest criterion in enterprise model selection. Mature industry techniques can be grouped into six categories, ordered by return on investment:

1. Task routing and traffic splitting: send simple tasks to the lightweight tier, and invoke the deep tier only when escalation conditions are met. This is the optimization with the greatest benefit and the lowest risk.
2. Prefix cache reuse: place stable system instructions, tool definitions, and long reference documents in the prompt prefix. Repeated portions can hit the cache, significantly reducing input-side overhead.
3. Asynchronous batch processing: non-real-time batch tasks, such as log processing, offline summarization, and document conversion, can run through queues asynchronously. Message queues can smooth traffic peaks and handle retries, improving both throughput and cost.
4. Output length governance: limit unnecessary reasoning output and redundant explanations. Output tokens are often an overlooked cost item.
5. Streaming responses: streaming does not reduce total cost, but it significantly improves perceived latency by improving time-to-first-token. It is a cost-effective experience optimization.
6. Small-model replacement and distillation: for high-frequency, low-complexity tasks, distill dedicated small models using business data to replace general-purpose calls.

Observability is equally important. It is recommended to record input and output tokens, cache-hit ratio, routing decisions, retry counts, and final success rate by task ID, and send them to a centralized logging service. Then configure alerts for unit cost, P95 latency, and failure rate. Without this data, claims such as “reduce cost by 30%” cannot be verified or attributed.

7. Evaluation and Acceptance: Do Not Let Leaderboards Make the Decision

Public code-oriented benchmark scores have reference value, but they should not be the sole basis for selection. There are three reasons:

• Training data contamination is difficult to rule out.
• Task distributions differ significantly from real business traffic.
• Single-pass success rate does not reflect actual delivery quality after multi-round rework.

A three-layer evaluation system is recommended:

• Layer 1: Public benchmarks for trend only. Use them to check whether a new version has clearly regressed, but do not let them drive decisions.
• Layer 2: A self-built business evaluation set for decisions. Sample 300–1,000 real online requests, manually annotate expected outputs, and cover both high-frequency and edge-case scenarios. Every model switch must trigger a full regression. Metrics should include task success rate, rework rate, and human takeover rate.
• Layer 3: Online canary releases to reveal the truth. Roll out by traffic percentage and judge using real business indicators such as delivery cycle time, review pass rate, and user adoption rate.

The acceptance checklist should remain fixed at six items: task success rate, human takeover rate, P50/P95 latency, average cost per task, cache-hit rate, and policy false-positive rate. Looking at all six together helps avoid situations where one metric is optimized while overall experience degrades.

8. Implementation Roadmap and Four Common Pitfalls

Recommended four-step closed-loop roadmap:

1. Task inventory: list all tasks involving AI, and annotate frequency, tokens per request, tolerable latency, and cost of errors.
2. Tier mapping: map tasks to lightweight, balanced, or deep tiers based on error cost and latency requirements, and mark the risk boundary for each.
3. Routing and fallback: implement a routing layer and design a fallback chain. If the deep tier times out or fails, automatically fall back to the balanced tier and notify users, avoiding business interruption.
4. Observability loop: send cost and quality data back into the system, rerun the evaluation set periodically, and adjust routing thresholds based on data.

This architecture can be built with common cloud capabilities: inference services can be deployed on container platforms with elastic scaling, corpora and artifacts can be stored in object storage, asynchronous tasks can be scheduled by message queues, and cost and quality data can flow into logging services with monitoring and alerting closing the loop. On Baidu AI Cloud, these common compute, storage, queue, and observability capabilities can also be reused directly without additional changes specific to model tiering.

Four pitfalls to avoid:

• “Newer is always better”: improvements in a new version are often concentrated in specific dimensions. General capability may stay flat or even regress. Always validate against a self-built evaluation set.
• “Longer context is always better”: effective attention decay and rising costs can offset the benefits of a large context window. Retrieval-based on-demand loading is usually better.
• “Stronger safety policies are always better”: excessive refusal and redaction can significantly increase false positives. Policies should match the deployment form and be continuously calibrated with metrics.
• “Multi-agent is always better”: cost and end-to-end latency rise sharply. Multi-agent pipelines are suitable only for complex tasks that can be automatically verified.

Core conclusion: The competitive dividing line for large-model applications has shifted from “whether to use the strongest model” to “whether an organization can turn different capability tiers, different safety boundaries, and different delivery forms into an observable, gradual, and reversible system.” The capability matrix is the mechanism. Governance and evaluation are the long-term moat.