Context Governance and Evaluation for Multi-Agent LLM Collaboration
This article walks through how to build and operate multi-agent LLM systems after a new model release, when raw capability gains need to be turned into real engineering value. It covers hierarchical main-worker orchestration and model tiering, context governance for long-horizon tasks (externalizing state, summarizing tool output, and budget-driven tiered summarization), behavioral evaluation (turning vague "productivity" into measurable metrics, enforcing acceptance through independent verification, and a reusable evaluation harness), and capacity/cost control with a canary rollout and observability. You'll find runnable Python examples, a scoring rubric, and a pre-launch checklist.
Introduction: After a New Model Ships, the Hard Part Is Using It Well
Every model release pushes developers back to the same question: once capability improves, how do you turn that into engineering value?
A typical scenario plays out after a new model version goes live. Community feedback starts to diverge. On launch day, the discussion centers on quotas running out and constant rate limits. Two days later, sentiment improves and the feedback shifts toward concrete usage experience. A heavy user summarized the new version like this:
- Compared to the previous version, behavior differs noticeably: it rushes to wrap up before the task is actually done;
- Context grows fast, so frequent summarization and compression is needed to keep moving;
- Any improvement in reasoning ability is not clearly felt.
This experience is representative and contains at least four signals: capacity supply, agent architecture, context governance, and capability evaluation. This article uses these four points as its main thread, drawing on hands-on experience building a hierarchical multi-agent orchestration system to break the problem down layer by layer and propose engineering solutions.
1. Hierarchical Multi-Agent Orchestration: Design the Architecture Before Choosing Models
1.1 The Basic Shape of a Main-Worker Architecture
Long-horizon tasks—code, 3D creation, financial research—typically contain many subtasks that can run in parallel. A common industry practice is hierarchical multi-agent orchestration: a main agent decomposes the goal, dispatches tasks, and aggregates results, while several worker agents each own a subtask, such as retrieving material, editing code, generating assets, or running consistency checks.
Illustrative code:
import asynciofrom dataclasses import dataclass, field@dataclassclass Task:goal: str # task goalacceptance: list[str] # acceptance criteria used to constrain "premature finishing"max_steps: int = 12 # max steps per taskbudget: int = 20_000 # context budget; compaction triggers once this is exceededasync def run_worker(t: Task, llm, verify) -> dict:"""A single worker: loops through "think - act - observe" until the criteria are met or the budget runs out."""context, step, cost = f"Goal: {t.goal}\nAcceptance criteria: {t.acceptance}", 0, 0while step < t.max_steps:step += 1resp = await llm(context, budget=t.budget)cost += resp.tokenscontext += resp.outputif resp.action == "finish":report = await verify(t, context) # independent verification; do not trust the model's self-reportreturn {"task": t.goal, "ok": report, "steps": step, "tokens": cost}obs = await resp.call_tool() # execute tool call (search, read/write files, render, etc.)context = append_observation(context, obs) # see the second section for summary compactionreturn {"task": t.goal, "ok": False, "reason": "step_limit", "tokens": cost}async def orchestrate(goal: str, plan: list[Task], llm, verify):"""Main agent: decompose the goal -> dispatch in parallel -> verify results -> synthesize delivery."""tasks = plan if plan else await llm.plan(goal)results = await asyncio.gather(*(run_worker(t, llm, verify) for t in tasks))pending = [r for r in results if not r["ok"]] # failed or questionable tasks go to retryfor _ in range(2):if not pending:breakpending = await asyncio.gather(*(run_worker(tasks[0], llm, verify) for _ in pending))return await llm.synthesize(goal, results, pending)
Compared with cramming everything into a single context, hierarchical orchestration offers three benefits:
- Context isolation: each worker sees only its own subtask, so the main context is never drowned in intermediate details;
- Parallel acceleration: independent subtasks run concurrently, shortening end-to-end latency;
- Fault containment: a single worker failure affects only its local scope, and the main agent can retry or replan.
1.2 High-Low Pairing: Use Different Model Tiers at Different Levels
The key to controlling cost is not only using cheaper models, but placing model capability in the right position. In practice, teams usually adopt a “high-low pairing” and assign by level:
- Planning and aggregation layer: handled by the main agent, involving goal decomposition, solution tradeoffs, and final integration—recommend the tier with stronger reasoning ability;
- Execution layer: worker tasks have clear boundaries and shorter context, so a faster, cheaper tier is fine;
- Verification layer: result verification requires some judgment, so choose a mid-tier, or distinguish roles on the same model with different prompts.
After layering, set up a unified call-abstraction layer to separate “model capability” from “orchestration logic.” Then when a tier’s price goes up, it gets rate-limited, or its quality disappoints, you just switch it in configuration instead of touching business code:
class LLMClient:"""Unified call layer: maps model tiers by role, with retry, timeout and fallback."""def __init__(self, registry: dict[str, str], call, retry: int = 2, timeout: float = 60):self.roles, self.call = registry, call # {"main": "...", "worker": "...", "verify": "..."}self.retry, self.timeout = retry, timeoutasync def __call__(self, role: str, *args, **kwargs):model = self.roles.get(role, self.roles["worker"])for attempt in range(self.retry + 1):try:return await self.call(model, *args, timeout=self.timeout, **kwargs)except Exception:if attempt == self.retry:# after all retries fail, fall back to a faster standby tierreturn await self.call(self.roles.get("fallback", model),*args, timeout=self.timeout, **kwargs)
At the same time, the main agent and workers should ideally exchange data in a structured format. Have the planning result land as verifiable JSON before handing it to the main agent:
{"plan_id": "p-20250601-001","goal": "Complete the quarterly industry analysis report","workers": [{"id": "w1", "owner": "main", "model_tier": "worker","goal": "Retrieve and summarize this quarter's public industry data","acceptance": ["Data sources are traceable", "At least 8 key metrics"], "budget": 20000},{"id": "w2", "owner": "main", "model_tier": "verify","goal": "Cross-check whether the data cited in the report matches its sources","acceptance": ["List each inconsistency"], "budget": 10000}],"budget": {"total_tokens": 400000, "max_rounds": 2}}
Structured planning makes it possible to query and audit how far the model has progressed, which tier executes each step, and how the budget is allocated. This is closer to an operable engineering system than relying on natural-language descriptions.
2. Context Governance: The Real Bottleneck in Long-Horizon Tasks
2.1 Why Context Swells
“Not proactive—always rushing to wrap up” and “context grows very fast” look like two problems, but in long-horizon tasks they are often cause and effect. As context keeps accumulating, it crowds out the space available for reasoning, and the model tends to produce a “seemingly complete” answer quickly rather than dig deeper. This is an engineering hypothesis that can be derived rather than assumed—but verifying it requires a controlled experiment, not intuition. Context swell usually comes from five kinds of content:
- Tool returns: a single search or file read can bring in thousands of tokens;
- Intermediate reasoning: chain-of-thought that piles up turn after turn;
- Repeated information: the same document read in over and over;
- Structured dumps: raw JSON, logs, and stack traces fed straight into the window without processing;
- Historical dialogue: accumulated chit-chat and probing that carries across tasks.
2.2 A Three-Layer Governance Strategy
Layer 1: Externalize state. Anything you “need for the next step but don’t need to read back verbatim” should go into external files, a database, or object storage, keeping only a path or ID in the context. This way you can retrieve it on demand and preserve task continuity even after the context is compressed or rebuilt.
Layer 2: Summarize tool output. Read and retrieval operations should return structured results with a summary, not raw long text:
def summarize_tool_result(raw: str, limit: int = 800, keywords: list[str] | None = None):"""Tool-result summary: structured fields + keyword-matched snippets, rather than re-injecting the full raw text."""if len(raw) <= limit:return rawhead = raw[: limit // 2]kws = keywords or []hits = [line for line in raw.splitlines() if any(k in line for k in kws)]return f"[summary] {head}...\n[matched snippets]\n" + "\n".join(hits[:5])[:limit]
Layer 3: Tiered summarization with budget control. Mainstream agent frameworks generally adopt a “sliding window + summary” approach. This article extends it into a three-part structure that carries task state:
- Continuously compress older history with rolling summaries, each capped at a fixed token count;
- Keep the most recent several turns verbatim in a sliding window, with tool outputs replaced by summaries;
- Maintain task state (goal, completed items, todos, key findings, files produced) separately, and inject it with every request after each compression.
import tiktoken # illustrative only; in production you can swap in your own tokenizer or estimate by character countENC = tiktoken.get_encoding("o200k_base")def count_tokens(messages: list[dict]) -> int:return sum(len(ENC.encode(m["content"])) for m in messages)def rolling_summary(history: list[dict], keep: int = 6, max_summary: int = 600):"""Sliding window + summary: keep the most recent `keep` turns verbatim, merge older content into a summary."""if len(history) <= keep:return historyold = [m["content"] for m in history[:-keep]]merged = "\n".join(old)summary = merged if len(merged) <= max_summary else merged[:max_summary] + "...(truncated)"return [{"role": "system", "content": f"[early-context summary]\n{summary}"}] + history[-keep:]class ContextManager:"""Three-tier context management with task state: persist state before compaction to avoid information loss."""STATE_KEYS = ("task", "done", "todo", "findings")def __init__(self, budget: int = 8_000, state_sink=None):self.budget, self.sink = budget, state_sink # sink: write to a database or object storagedef build(self, history: list[dict], state: dict):ctx = []used = count_tokens(history)# 1) task state: always retained, extracted from raw history during compactionif state and self.sink:self.sink.save(state) # persist to support restart and auditingctx.append({"role": "system", "content": "Task state: " + str({k: state.get(k) for k in self.STATE_KEYS})})# 2) compacted historyctx += rolling_summary(history)used += sum(count_tokens([m]) for m in ctx)# 3) most recent raw messages: backfill from newest to oldest until the budget is reachedrecent = []for msg in reversed(history):n = count_tokens([msg])if used + n > self.budget:breakrecent.insert(0, msg)used += nreturn ctx + recent
In addition, tool results should pass through summarize_tool_result before entering history. This compresses the context and reduces interference from irrelevant noise in later reasoning.
2.3 The Right Way to Compress
Two points are easy to overlook:
- Compression should trigger on the token budget, not on a fixed number of turns. A single tool call can double the context, so turn-based compression often kicks in too late;
- Compression is lossy. Anything dropped in a summary must be recoverable—once task state is written to object storage or a database, the main agent can fetch the original artifacts by path at any time.
3. Behavioral Evaluation: How to Tell Whether It Is “Not Pulling Its Weight”
3.1 First Define “Proactive” as Something Measurable
“Not pulling its weight” is a subjective impression. To evaluate it, convert it into measurable metrics:
- Acceptance-criteria satisfaction rate: how many of the preset criteria are met before the task ends;
- Average steps and tool-call count: too few steps combined with insufficient output may signal premature finishing;
- Rework rate: the share of deliverables bounced back by the verification stage;
- Output per unit cost: average token spend and completion quality per normalized task;
- Context efficiency: the number of compactions needed to finish the same task, or the peak context size.
At the same time, separate “model changes” from “system changes”: keep the prompts, tool set, parsing logic, and system configuration completely identical, and repeat at least three times and average. Otherwise, the differences you observe may come from the evaluation environment rather than the model itself.
3.2 Use Independent Verification to Constrain “Rushing to Finish”
Rather than asking the model to be “more proactive,” make the system enforce acceptance. Every task defines explicit completion criteria, checked by an independent verification role instead of trusting the model’s own “done” claim. When necessary, add an adversarial “critic” role that specifically hunts for flaws in the deliverable:
async def verify(worker_llm, critic_llm, task: Task, report: str) -> bool:"""Independent acceptance check: the verifier role only judges "whether the criteria are met" and does not participate in generation."""checklist = "\n".join(f"- {item}" for item in task.acceptance)prompt = f"Task: {task.goal}\nCriteria:\n{checklist}\nDelivery report:\n{report[:4000]}\n" \"Give a met/not-met verdict for each item with reasoning, and output JSON: {\"pass\": true/false, \"gaps\": [...]}"verdict = await worker_llm(role="verify", prompt=prompt, response_format="json")if verdict.get("pass"):return True# not passed: the "critic" role points out the weakest links, and the worker reworks based on thatgaps = await critic_llm(role="verify",prompt=f"The following delivery failed acceptance. Unmet items: {verdict.get('gaps')}\nReport:\n{report[:3000]}\n""Identify the 1-3 most critical problems and suggest fixes.")raise RetryWithFeedback(gaps) # caught by run_worker, which re-executes with the feedback
3.3 A Reusable Evaluation Harness
import time, json, statisticsfrom dataclasses import dataclass, field@dataclassclass Case:id: strdomain: str # code / creation / research ...prompt: strrubric: list[str] # scoring rubric, each item 0-2 pointsweight: float = 1.0 # domain weight, prevents a single domain from dominating the total@dataclassclass Run:case: Casesteps: inttokens: intcompactions: intwall: floatoutput: strscores: list[int] = field(default_factory=list)async def evaluate(cases: list[Case], orchestrator, judge, repeats: int = 3) -> dict:runs: list[Run] = []for case in cases:for _ in range(repeats): # repeat execution to observe stabilityt0 = time.perf_counter()res = await orchestrator(case.prompt)scoring = await judge(prompt=f"Task: {case.prompt}\nRubric:\n{case.rubric}\nOutput:\n{res.output[:6000]}\n""Score each item on 0/1/2 and output only a JSON array.",response_format="json")runs.append(Run(case, res.steps, res.tokens, res.compactions,time.perf_counter() - t0, res.output, scoring))# average the multiple scores of the same case to reduce single-run randomnessgrouped: dict[str, list[Run]] = {}for r in runs:grouped.setdefault(r.case.id, []).append(r)return summarize(list(grouped.values()))def summarize(groups: list[list[Run]]) -> dict:"""Produce a horizontally comparable scorecard: quality, stability, steps, context efficiency and cost."""report = {}for g in groups:scores = [statistics.mean(r.scores) for r in g]report[g[0].case.id] = {"quality": round(sum(scores) / (2 * len(g[0].case.rubric)), 3),"stability": round(1 - statistics.pstdev(scores) / 2, 3),"avg_steps": round(statistics.mean(r.steps for r in g), 1),"avg_tokens": int(statistics.mean(r.tokens for r in g)),"compactions_per_run": round(statistics.mean(r.compactions for r in g), 2),"avg_seconds": round(statistics.mean(r.wall for r in g), 1),}return report
Design the rubric per task type. For a “long-horizon research report” task, for example:
| Dimension | Key points | Score |
|---|---|---|
| Conclusion correctness | Whether key conclusions are grounded and self-consistent | 0–2 |
| Coverage completeness | Whether preset key points and acceptance items are covered | 0–2 |
| Process soundness | Whether there was multi-round verification rather than a single-shot draft | 0–2 |
| Deliverability | Whether structure, format, and readability meet delivery standards | 0–2 |
Breaking “improved reasoning ability” into three things to verify separately avoids self-contradictory evaluation conclusions:
- Complex reasoning: logical deduction and solution tradeoffs under multiple constraints;
- Long-horizon planning: coherent, multi-step execution across tools with self-correction;
- Domain knowledge: factual accuracy and terminology conventions in a specific field.
A model can very plausibly gain stronger long-horizon execution while complex reasoning stays flat. If you judge only by feel, it is easy to be misled by “crisper answers” or “fewer steps”—which is exactly what premature finishing looks like. To judge an agent, look at “task completion quality × stability,” not the fluency of a single response.
4. Capacity and Cost Control: Aligning Quotas With the Business
Quotas running out during the early launch period is, at its core, a mismatch between the supply cadence and how the system calls the model. It can be eased from three angles:
- High-low pairing: let only the top tier handle planning and aggregation, and route the many repetitive subtasks to a faster, cheaper tier. In common configurations, this lower tier’s cost per call is often around half that of the top tier—check each platform’s billing documentation for the exact figures.
- Result caching: reuse cached responses for prompts with identical prefixes and cache results for repeated queries to avoid re-spending.
- Budget and routing: set a budget cap per task and per user; once exceeded, downgrade to a lower tier or queue the request. Combine this with monitoring and alerts to observe the actual spend distribution across tiers and dynamically tune the high-low ratio.
5. Engineering Rollout: Launch, Canary, and Observability
After a new model goes live, do not switch everything over at once. A safer path is to canary first, then scale up:
- Run an A/B comparison of old and new versions on a fixed evaluation set, and read the scorecard rather than individual cases;
- Ramp up in steps of 5%, 20%, and 50%, while watching error rate, latency, and unit cost;
- Keep a one-click rollback switch—model tier, temperature, and prompt version should all support hot-swapping.
Observability is equally indispensable. Every agent run should record the task ID, the call relationships between the main agent and workers, the token spend per step, the number of compactions, verification results, and the location of the final artifacts. This information supports debugging and is the foundation for cost accounting and continuous tuning.
In terms of deployment shape, such systems are usually a combination of a “stateless orchestration service + stateful storage”:
- Container platform: hosts the orchestration service and worker pool, with elastic scaling to handle peaks;
- Object storage: holds intermediate artifacts, reports, and the raw pre-compaction context;
- Message queue: decouples task scheduling, absorbs traffic spikes, and supports failure retries;
- Logging and monitoring/alerting: uniformly collects the call trace and resource metrics, surfacing anomalies promptly.
These capabilities all have corresponding general-purpose product categories on Baidu AI Cloud, which can serve as a reference for teams building an agent-collaboration platform.
6. Pre-Launch Checklist
- Clear responsibility boundaries between the main agent and workers, with tasks dispatched in a structured format
- Every task has explicit completion criteria and is independently verified, not self-declared complete
- Context compaction triggers on the token budget, and task state is externalized and recoverable
- Tool output is summarized to avoid dumping raw long text straight into the window
- The evaluation set covers multiple domains and is run repeatedly to observe stability
- Old and new versions are A/B compared under the same system configuration
- Model tier, temperature, and prompt version support hot-swapping and rollback
- Every call is logged with metrics and cost accounting
Conclusion
The capability gains from a new model only pay off reliably under engineering constraints. Hierarchical orchestration solves “who does the work and who verifies it,” context governance solves “remember and keep moving,” and the evaluation system solves “is it good and is it stable”—all three are indispensable. The way to explore the edge of capability is to push one more step in a direction that seems impossible. For engineering teams, that step is simple: build the evaluation and governance capabilities first, then welcome the next model version.
