Evaluating LLM Reasoning and a Practical Guide to Multi-Agent Collaboration

Introduction: From Benchmark Obsession to Engineering Evaluation

Every time a new generation of large language models (LLMs) is released, the developer community is flooded with benchmark comparisons and anecdotal reviews. However, fragmented user experiences rarely answer the core engineering question: Which model should I use for my specific business scenario, how should I combine models, and how do I control costs?

Recent discussions around a specific new model were filled with emotional complaints, but they also surfaced three valuable technical topics. First, how to scientifically evaluate a model’s high-level reasoning capabilities, especially in mathematics. Second, how to build a tiered model selection strategy that aligns model capability with task value. Third, how to use multi-agent collaboration to maximize cost-performance under strict quota and rate-limit constraints. This article expands on these three areas.

1. Evaluating Mathematical and Reasoning Skills: A Layered Framework

1.1 The Boundary Between Closed-Ended and Open-Ended Problems

The industry generally divides mathematical capability into two categories:

• Closed-ended problems: These have a single correct answer and a clear verification path, such as competition problems or standard proofs. Current flagship models show little difference on these tasks because results are formally verifiable and training data is abundant.
• Open-ended mathematical problems: These lack a standard answer or have an enormous solution space. Models must generate conjectures, construct arguments, and self-audit for flaws. These tasks demand “research-grade reasoning” and are the true differentiator between top-tier models and ordinary flagships. Only a few models can reliably handle open-ended problems today.

This leads to a practical evaluation principle: Use open-ended tasks to distinguish top-tier models, and use closed-ended tasks as a baseline screening test. If your team focuses on algorithms and engineering, you can build a private evaluation set from “hard problems you have actually encountered in the past.” This will be more relevant than any public leaderboard.

1.2 Reasoning Chain Quality Matters More Than the Final Answer

Relying solely on whether the final answer is correct is insufficient. Two models may both get the answer right, but differences in their Chain-of-Thought (CoT) heavily impact the engineering experience:

• Conciseness: Does the reasoning path go straight to the point, or does it ramble and burn tokens through trial and error?
• Readability: Are the intermediate steps structured like an engineer’s analysis notes, allowing for efficient manual review?
• Stability: How much variance is there across multiple samples? Does the model occasionally have a flash of insight but fail most of the time?

In practice, you see two distinct styles. One model produces extremely long and complex reasoning chains; the result is fine, but the computational cost and reading effort are high. Another produces shorter, clearer chains that resemble natural human reasoning. For scenarios requiring human review, the latter usually offers a better overall experience. Therefore, the evaluation metric should include the combined cost per correct solution, factoring in token consumption, latency, and review time.

1.3 Building a Reproducible Internal Evaluation Process

Here is a lightweight evaluation workflow:

1. Select 20–50 real business problems, such as difficult bug localization, algorithm design, or architecture trade-offs.
2. Fix the prompt and sampling parameters; run each task 3–5 times independently and take the majority result.
3. Record four data points: accuracy, reasoning chain length, end-to-end latency, and token cost.
4. Compile a “task type × model” score matrix rather than a single aggregated leaderboard.

2. Tiered Model Selection: Matching Model Level to Task Value

A key consensus is that no single model route dominates in all scenarios; different technical approaches have distinct strengths. From an engineering standpoint, a tiered configuration is more rational:

Tier	Positioning	Typical Tasks
Flagship / Research	Tackling open-ended, high-complexity problems	Mathematical derivation, complex architecture design
Main / Workhorse	Core daily development tasks	Feature implementation, code review, refactoring
Economy / Cost-efficient	High-frequency, low-risk tasks	Format conversion, comment generation, batch fixes

Three principles should guide model selection:

1. Difficulty escalation: Try the economy tier first. If it fails more than a set threshold, automatically escalate to the main tier.
2. Risk stratification: Tasks involving changes to core logic should directly use the main tier or higher. Do not experiment with cheaper models.
3. Budget allocation: Reserve scarce flagship-tier quotas for genuinely high-value problems. Avoid using a sledgehammer to crack a nut.

3. Multi-Agent Collaboration: Distributing Work to the Right Executors

3.1 The Orchestrator-Worker Architecture

A particularly effective pattern from community discussions is to have one main agent handle orchestration and core implementation, while a child agent handles peripheral tasks like bug hunting, code review, and small feature additions. This is the classic Orchestrator-Worker pattern:

• Orchestrator (Main Agent): Understands the requirement, decomposes tasks, performs final integration, and makes decisions.
• Worker (Child Agent): Executes delegated subtasks and returns only the conclusion and deliverable artifacts.

This yields three main benefits. The orchestration model’s context window is preserved by filtering out low-value information. Low-cost models handle a large volume of mechanical work. Subtasks can run in parallel, reducing overall latency.

3.2 Delegation Rules Should Be Explicitly Configured

Delegation should not rely on the model’s discretion. It should be explicitly defined in a project-level agent configuration file:

# Project Agent Collaboration Rules (Example)
## Task Delegation Policy
- Code review, bug localization, standalone small features: delegate to review-worker (economy-tier model)
- Architecture changes, core algorithms, cross-module refactoring: handled by the main agent directly
- Delegation must include: file path, reproduction steps, acceptance criteria
- The worker returns only: conclusion + diff, without lengthy analysis

# Pseudo-code: Orchestrator Delegation Logic
def route_task(task):
    if task.type in {"code_review", "bug_hunt", "small_feature"}:
        return spawn_worker(model="economy-tier", task=task)
    elif task.difficulty == "open" or task.risk == "high":
        return solve_with_main_model(task)
    else:
        return try_economy_first_then_escalate(task)

3.3 Three Practical Considerations

• Define acceptance criteria upfront: Clearly state what “done” means when delegating. This prevents the worker from going off-track and causing rework.
• Trust-level classification: Review-based results should be human-verified through sampling. Build a trust profile for each model over time.
• Minimize context: Send only the necessary files and constraints. This reduces token usage and avoids information leakage.

4. Quotas and Rate Limits: Treating Quota as a Schedulable Resource

4.1 Understanding the Two-Layer Rate Limit Structure

Most subscription-based or platform-based services impose two layers of limits: a short rolling window (e.g., a few hours) and a long window (e.g., a weekly cap). A common pitfall is that the long-term limit may be equivalent to only a few fully-saturated short windows. Without an overall plan, one intense overnight session can exhaust the entire weekly quota.

A scenario from typical community discussions is illustrative: an agent is set to run overnight, hits the short-window limit in the early morning, resumes after the window resets, and then quickly triggers the weekly cap. This is fundamentally a lack of budget scheduling, not merely “insufficient quota.”

4.2 Engineering Countermeasures

1. Usage monitoring: Periodically collect the remaining percentage of both the short-window and long-window quotas. Log this data or connect it to an alerting system with threshold-based notifications.
2. Priority-based task queue: Divide the task queue into “must complete today” and “can be deferred.” When quota is tight, only high-priority tasks are released.
3. Degradation plan: High-value tasks use the main tier; lower-priority tasks automatically switch to the economy tier or a local smaller model as a fallback.
4. Avoid unattended peak usage: Do not schedule high-intensity tasks during unattended hours, such as late at night. If a limit is hit, there is no human intervention, and the waiting cost is highest.
5. Multi-channel backup: If the primary service quota is exhausted or unavailable, switch to an alternative model service, such as a cloud provider’s model platform, to keep the workflow running without interruption.

5. Conclusion: From Complaints to Methodology

Looking back at the emotional complaints, such as “a weekly quota only lasts a few short windows” or “let’s use it as a temporary stopgap,” the underlying issue is universal: finding a sustainable balance between quality, speed, and cost.

Instead of obsessing over “which model is the strongest,” teams should invest in three things that deliver long-term value:

• A private evaluation set to make every model selection data-driven.
• A multi-agent delegation specification so cheap models handle the grunt work and expensive models tackle the hard problems.
• A quota scheduling mechanism to ensure limited resources always serve the highest-priority goals.

When model iteration outpaces a team’s ability to adapt, an engineering-focused “model usage methodology” provides more durable returns than chasing the latest release. Whether you call model APIs directly or build applications on a cloud-based model service such as Baidu AI Cloud, this framework for evaluation, collaboration, and scheduling applies equally.