A Developer Guide to LLM Math Evaluation and Multi-Agent Deployment
This article presents a repeatable process for evaluating LLM math reasoning on your own test sets, then explains how to configure main-agent and sub-agent collaboration, handle quota limits with checkpointing and failover, and apply a selection checklist and troubleshooting sequence. It emphasizes structured records, cost and stability metrics, and reproducible deployment decisions.
A Developer Guide to LLM Math Evaluation and Multi-Agent Deployment
1. Why Build Your Own Math Evaluation
Scores on public leaderboards change with model versions, prompts, and evaluation settings. Copying leaderboard conclusions directly makes it easy to choose the wrong model. Developers need a process that can be run repeatedly on their own questions and use recorded data to make decisions.
Open-ended math problems require models to perform multi-step deduction, invoke theorems, and check conditions. Therefore, evaluation cannot rely only on whether the final answer is correct. The following is an evaluation method that can be executed directly.
| Item | Recommended practice |
|---|---|
| Test set | Use public competition problems or a self-built question bank, and state the source and license. Categorize questions by proof, calculation, and modeling. |
| Sample size | Use at least 100 questions per category and report confidence intervals for accuracy. With too few samples, the difference between two models may be only random fluctuation. |
| Core metrics | Final-answer accuracy, verifiability rate of the derivation process, and average output tokens per question. |
| Stability | Sample the same question 5 times and calculate the answer consistency rate. Models with low consistency are harder to use directly in batch tasks. |
| Run conditions | Record model identifier, version number, call date, temperature parameter, and maximum output length for reproducibility. |
| Human review | Spot-check proof questions, with at least two people independently scoring and calculating the disagreement rate. |
Evaluation results need to answer a specific question: whether a longer chain of thought improves accuracy. The method is to bucket questions by output token length and compare accuracy and cost in each bucket. If the accuracy gain is smaller than the cost increase, choosing a model with shorter output is more reasonable. This conclusion applies only to your question set and cannot be directly generalized to other tasks.
It is recommended to store the results of each run as structured records, for example one line per question, including question ID, model identifier, version, sample index, answer, scoring result, and token count. This way, when changing models or adjusting prompts later, you can directly compare historical data without rebuilding the process.
2. Architecture and Configuration for Multi-Agent Collaborative Development
A single model can hardly handle overall architecture design and detailed review at the same time. A common approach is a “main agent plus sub-agents” structure. The main agent uses a high-reasoning-tier model and is responsible for system design, core business logic, and complex defect localization. Sub-agents use faster and lower-cost models and are responsible for syntax checking, code style checking, unit test generation, and small local feature changes.
Below is a configuration template. The model identifiers, budgets, and retry parameters are placeholders and need to be replaced with real content from the platform documentation.
{"agent_system": {"main_agent": {"model": "<HIGH_REASONING_MODEL_ID>","role": "architect_and_core_developer","max_output_tokens": 8192,"budget_usd_per_task": 5.0,"retry": {"max_attempts": 3, "backoff": "exponential", "base_seconds": 2}},"sub_agents": [{"name": "code_reviewer_agent","model": "<LIGHTWEIGHT_MODEL_ID>","role": "code_reviewer","tasks": ["syntax_check", "style_check", "unit_test_generation"],"max_output_tokens": 2048,"budget_usd_per_task": 0.5}]},"logging": {"level": "info", "redact_fields": ["api_key", "source_code_snippet"]},"permissions": {"api_key_scope": "inference_only", "file_write": "sandbox_branch_only"}}
Pay attention to the following during implementation.
- Use model identifiers from the platform console or official documentation, and do not reuse the placeholder names in the example.
- Grant API keys only inference permission, store them in a secret management service, and do not write them into code repositories or logs.
- When sub-agents modify code, they can write only to an independent branch, which is merged after review by the main agent or a human.
- Redact keys and sensitive source code snippets in logs.
- Set a cost cap for each task category. When the cap is exceeded, stop execution and output a report.
- Track call counts, latency, and error codes for each task category. These data are the basis for adjusting the division of labor between the main agent and sub-agents.
The stability of a multi-agent system depends on observability. It is recommended to monitor at least three types of metrics: task status distribution, call frequency per unit time, and error code distribution. When the error rate of a sub-agent continues to rise, first check the input format and prompts, and then consider changing the model.
3. Impact of Quota Limits and How to Respond
Quota limits are most likely to cause interruptions in long unattended nighttime tasks. For example, large-scale code refactoring or full test case generation may last for hours. Suppose a platform sets a 5-hour rolling limit and a 7-day periodic limit; this is only an example, and actual values must be based on the platform documentation.
Rolling limits and periodic limits are calculated differently. If the periodic limit allows only about 15 hours of effective usage, then only three full 5-hour windows can be used in a week. If a task is interrupted at the end of the first window and the periodic limit is triggered again soon after recovery, you may still be unable to continue when you wake up in the morning. Therefore, when designing automated processes, both limits must be included in the decision conditions.
There are three ways to respond to quota limits.
- Task tiering. Assign tasks that require deep reasoning to advanced models, and assign batch generation, format conversion, and mechanical review to lightweight models or local open-source models. After tiering, advanced quota is used only for steps that truly need it.
- Checkpoint and resume. Read progress before each task executes and save results immediately after execution. When a quota error occurs, save the current state and resume time, and then pause the entire batch. Below is a simplified implementation approach.
def run_batch(tasks, store, client):for task in tasks:state = store.load(task.id) # Load saved progressif state.get("status") == "done":continue # Skip completed tasks to avoid recomputationtry:result = client.call(task.payload,idempotency_key=task.id # Repeated requests with the same key are computed once)store.save(task.id, {"status": "done", "result": result})except QuotaExceeded as e:store.save(task.id, {"status": "paused","resume_after": e.retry_after, # Read resume time from the error or response header"partial": state.get("partial"),})break # Stop the entire batch when quota is exhaustedexcept TransientError:store.save(task.id, {"status": "retry_pending"})# An external scheduler re-enqueues with exponential backoff
- Multi-platform failover. Design a unified model gateway in the business system and wrap different platform interfaces into the same abstraction. Below is an interface sketch.
from typing import Protocolclass ModelProvider(Protocol):name: strdef complete(self, prompt: str, max_tokens: int) -> str: ...def call_with_failover(providers: list[ModelProvider], prompt: str, max_tokens: int) -> tuple[str, str]:last_error = Nonefor provider in providers:try:return provider.complete(prompt, max_tokens), provider.nameexcept (QuotaExceeded, ServerError) as e:last_error = e # Switch only on quota errors or persistent server errorscontinueraise last_error
Before switching platforms, confirm two things. First, the output format for the same task may differ across platforms, so validate it with a regression test set. Second, record the platform name after switching for later troubleshooting. Cross-platform switching may also raise data compliance issues, so each platform’s data retention policy should also be included in the selection scope.
4. Selection Checklist and Troubleshooting Steps
During selection, check the following items one by one.
- Whether math evaluation has been run at least three times on a self-built question set.
- Whether the cost of output tokens has been included in the monthly budget.
- Whether the platform provides stable version numbers and advance notice of model deprecation.
- Whether quota definitions are documented, including rolling window, period length, concurrency, and overage behavior.
- Whether the model supports structured output and tool calling, which affects whether sub-agents can work reliably.
- Whether data retention and privacy terms meet the team’s compliance requirements.
When problems occur, troubleshoot in the following order.
- Check the error code first to distinguish quota, authentication failure, timeout, and content filtering. Different errors require completely different handling.
- For quota errors, read the resume time and confirm that checkpoint data has been saved.
- When results are abnormal, compare the recorded model version and temperature parameter to determine whether a version change caused the issue.
- When tasks are executed repeatedly, check whether the idempotency key is effective and whether there is latency in storage writes.
- When costs exceed the budget, count token usage per task and identify excessively long outputs or repeatedly passed context.
5. Conclusion
Math reasoning capability, collaboration efficiency, and quota stability all need to be judged with repeatable data. The evaluation table, configuration template, and checkpoint-resume code above are starting points that can be adjusted directly. Specific parameters need to be modified according to your own questions, codebase, and budget.
It is recommended to continuously record weekly quota usage, task success rate, and average cost per task. Observe whether these metrics change with model versions or platform policy changes, and then adjust model division of labor and failover strategy accordingly.
