How to Evaluate LLM Math Reasoning, Interpret Throughput Metrics, and Build Rate-Limited Multi-Agent Workflows
This article explains how to evaluate LLM math reasoning with reproducible test conditions and scoring rules, how to interpret throughput claims with the right measurement context, how to troubleshoot and design API rate limiting, and how to build multi-agent workflows with concurrency controls and token budgets. It includes practical checklists, a scoring table, and a Python example for retry and budget management.
Introduction
LLM evaluation has expanded from text generation to math reasoning, code generation, and multi-agent collaboration. For developers, the real challenge is not simply knowing which models exist. It is being able to judge whether a model fits a given task with reproducible methods, understand the performance numbers published by providers, keep tasks running reliably under quota limits, and divide work across multiple models while controlling cost.
This article addresses four questions: how to evaluate math reasoning, how to interpret compute metrics, how to troubleshoot and design API rate limiting, and how to build multi-agent workflows. Each section provides executable checks or code that readers can adapt to their own scenarios.
1. How to Evaluate LLM Math Reasoning
Many teams ask directly, “Which model is best at math?” But that question has no comparable answer unless the evaluation criteria are consistent. The same model can perform very differently under different temperatures, output lengths, and prompts. Evaluation conclusions must therefore include the test conditions; otherwise, they cannot be reproduced by others or used for decision-making.
Before starting an evaluation, confirm the following:
- Task type: numerical calculation, symbolic derivation, theorem-proving draft, or long-form lecture-style output. Different task types should be counted separately.
- Model information: model name, specific version, and call date. Providers may update models without notice, so the date matters.
- Inference parameters: temperature, maximum output length, and whether tool calling is enabled. Use the same parameters for all models when possible.
- Prompts: save every test prompt in full, including system prompts and examples.
- Scoring criteria: define the conditions for each score and have two or more reviewers score independently.
- Human review rules: decide which side takes precedence when automated scoring disagrees with human judgment, and define the review sampling ratio.
- Failure sample storage: keep incorrect answers, derivations, and review comments in full for later analysis.
Below is a scoring table that can be used directly.
| Dimension | Max score | Criteria |
|---|---|---|
| Correct conclusion | 4 | The final answer matches the reference answer, or the proof can be independently verified |
| Coherent reasoning | 3 | Every step is justified, with no skipped steps or circular reasoning |
| Consistent notation | 2 | Symbols are consistent throughout, and LaTeX compiles correctly |
| Complete structure | 1 | Includes necessary parts such as definitions, lemmas, proof, and conclusion |
Evaluation results should be summarized by task type, with sample counts and variance. When the sample is too small, a small difference between two models may be random fluctuation. If only a dozen questions were tested, the conclusion should say “performed better on this sample” rather than “is clearly stronger.”
In addition, for open-ended theorem proving, a model’s output may look complete while containing logical gaps. Reviewers should focus on the justification for each step, not just length and formatting.
2. How to Read High-Throughput Compute Metrics
Providers or hardware vendors often publish numbers such as “tokens processed per second per rack.” Such numbers are useful, but only when the measurement conditions are included. At minimum, confirm the following:
- Model size and precision: parameter count and quantization method.
- Request shape: batch size, context length, and the ratio of input to output.
- Concurrency: how many requests are processed at the same time.
- Hardware and software: accelerator model and inference framework version.
- Measurement definition: average or peak, whether first-token latency is included, and the measurement duration.
High throughput does not mean that an individual request is fast. Under high batch concurrency, each request may wait in a queue, and first-token latency and tail latency (P99) can rise significantly. For interactive applications, pay attention to first-token latency, P99 latency, and cost per task at the same time. For offline batch jobs, throughput and unit cost are more important.
When evaluating a compute plan, convert metrics into “the cost of completing one task.” For example, calculate the average input and output tokens consumed by a code review, convert them into a cost based on pricing, and divide by the number of successfully completed tasks. This number is closer to the actual basis for decision-making than raw TPS.
If self-deploying, also consider VRAM usage. When VRAM is insufficient, the system may be forced to reduce batch size or shorten context, which lowers throughput. Therefore, load testing should use the same context length and concurrency pattern as the production environment.
3. How to Troubleshoot and Design API Rate Limiting
Start with the conclusion: a rate-limiting strategy should support fine-grained token-bucket controls, allow long-window quotas to be purchased on demand, and provide observable usage alerts. Only when all three are present can long-running tasks complete reliably.
Many providers set both short-window and long-window quotas, such as a short-period window and a longer-period window. Problems often arise from the ratio between the two. Suppose the long-window quota is only equivalent to three short-window quotas. A task that needs to run overnight will repeatedly exhaust the short-window quota until the long-window quota is depleted, and the task will be interrupted at night. In this case, even if no individual short window exceeds its limit, the overall task still cannot complete.
When troubleshooting rate limiting, proceed in the following order:
- Confirm the returned status code. HTTP 429 usually means requests are too frequent or quota is insufficient; distinguish between these two cases.
- Read the
Retry-Afterfield in the response headers. If present, wait for the time it specifies. - Record the time, token count, and return code of each call to identify when the limit was exceeded.
- Compare the usage curves for short and long windows to determine which layer is the bottleneck.
- If the quota structure does not match the task, adjust the task pace or request a more suitable quota.
On the client side, use a token bucket to control request rate. A token bucket has two key parameters: bucket capacity determines the allowed burst of requests, and refill rate determines the long-term average rate. For example, a bucket capacity of 10 and a refill rate of 2 tokens per second allow short bursts while keeping the long-term rate predictable. Set the parameters according to the provider’s actual limits and your business peak, and verify them in load testing.
When rate limited, retry with exponential backoff and random jitter. Fixed-interval retries can cause many clients to request again at the same time, creating new congestion. Backoff can increase as 1 second, 2 seconds, and 4 seconds, with a small random offset added to each result. After reaching the maximum number of retries, mark the task as pending rather than waiting indefinitely.
For monitoring, set multi-level alerts. For example, send a reminder when quota usage exceeds 70%, pause new subtasks when it exceeds 85%, and keep only critical tasks when it exceeds 95%. Thresholds should be adjusted according to task urgency.
If the existing quota structure does not fit long-running tasks, present concrete requirements to the provider, including expected average daily token volume, peak concurrency, task runtime, and acceptable interruption frequency. Specific data is more likely to lead to a reasonable quota plan than a vague “the quota is insufficient.”
4. How to Build Multi-Agent Collaboration
A common multi-agent approach is to have a main controller model understand the goal, decompose tasks, and aggregate results, while several submodels perform relatively simple work such as code review, defect localization, and small feature implementation. The main controller usually uses a stronger model, while submodels use lower-latency, lower-cost models. Model selection should not rely only on parameter count or leaderboards; it should be based on pass rate and unit cost on your own task set.
Proceed in the following steps:
- Model selection: test candidate models on a small set of real tasks and record pass rate, latency, and cost.
- Decomposition: have the main controller output a structured subtask list as a JSON array, where each item contains a unique ID and a specific instruction.
- Concurrency control: limit the number of simultaneously running subtasks to avoid exceeding quota in a burst.
- Cost ceiling: set a token budget for the entire task and stop when it is exceeded.
- Failure fallback: retry failed subtasks first; if they still fail, record the reason and skip them, and let the main controller decide whether to decompose again.
- Result validation: perform format checks and basic validation on submodel outputs, such as whether code compiles or passes tests.
The example below relies only on the Python standard library. For real calls, replace the call function with an actual HTTP client, and have the client raise RateLimitError when it receives a 429 response while returning the token usage for that call.
import jsonimport randomimport timefrom dataclasses import dataclassclass RateLimitError(Exception):"""Raised by the client when an HTTP 429 response is received."""@dataclassclass Budget:max_tokens: intused: int = 0def can_afford(self, estimate: int) -> bool:return self.used + estimate <= self.max_tokensdef charge(self, tokens: int) -> None:self.used += tokensdef call_with_retry(call, prompt, budget, estimate=500, max_retries=4, base_delay=1.0):"""Call the model, retry with exponential backoff on rate limits, and respect the token budget."""for attempt in range(max_retries + 1):if not budget.can_afford(estimate):raise RuntimeError("Insufficient token budget; stopping calls")try:text, tokens = call(prompt)budget.charge(tokens)return textexcept RateLimitError:if attempt == max_retries:raisedelay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)time.sleep(delay)raise RuntimeError("Unreachable code")def parse_plan(text):"""Validate the task plan returned by the main controller."""plan = json.loads(text)if not isinstance(plan, list):raise ValueError("The plan must be a JSON array")for item in plan:if not isinstance(item, dict) or not {"id", "instruction"}.issubset(item):raise ValueError("Each subtask must contain id and instruction fields")return planclass AgentOrchestrator:def __init__(self, main_call, sub_call, budget):self.main_call = main_callself.sub_call = sub_callself.budget = budgetdef dispatch(self, task):prompt = ("Break the following task into subtasks. Output only a JSON array ""where each item contains id and instruction fields.\nTask: " + task)raw = call_with_retry(self.main_call, prompt, self.budget)plan = parse_plan(raw)results = []for item in plan:try:out = call_with_retry(self.sub_call, item["instruction"], self.budget)results.append({"id": item["id"], "status": "ok", "output": out})except (RateLimitError, RuntimeError) as exc:results.append({"id": item["id"], "status": "failed", "error": str(exc)})return resultsdef fake_main(prompt):plan = [{"id": 1, "instruction": "Review boundary conditions in utils.py"}]return json.dumps(plan, ensure_ascii=False), 120def fake_sub(prompt):return "No obvious boundary issues found", 80if __name__ == "__main__":orchestrator = AgentOrchestrator(fake_main, fake_sub, Budget(max_tokens=1000))for row in orchestrator.dispatch("Check boundary conditions in the code"):print(row)
There are several points to note in this code. The budget check uses an estimate before the call, avoiding the situation where an over-budget call is discovered only afterward. When a subtask fails, only its status is recorded without interrupting the whole workflow, so the main controller can decide whether to decompose again based on the failure record. parse_plan performs basic structural validation, but whether the semantics are correct still requires the result validation step.
In production, also add the following: record token usage and latency for each subtask for cost analysis; set a length limit for main controller output; run tests on code output from submodels instead of merging it directly; and redact logs to avoid writing secrets or user data into logs.
Conclusion
Comparing LLM math reasoning capabilities requires fixed evaluation conditions and reproducible scoring rules. Compute metrics are useful only when their measurement definitions are included. API rate-limiting problems often come from the ratio between short- and long-window quotas; token buckets, backoff retries, and tiered alerts can minimize the impact. The key to multi-agent collaboration is clear division of work, controlled concurrency and budget, and validation of subtask results.
Here is a checklist that can be executed directly:
- Before evaluation, record model versions, parameters, prompts, and scoring rules.
- During load testing, use the same context length and concurrency pattern as production.
- Use cost per task, not a single throughput number, for decision-making.
- Implement token-bucket rate limiting, exponential backoff, and usage alerts for API calls.
- Set token budgets, failure fallback, and result validation for multi-agent tasks.
By following these steps, teams can use LLM capabilities reliably in real engineering work at controllable cost.
