Evaluating LLM Capabilities and Managing Usage Quotas in Development Workflows
This guide explains how development teams can evaluate large language models through a repeatable process. It covers splitting evaluation by task type, building fixed question sets, recording results along with their verification methods, and understanding usage limits. It also describes how to assign tasks to model tiers based on complexity and remaining quota.
1. Why Model Evaluation Should Be a Process
Judging a model by impression is unreliable, because conclusions are easily swayed by a few individual cases. A correct answer to a hard problem may be luck, and a wrong answer may only mean the prompt was unclear. Teams need a repeatable method: define the task type first, fix a set of evaluation samples, record the results, and include cost in the same table.
Cost has two parts. One is the rework caused by output quality. The other is resource cost, which includes call quotas, waiting time, and context length. Looking at only one of them often leads to biased conclusions.
2. Split Evaluation Dimensions by Task Type
The same model can perform very differently across tasks, so break tasks down before evaluating them.
2.1 Math Reasoning Tasks
Math tasks fall into three broad categories:
- Problems with standard answers, such as competition and calculation problems, where results can be compared directly.
- Proof problems, where a correct conclusion is not enough. Each step of the derivation must also be checked.
- Open-ended problems with no accepted answer. Here, the question is whether the approach is valuable and whether the conclusion can be verified independently.
Common verification methods include the following:
- Numerical verification: substitute concrete values and check whether the conclusion holds.
- Symbolic computation: use a general-purpose computer algebra library to simplify or solve the problem, then compare the results.
- Independent reproduction: have another engineer, or a different method, derive the result again.
- Formal verification: use a formal proof tool to check key steps.
A fluent derivation does not mean the conclusion is correct. Judge the work on the steps that can be verified.
2.2 Engineering Tasks
Common engineering tasks and their evaluation metrics are listed below:
- Defect localization: given reproduction steps and logs, identify the root cause. The key metrics are localization accuracy and average time taken.
- Code review: the key metrics are the issue detection rate and the false positive rate.
- Small feature increments: the key metrics are the test pass rate and whether the scope of changes stays controlled.
- Technical documentation and explanations: the key metrics are factual accuracy and consistency with the code.
Count each task type separately. Do not use a single overall score to represent all capabilities.
3. Build a Reproducible Evaluation Process
The evaluation process should include the following five steps.
- Build a question set. Prepare 20 to 50 samples for each task type, covering easy, medium, and hard tiers.
- Fix the inputs. Use a unified prompt template and context length. Do not rewrite questions during evaluation.
- Sample multiple times. Run each question several times at minimum and count the passes. A single result only describes one case.
- Blind review. Reviewers should not know which model or tier produced each result, which reduces subjective bias.
- Record versions. Log the model version, call time, and parameter settings so the evaluation can be reproduced later.
Each evaluation record can use a structure like this:
record = {"task_id": "math-017","tier": "high_reasoning","attempts": 5,"verified_correct": 3,"verification": "symbolic_check","notes": "The 4th attempt reached the correct conclusion, but the intermediate derivation skipped the boundary conditions"}
Keep both the number of passes and the verification method. Recording only “correct” or “incorrect” discards much useful information.
4. Understand the Structure of Usage Limits
Usage limits are usually not a single number. Common forms include three types:
- Short-period windows: for example, rolling hourly limits that control instantaneous peaks.
- Long-period windows: for example, weekly limits that control total volume.
- Remaining balance: the quota left in a prepaid plan or subscription.
These limits apply at the same time, and when any one of them runs out, the task stops. Planning should therefore focus on the tightest window. Track the remaining ratio of each window separately instead of looking only at the total balance.
A short window being consumed quickly does not mean a long window will be consumed at the same rate. Estimate the remaining time of a long window from the recent average daily usage:
days_left = weekly_remaining / daily_average
If the estimate is clearly shorter than the remaining period, adjust task allocation in advance. Do not wait for tasks to fail midway.
5. Assign Tasks to Tiers by Complexity
The basic principle is that the more complex the task, the stronger the reasoning ability it needs. Calls to the strongest tier, however, should be kept under control.
- High-complexity tasks: design decisions, difficult defects, and proof problems. Use the most capable tier and keep the volume under control.
- Medium-complexity tasks: routine feature implementation and test completion. Use a balanced tier.
- Low-complexity tasks: format conversion, batch checks, and repetitive edits. A low-cost tier is sufficient.
The following illustrative code shows how to choose a tier based on remaining quota and task complexity:
def pick_tier(remaining_ratios, complexity):# Use the tightest remaining ratio across all limit windowstightest = min(remaining_ratios)if complexity == "high" and tightest > 0.3:return "high_reasoning"if tightest > 0.1:return "balanced"return "low_cost"
When handing tasks to sub-agents, keep the following in mind:
- Describe the input, the output format, and the acceptance criteria clearly.
- Pass only the context needed to complete the task. Do not send the full session history, because it wastes quota.
- Have the parent agent or a human spot-check the sub-agent’s results. Merge them into the main branch only after they are confirmed correct.
6. Common Mistakes
- Drawing conclusions from a single case. With too few samples, the conclusion is not reliable.
- Treating “looks reasonable” as “verified.”
- Ignoring usage windows until a task fails midway through.
- Using the same tier for all tasks. This wastes quota and slows down simple tasks.
- Treating public leaderboards as universal answers for every scenario. Leaderboards reflect performance on specific test sets, which may not match your business scenario.
7. Implementation Checklist
- Tasks are grouped by type and complexity
- Each task type has at least one fixed question set
- Each result has an independent verification method
- The remaining quota of each limit window can be checked in real time
- The tiering rules are documented in the team’s documentation
- Quality and cost data are reviewed once a week
