LLM Selection and Quota Scheduling Practices for Long-Running Tasks
This article explores strategies for selecting large language models (LLMs) and scheduling quotas for long-running tasks, addressing the common issue of uncontrollable completion times due to uncertain rate limits. It covers building custom test sets for model evaluation, defining quota metrics, task grading, primary-auxiliary model routing, and breakpoint resumption. Additionally, it provides insights into cost modeling, monitoring, and a step-by-step implementation guide to help development teams optimize quota utilization and task completion times.
I. Why Overnight Tasks Go Out of Control
Many developers run background tasks such as code review, cross-file refactoring, or mathematical reasoning using LLMs, expecting to collect results the next day. However, these tasks often trigger hourly limits within a few hours, entering a waiting state. After the quota resets, execution resumes only to hit longer-period limits.
The core contradiction is that uncertain quotas lead to uncontrollable completion times, disrupting R&D rhythms. Teams relying on external model APIs or subscription services for deep development face this issue, necessitating a reusable approach for model selection and scheduling.
II. Evaluating Models with Custom Test Sets
Public leaderboards can serve as a reference, but they cannot replace your business data. It is recommended to extract samples from your codebase and historical tasks to build a regression test set. Using the same set of questions and scoring rules for each evaluation ensures comparable results.
2.1 Mathematical and Logical Reasoning
- Test Set Composition: Include university math, discrete math, and algorithm problems. Each category should have at least 30 questions, totaling no less than 100.
- Scoring Criteria: A correct answer scores 1 point. Critical missing steps or wrong premises in the derivation score 0.5 points. A correct conclusion without a verifiable derivation process scores 0.
- Pass Threshold: A total score of 80% of the question volume, with a score rate of no less than 70% for each category, is required to enter the candidate pool.
- Reproduction Steps: Fix the prompt template, question order, and sampling parameters (if supported by the API). Run each question 3 times and take the majority consistent result. Record questions with inconsistent conclusions across the three runs as evidence of model instability.
2.2 Long Context and Code Understanding
- Issue: Feeding a large repository as a whole might exceed the context window or dilute key information.
- Approach: Split the repository by modules, attaching module descriptions and interface lists to each batch. Require the model to list doubts first before giving conclusions.
- Metrics: Issue hit rate (proportion of manually annotated real defects found), false positive rate (proportion of items flagged as defects but actually invalid), and average time per batch.
2.3 Reasoning Chain Length and Duration
Equally correct answers may have reasoning chains that differ in length by multiples, varying duration and quota consumption. Record the output token count and end-to-end duration for each question, calculating the “tokens consumed per correct answer” as an efficiency metric.
III. Quota Scheduling: Understand the Rules, Then Schedule Tasks
3.1 Define Quota Metrics Clearly
Different services have different limit structures. Before scheduling, confirm the following information from the official documentation or the console, and document it for the team:
- Service and package name
- Minute or hour-level window limits
- Daily reset time (mind the time zone)
- Long-term total quota and refresh cycle
- Metering method: by request count, token count, or duration
If the official documentation does not disclose a certain item, use actual console measurements and record the measurement date.
3.2 Task Grading Table
| Level | Example Task Types | Recommended Model | Failure Cost | Scheduling Period |
|---|---|---|---|---|
| A | Complex reasoning, cross-module refactoring, security-related review | Most capable model | High | Prioritize when quota is sufficient |
| B | Unit test generation, defect localization, interface documentation | Mid-tier model | Medium | Regular periods |
| C | Format adjustment, comment completion, code completion | Lightweight model | Low | Periods with tight quota |
3.3 Primary-Auxiliary Model Routing Rules
- The primary model is only responsible for planning, task decomposition, and final review.
- Parallelizable subtasks (defect localization, writing unit tests, error checking) are assigned to auxiliary models.
- Outputs from auxiliary models must pass machine validation (compilation, unit tests, static checks) before being handed over to the primary model.
- If a subtask fails twice, escalate it to the primary model. Do not retry indefinitely.
- Calculate the primary model’s call proportion weekly and compare it with the previous cycle. Any reduction should be validated using custom evaluation data; do not preset target numbers.
3.4 Breakpoint Resumption Checklist
Batch processing tasks are executed in shards. Check the following items before and after each shard runs:
- The input for the current shard has a recorded hash value for identification during re-runs.
- Send the first 5% of data first to record response quality and quota consumption per shard.
- Estimate whether the remaining quota can complete the remaining shards. If not, pause.
- When quota is insufficient, switch to an evaluated backup model or save progress and wait for reset.
- Write each shard’s result to an independent file. Do not overwrite completed results.
- Verify the validation results of completed shards before resuming.
IV. Cost Modeling and Monitoring
Comparing unit prices alone is insufficient for judging model selection. The total cost of a single task can be calculated as follows:
Total Cost = Request Cost + Tool Call Cost + Retry Cost + Review Cost
Where, Review Cost = Review Duration × Manual Labor Unit Price.
Retry cost needs to be estimated separately. Let p be the failure rate of a single request for a certain task type. Assuming requests are independent and retried until success, the average number of requests is 1/(1−p). For example, if p=0.2, an average of 1.25 requests is needed, which is 25% more than a single request. The more unstable the model, the larger this multiplier. Therefore, a model with high output quality fluctuations may not have a lower total cost than a more expensive but more stable model.
The monitoring dashboard should at least include: daily quota utilization, success rate for each task type, average cost per correct output, and the distribution of waiting times.
4.1 From Point Tools to Agent Workflows
Most teams remain at the stage of “copying code to the model for modification.” A more mature approach is to let the model read the repository, review historical commits, run tests, and have the primary agent dispatch subtasks, with developers only reviewing the final results. This demands higher instruction following and context management, so the completion rate of multi-step tasks should be included in the evaluation set during model selection.
4.2 Update Test Sets with Model Updates
After a new model version or a new service provider’s product is released, run the regression test set completely to compare accuracy, latency, and quota consumption. This requires an initial time investment but can early detect quality degradation caused by model upgrades.
V. Implementation Sequence Recommendations
In the first week, document quota metrics and complete task grading. In the second week, establish the evaluation set and complete the first round of comparison. From the third week onwards, enable primary-auxiliary routing and breakpoint resumption, and review the monitoring dashboard weekly. By following this sequence, the team can observe changes in quota utilization and task completion time within a short period.
