LLM Wish Accuracy and Calling Cost: How to Evaluate Models and Control Token Spend

    A practical guide to judging large language models by "wish accuracy"—the first-pass success rate of a single prompt—instead of list price. It shows with a worked cost model why accuracy differences amplify token spend, how to build a reproducible evaluation suite from real business tasks, which evaluation pitfalls to watch for, and how layered routing between strong planning models, cheap execution models, and deterministic verification keeps cost per successful task under control.

    1. Introduction: the awkward middle ground of LLM selection

    Most developers have run into this: you face a model that is good enough but not good to use. Dropping it feels wasteful, committing fully feels risky.

    Models in this category typically handle entry-level work well. Format conversion, boilerplate code, and common troubleshooting often take a single conversation, and they are genuinely faster than weaker models. But as soon as the task involves coordinating multiple files, respecting hidden constraints, or reasoning through deeper logic, problems cluster. The proposed solution looks plausible but is subtly wrong, so you keep adding context and correcting course, until the token spend on that one task exceeds what a weaker model would burn making repeated lucky guesses under the same budget. Conversely, some plans that reset their quota on a billing cycle look generous on paper, yet a long-running task can exhaust them early.

    That combination—capability sitting right in the middle, cost with no advantage—is what produces the feeling of being stuck, neither up nor down. It usually traces back to three causes:

    • Selection based on single-turn impressions: judgments come from a handful of conversations, with no evaluation set covering real business work;
    • Reading the price tag instead of actual spend: the hidden cost of multi-turn retries is ignored;
    • No quota governance: calls have no budget cap and usage is not aggregated, so problems surface late.

    This article addresses three related questions:

    1. What is “wish accuracy,” and how does it amplify cost?
    2. How do you design a reproducible, measurable model evaluation system?
    3. Within a limited quota, how do task decomposition and scheduling architecture keep cost in an acceptable range?

    2. Wish accuracy: the core yardstick of model capability

    2.1 What “making a wish” means

    In LLM-driven coding and automation, “making a wish” means handing the model your requirements, context, and constraints in natural language all at once, expecting runnable code or a deliverable straight out—like making a wish to the model.

    The point of the metaphor is whether the wish comes true. The more accurate the wish, the higher the one-shot pass rate and the less trial and error. An inaccurate wish adds cost to every turn, for two reasons:

    • Every turn must carry the entire prior conversation and code as context, so input grows linearly with turn count and cumulative consumption approaches quadratic growth;
    • Every wrong output also consumes human review and feedback time.

    So the core value of a large model is not just that it can compute, but that it understands. It is widely accepted in the industry that as model size and training adequacy increase, complex instruction following, multi-step reasoning, and cross-file association generally improve too—showing up as more accurate wishes and fewer retries. But this advantage is not fixed: once a task falls outside the model’s capability boundary, even a strong model sinks into inefficient trial and error and the cost advantage disappears. Model selection therefore has to be grounded in specific tasks, not in a generic strong-versus-weak judgment.

    2.2 The amplification effect on cost: a worked example

    To make the impact of accuracy concrete, consider a simplified model (the numbers are illustrative and do not represent measured results for any real product).

    Assume an average of 8,000 input tokens and 1,000 output tokens per turn, billed at 2 CNY per 1,000 input tokens and 10 CNY per 1,000 output tokens. Ignoring cache hits, one turn costs about 16 + 10, i.e. 26 CNY.

    Wish accuracy (single-turn first-pass rate) Avg turns Cumulative input (illustrative) Cost per task (illustrative) Tasks completed with 100 CNY (illustrative)
    70% 1.43 ~15K tokens ~CNY 48 ~2
    50% 2.00 ~24K tokens ~CNY 78 ~1
    30% 3.33 ~53K tokens ~CNY 173 Budget insufficient; task cannot complete

    Calculation: average turns is 1 divided by the single-turn pass rate, the expectation of a geometric distribution. Cumulative input after n turns is an arithmetic series sum: n × 8,000, plus n × (n − 1) / 2 × 1,000.

    Two things stand out:

    • Raising wish accuracy from 30% to 70% cuts cost per task by roughly 70%. A low-accuracy model exhausts the same budget with nothing to show, while a high-accuracy model has already produced a deliverable—so the practical value of one budget can differ severalfold;
    • Once the cost of a single task exceeds the budget cap, a larger budget only decides whether you give up early or late. It does not change the fact that the task will not complete.

    2.3 A composite view: capability is multidimensional

    “Wish accuracy” therefore cannot stay a subjective impression; it should be decomposed into measurable dimensions. The three layers most directly tied to retry cost are:

    1. Instruction alignment: does it grasp both explicit requirements and implicit intent, and proactively clarify ambiguity, rather than patching in the wrong direction;
    2. Execution correctness: does the code pass compilation, tests, and static checks, and are constraints (interface contracts, data formats) strictly honored;
    3. Context stability: as turns and files accumulate, does it still hold on to the key constraints instead of being distracted by secondary information.

    Conventional dimensions such as knowledge coverage, refusal and hallucination tendencies, and output consistency should also be included. Only when these dimensions are made explicit do evaluation conclusions map onto real usage, and selection decisions rest on evidence.

    3. Building an evaluation system you can actually run

    3.1 Replace single-turn impressions with a task pool

    The first principle of evaluation is to let real tasks speak. Pull representative tasks from your own tickets, commit history, and documentation Q&A, and de-identify them. Layer tasks by difficulty and type, for example:

    • Entry level: format conversion, boilerplate code, small single-file changes;
    • Intermediate: defect localization, cross-file refactors, interface integration;
    • Expert level: complex refactoring, long-context retrieval, design decisions that touch business rules.

    Prepare at least 20 to 50 tasks per layer, which keeps the estimation error of the single-turn pass rate within about ten percent. Write explicit acceptance criteria for every task category in advance—this is what keeps evaluation results from collapsing into subjective argument.

    3.2 An executable evaluation flow

    The recommended approach is “automated judging first, human review as a backstop.” Illustrative code:

    1. from dataclasses import dataclass
    2. import statistics
    3. @dataclass
    4. class Task:
    5. task_id: str
    6. category: str # entry / medium / expert
    7. prompt: str # task description and constraints only; no acceptance details leaked
    8. context: str = "" # optional prior code or long documents
    9. judge: callable = None # automated judge returning (passed, detail)
    10. price_in: float = 2.0 # input token price (CNY per 1K tokens)
    11. price_out: float = 10.0 # output token price (CNY per 1K tokens)
    12. def evaluate(model_client, tasks, max_turns=6, max_budget=50.0, k=1):
    13. """model_client.chat returns a dict:
    14. {"text": reply, "in_tokens": int, "out_tokens": int}
    15. """
    16. records = []
    17. for task in tasks:
    18. for _ in range(k): # k>1: repeat sampling to observe output stability
    19. history, spent, turns, passed = [], 0.0, 0, False
    20. for _ in range(max_turns):
    21. reply = model_client.chat(
    22. task.prompt, history, context=task.context
    23. )
    24. spent += (
    25. reply["in_tokens"] * task.price_in
    26. + reply["out_tokens"] * task.price_out
    27. ) / 1000.0
    28. history.append(reply["text"])
    29. turns += 1
    30. # automated judging: unit tests / compile checks / structured field comparison, etc.
    31. if task.judge and task.judge(reply["text"])[0]:
    32. passed = True
    33. break
    34. if spent >= max_budget: # budget guardrail: stop as soon as the cap is hit
    35. break
    36. records.append({
    37. "task_id": task.task_id, "category": task.category,
    38. "passed": passed, "turns": turns, "cost": spent,
    39. })
    40. return records
    41. def summarize(records):
    42. by_cat = {}
    43. for r in records:
    44. s = by_cat.setdefault(r["category"], [])
    45. s.append(r)
    46. report = {}
    47. for cat, items in by_cat.items():
    48. hit = [r for r in items if r["passed"]]
    49. report[cat] = {
    50. "pass_rate": len(hit) / len(items),
    51. "avg_turns": statistics.mean(r["turns"] for r in items),
    52. # actual spend per successful task; failed-task spend counts as sunk cost
    53. "cost_per_pass": (
    54. sum(r["cost"] for r in items) / len(hit) if hit else None
    55. ),
    56. }
    57. return report

    This flow produces two key classes of data:

    • Success rate and average turns: characterizes wish accuracy;
    • Actual spend per successful task: characterizes cost efficiency.

    Print the summary at the end of each run as well. That fits an inline command workflow and makes the evaluation report easy to read quickly in a local terminal.

    3.3 Three evaluation traps that are easy to miss

    1. Long-context degradation: rerun the evaluation after injecting files into the context that resemble the target file and are topically close but irrelevant. A clear drop in pass rate means the model is easily distracted by similar content—and the risk of wrong edits in real development is not trivial.
    2. Output stability: sample the same task multiple times. If the success probability swings widely between 30% and 70%, plan against the expected value rather than the best single run.
    3. Test set contamination: evaluation tasks may already sit in the model’s training data. Interpret results on public question banks cautiously, and prefer self-built private tasks that are refreshed regularly.

    4. From list price to real spend: a systemic view of cost and quota

    4.1 Cost is more than the price tag

    Teams often think of model cost in terms of unit price or subscription price. Real spend includes at least three categories:

    Cost type Typical components Optimization approach
    Explicit cost API charges, subscription fees, storage and network Accurate wishes, concise output, result caching
    Hidden cost Human review and rework, context preparation, waiting after quota exhaustion Automated acceptance, layered scheduling, budget alerts
    Risk cost Defect fixing and incident handling after a wrong solution reaches production Mandatory verification gates, gradual rollout

    4.2 Pay-as-you-go billing vs. subscription quota

    The two mainstream billing models differ structurally:

    • Pay-as-you-go: billed on actual input and output tokens, cost grows linearly with call volume and is predictable. Under the same budget, a model with higher output efficiency typically gets more done for the same money.
    • Monthly subscription: a fixed price grants a certain quota, reset per cycle. The user’s concern is whether the quota is enough and how long it lasts, not the price of a single call.

    Each has tradeoffs:

    • Steady, predictable usage suits subscriptions, and fixed spend simplifies budget planning;
    • Usage that swings with business demand, and scenarios sensitive to quota caps and concurrency limits, suit pay-as-you-go better.

    Conversely, if a model’s cost per task is high, subscription quota drains faster, and even moderately complex tasks become hard to sustain.

    Also note that quota rules and entitlement scope for subscription services can change with the terms of service. Architecturally, stay portable: avoid binding business flows tightly to a single billing model.

    4.3 Making cost calculable and controllable

    Build a cost governance loop around four steps:

    1. Budget guardrails: set a maximum turn count and maximum spend per session and per task, terminate automatically on breach, and emit current progress—so the model “stops with partial results in hand” instead of burning tokens on nothing;
    2. Usage aggregation: record token consumption and cost per call along application, environment, and user dimensions, producing a traceable call ledger;
    3. Monitoring and alerts: set thresholds for daily/monthly spend, call failure rate, and quota utilization, and notify promptly on anomalies;
    4. Cost attribution: periodically compare cost per successful task across scheduling strategies and keep tuning routing rules.

    LLM calls usually run in the cloud, and platforms such as Baidu AI Cloud typically offer baseline capabilities like logging, monitoring and alerting, and object storage, which can hold call records, raise real-time alerts, and archive evaluation results respectively. That keeps cost transparent and controllable, and makes retrospectives and optimization easier.

    5. Engineering architecture: amplifying model value with layered scheduling

    5.1 Strong-and-weak collaboration: let strong models do the hard, low-volume parts

    Since the value of a strong model lies in wish accuracy, it does not need to serve every request. A fairly mature pattern is separating planning from execution:

    • Strong model plans: task decomposition, key decisions, acceptance criteria. Call volume is small, but it determines whether the whole task succeeds;
    • Weak model executes: carries out localized implementation in batches from the refined instructions, with low per-call cost and high throughput, scaling elastically and running in parallel on container platforms;
    • Deterministic tools verify: compilation, unit tests, static checks, and format validation act as automated acceptance gates, sending failing results back for correction automatically.

    The finer the task is decomposed, the more likely the weak model’s wish lands, and the better the overall cost-performance. Strong models are not omnipotent, though: when a task needs cross-module architectural tradeoffs or long chains of logical reasoning, they can loop on mistakes too. Identify that early and switch to structured reasoning methods or human involvement rather than continuing to burn quota in a loop.

    5.2 Layered routing and fallback

    Illustrative pseudocode:

    1. def route(task, executor, planner, budget_left):
    2. # pick the execution model by task complexity; planner only steps in when decomposition is needed
    3. plan = task.spec if task.atomic else planner.split(task)
    4. if len(plan.steps) == 1 and task.risk == "low":
    5. result = executor.run(plan.steps[0])
    6. else:
    7. # non-atomic tasks: split into subtasks, run them in parallel, then assemble
    8. results = parallel_map(executor.run, plan.steps)
    9. result = assemble(results)
    10. # fallback: if automated verification fails and quota allows, escalate to a higher-tier model
    11. if not verify(result) and budget_left > ESCALATE_COST:
    12. result = executor.run(task, model_tier="high")
    13. # when quota is tight, return intermediate results for a human or a queue instead of dropping them silently
    14. log_usage(task.id, result)
    15. return result

    Routing policy should not be set once and left alone. Review and adjust it periodically based on production success rates, average turns, and cost data, closing the loop of “evaluate → schedule → re-evaluate.”

    5.3 Treat quota as a first-class resource

    In LLM application architecture, call quota has become a key resource alongside compute and storage, and governance can borrow from mature cloud practices:

    • Quota allocation: set call quotas and priorities per business line and per environment, with high-priority tasks served first when quota is tight;
    • Observability: bring token consumption, call success rate, and average latency into one monitoring dashboard;
    • Context engineering: inject relevant snippets on demand via retrieval, summarize and compress conversation history, and trim redundant code to reduce input size per call;
    • Versioning and replaceability: hide provider differences behind a unified model access layer, and keep prompt templates under version control. Before switching to a new model, run regression evaluation to confirm wish accuracy has actually improved.

    6. Common mistakes and selection advice

    6.1 Four common mistakes

    1. Generalizing from a few samples: a few successful sessions lead to “this model is strong,” while failure cases and long-context scenarios are ignored;
    2. Reading only the price tag: the cost amplification from multi-turn retries is overlooked, along with constraints such as quota cycles and rate limits;
    3. One model for everything: all tasks hit the same model, so you either waste budget or fall short on capability;
    4. Evaluating once and stopping: models keep iterating, so the evaluation set and scheduling policy must be updated in step.

    6.2 A selection action checklist

    • Build a layered task pool from real business work, and define automatically executable acceptance criteria per task category first;
    • Focus core metrics on single-turn first-pass rate, average turns, and cost per successful task;
    • Include explicit, hidden, and risk costs in the cost model;
    • Schedule by task complexity: strong models plan, weak models execute, deterministic tools accept;
    • Build cost monitoring and alerting on top of budget guardrails and usage aggregation;
    • Verify switching decisions with regular regression evaluation, and keep models replaceable.

    7. Conclusion

    The awkward middle ground is better attributed to an evaluation system that is not detailed enough and a cost structure that is not transparent than to any single model being insufficiently strong. Wish accuracy decides how much trial and error you pay for, quota efficiency decides how far a budget reaches, and task decomposition plus layered scheduling decide where each unit of spend actually goes. Put all three into a quantifiable evaluation framework, and selection turns from “wishing on a feeling” into an engineering decision backed by data—delivering reliable results continuously within a limited budget.