Choosing LLMs When Compute Is Scarce: Intelligence per Dollar
As demand for inference outpaces compute supply, model selection is shifting from peak capability to cost-effective, verifiable work. This article covers task-level evaluation, intelligence per dollar, reliability and refusal, high-risk controls, domain and long-context engineering, and layered routing so teams can ship under fixed budgets.
Background: From Capability Ceiling to Unit Cost
For the past two years, much of the large-model discussion focused on whose capability ceiling was higher. As models grow and inference volume rises, compute supply has not kept up with demand. For developers and enterprises, the practical question is which model completes more useful work under a fixed budget.
That shift is from an intelligence race to an efficiency race. Leaderboard scores alone are not enough. Accuracy, latency, cost per call, and the cost of recovering from failures need to be counted together.
Dimension 1: Task Split — There Is No Universal Winner
The same model can perform very differently across coding scenarios. Success rates may diverge on multi-file fixes, single-file completion, and in-terminal command execution. Each class of work stresses context length, tool-call stability, and error recovery differently.
Split business work into subclasses and test each one. A simplified evaluation sketch:
task_set = list of subtasks split by business needfor model in candidate_models:for subtask in task_set:result = run(model, subtask, fixed_prompt_template)record(model, subtask, passed, latency, cost)aggregate: pass rate per subtask; pick the best model per subtask
Prompt templates, sampling parameters, and the evaluation set must stay identical, or comparisons are not trustworthy.
Dimension 2: Intelligence per Dollar
Intelligence per dollar is the number of valid outcomes you get per unit cost. It is closer to production than accuracy alone. Two models with the same accuracy can have very different economics at scale if inference cost differs by several times.
A simple estimate:
Cost per valid result = total inference spend / number of tasks that pass validation
Include retries and human review. A cheap model that needs many retries may not be cheaper in practice.
Sketch:
def cost_per_valid_result(records):total_cost = sum(r.cost for r in records)valid = sum(1 for r in records if r.passed)if valid == 0:return float("inf")return total_cost / valid
Dimension 3: Reliability and Admitting Uncertainty
When a problem is outside training data or knowledge, models typically either give a plausible but wrong answer or say they cannot tell. The first misleads; the second needs a refusal policy.
Build three question sets: routine items with a gold answer; items with no answer or an outdated answer; items that need external sources. Track wrong-answer rate, appropriate refusal rate, and over-refusal rate together. Cutting wrong answers while flooding the product with refusals still fails the business.
For commercial use, keep human review on critical outputs, or constrain generation with retrieved evidence and require a source for each claim.
Dimension 4: Evaluating and Controlling High-Risk Capabilities
As capabilities rise, risk assessment in areas such as cybersecurity and biosecurity is becoming routine. Users should check three things: whether the provider published a risk-assessment method; whether the business touches sensitive domains; whether the system has audit logs, permission isolation, and alerts for abnormal behavior.
When wiring a model into internal systems, follow least privilege. Tool calls should reach only named resources, and every action should be logged. Irreversible actions such as delete, send, or deploy must require human confirmation.
Professional Domains and Long Documents
General models are stable on general knowledge. In specialist domains, results usually depend on data quality and retrieval. For professional documents, regulations, and technical manuals, common practice includes:
- Semantic chunking that keeps heading hierarchy and table structure;
- A vector index plus keyword search for hybrid retrieval;
- Prompts that require citing paragraph numbers;
- Map-reduce style section summaries so key facts are not lost in long context.
These engineering steps often outperform simply switching to a larger model.
Engineering When Compute Is Tight
With limited compute, stop calling the strongest model every time. Typical layers:
- Model routing: small models for simple work, escalate for hard work;
- Result cache: return already-validated answers for repeats;
- Batching: merge non-real-time jobs to raise utilization;
- Rate limits and degradation: keep the core path up at peak.
Routing sketch:
def route(task):if task.complexity == "low" and cache.hit(task):return cache.get(task)if task.complexity == "low":return small_model.run(task)result = large_model.run(task)if not validator.check(result):return human_review_queue.push(task)return result
The validator is essential. It can be rules, unit tests, format checks, or a second review, and it should block unreliable output.
Launch Checklist
Before production, confirm:
- Evaluation by business subtask, not only an overall score;
- Cost per valid result, including retries and human review;
- Refusal behavior on unknown questions;
- Output checks and human confirmation points;
- Call logs, with periodic review by cost and quality;
- A degradation plan for peak traffic.
Summary
Compute scarcity will last. Productivity gains will show up more in automating repetitive, process-shaped work. Treat how much verifiable work you get per dollar as the core metric, and pair it with layered routing and strict validation so models stay useful under a controlled cost.
