Near-Zero-Cost AI Agents: The Battle Over the Default Model
This article examines why the debate over near-zero-cost AI centers on which model becomes an agent's default rather than on free credits. It breaks down the token-economics levers pushing costs toward zero—sparse activation in Mixture-of-Experts models, prefix/context caching, and inference-side optimizations such as batching, quantization, and speculative decoding—and shows how to build a cost model around cost per successful task and retry amplification. It lays out four hard constraints for choosing an agent's default model (time-to-first-token, structured-output reliability, effective long-context use, and safety boundaries), explains how to read third-party call-volume rankings without misjudgment, and covers emerging risks including credit abuse, cost observability, compliance, security, and cloud deployment. It closes with role-specific action checklists for developers, platform providers, and enterprise decision-makers.
1. The phenomenon: what’s actually being debated is “default,” not “free”
When an open-source agent project designates a particular LLM as its official recommended default model and bundles it with generous free usage credits at launch, the attention this draws has less to do with the word “free” and more to do with the word “default.”
An agent typically runs on a “local gateway + cloud inference” pipeline: the local process intercepts terminal commands, reads and writes files, and drives UI actions, while the cloud model handles planning and decision-making. In this pipeline the model is something that gets “called,” and which model gets called is usually determined by the default configuration, the built-in routing table, example code, and documentation. In the out-of-the-box stage, the vast majority of users won’t actively compare models; they simply accept the default. So “becoming the default” is equivalent to capturing the agent’s distribution entry point—this is the third battlefield of model competition.
Model competition can roughly be split into three layers:
- Capability layer: the absolute level of reasoning, code, tool calling, and long-context understanding;
- Cost layer: per-token price, cache-hit discounts, and the hidden amplification from failed retries;
- Distribution layer: who gets written into the default config, SDK examples, tutorials, and templates.
Over the past two years the industry debate has concentrated on the first layer. The real significance of this round of changes is that the second and third layers are beginning to influence the landscape independently. When a model is good enough on both cost and engineering fit, it doesn’t need to beat every competitor on the leaderboard—as long as it becomes the default in some high-traffic scenario, it can capture an order-of-magnitude share of calls.
One point deserves a clear head: a default position comes from a combination of engineering and commercial choices and is not equivalent to a capability verdict. When a community first picks a default model, the choice is often also influenced by factors like “low onboarding cost, permissive licensing, ample credits, acceptable latency.” As load grows, whether it remains the optimal default should be answered again by measured data.
2. Why costs approach zero: a look inside token economics
“Near-zero cost” isn’t marketing rhetoric; it’s the result of several mutually independent technical levers stacking together. Understanding them lets you judge whether you can replicate the same cost structure.
2.1 Sparse activation: decoupling parameter count from inference compute
The key to the Mixture-of-Experts (MoE) architecture is that “total parameter count” and “parameters actually activated per token” are two different things. If a model with trillions of parameters activates only a small subset of experts on each inference, the compute for a single forward pass can be well below that of a dense model of comparable scale. This directly lowers two metrics: compute cost (which determines unit price) and time-to-first-token (which determines perceived responsiveness).
It’s worth stressing that sparse activation lowers average inference cost; it doesn’t automatically mean stronger capability or a guaranteed lower price. The balance of expert routing, load distribution under batching, and the cold start of tail experts all affect real throughput. To judge whether it fits, engineers should look at measured P50/P95 latency and real bills under their own traffic profile, not at architecture names.
2.2 Prefix reuse: context caching changes the billing structure
Agent workloads have a natural property: a large fraction of the input is repeated. System prompts, tool definitions, project background, and coding conventions are nearly byte-for-byte identical across dozens of calls. Mainstream inference services generally offer context/prefix caching: they keep the KV state of a stable prefix in GPU memory or tiered storage, so the input billing for subsequent hits is well below full input, and time-to-first-token drops.
This lever’s payoff is often underestimated. Take a typical coding-agent session: if a fixed prefix accounts for 60–70% of input tokens per turn and the hit rate reaches 90%, the nominal cost on the input side can drop by nearly half or more. But caching is no free lunch:
- Caching has invalidation conditions: once the prefix content changes (even a single space, or a bumped version number), the whole cached segment is invalidated;
- It has a lifetime: after an idle timeout it must be refilled, so long-idle sessions see hit rates fall back;
- It has precision and isolation requirements: when sharing a cache across tenants, you must guarantee isolation to avoid cross-session data leakage.
The corresponding engineering action is to turn prompts into versionable, stable assets and to push volatile content to the end.
# Example: put a stable prefix first and volatile content last to improve cache hit rateSYSTEM_PREFIX = load_versioned_prompt("agent_system_v7") # version-controlled; changes go through reviewTOOL_SCHEMA = build_tool_schema() # put tool definitions first and keep them stableHISTORY = session.short_term(max_turns=8) # short-term dialogue, rolling windowMEMORY = retriever.topk(user_id, query) # inject long-term memory via on-demand retrievalprompt = SYSTEM_PREFIX + TOOL_SCHEMA + HISTORY + MEMORY # fixed order to avoid prefix drift
2.3 Inference-side optimization: batching, quantization, and speculative decoding
Beyond architecture, the engineering quality of the inference service also determines unit price: continuous batching improves GPU utilization, quantization reduces memory-bandwidth pressure, speculative decoding uses a small model to draft and a large model to verify in order to cut the number of serial decode steps, and PD-separated deployment splits prefill and decode so each can scale independently. These capabilities are transparent to the user but show up on the price and latency curves.
To judge whether a model is really “cheap,” build your own cost model rather than looking only at the list price:
# Example: cost accounting for a single session (all numbers are hypothetical; method demo only)def session_cost(calls, in_tokens, out_tokens, cached_in, price):miss_in = in_tokens - cached_intotal = 0.0for _ in range(calls):total += (miss_in * price["in_miss"]+ cached_in * price["cache_hit"]+ out_tokens * price["out"]) / 1_000_000return total
What actually determines the budget is two derived metrics: cost per million effective tasks (total_spend / successful_tasks) and the retry amplification factor (actual call rounds / theoretical optimal rounds). A model with a low price but a low one-shot success rate—one that needs multiple rounds of self-correction—can end up with a higher total cost.
3. Why agents can’t look at price alone: four hard constraints
Making a model the agent’s default essentially makes its weaknesses the default too. Selection has to clear at least four gates:
1. Time-to-first-token (TTFT) determines responsiveness. Agents often sit in a “step-by-step confirmation” interaction state: the user types a command, waits for a plan, reviews a diff. If every step takes several seconds of waiting, the sense of fluidity collapses immediately. Sparse activation and cache hits mainly improve TTFT, which is why it’s closer to user perception than “total throughput.” When evaluating, record P50, P95, and time-to-first-token together, not just the average.
2. Reliability of structured output and tool calls. Agents depend on strong structured protocols like JSON output, function calls, and diff patches. A single formatting error can trigger retries and context bloat. The evaluation method should include format-compliance rate, argument-correctness rate, and the recognition rate for dangerous operations, rather than only the quality of the generated text.
3. “Effective” not just “available” in long context. A larger context window doesn’t mean better retrieval. You need to test key-information location, cross-segment conflict resolution, and recall of information at the end of a long document. The approach is to bury a known “needle” in a long context and measure recall and mis-citation rates.
4. Instruction following and safety boundaries. When the input simultaneously contains untrusted content like code, web pages, and logs, whether the model will execute malicious instructions embedded in them (prompt injection), and whether it will perform deletion or push operations without confirmation, are all questions the default model must answer. Permission control, whitelisted commands, and double confirmation for dangerous actions cannot rely on the model’s own discretion.
4. Ecosystem shifts: how to read the call-volume rankings
The call-volume rankings from third-party model aggregation and routing platforms are a useful signal for observing market structure, but reading them requires avoiding three common misjudgments:
- Call volume ≠ a quality ranking: it simultaneously reflects price, free credits, distribution position, and developers’ curiosity-driven trials. A short-term spike often includes a large amount of trial traffic.
- High concentration ≠ technical monopoly: aggregation platforms themselves do routing and fallback, and the same developer may long-term integrate three or four models at once.
- A change in ranking ≠ a drop in cost: what’s really worth tracking is the cost-per-task curve, not the nominal unit price.
For all agent projects, the right way to use these signals is to stay vendor-neutral. Abstract model access into an adapter layer, make model parameters, timeouts, retries, and fallback policies configurable, and turn switching cost from “changing code” into “changing config.”
# Example: vendor-neutral routing and fallback to avoid binding to a single modeldef route(task):if task.risk_level == "high" or task.tool_calls_required > 3:return "strong_model" # route key decisions to the strong modeldraft = small_model(task) # try the low-cost model first for low-risk tasksreturn "strong_model" if verifier.reject(draft) else draft
Layered routing is one of the most effective ways to control cost: hand simple, high-frequency, low-risk tasks (file classification, format conversion, summarization, command completion) to a low-cost model, and hand complex planning, code refactoring, and incident triage to a strong model, with a verifier pass to intercept errors—this usually raises the success rate at unchanged or even lower total cost.
5. New risks brought by near-zero cost
Falling prices amplify both upside and risk at the same time, so guardrails need to be designed in advance.
Credits and abuse. Large free credits are essentially customer-acquisition subsidies; they usually come with rate limits, concurrency caps, and anti-abuse policies, and they tighten as load grows. Building free credits into architectural assumptions (for example, assuming unlimited retries) is dangerous. Production systems must have usage budgets, alerts, and circuit breakers, treating “budget exhausted” as a class of failure that has to be handled.
Cost observability. Agent costs are hard to account for per request; they should be attributed along session/task dimensions, recording per-task token consumption, cache-hit rate, retry count, tool-call count, and final success or failure. Without this set of metrics, you can’t judge whether an optimization is effective.
Dependency and compliance. Models with open weights or that are locally deployable reduce concerns about data leaving borders and data sovereignty, but they also bring responsibility for version management, license terms, and security patches. On the enterprise side, you need to define model-version pinning, change review, and rollback processes.
Security surface. Cheap, low-latency models encourage stuffing more untrusted content into the context, and the attack surface for prompt injection grows accordingly. The default configuration should include least-privilege tool permissions, confirmation for write operations, audit logs, and redaction of sensitive information.
Cloud deployment. Whether you self-host inference or use a managed service, mainstream cloud platforms (such as Baidu AI Cloud) generally provide common capabilities like model services, containerized deployment, gateways and load balancing, object storage, logging, and monitoring/alerting, used to support the caching layer, canary releases, and cost accounting. For specific parameters, billing, and quotas, refer to the official documentation; don’t hardcode any unverified specs into your architecture design.
6. Action checklists for different roles
For developers: version your prompts and put fixed content first; enable prefix caching and monitor the hit rate; replace “one-size-fits-all” with layered routing; build your own scenario eval set (30–50 real tasks is enough to start) and rerun it on every model switch; record P95 of TTFT.
# Example: a minimal self-built eval set, run on every model switchCASES = [{"input": "...", "expect_schema": "plan", "max_ttft_ms": 800, "tools": 0},{"input": "...", "expect_schema": "patch", "must_pass_tests": True, "tools": 2},]def run(model):return [{"case": c, "ok": check(c, model(c)),"ttft_ms": measure_ttft(model, c)} for c in CASES]
For platform providers: the default model should be switchable and its default value periodically reviewed; put credit policies, rate limits, and data-usage policies into the documentation; provide a usage dashboard so users can see cost attribution clearly; set default confirmation for high-risk actions.
For enterprise decision-makers: change the discussion metric from “token unit price” to “cost per successful task”; run a small-scale A/B before deciding the default; preserve multi-vendor capability at both the contract and technical levels; bring model-version changes into change management.
7. Conclusion: free is the phenomenon, approaching-zero marginal cost is the result
When the default model starts to approach free, the industry won’t return to the old narrative of “compute scarcity”; instead, competition is pushed to the next level: when intelligence itself becomes a commons, the moat shifts to scenario data, workflow depth, reliability engineering, and user trust.
For developers this is clearly good news—the cost of trial and error for agents drops sharply, moving from “calculate tokens first, then start” to “get it running first, then optimize.” But technical judgment matters more than ever: cheapness induces over-calling, defaults mask unsuitability, and caching creates hidden dependencies. What really determines the success of an agent project is still those unglamorous things—stable structured output, measurable success rates, controllable permission boundaries, and a cost-governance mechanism that still runs when the model is free.
