# Weekly LLM Trend Tracking: Token Cost and Agent Capability Metrics
Tracking large language models week by week: how to normalize per-token list prices into cost per task, account for context replay and output ratios in agent workloads, set step and token budgets, interpret call-volume share shifts, and ground capability changes in a stable internal eval set. Includes worked cost tables, sample Python and YAML, and a five-step weekly review loop.
1. Why Weekly Trend Tracking Matters
The LLM market has shifted from annual generational releases to weekly iteration. Post-trained variants of top-tier models ship continuously and weights are increasingly released; mid- and low-tier models compete on lower unit prices and faster inference, specifically to drive down the cost of agent workloads; leading vendors are also rolling out personal assistant products that stay resident in the cloud and keep working while the user is offline. Supply, capability, cost, and call structure are all moving at once.
For engineering and product teams, making selection decisions on a quarterly cadence usually means being one release behind, and comparing list prices in a single pass easily produces the wrong conclusion. The value of weekly tracking is that it records every round of change against a consistent measurement baseline, so questions like “why did last week’s setup suddenly stop making economic sense?” have a clear answer.
A complete tracking cycle should cover five stages and close the loop:
- Collect: capture usage data from your own gateway, plus model release information and benchmark results.
- Compute: convert to cost per task and refresh the cost-performance frontier.
- Analyze: run period-over-period comparisons and version migration analysis to locate the source of change.
- Decide: confirm whether to switch the default model, or adjust routing and step budgets.
- Review: validate the decision in the next cycle and recalibrate your eval set and budget baselines.
2. The Four-Layer Signal Model: What to Watch
The useful signals in raw observation logs abstract into four layers. They corroborate one another; reading any single layer in isolation invites a one-sided conclusion.
- Supply layer — capability and ecosystem availability
- New weights released or opened up mean self-hosting and private deployment become more viable.
- Context window size and reasoning modes set the ceiling for long-running tasks.
- Open-weight releases usually ship alongside model cards, quantized builds, and inference framework support, which affects time to production.
- As overseas services tighten controls on model distillation behavior and mid-to-high-tier supply from Chinese vendors remains relatively thin, updates to high-tier Chinese models are the most direct signal on the supply side.
- Capability layer — fit for your target tasks
- General intelligence indices and third-party composite scores are useful as horizontal reference only.
- Vertically, compare score changes between versions within the same generation.
- The final judgment has to rest on your own task-specific eval set.
- Cost layer — real bills, not list prices
- Input/output unit prices are only the starting point; in agent scenarios, context replay multiplies input-side cost several times over.
- Watch the cost spread across tiers: in the original observations, two similarly positioned high-tier models differed by several times in actual cost per task.
- Low-tier models pushing agent task costs down sharply is the textbook result of deliberate cost tiering.
- Demand layer — shifts in call structure
- Total call volume, request counts, per-model share, and period-over-period change indicate which direction the market is migrating.
- This data usually comes only from public statistics published by third-party aggregation platforms, with limited methodology and coverage. It is good for trends, not for absolute values.
3. Cost Normalization: From List Price to Cost per Task
List prices are typically quoted per million tokens, separately for input and output. Comparing unit prices directly ignores the largest hidden cost in agent scenarios: context replay. The same conversation history is resent at every reasoning step, so the more model calls a task makes, the larger the cumulative input token count.
3.1 The context amplification factor
Let the average input tokens be I and output tokens be O, with an output-to-input ratio r = O / I. The cost of a single reasoning step is:
cost_step = (I × p_in + O × p_out) / 1,000,000
where p_in and p_out are the prices per million input and output tokens. For an N-step task, the total cost is:
cost_task = N × I × (p_in + r × p_out) / 1,000,000
For the same task, the input token volume and step count are determined by the prompt and the orchestration logic. As a result, the ratio of task costs between two models is approximately the ratio of their price-weighted unit prices:
price_eff = p_in + r × p_out
This has three implications:
- List price differences get amplified: the price gap carries through to the task level unchanged, and the larger r is, the more visible it becomes.
- List price does not determine the actual gap: vendors apply different context management strategies, so task costs can still differ materially at the same list price.
- r is the key selection parameter: tasks with high output share — long chain-of-thought reasoning, code generation, structured output — are more sensitive to the output price.
Cache hits dilute input cost further. Illustrative code:
# Illustrative algorithm: prices are hypothetical and only demonstrate the calculation.def task_cost(input_tokens: int, output_tokens: int, p_in: float,p_out: float, cached_input: int = 0) -> float:"""Compute task cost, expressed in the per-million-token pricing unit.Args:input_tokens: Cumulative input tokens (including multi-step replay)output_tokens: Cumulative output tokensp_in/p_out: Input/output price per million tokenscached_input: Input tokens served from cache (billed at the cache rate)"""billable_in = max(input_tokens - cached_input, 0)return (billable_in * p_in + output_tokens * p_out) / 1_000_000def eff_price(p_in: float, p_out: float, ratio: float,cache_hit: float = 0.0, p_cache_factor: float = 0.0) -> float:"""Compute the effective unit price: folds the output ratio and cachehits into the input side.ratio = output tokens / input tokenscache_hit = share of input tokens served from cachep_cache_factor = cache price as a multiple of the input price (e.g. 0.25means billed at one quarter)"""return p_in * ((1 - cache_hit) + cache_hit * p_cache_factor) + ratio * p_out
3.2 Quantifying the difference with illustrative data
Consider an illustrative case: 200,000 input tokens and an output-to-input ratio of 0.15, i.e. 30,000 output tokens.
| Dimension | Tier A (high price) | Tier B (mid price, same tier) | Tier C (low-cost tier) |
|---|---|---|---|
| Input price (RMB / million tokens) | 80 | 40 | 1.2 |
| Output price (RMB / million tokens) | 400 | 160 | 4.8 |
| Cost per task (RMB) | 76.0 | 32.8 | 0.384 |
| Multiple of Tier C | ~198× | ~85× | 1× |
Task costs between products in the same tier can differ by more than twofold. The low-cost tier cuts agent cost by an order of magnitude not through discounting, but by driving both input and output unit prices down to roughly one hundredth of the previous level.
3.3 Matching cost to task type
A cost advantage only matters if it lands on the right workload. Output-heavy long-chain tasks are more sensitive to unit price; tool-call-heavy, short-step tasks are more sensitive to the quality of the model’s decisions. The sensible approach is to group workloads by task profile first, then pick a tier per group, rather than switching everything to a single “cheapest” model.
4. Agent Tasks: Step Budgets and Context Governance
4.1 Agent cost grows with steps
Agent tasks involve multiple reasoning passes: planning, tool invocation, result verification, failure retries. Cost grows roughly linearly with step count, but two amplifiers are easy to lose control of:
- Failure retries: repeated attempts when output formatting is invalid or tool-call arguments are malformed.
- History accumulation: the full history is resent at every step, so per-step input tokens grow with the step count.
4.2 Engineering controls
- Dual caps on steps and budget: set a maximum step count and a token budget; terminate on whichever is hit first and return partial results.
- Context compression: retain key decisions, tool return summaries, and the pending-task list; fold away redundant intermediate output.
- Prompt caching: place system prompts, tool descriptions, and stable long-lived knowledge in the prefix, and keep their content and ordering stable to raise the cache hit rate.
- Structured result validation: require output to match a fixed schema; on validation failure, retry with explicit error feedback and cap the number of retries.
- Tiered model routing: send light tasks such as planning, extraction, and formatting to lower-tier models, and reserve key judgment for higher-tier models.
# Illustrative pseudocode: step budget + context compression + schema validationMAX_STEPS = 30MAX_TOKENS = 2_000_000CTX_LIMIT = 60_000def run_agent(task, planner, worker):history = [{"role": "system", "content": SYSTEM_PROMPT},{"role": "user", "content": task}]used = 0plan = planner(history) # High tier: decompose the task, pick a strategyfor step in range(MAX_STEPS):# Guard: return available progress once the budget or step cap is reachedif used >= MAX_TOKENS or step >= MAX_STEPS:return {"status": "partial", "plan": plan, "history": history}history = compress(history, keep=CTX_LIMIT) # Summarized compression, caps per-step inputaction = worker(history, plan) # Low tier: execute one stepused += action.usageobservation = call_tool(action) # Tool execution and result write-backif not valid_schema(observation): # Schema validationobservation = {"error": "invalid", "hint": "Return the agreed fields"}history.append({"role": "assistant", "content": action.text})history.append({"role": "tool", "content": observation})if done(observation): # Termination checkreturn {"status": "done", "result": observation, "tokens": used}return {"status": "partial", "history": history}
4.3 Estimating long-task cost
Plug assumptions such as “about 20,000 context tokens per step, an output-to-input ratio of 0.15, and an average of 20 steps to finish a task” into the effective unit price formula and the total cost of a long task follows:
| Tier | Effective unit price (RMB / million tokens) | Cost per step (RMB) | Cost of a 20-step task (RMB) |
|---|---|---|---|
| Tier A (high price) | 140.0 | 2.80 | 56.0 |
| Tier B (mid price) | 64.0 | 1.28 | 25.6 |
| Tier C (low-cost) | 1.92 | 0.0384 | 0.768 |
Long-task cost is highly sensitive to unit price, while step count and context length are variables you can actively reduce on the engineering side.
5. Always-On Assistants: From Q&A to Continuous Execution
Leading vendors have successively shipped personal assistant products that stay resident in the cloud and keep processing email, calendars, and document analysis while the user is offline. This form factor moves LLM applications from single-turn Q&A to long-horizon delegation.
For product and engineering teams, the capability requirements abstract into the following:
- Triggering: both scheduled execution and event-driven execution, with tasks advancing automatically while the user is offline.
- Tool authorization: connecting to external systems such as email, calendar, and financial data requires least privilege and tiered authorization.
- State persistence: save task progress and user preferences across sessions instead of re-deriving context from scratch each time.
- Cost control: the call volume of resident tasks depends on task complexity and user scale, so budget caps and degradation policies are required.
- Observability and traceability: every automated execution needs an audit trail for review and incident triage.
Building this class of application on Baidu AI Cloud can draw on a composition of general-purpose cloud capabilities:
- Host the agent runtime and tool gateway on a container platform or function compute.
- Drive task queuing and peak shaving with a message queue.
- Store task snapshots and run logs in object storage.
- Track task success rate, token consumption, and abnormal interruptions with logging services and monitoring alerts.
# Illustrative configuration: budget and scheduling policy for an always-on assistant taskassistant:schedule:cron: "0 */2 * * *" # Routine execution every 2 hourstriggers:events: [mail_received, calendar_changed]budget:tokens_per_run: 300000 # Token cap per runmax_steps: 25 # Max steps per rundaily_tokens: 3000000 # Daily cumulative cap for the taskdegrade: # Degradation policy once the budget is exhaustedmodel: fast_tiermode: summarize_only # Summarize only, no write operationspermissions: # Tool permissions granted on a least-privilege basismail: readcalendar: read, writeaudit:log_retention_days: 180 # Execution log retention period
6. Reading Call-Volume Data: Structure Matters More Than Absolute Scale
Call-volume data from third-party aggregation platforms is usually published weekly and includes total tokens, text request counts, and token share for the major models. Its value lies in observing migration direction and pace. Four cautions apply when interpreting it.
6.1 Watch average tokens per request
When total token growth consistently outpaces request growth, the average tokens per request is rising. Three causes are common:
- Longer context windows carry more history and reference material per input.
- More agent-style calls split a single task across more request rounds.
- A higher share of multimodal and long-document processing.
If a model’s total tokens surge while request counts stay flat, individual requests are almost certainly getting heavier rather than the user base growing.
6.2 Separate new capacity ramp-up from share migration
When a new version ramps up, older versions’ tokens typically contract at the same time. If the aggregation is keyed by model slug, the old/new crossover is split into two curves and is easily misread as “a model declining.” The correct approach is to merge old and new versions of the same model before computing migration.
6.3 Watch for share cliffs
A sharp drop in an individual model’s share usually corresponds to one of three situations: a default-version switch, falling behind on capability benchmarks, or a change in pricing or availability. Cross-validate against the release cadence and your own test results instead of concluding outright that a model is unusable.
6.4 Understand the scope boundary
Aggregated data comes from a single platform’s call records and covers traffic routed through that platform only. It excludes direct API access, private deployments, and internal calls. It will mislead if used to judge “total industry demand,” but it is reliable for observing relative migration between comparable models.
7. Grounding Capability Changes in Your Own Eval Set
The gap between leaderboard scores and real tasks is the most common trap in model selection. Build a small, stable internal eval set covering the typical scenarios in your business:
- Long-horizon coding: cross-file changes, dependency understanding, and regression verification.
- Structured extraction: fixed-schema output, field completeness, and accuracy.
- Long-context retrieval: locating and synthesizing key information inside very long documents.
- Tool calling: multi-step success rate and argument validity.
- Content generation: style consistency, factual accuracy, and human preference ratings.
Once defined, keep the eval set stable. When models are updated, replace the model and not the eval set, so that longitudinal data stays comparable. An illustrative evaluation flow:
# Illustrative evaluation flow: one eval set, multiple samples, ranked by pass rate and costdef evaluate(model, cases, samples: int = 3):passed, tokens = 0, 0for case in cases:for _ in range(samples):resp = model.run(case["input"], schema=case.get("schema"))tokens += resp.usagepassed += case["check"](resp.output)total = len(cases) * samplesreturn {"model": model.name,"pass_rate": passed / total, # Pass rate, reduces variance from a single sample"avg_tokens": tokens / total, # Average tokens per task, used for cost normalization}def rank(models, cases, price_in, price_out, ratio=0.15):results = [evaluate(m, cases) for m in models]for r in results:# Cost per task = average tokens per task × effective unit pricer["cost"] = r["avg_tokens"] * eff_price(price_in[r["model"]],price_out[r["model"]], ratio) / 1_000_000# Sort by pass rate descending, then cost ascending, to get the cost-performance frontierreturn sorted(results, key=lambda x: (-x["pass_rate"], x["cost"]))
8. Common Pitfalls
- Looking only at list prices: ignoring context replay and output share severely understates task cost.
- Ignoring migration cost: switching models requires re-tuning prompts, tool schemas, and retry policies, and migration cost sometimes exceeds the savings from the price gap.
- Overlooking stability: eval-set pass rate is an average, and failure retries inside long tasks can multiply the real cost.
- Leaving step count uncapped: an agent without step and budget limits can burn a whole month’s budget on a single bad run.
- Blindly chasing the top tier: in most business scenarios, mid- and low-tier models with good context management already meet the requirement.
- Ignoring compliance differences around open weights: self-hosted weights offer better data control, but operations and tuning costs must be counted in total cost of ownership.
9. Summary
The pace of weekly model iteration will not slow down. The teams that keep capturing value are usually not the ones that adopt the newest model first, but the ones that build a measurement system first. Work on a weekly cadence and follow this checklist:
- Define the key metrics across the four signal layers and fix the collection methodology.
- Build a cost-per-task model that folds in output share and cache hits.
- Set three caps for agents: steps, tokens, and daily budget.
- Use a stable internal eval set for longitudinal version comparison and produce a cost-performance frontier.
- Combine period-over-period call-structure analysis to judge migration direction and adjust routing in time.
Turning every model update into an engineering decision that is measurable, comparable, and reversible is what keeps cost and quality in balance through fast iteration.
