Rebuilding AI Application Metrics: From User Count to Compute and Output
A recent earnings report drew more attention to an AI coding assistant's usage figures than to overall revenue: the product consumes roughly 100 trillion (100T) tokens per day despite having only about one-twentieth the daily active users of a general-purpose assistant with roughly 200 million DAU. This article explains why agentic products consume far more compute per user than chat-style products, why user count has become an inadequate standalone metric for AI applications, and how teams can rebuild their measurement system around a three-layer framework—usage, efficiency, and business metrics. It proposes "value output per unit of compute" as a north star metric, supported by quality, experience, and cost guardrails, and provides a unified instrumentation schema plus a reusable SQL query template for tracking compute cost and output at the task level.
1. The Signal: Token Consumption and User Scale Are Drifting Apart
In a major internet company’s latest earnings report, the figure that drew the most attention was not overall revenue but the usage data of an AI coding assistant. The publicly disclosed numbers show:
- Usage leader: The product consumes about 100 trillion (100T) tokens per day on average—more than a general-purpose AI assistant with roughly 200 million daily active users.
- Different reporting standards: General-purpose assistant products typically do not disclose token consumption separately. One cloud provider reported a company-wide average of about 180 trillion daily token calls in June of the same year, but it is difficult for outsiders to determine how much of that came from consumer-facing assistant applications.
- Large scale gap: This coding assistant has just over 10 million DAU, while the general-purpose assistant has roughly 200 million—about one-twentieth of the latter.
It is worth noting that there is still room for debate about whether tokens from two products with different form factors can be compared directly. But the comparison “an app with 10 million DAU consumes more compute than one with 200 million DAU” at least shows that, on the dimension of token consumption, user scale and resource investment have already diverged noticeably.
1.1 An Intuitive Explanation: Different Product Forms, an Order-of-Magnitude Gap per User
Let’s run a simplified estimate. Suppose an ordinary chat user sends about 20 rounds of questions and answers per day, with an average of 3,000 tokens of context per round, for a per-user daily consumption of about 60,000 tokens. Now suppose an agent user runs 3 tasks per day, each averaging 20 steps of reasoning and tool calls, each step carrying about 40,000 tokens of context; after accounting for roughly 15% retry overhead, that comes to about 2.76 million tokens per user per day:
Coding assistant: 10,000,000 × 2,760,000 ≈ 2.76 × 10^13 tokens/dayGeneral AI assistant: 200,000,000 × 60,000 ≈ 1.20 × 10^13 tokens/day
The ratio is about 2.3—a product with only about 5% of the users consumes roughly 2.3 times the compute.
The parameters here are illustrative only; real values need to be calibrated against actual instrumentation data. But this estimate reveals a pattern: when per-user compute consumption between two product types differs by one or two orders of magnitude, “who has more users” no longer equals “who consumes more compute and drives larger compute resources.”
1.2 Reporting Standard Differences: No Simple Side-by-Side Comparison
The case above compares products of different forms. When comparing token usage across vendors, you also need to account for the fact that reporting standards are not fully consistent: how input and output tokens are converted, whether cache-hit input is counted, and how multimodal content is converted all vary from vendor to vendor. Treating token counts as a universal yardstick and comparing them directly easily leads to misleading conclusions.
2. Mechanism Analysis: Why Agents Are More Compute-Hungry Than Chat Products
Token consumption can be broken down into four factors:
Total tokens = number of calls × average context size per call × output length × (1 + retry amplification factor)
For a chat product, “number of calls” roughly equals the number of user questions. But a single agent task issues multiple consecutive model calls within a loop of “think—call tool—observe result—think again,” so all four factors are amplified at once.
2.1 The Task Loop: Call Count Grows Multiplicatively
The typical execution flow of an agent looks like this:
User issues a task→ Plan: break it down into steps→ Execute: call tools such as search, read file, run command, submit code→ Observe: feed tool results back into the context→ Reflect: judge whether the result meets requirements, and decide to continue, correct, or finish→ (loop several rounds) output the final result
The user issues a single instruction, but the model must make decisions across multiple consecutive rounds; the number of loop rounds typically ranges from single digits to several dozen.
2.2 Context Expansion: Input Token Size Keeps Rising
An agent’s context contains at least four parts: system prompts and tool descriptions, conversation and task history, tool call results from each step, and code repository structure or long document fragments. The context only grows during a task, never shrinks, and the same historical content is fed to the model repeatedly at every subsequent step.
Linear growth can become even steeper:
- Secondary growth from retries: When a tool call fails, parameter validation fails, or the result falls short of expectations, the model must re-plan based on the error message and execute again, which can double the total number of steps.
- Amplification from acceptance feedback: Take code generation as an example. After the model writes code and runs tests, it pastes the entire failure log back into the context to keep fixing it—a single log can easily be thousands of tokens.
2.3 Output Tokens and Result Verification: Hidden Costs Count Too
An agent does not just output final text; it also produces structured tool-call parameters, intermediate plans, and internal reasoning content generated by reflection and planning models. In addition, agents often treat automatic checks such as test results, code reviews, and static analysis as part of the completion loop, and these steps consume compute as well. This investment is critical to whether a task is truly complete, and it should not be dismissed as waste when evaluating value.
2.4 Estimation Model: Breaking Consumption into Optimizable Factors
# Estimate per-user daily token consumption (illustrative; calibrate parameters with real instrumentation data)def estimate_daily_tokens(chat_rounds: int, # average daily chat roundstasks: int, # average daily agent taskssteps: int, # average reasoning/tool-call steps per taskbase_ctx: int, # base context per step (system prompt + tool descriptions)growth: int, # context added per step, such as new tool resultsoutput_tokens: int, # average output per step (including tool-call params, internal reasoning)retry_rate: float, # extra overhead ratio from failure retries) -> int:chat = chat_rounds * base_ctx # simplification: similar context size across chat roundsfirst_step = base_ctx + output_tokenslater_step = base_ctx + growth + output_tokensagent = tasks * (first_step + (steps - 1) * later_step) * (1 + retry_rate)return chat + agent
From this model, we can see that context grows roughly quadratically with the number of steps: the agent accumulates history step by step, and each additional step re-feeds all prior steps’ content to the model. This also explains why a single long task can equal a whole day of an ordinary user’s conversation consumption.
3. Metric Breakdown: Why “User Count” Is No Longer an Effective Metric
- Per-user consumption variance is too large: A single chat-style Q&A and a multi-step agent task can differ by one to two orders of magnitude in token consumption. Counting only “how many people are using it” essentially erases the variable with the largest variance.
- User scale does not equal resource commitment: The per-token compute cost is rigid. Serving 200 million light users and serving 10 million heavy users may require compute on the same order of magnitude—or the latter may require even more. For capacity planning and budget management, user count no longer has predictive power.
- It creates wrong incentives: Judging success by token usage incentivizes teams to “inflate the numbers” by adding steps and lengthening context; judging success by user count, in turn, masks the compute pressure from heavy users. Both can steer evaluation away from true value.
It should be emphasized that this does not mean user count no longer matters. User scale, retention, and paid conversion remain core metrics for the health of an AI application. The problem is that user count can no longer, on its own, represent investment scale, cost level, and commercial potential.
4. System Rebuild: Three-Layer Metrics and a North Star Metric
4.1 Three-Layer Metric Framework
| Layer | Core Question | Representative Metrics |
|---|---|---|
| Usage | Who is using it, and how much | DAU, penetration rate, tasks per user, task frequency distribution, retention |
| Efficiency | How much compute does it take to get one thing done | Task success rate, first-pass rate, tokens per successful task, average steps, retry rate, cache hit rate, end-to-end latency |
| Business | What did the compute investment bring back | Cost per successful task, value output per unit of compute, compute cost as a share of revenue, marginal gross margin |
The three layers form a “funnel”: the usage layer determines scale, the efficiency layer determines unit cost, and the business layer determines whether the investment is sustainable. Looking at any single layer in isolation can lead to a one-sided conclusion.
4.2 Rigorous Definitions of a Few Key Metrics
- Task success rate: The proportion of tasks whose final state meets preset acceptance conditions (e.g., tests pass, review passes, user confirms)—not the proportion of cases where “the model gave an answer.”
- First-pass rate: The proportion of tasks that meet acceptance conditions on first delivery, used to measure rework cost.
- Tokens per successful task: Total token consumption divided by the number of successful tasks. It loads the cost of failures and retries into the cost side of the denominator, making it closer to real value output than “tokens per call.”
- Compute efficiency: The number of successful tasks or business value produced per unit of compute, used to compare efficiency across different product forms and model combinations.
4.3 The North Star Metric: Value Output per Unit of Compute
We recommend adopting “value output per unit of compute” as the overarching metric:
Value output per unit of compute = Σ(successful tasks × task value weight) / compute cost
Here, the task value weight can be tiered by task type—such as core business automation, assistive generation, and simple Q&A—while compute cost is uniformly converted into resource expenditure under the same standard. The resulting metric satisfies three requirements at once:
- It measures output by “tasks” rather than by “people”;
- It puts cost explicitly in the denominator;
- It is comparable across different product forms.
4.4 Guardrail Metrics: Preventing Single-Metric Optimization from Going Off Track
A north star metric can easily be “optimized” into a numbers game, so it must be paired with guardrail metrics:
- Quality guardrails: task success rate, human rework rate;
- Experience guardrails: first-token latency, task completion time P95, failure rate;
- Cost guardrails: tokens per successful task, cost per successful task, peak utilization.
Before any optimization ships, both the north star and guardrail metrics should be observed together, to avoid a false gain where “unit cost went down but tasks got done incorrectly.”
5. Data Foundation: Unified Measurement Standards and Instrumentation Design
5.1 Define Standards First
- Token: Specify how input, output, and cache-hit input are counted, and how reasoning content is handled, using the same conversion rule company-wide.
- Session: One continuous usage process.
- Task: A single verifiable goal from the user, composed of several steps.
- Step: One model inference, which may carry several tool calls.
We recommend locking these three levels down in the data dictionary before discussing metrics. Otherwise the same “call volume” means different things to different teams, and no amount of data can be aggregated.
5.2 Unified Instrumentation Fields
{"event_id": "uuid","session_id": "session unique identifier","task_id": "task unique identifier","step_index": 3,"user_type": "free|paid|enterprise","model_role": "planning|coding|verification","input_tokens": 41230,"cached_input_tokens": 36800,"output_tokens": 1520,"reasoning_tokens": 2100,"tool_calls": ["search", "read_file", "run_tests"],"latency_ms": 8600,"status": "success|failed|retry","task_result": "accepted|rejected|timeout","ts": 1738368000000}
5.3 Collection Pipeline
Client/server instrumentation→ Message queue (peak shaving, decoupling)→ Stream processing (aggregate by session and task, real-time dashboard)→ Data warehouse/OLAP (multi-dimensional analysis, cost accounting, long-term retention)
For long-task, compute-heavy scenarios like coding agents, we recommend treating the “task” as the smallest unit of cost attribution. This lets you both calculate how much compute each task consumed and what it accomplished, and pinpoint the task types and user groups with high consumption.
6. Metric Standards Reference: A Ready-to-Reuse Query Template
The following SQL is illustrative; field names need to be adjusted to your actual table structure:
-- Core metrics: successful tasks, token consumption, value output per unit of computeWITH task_stat AS (SELECTt.task_id,t.task_type,t.task_result, -- accepted / rejected / timeoutSUM(e.input_tokens) AS input_tokens,SUM(e.cached_input_tokens) AS cached_tokens,SUM(e.output_tokens) AS output_tokens,SUM(e.reasoning_tokens) AS reasoning_tokens,COUNT(*) AS steps,SUM(e.latency_ms) AS total_latencyFROM dwd_model_call_event eJOIN dim_task t ON e.task_id = t.task_idWHERE e.dt BETWEEN '2025-09-29' AND '2025-10-05'GROUP BY t.task_id, t.task_type, t.task_result)SELECTtask_type,COUNT(*) AS tasks,SUM(CASE WHEN task_result = 'accepted' THEN 1 ELSE 0 END) AS accepted_tasks,SUM(input_tokens + output_tokens + reasoning_tokens) AS tokens,SUM(input_tokens + output_tokens + reasoning_tokens)/ NULLIF(SUM(CASE WHEN task_result = 'accepted' THEN 1 ELSE 0 END), 0) AS tokens_per_accepted,AVG(steps) AS avg_steps,AVG(total_latency) AS avg_latency_msFROM task_statGROUP BY task_type;
The value of a query like this is that once “tokens per successful task” is broken down by task type, the team can see exactly which kinds of tasks the compute is being spent on and which tasks are worth continued investment.
7. Common Pitfalls and Optimization Directions
- Token-only thinking: Treating “how many tokens are consumed” itself as a scorecard incentivizes teams to drive high consumption rather than high value. The right question is “how many things get done per unit of compute.”
- Looking only at averages: Heavy users and long tasks pull averages up sharply, so it is better to look at P50, P90, and per-user percentiles at the same time.
- Ignoring failure cost: Failed and timed-out calls also consume tokens. We recommend breaking out a “wasted compute ratio”—the share of total consumption spent on failures and repeated attempts—as a dedicated optimization target.
On this basis, several general optimization approaches can be considered:
- Context engineering: Put only content relevant to the current step into the context, and summarize and compress historical tool results;
- Cache-friendly design: Keep the prefix of system prompts and tool descriptions stable to improve the chances of prefix cache hits;
- Fast/slow tiering: Assign lightweight steps such as intent recognition and result verification to smaller models, and complex planning and generation to larger models;
- Global budget: Set a step limit and a token limit for each task; once exceeded, route to human confirmation or fall back to a degraded mode.
8. Conclusion: Shift the Focus from “How Many People Use It” to “How Much Gets Done”
Token consumption is worth attention not because it is important in itself, but because it is the first thing to make the compute investment of AI applications observable, attributable, and comparable. When a product with 10 million DAU consumes more compute than one with 200 million DAU, what is truly being rewritten is not the ranking of some product, but the way the entire industry evaluates AI applications.
For technical teams, we recommend landing this in three steps:
- Unify standards: Write the definitions of the four levels—token, session, task, and step—into the data dictionary;
- Build solid instrumentation: Use the task as the smallest cost unit and connect the “consumption—output” chain;
- Establish a metric system: Use value output per unit of compute as the north star, backed by quality and experience guardrail metrics.
A pragmatic judgment is this: when measuring AI applications, “how many people use it” still matters, but it is giving way to “how much compute was used and how much got done.” This shift has a far greater impact on capacity planning, cost governance, and product decisions than any single profit figure on a report—and is far more worth watching over the long term.
