LLM Usage Tracking: Metric System Design and Implementation in Practice

1. Why an “Adoption Signal” Tracking System Is Needed

As large language models move from early experimentation to large-scale production use, “how well is a given model actually being adopted” has become a shared concern for engineering teams, researchers, and decision-makers. But adoption is inherently a multi-dimensional problem, and any single metric is easy to misjudge:

• A rise in API call volume may simply reflect longer context per task, not more users coming in;
• A rise in user count may be driven by a short-term campaign and may not represent genuine stickiness;
• A rise in spending may stem from pricing adjustments or a drop in cache hit rate, not from expanded usage.

A mature approach, therefore, is to build a multi-dimensional metric system in which four categories of signals—absolute scale, growth rate, cost intensity, and relative position—cross-validate each other. A typical case illustrates the value: a model on a certain intelligent coding platform showed steadily rising connected users, cumulative spending, and token consumption, yet its ranking and share across the whole market stayed flat for a long time. Looking only at absolute values would lead to the optimistic conclusion that “adoption keeps strengthening.” Only by introducing relative metrics can you identify the real structure: “growth diluted by simultaneous expansion of the overall market.” This article offers an actionable scheme around exactly this kind of problem.

2. Definitions and Calibers of the Four Core Metric Categories

1. User Scale

• Stock metrics: connected/active user count, measuring reach;
• Increment metrics: daily new users;
• Rate metrics: day-over-day growth rate = daily new users ÷ previous day’s stock.

Note the distinction between “connected users” and “active users”: the former is a cumulative caliber that only increases; the latter is a daily caliber that reflects real usage. A long-term divergence between the two often means strong acquisition but weak retention.

2. Cost Consumption

• Cumulative spending: reflects total historical investment;
• Daily spending increment: reflects the day’s consumption rate;
• Per-capita calibers: cumulative spending ÷ user count, daily spending ÷ active users, used for cross-period and cross-object comparison.

3. Usage Intensity

Centered on token consumption, including cumulative consumption, daily increment, and growth rate. Tokens are a fairer measure of intensity than “call count,” because they eliminate the incomparability between “high-frequency small requests” and “low-frequency large tasks.”

4. Relative Position

• Share = own usage ÷ total market usage;
• Ranking: the ordinal expression of share.

Anonymized Snapshot Example

Below is a typical single-day tracking snapshot (data from an anonymized example of a public tracking scenario on a certain platform):

Metric	Snapshot Value	Daily Increment	Day-over-Day
Connected users	432K	+18K	~+4.3%
Cumulative spending	$980K	+$38K	~+4.0%
Cumulative token consumption	251B	+12B	~+5.0%
Ranking / Share	9th	—	0.5% (flat)

From this single row of snapshot alone, several insights can be derived through simple calculation: cumulative per-capita spending is about $2.27 per user; the daily spending increment corresponds to about $0.09 per person per day against the previous day’s stock of users; and “all three absolute metrics rising while share stays motionless” is exactly the input for the divergence analysis below.

3. From Raw Logs to Daily Snapshots: Collection and Computation Pipeline

A reusable pipeline has five layers:

1. Data source layer: API gateway access logs, billing records, client-side instrumentation;
2. Collection buffer layer: logs are collected uniformly and written to a message queue, smoothing peak traffic;
3. Computation layer: streaming tasks for real-time observation, batch tasks for end-of-day aggregation and backfill;
4. Storage layer: metrics written to a time-series database, raw details archived to object storage;
5. Service layer: provides snapshot queries, dashboards, and alerts.

Daily-granularity aggregation can be illustrated with a generic SQL snippet:

SELECT
    date_trunc('day', request_time)        AS stat_day,
    count(DISTINCT user_id)                AS active_users,
    sum(prompt_tokens + completion_tokens) AS total_tokens,
    sum(billed_amount)                     AS total_cost
FROM model_call_logs
WHERE model_id = 'model-a'
GROUP BY 1
ORDER BY 1;

On this basis, tracking snapshots are generated from two adjacent days’ aggregation results:

def build_snapshot(today: dict, yesterday: dict) -> dict:
    """Generate a daily tracking snapshot from two adjacent days' aggregation results"""
    def rate(cur, prev):
        return (cur - prev) / prev if prev else None
    return {
        "users":        today["active_users"],
        "users_delta":  today["active_users"] - yesterday["active_users"],
        "users_growth": rate(today["active_users"], yesterday["active_users"]),
        "tokens_delta": today["total_tokens"] - yesterday["total_tokens"],
        "tokens_growth": rate(today["total_tokens"], yesterday["total_tokens"]),
        "cost_delta":   today["total_cost"] - yesterday["total_cost"],
    }

This kind of pipeline does not need to be built from scratch and can be constructed directly on a cloud platform: for example, on Baidu AI Cloud, you can use logging services to collect and search call logs, a message queue to absorb peak traffic, monitoring and alerting products to host metric storage, visualization, and threshold alerts, and a container platform to run custom aggregation computation tasks. Each layer is a general industry capability category, and the combination can be flexibly adjusted to a team’s technology stack.

4. A Framework for Interpreting the “Growth Up, Share Flat” Divergence

This is the most common and most easily misread pattern in adoption tracking, and it is best analyzed in four steps.

1. Mathematical Decomposition

Let share S = own usage ÷ total market usage. Then the approximate relationship of its change is:

ΔS / S ≈ g(own growth rate) − g(market growth rate)

That is: share is a relative concept, determined by the growth-rate difference between oneself and the overall market, not by one’s own growth rate alone. Looking back at the snapshot: user growth of about 4.3% and token growth of about 5.0%, while share held steady at 0.5%. One can infer that the market’s overall growth rate during the same period was roughly of the same order—absolute rise, relative standstill.

2. Trend Judgment of the Growth-Rate Difference

A single-day snapshot is noisy, so you should observe the slope of “own growth rate − market growth rate” over several consecutive days: only sustained positive values will push share upward; fluctuations around zero maintain the status quo; sustained negative values mean that even if absolute values are still growing, it is merely “shrinking growth.”

3. Decomposing Growth Quality

• Marginal consumption intensity: the ratio of daily spending increment to daily new users (about $2.1 per new user in this example), which can be compared with the average daily spending of existing users to judge whether new users are “deep users” or “registered but idle”;
• Token/user trend: a sustained rise in tokens per user indicates deepening task complexity or dependence, a signal with more substance than user count;
• Retention cross-validation: if per-capita metrics rise while share does not, growth quality is acceptable and the bottleneck is acquisition speed.

4. Ranking Rigidity and Competitive Density

Ranking is the ordinal expression of share, and its movement depends on the density of gaps between adjacent positions: if the gap to 8th place is very small, a minor perturbation can swap positions; if the gap is significant, the ranking shows “rigidity” and stays flat for a long time. Therefore, when assessing relative position, present the share gap as well, rather than looking only at the rank number.

5. Dashboards and Alerts: Letting the System Run Automatically

A recommended three-tier dashboard structure: the top holds core metric cards (the four snapshot items plus day-over-day change); the middle holds trend lines (daily series of users, tokens, and spending with a 7-day moving average); the bottom holds the relative position section (share curve and gaps to adjacent competitors). All trend lines must be labeled with both caliber and statistical window to avoid misreading.

Alerts should be based on statistical anomalies rather than fixed thresholds. Example rules:

alerts:
  - name: abnormal_user_growth
    expr: abs(zscore(users_growth, window=14d)) > 3
  - name: share_breakout_notice
    expr: market_share > 0.006 for 3d
  - name: abnormal_cost_per_user
    expr: abs(zscore(cost_per_active_user, window=14d)) > 3

The abnormal cost per user deserves special attention: it often exposes cache failures, sudden task-structure shifts, or abusive behavior before an incident does.

6. Common Pitfalls in Engineering Practice

1. Caliber consistency: token statistics must specify whether they include cache hits, tool-call round trips, and failed requests; caliber drift is the number-one cause of trend breaks;
2. Late-arriving data backfill: cross-timezone and asynchronous billing cause data to arrive late, so snapshots must support idempotent recalculation by statistical day rather than simple accumulation;
3. Deduplication logic: user counts must be deduplicated by a stable subject identifier; beware of inflation caused by the same subject using multiple devices and accounts;
4. Privacy compliance: user-level statistics should be aggregated and de-identified, and detailed logs should have a retention period;
5. Sample representativeness: data from a single platform can only reflect local adoption and cannot be extrapolated to the whole; conclusions should be limited to “within that platform’s caliber.”

7. Summary

A reliable LLM adoption tracking system is essentially a combination of “four metric categories cross-validating each other + a standardized daily aggregation pipeline + a divergence analysis framework”: absolute values answer “how big,” growth rate answers “how fast,” cost answers “how heavy,” and share answers “what the relative position is.” When the “growth up, share flat” pattern appears, the correct conclusion is neither simply optimistic nor pessimistic, but rather to narrow the question down to “whether the difference between own growth rate and market growth rate is improving.” By turning this system into automated daily snapshots and alerts, you can continuously obtain a quantitative judgment of adoption trends at very low marginal cost.