Measuring the True Value of AI Models with Retention Data: Methods and Limitations

    This article explores how to evaluate the true value of AI models using user retention data rather than relying solely on traditional benchmarks. It discusses the limitations of static evaluations and details a methodology for calculating weekly retention for heavy single-model users, providing actionable insights for developers to assess long-term model performance and user preference.

    1. Why Measure Model Value with Retention Data

    One-off evaluations typically score models on fixed datasets, measuring their capability upper limits under specific conditions. When developers choose a model for real-world workflows, they care about something else: can this model consistently solve everyday problems and make users want to use it repeatedly?

    Retention metrics shift the perspective from “what the model can do” to “what happens after users choose it.” If a model is repeatedly selected for real tasks, it indicates that it has reached an acceptable balance for users among cost, speed, result quality, and usage habits. This signal comes from real behavior, and the sample size is much larger than that of manually constructed test sets.

    Retention itself does not equal quality. It is simultaneously influenced by factors such as quality, price, availability, default settings, and migration costs. Therefore, retention is better suited as a dimension within an evaluation system, rather than the sole conclusion.

    2. Three Limitations of Traditional Evaluation

    1. Disconnect between datasets and real tasks. Benchmark questions are typically independent, standardized, and auto-gradable. Real development tasks, however, involve multi-file contexts, ambiguous requirements, and iterative modifications.
    2. Scores do not reflect usage costs. Two models with similar scores might differ significantly in per-call pricing, response latency, and failure retry rates.
    3. Static results lack a time dimension. Model versions update frequently, and user experience may change over time, which a one-time evaluation struggles to capture.

    Retention data can fill these gaps. It records long-term selection behavior and can be updated on a rolling weekly basis.

    3. Defining Weekly Retention for Heavy Single-Model Users

    To quantify “continuous use,” three concepts need to be clearly defined first:

    • Heavy single-model user: A user who sends over 80% of their requests to the same model within a given week.
    • Primary model of the week: The model that is predominantly used as defined above.
    • Next-week retention: The proportion of users who still use this model as their primary model in the following week. Here, “using it as the primary model” can be defined as the model accounting for more than 50% of requests. This threshold should be clearly stated in the report.

    This definition serves two purposes. First, it only counts users with a clear preference for a specific model, excluding cases where users frequently switch or mix multiple models. Second, it ties retention to selection behavior; the higher the proportion, the stronger the stable usage habits formed by the users.

    The following example calculates this metric based on request logs. The logs contain user ID, week number, model name, and request count.

    ```python
    import pandas as pd

    def weekly_heavy_retention(df, heavy_thr=0.8, keep_thr=0.5):

    1. # df fields: user_id, week, model, requests
    2. per_model = df.groupby(["user_id", "week", "model"], as_index=False)["requests"].sum()
    3. totals = (per_model.groupby(["user_id", "week"], as_index=False)["requests"]
    4. .sum()
    5. .rename(columns={"requests": "total"}))
    6. per_model = per_model.merge(totals,