The Hidden Variable in AI Assistant Training: Why a "Perfect" Simulated User Skews Your Model

Nearly every team that trains or evaluates AI assistants follows the same routine. They pick a large language model that is capable enough to pass as a knowledgeable human, instruct it to role-play an ordinary user, and then let it hold multi-turn conversations with the agent under training or under evaluation. The resulting dialogues become training signals or evaluation scores.

At first glance, the approach seems sound. A model cast as a user has seen a great deal: it can understand complex intent and express requirements in clear, well-organized language, which makes it look like an ideal sparring partner. Yet that seemingly reasonable choice introduces the biggest hidden risk in the whole training and evaluation system.

1. The Overlooked Premise: The User Changes the Task Itself

An agent’s capability is essentially a measure of how well it solves problems within a particular interaction distribution. That distribution is shaped by both sides of a conversation: the agent being tested and the user talking to it. Replacing the user with a model is not simply a matter of swapping in a new sparring partner. It rewrites the task’s difficulty, its information structure, and the standards by which the agent is judged.

The core problem is a capability mismatch. A model asked to play a user tends to be eager to help, clear in its phrasing, logically consistent, and quick to answer. Real people often behave quite differently. They may describe their needs vaguely, keep key constraints to themselves, lose patience under repeated questioning, or simply reply, “Whatever you think is best.”

The gap between these two kinds of user is a difference in distribution, not in degree. It determines whether the input an agent receives is clean or noisy, and whether each task is clearly defined or requires active clarification. Training or evaluating with the wrong user therefore skews the results in a systematic way.

2. A Data Set That Shows the Scale of the Distortion

To see how serious the distortion can be, consider a comparison experiment that isolates the user variable. The same agent was given the same set of tasks, with only the source of the user changed. The user came from three sources: real human users, simulated users played by several of the most capable commercial LLMs, and simulated users played by several lightweight models with weaker capabilities.

The results were striking:

• With real human users, the agent achieved a task success rate of about 63.6%.
• With top-tier commercial LLMs playing the user, success jumped to 82.4%–84.4%.
• With lightweight open-source models playing the user, success fell to 45%–48%.

The tasks were the same and the agent was the same. Only the user changed, yet the success rate swung by nearly 40 percentage points, from 45% to 84%. In other words, the simulated user you rely on to train or evaluate an agent is quietly redefining the difficulty baseline of the tasks themselves. You think you are measuring the agent’s true capability, but you are actually observing a heavily distorted picture of it.

3. Three Root Causes of the Distortion

Solving the problem starts with understanding where the distortion comes from. Broadly, it stems from three sources.

A stronger model cooperates more, so the task becomes easier. A top-tier model playing a user will proactively fill in information the agent is missing, restate vague requests in standard language, and offer hints when the agent gets stuck. In effect, it is an easy-to-get-along-with user that does much of the clarification, inference, and error tolerance the agent should be handling on its own. This quietly simplifies the task and inflates the scores.

A weaker model causes more disruption, so the task becomes nearly impossible. A lightweight model playing a user often fails to stay in character, expresses itself incoherently, gives irrelevant answers, and may even produce conversational noise that real users would never generate. The agent is then working with an input distribution that is distorted and, in places, impossible to resolve, which unfairly drags scores down.

Implicit criteria and natural noise are lost at the same time. Real users often have expectations they never state out loud, and those unspoken constraints are exactly what the agent is supposed to recognize on its own. Real users also bring natural vagueness, hesitation, and shifts in mood. When a user is replaced by a model, this real-world noise is either smoothed away or amplified into unnatural interference. Either way, the result drifts further from the real distribution.

4. From Training to Evaluation: Distortion Spreads Across the Pipeline

The fidelity of user simulation affects more than a single evaluation score. Distortion travels along the whole development chain.

In training, if the conversation data comes from an overly cooperative model playing the user, the agent learns a flawed world model in which users are always friendly and requirements are always clear. Once deployed and faced with real users who express themselves vaguely, it will lack the ability to clarify and infer. It will look unprepared from the first conversation.

In evaluation, if leaderboards and acceptance criteria are built on top of a “smart user,” teams will become overly optimistic about the agent’s capabilities. They may believe the product is ready, only for it to fail in real-world use. The reverse is also true: evaluating with an excessively noisy model undermines team confidence and hides genuine progress.

In short, user simulation is not a minor detail in the training process. It is an invisible variable that determines whether the whole system can be trusted, like a ruler that has never been calibrated. Every number it produces should be treated with skepticism.

5. How to Improve Simulation Fidelity: Practical Approaches

Since the problem is that simulated users do not behave enough like real people, the path forward is clear: bring the simulated distribution as close to the real one as possible. The following approaches are general enough to apply across teams and practical enough to implement.

1. Calibrate the simulator with real user data. Collect anonymized logs of real conversations and measure how human users typically express themselves, how often and at what point they ask for clarification, how often they give up, and how their implicit constraints are distributed. Use these statistics to constrain or calibrate the model playing the user, instead of letting it improvise freely.

2. Inject controllable noise and uncertainty. A simulated user should be allowed to be imperfect. It might sometimes express itself vaguely, need to be asked further before it shares additional information, or show impatience after being asked the same thing too many times. What matters is that this noise is a configurable variable, not something left to random generation.

# Example: adding controllable "imperfection" parameters to a user simulator
simulator_config = {
    "clarity": 0.6,              # probability that the user expresses itself clearly; lower values mean more vagueness
    "patience": 2,               # number of clarifying questions tolerated before replying something like "whatever works"
    "hidden_constraints": True,  # whether the user keeps implicit criteria that it never states
    "noise_level": "natural",    # noise should resemble real conversations rather than extreme randomness
}

3. Preserve and test implicit constraints. During evaluation, deliberately include tasks in which the user never states its criteria upfront. This lets you observe whether the agent can proactively clarify and infer intent, rather than waiting for requirements to be handed to it.

4. Build a control user set from multiple sources. Do not rely on a single model to play the user. Keep real human samples alongside samples from models of different capability levels, report the results for each source separately, and treat the differences between them as a stability metric worth monitoring.

5. Establish a difficulty baseline rather than reporting absolute scores alone. Give the same tasks to the agent while switching the user source, and track how much the scores move. A smaller spread indicates that the agent is more robust across different kinds of users. A larger spread is a warning sign that its performance may be propped up by one particular user distribution.

6. What This Means for Training and Evaluation

To return to the question we started with: why not let the smartest model play the user when training an AI assistant? Because it is too capable and too cooperative. It artificially lowers task difficulty and distorts both the agent and the evaluation at the same time.

The more valuable approach is to treat user simulation as an engineering problem that needs careful design and ongoing calibration. Use the real distribution as an anchor, allow simulated users to be imperfect, preserve the real world’s vagueness and unspoken constraints, and compare results at the level of the whole distribution. Only then will the agents you train hold up in real-world use, and only then will your evaluation scores genuinely reflect what the product can do.

The key to a stronger AI assistant is not a smarter sparring partner but a more realistic user. When simulated users come close enough to real ones, training signals and evaluation results become reliable at the same time, which puts development on a steadier footing over the long run.