Making “Fake Users” Behave Like Real People: Simulation Bias in Agent Training

An AI agent that must complete tasks through multi-turn dialogue usually depends on a “user simulator” during training and evaluation. Another model plays the user: it states a need, provides additional information, accepts or rejects proposed solutions, and responds to follow-up questions.

This approach is inexpensive and repeatable, but it has a structural problem that is easy to overlook: the simulator’s communication habits can rewrite the task itself.

A helpful, complete, logically consistent model-user does not come from the same distribution as a real person who speaks vaguely, keeps their criteria to themselves, and becomes dismissive when asked too many questions. Strategies that work well with the former often fail with the latter.

Where Simulation Bias Comes From

A common industry pipeline selects a capable conversational model, gives it a role description, and has it interact with the agent being trained. This seems reasonable because a strong model can supposedly play almost any role.

The problem is that these models tend to clarify requirements, list constraints, and make the next step explicit. They rarely exhibit four common patterns of real-user behavior.

First, users often provide incomplete descriptions. They may state only part of a goal, such as “Take care of last week’s report,” without specifying the date range, definitions, output format, or acceptance criteria.

Second, users often have implicit standards. They may know what they prefer but fail to state it. Only after seeing a result will they reject it with comments such as, “That’s not what I had in mind.”

Third, users have limited interaction budgets. After one or two rounds of follow-up questions, they may answer “Whatever,” “You decide,” or simply end the conversation.

Fourth, users are inconsistent. They may change their mind, forget earlier context, or combine two unrelated requests in a single sentence.

When a simulator systematically avoids these behaviors, the agent learns not how to make progress under uncertainty, but how to execute a process cleanly when all relevant information is already available. Evaluation scores become inflated. Once deployed, the agent may not know how to ask useful questions, make reasonable inferences, or provide a fallback plan when information is missing.

How the Simulator Rewrites Task Difficulty

One line of research uses the same agent and the same task set while changing only the source of the “user.” When interacting with real people, the agent achieves a moderate success rate. When a highly capable commercial conversational model plays the user, the success rate rises significantly, with the difference reaching roughly 20 percentage points. When a weaker open-source model plays the user, the success rate falls by a similar order of magnitude.

This comparison is not primarily about which model is smarter. It shows that the evaluation conditions have changed.

A strong model-user effectively puts the agent in easy mode. The requirements are automatically completed, the cost of clarification is absorbed by the simulator, and failure paths are politely avoided.

A weak model-user may create a different kind of difficulty: irrelevant answers, poor task-state maintenance, or noise unrelated to the goal. In that case, the agent’s failure mixes two causes: the agent may be inadequate, but the simulated user may also be impossible to work with.

Neither score is a reliable estimate of capability on the real-user distribution.

User simulation should be treated as part of the environment dynamics. If the environment is too easy, the policy overfits to the assumption that the other party will state every constraint. If it is too difficult and its noise does not resemble real behavior, the policy may instead overfit to the assumption that the other party is completely uncooperative.

External validity improves only when the simulator’s information-revelation patterns, impatience curve, and refusal behavior are reasonably close to those of real users.

How Training Signals Become Contaminated

In reinforcement learning and preference optimization, the user simulator serves as both part of the environment and a source of reward signals.

If the simulator is always cooperative, the agent can receive consistently positive feedback simply by producing long clarification checklists or formulaic confirmations. This is a form of reward hacking against the simulator. The policy optimizes for “making the fake user satisfied,” rather than reaching an acceptable result with a real user in fewer interactions.

Common symptoms include:

• The number of clarification turns is low on simulated tasks but rises sharply with real users.
• Simulated users prefer a complete solution in one step, while real users often need a minimally executable version first, followed by iteration.
• Simulated users rarely change requirements midway through a task, while a single change in production causes the agent to restart from scratch.

When these symptoms appear, adding more simulated conversations only amplifies the same bias.

A more subtle problem occurs when training and evaluation use the same type of simulator. If both the training and test sets are generated by “perfect users,” internal metrics may look strong while failing to measure performance on ambiguous, out-of-distribution language.

A better approach is to separate user strategies from task sets. Keep the tasks fixed, sample user behavior from an independent distribution, and retain a small set of real conversations as an anchor for calibrating simulation scores.

Building More Realistic User Models

A user should not be modeled as an all-knowing collaborator. Instead, model the user as an information process with constraints.

The following dimensions are generally sufficient for a useful first simulation specification and do not depend on any particular vendor interface.

Information-Revelation Strategy

Provide only part of the goal at the beginning. Release important constraints by turn, or only after the agent asks the right question.

Slots that have not been requested should remain unknown. The simulator should not automatically fill them in.

Implicit Preferences

Define an acceptance checklist for each task, but keep it hidden from the agent. After seeing a candidate result, the simulator can respond with a short rejection or a partial revision request instead of revealing every criterion at once.

Interaction Budget

Set a maximum number of useful replies and an impatience threshold. After the threshold is reached, responses should become shorter. The user might say “You decide” or “Anything is fine,” or accept a less-than-perfect but deliverable result.

This penalizes pointless chains of follow-up questions and rewards strategies that identify the most important uncertainty.

Language Noise

Allow ambiguous references, typos, colloquial wording, and sentences that combine two requests. The noise should have structure: it should still point toward the same task rather than becoming random nonsense.

The kind of non-cooperation produced by a weak model should not automatically be treated as more realistic.

Inconsistent State

With a low probability, change a non-core constraint during the conversation or forget an option that was just confirmed. The agent should detect the conflict and perform a minimal rollback rather than assuming that all earlier context remains valid.

These rules can be described through configuration instead of giving another large model complete freedom to improvise the role. For example:

user_sim:
  reveal: delayed          # Release constraints with a delay
  hidden_criteria: true    # Do not volunteer acceptance criteria
  patience_turns: 2        # Responses become shorter afterward
  shorthand_after: 3
  contradiction_prob: 0.1
  style: terse             # Short, colloquial, minimally explanatory
  must_not:
    - volunteer_full_spec
    - solve_task_for_agent

The must_not section is especially important. Once the simulator starts planning for the agent or rewriting the request as a full specification, it is no longer acting as a user. It has become a second assistant.

How to Evaluate Without Misleading Yourself

Evaluation should use at least three layers rather than reporting a single overall success rate.

Layer 1: Fixed Tasks, Variable Users

Run the same task set with at least three user profiles: highly cooperative, moderately ambiguous, and low patience. Report the range between the highest and lowest success rates.

If the range is large, the score is reflecting user difficulty more than agent capability.

Layer 2: Real-User Anchors

Use real people for a small subset of tasks and compare their results with the simulated-user results.

A consistently positive gap suggests that the simulator is too easy. A consistently negative gap, especially when failures are caused by irrelevant or incoherent responses, suggests that the simulator is not sufficiently cooperative.

The goal is not to make the two scores identical. The goal is to make the gap stable, explainable, and trackable as the product evolves.

Layer 3: Process Metrics

In addition to final task completion, record:

• The number of effective clarification turns
• The rate at which key slots are identified correctly on the first attempt
• Recovery performance after the user responds with “Whatever”
• The number of rework steps after a mid-task constraint change

Outcome metrics alone can hide a process that succeeds only because the simulator supplies missing information.

When labeling failures, assign responsibility explicitly:

• The information was insufficient, but the agent failed to ask: strategy failure
• The user provided a constraint, but the agent ignored it: execution failure
• The simulator became inconsistent beyond its configured probability: environment failure

Environment failures should not count against model capability. Otherwise, a “hard mode” simulator turns its own noise into an apparent agent defect.

Improving the Training Loop

Start by generating user trajectories that are imperfect but still workable through rules and templates. Use a stronger model only to polish colloquial wording; do not let it decide when information is released.

The information schedule should be controlled by the task specification and a random seed. The language model should vary the phrasing, not redefine the interaction structure. This preserves linguistic diversity without allowing a strong model to turn every task back into easy mode.

The curriculum can be organized by information completeness, but the final training distribution must include delayed information and low patience. The agent should not converge only on simple, fully specified interactions.

Rewards should be tied as closely as possible to verifiable task state:

• Whether a file was generated
• Whether the fields satisfy hidden acceptance criteria
• Whether the user accepts the result within the interaction budget

Do not use conversational politeness or fluency as the primary reward. Otherwise, the policy will learn to match the simulator’s language preferences rather than complete useful work.

After deployment, sample ambiguous openings, mid-conversation changes, and early terminations from real conversations and feed them back into the system for distribution analysis and failure classification. Do not use unredacted conversations directly as training data.

The purpose of this feedback loop is to update the probabilities in the user configuration, not to chase the surface wording of any single conversation.

If cloud-based training and evaluation pipelines are used, object storage can hold task specifications and hidden acceptance checklists. Message queues can split evaluation jobs across different user configurations for parallel execution. Logging and monitoring can track whether the success-rate range and the gap from real-user anchors are getting worse.

These are general infrastructure capabilities. The central issue remains how the data is stratified, not which conversational model is assigned to play the user.

Conclusion

An agent becomes more robust only when its counterpart behaves like a real user, not like another perfect assistant.

Using an overly capable model as the user puts evaluation into easy mode. Using an incoherent, uncooperative weak model puts it into a distorted form of hard mode. Both conditions can cause teams to misjudge the agent’s capability boundaries.

A more reliable simulation environment makes information revelation, implicit acceptance criteria, and interaction budgets explicit. It calibrates scores against real-user anchors and ties rewards to verifiable outcomes.

Only then can user simulation evolve from a cooperative performance into a training environment whose lessons are more likely to transfer to real interactions.