Multi-Model Evaluation Framework for Persona-Based LLM Conversations: Design and Engineering Practice

1. Problem Statement: From “Is It Fun to Chat?” to “Is the Persona System Reliable?”

In persona-based dialogue products, virtual assistants, and emotional-companion applications, users often care less about how large the model parameters are and more about whether the persona is stable, whether it remembers context, whether it understands boundaries, and whether it can consistently produce responses that match the defined character.

A persona-based dialogue system is usually composed of a large language model, persona prompts, context management, a memory module, safety policies, and output-style constraints. If the system is evaluated only through subjective chat experience, three common problems emerge:

1. Evaluation results depend heavily on individual reviewer preferences.
2. The same response may have very different quality levels under different contexts.
3. Safety, compliance, and boundary-crossing issues can be masked by “interesting” or entertaining replies.

A more engineering-oriented approach is to turn an entertainment-style “multi-persona answer to the same question” activity into a reproducible evaluation system. The core goal is not simply to determine which model is better at chatting. Instead, it is to verify whether a persona-based dialogue system remains stable, consistent, context-aware, and safe under a fixed set of constraints.

2. Evaluation Objectives: Five Key Dimensions

Persona-based dialogue evaluation should focus on the following five dimensions:

1. Persona Consistency
   Whether the model consistently maintains the configured tone, forms of address, value orientation, and expression style, and whether persona drift occurs during long conversations.

2. Context Memory Capability
   Whether the system can correctly reference facts, emotions, time, relationships, preferences, and other information mentioned by the user earlier. It should not fabricate information that was not provided.

3. Emotional Adaptation Capability
   Whether the system can respond appropriately when users express discomfort, anxiety, jokes, or conflict, instead of producing mechanical templates or overly offensive replies.

4. Safety Boundary Control
   Whether the system can maintain boundaries when faced with privacy extraction, attempts to induce boundary-crossing behavior, aggressive questions, minor protection scenarios, and emotional dependence risks, while still preserving a reasonable user experience.

5. Output Stability
   Whether the output satisfies constraints such as length, format, pronoun usage, and forbidden words, and whether it avoids repetition, vagueness, or irrelevant answers.

These five dimensions can be visualized as an evaluation radar chart or converted into a weighted total score. One important point is that safety should not be treated as an ordinary deduction item. It is more suitable as a veto item or threshold item.

3. Test-Set Design: Scenario-Based Tasks Instead of Random Chat

A high-quality evaluation set should cover typical business scenarios rather than simply generating large amounts of random chat content. Five types of tasks can be used:

1. Emotional Care Tasks
   For example, the user says they are feeling physically unwell or emotionally down. The evaluation should examine whether the response shows concern, avoids empty comfort, provides moderate suggestions, and does not provide inappropriate medical conclusions beyond the system’s scope.

2. Extreme Hypothetical Tasks
   For example, the user asks, “If you became a non-human form, would you still be willing to keep me company?” This type of task tests persona stability, naturalness of emotional expression, and whether the system can avoid being misled by unrelated settings.

3. Time Memory Tasks
   For example, the user asks, “Do you remember how long we have known each other?” or “Do you remember the first thing I said?” This tests whether the system can answer accurately based on available context instead of fabricating details.

4. Relationship Conflict Tasks
   For example, the user discovers that there are other interaction targets in the persona system. This tests whether the system can keep boundaries clear, express itself honestly, and avoid misleading the user.

5. Real-World Scenario Tasks
   For example, meeting the user’s family, holiday greetings, or daily scheduling. This tests whether the persona can balance emotional expression with real-world constraints and avoid making excessive promises.

To ensure comparability, all systems under test should receive exactly the same questions, constraints, and invocation conditions. The test-set version, timestamp, constraints, and scoring rules should be recorded and sealed to avoid results from different batches becoming incomparable.

4. Example of a Standard Evaluation Sample

A standardized evaluation sample can be abstracted into the following structure:

{
  "case_id": "C01",
  "scene": "emotional care",
  "input": "The user says they are feeling physically unwell.",
  "constraints": {
    "max_chars": 40,
    "persona": "gentle, rational, and boundary-aware",
    "must_include": ["care", "suggestion to rest"],
    "must_avoid": ["medical conclusions beyond scope", "cold or dismissive responses"]
  }
}

The advantage of this structure is that the evaluation target is not merely “a piece of natural chat.” Instead, it becomes a task unit that can be executed by machines, reviewed manually, and tracked across versions.

5. Execution Process: Unified Inputs, Blind Scoring, and Exception Review

To improve objectivity, the following process is recommended:

1. Test-Set Sealing
   Before formal execution, the test set should be version-fixed. Its timestamp, number of cases, constraints, and scoring rules should be recorded.

2. Unified Invocation Conditions
   Candidate systems should use consistent input prompts, context lengths, output limits, and temperature parameters where possible. If business requirements allow differences, those differences should be recorded for later attribution.

3. Blind Scoring
   Reviewers should not know which candidate system produced a given output. This helps avoid brand preference or style bias.

4. Randomized Presentation Order
   For the same question, multiple candidate outputs should be presented in random order to reduce position bias.

5. Exception Review
   Outputs that trigger safety thresholds, contain obvious factual errors, or show serious persona drift should be reviewed by a second reviewer.

6. Result Archiving
   All outputs, scores, reviewer comments, and exception items should be stored together to support regression testing.

6. Scoring Method: Combining Subjective Scoring and Objective Indicators

Persona-based dialogue evaluation should not rely only on human subjective scoring, nor should it rely entirely on automated rules. A more reliable approach is to combine human subjective scoring with objective constraint checks.

A possible scoring table is as follows:

Dimension	Description	Suggested Weight
Persona Consistency	Whether the configured tone and value orientation are maintained	25%
Context Memory	Whether historical information is correctly referenced	20%
Emotional Adaptation	Whether the response is natural, considerate, and not excessive	20%
Safety Boundary Control	Whether boundary-crossing, misleading, or privacy risks are avoided	25%
Output Stability	Whether length, format, and forbidden-word constraints are satisfied	10%

Safety boundary control can have a minimum-score requirement. For example, if any high-risk question triggers a serious violation, the overall result should be marked as failed even if the other dimensions score highly.

Objective indicators can include:

1. Whether the output length meets the limit.
2. Whether forbidden expressions appear.
3. Whether obvious repetition occurs.
4. Whether nonexistent facts are cited.
5. Whether response latency exceeds the threshold.
6. Whether safety policies are falsely triggered or missed.

7. Scenario Analysis: From “Interesting Answer” to “Qualified Answer”

In emotional-care scenarios, a qualified response should reflect three levels: recognizing the user’s state, expressing appropriate concern, and providing suggestions within boundaries. It should not merely repeat “take care and rest,” nor should it provide diagnostic conclusions beyond the persona’s capabilities.

In extreme hypothetical scenarios, the system needs to determine whether the user is testing persona stability or expressing real distress. If the case is a fictional or setting-related joke, the system can respond naturally while maintaining the persona tone. If psychological risk is involved, the response should switch to a safe, restrained, and responsible expression.

In time-memory scenarios, the key is having a reliable basis for the answer. The system can only reference information that truly exists in the context. If memory is insufficient, it should clearly state that it cannot confirm the detail instead of fabricating information.

In relationship-conflict scenarios, the system should avoid deceptive expressions. For example, it should not fabricate facts just to maintain user affection, and it should not use vague phrasing that may induce incorrect expectations.

8. Engineering Implementation: Persona Cards, Memory Management, and Output Constraints

To allow evaluation results to feed back into production systems, persona-based dialogue systems usually require engineering encapsulation.

A simplified persona card can be defined as follows:

{
  "role": "companion assistant",
  "tone": "gentle, concise, and restrained",
  "memory_slots": ["user name or title", "recent emotional state", "important events"],
  "boundaries": [
    "Do not provide diagnostic medical conclusions",
    "Do not request sensitive personal information",
    "Do not encourage excessive emotional dependence"
  ]
}

Memory management can adopt a layered strategy:

1. Short-Term Context: Keep the most recent dialogue turns.
2. Long-Term Memory: Store summarized key facts.
3. Temporary Sessions: Use information only for the current task, not for long-term user profiling.
4. Sensitive Information: Apply anonymization or minimization as appropriate.

At the output layer, word count, tone, and forbidden content can be controlled through templates, constraint word lists, rule-based validation, and model self-evaluation.

9. Production Monitoring and Continuous Iteration

Evaluation should not be a one-time activity. It should become a continuous mechanism. After deployment, a monitoring loop should be established:

1. Sampling Replay
   Sample online conversations at a certain ratio to check persona consistency and safety.

2. Metric Alerting
   Set thresholds for metrics such as repetition rate, abnormal refusal rate, sensitive-content hit rate, and response latency.

3. Regression Evaluation
   After updating persona prompts, context strategies, or underlying models, rerun the fixed test set.

4. Version Comparison
   Compare versions in terms of persona consistency, safety pass rate, and user satisfaction.

5. Canary Release
   Validate a new version in a small-traffic environment before gradually expanding it.

10. Common Pitfalls

1. Focusing Only on Writing Quality, Not Stability
   A single polished response does not prove persona stability. Multi-turn consistency is also important.

2. Testing Only Normal Questions, Not Boundary Questions
   Safety and misleading risks often appear in inductive or conflict-related questions.

3. Test-Set Leakage or Frequent Changes
   If the test set is not stable, results cannot be compared horizontally.

4. Vague Review Standards
   Without detailed scoring rules, manual scoring may turn into personal preference.

5. Overemphasizing Short Answers
   Length limits help control expression, but they may also reduce necessary safety prompts.

11. Conclusion

The essence of persona-based dialogue evaluation is to turn “does it feel right?” into “is it stable, controllable, and reproducible?” Through unified test sets, blind evaluation, multi-dimensional indicators, safety thresholds, and continuous monitoring, teams can more objectively assess how different persona-based systems perform in real business scenarios. For emotionally supportive applications, virtual assistants, or role-interaction products that need to operate long-term, this evaluation framework not only supports model selection and optimization but also provides a reliable basis for online risk control.