Technical Guide to Creating AI Virtual Characters via LLM-Based Secondary Design

1. Introduction: Why LLM-Based Virtual Character Secondary Design Matters

Demand for AI virtual characters with distinctive visual styles is rising across short-video platforms, live streaming, and gaming. Rather than building characters from scratch, secondary design based on existing character settings delivers diverse visual concepts faster and at lower cost.

In practice, however, three core challenges frequently arise: maintaining consistency across multiple scenes, controlling the cost of multi-model orchestration, and crafting prompts that directly determine output quality. This article addresses each of these problems and provides an end-to-end, executable methodology—from setting extraction and prompt strategy through multi-model pipelines to text-to-image integration.

2. Structured Extraction and Validation of Original Character Settings

2.1 Breaking Down Setting Documents into Structured Data

High-quality secondary design starts with converting original character documentation into machine-readable structured data. A complete virtual character setting typically includes four categories of core elements:

• Appearance: hair color, eye color, outfit, signature accessories
• Personality tags: keywords such as “energetic” or “tsundere”
• Backstory: world-building and character identity
• Visual style: realistic, cel-shaded anime, watercolor, etc.

Organize these into a JSON or Markdown document, for example:

{
  "character_name": "Example Character",
  "appearance": {
    "hair": "pink twin tails",
    "eyes": "amber",
    "outfit": "academy-style uniform"
  },
  "personality": ["energetic", "curious"],
  "style": "Japanese cel-shaded anime"
}

2.2 Identifying and Correcting Setting Inconsistencies

Original settings often contain vague descriptions or conflicting elements—for instance, a hair color that clashes with eye color within the given context, or clothing that does not fit the world-building. Before feeding data into any model, perform item-by-item consistency checks. This step is fundamental but directly determines the quality floor of secondary design.

3. How LLMs Convert Character Settings into Visual Prompts

3.1 Core Role of the LLM: Text Transformation, Not Direct Image Generation

A large language model does not generate images itself. Its core value lies in converting structured character settings into natural-language prompts that text-to-image models can understand. Specifically, the LLM must handle three tasks:

• Style expansion: add details such as lighting and pose while preserving core features
• Diversity generation: produce multiple distinct descriptions from the same setting
• Conflict resolution: intelligently fill gaps or suggest edits when the original setting is contradictory

3.2 Key Prompt Engineering Strategies

Prompt quality directly determines output quality. Recommended strategies include:

• Role-play instruction: explicitly state “You are a senior anime character designer” to steer output toward a professional perspective
• Few-shot guidance: attach one or two example outputs to align the model’s style
• Explicit hard constraints: declare things like “keep hair as pink twin tails” and “lock eye color to amber” to prevent setting drift
• Layered description structure: start with overall composition, then local details, then lighting and atmosphere
• Negative prompts and weight syntax: use negative prompts to exclude low-quality elements, e.g., (low quality:1.4), (bad anatomy:1.3); assign weights to key elements, e.g., (pink twin tails:1.2), (amber eyes:1.2)

Example prompt template:

You are a professional anime character designer. Based on the following settings, create 3 scene descriptions (school / starry sky / rainy day). Requirements: maintain pink twin tails and amber eyes, Japanese cel-shaded anime style. Negative prompt: (worst quality:1.4), (deformed:1.3). Output each description in the order: composition → outfit → expression and pose → lighting.

4. Designing and Deploying a Multi-Model Collaboration Pipeline

4.1 Why Multi-Model Orchestration Is Needed

No single model can simultaneously deliver creative diversity, logical consistency, and fluent language. A common division of labor is:

• Creative divergence model: generates multiple candidate descriptions
• Logic validation model: checks candidates against the original setting for consistency
• Polishing model: refines passing candidates into final, production-ready prompts

4.2 Pipeline Workflow and Error Handling

A typical pipeline proceeds as follows:

1. Input stage: feed the structured setting into the creative divergence model to generate N candidate descriptions
2. Validation stage: send each candidate together with the original setting to the validation model; flag inconsistencies; if validation fails, trigger a retry (recommended maximum of 3 attempts); if the retry threshold is exceeded, escalate for manual review
3. Refinement stage: pass validated descriptions to the polishing model to produce final high-quality prompts
4. Human review: keep a manual checkpoint at key stages

Pseudocode example:

candidates = creative_model.generate(setting, n=5)
for c in candidates:
    result = check_model.validate(c, setting, max_retries=3)
    if result.passed:
        final_prompts.append(polish_model.refine(result.prompt))
    elif result.retries >= 3:
        alert_human_review(c, result.issues)

This workflow can be automated through cloud functions or a workflow orchestration engine.

4.3 General Considerations for Model Selection

When choosing models for collaboration, evaluate: instruction-following capability, creative richness, depth of context understanding, and output stability. Run small-scale comparative tests using setting-consistency scores and description completeness as core metrics, then select the model combination that best fits your use case—along with a cost estimate.

5. From Text to Vision: Text-to-Image Integration and Consistency Control

5.1 Connecting to Text-to-Image Models

The natural-language descriptions produced by the LLM must be converted into a format that text-to-image models accept, including keyword weights and negative prompts. Add a format-conversion step after LLM output to automatically transform the text into tag-style prompts compatible with mainstream cloud services or open-source inference frameworks.

5.2 Practical Techniques for Character Consistency Across Scenes

Maintaining character consistency across multiple scenes is a well-known industry challenge. Common techniques include:

• Reference image injection: supply a fixed character reference image during generation to stabilize facial features and key traits
• Seed locking: fix the random seed to reduce stochastic variation
• LoRA fine-tuning: train a dedicated style model on 10–20 character samples using frameworks such as kohya_ss in an environment with at least 8 GB of VRAM. This fundamentally improves consistency. Ensure all training data is properly authorized and compliant.

6. Practical Considerations and Compliance Notes

1. Copyright compliance: secondary design requires legal authorization over the original setting; training data for LoRA must also be verified for copyright
2. Iteration loop: establish a “generate → feedback → optimize” cycle to continuously refine prompt templates
3. Cost control: multi-model orchestration increases API call expenses; use caching and batch processing to optimize
4. Quality baseline: define explicit consistency scores and description-completeness standards as the basis for automated filtering
5. Data security: storage and use of character reference images and training samples must comply with applicable data security regulations
6. Human oversight: all AI-generated content should undergo human review to ensure compliance with public-distribution content standards

7. Conclusion

LLM-based secondary design of virtual characters is a comprehensive practice that combines prompt engineering, multi-model orchestration, and text-to-image technology. By extracting settings in a structured way, designing prompts with negative terms and weight syntax, building pipelines with exception fallback, and applying LoRA fine-tuning plus reference-image injection for consistency, developers can produce AI virtual character assets efficiently and reliably. This approach holds strong potential in short-video production, virtual streaming, game pre-production, and related fields—and is well worth continued exploration.