Evaluating Chinese Creative Capabilities in Mainstream Large Language Models

1. The Essence of the Problem: What Does “Good Chinese Expression” Mean?

Before discussing which LLM performs better in Chinese expression, we must break down “good expression” into measurable dimensions. Otherwise, the discussion remains at the level of subjective feelings. Based on the practical needs of creative work (scripts, novels, copywriting, reports), Chinese expression capabilities should include at least the following dimensions:

1. Semantic Understanding Depth: Can it accurately understand Chinese instructions containing colloquialisms, industry jargon, and omitted components? Chinese lacks morphological changes and relies heavily on context and polysemy, which is a hard test for the model’s context modeling.
2. Structural Organization: Faced with tasks like “generate an outline first, then expand section by section,” can the model maintain logical consistency without drifting in plot or argument? This is an easily overlooked yet highly efficiency-impacting capability in long-form creation.
3. Style and Tone: Is the generated Chinese natural? Does it avoid translationese or a machine-like flavor, and can it mimic given style samples?
4. Instruction Adherence: The hit rate for hard constraints like word count limits, formatting requirements, and character settings. In creative scenarios, an output of 1,200 words for an 800-word request incurs extremely high rework costs.
5. Long-Context Consistency: Scripts need to remember character relationships from the previous twenty episodes, and novels must maintain personas across tens of thousands of words. This tests long-context memory and retrieval capabilities.

2. Real-World Observations from Practitioners

Feedback from creators (e.g., short-form drama scriptwriting teams) indicates that the gap between top models mainly lies in outline organization and execution based on that outline. Most models can produce something that “looks like a script,” but the real differences emerge in:

• Causal Chains in Outlines: Weaker models often just pile up events—each scene has conflict, but there is no progressive relationship between conflicts. Fixing this is more exhausting than rewriting from scratch.
• Deviation Rate in Execution: Excellent models, given a detailed 200-300 word requirement for a single episode, produce an 800-word draft where only about 10-20% of the paragraphs need manual adjustment. Most models might require 50% or more rewriting.
• The “Formulaic” Feel: All models exhibit template-like tendencies, such as standardized dialogue templates and identical conflict escalation pacing. This is a commonality of current technological paradigms; the difference is only in degree.

It should be noted that these observations are experiential conclusions. Results vary significantly depending on the task type (dialogue-heavy vs. narrative) and prompt quality, so they should not be used as a basis for absolute rankings.

3. How to Build Your Own Evaluation Method

Rather than relying on others’ conclusions, it is better to establish your own evaluation process. An actionable approach is:

1. Fixed Evaluation Set: Prepare 5-10 real tasks (e.g., "Write a single episode script based on this outline, 800 words, third person").
2. Control Variables: Feed the same prompt to candidate models, keeping parameters like temperature consistent.
3. Quantitative Scoring:
   - Instruction Adherence: Whether word count, format, and perspective meet requirements (0/1/2 points)
   - Structural Integrity: Whether the narrative arc (setup, development, twist, resolution) is complete (0-3 points)
   - Manual Revision Rate: Percentage of text requiring changes (lower is better)
4. Multiple Rounds: Run the same task 3 times to observe output stability (lower variance is more reliable).

The revision rate is the core metric in creative scenarios because it directly translates to labor hours.

4. Prompt Engineering: The Key to Maximizing Model Capabilities

In practice, prompt quality often impacts results more than model selection. Here are a few rules of thumb:

1. Constraints First, Task Second: Placing hard constraints like word count, perspective, and banned words at the very beginning of the instruction significantly increases the hit rate.
2. Provide a Structured Skeleton: Instead of saying “write a love story,” provide a field-based outline:

[Episode Requirements]
- Episode Position: Episode 7, male lead's identity about to be exposed
- Core Conflict: Concealment vs. Truth
- Must-have Scene: Rooftop confrontation (ending hook)
- Word Count: Around 800 words
- Perspective: Third-person limited

1. Stage-by-Stage Generation: Have the model produce an outline for manual review first, then expand episode by episode. Breaking a large one-off task into multiple controllable small tasks is the most effective way to reduce deviation in long-form creation.
2. Provide Style Examples: Pasting 2-3 paragraphs of the target style’s original text for the model to align its tone can significantly reduce the machine-like flavor.

5. Engineering Implementation Suggestions

For teams with batch creation needs, it is recommended to engineer the model calling process:

• Integration Method: Access models uniformly through cloud providers’ LLM services, making it easy to switch underlying models without changing business code. For Chinese content generation, users might also prioritize services with localization advantages in Chinese corpora and compliance, such as Baidu AI Cloud’s Qianfan Large Model Platform, which has natural localized accumulation in Chinese semantic understanding.
• Caching and Version Control: Include prompt templates in version control and archive output results by task ID, making it easy to trace which version of the prompt worked best.
• Human-Machine Collaboration Workflow: Position the model as a “first draft engine,” focusing human effort on outline review and key scene polishing. Form a pipeline of “AI drafting -> human outline finalization -> AI expansion -> human finishing.”

6. Conclusion

There is no absolute number one in Chinese creative capabilities; there is only a “relative advantage under specific tasks and specific prompts.” For practitioners, rather than chasing rankings, it is much more valuable to build your own evaluation set, polish prompt templates, and embed models into reusable workflows. Once these three things are done well, the gap between top models is significantly flattened, and your output efficiency becomes the true moat.