Testing LLM Chinese Generation: Why an Acrostic Poem Challenge Reveals Significant Gaps
This article explores the effectiveness of using Chinese acrostic poems to evaluate large language models (LLMs). By testing models on a highly constrained poetry generation task, we analyze why general benchmark scores do not always translate to fine-grained language capabilities. We break down the evaluation criteria, common failure modes, and provide a structured, reusable methodology for developers to assess LLM instruction following, imagery control, and format discipline.
I. Why Use an Acrostic Poem to Test LLMs?
Among Chinese generation tasks, the acrostic poem is an underrated “high signal-to-noise” test. It may seem like a simple word game, but a 28-character poem (such as a seven-character quatrain) simultaneously compresses three core capability checkpoints:
- Instruction Following: The model must strictly start each line with specific characters (e.g., “静”, “水”, “流”, “深”). This is the hardest constraint; it is either right or wrong, leaving almost no room for subjective debate.
- Imagery Control: The prompt specifies the recipient’s identity—for instance, a retired physics teacher. The poem must align with the idiom’s underlying meaning (reserved, profound, wise but unassuming) and the professional aura of a teacher, testing the model’s grasp of semantic fields and emotional tone.
- Format Discipline: Character count, line breaks, and rhyming must be fundamentally standardized without messy line lengths or obvious padding. This tests the model’s internalization of Chinese poetic norms.
Because of these layered requirements, this challenge can easily separate models that merely “can write a poem” from those that do it well, revealing visible gaps in performance.
II. The Most Striking Phenomenon: Model Origin Does Not Dictate Capability
In this informal comparison, a counterintuitive result emerged: the performance distribution did not align with the “domestic vs. overseas” model camps. The best-performing model received near-perfect evaluations, while two well-known overseas models landed at the bottom with average scores around 0.4 and 0.2.
This result indicates at least two things:
First, there is no inherent link between a model’s origin and its fine-grained Chinese generation capability. The Chinese systems of phonology, imagery, and literary allusions are highly specialized. Without deep alignment and refinement on Chinese corpora, even a generally strong model can fail at character-level constraints and thematic resonance.
Second, the essence of the score gap is not simply “knowing how to write poetry,” but the product of three capabilities. A model scoring 0.9 in instruction following, 0.5 in imagery fit, and 0.8 in format discipline might offer a worse overall experience than one scoring 0.7 across the board. The bucket effect is particularly pronounced in creative tasks.
III. Breakdown of Common Failure Modes
Summarizing the typical errors from this testing round, we can categorize them into five types:
- Character Displacement: Forgetting the required starting character in the fourth line, or placing the correct character in the wrong position. These errors expose the model’s instability in handling “character-by-character anchoring” constraints.
- Semantic Drift: The first two lines address the prompt, but the last two suddenly slide into generic blessings, completely disconnecting from the imagery of a “physics teacher” or the theme of “still waters run deep.” This indicates a lack of global semantic planning.
- Padding and Repetition: Forcing function words to make a rhyme, or repeating the same imagery across lines (e.g., mentioning the “moon” twice). This reflects a local greedy strategy during generation.
- Mismatched Atmosphere: The text is technically correct, but the overall tone reads like doggerel rather than a dedicated gift poem, lacking the implicit subtlety required by the theme. This is the most implicit metric, heavily reliant on “language sense.”
- Format Divergence: Unauthorized outputs like explanations, added quotation marks, or altered sentence lengths. This shows a lack of boundary awareness regarding the required output format.
IV. How to Build a Reusable Evaluation Method
If developers want to engineer this type of testing, they can refer to the following workflow:
- Design prompts with incremental constraints: Pure character constraints → Character + identity constraints → Character + identity + emotional tone. Observe where capability drops off step-by-step.
- Replace overall impression scores with structured scoring. For example:
{"instruction_following": {"weight": 0.4, "checks": ["All four starting characters correct", "No extraneous explanatory text"]},"imagery_fit": {"weight": 0.35, "checks": ["Resonates with recipient identity", "Matches idiom meaning"]},"format_discipline": {"weight": 0.25, "checks": ["Seven characters per line, four lines", "Rhyme is generally regular"]}}
- Sample multiple times for the same prompt and track the pass rate rather than relying on a single best result. This prevents mistaking random variance for actual capability.
- Combine blind human evaluation with automated checks: Character anchors and formats can be verified via scripts, while imagery and literary grace should be handed over to blind reviewers for scoring.
V. Conclusion
The value of a “small challenge” like an acrostic poem lies in its ability to expose the weaknesses most easily masked by marketing rhetoric: fine-grained constraint following and Chinese language sense. For developers, it is worth building a high-constraint test set during model selection. No matter how impressive a model’s general benchmark scores are, the actual business experience is often determined by these subtle, hidden capabilities. Still waters run deep, and the true depth of a model can only be fathomed through these seemingly quiet challenges.
