Evaluating Text Generation with Multiple Constraints: A Practical Guide to Acrostic Poetry Tests
This article presents a practical framework for evaluating language models on tasks with multiple simultaneous constraints, using acrostic poetry (藏头诗) as a concrete test case. It explains why acrostic poems are well-suited for exposing model weaknesses in instruction following, semantic control, and format discipline. The guide covers designing evaluation dimensions (rule-based, semantic, and subjective), separating hard constraints from soft quality scoring, using judge models with attention to common biases, ensuring reproducibility, analyzing failure modes, and building a repeatable evaluation pipeline. The goal is to help practitioners create fair, transparent, and actionable model evaluations for constrained generation tasks.
Why Acrostic Poems Are a Suitable Evaluation Task
An acrostic poem may appear to only require each line’s first character to spell out a word, but it simultaneously imposes several layers of constraints. The first layer is structural: exact line count, fixed character count per line, punctuation, and optional stanza breaks. The second is semantic: the phrase formed by the first characters must resonate with the poem’s imagery as a whole—each line cannot drift into unrelated content. The third is audience-related: the intended recipient, occasion, and tone all influence word choice. The fourth concerns factual accuracy: if the prompt refers to a person’s identity or background, the poem must not invent unsupported details.
Because these constraints stack together and the text is short, any slip becomes immediately visible. Thus, an acrostic poem can simultaneously test three things: how well the model follows instructions, how well it controls imagery, and how disciplined it is with output format. Using such a task as an evaluation sample exposes problems more clearly than free-form writing.
For example, ask the model to write a four-line acrostic poem for a retired teacher, hiding the four characters 山高水长 (mountains high, water long). The model must produce: exactly four lines, each line starting with 山, 高, 水, 长 respectively; content that shows respect for a teaching career; no fabricated details about the person’s life; and language that is fluent and somewhat evocative. Any missed requirement will show up in the scoring.
Designing Evaluation Dimensions
Dimension design is the foundation of any evaluation. Too few dimensions fail to distinguish model differences; too many make it hard for human raters to agree. In practice, dimensions can be grouped into three classes.
The first class comprises hard metrics that can be checked by rules: line count, first characters, character count range, and prohibited characters. These have binary outcomes and require no subjective judgment.
The second class includes metrics that require semantic understanding but have relatively clear boundaries: content compliance, whether the output includes irrelevant information, and whether factual descriptions match the given material. These are well-suited for a judge model, provided clear criteria are attached.
The third class covers subjective quality metrics: naturalness of language, depth of imagery, and emotional resonance. These are the hardest to quantify and should be described with a graded scale, with periodic human sampling.
A reference dimension table follows.
| Dimension | What It Checks | How It Is Judged |
|---|---|---|
| Audience fit | Whether address, tone, and wording match the recipient’s identity and occasion | Judge model plus human sampling |
| Content compliance | Whether the output contains inappropriate or sensitive expressions | Rule filtering plus judge model |
| Factual accuracy | Whether identity, experience, and time references align with the provided material | Judge model compares against material |
| Format constraints | Line count, character count, first characters, punctuation | Rule-based validation |
| Language quality | Grammar, word choice, rhythm | Judge model with a graded scale |
| Expressive impact | Whether imagery is concrete and emotions feel natural | Human primary, judge model auxiliary |
| Material selection | Whether imagery stays on topic and avoids empty clichés | Judge model plus human sampling |
Each dimension should have a clear description of what each score level means. For example, language quality can be divided into five levels: frequent errors that hinder comprehension; mostly grammatically correct but stiff wording; fluent but lacking rhetorical finish; fluent with some rhetorical devices; concise and rhythmically strong. The clearer the differences between levels, the closer different raters’ scores will converge.
Separating Hard Constraints from Soft Quality
The most error-prone step in an evaluation pipeline is handing hard constraints over to subjective judgment. A poem with a wrong first character but otherwise well-written should not receive a high overall score. The correct approach is to first run rule-based checks; any sample that fails is directly marked as invalid, and only the remaining samples proceed to semantic scoring.
Below is a sample validation function for line count and first characters:
import redef check_acrostic(poem, head, line_count):# Strip blank lines and leading punctuation/whitespacelines = []for raw in poem.split("\n"):text = re.sub(r"^[\s\W_]+", "", raw)if text:lines.append(text)if len(lines) != line_count:return False, f"Expected {line_count} lines, got {len(lines)}"initials = "".join(line[0] for line in lines[:len(head)])if initials != head:return False, f"Expected first characters '{head}', got '{initials}'"return True, "Passed"
The key principle is “check first, then score.” If a hard constraint fails, the sample’s total score should be zero (or the sample should be excluded), and the failure reason should be recorded for later statistical analysis of the most common error types.
Soft quality aggregation can then use a weighted average. Assuming each dimension score is normalized to [0, 1], the total score can be computed as:
def total_score(dims, weights, hard_pass):if not hard_pass:return 0.0weighted = sum(weights[k] * dims[k] for k in weights)return weighted / sum(weights.values())
Weights should be determined by the task’s goal. For customer-facing copywriting, audience fit and content compliance may carry higher weights; for creative writing assistance, expressive impact may be heavier. Once weights are set, they should be disclosed in the report and not adjusted afterward to fit a desired conclusion.
Notes on Using Judge Models for Scoring
Using another language model as a judge is now common, but it introduces several systematic biases that need to be actively managed.
The first is self-preference bias: a judge model tends to favor outputs that resemble its own style. The remedy is to choose a judge that is unrelated to the model being evaluated, and to state the judge’s identity and version in the report.
The second is position bias: in pairwise comparisons, the judge may prefer the first answer. You can shuffle the order and average the results over multiple runs.
The third is length bias: judges tend to rate longer, more complex texts as better. For a constrained task like acrostic poems, length bias is less pronounced, but for other tasks it must be explicitly controlled.
The fourth is scoring scale drift: a judge may not use the same scale across different batches. It is advisable to include several fixed benchmark samples of known quality in each batch to monitor stability.
To check agreement between the judge and humans, sample a fraction of outputs and have two or more humans score them independently, then compute a consistency metric such as Cohen’s kappa or the intraclass correlation coefficient. If consistency is low, the scoring rubric is not clear enough and should be revised rather than trusting the judge’s results.
Ensuring Reproducibility
Evaluation results are only meaningful if the conditions can be reproduced. Model providers may update weights or change service policies without notifying users, so models with the same name can produce different results on different dates.
In practice, at least the following information should be recorded:
- Collection date and time (to the day);
- The model’s server-side version identifier, if the API provides such a field;
- Sampling parameters: temperature, maximum output length, random seed (if supported);
- The complete system prompt and user prompt;
- The judge model’s version and scoring prompt;
- The version number of the sample set and the number of samples.
When reporting conclusions, state that the results are valid only for the collected version. To compare results over time, keep a fixed set of benchmark samples and rerun them each time to distinguish true capability changes from environmental changes.
Sample size also affects reliability. Testing each model on a single prompt yields scores that may mainly reflect the prompt’s incidental features. A more robust approach is to prepare multiple similar tasks and generate several responses per task, then compute means and dispersion. When the difference between two models is smaller than random variation, state that the difference is not significant rather than ranking them.
Common Failure Modes and Analysis
The value of an evaluation goes beyond ranking; it lies in identifying why a model fails. In acrostic poetry tasks, common failure modes include the following.
The first is correct form but empty content. The first characters are right and the line count is right, but each line is a generic compliment with no concrete imagery. This suggests the model handles format well but lacks material selection ability. To analyze it, count the frequency of vague words or have a judge explicitly determine whether each line contains concrete imagery.
The second is fluent language but disconnected from the prompt. The verse reads beautifully but has nothing to do with the recipient’s identity—it might even sound like it was written for a young person. This is a failure of audience fit, often because the model understands the instruction only at a superficial level.
The third is sacrificing semantics to force the first character. To start a line with the required character, the sentence structure becomes awkward or grammatically flawed. This exposes the tension between constraint and expression, and can be captured by the language quality dimension.
The fourth is fabricating facts. For example, if the prompt only states that the person is a retired teacher, the model might invent specific years of service or awards. These errors are risky when the output is used externally, so they should be checked carefully under the factual accuracy dimension, and offending samples should be listed separately.
When analyzing, do not look only at total scores. Two models may have similar totals while one excels in format but fails on facts, and the other does the opposite. Plotting each dimension’s score as a radar chart or grouped bar chart reveals each model’s strengths and weaknesses more clearly.
Practical Implementation Recommendations
Organizing the above methods into a repeatable pipeline usually involves these steps:
- Prepare a sample set; each item includes the task description, constraints, and reference material.
- For each model under evaluation, generate responses with fixed parameters and save the raw output.
- Run rule-based validation on the output and record failure reasons.
- For outputs that pass validation, call the judge model to score according to the rubric and save the judge’s reasoning.
- Sample a percentage for human review and compute consistency.
- Aggregate per-dimension results and generate a report, noting versions, dates, and limitations.
In this pipeline, the input and output of each step should be saved in a structured format. At minimum, one record should contain: sample ID, model identifier, generated text, validation result, per-dimension scores, judge rationale, and collection time. Structured records facilitate later tracing and quick reruns after model updates.
Finally, it is worth emphasizing that evaluation is meant to help decide whether a model is suitable for a particular task, not to produce a universal ranking. When choosing a tool or model, evaluate it with representative samples from your actual use case, and rerun periodically to track capability trends. If you clearly break down the constraints, write a transparent scoring rubric, and specify the scope of validity, the evaluation conclusions will stand up to scrutiny.
