LLM Evaluation and Model Selection for Long-Constraint Copy Generation

    In real content operations, models rarely receive an open prompt—they get structured briefs with role definitions, source material, hard formatting rules, tone boundaries, and expression red lines that must not be crossed. This article argues that constraint-following, not prose quality, is the real dividing line, and presents a repeatable evaluation method built on six checkable constraints, a two-stage pipeline that combines deterministic scripts with a judge model, and a seven-dimension scoring rubric. It examines failure modes such as leaked reasoning chains, instruction decay, and factual hallucination, then turns the findings into a selection workflow that moves from total-score rankings toward constraint-following ability, output compliance, and failure cost.

    1. The Core Problem: Constraint-Following Is the Real Dividing Line

    In public discussions of LLM evaluation, “how good is the writing” tends to come up first. But in real content operations, a model rarely receives a one-liner like “write a restaurant review.” Instead, it gets a structured writing brief: a role definition, the available source material, hard formatting requirements, tone boundaries, and expression red lines that must not be crossed. Whether the model honors each of these items, one by one, decides whether the output is “ready to publish” or “still needs a human rewrite.”

    The difficulty lives precisely in the constraints. When hard requirements pile up to five or six and start to constrain one another, instruction decay shows up all at once. A model may hold the title length through the first half of the text, then forget to state a clear conclusion at the end; it may faithfully use the source material early on, then slip in an evaluative judgment when it reaches the competitor comparison. Open-ended writing rewards flair; long-constraint writing rewards the ability to keep its promises. So an effective evaluation method cannot look only at whether the language flows—it has to break constraint-following into items that are detectable, attributable, and regression-testable.

    2. Task Design: Translating “Can It Write” into Six Judgeable Constraints

    2.1 Source Material and Role

    To stay close to real content production, this article uses a restaurant-review note as its example. The material below has been anonymized and de-identified and is used only to demonstrate the evaluation method.

    The task asks the model to play a food blogger who is “picky but fair,” and forbids a promotional, ad-copy tone. The material describes a long-running neighborhood shop:

    • Its signature dish is wood-fired wontons, 18 yuan for 12 pieces;
    • The wait starts at 40 minutes;
    • The owner has a bad temper but remembers regulars’ tastes;
    • A business license from 1998 hangs on the wall;
    • The trendy internet-famous shop next door uses pre-made meal packs and sells half a portion for 29 yuan;
    • It accepts only cash and WeChat Pay.

    The material deliberately plants three “hooks”: the wait and the bad temper are drawbacks that must be reported honestly; 1998 is the only usable year; and the neighboring shop’s price is a fact, not an emotion. This design tests whether the model makes the right tradeoff between “writing it well” and “staying true to the facts.”

    2.2 Six Hard Constraints and Their Failure Modes

    Constraint Requirement Suggested check Typical failure
    C1 Title ≤ 20 characters and includes a specific number Script Missing the number, or exceeding the length
    C2 The owner’s bad temper and the wait time must be included; write the drawbacks as they are Keyword pre-screen + judge Dropping the drawbacks, or softening them into “the owner is very friendly”
    C3 Only years given in the material may be used; no self-derived calculations Regex-extract four-digit years and check against a whitelist Deriving 1998 + 28 = 2026, or fabricating “ironclad proof”
    C4 Competitor comparisons state facts only, no disparagement Script scan for disparaging words + judge for tone Writing “the meal-pack tax” or “fleecing customers”
    C5 The ending must give a clear “worth it or not” judgment Judge Leaving only “judge for yourselves”
    C6 Body of 400–500 characters, no more than 6 emoji Script Over the length, or too many emoji

    There is tension between C2 and C4: writing the drawbacks honestly can easily slide into emotional evaluation, which tests whether the model can “state facts without judging.” C5 demands a clear stance while C4 demands restraint, so the two hold each other in check. An ideal output is roughly: the wait starts at forty minutes and the owner has quite a temper, but the wontons remember regulars’ tastes, and the price is right there—whether it is worth it depends on whether you are willing to trade your time for this one bowl.

    2.3 Fix the Evaluation Protocol First

    Before running a batch, the protocol must be fixed, otherwise results cannot be compared.

    • Collection and versioning: Record the collection date, the model identifier returned by the API, the full prompt, and sampling parameters such as temperature and max tokens, all saved to disk alongside the results. Sampling parameters should be fixed at the lowest randomness. Even when deep-reasoning capabilities are invoked, the randomness of the reasoning process itself can still introduce variance.
    • Judgment protocol: In C6, the 400–500 character count applies to the body only and excludes the title; the title is judged separately by C1. Emoji are counted in the body by default; if an emoji appears in a title, it can be observed separately under a platform-adaptation dimension. Whether the title counts toward the character total and how emoji are counted must be written into the scoring rubric in advance, and every model under evaluation must face the same interpretation.
    • Request method: Route all requests through the same chat-completions-style endpoint of one compatible gateway; switching candidate services then requires only changing the base_url, API key, and model name. The client can use a mature SDK or a lightweight community library, avoiding custom request logic that would introduce extra variables.
    • Data granularity: Save one raw JSON per call, containing at least the raw response text, token usage, latency, and request ID, to support offline review and after-the-fact tracing.
    • Engineering reliability: Batch concurrency needs rate limiting, timeouts, and retry policies in place, and the actual concurrency of each round should be recorded, so that the stress test itself does not become a source of error.

    3. A Two-Stage Pipeline: Deterministic Checks Plus a Judge Model

    This article uses a two-stage pipeline of “deterministic checks + judge model.” For engineering, the implementation can be split into four actions—unified calls and logging, script validation, model judging, and aggregation with attribution—but the evaluation logic remains two-stage: first use rules to handle hard constraints that can be judged automatically, then use the judge model to handle semantic and tone judgments.

    1. unified call (fixed params) → per-item raw JSON saved to disk
    2. → Stage 1: script deterministic validation (length/emoji/year/disparaging words/output structure)
    3. → Stage 2: judge model scores seven dimensions per a fixed rubric (returned as JSON)
    4. → aggregate, cross-compare, and attribute failure samples
    1. Script validation: Constraints with clear, automatically executable rules—C1, C3, the keyword layer of C4, and C6—go to the script. Scripts are stateless and reproducible and are not affected by judge preferences, leaving the judgment space to the model.
    2. Judge scoring: Semantic constraints—C2, the tone of C4, C5, and the seven-dimension scores—go to the judge model. The judge prompt must include the role, the material, the six constraints, the seven-dimension definitions, and a unified judgment scale, and must return JSON for easy machine aggregation.
    3. Consistency check: When the script’s answer conflicts with the judge’s opinion, the script wins. Scripts are explainable and regression-testable; judge scores can only serve as supporting evidence.
    4. Sampling and stability: This round’s main leaderboard can use single-round collection, but a score is a point estimate rather than a distribution. When rank differences are small, recheck with multiple samples; a more robust approach is to compute the mean and variance over no fewer than 3–5 runs.

    3.1 Key Detail: Should the Reasoning Chain Count as Body Text

    Some models first output a block of thinking wrapped in reasoning tags, then the body. This creates a dilemma for the pipeline:

    • Strip it, then send to the judge: Scoring is cleaner, but stripping is itself a text transformation, and the stripping boundary is hard to define.
    • Send as-is to the judge: The scoring protocol matches the raw output, but the judge may count the thinking chain toward the character total, or judge the hesitation and self-correction inside the chain negatively.

    We ultimately chose to send the raw text and let the judge distinguish the body itself, stating explicitly in the prompt: “The reasoning block does not count toward the character total but is treated as part of the output structure.” The cost is that some outputs with reasoning blocks score noticeably lower on the platform-adaptation dimension.

    In an internal stability retest with a collection date of 2026-05-12, the candidates included anonymized model A (version: model-a-2026-04, samples: 6) and anonymized model B (version: model-b-2026-05, samples: 6). The normalization method was: each dimension first receives a 0–1 raw score from the judge model; items that fail script validation are counted as 0; outputs containing an un-stripped reasoning block are first treated as 0 on the platform-adaptation dimension; the final weighted total is the sum of each dimension’s score multiplied by its weight, with missing dimensions treated as 0. The result showed that model A’s outputs with thinking chains scored clearly lower on the platform-adaptation dimension, indicating that output format is an admission requirement, not a deduction that can be offset by writing quality.

    A more robust engineering approach is to switch to an interface that returns reasoning_content and content separately. Once clean body text cannot be obtained, it should be judged non-compliant outright and excluded from the writing-style comparison, and regex stripping plus output validation should be added to the publishing pipeline as a fallback.

    3.2 Illustrative Validation Code

    Run notes: The following code is for evaluation illustration only. It depends on the Python standard library re and statistics, and does not depend on third-party packages; json can be used as needed for actual logging. The current emoji counting and reasoning-tag stripping are simplified implementations; production environments should replace them with a mature emoji-parsing library and a more robust reasoning-block parser, and should always retain the raw output for auditing.

    1. import json
    2. import re
    3. import statistics
    4. # 1) Rule validation: C1 / C3 / C4 (disparaging words) / C6
    5. def strip_think(text: str) -> str:
    6. """Remove reasoning blocks and keep the body. For manual review only, not for automatic scoring."""
    7. return re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()
    8. def count_chars(text: str) -> int:
    9. """Ignore line breaks and whitespace; count each emoji as one character. C6 length counts the body only."""
    10. return len(re.sub(r"\s", "", text))
    11. def emoji_count(text: str) -> int:
    12. """Rough emoji count by common Unicode ranges; replace with a mature library in production."""
    13. return len(re.findall(
    14. r"[\U0001F1E6-\U0001F1FF\U0001F300-\U0001FAFF\u2600-\u27BF]",
    15. text,
    16. ))
    17. ALLOWED_YEARS = {"1998"} # C3: the only year given in the material; use a plain year set
    18. DISPARAGING = {"智商税", "割韭菜", "坑", "垃圾"}
    19. def contains_disparaging(text: str) -> bool:
    20. """Check by substring so that multi-character disparaging words are matched."""
    21. return any(word in text for word in DISPARAGING)
    22. def check_hard_constraints(body: str, title: str) -> dict:
    23. # Extract only four-digit years and return strings like "1998"
    24. years = set(re.findall(r"(?:19|20)\d{2}", body + title))
    25. return {
    26. "title_len_ok": len(title) <= 20, # C1
    27. "title_has_number": bool(re.search(r"\d", title)), # C1
    28. "length_ok": 400 <= count_chars(body) <= 500, # C6, body only
    29. "emoji_ok": emoji_count(body) <= 6, # C6
    30. "year_ok": years <= ALLOWED_YEARS, # C3
    31. "no_disparage": not contains_disparaging(body + title), # C4
    32. }
    33. # 2) Seven-dimension weighted aggregation
    34. DIMS = ["compliance", "consistency", "disclosure",
    35. "evidence", "persona", "platform", "value"]
    36. WEIGHTS = {
    37. "compliance": 0.30, # hard constraints
    38. "platform": 0.20, # output format and platform fit
    39. "value": 0.15, # clear conclusion
    40. "consistency": 0.10,
    41. "evidence": 0.10,
    42. "persona": 0.10,
    43. "disclosure": 0.05,
    44. }
    45. def weighted_score(scores: dict) -> float:
    46. """scores are seven-dimension [0,1] values; missing dimensions count as 0 to prevent silent omissions."""
    47. return round(sum(WEIGHTS[k] * scores.get(k, 0.0) for k in DIMS), 3)
    48. def summarize(rows: list) -> None:
    49. """Print rankings, dimension means, and ranges to spot dimensions with insufficient discrimination."""
    50. ranked = sorted(rows, key=lambda r: weighted_score(r["scores"]), reverse=True)
    51. for i, r in enumerate(ranked, 1):
    52. print(i, r["model"], weighted_score(r["scores"]))
    53. for dim in DIMS:
    54. vals = [r["scores"].get(dim, 0.0) for r in rows]
    55. if vals:
    56. print(dim, f"mean={statistics.mean(vals):.2f}",
    57. f"range={max(vals) - min(vals):.2f}")

    4. The Seven-Dimension Scoring System: Look Beyond the Total

    4.1 Dimension Definitions

    Dimension Focus Typical point loss
    Compliance Whether the six hard constraints are met one by one Title missing a number, omitting the owner’s temper or the wait time, length out of bounds
    Consistency Whether the output maps one-to-one to the material, with no self-contradiction Turning “bad temper” into “warm and friendly,” inventing an address or opening hours out of thin air
    Disclosure Restraint where the material provides no coverage Using “allegedly” or “as the story goes” to manufacture unverified endorsement
    Evidence Whether conclusions are supported by the material and verifiable by readers Writing only “the taste is amazing” without any verifiable fact like price or year
    Persona Whether it fits “picky but fair,” and drawbacks are not concealed Reading like a press release, or a rant throughout that loses the “fair”
    Platform fit Whether the title, opening, and overall style fit image-and-text recommendation platforms Piled-up long sentences, excessive emoji, lack of scene-setting
    Value judgment Whether the ending gives a clear “worth it or not” conclusion Being ambiguous, pushing the judgment back to the reader

    Each of the seven dimensions takes a value from 0 to 1. The first version can use equal weighting, but equal weighting masks differences in importance between dimensions. For real campaigns, compliance and platform fit usually matter more than disclosure.

    4.2 Observed Results: The Loss Distribution Is What Matters

    From the evaluation results, the gap between the first-place and tenth-place average scores is often not large. On a 0–1 scale, this gap is hard to explain as “writing quality,” but it better reflects “whether the model keeps its promises.” A few observations:

    1. Compliance is the main discriminator. Lower-ranked models mostly lose points on title numbers, mandatory drawbacks, or the concluding judgment, rather than writing less-beautiful sentences.
    2. Output format is an admission threshold. If an output with a reasoning block is compared directly against others without processing, it amounts to using writing style to compensate for a structural defect. This is not “how well it is written” but “whether it can be used directly.”
    3. Discrimination is unevenly distributed. Among the seven dimensions, consistency, disclosure, evidence, and persona often score high overall and close to one another, contributing little to ranking; what truly opens up the gap is compliance, platform fit, and value judgment. Evaluators should adjust weights accordingly, rather than letting all dimensions “split” the conclusion evenly.

    5. A Closer Look at Failure Modes: Three Typical Problems and Minimal Fixes

    First, leaked reasoning chains. The model outputs its reasoning process as part of the body, so the body is interrupted by lengthy analysis and repeated deliberation. If the publishing pipeline does nothing, this content both fails to read well and violates the basic expectation that “the body is clean and usable.” For evaluation, it is not a matter of losing a few points but of protocol contamination. The minimal fix is to add reasoning-block stripping and body validation to the publishing pipeline, and to set output structure as an admission requirement on the evaluation side.

    Second, instruction decay. The later and more mutually constraining the constraints in a prompt, the higher the chance they are dropped. Models often “remember the title length but forget the concluding stance.” Three mitigations:

    • Number the constraints and restate them one by one;
    • Require the model to fill in a constraint checklist before writing the body;
    • Re-run failure samples in a targeted way and record the failure type.

    But re-running only reduces sporadic failures; it cannot replace a model’s own long-constraint following ability.

    Third, factual hallucination and overreach. A model’s pretraining knowledge is often “richer” than the material, so it tends to fill in years, addresses, and per-person costs on its own, or to make judgments about competitors beyond the material. This kind of failure is the most insidious, because it reads flawlessly yet is a hard flaw at the level of content compliance and brand trust. There is only one principle for handling it: years and facts outside the material must not appear in the body, and competitors are stated as facts only.

    6. Selection Guidance: From “Total-Score Ranking” to “Constraint-Following Ability”

    Back to the original intent of evaluation—providing a basis for selection. Judge in the following order.

    Step 1: Check whether compliance passes. Meeting all hard constraints is the admission line; those who fail do not enter the writing-style comparison.

    Step 2: Look at weighted dimensions, not equal weights. Equal averaging makes a small score gap look like “everyone is about the same,” while weighting brings the gap back to its true weight. An example weighting for a “ready to publish directly” scenario:

    Dimension Equal weight Suggested weight Reason
    Compliance 0.143 0.30 If hard constraints fail, the content cannot be used directly
    Platform fit 0.143 0.20 Includes output format; decides whether it can enter the publishing pipeline
    Value judgment 0.143 0.15 Directly affects user decisions
    Consistency 0.143 0.10 A mismatch between description and material damages trust
    Evidence 0.143 0.10 Determines content credibility
    Persona 0.143 0.10 Affects account tone
    Disclosure 0.143 0.05 Can be relaxed when material is simple and needs no endorsement

    Step 3: Look at the cost of failure. A model weak at constraint-following is not necessarily unselectable, but it must be paired with a stable post-processing and retry mechanism. By comparison, models with controllable output structure cost less in pipeline integration and ongoing maintenance.

    Step 4: Use regression, not a one-time leaderboard. Vendors may silently update weights; last month’s high score may not hold this month. Selection conclusions should serve as a regression baseline, re-running the same set of constraint tasks monthly and watching changes in rank and key dimensions, rather than a one-time leaderboard.

    Common selection mistakes include: looking only at the total-score ranking, doing only a single sample, letting the same model be both contestant and judge, not fixing sampling parameters, not recording the model version, and ignoring failure-sample attribution. All of these strip the evaluation of reproducibility.

    7. Putting It into Practice: Building Evaluation Capability into the Platform

    Evaluation is not just “running a score once” but becoming a sustainable engineering capability:

    • Task-library accumulation: Turn the six hard constraints, the material, and the scoring rubric into a versionable task template; adding a new candidate model requires only changing the base_url and model name to re-run.
    • Automated pipeline: Chain batch running, validation, scoring, and aggregation into one pipeline, write results to a data warehouse or object storage, and add monitoring and alerting to catch abnormal score drops.
    • Rules first, model as supplement: Script validation can run offline in bulk—low cost and reproducible; the judge model suits semantic judgment, but avoid “the same model as both contestant and judge,” which tends to overestimate same-origin models.
    • Sample management and de-identification: The material may involve real merchant information; de-identify before saving to disk and restrict data access permissions.
    • Publishing fallback: Even if evaluation passes, keep deterministic checks for length, sensitive words, year, and output format in the publishing pipeline to reduce the probability of production incidents.

    8. Conclusion

    In long-constraint copy tasks, the key to evaluation is not simply judging “who writes better” but judging “who keeps its promises better.” Once model output is broken into dimensions such as compliance, platform fit, and value judgment, failures acquire an attribution path, and selection can shift from “looking at the leaderboard” to “looking at constraint-following ability, output format, and failure cost.” On this basis, using a sustainable regression evaluation to counter version drift is what allows LLM evaluation methods to truly serve content production and business decisions.