Evaluating Long-Form Generation LLMs: Methodology and Model Selection in Practice

    A practical guide to building a reproducible evaluation pipeline for long-form text generation. It covers designing hard, checkable constraints, defining scoring dimensions with weighted rubrics, controlling judge-model bias, batch-calling and auto-scoring candidates with Python, and translating evaluation results into domain-specific model selection decisions.

    1. Background: Long-Document Generation as a Stress Test

    Many people judge whether a large language model “is good enough” based on whether single-turn Q&A feels fluent or whether short copy looks polished. That approach is convenient, but it offers limited value for business scenarios that demand sustained, multi-constraint work. The capability that actually separates models is this: given a task with messy information and mutually constraining requirements, can a model reliably produce a long-form, internally consistent draft that is ready to enter a review process?

    An e-commerce promotion retrospective report is a typical stress test. Such a document has to handle several problems at once:

    • Good and bad results must appear side by side in the same document; neither side can be hidden behind the other.
    • Root-cause analysis must land on the process, not on individuals, or the document turns into a blame-assignment artifact.
    • “Unglamorous but real” facts, such as temporary staff reassignment, must feed into the delay analysis.
    • The document follows a fixed format, usually four parts: Results Review, Keep, Problem, and Try.
    • Follow-up actions must be executable, for example “lock resource scheduling in advance and name the change owner during requirements review,” rather than “strengthen communication.”
    • Self-flagellating phrasing such as “intense soul-searching” or “deep reflection” must not appear anywhere.

    As an example, six mainstream models completed the same task in parallel, with six hard requirements and a judge model scoring across seven dimensions (each out of 1.0). The results are shown below. For readability, models are labeled A through F.

    Model Audience Fit Compliance Factual Accuracy Tone Format Language Persuasiveness Material Selection Average
    A 0.94 0.98 0.98 0.78 0.94 0.88 0.93 0.92 0.92
    B 0.88 0.86 0.62 0.99 0.89 0.87 0.84 0.85 0.85
    C 0.84 0.81 0.46 1.00 0.86 0.76 0.68 0.77 0.77
    D 0.78 0.69 0.28 0.86 0.82 0.75 0.57 0.68 0.68
    E 0.65 0.88 0.48 0.32 0.86 0.47 0.58 0.61 0.61
    F 0.55 0.30 0.20 0.65 0.45 0.55 0.35 0.25 0.33

    A few details are worth noting:

    1. The gap between the highest and lowest scores is 0.59.
    2. Model F, at the bottom of the average, did not deliver a complete draft; its body stopped mid-sentence, so much of its low score comes from “task not completed.”
    3. Model B, right behind the leader with an overall close score, scored only 0.62 on factual accuracy, showing that a document which “looks excellent” can hide substantive errors.

    2. Method Overview: A Closed Loop from Task Definition to Selection

    Turning the subjective notion of “document quality” into a quantifiable, reproducible evaluation can be broken into five stages:

    1. flowchart LR
    2. A["1 Define the task and hard requirements"] --> B["2 Batch-call candidate models"]
    3. B --> C["3 Persist raw outputs to disk"]
    4. C --> D["4 Judge model scores by dimension"]
    5. D --> E["5 Interpret results and select a model"]
    6. E -. "Problem-driven iteration" .-> A

    Key points for each stage:

    • Define the task: Write business requirements as clear, decidable rules. The vaguer the rules, the more scoring depends on the judge model’s subjective preference, and the less trustworthy the results.
    • Batch calls: Give every candidate the same prompt and parameters, and sample repeatedly when necessary to separate genuine capability differences from random variance.
    • Persist to disk: Fully retain raw outputs, call timestamps, usage data, and so on, so results can be reviewed and reproduced.
    • Scoring: Use a judge model to score by dimension, paired with bias-control measures.
    • Interpretation: Weigh score gaps, failure modes, and cost against the business context, rather than staring only at the overall ranking.

    3. Task Design: Turning Vague Requirements into Decidable Hard Constraints

    The quality of the evaluation task directly determines how trustworthy the conclusions are. A good task design meets three criteria:

    1. Aligned with real business: Close to an actual deliverable, so a model cannot score by producing “polished empty talk.”
    2. Specific and checkable constraints: Every requirement can be verified line by line in the draft.
    3. Graded difficulty: Both capable and weaker models must separate clearly, and specific failure modes must be exposed.

    3.1 Six Typical Categories of Hard Constraints

    Constraint Type Example How to Judge
    Side-by-side presentation Delay and above-target conversion must both appear Marked as a violation if either side is singled out or downplayed
    Attribution target A first-day full-discount price display error (lasting about 2 hours, affecting roughly 300 orders) must be attributed to the process Check whether an individual is named or implicated
    Factual completeness The resource fact that 2 people were reassigned for 3 days must be included in the delay analysis Missing or mentioned only in passing counts as incomplete
    Structural format Organized into the four parts: Results Review / Keep / Problem / Try All parts present, in the correct order, with no additions or omissions
    Actionable actions Each improvement action must include an owner, a deadline, and an acceptance method Any vague statement like “strengthen communication” fails
    Tone and style No self-flagellating phrasing such as “intense soul-searching” or “deep reflection” Determinable by keyword or sentence-pattern matching

    3.2 Why These Constraints Effectively Separate Models

    • Stance constraints (side-by-side presentation, attribution): Test how faithfully the model follows instructions. Models tend to “hide deductions” or produce “people-pleasing” text; softening negative information violates the constraint.
    • Structural constraints: Test state retention over a long output. As the article reaches its later sections, earlier formatting requirements are easily forgotten.
    • Actionability constraints: Test the ability to turn knowledge into action. A model knows “communication matters” but cannot supply an owner, a deadline, and an acceptance standard.
    • Style constraints: Test compliance with negative instructions. Many models follow “do not…” instructions noticeably worse than positive ones.

    When these constraints stack up in a single long-form generation, weak spots are amplified quickly.

    3.3 Conflicts Among Constraints Are Themselves a Test

    In the example task, the six constraints pull against each other:

    • To report the delay honestly, you cannot emphasize only above-target conversion.
    • To attribute the issue to the process, you cannot “close the case quickly” with individual error.
    • To give staff reassignment its due place, you cannot pin the entire delay on design revisions.
    • To stay actionable, you cannot write “strengthen communication.”

    Meeting one or two of these is easy. The difficulty lies in holding every line at once, which is exactly what real business documents require.

    4. Scoring System: Breaking Down Dimensions and Designing Weights

    4.1 The Meaning and Judging Method of the Seven Dimensions

    Dimension What It Examines Typical Deductions
    Audience Fit Whether it meets the information needs of different readers such as management, execution teams, and collaborators Only a detailed running log with no conclusion, or only slogans with no evidence
    Compliance Whether it crosses disclosure boundaries or expression red lines Disclosing unnecessary internal details, or making inappropriate remarks
    Factual Accuracy Whether data, attribution, and conclusions match the given material Distorted numbers, misattribution, fabricated causes
    Tone Whether it fits the organization’s expression style and values Frivolous tone, or drifting into self-flagellating prose
    Format Paragraph structure, hierarchy, and degree of itemization Missing sections, scrambled order, vague action items
    Language Fluency, word accuracy, and information density Repetition and verbosity, empty talk, misuse of terminology
    Persuasiveness Whether the argument is complete and conclusions are supported Judgments without evidence, internal contradictions
    Material Selection Whether representative key events and data are chosen Listing irrelevant details, missing key milestones

    Dimension settings should correspond to business goals:

    • For a retrospective aimed at management, Audience Fit and Persuasiveness should carry higher weight.
    • For material intended for external disclosure, Compliance and Factual Accuracy need a substantial weight increase.

    It is recommended to state weights explicitly in the prompt to avoid the total score being dominated by a single dimension.

    4.2 Sample Weight Configuration

    1. {
    2. "dimensions": {
    3. "audience_fit": 0.15,
    4. "compliance": 0.15,
    5. "factual_accuracy": 0.25,
    6. "tone": 0.10,
    7. "format": 0.10,
    8. "language": 0.10,
    9. "persuasiveness": 0.10,
    10. "material_selection": 0.05
    11. },
    12. "hard_rules": [
    13. "Present delay and above-target conversion side by side; neither may be singled out or downplayed",
    14. "Attribute the price display issue to a process stage, not to an individual",
    15. "Reflect staff reassignment in the delay analysis",
    16. "Output in the four parts: Results Review, Keep, Problem, Try",
    17. "Each action item must include an owner, a deadline, and an acceptance method",
    18. "No self-flagellating or pledge-style phrasing"
    19. ],
    20. "scoring": "First check each hard requirement; if any one fails, that dimension is marked non-compliant, then scored by description"
    21. }

    4.3 Mandatory Zero Scores: Preventing “Boilerplate” from Scoring High

    A document that is neatly formatted, fluent, and long enough can score decently across several dimensions. If it omits key facts, misattributes causes, or skips a required section yet still earns a mid-to-high score, the evaluation is distorted. Mandatory zero-score rules are therefore needed:

    • If either the delay or the above-target conversion is missing, Audience Fit scores 0.
    • If the price issue is attributed to an individual, Compliance scores 0.
    • If any required section is missing, Format scores 0.
    • If the body is not fully output (for example, cut off midway), the entire answer is deemed non-compliant and excluded from ranking, or listed separately.
    • If action items contain vague statements, Persuasiveness is docked proportionally.

    5. Judge-Model Scoring: Process Design and Bias Control

    5.1 Basic Process

    1. Persist raw outputs to disk: Save the model’s returned text directly, with no preprocessing, so scoring is based on real output.
    2. Score one by one: Call the judge model separately for each document, producing a structured JSON result.
    3. Parse and retry: When parsing fails, fix the prompt and retry.
    4. Aggregate and rank: Compute the weighted total by weight, while retaining the per-dimension breakdown.

    5.2 Four Common Biases and Countermeasures

    • Position bias: The judge model tends to give higher scores to answers placed first or last. Countermeasure: score each document independently, rather than using a “compare and rank multiple documents” approach.
    • Length bias: Long, ornate answers tend to be favored. Countermeasure: state clearly in the scoring instructions that “concise and accurate takes priority over length.”
    • Self-preference: The judge model may favor answers stylistically similar to its own output. Countermeasure: cross-validate with judge models from different sources.
    • Dimension confusion: Judgments on different dimensions may influence each other. Countermeasure: require independent per-dimension scoring with reasons, avoiding an “overall impression” dominating the score.

    One point to view objectively: a judge model is not perfectly fair, and some dimensions leave room for subjectivity. But as long as the gap is large enough, as in this example where the highest and lowest differ by 0.59, the ranking signal itself is already clear enough to support a selection decision.

    5.3 A Standardized Structure for Scoring Results

    1. {
    2. "model": "candidate-model",
    3. "sample_id": 3,
    4. "scores": {
    5. "audience_fit": {"score": 0.0, "reason": "Omitted the above-target conversion side and only emphasized the delay"},
    6. "compliance": {"score": 1.0, "reason": "Attributed the price issue to the process, not to an individual"},
    7. "factual_accuracy": {"score": 0.8, "reason": "Data matches the given material; staff reassignment is mentioned only in one sentence in the delay analysis"},
    8. "tone": {"score": 0.9, "reason": "Professional tone, no self-flagellating phrasing"},
    9. "format": {"score": 1.0, "reason": "All four parts present and in the correct order"},
    10. "language": {"score": 0.8, "reason": "Some repetitive phrasing"},
    11. "persuasiveness": {"score": 0.6, "reason": "Two actions in the Try section lack an acceptance method"},
    12. "material_selection": {"score": 0.8, "reason": "Kept key milestones but downplayed the positive results"}
    13. },
    14. "hard_rule_violations": ["side-by-side presentation"],
    15. "overall_comment": "Complete structure and appropriate tone; main deductions come from failing to present results side by side and from non-actionable items"
    16. }

    6. Engineering Implementation: Batch Calls and Automated Review

    6.1 Batch-Calling Candidate Models

    The code below works with any model service that provides a chat-completion-style compatible API. In environments such as Baidu AI Cloud, fill in the connection parameters, such as the model call name and keys, using the information provided by that environment:

    1. import os
    2. from concurrent.futures import ThreadPoolExecutor
    3. from openai import OpenAI
    4. # Inject connection info via environment variables to keep keys out of code and repos
    5. client = OpenAI(
    6. api_key=os.environ["LLM_API_KEY"],
    7. base_url=os.environ["LLM_BASE_URL"],
    8. )
    9. CANDIDATES = [
    10. "model-a", "model-b", "model-c",
    11. "model-d", "model-e", "model-f",
    12. ]
    13. PROMPT = open("prompt.txt", encoding="utf-8").read()
    14. def call_once(model: str, sample_id: int) -> dict:
    15. """Call a single model and return the raw output; exceptions propagate for the caller to retry"""
    16. resp = client.chat.completions.create(
    17. model=model,
    18. messages=[{"role": "user", "content": PROMPT}],
    19. temperature=0.0, # Evaluation requires reproducibility, so sampling temperature is set to 0
    20. )
    21. return {
    22. "model": model,
    23. "sample_id": sample_id,
    24. "content": resp.choices[0].message.content,
    25. "usage": resp.usage.model_dump(), # Record token usage for cost accounting
    26. }
    27. results = []
    28. with ThreadPoolExecutor(max_workers=6) as pool:
    29. futures = [pool.submit(call_once, m, i) for m in CANDIDATES for i in range(3)]
    30. for f in futures:
    31. try:
    32. results.append(f.result())
    33. except Exception as e:
    34. print(f"Call failed: {e}") # In production, log full context and auto-retry
    35. # Persist raw outputs fully: saving failed samples matters too, as they are part of the evaluation
    36. with open("raw_outputs.jsonl", "w", encoding="utf-8") as f:
    37. for r in results:
    38. f.write(json.dumps(r, ensure_ascii=False) + "\n")

    Points to note in the implementation:

    • Repeated sampling: Sample each model multiple times (3 times in the example) to separate capability differences from random variance.
    • Consistent parameters: All candidates use the same prompt and the same temperature. Some services may not support a custom temperature; in that case, record the parameters actually used consistently.
    • Persist raw output: Save raw output rather than cleaned text, so the evaluation conclusions hold up to review.
    • Keep failed samples: Samples that fail to call or produce incomplete output should also be recorded, not silently discarded.

    6.2 Automated Scoring with a Judge Model

    1. import json
    2. import time
    3. JUDGE_MODEL = "judge-model"
    4. RUBRIC = """
    5. You are a senior technical reviewer. Score the given document on each of the following 8 dimensions (0.00~1.00, up to two decimals):
    6. audience fit, compliance, factual accuracy, tone, format, language, persuasiveness, material selection.
    7. Scoring instructions:
    8. 1. Hard requirements: present delay and above-target conversion side by side; attribute the price issue to the process, not an individual;
    9. reflect staff reassignment in the delay analysis; output in the four-part structure;
    10. each action item must include an owner, a deadline, and an acceptance method; no self-flagellating phrasing.
    11. 2. If any hard requirement is not met, list it in hard_rule_violations and dock the corresponding dimension.
    12. 3. Concise and accurate takes priority over length and ornate wording; judge each dimension independently and give a brief reason.
    13. 4. Output strictly the following JSON and nothing else:
    14. {"scores": {"<dimension>": {"score": 0.0, "reason": "..."}},
    15. "hard_rule_violations": ["..."], "overall_comment": "..."}
    16. """
    17. WEIGHTS = {
    18. "audience_fit": 0.15, "compliance": 0.15, "factual_accuracy": 0.25, "tone": 0.10,
    19. "format": 0.10, "language": 0.10, "persuasiveness": 0.10, "material_selection": 0.05,
    20. }
    21. def judge_one(doc: dict) -> dict:
    22. resp = client.chat.completions.create(
    23. model=JUDGE_MODEL,
    24. messages=[
    25. {"role": "system", "content": RUBRIC},
    26. {"role": "user", "content": doc["content"]},
    27. ],
    28. temperature=0.0,
    29. response_format={"type": "json_object"}, # Supported by some compatible APIs; remove it if not supported
    30. )
    31. return json.loads(resp.choices[0].message.content)
    32. report = []
    33. for doc in results:
    34. for attempt in range(3):
    35. try:
    36. verdict = judge_one(doc)
    37. break
    38. except (json.JSONDecodeError, KeyError):
    39. time.sleep(2 ** attempt) # Exponential backoff retry
    40. else:
    41. verdict = {"scores": {}, "hard_rule_violations": ["Judge output parsing failed"]}
    42. weighted = sum(WEIGHTS.get(k, 0) * v["score"] for k, v in verdict["scores"].items())
    43. report.append({"model": doc["model"], "sample_id": doc["sample_id"],
    44. "weighted_score": round(weighted, 4), **verdict})

    Notes:

    • Separated weights: Weights are maintained centrally in code, decoupled from the scoring prompt, so adjusting weights requires no change to the model instructions.
    • Exception fallback: On parsing failure, retry with exponential backoff; if retries are exhausted, record the sample as anomalous to keep bad data out of the aggregate.
    • Traceable results: The final report keeps each dimension’s score, the violations, and the overall comment, making it easy to trace the reason for each deduction.

    6.3 Additional Engineering Recommendations

    • Concurrency control: Set the concurrency level according to the server’s rate limits to avoid throttling; introduce a request queue when necessary.
    • Cost control: Long-form generation and judge scoring both consume many tokens. Validate the prompt and scoring standard on a small number of samples first, then scale up to the full set.
    • Human review: Sample the automated scores for human review, especially the factual accuracy dimension. Automated scoring is for large-scale screening; final conclusions still need human judgment.

    7. Result Interpretation: What to Look At Beyond the Score

    1. Look at the gap first, then the ranking.
    A 0.59 gap means the top and bottom models are not on the same usability tier. At this magnitude, subjective error on any single dimension is not enough to overturn the overall conclusion.

    2. Dimensional weaknesses matter more than the total.

    • B’s factual accuracy score of only 0.62 means the documents it produces “read as very professional” yet may contain data or conclusions that do not survive checking. Once such documents enter review, rework costs are often higher than a straightforward rewrite.
    • E’s tone score of only 0.32 shows its expression style is severely misaligned with the organization’s requirements.

    3. Failure modes define the boundaries of applicability.

    • Output cut off midway (like F): unsuitable for long-form scenarios requiring a complete deliverable.
    • Weak on facts (like B): unsuitable for scenarios with high data-accuracy requirements, even when the language flows well.
    • Too much vague boilerplate (like E): unsuitable for scenarios that need to be turned into concrete execution.

    4. Evaluation conclusions have applicability limits.
    An evaluation reflects only performance under “this type of task, this set of constraints, this one judge.” Model versions keep updating, and the evaluation set should be expanded and refreshed periodically. Conclusions should note the evaluation date and task type.

    8. From Evaluation to Selection: Matching Model Capabilities by Domain

    Back to the question of “which model suits which domain.” Rather than claiming some model “can do anything,” map business needs onto capability dimensions:

    Business Domain Core Requirements Key Dimensions to Examine Selection Advice
    Operations and project retrospectives Balanced stance, objective attribution, actionable follow-ups Audience Fit, Factual Accuracy, Persuasiveness Prefer models with high total scores and stable factual accuracy
    Enterprise documents and report writing Structural compliance, consistent style, high information density Format, Language, Audience Fit Focus on long-form structural retention; verify stability with repeated sampling
    Data analysis and business insight Conclusions consistent with data, no fabricated causes Factual Accuracy, Material Selection, Persuasiveness Use models with low factual accuracy cautiously; conclusions need human review
    Customer service and knowledge Q&A Compliance, faithfulness to the knowledge base, no over-promising Compliance, Factual Accuracy Prefer models stable on compliance and factual accuracy, paired with retrieval-augmented generation
    Coding and automation scripts Correct, runnable code Language, Persuasiveness (logical consistency) Set up a separate code-execution evaluation; do not rely on text scores alone
    Marketing copy and creative work Language expressiveness, style fit Language, Tone Relatively lenient on factual accuracy, but do not relax compliance review

    A “three-step method” is recommended for selection:

    1. Map the scenario constraints: List the requirements you must hold to and the output spec, as the basis for the evaluation prompt.
    2. Build a small evaluation set: Extract 10–30 typical tasks from real business, covering easy, medium, and hard difficulty.
    3. Weighted scoring and cross-validation: Set weights by business priority, re-test with a different judge model, and sample for human review.

    It is worth stressing that the goal of evaluation is to choose the right tool and define boundaries, not to find an “all-capable model.” Striking a balance between cost and effectiveness is the core of engineering practice.

    9. Conclusion

    Evaluating long-form generation tasks is fundamentally about answering one question: when requirements pull against each other and the length keeps growing, can the model still hold every line? The approach in this article can be summarized in four points:

    • Write constraints clearly, so judgment can be objective;
    • Break dimensions down finely, and align weights with business;
    • Control judge bias, and trust the ranking signal;
    • Failure modes reveal applicability boundaries better than the total score.

    Solidify this workflow, and you can keep producing reproducible, low-cost selection evidence in an environment where models update frequently.