LLMs for Newsrooms: Scenario-Based Evaluation and Engineering Practices

    This article presents a scenario-based evaluation framework and an engineering blueprint for deploying large language models (LLMs) in news editorial workflows. It explains why news production is a specialized, time-sensitive, zero-tolerance pipeline where general-purpose models struggle, then breaks the workflow into five core tasks—interview outline design, article writing, proofreading, headline generation, and short-video rewriting—with recommended metrics and reference thresholds. It draws out four key findings, lays out a four-layer engineering approach for moving from "usable" to "reliable," describes scenario-based model selection and routing strategies, and defines compliance and data-security boundaries. Media teams will gain practical guidance on building their own evaluation sets, constraining facts with retrieval augmentation and citation checks, keeping human review at critical checkpoints, and applying automated validation before distribution.

    When media organizations bring LLMs into their workflows, a few recurring questions come up: How do you evaluate a model for editorial tasks? How do you control hallucination? How do you choose a model for headline generation? How do you reduce missed errors during proofreading? What should you watch for when selecting a model? This article offers a scenario-based evaluation and engineering implementation plan built around the news editorial workflow, helping teams move from “usable” to “reliable.”

    1. Why News Editing Is a “Hard Problem” for LLMs

    News content production may look like “writing articles,” but it is really a highly specialized, time-sensitive, zero-tolerance pipeline: from topic planning, interview outline design, and material collection, to drafting, fact-checking, proofreading, and headline generation, and then to summary cards, short-video scripts, and multi-format distribution. Each stage places different demands on factual accuracy, style conventions, news-value judgment, and timeliness.

    If a general-purpose LLM is dropped straight into the editorial workflow, a series of problems tends to surface: the model can produce fluent text yet confidently fabricate facts; it can rewrite sentences yet fail to understand “why this news matters”; it can perform well on a single task yet fail to stay reliable across the whole pipeline. So before implementation, teams must define the model’s capability boundaries: LLMs are well suited to assisted generation, structured organization, and initial screening, but should not be solely responsible for final factual judgment.

    2. Scenario-Based Evaluation: Mapping Capabilities Across the Workflow

    The prerequisite for sound evaluation is to break things down “by scenario” rather than assigning a single overall score. Below is an engineering-oriented evaluation template. For each task category, aim for 100–300 samples, drawn from the organization’s real articles, historical revision samples, anonymized interview material, and edge cases. Annotation should use dual review, and scoring should combine rule-based checks with human ratings. The thresholds in the table are not universal industry standards; they need to be calibrated against the outlet’s positioning, article type, and publishing risk.

    Evaluation Task Core Capabilities Examined Recommended Metrics Reference Passing Threshold
    Interview outline design Topic decomposition, layered coverage, contrasting perspectives, question executability Key-point coverage, factual error rate, structural completeness Key-point coverage ≥85%, factual errors should be 0
    News article writing Genre control, information organization, logical coherence, factual consistency Factual accuracy, citation traceability, genre compliance rate Factual accuracy ≥95%, missing-citation rate <2%
    Manuscript proofreading Typos, factual errors, internal contradictions, data inconsistencies Error recall, false-positive rate, contradiction detection rate Key error recall ≥90%, false-positive rate ≤5%
    Headline generation Core-fact compression, appeal, avoiding exaggeration Core-fact coverage, headline consistency, exaggeration risk Core-fact coverage ≥90%, exaggeration risk should be 0
    Short-video script rewriting Colloquial tone, rhythm, shot adaptation, no factual distortion Factual consistency, spoken fluency, human edit rate Factual consistency ≥95%, human edit rate ≤20%

    The key insight is that scores across these five task types are usually uneven. Writing tasks tend to earn higher marks because the stylistic patterns are mature and the training data is abundant. By contrast, high-precision, low-tolerance tasks such as proofreading and headline generation often score noticeably lower. This shows that a model’s “generative ability” is not in sync with its “discrimination ability” and “judgment.” Precisely where editorial work can least afford mistakes, the model tends to be weakest.

    3. Four Conclusions Worth Confronting

    Using the evaluation criteria above and running small-scale internal tests on 100–300 samples per task type, a few trends typically emerge. These conclusions are meant for general capability comparison and selection reference within standard news content production; they do not replace an individual organization’s own retesting. For high-risk topics, investigative reporting, politically sensitive content, and specialized fields such as healthcare and finance, a dedicated acceptance mechanism should be established separately.

    1. No single LLM currently completes the full editorial workflow at high quality.
    Strong performance on individual tasks does not equal end-to-end reliability. Any solution claiming “end-to-end automated editorial production” should be treated with caution. A more sensible approach is to decompose tasks and combine stages, letting the model handle assisted generation, initial screening, and structured organization rather than expecting one model to do everything.

    2. Hallucination remains unsolved, and errors are becoming more subtle.
    A model’s fluent expression can actually mask factual errors. The more “smooth” it reads, the more likely people are to lower their guard. Fact-checking must therefore exist as an independent stage and cannot rely on the model to be “self-aware.” Combining closed material sets, mandatory citations, citation verification, automatic interception, and human review can reduce the probability that hallucinations enter the production pipeline, but it cannot guarantee complete elimination.

    3. A “cold” model struggles to judge news value.
    Value judgments—timeliness, degree of public concern, scope of impact, and the outlet’s editorial stance—depend heavily on context, audience, and editorial experience. General-purpose models lack a stable “nose for news” and need external rules, domain knowledge, and human oversight. This is especially true for headline generation and topic ranking: don’t judge only by language fluency, but also by whether the output is accurate, restrained, and verifiable.

    4. Select LLMs according to the editorial scenario.
    Matching different tasks to different models is the most pragmatic strategy. For example, interview outlines can favor models with strong reasoning and structuring capabilities; article writing can use long-context models combined with RAG; proofreading can combine a rule engine with a high-precision small model; and headline generation suits a pipeline of multi-candidate generation plus rule filtering plus human confirmation. Combined with routing and orchestration, this balances cost, speed, and quality.

    4. Engineering Implementation: From “Usable” to “Reliable”

    To turn evaluation conclusions into a running system, four safeguards need to be built at the engineering level.

    4.1 Build Your Own Scenario-Based Evaluation Set

    Don’t rely only on public leaderboards. Media organizations should build their own evaluation sets based on real articles, ideally covering three sample types: high-quality originals, historically returned/edited drafts, and risk-edge drafts. For each task type, define quantifiable scoring dimensions such as factual accuracy, citation traceability, key-point coverage, format compliance, and human edit distance.

    A checklist for building the evaluation set:

    • At least 100 samples per task type;
    • Coverage of routine articles, breaking news, in-depth reports, short-video scripts, and similar types;
    • Anonymization applied to samples;
    • Gold-standard answers or scoring criteria annotated;
    • Reasons for human edits recorded to support continuous iteration.

    4.2 Constrain Facts with Retrieval Augmentation

    Make the model “answer based on the given materials” rather than “generate from memory,” and require citation sources in the output. This reduces the risk of unfounded generation, but it still needs to be paired with citation verification and human review.

    Note the applicable boundaries: RAG is not a cure-all. If the material is missing, cannot be retrieved, conflicts with itself, or the model misinterprets the citation relationships, errors can still occur. It is therefore advisable to add three types of checks:

    • Whether the citation exists;
    • Whether the citation is out of range;
    • Whether key numbers, dates, and locations correspond to the material.

    4.3 Human-Machine Collaboration: Keep the Critical Checkpoints

    In particular, human review must be retained at the stages of fact-checking, news-value judgment, headline-exaggeration risk, and sensitive-information identification. The model should be positioned as an “efficient assistant,” not a “final editor.”

    A recommended human review checklist:

    • Do key facts have sources?
    • Are numbers, dates, and locations consistent?
    • Are quotations accurate?
    • Are there privacy, copyright, or interviewee-consent issues?
    • Does the headline accurately summarize the core facts?
    • Is there any sensationalism, exaggeration, or stance bias?

    4.4 Structured Output and Automated Validation

    Use programs to run format and consistency checks on the model’s output, intercepting non-compliant results before distribution. Common automated validation rules include:

    Validation Check Trigger Condition Handling
    Citation out of range Citation number <1 or > the number of materials Intercept and escalate to human review
    Output without citation Factual content has no citation number at all Return for regeneration
    Headline exaggeration Unsupported expressions such as “most,” “only,” or “unprecedented” appear Intercept or downgrade to human review
    Number inconsistency Numbers in the output differ from the material Escalate to human verification
    Missing format Key fields such as headline, summary, or storyboard are absent Block the publishing flow

    The following example code shows a minimal implementation pattern for “initialize the model call + constrain with materials + structured validation.” The example uses an OpenAI-compatible interface for illustration; in practice it can be swapped for the SDK of a model platform such as Baidu AI Cloud Qianfan. The key steps are installing dependencies, configuring authentication, and initializing the model.

    1. # Example: pip install openai
    2. import os, re
    3. from openai import OpenAI
    4. client = OpenAI(
    5. api_key=os.getenv("LLM_API_KEY"),
    6. base_url=os.getenv("LLM_BASE_URL") or None,
    7. )
    8. # Explicitly initialize the model call function to avoid an undefined llm
    9. llm = client.chat.completions.create
    10. def call_llm(prompt: str, materials: list[str]) -> str:
    11. """Call the LLM, forcing answers based only on the given materials."""
    12. context = "\n\n".join(f"[{i+1}] {m}" for i, m in enumerate(materials))
    13. system = "You are a rigorous editorial assistant. Answer only based on the given materials and never fabricate facts."
    14. user = f"Materials:\n{context}\n\nTask: {prompt}\nRequirement: annotate a citation number after each fact, e.g., [1]."
    15. resp = llm(
    16. model=os.getenv("LLM_MODEL", "ernie-4.0-8k"),
    17. messages=[
    18. {"role": "system", "content": system},
    19. {"role": "user", "content": user},
    20. ],
    21. temperature=0.2,
    22. )
    23. return resp.choices[0].message.content or ""
    24. def validate_output(text: str, ref_count: int) -> dict:
    25. """Structured validation: check whether citations exist and are within range."""
    26. cited = [int(x) for x in re.findall(r"\[(\d+)\]", text)]
    27. out_of_range = [c for c in cited if c < 1 or c > ref_count]
    28. return {
    29. "has_citation": len(cited) > 0,
    30. "invalid_citations": out_of_range,
    31. "passed": len(cited) > 0 and not out_of_range,
    32. }
    33. materials = [
    34. "Rainfall in a region hit a ten-year seasonal high, with a peak hourly intensity of 60 mm.",
    35. "The relevant authorities have activated an emergency response and relocated residents in low-lying areas.",
    36. ]
    37. draft = call_llm("Generate a news headline and a one-sentence summary from the materials above.", materials)
    38. result = validate_output(draft, len(materials))
    39. if not result["passed"]:
    40. draft = "[Pending Human Review] " + draft
    41. print(draft, result)

    5. Scenario-Based Selection and Routing Strategy

    Different editorial tasks demand different model capabilities, so it is not advisable to cover every stage with the same model. The following routing strategy can serve as a reference:

    Task Recommended Routing Key Requirements
    Interview outline design Strong reasoning general model + domain knowledge base Cover topics, question executability, avoid vagueness
    Article writing Long-context model + RAG Traceable facts, clear structure, accurate genre
    Proofreading Rule engine + high-precision model Low false positives, localizable, reviewable
    Headline generation Multi-candidate generation + rule filtering + human confirmation Accurate, concise, no exaggeration
    Short-video rewriting Transcription model + fact verification + storyboard template Colloquial, rhythmic, no factual distortion

    If the organization has in-house vertical models—for example, in finance, sports, or local-interest news—they can be prioritized for the corresponding scenarios. If no vertical models are available, the gaps can be bridged through prompt templates, knowledge-base retrieval, and workflow orchestration.

    6. Compliance and Data-Security Boundaries

    When object storage holds material and manuscripts, technical availability is not the only concern; requirements for copyright authorization, interview-material privacy, and data retention and anonymization must also be met. At a minimum, the following mechanisms are advisable:

    • Maintain an authorization ledger for articles, images, video, and interview recordings;
    • Apply tiered authorization and least-privilege access to interviewee information, minors’ information, and sensitive leads;
    • Anonymize personal data in evaluation sets and training prompts;
    • Set material retention periods and deletion mechanisms;
    • Enable object-storage access logging and auditing;
    • Run copyright, privacy, source-protection, and factual-compliance checks before publishing.

    For high-risk content such as breaking news, disaster reporting, public safety, healthcare, and financial investment, fully automated publishing is not advisable; human final review should be retained.

    7. Summary: A Selection Checklist and Common Pitfalls

    When evaluating an LLM editorial solution, media organizations can use the following checklist:

    • Are tasks such as interview, writing, proofreading, headlines, and short video evaluated separately?
    • Is there a scenario-based evaluation set built from real articles?
    • Are metrics such as factual accuracy, citation traceability, and human edit rate clearly defined?
    • Is generation based on materials and citation output enforced?
    • Are there automatic interception rules for citation out-of-range, missing format, and headline exaggeration?
    • Is human review retained at the fact-checking and value-judgment stages?
    • Are copyright, privacy, interview consent, and data-retention issues addressed?
    • Can it route to different models, rules, or human stages depending on the task?

    Common pitfalls include: judging only by public leaderboards; pursuing “one model that automates the whole workflow”; letting the model publish unverified articles directly; using a general-purpose model to judge news value; ignoring headline exaggeration and copyright/privacy risks; and failing to establish a continuous evaluation and iteration mechanism.

    Bringing LLMs into news editing is not about “replacing editors,” but about “reshaping the workflow.” The genuinely valuable approach is to place the model at the right task stage, use scenario-based evaluation to define capability boundaries, use retrieval augmentation and citation checks to constrain facts, use automated validation to intercept format and logic risks, and use human-machine collaboration to hold the line on news value and compliance. For media organizations, mastering evaluation methods and engineering patterns matters far more than chasing a single leaderboard ranking.