How to Evaluate AI Writing: A Practical Evaluation Framework
AI-generated text often reads smoothly while carrying little verifiable value, so judging it by whether it "reads like a human wrote it" misses the point. This article proposes a practical framework for evaluating AI writing instead. It defines where AI writing sits relative to first-hand observation, draws the boundary between tasks AI can and cannot own, and introduces a five-dimensional model covering factuality, traceability, experiential authenticity, structure and information density, and risk compliance. It then shows how to automate a first-pass fact-alignment pipeline in Python, how to push verifiability upstream through output contracts and controlled context, how to set acceptance criteria per scenario, and which metrics and regression practices make quality observable over time. The core message is to judge three things: whether claims are grounded, whether the grounding is checkable, and whether content that should be left blank is honestly left blank.
1. Where the Problem Starts: An Overlooked Premise
After taking part in on-location filming and first-line hands-on work, a researcher made a sharp observation: the sensation of a sunburned arm can hardly be conveyed accurately through text or images. He had read about the “primacy of bodily perception” in the phenomenological tradition and agreed with it in theory, but only after spending a few days in the sun did he understand that a gap separates theory from physical experience.
This observation is critical for evaluating AI writing, because AI writing sits even further back than “reading a report from your desk”: it is a re-organization of an existing text distribution, not an observation of the world. Even a poor human writer at least stands at the edge of the scene; the model has never been to the scene at all and holds only descriptions of how others described it.
From this, the first principle for evaluating AI writing follows: there is no necessary link between fluency and authenticity, and the two are often negatively correlated. A language model’s training objective pushes it toward output that “sounds most plausible,” and the most plausible phrasing tends to be full of smooth but empty transitions, flattened individual differences, and details filled in for narrative completeness. Such text reads the most comfortably but has the lowest verifiable value.
So the right question is not “Is the AI writing good?” but rather: does every assertion in this text have a traceable source, and is every piece of experiential content that should not be machine-generated honestly left blank?
2. Capability Boundaries: What AI Can and Cannot Own
Before designing evaluation criteria, you must split the task to either side of the capability boundary. Otherwise you hit the most common failure mode: using AI in the wrong place, then evaluating it on the wrong dimension.
Tasks suitable for AI writing:
- Structured re-organization: turning scattered material into outlines, meeting notes, or itemized explanations
- Multi-source summarization and alignment: compressing several materials into a unified overview while flagging disagreements
- Format and style conversion: rewriting technical documents, unifying terminology, translating across languages, and producing layered explanations for different readers
- Templated output: API documentation skeletons, release notes, test-case descriptions, and readable summaries of logs and alerts
- Draft acceleration and reverse review: generating first drafts, or checking a human-written document for omissions and internal contradictions
Parts that are unsuitable for AI writing—or at least cannot be finalized by it independently:
- First-hand experiential description: the smell of a scene, physical sensations, degree of fatigue, and the concrete unfolding of unexpected events
- First-hand data and time-sensitive facts: numbers, dates, prices, people, and places that are not in the training corpus
- Accountability- and commitment-related statements: liability determinations, compensation terms, compliance commitments
- Judgments that require a named author to bear responsibility: conclusions must be owned by an identifiable party, and the model bears no consequences
Mixing these two categories is the structural cause of most AI writing incidents—not poorly written prompts.
3. A Five-Dimensional Evaluation Model
Break the evaluation into five orthogonal dimensions, and give each one a decidable pass condition, so you avoid irreproducible conclusions like “it feels pretty good overall.”
| Dimension | Core question | How to judge |
|---|---|---|
| Factuality | Do assertions match the material? | Align each one against the source; count hits and conflicts |
| Traceability | Can each assertion point to a specific origin? | Citation coverage; share of unsourced assertions |
| Experiential authenticity | Are there sensory details fabricated by the machine? | Whether combinations of sensory + concrete quantities are traceable |
| Structure and information density | Is there real information gain? | Ratio of redundant sentences, density of filler phrases, number of points per paragraph |
| Risk and compliance | Does it contain sensitive information or absolute promises? | Rule library + sensitive-entity scan + human review |
The third dimension is the easiest to miss and best reflects the researcher’s insight. A detection approach is to run co-occurrence statistics on embodied vocabulary such as “sunburned,” “aching,” “cold wind,” and “before dawn,” together with specific quantitative expressions such as “lasted,” “about,” “three hours,” and “forty degrees.” Any fragment where such co-occurrence appears but no corresponding evidence exists in the source library should be flagged as high risk and sent to human review. Real sensory detail always comes from observation records, whereas fabricated sensory detail always carries both “vivid” and “precise” traits at once—the fingerprint left by the machine’s filling-in process.
4. Automating the Evaluation: A Fact-Alignment Pipeline
Human review does not scale, so you must use a program for the first layer of filtering. The core idea is to break the generated output into atomic claims and then re-check each one against its source.
from dataclasses import dataclass, field@dataclassclass Claim:text: strhas_number: bool = False # contains numbers, dates, unitshas_sensory: bool = False # contains sensory/embodied descriptionneeds_source: bool = True # whether evidence support is required@dataclassclass Verdict:claim: Claimevidence: list[str] = field(default_factory=list)score: float = 0.0level: str = "manual" # pass / manual / blockdef extract_claims(draft: str, sensory_lexicon: set[str]) -> list[Claim]:"""Split into sentences and tag claim attributes; rules can be extended with a business lexicon."""claims = []for sent in split_sentences(draft):tokens = set(tokenize(sent))claims.append(Claim(text=sent,has_number=bool(NUMBER_PATTERN.search(sent)),has_sensory=bool(tokens & sensory_lexicon),))return claimsdef align_with_sources(claim: Claim, index) -> tuple[list[str], float]:"""Retrieve evidence from the source index; return matched snippets and the highest relevance score."""hits = index.search(claim.text, top_k=5)return [h.snippet for h in hits], (hits[0].score if hits else 0.0)def review(draft: str, index, threshold: float = 0.72) -> list[Verdict]:results = []for c in extract_claims(draft, SENSORY_LEXICON):evidence, score = align_with_sources(c, index)if c.has_number and not evidence:level = "block" # key numbers must have first-hand evidenceelif c.has_sensory and score < threshold:level = "block" # embodied details without a source are always blockedelif score >= threshold:level = "pass"results.append(Verdict(c, evidence, score, level))return results
Three implementation points:
- Threshold zoning, not a single cut. Use strict blocking for numeric and embodied claims, and relax the bar for explanatory and transitional text; otherwise reviewers drown in false positives and quickly stop reading the report.
- Conflict is more dangerous than absence. A retrieval hit does not mean agreement. If the source says “about 200 participants” while the draft says “300,” the relevance score may still be high, so you must additionally perform an extractive comparison of numbers and units and cannot rely on similarity alone.
- The report must be directly actionable. Fix the output structure as “claim text / verdict level / evidence snippet / recommended action,” so reviewers do not have to hunt back through the draft to find the location.
5. Governing from the Generation Side: Output Contracts and Context Constraints
Evaluation is an after-the-fact action; it is more costly to push verifiability forward into the generation stage. This is fundamentally an interface-design problem, not a matter of rhetoric.
First, define material boundaries and allow refusal. Write clearly in the instructions: “Answer only from the attached materials; for information not in the materials, output a to-be-confirmed item.” Without this clause, the model fills the gap with general knowledge, and the more naturally it fills it, the harder the gap is to detect.
Second, require structured output that keeps conclusions and evidence in separate columns. For example:
{"summary": "The redesign reduced first-screen render time","claims": [{"text": "First-screen render time decreased", "value": null, "unit": null,"source_id": null, "status": "unverified"}],"open_questions": ["The magnitude of the decrease and the measurement basis are not given in the materials"],"provenance_required": true}
The status field makes “unverified” a first-class citizen. The easiest mistake a person makes when reading a draft is to treat a confidently worded sentence as a verified fact; an explicit status flag sharply reduces this kind of misjudgment.
Third, inject controlled context. The glossary, definitions of terms and metrics, a list of forbidden phrasings, and the target-reader level should be delivered with the request in a structured way, rather than relying on hand-written prompts each time. Using three different phrasings for the same term across paragraphs is the most common low-quality signal in AI writing—and also the easiest to catch automatically with a consistency check.
Fourth, the sampling strategy should serve stability. Output aimed at factual statements should favor more deterministic settings, and key drafts should be generated multiple times for comparison: if two generations from the same material differ factually, the output is sensitive to sampling, and neither version should be adopted directly.
6. Acceptance Criteria by Scenario
A single scorecard cannot solve every scenario; acceptance thresholds must vary with the use case.
- Technical documentation and API specifications: judged by executability. Whether examples run, whether parameter names match the definitions, and whether error codes are covered matter far more than elegant prose.
- Data reports: numbers and measurement basis come first. Every number must trace to a query or data source, with units, time windows, and sample scope stated explicitly.
- Oral records and field reporting: first-hand experience cannot be outsourced. Here the reasonable role for AI is transcription and organization, deduplication, structured annotation, and cross-checking; the narrative subject must be the real interviewee. Once you let the model “fill in the on-site feeling,” the ethical foundation of the work is hollowed out.
- External copy and operational content: compliance first. Absolute statements, effect guarantees, and unverified comparative claims should be hard-blocked by the rule library.
- Ops scripts and automation configs: judged solely by testing and runnability. Review looks at behavior, not wording.
7. Continuous Measurement: Making Quality an Observable Metric
A one-off review can only tell you how this one piece turned out; a team needs trends. Track at least the following metrics:
- Fact-error density: number of confirmed factual errors per thousand words, counted separately by scenario
- Unsourced-claim share: proportion of to-be-confirmed claims output by the automated pipeline
- Human-edit rate: edit distance between the final and first drafts, split into structural changes and wording changes
- Rework rounds: number of review cycles needed to go from first draft to publishable
- First-pass rate: share of pieces that can be published without substantive revision
The supporting mechanism is a regression set: accumulate every problem fragment, verdict, and correct phrasing found during each review into fixed test cases. After changing a prompt template or tuning pipeline rules, run the regression set before shipping; otherwise it is easy to fall into a loop of “fixing scenario A and breaking scenario B.”
8. A Practical End-to-End Workflow
Chain the steps above into a closed loop:
material collection → material ingestion (object storage + retrieval index)→ generation (with citations, status flags, and refusal allowed)→ automated validation (rule blocking + evidence alignment + consistency checks)→ human review (factual owner + domain reviewer + language editor)→ publish and archive (retain material fingerprints and generation parameters)→ spot-check and feedback → regression set
Material ingestion deserves emphasis: the ceiling on traceability is set by how you manage your materials. If source material is scattered across personal folders and chat logs, then no matter how clever the alignment algorithm is, it has nowhere to start. Unified ingestion that preserves source identifiers and collection timestamps is the precondition for the entire evaluation system to hold. Preserving generation parameters and the version of materials used during archiving is the only reliable basis for later retrospectives and dispute handling.
9. Conclusion
Return to that first observation about the sunburn. What it reveals is not that “AI writing is no good,” but that the responsibilities of AI writing must be redrawn: the machine excels at organization, compression, transformation, and consistency checking, and the quality of this work can be measured objectively. Experience, presence, and responsibility, by contrast, must be supplied by people—they cannot be replaced by paraphrase and should not be faked.
So the right posture toward evaluating AI writing is not judging whether it writes more or less like a human, but checking three things: whether assertions are grounded, whether the grounding is checkable, and whether the places that should be left blank are honestly left blank. Turn these three things into stages on a pipeline that can be executed, measured, and regression-tested, and AI writing can shift from an output that “looks pretty good” into a productivity tool a team can genuinely rely on.
