画板 1

    Historical Figure Record Governance: From Information Extraction to Audit

    5 hr ago0 views

    Biographical prose reads smoothly, but systems cannot use it to answer questions such as who held which position in a given year, whether dates and events contradict each other, or whether name variants from different sources refer to one person. This article turns that gap into five testable governance goals, then walks through a minimum data model (person entities, fact triples, and a separate table for evaluations), a seven-step extraction pipeline with runnable Python time-normalization code and unit tests, entity alias merging, multi-source conflict detection, configurable and calibrated confidence scoring, timeline ordering and rendering rules, PostgreSQL schema and retrieval SQL with evidence, handling of sensitive public information, and a four-phase rollout with exit criteria. It closes with five common pitfalls and an FAQ. The same governance skeleton applies to any record with multiple sources, versions, and relationships, including institutions, events, and business entities.

    How to Govern Historical Figure Records: From Information Extraction to Audit

    Why raw biographical text can’t be used as-is: define acceptance criteria first

    A complete biography reads smoothly, but a system cannot answer questions like these: Did this person hold a given position between 1971 and 1986? Do their birth and death dates, education, and appointments contradict each other? Can aliases and position titles from different sources be merged into one person?

    So the governance goal should be written as five testable acceptance criteria, not as “tidy up the text”:

    1. Entity-based: People, places, organizations, positions, and events are addressable, and alias-merge accuracy is measurable.
    2. Relation-based: Person–position, person–kin, and person–organization relationships can be queried.
    3. Time-aware: Every fact carries a value, a precision, and a validity interval, so facts can be sorted and compared.
    4. Provenance-aware: Every fact is bound to a document ID, a paragraph offset, a source tier, and a confidence score.
    5. Service-oriented: Retrieval, Q&A, the graph, and review all use the same fact store, and every answer can point back to its evidence.

    What does the minimum data model for a person record look like?

    Person entity: separate the display name from the retrieval name

    1. {
    2.   "entity_id": "person_0001",
    3.   "entity_type": "Person",
    4.   "display": "Person A",
    5.   "original_variants": ["Person A", "A. (romanized form)", "General A"],
    6.   "birth": {"date": "1951-07-03", "precision": "day", "place_id": "place_0007", "confidence": 0.96},
    7.   "death": {"date": "2014-10-04", "precision": "day", "place_id": "place_0012", "confidence": 0.94},
    8.   "nationality": "Country X",
    9.   "education": [
    10.     {"institution_id": "org_0021", "faculty": "School of Law", "start": null, "end": null, "precision": "unknown"}
    11.   ]
    12. }

    Key points:

    • display is used only for display.
    • original_variants take part in matching.
    • Empty values must be written explicitly as null, not omitted. Otherwise, later sorting and statistics will treat “unknown” and “none” as the same thing.

    Fact triples: one primary key per fact

    1. {
    2.   "fact_id": "fact_10001",
    3.   "subject_id": "person_0001",
    4.   "predicate": "held_position",
    5.   "object_type": "Position",
    6.   "object_value": "Head of State of Country X",
    7.   "valid_from": "1971-04",
    8.   "valid_to": "1986-02-07",
    9.   "from_precision": "month",
    10.   "to_precision": "day",
    11.   "place_id": "place_0012",
    12.   "source_id": "doc_1001",
    13.   "evidence_segment_id": "seg_0004",
    14.   "confidence": 0.88,
    15.   "status": "verified",
    16.   "created_at": "2026-08-26T10:00:00Z"
    17. }

    With this design, if you later find that a term’s start month is wrong, you only change that one fact and create a new version. There is no need to rewrite the whole record.

    Facts and evaluations must live in separate tables

    • person_facts stores objective statements.
    • person_evaluations stores evaluations, and must always carry author, published_at, stance, and source_id.

    Before deciding which table a sentence belongs to, classify it as a statistical description, a factual statement, or a source’s opinion. For example, a sentence such as “the national economy was in difficulty during their time in office” needs this classification first. Otherwise, opinionated judgments will end up mixed into retrieval results.

    From text to fields: how to put information extraction into practice

    The seven-step pipeline

    Clean and denoise → split into paragraphs and generate a segment_id plus character offsets → identify candidate entities → extract relation candidates → normalize time and place → bind evidence → write to the fact store (unresolved items go to a manual review queue).

    Time normalization: a minimal, runnable implementation

    The code uses only the Python 3.10 standard library. Its inputs are the raw expression and a context (current year, birth year). When parsing fails, it raises an exception rather than returning the string "unknown". That keeps dirty values out of the fact table.

    1. import re
    2. from datetime import date
    4. PRECISION_RANK = {"day": 3, "month": 2, "year": 1, "unknown": 0}
    6. class TimeParseError(ValueError):
    7.     """Raised for unparseable time expressions; record them in time_review_queue, not in the fact table."""
    10. def normalize_time(text, context=None):
    11.     # Matches the Chinese date expressions found in the source documents.
    12.     ctx = {"current_year": None, "birth_year": None}
    13.     ctx.update({k: v for k, v in (context or {}).items() if v is not None})
    14.     if text is None:
    15.         raise TimeParseError("empty text")
    16.     s = re.sub(r"\s+", "", text.strip())
    18.     m = re.fullmatch(r"(\d{4})年(\d{1,2})月(\d{1,2})日", s)
    19.     if m:
    20.         y, mo, d = (int(x) for x in m.groups())
    21.         try:
    22.             iso = date(y, mo, d).isoformat()
    23.         except ValueError as exc:
    24.             raise TimeParseError(f"invalid date: {s}") from exc
    25.         return {"value": iso, "precision": "day", "year": y}
    27.     m = re.fullmatch(r"(\d{4})年(\d{1,2})月", s)
    28.     if m:
    29.         y, mo = int(m.group(1)), int(m.group(2))
    30.         if not 1 <= mo <= 12:
    31.             raise TimeParseError(f"invalid month: {s}")
    32.         return {"value": f"{y:04d}-{mo:02d}", "precision": "month", "year": y}
    34.     m = re.fullmatch(r"(\d{4})年", s)
    35.     if m:
    36.         return {"value": m.group(1), "precision": "year", "year": int(m.group(1))}
    38.     m = re.fullmatch(r"同年(\d{1,2})月(?:(\d{1,2})日)?", s)  # "same year" as the current year in context
    39.     if m:
    40.         if ctx["current_year"] is None:
    41.             raise TimeParseError("context.current_year required for 同年")
    42.         mo = int(m.group(1))
    43.         if not 1 <= mo <= 12:
    44.             raise TimeParseError(f"invalid month: {s}")
    45.         y = ctx["current_year"]
    46.         if m.group(2):
    47.             return normalize_time(f"{y}年{mo}月{m.group(2)}日", ctx)
    48.         return {"value": f"{y:04d}-{mo:02d}", "precision": "month", "year": y, "inferred": "context_year"}
    50.     m = re.fullmatch(r"(\d{1,3})岁", s)  # age
    51.     if m:
    52.         if ctx["birth_year"] is None:
    53.             raise TimeParseError("context.birth_year required for 年龄表达")
    54.         y = ctx["birth_year"] + int(m.group(1))
    55.         return {"value": f"{y:04d}", "precision": "year", "year": y, "derived": "age_to_year"}
    57.     m = re.fullmatch(r"(\d{1,3})年", s)   # a duration, not a point in time
    58.     if m:
    59.         return {"value": None, "precision": "unknown", "duration_years": int(m.group(1))}
    61.     raise TimeParseError(f"unsupported time expression: {text}")

    How to write the unit tests

    1. import pytest
    3. @pytest.mark.parametrize("text,ctx,expect", [
    4.     ("1951年7月3日", {}, ("1951-07-03", "day")),
    5.     ("1971年4月", {}, ("1971-04", "month")),
    6.     ("1971年", {}, ("1971", "year")),
    7.     ("同年2月3日", {"current_year": 1986}, ("1986-02-03", "day")),
    8.     ("19岁", {"birth_year": 1951}, ("1970", "year")),
    9. ])
    10. def test_supported(text, ctx, expect):
    11.     out = normalize_time(text, ctx)
    12.     assert (out["value"], out["precision"]) == expect
    14. @pytest.mark.parametrize("text,ctx", [
    15.     ("终身", {}), ("28年", {}), ("同年2月3日", {}), ("1971年13月", {}), ("", {}),
    16. ])
    17. def test_unsupported_or_invalid(text, ctx):
    18.     with pytest.raises(TimeParseError):
    19.         normalize_time(text, ctx)

    Fallback strategy:

    • A fact whose time fails to parse is given the status needs_review and written to time_review_queue(doc_id, segment_id, raw_text, reason).
    • Such facts do not enter the timeline or the Q&A results.
    • If more than 5% of the facts in a single document fail to parse, pause ingestion of that batch and trigger an alert.

    Entity merging

    • Structure each alias as {text, type}, where type is formal, foreign_name, alias, or nickname.
    • Match in this order:
      1. Exact match on the canonical name.
      2. Normalized alias match (remove spaces, fold case, unify transliteration of foreign scripts).
      3. Semantic-similarity recall, followed by human confirmation.
    • Two different people who share a name must be told apart using context such as time, place, and organization. String similarity alone is not enough.

    How should you handle contradictions between multiple sources?

    Source tiers are only for weighting, not for single-source veto power

    • High trust: official registries, authoritative chronologies, original archival documents.
    • Medium-high trust: academic publications.
    • Medium trust: news reports and published biographies.
    • Low trust: unsigned compilations and content aggregated from the web.

    A low-trust source can point to counter-evidence against the main evidence. However, on its own it cannot overturn a fact that is already backed by a high-trust source.

    Conflict detection rules

    • Same attribute, different values: with the same subject + predicate and overlapping time intervals, object_value does not match.
    • Invalid interval: valid_to < valid_from.
    • Lower precision: new evidence is less precise than the existing fact (for example, month-level versus day-level). Mark it for review.
    • Missing information: a key predicate has no time, place, or source.
    • Confidence anomaly: a low-trust source claims a certainty of more than 0.95.

    Confidence: configurable weights plus probability calibration

    In the original formula, the terms had different units, so adding them directly was meaningless. In production, this should become a linear score followed by a sigmoid. The weights should go in a configuration file, not be hardcoded in the code:

    1. z = w0
    2.   + w_source   * source_level_score      # high=1.0, medium-high=0.75, medium=0.5, low=0.2
    3.   + w_support  * cross_source_support    # independent sources / 3, capped at 1
    4.   + w_temporal * temporal_consistency    # no conflict with the confirmed timeline = 1, conflict = 0
    5.   + w_clarity  * entity_clarity          # unique entity match = 1, ambiguous = 0.3
    6.   - w_contra   * contradiction_count     # deduct 0.8 per conflict
    7.   - w_vague    * vagueness_score         # proportion of hedging words such as "about", "reportedly"
    8. confidence = sigmoid(z)

    Calibration and thresholds

    • Use 300–500 human-annotated facts as a validation set.
    • Compute the Brier score and the reliability curve on it.
    • If facts with a model confidence of 0.8 are actually correct only 60% of the time, recalibrate with isotonic regression.
    • There is no universal threshold. Instead, pick the target precision for the use case (for example, 0.95 for Q&A), then take the lowest threshold on the validation set that meets it as the release cutoff.
    • The example value 0.88 only shows the field format. Do not use it directly in production.

    How should you sort and render the timeline?

    Each event has these fields: event_id, subject_id, event_type, title, start, end, from_precision, to_precision, place_id, related_entity_ids, source_id, confidence, and conflict_flag.

    Define the event types as a fixed enum: birth, education, appointment, departure, migration, kin, social, political, medical, and death.

    Rendering rules:

    1. Primary sort key: (start_year, start_month, start_day). When keys are equal, more precise dates come first (day > month > year), then higher confidence comes first.
    2. Padding: fill missing precision with the placeholder 00, and show the date with lower precision in the display, for example 1971-04 (month-level).
    3. Missing start date: do not place the event in the timeline. Put it in an “unknown timing” group and show it greyed out.
    4. Conflicts: show a warning marker on events that have conflicts. Expanding the marker shows the competing candidate values and where each one comes from.
    5. Derived times: times derived from age mark themselves as “inferred”.

    Example output:

    1. 1951-07-03   Birth · place_0007 · source: doc_1001 (high) · confidence 0.96
    2. 1971-04 (month-level)  Took office in a position · source: doc_1001 (medium-high) + doc_1042 (medium) · 0.88
    3.              ⚠ Conflict: another source records 1971-05 (doc_2087, medium trust)
    4. 1986-02-07   Left office and left the country · source: doc_1103 (high) · 0.93
    5. [Undated]    A public appearance · original text says "several years later" · awaiting manual verification

    How should you design storage and retrieval so it scales?

    Keep the following principles:

    • The raw text layer is immutable, with a versioned snapshot for each revision.
    • Every structured fact must be traceable back to a paragraph offset.
    • The fact layer uses relational tables plus search indexes.
    • Aliases live in their own table so that merge-based search is supported.
    1. -- PostgreSQL syntax
    2. CREATE TABLE person_entities (
    3.     entity_id    TEXT PRIMARY KEY,
    4.     entity_type  TEXT NOT NULL,
    5.     display_name TEXT NOT NULL,
    6.     status       TEXT NOT NULL DEFAULT 'active'
    7. );
    9. CREATE TABLE person_aliases (
    10.     entity_id TEXT NOT NULL REFERENCES person_entities(entity_id) ON DELETE CASCADE,
    11.     norm_text TEXT NOT NULL,
    12.     alias_type TEXT NOT NULL CHECK (alias_type IN ('formal','foreign_name','alias','nickname')),
    13.     PRIMARY KEY (entity_id, norm_text)
    14. );
    16. CREATE TABLE fact_triples (
    17.     fact_id      TEXT PRIMARY KEY,
    18.     subject_id   TEXT NOT NULL REFERENCES person_entities(entity_id) ON DELETE CASCADE,
    19.     predicate    TEXT NOT NULL,
    20.     object_type  TEXT NOT NULL,
    21.     object_value TEXT,
    22.     valid_from   TEXT,
    23.     valid_to     TEXT,
    24.     from_precision TEXT NOT NULL DEFAULT 'unknown',
    25.     to_precision   TEXT NOT NULL DEFAULT 'unknown',
    26.     place_id     TEXT,
    27.     source_id    TEXT NOT NULL REFERENCES source_documents(source_id),
    28.     evidence_segment_id TEXT,
    29.     confidence   REAL NOT NULL CHECK (confidence BETWEEN 0 AND 1),
    30.     status       TEXT NOT NULL CHECK (status IN ('candidate','verified','needs_review','retracted')),
    31.     CHECK (valid_to IS NULL OR valid_from IS NULL OR valid_to >= valid_from),
    32.     created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
    33. );
    35. CREATE INDEX idx_fact_subject_pred_time ON fact_triples (subject_id, predicate, valid_from);
    36. CREATE INDEX idx_fact_object_value       ON fact_triples (object_value);
    37. CREATE INDEX idx_fact_status_conf        ON fact_triples (status, confidence);
    38. CREATE INDEX idx_alias_norm              ON person_aliases (norm_text);
    39. CREATE INDEX idx_fact_segments           ON fact_triples (evidence_segment_id);

    This retrieval SQL is ready to run, and returns results that can be passed straight to the Q&A layer:

    1. SELECT f.fact_id,
    2.        f.predicate,
    3.        f.object_value        AS position_name,
    4.        f.valid_from, f.valid_to,
    5.        f.from_precision, f.to_precision,
    6.        round(f.confidence::numeric, 3) AS confidence,
    7.        d.title               AS source_title,
    8.        d.source_level        AS source_level,
    9.        seg.start_offset      AS evidence_offset,
    10.        seg.text              AS evidence_text
    11. FROM fact_triples f
    12. JOIN source_documents d ON d.source_id = f.source_id
    13. LEFT JOIN segments seg  ON seg.segment_id = f.evidence_segment_id
    14. WHERE f.subject_id = $1
    15.   AND f.predicate IN ('held_position', 'resigned_position')
    16.   AND f.status = 'verified'
    17.   AND f.confidence >= 0.75
    18. ORDER BY f.valid_from ASC, f.confidence DESC
    19. LIMIT 50;

    Search uses this fallback order:

    1. Formal name
    2. Aliases and nicknames
    3. Structured filters on time, place, and organization
    4. Full-text search, as a last resort

    Results below the threshold are returned only as “leads”, clearly labeled as such. They are never mixed in with verified facts.

    How do these capabilities create business value?

    Four services share the same fact store:

    • The timeline page
    • The person relationship graph (kinship, appointments, organizations), with filters by relation type and time range
    • Fact Q&A
    • The review and correction console

    Q&A must put structured facts first, together with the original text segment, the source, and the confidence score. When there is a conflict, it should return the candidate values along with the answer. It should not generate a fluent but uncertain narrative. This is the most effective way to suppress hallucinations.

    The editorial log records each change as log_id, entity_id, field, old_value, new_value, operator, reason, source_id, reviewed, and timestamp. It is append-only and never overwrites entries.

    Handling checklist for public information on historical political figures

    Can be collected:

    • Official positions and terms of office
    • Public birth and death information
    • Public education history
    • Public family relationships: the names of parents, spouse, and children, and the relationships themselves
    • Public organizational affiliations
    • Public event participation
    • Source details and publication date

    Require manual review before being stored:

    • Relatives’ specific home addresses and contact details
    • Health details
    • Court case details
    • Any information involving minors
    • Key dates or relationships backed by only one low-trust source
    • Matching names to identities across international sources

    Never collected or modeled:

    • Private home addresses, phone numbers, ID document numbers, and bank accounts
    • Content of communications
    • Raw medical records
    • Other people’s privacy-sensitive personal information
    • Unverifiable rumors and judgmental claims

    Objection, deletion, and retraction requests:

    When an outside party asks for a correction or a deletion, freeze the external visibility of the related facts first (set their status to pending_review). Reviewers then confirm the request within an agreed time limit.

    • If the request is valid, run the correction or deletion process and keep a record of it.
    • If not, document the reason and reply to the requester.

    Source retraction:

    • When a source is proven unreliable, use its source_id to set all candidate facts it produced to retracted in bulk.
    • Then rebuild the graph edges, timeline events, and Q&A caches that depend on those facts.
    • Published content must be issued as a new version, and the page must keep a “revision note”.

    Audit requirements:

    • Changes to sensitive fields always require two-person review.
    • Audit logs are stored separately from the business database and are append-only. Their retention period is set by internal compliance policy.
    • Export interfaces leave out unverified and sensitive fields by default.

    Phased rollout and acceptance criteria

    Phase Main tasks Exit criteria (example)
    1. Archive standardization Build the document store, denoise, tag source tiers, generate segment IDs and version snapshots 100% paragraph coverage; at least 99% of facts can be traced back to an offset
    2. Modeling Define the enums for entities, relations, and predicates; define alias rules and source binding Data dictionary review passed; no duplicate meanings in the predicate enum
    3. Extraction and quality control Extract time, place, organization, kin, and appointment data; perform manual spot checks On a 200-sentence golden set: fact extraction precision ≥ 0.92, time normalization failure rate ≤ 5%, source-binding coverage ≥ 98%, and at least 90% of conflicts resolved
    4. Service rollout Build retrieval, timeline, relationship graph, Q&A, review console, and monitoring Retrieval P95 ≤ 500 ms; 100% of answers include evidence; Brier score lower than baseline after calibration; retraction drill passed

    Five common pitfalls

    1. Summarizing without structuring: you end up unable to run statistics or link related information.
    2. Recording conclusions without sources: fixing mistakes later becomes extremely costly.
    3. Mixing evaluations and facts in one table: opinionated judgments leak into retrieval results.
    4. Ignoring time precision: the timeline ends up sorted incorrectly.
    5. Treating version control as an extra feature: a single bad edit then pollutes every downstream application.

    FAQ

    Should the model use documents or relational tables?
    Use relational tables for facts and relationships, because they make constraints and interval checks easy. Store raw text and paragraphs in object storage with a document index. Don’t force one database to do everything.

    How do you measure extraction quality?
    Build a golden set of 200–500 sentences. Compute precision and recall for each predicate, and track the time-normalization failure rate separately. This stops an overall metric from hiding systematic errors in a particular field type.

    When sources conflict, which one should you trust?
    No single authoritative source decides. Combine source-tier weighting, the number of independent sources, time consistency, and confidence calibration. If you still cannot decide, keep the competing candidates and flag the conflict instead of forcing a choice.

    How do you prevent Q&A hallucinations?
    Use retrieval first and always attach evidence. Set a threshold for releasing answers, and refuse to answer when there is no evidence. Bind the answer cache to the fact version so that answers become invalid automatically when a fact is retracted.

    How do you audit version changes?
    Use an append-only log and field-level diffs. Require a reason and a source for every change, and two-person review for sensitive fields. Store audit logs separately.

    Summary

    The hard part of governing person records is not how much material you have. It is whether the data can be computed on, reused, and audited. Turning biographical text into a knowledge asset depends on four things working together: entity modeling, relationships, time handling, and provenance. You also need runnable extraction code, confidence scores you can calibrate, a retrieval layer with indexes, clear boundaries around sensitive fields, and acceptance metrics for each phase.

    This approach is not limited to historical figures. The same governance skeleton works for any record with multiple sources, versions, and relationships, including institutions, events, and business entities. The goal is to turn every piece of text into a knowledge unit that can be queried, verified, and reused.

    09040904

    Top articles