Structured Modeling and Validation of Person Knowledge Entries

1. The Real Problem: From a Paragraph to Maintainable Data

Consider a typical person entry. The following sample has been anonymized: a 20th-century French writer and critic, who also worked as a screenwriter and actor, served for a long time in the private secretariat of a senior official, wrote film and book reviews for the literary supplement of a major newspaper, and produced work across four directions: literary criticism, novels, diary-style works, and film/TV scripts. At the end of the entry, there is also a table showing “year — work — participation role.”

This kind of text is very common in information extraction engineering. It mixes four completely different types of data into one piece of natural language:

• Entity attributes: name in multiple languages, nationality, birthplace, birth and death dates, and occupation set.
• Event timeline: employment periods, editorship terms, years of performance and screenwriting, death date, and burial location.
• Work set: many-to-many relationships between works and people, with explicit role labels. The same work may involve a person both as a screenwriter and as an actor.
• Person relationships: kinship, social relationships, and collaboration relationships.

Storing all of this as one large text field may work in the short term, but it will fail in the long term: the fields cannot be counted, timelines cannot be compared, people with the same name cannot be disambiguated, and errors cannot be traced back to the source text.

Therefore, the engineering goal should be clear: decompose biography text into four layers — entities, attributes, typed events, and role-bearing relationships — and ensure that every assertion can be traced back to a source text snippet and a source identifier.

2. Data Model: Three-Layer Separation Instead of One Wide Table

2.1 Layering Principles

1. The main table should only contain stable attributes.
   Frequently changing content, such as summaries, tags, and popularity metrics, should not be placed in the main table. This avoids version conflicts.

2. The timeline should be an independent table.
   One person corresponds to N events. These N events are the foundation for consistency validation and timeline visualization.

3. Relationships should be independent and role-based.
   Work relationships must explicitly record the role, such as screenwriter, actor, or author. Otherwise, vague expressions like “participated in a film” will be misused downstream.

4. Keep the original text traceable.
   Every derived data item should be attached to raw_text and a source identifier. This is the prerequisite for review and correction, and it is also the most reliable way to prevent hallucinations when working with large language model extraction.

2.2 Schema Sketch

-- Person master table: stable attributes only
CREATE TABLE person (
    person_id      VARCHAR(32) PRIMARY KEY,
    name_zh        VARCHAR(128) NOT NULL,
    name_native    VARCHAR(256),            -- original name, multilingual
    gender         CHAR(1)      NOT NULL,   -- M/F/U
    birth_place    VARCHAR(256),
    birth_date     DATE,
    death_date     DATE,
    occupation_set VARCHAR(512),            -- normalized occupation code set
    version_no     INT  NOT NULL DEFAULT 1,
    status         VARCHAR(16) NOT NULL,    -- draft/reviewing/published/offline
    updated_at     TIMESTAMP
);
-- Timeline event table: appointments, awards, publications, performances, death, etc.
CREATE TABLE person_event (
    event_id   BIGSERIAL PRIMARY KEY,
    person_id  VARCHAR(32)  NOT NULL,
    event_type VARCHAR(32)  NOT NULL,       -- controlled enumeration
    title      VARCHAR(256),
    org_name   VARCHAR(256),
    start_date DATE,                        -- if only year is known, normalize to first day of the year and mark precision
    end_date   DATE,                        -- NULL if ongoing
    date_prec  VARCHAR(8),                  -- day/month/year/decade
    raw_text   TEXT,
    source_id  VARCHAR(256) NOT NULL
);
-- Work and relationship table: role-based many-to-many
CREATE TABLE person_work (
    person_id  VARCHAR(32) NOT NULL,
    work_name  VARCHAR(256) NOT NULL,
    work_type  VARCHAR(32)  NOT NULL,       -- literary essay/novel/diary-style work/film
    role       VARCHAR(32)  NOT NULL,       -- screenwriter/lead actor/author/co-director
    work_year  SMALLINT,
    award_name VARCHAR(128),                -- award information, nullable
    PRIMARY KEY (person_id, work_name, role)
);

One often-overlooked detail is the date_prec field. In biography text, “late 1950s,” “1962,” and “March 22, 1996” have very different precision levels. If all of them are stored as plain DATE values, downstream sorting and statistics will suffer from systematic bias.

3. Extraction Pipeline: Rules First, Models as Fallback, Human Review for Edge Cases

3.1 Five Stages

1. Chunking and field locating
   Preferably split content according to the inherent layout of encyclopedia entries: abstract paragraph, experience section, family section, works section, and tables. Layout priors can significantly improve accuracy and are more stable than extracting everything from the whole document without structure.

2. Temporal expression normalization
   Convert expressions such as “served as editor-in-chief of a journal from 1949 to 1953” or “late 1950s” into ISO 8601 intervals with precision markers.

3. Occupation and organization normalization
   Map free-text occupations to a controlled enumeration. Normalize organization names, including abbreviations, full names, and historical names.

4. Table structuring
   Person work tables often contain merged cells and cross-row notes. First restore the two-dimensional grid, then parse it according to three semantic columns: year, work name list, and role.

5. Entity linking and deduplication
   Disambiguate people with the same name using birth and death years, birthplace, and kinship relationships. If external authoritative identifiers are available, prioritize alignment by those identifiers.

3.2 Normalization Example

{
  "person": {
    "name_native": "<native name>",
    "birth_year": 1914,
    "death_year": 1996,
    "nationality": "FR",
    "occupations": ["WRITER", "CRITIC", "SCREENWRITER", "ACTOR"]
  },
  "events": [
    {"type": "GRADUATE_DEGREE", "field": "LAW", "start": "1938-01-01", "prec": "year"},
    {"type": "PRIVATE_SECRETARY", "org": "<senior official's office>", "start": "1944-01-01",
     "end": "1949-12-31", "prec": "year", "evidence": "worked in the private secretariat of ... from 1944 to 1949"},
    {"type": "EDITOR_IN_CHIEF", "org": "<monthly magazine>", "start": "1949-01-01",
     "end": "1953-12-31", "prec": "year"},
    {"type": "COLUMNIST", "org": "<newspaper literary supplement>", "start": "1954-01-01", "prec": "decade"},
    {"type": "DEATH", "start": "1996-03-22", "prec": "day"}
  ],
  "works": [
    {"name": "<literary essay>", "type": "ESSAY", "year": 1945, "role": "AUTHOR"},
    {"name": "<novel>", "type": "NOVEL", "year": 1959, "role": "AUTHOR",
     "award": "<literary award>"},
    {"name": "<film A>", "type": "FILM", "year": 1950, "role": "ACTOR"},
    {"name": "<film B>", "type": "FILM", "year": 1962, "role": "SCREENWRITER"}
  ]
}

The most important design in this structure is the evidence field. It allows an assertion, such as “served as editor-in-chief from 1949 to 1953,” to be traced back word for word. Reviewers can locate disputed points without rereading the entire text.

4. Validation Rules: From “Looks Correct” to “Machine-Verified Correct”

The most frequent errors in biography data are not missing extractions. They are extractions that are present but mutually contradictory. At minimum, the following categories of validation should be implemented.

import datetime as dt
def validate_person(p, events, works):
    errs = []
    # 1) Basic format and logic: birth/death order, plausible lifespan
    if p["birth_date"] and p["death_date"]:
        if p["birth_date"] > p["death_date"]:
            errs.append(("E_BIRTH_AFTER_DEATH", p["person_id"]))
        elif (p["death_date"] - p["birth_date"]).days > 365 * 130:
            errs.append(("E_AGE_OUT_OF_RANGE", p["person_id"]))
    # 2) Self-consistency of event intervals
    for e in events:
        if e["start_date"] and e["end_date"] and e["start_date"] > e["end_date"]:
            errs.append(("E_EVENT_RANGE_REVERSED", e["event_id"]))
    # 3) Events cannot start before birth year or after death year, except award or honorary events
    for e in events:
        if e["event_type"] in ("PRIVATE_SECRETARY", "EDITOR_IN_CHIEF", "COLUMNIST"):
            if p["birth_date"] and e["start_date"] < p["birth_date"].replace(year=p["birth_date"].year + 14):
                errs.append(("E_EVENT_TOO_EARLY", e["event_id"]))
    # 4) Mutually exclusive roles of the same type at the same organization should not overlap
    by_org = {}
    for e in events:
        if e["event_type"] == "EDITOR_IN_CHIEF":
            by_org.setdefault(e["org_name"], []).append(e)
    for org, lst in by_org.items():
        lst.sort(key=lambda x: x["start_date"] or dt.date.min)
        for a, b in zip(lst, lst[1:]):
            if a["end_date"] and b["start_date"] and a["end_date"] >= b["start_date"]:
                errs.append(("E_OVERLAP_SAME_ORG", org, a["event_id"], b["event_id"]))
    # 5) Work year must fall within the person's plausible activity period
    for w in works:
        if w["year"] and p["birth_date"] and w["year"] < p["birth_date"].year + 15:
            errs.append(("E_WORK_YEAR_IMPOSSIBLE", w["name"]))
    # 6) Required fields and enumerations
    if not p["occupations"]:
        errs.append(("E_OCCUPATION_EMPTY", p["person_id"]))
    if p["gender"] not in ("M", "F", "U"):
        errs.append(("E_GENDER_ENUM", p["person_id"]))
    return errs

In addition to machine rules, three human intervention strategies are needed:

• Low-confidence extraction: fields whose model confidence score is below a threshold should be forced into a review queue.
• Cross-source conflicts: if two sources provide different values for the same attribute, the item should require human review.
• High-impact entities: high-traffic people should be prioritized for careful review.

Everything outside these three categories should be published automatically. Otherwise, review costs will grow linearly with the number of entries.

5. Storage and Retrieval: Structured Querying and Semantic Recall Together

Person knowledge entries usually serve two access patterns:

• Precise condition filtering, such as “French writers who served as editor-in-chief between 1940 and 1960.”
• Fuzzy semantic querying, such as “writers who created diary-style works and participated in film screenwriting.”

A single storage solution is difficult to satisfy both needs. A practical design includes:

• Structured storage: use a relational database for the main table, event table, and relationship table. This ensures transactional consistency and makes validation rules executable.
• Retrieval layer: write summaries, work names, and event descriptions into a search service. Build inverted indexes to support keyword search and range filtering.
• Semantic layer: generate embeddings for summaries and key snippets. Use hybrid retrieval with keyword recall, vector recall, and fusion ranking, such as Reciprocal Rank Fusion, to handle cases like “diary-style works” that may not match literally but are semantically related.
• Raw and derived separation: store raw snapshots in object storage and store derived structured data in the database. Link the two through source identifiers so the pipeline can be rerun.
• Observability: connect extraction latency, validation failure rate, and index latency to logging and monitoring systems. A sudden drop in rule hit rate is often the first signal of upstream page layout changes.

6. Operational Loop: Turning One-Off Extraction into a Sustainable Pipeline

The recommended workflow is a six-stage loop: discovery, extraction, validation, review, publication, and feedback. Quantifiable quality gates should be defined.

Metric	Definition	Target
Field completeness rate	Non-empty required fields / total required fields	≥ 98%
Validation pass rate	Percentage of entries passing validation on first attempt	≥ 95%
Extraction precision	Percentage of correct assertions in sampled review	≥ 97%
Manual rejection rate	Percentage of rejected entries in the review queue	≤ 5%
Deduplication accuracy	Percentage of correct merge operations	≥ 99%
Traceability coverage	Percentage of assertions carrying evidence	100%

Three practical conclusions are worth emphasizing.

First, do layout parsing before semantic extraction. This is more controllable than a single end-to-end extraction pass. When errors occur, the localization cost is much lower.

Second, define controlled vocabularies early. Once occupations, organizations, and work types are allowed to become free text, downstream statistics and aggregation will fail.

Third, every assertion must carry a source text snippet and source identifier. This is essential for review efficiency, content compliance, and dispute resolution.

With this model and workflow, a biography text of several thousand words can be reliably reduced to dozens of structured assertions. These assertions can then support downstream capabilities such as timeline visualization, relationship graphs, condition-based filtering, and semantic question answering, while keeping the marginal cost of adding new entries close to zero.