Building and Retrieving a Person-Centric Knowledge Base from Unstructured Profiles

This article starts from a typical person-profile text and discusses how to convert unstructured narrative into engineering-usable entity knowledge. The discussion covers problem definition, data model design, the processing pipeline, retrieval and recall, quality evaluation, operations and governance, common pitfalls, and a closing summary.

1. Problem Definition: From a Paragraph to a Queryable Table

The raw input is often just natural language, such as a person’s full name, date of birth, place of birth, and occupation. This kind of text has four typical characteristics:

1. Each record contains little information, but the number of instances can be very large, so manual curation cost grows linearly with scale.
2. Expressions are highly free. For example, “born on [date] in [place]” and “native of [place], born in [year]” express the same semantics.
3. The same entity may have multiple references: full name, common name, abbreviation, and transliteration variants.
4. Attributes may change over time. If only the current value is stored without versioning, historical queries cannot be traced.

Therefore, the engineering goal is not simply to store the paragraph, but to extract a set of entity-attribute triples and give them a unique identifier, source evidence, and change history.

2. Data Model Design: Separate Entities, Attributes, and Sources

A three-layer structure is recommended to avoid coupling facts with sources in the same row.

Layer 1: Entity master table. Each entity has a stable identifier. Once generated, an identifier should not be reused.

CREATE TABLE entity (
  entity_id      BIGINT PRIMARY KEY,
  canonical_name VARCHAR(200) NOT NULL,
  aliases        JSON NOT NULL,        -- aliases, short names, transliteration variants
  entity_type    VARCHAR(32) NOT NULL, -- person / org / place / work
  first_seen_at  TIMESTAMP NOT NULL,
  updated_at     TIMESTAMP NOT NULL
);

Layer 2: Attribute table. Attributes are expressed by type, value, and value-domain semantics. Date values should use a unified ISO 8601 representation, and coordinates should be stored separately from place names.

CREATE TABLE entity_attribute (
  attr_id     BIGINT PRIMARY KEY,
  entity_id   BIGINT NOT NULL,
  attr_type   VARCHAR(64) NOT NULL,  -- birth_date / birth_place / occupation
  value_text  VARCHAR(500),
  value_date  DATE,
  value_json  JSON,
  valid_from  DATE,
  valid_to    DATE,                  -- NULL means currently valid
  confidence  DECIMAL(5,4) NOT NULL
);

Layer 3: Evidence table. Every attribute must be able to answer: which text did it come from, when was it extracted, and which process confirmed it?

CREATE TABLE evidence (
  evidence_id   BIGINT PRIMARY KEY,
  attr_id       BIGINT NOT NULL,
  source_uri    VARCHAR(1000),
  source_offset JSON,               -- character offsets in the source text
  quote_text    VARCHAR(1000),
  extractor_ver VARCHAR(32),
  created_at    TIMESTAMP
);

There are three key design principles.

First, attributes may change, but primary keys remain stable. Updating an occupation should not cause the entity to be regenerated.

Second, validity intervals should be explicit. The valid_from and valid_to fields make the current value a range query result rather than an overwrite.

Third, evidence should not be physically deleted. Even if extraction is later judged to be incorrect, the record should be marked as invalid rather than deleted, so auditability and model iteration are preserved.

3. Processing Pipeline: From Text to Triples

The pipeline is best split into six independently scalable stages. Stages should be decoupled through a message queue so that retry handling and backlog observation can be done properly.

def pipeline(raw_text: str) -> list[dict]:
    text   = normalize(raw_text)          # 1 Normalization: full-width/half-width, whitespace, variant characters
    chunks = sentence_split(text)         # 2 Sentence splitting: retain character offsets
    spans  = recognize_entities(chunks)   # 3 Entity recognition: return types and positions
    links  = link_to_kb(spans)            # 4 Entity linking: map to existing entity_id
    trips  = extract_relations(chunks, links)  # 5 Relation and attribute extraction
    return validate(trips)                # 6 Validation: rule-based and model-based checks

Each stage has its own failure modes and needs separate fallback designs.

The normalization stage handles the most common dirty-data sources. For example, date formats such as “March 30” and “03/30” must be unified into the same internal representation before being written to storage.

The sentence-splitting stage must preserve original character offsets. Otherwise, the evidence located in the third stage cannot be traced back to the source text.

The entity recognition stage should prioritize high-recall strategies for person, place, and time entities. Precision issues can be handled by later validation rather than discarding candidates during recognition.

The entity linking stage is usually the most error-prone part of the entire pipeline. Homonyms, the same person with multiple transliterations, and ambiguous references are typical difficulties. In practice, a combined score based on name similarity and attribute consistency is commonly used, with a manual review queue for low-score samples.

def link_score(name_sim: float, attr_agree: float) -> float:
    # Attribute consistency is weighted more heavily:
    # relying only on name similarity easily leads to incorrect linking
    return 0.35 * name_sim + 0.65 * attr_agree

Candidate triples should not be written directly into the main database. A better approach is to place them in a staging area first. They should pass both rule-based validation and manual sampling before being promoted to production data. Rule-based validation may include checks such as whether dates are valid, whether place names belong to a known administrative-region set, and whether occupations belong to a controlled vocabulary.

4. Retrieval and Recall: Support Both Exact and Fuzzy Queries

Structured storage solves precise queries, while inverted indexes solve text matching. Both must coexist.

• Exact path: Query the database directly by entity identifier or attribute type, for example, “retrieve all person entities whose birth date falls within a given range.”
• Fuzzy path: Build an inverted index over entity names, aliases, and attribute text, supporting prefix matching, pinyin matching, and typo-tolerant matching.
• Hybrid path: First use the inverted index to recall candidates, then filter and rank with structured conditions, balancing coverage and accuracy.

def search(keyword: str, limit: int = 20):
    candidates = inverted_index.match(keyword, top_k=200)   # high recall
    ids = [c.entity_id for c in candidates]
    rows  = db.query("SELECT * FROM entity WHERE entity_id = ANY(%s)", ids)
    return rerank(rows, keyword)[:limit]                    # fine-grained ranking

High recall, lower precision, and then reranking are a common paradigm for this type of retrieval system. Loosening the recall threshold and moving filtering later is usually easier to tune than doing the reverse.

5. Quality Evaluation: Replace Subjective Judgment with Quantifiable Metrics

The evaluation system should cover at least four dimensions.

1. Extraction: Attribute-level precision, recall, and F1. These should be calculated separately by attribute type so that averages do not hide weaknesses.
2. Linking: Entity-linking accuracy and misattachment rate. Misattachment usually has a larger downstream impact than lower accuracy.
3. Consistency: The proportion of conflicting attributes from multiple sources for the same entity, and whether such conflicts are correctly marked for review.
4. Timeliness: Latency percentiles from new data generation to availability in storage.

In addition to offline metrics, a manually labeled gold set should be maintained and expanded as versions evolve. Every rule change or model change should be compared against the gold set to prevent “fixing one case while breaking ten cases.”

6. Operations and Governance: Treat the System as a Long-Term Asset

• Object storage preserves raw text and evidence snapshots, ensuring replayability and traceability.
• A message queue decouples stages, absorbs traffic spikes, and provides retry and dead-letter channels for failed messages.
• A container platform hosts extraction and validation services, scaling them independently by stage. Recognition workloads usually consume more compute than validation workloads.
• A logging service records the processing path and latency of each record, making it easier to trace back by entity identifier.
• Monitoring and alerting should focus on backlog volume, failure rate, end-to-end latency, and conflict rate. A sudden increase in conflict rate often indicates that the format of an upstream data source has changed.

If the goal is to reduce the cost of building and maintaining infrastructure, these five capabilities can be provided by general managed cloud services rather than self-built clusters. At the governance level, three things must be clarified: data retention periods, access control for sensitive attributes, and a review process for rule changes. A knowledge base lacking these three items often degrades into an untrustworthy data swamp within about six months.

7. Common Pitfalls

• Premature ingestion: Skipping staging and validation allows incorrect data to spread. Cleaning up contaminated data usually costs much more than preventing it.
• Over-normalization: Forcing place names or occupations into a small set of enumerated values makes new values impossible to represent.
• Ignoring versions: Storing only current values prevents the system from answering how a record was historically represented.
• Optimizing only recall: If recall increases but precision collapses, the returned results become unusable.

Conclusion

Turning a short person profile into an engineering asset is not mainly about how complex the model is. The key is whether the data model can evolve, whether the pipeline is observable, and whether quality metrics can support regression testing. First, design the three-layer entity-attribute-source structure and the evidence-retention mechanism. Then, introduce extraction and validation capabilities stage by stage. Finally, use the gold set and monitoring metrics to drive long-term iteration. This methodology can be applied to other entity-centric knowledge scenarios, such as organizational structures, product catalogs, and geographic information systems.