Building a Knowledge Graph Extraction System in Three Hours with an LLM Agent
This article explains why knowledge graph extraction is a strong test of AI coding agents, how a long-horizon agent can internalize the plan-generate-evaluate loop, and a four-stage method for delivering an end-to-end pipeline from schema design to visualization.
Why knowledge graph extraction is a good test
Knowledge graph extraction is a typical long-horizon, complex engineering task. It is a useful way to measure whether an AI coding agent can ship a real system, because the pipeline spans multiple stacks. A failure in any stage breaks the whole flow:
- Ontology design. You first define a domain schema, such as entity types and relation constraints for “movie–director–actor.” The ontology sets the graph’s expressiveness and is the foundation for everything that follows.
- Information extraction. From unstructured text you identify entities and extract relations. This involves named entity recognition, relation classification, and coreference resolution, plus ambiguity, aliases, and noisy text.
- Graph modeling and storage. Extracted results must land in a graph database. You need node labels, properties, edge direction, and indexes so query performance does not collapse later.
- Visualization and interactive query. The frontend should render the graph, support filters, and expand nodes so users can actually use the system.
These stages are tightly coupled: ontology changes ripple into extraction rules, the graph model, and the UI. The work cannot be split into a few independent “write a demo” tasks; it has to be delivered as one complete system.
Traditional development vs. an agent-owned closed loop
In the past, this kind of system usually needed a human-built engineering scaffold: write a planning document, have the model generate code by module, then run tests, collect errors, and feed them back—repeat. That is the familiar plan–generate–evaluate loop, which developers used to encode explicitly as code or a workflow.
In this practice, a key change appeared: when the model’s long-horizon reasoning and coding ability are strong enough, that loop can live inside the model’s own execution. The developer only needs a clear goal and acceptance criteria up front. The agent then writes code, runs tests, finds defects, and self-corrects in the same context until a runnable system is delivered.
The run here lasted about three hours, used nearly 800,000 tokens, and delivered a fully automatic path from text input to graph visualization. That duration and token use suggest the model was doing project-level development, not autocomplete-style coding.
Four stages of a long-horizon task
From this practice, the agent’s process can be summarized in four stages. The same method can be reused on other projects.
Stage 1: Goal definition and constraints
Start with a structured brief, not a vague one-liner. Example skeleton:
Goal: Build a knowledge graph extraction and visualization systemOntology requirements:- Entity types: movie, director, actor, genre- Relation types: directed, acted_in, belongs_to- Each entity type must have a unique identifier and at least one core propertyAcceptance criteria:- Extraction quality passes manual spot checks- Graph database supports multi-hop queries- Frontend supports click-to-expand nodes and relation filtersDeliverables: runnable complete code + startup instructions
Clear acceptance criteria matter: they act as a built-in evaluator. Without them, the agent tends to spin on local optimizations.
Stage 2: Ontology first
Have the agent produce an ontology document and schema validation logic before writing extraction code. In this run, the model first generated a JSON ontology and a validity-check function; only after that was it allowed to write extraction code. That human checkpoint on design avoids baking a bad schema into downstream modules.
Stage 3: Extraction pipeline and graph ingest
For extraction, the agent designed a two-stage strategy: coarse entity recognition, then relation alignment and disambiguation. For ingest it used a standard graph model—nodes with properties, typed edges, indexes on high-frequency query paths. It wrote unit tests and ran several self-check rounds, which is the internalized self-evaluation loop.
Stage 4: Visualization and end-to-end integration
The frontend used a graph rendering library with force-directed layout, node expansion, and relation filters. The last step was end-to-end: feed real text and check data consistency from extraction to display. This stage surfaced the most issues and most strongly tested cross-module debugging.
Reusable lessons
- Goal clarity drives how long the agent can work autonomously. The more structured the brief, the more rounds it can run without human intervention.
- Put acceptance criteria first. The evaluator need not be explicit code, but the criteria must be stated at start or the loop has no convergence direction.
- Manually review high-leverage artifacts such as ontology/schema. They shape the whole system; one error is amplified downstream.
- Budget tokens. Cost on long-horizon work comes mainly from self-correction rounds. Staged acceptance cuts wasted iteration.
- Do not assume full automation. Human checkpoints at stages 2 and 4, at small cost, made overall quality more predictable.
Closing
This three-hour run supports a trend: as models move from single-turn code generation to autonomous delivery of long-horizon tasks, problems that once needed heavy engineering scaffolds are being absorbed by in-model planning and self-evaluation. For developers, the core skill is shifting from “how to orchestrate the model” to “how to define the problem, design acceptance criteria, and hold the key design points.” Get those three right, and the agent can be a genuine engineering delivery partner.
