Long-Running AI Agent Harness Design: From Scaffolding to a Self-Closing Loop
An engineer's field notes on designing the harness — the scaffold around a large language model — for long-running autonomous coding agents. It argues that a harness should be treated not as a permanent patch for the model's weaknesses but as a provider of verifiable ground truth, and it separates capability compensation from execution-environment interfaces. Using an ontology-based knowledge graph extraction and visualization system as a deliberately hard yet verifiable test case, the article walks through research-first design, an eight-phase plan with executable acceptance signals, and how a plan-generate-evaluate loop can emerge within a single context when objective execution signals exist. It closes with a human-owned checklist for moving from a working demo to production — idempotency, ontology migration, quality metrics, performance and indexing, security, observability, and deployment — along with three conclusions on decomposition, ground truth, and the shifting role of the engineer.
1. First, Define the Problem: What Does the Scaffolding Actually Fill?
Engineers who have built long-running Agent applications have mostly written a layer of code like this: split a large task into subtasks, retry failed steps a few times, write key state into files, and run summary compression when the context is about to overflow. The industry often calls this “peripheral program” a Harness — roughly, the model’s “outer shell.” The reason it exists is straightforward: the stronger a model’s reasoning and execution ability, the more it resembles a capable intern who has a poor memory and never self-checks. The Harness is the supervisor who takes notes for them, watches progress, and verifies results.
Breaking these scaffolds apart one by one, you find they compensate for a few typical weaknesses:
- Limited context capacity: you need an external file system as “long-term memory,” plus chunked reading and summary compression;
- Prone to error in a single reasoning pass: you need retries, self-checks, and multi-candidate voting to correct drift;
- Insufficient patience and consistency on long tasks: you need to manually split things into phases and deliver intermediate artifacts step by step;
- No objective execution environment: the model can only “imagine” that the code runs; you need the scaffold to actually execute it, collect the errors, and feed them back.
The first three essentially compensate for capability gaps in the model; the fourth compensates for a gap in the “execution environment.” This point is critical: capability gaps will gradually be filled as models iterate, but the execution-environment gap will not disappear on its own. An agent that cannot truly run code, access storage, or trigger tests needs an external scaffold to provide “ground truth” for it, no matter how strong the model is. So the right way to ask “can we remove half of the scaffolding” is not “should we remove it,” but “which parts are capability compensation, and which are environment interfaces.”
2. Picking the Topic: Why Use Knowledge Graph Extraction to Test Long-Range Capability
To test long-range autonomous programming capability, I chose a representative engineering problem: an ontology-based knowledge graph extraction and visualization system. I chose it not because it is “hard enough,” but because it is “hard and verifiable.”
The value of this kind of system is that it does not just store data — it requires the data to be constrained by an ontology. The ontology defines “what concepts exist in the world, what relationships are allowed between concepts, and what constraints each relationship is subject to.” Once these constraints are strictly enforced, the system gains the ability to improve itself: newly extracted triples can be validated automatically, conflicts can be detected, gaps can be located, and the ontology itself can evolve as the corpus grows.
From an engineering perspective, it strings together at least five capability domains:
- Semantic understanding: parse the ontology definition and understand classes, properties, relationships, and cardinality constraints;
- Information extraction: identify entities and relationships in unstructured text, and disambiguate and normalize them;
- Data modeling: map the ontology to nodes, edges, and properties in the graph store, and design indexes and uniqueness constraints;
- Engineering interfaces: provide CRUD for the ontology, extraction services, graph query services, and format export;
- Interactive presentation: present query results as force-directed graphs, adjacency lists, and property panels, supporting interactive exploration.
If any link fails, the whole system cannot be delivered. Worse, these links have a chain dependency: if the extraction is wrong, the graph data is dirty; if the ontology design is wrong, extraction validation gives false positives; if the interface contract is unstable, the frontend visualization gets reworked repeatedly. This kind of task, where “errors accumulate,” is exactly where long-range agents most easily lose control — and exactly what makes it suitable as a stress test.
3. Phase One: Let the Model Research First, Then Design
In past experience, models often start writing code the moment they get a requirement, get stuck on an unfamiliar dependency, and then go back to refactor. This time the approach was to explicitly require in the prompt “research first, architect first, then act,” and to constrain the output form.
The requirement I gave contained five parts: goal (a runnable end-to-end system), scope (ontology modeling, extraction, graph storage, query interfaces, visualization), non-goals (no user system, no multi-tenancy, no pursuit of distribution), acceptance method (test cases pass + end-to-end smoke flow works), and constraints (preferred technology choices, directory structure conventions, no introducing unverified dependencies).
The architecture layering the model then produced was roughly as follows:
┌──────────────────────────────────────────────────────────────┐│ Visualization: force-directed graph / adjacency list / ││ property panel / path query │├──────────────────────────────────────────────────────────────┤│ Service layer: ontology CRUD | extraction API | ││ graph query API | export API │├──────────────────────────────────────────────────────────────┤│ Domain layer: ontology schema engine ││ (validation / inference / versioning) ││ extraction pipeline ││ (candidate generation → disambiguation → ││ relation classification → validation) │├──────────────────────────────────────────────────────────────┤│ Storage layer: graph DB (nodes / edges / indexes) + ││ object storage (raw text) ││ relational DB (task / batch / audit metadata) │├──────────────────────────────────────────────────────────────┤│ Infrastructure: config, logging, metrics, ││ error codes, idempotency keys │└──────────────────────────────────────────────────────────────┘
What is worth emphasizing is that the extraction pipeline was split into four steps rather than “one call that directly outputs triples.” This design comes from a common lesson: when you make the model output structured results in one shot, errors appear in ways that are “plausible but impossible to locate.” Once you split it into candidate generation, entity disambiguation, relation classification, and ontology validation, each step can be measured for accuracy independently and retried independently.
At the same time, the model proactively abstracted the “ontology” into versionable configuration instead of hardcoding it into business code:
ontology:version: 1classes:Person:props: { name: string, birth_year: int }id_key: nameWork:props: { title: string, publish_year: int }id_key: titlerelations:DIRECTED: { from: Person, to: Work, cardinality: 1..n }ACTED_IN: { from: Person, to: Work, cardinality: 0..n }ADAPTED_FROM: { from: Work, to: Work, cardinality: 0..1 }constraints:- Person.birth_year: range [1800, current_year]- DIRECTED.only_source: Person
Externalizing the ontology into configuration brings three direct benefits: ontology evolution does not require code changes; validation logic can reuse the same definition; and different corpus batches can use different ontology versions for regression comparison.
The extraction interface contract was also constrained to an “evidence-traceable” form, where every triple must carry its location in the source text:
{"triples": [{"subject": {"type": "Person", "id": "Chen Kai", "surface": "Chen Kai"},"predicate": "DIRECTED","object": {"type": "Work", "id": "Not One Less", "surface": "Not One Less"},"evidence": {"doc_id": "d-1024", "span": [117, 143], "confidence": 0.91}}]}
The evidence field is not a formality. In a production environment, “why does this edge exist” must be traceable; otherwise, once data quality is questioned, the credibility of the entire graph collapses.
4. Eight-Phase Task Planning and Per-Phase Acceptance
After confirming the architecture, the model directly gave a phased execution plan in the conversation. I organized it as follows (a structured summary of the model’s self-drafted plan, for easier human review):
| Phase | Main Content | Acceptance Signal for the Phase |
|---|---|---|
| 1 | Backend skeleton, configuration and logging, error-code conventions | Service starts; health check returns normal |
| 2 | Ontology schema engine: loading, validation, versioning | Illegal ontology samples must be rejected with line-level errors |
| 3 | Extraction API: candidate generation, disambiguation, relation classification | On a fixed corpus, triple recall and precision meet target |
| 4 | Ontology CRUD and graph writing | Repeated writes produce no duplicate nodes (idempotency check) |
| 5 | Graph interchange format export (triple serialization) | Exported file can be read back by an independent parser with matching counts |
| 6 | Frontend scaffolding and interface integration | Lists, details, and error states each have corresponding UI |
| 7 | Graph visualization and interactive query | Zoom, drag, adjacency expansion, and path query all work |
| 8 | Integration tests and end-to-end smoke test | Full chain from raw-text upload to seeing an edge on the graph passes |
There is an easily overlooked engineering detail here: every phase is required to have an “executable acceptance signal,” not “looks finished.” The acceptance criterion for phase 2 is written as “an illegal ontology must be rejected,” not “the ontology engine is implemented”; phase 5 is written as “after export, it can be read back by an independent parser,” not “the export feature works.” The more objective the acceptance criteria, the higher the signal-to-noise ratio of the model’s subsequent self-checks.
The reason phase 5 emphasizes “read back by an independent parser” is that the most common failure mode for format-conversion tasks is self-consistent but non-compliant: your own export, your own parser, both ends aligned with each other, yet the file does not conform to a general specification and fails with any third-party tool. This kind of error can only be exposed by “verifying with a different, independent implementation.”
5. How the Loop Forms: Plan — Generate — Evaluate
The deepest impression this whole process left on me is: the “plan — generate — evaluate” loop was not explicitly written into code; it emerged naturally within the same context.
The model writes a piece of code, then runs it itself; if execution fails, it reads the error, locates the specific file and line, modifies it, and runs again; after unit tests pass, it moves on to the next module. It plays both the generator and the evaluator role itself. What the human needs to do is only to state the goal and acceptance criteria clearly at the very beginning, and then, over the following hours, keep supplying tokens and receiving periodic progress reports.
But an honest caveat must be added here: the loop only holds when there is an objective execution signal. In the code scenario, the compiler, unit tests, schema validation, and health checks are all “ground truth”; they will not give false feedback just because the model feels good about itself. Conversely, if the task lacks an executable signal — for example, “help me design a beautiful UI” or “evaluate whether this copy is well written” — then the model’s self-evaluation degenerates into self-confirmation, and the loop quickly spins in place. This is also why I insisted on picking a “hard and verifiable” engineering problem when choosing the topic, rather than one that “looks sophisticated but cannot be measured.”
To summarize this experience in one sentence: the first principle of scaffolding is not “patching up the model,” but “providing verifiable ground truth to the model.” The patching part will indeed shrink as model capability grows, but the verifiable part will become more important as task scale grows.
6. From “It Runs” to “Production-Ready”: What Humans Still Need to Add
Even if the model autonomously completes the main implementation, there are still several positions between a demo and production that humans must gatekeep. The following is a checklist I have repeatedly confirmed in practice:
1. Idempotency and replayability
Extraction is time-consuming, may fail, and needs to be batched. The write interface must support idempotency keys (such as a hash of doc_id + span + predicate) to guarantee that repeated runs of the same batch do not produce duplicate edges. The retry strategy must distinguish “retryable errors” (timeouts, rate limiting) from “non-retryable errors” (ontology validation failures).
2. Ontology migration
The ontology will definitely evolve. Adding relationships, modifying cardinality constraints, and splitting or merging classes all require migration scripts and rollback paths. My approach is to write the ontology version number into each data item’s metadata, so that any query can trace “which ontology version this edge was extracted under.”
3. Quality metrics, not feelings
An extraction system must have quantitative metrics: triple-level precision, recall, entity disambiguation error rate, and ontology validation rejection rate. Without a baseline, you cannot judge whether a model or prompt change is an improvement or a regression. Building a small, stable golden test set is the prerequisite for all subsequent iterations.
4. Performance and indexing
The bottleneck of graph queries is usually “not building indexes for high-frequency query paths” and “N+1 style per-row queries.” The visualization frontend is equally prone to pitfalls: once the node count reaches the thousands, blindly rendering the whole graph freezes the page, so you need to filter by degree, expand neighborhoods on demand, and throttle layout computations.
5. Security boundaries
Text extraction is essentially processing untrusted input. You must guard against prompt injection (instructions like “ignore the above” embedded in the corpus), limit extraction length, parameterize database writes, and keep the model service’s credentials in a secret manager rather than in code constants. All extraction and write actions should leave audit logs.
6. Observability
Every extraction batch should record elapsed time, token consumption, failure rate, and the proportion rejected by ontology validation. These metrics both locate problems and serve as the basis for cost accounting.
7. Handling production load
A truly long-running system usually needs the service containerized and deployed on a container platform, with object storage carrying the raw text and exported files, with logging and monitoring/alerting covering error rates and latency, and with a configuration center managing ontology version switches. On cloud platforms like Baidu AI Cloud, these capabilities are mostly offered as standard, general-purpose product categories (container platform, object storage, logging service, monitoring and alerting, secret management, etc.) that can be combined directly without building your own.
7. Three Conclusions from This Practice
First, the key to long-range tasks is not “how much code the model can write,” but “whether the task can be broken into phases with objective acceptance signals.” The quality of the decomposition determines the quality of the loop.
Second, whether the “plan — generate — evaluate” loop holds depends on whether the execution environment provides ground truth. Scaffolding will not disappear; its focus just shifts from “compensating for capability” to “providing verification.”
Third, human value moves up rather than being replaced. It moves up from “writing code line by line” to “defining goals, setting boundaries, designing acceptance criteria, establishing quality baselines, and holding the security line.” The reason this autonomous development over a few hours could be delivered is precisely that the up-front definition of goals and acceptance criteria was clear enough; if the requirement had been vague, the same model would only have produced a pile of unverifiable code faster.
For teams preparing to try long-range agents, my advice is: start with a “hard and verifiable” task, get the executable signals in place first, and only then talk about the degree of autonomy. Reverse the order, and what you get is often just automation that looks busy.
