Putting Agentic Coding to the Test: Building a Vector Database from Scratch with LLM Agents

1. Why “Building a Vector Database from Scratch” Is a Good Test of Agentic Coding

Agentic coding is fundamentally different from traditional code completion: the model has to drive a task forward autonomously through a “plan → code → run → observe → fix” loop, rather than passively responding to single-turn prompts. Measuring this capability with “can it write a function” is not enough; the real question is “can it deliver a complete, testable, runnable system.”

A vector database was chosen as the benchmark task for four reasons:

1. Enough technical depth. A usable vector database has to cover vector ingestion and validation, index construction, approximate nearest-neighbor search (graph-based or clustering/bucket-based designs), multiple distance metrics (cosine, inner product, Euclidean), persistence, and metadata filtering. That spans data structures, numerical algorithms, storage, and interface design—a comprehensive checkup of overall backend engineering ability.

2. Objective acceptance criteria. Retrieval quality is measurable with hard metrics such as recall@k, throughput, and P99 latency. There is no vague “it looks like it runs” middle ground, which effectively prevents demo-grade code from passing as a complete implementation.

3. Abundant public material, demanding engineering details. The core principles are well covered by papers and tutorials, but details like race control between concurrent writes and index updates, memory layout, and bulk import force the model to genuinely understand and combine knowledge rather than recite it.

4. A manageable workload. The task can be completed within a limited number of iterations, yet it is large enough to expose context drift and consistency problems in long-horizon work.

2. Methodology: Unified Constraints and Evaluation Dimensions

To keep the findings reproducible, these constraints were fixed before testing:

• One programming language and runtime for all runs; calling any existing vector search implementation was forbidden, and the core index had to be built from scratch.
• A single requirements document, spelling out the interface contract, quantified metrics, and acceptance checklist.
• The same iteration count and execution budget for every run, so results would not be skewed by “throwing more time at it.”
• Only general-purpose tooling—compiler, test framework, benchmark harness—with no human hints during execution.

Runs were scored on six dimensions:

1. Requirement understanding and task decomposition — does the run produce an architecture description, module boundaries, and a milestone plan before writing code, or does it dive straight in?
2. Algorithm correctness — are index construction and search results correct, and does recall meet the bar?
3. Engineering quality — are exception handling, logging, configuration management, and code layering done properly?
4. Autonomous verification — can the run write and execute its own tests and trace problems from failure results?
5. Iterative repair — how well does it self-correct compile errors, failing tests, and missed performance targets?
6. Long-horizon consistency — do interface contracts stay aligned with the early design after many rounds of iteration?

3. Key Observations During Execution

Phase 1: Planning and design. The better runs open with a concise architecture document: the data model, a layered structure across storage, index, and interface layers, and acceptance criteria for each milestone. The quality of this step largely determines what follows—runs that planned well showed noticeably less rework later on, while runs that skipped planning and started coding often had to tear things down and start over during integration.

Phase 2: Implementation. The most common failure mode here is “hallucinated dependencies”: citing function signatures that don’t exist, or assuming an interface behaves a certain way without evidence. With off-the-shelf retrieval libraries banned, these problems cluster in the compile and test stages—and what’s really being measured is whether the model can correct itself based on the error messages.

Phase 3: Testing and repair. Many runs cover only the happy path—insert a few vectors, run a query, compare results. Edge cases such as querying an empty store, writing duplicate IDs, dimension mismatches, and recall degradation as the dataset grows are typically only addressed when the acceptance checklist explicitly requires them.

Phase 4: Performance optimization. Once “functionally usable” is reached, most runs stall. Whether the model proactively analyzes bottlenecks (for example, the linear complexity of a full brute-force scan) and introduces an approximate retrieval structure to trade accuracy for speed is the dividing line between “works” and “works well.”

4. Typical Risk Points

1. Hallucinated interfaces and dependency misuse — calling non-existent functions or misreading interface semantics. Compile-time problems surface quickly; semantic misuse is subtler and has to be caught by tests.
2. Textbook implementations — porting pseudocode straight from papers while ignoring engineering constraints (such as concurrent index updates during writes), producing an outcome where “the algorithm is correct but the system is unusable.”
3. Long-horizon goal drift — as iterations accumulate, interface naming becomes inconsistent and inter-module contracts quietly drift away from the original design; periodic reviews are needed to pull things back on track.
4. Inflated metrics — the model may self-test with a dataset that is too small or parameters that are too loose, making recall numbers look good. Acceptance must be re-verified on an independent dataset.

5. What the Acceptance Scaffold Looks Like

# Minimal interface contract (illustrative): the agent implements the internal index itself
class VectorStore:
    def add(ids, vectors) -> None              # Batch write
    def search(query, top_k, filters) -> list  # Nearest-neighbor search + metadata filtering
    def persist(path) -> None                  # Persist a snapshot
    def load(path) -> None                     # Restore from a snapshot

# Recall acceptance (illustrative): re-verified against an independent benchmark dataset
def recall_at_k(ground_truth, results, k):
    top_k = set(results[:k])
    return len(top_k & set(ground_truth)) / len(ground_truth)

Prepare three benchmark datasets—small (easy to debug), medium (validates algorithm behavior), and large (exposes performance bottlenecks)—and fix the random seeds so results stay comparable across runs.

6. From Evaluation to Production: Engineering Recommendations

1. Front-load the acceptance criteria. Write recall, latency, and concurrency targets into the very first requirements document; the more precise the metrics, the more convergent the output.
2. Drive the agent with tests. Requiring “tests first, implementation second, fix on failure” sharply reduces wasted iterations and gives acceptance an objective basis.
3. Insert human checkpoints. Review at three points—architecture design, core algorithm selection, and final acceptance—and let the agent run autonomously everywhere else. That balances efficiency and quality.
4. Manage context and delivery cadence. Break complex work into milestone-based deliveries, accumulating documentation and tests at each stage, to avoid losing information in a single overly long session.
5. Lean on cloud services for the periphery. Keep the core logic self-built, but use your cloud provider’s general-purpose products for the surrounding infrastructure—object storage, container platforms, logging services, and monitoring and alerting, such as those offered by Baidu AI Cloud—so engineering effort stays focused on the part that actually differentiates the system.
6. Never skip process gates. Agent-generated code goes through the same branch management, code review, and CI gates as anyone else’s; manage it like a “high-output but needs mentoring” new teammate.

7. Conclusion

• In pure backend scenarios, the current generation of LLMs is already capable of 0-to-1 prototyping through agentic coding: architecture, core algorithms, interfaces, and tests can all be brought up with light human involvement.
• A clear gap remains before “ship without review”; the weak spots concentrate in long-horizon consistency, proactive performance optimization, and completeness of edge-case handling.
• The right positioning is an extremely efficient pair programmer—humans own requirement definition, acceptance criteria, and key design decisions, while the agent carries the high-frequency labor of coding, testing, and repair. A clean division of labor between human and machine is what turns this wave of capability into real engineering productivity.