Building a Vector Database from Scratch: An Agentic Coding Evaluation

I. Motivation: Building a Vector Database as an Engineering Test

This evaluation was originally planned as a video, but considering the high information density of the technical details and the need for future retrieval, it was ultimately decided to document the process in text. This test answers one very specific question: In a pure backend, greenfield scenario without legacy code burdens, how far can a large language model go when using Agentic Coding to build a vector database from scratch?

Here is the upfront conclusion: When requirement boundaries are clear and acceptance criteria are quantifiable, frontier LLMs can end-to-end complete the vast majority of skeleton design and functional implementation, including project scaffolding, data structures, API contracts, and unit tests. However, in the areas of boundary correctness for approximate retrieval algorithms and performance tuning, human review remains indispensable. It must be stated that all observations below come solely from this specific test and do not constitute a universal conclusion about any model’s capabilities.

II. Why This Task is Ideal for Evaluating Agentic Coding

A good “test question” needs to meet four conditions simultaneously, and a vector database hits all of them:

1. High Knowledge Density: It spans algorithms (Approximate Nearest Neighbor search), data structures (Hierarchical Navigable Small World graphs, inverted buckets), API design (CRUD and query semantics), and systems engineering (persistence, concurrency, memory management). It tests both the breadth and depth of the model.
2. Completely Objective Acceptance: Recall rate, latency, and correctness can all be measured numerically, avoiding subjective judgments like “the code looks like it runs.” The results of any approximate index can be reconciled against the “gold standard” of brute-force search.
3. Pure Greenfield: No legacy code, implicit conventions, or historical baggage. This eliminates interfering variables like “failing to understand the old system” and focuses purely on the model’s engineering capabilities.
4. Close to Real-World Business: Scenarios like Retrieval-Augmented Generation (RAG), semantic deduplication, image search, and recommendation retrieval all use vector search. Test conclusions can easily be transferred to real-world projects.

III. Test Design: Defining the “Syllabus” First

The upper limit of Agentic Coding effectiveness is often determined the moment the task brief is written. This task brief contained four parts:

• Requirement Boundaries: Clearly defined implementation scope—in-memory vector storage, three distance metrics (Cosine, Inner Product, Euclidean), Top-K search, scalar filtering, and disk persistence. Explicitly excluded distributed and multi-replica capabilities to prevent scope creep.
• API Contracts: Pre-defined interface signatures for insertion, deletion, query by ID, and KNN search to prevent the model from improvising interfaces that are difficult to reconcile.
• Acceptance Thresholds: All functional tests must pass; under agreed data scale and dimensions, the recall rate of the approximate index against brute-force search must not fall below a set threshold; single-query latency must meet the upper limit.
• Feedback Loop: After each iteration, acceptance tests are automatically executed, and the complete output of failed test cases is fed back to the model, allowing it to self-correct based on evidence rather than guessing.

Evaluation used six dimensions: functional correctness, recall rate, performance, architectural soundness, code maintainability, and autonomous debugging capability. To provide a more three-dimensional observation, two anonymous frontier models were tasked independently under the exact same brief and environment configuration. The purpose was to compare behavioral style differences rather than to provide a ranking.

IV. Building a Vector Database from Scratch: Four Technical Milestones

Whether implemented by a human or a model, a functional vector database generally goes through four milestones.

4.1 Milestone 1: Distance Metrics and Brute-Force Baseline

Everything starts with correctness. Brute-force search is implemented first; it serves as the functional baseline and the reference for verifying the recall rate of subsequent approximate indexes. The core logic is very straightforward:

import math
from dataclasses import dataclass
@dataclass
class Record:
    id: str
    vector: list[float]
    metadata: dict
def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return 0.0 if na == 0 or nb == 0 else dot / (na * nb)
def knn_search(store: list[Record], query: list[float], top_k: int = 10):
    ranked = sorted(store, key=lambda r: cosine(query, r.vector), reverse=True)
    return ranked[:top_k]

It is worth noting that both models spontaneously chose the “brute-force first, optimize later” path—this is correct engineering intuition: without a correct baseline, subsequent approximate optimizations cannot be verified. Recall reconciliation can also be done in a few lines of code:

def recall_at_k(result_ids, truth_ids):
    hit = len(set(result_ids) & set(truth_ids))
    return hit / max(len(truth_ids), 1)

4.2 Milestone 2: Approximate Search Index

The complexity of brute-force search grows linearly with data volume, and latency quickly becomes unacceptable, necessitating Approximate Nearest Neighbor (ANN) indexes. There are two common industry routes:

• IVF (Inverted File Index): First clusters all vectors (e.g., using k-means) into nlist buckets. During a query, only the nprobe buckets closest to the query are scanned. It is simple to implement and memory-friendly, but recall rate is sensitive to the nprobe value.
• HNSW (Hierarchical Navigable Small World): Builds a multi-layer neighbor graph. Sparse upper layers handle “long-distance jumps,” while dense bottom layers handle “precise approximation.” ef_construction controls graph build quality, and ef_search controls query precision. Recall and latency performance are good, at the cost of high build overhead, large memory footprint, and tricky deletion operations.

4.3 Milestone 3: Quantization and Memory Trade-offs

As vector scales continue to rise, Product Quantization (PQ) or Scalar Quantization (SQ) can compress memory exponentially but introduce precision loss. Whether to introduce quantization and at which layer is a typical “trade-off decision”—this is exactly where the models diverged most in this test. Some models tended to stack all optimizations at once, which ironically introduced recall regressions that were difficult to locate.

4.4 Milestone 4: Engineering—CRUD, Filtering, Persistence, and Concurrency

• Deletion: Deletion in graph indexes usually adopts “soft delete + deferred compaction.” Directly removing nodes would destroy graph connectivity.
• Scalar Filtering: Divided into pre-filtering and post-filtering strategies. Pre-filtering filters out a candidate set based on metadata before searching; post-filtering over-samples Top-K results and then filters. Each has its pros and cons under different selectivity rates.
• Persistence: A common approach is “full snapshot + incremental log.” On startup, it loads the snapshot first and then replays the log to control recovery time.
• Concurrency: Read-write locks are the most basic solution. An advanced approach uses segmented immutable structures (borrowing from LSM concepts), utilizing copy-on-write to make the read path nearly lock-free.

V. Observations: Typical Behavioral Profiles of Agentic Coding

Reviewing the complete execution trajectories of the two models reveals several highly patterned behaviors.

What they did well: In the planning phase, they output architectural proposals and milestone breakdowns first, rather than diving straight into coding. The quality of scaffolding, data structures, API layers, and test cases generated in one pass was quite high. When encountering errors, they could autonomously read stack traces, insert logs, and converge on a fix.

Where they stumbled:

1. API Hallucinations: Occasionally referenced non-existent library functions or parameter signatures, concentrated on obscure numerical computing interfaces. The countermeasure is to explicitly write out API contracts in the task brief, which significantly reduces such errors.
2. Over-engineering: Independently introduced abstraction layers, configuration systems, and even plugin mechanisms outside of the clear acceptance criteria. For a small greenfield project, this “elegance” actually slows down iteration.
3. Long-Task Drift: As the task chain grew longer, the model began to “forget” early constraints, such as quietly changing the agreed-upon filtering semantics. The countermeasure is to make acceptance tests a fixed action in each iteration, using machine-executable assertions to replace the model’s “memory.”
4. Performance Tuning as a Watershed: When upgrading from brute-force to graph indexes, improper handling of boundary conditions (empty buckets, duplicate vectors, extreme search parameters) causes recall regressions. These issues often don’t turn tests “red”; only reconciliation against the brute-force baseline can expose them.

The style differences between the two models were also representative: one took “small steps,” with each change accompanied by tests and self-checks, taking longer but requiring less rework; the other took “strides,” preferring to output a complete implementation at once, looking impressive, but requiring significantly more debugging rounds. The styles themselves are neither good nor bad, but the corresponding human intervention strategies are completely different—the former is suitable for letting go, while the latter requires setting up checkpoints early.

VI. Conclusions and Applicable Boundaries

Returning to the initial judgment, the conclusion can be distilled into three points:

1. Pure backend greenfield + quantifiable acceptance criteria is the current “sweet spot” for Agentic Coding. Models can handle the vast majority of skeleton and functional implementation, shifting the human role from “writing code” to “defining standards and evaluating proposals.”
2. The deep waters of correctness and performance still require human gatekeeping. Boundary correctness of approximate algorithms, quantization trade-offs, and concurrency correctness—these “silent failures” are exactly where human review adds value.
3. Conclusions belong only to this test. Change the question or the acceptance criteria, and the conclusions may differ. This is why “syllabus design” is more worth investing in than “worrying about which model to use.”

VII. Implementation Suggestions for Engineering Teams

1. Write requirements as executable acceptance tests rather than prose descriptions—assertions are the least ambiguous requirement documents.
2. Push forward milestone by milestone, committing to version control at each milestone. Roll back to the last stable point whenever a problem occurs.
3. Keep the feedback loop fast and accurate: Tests should be runnable with one click, and outputs should directly pinpoint failed test cases.
4. Control context: Let the agent read files on demand rather than stuffing the entire codebase into the prompt.
5. Set human checkpoints: Architecture reviews, algorithm correctness, and security and data compliance. These three checkpoints are not recommended to be entirely handed over to the model.
6. Leverage general cloud capabilities for deployment verification: Deploy the implementation to a container platform for integration and stress testing. Observe recall and latency curves through logging services and monitoring alerts, and feed them back into the next iteration. Taking Baidu AI Cloud as an example, its general capabilities such as containers, logging, and monitoring alerts can cover these verification needs without binding to any proprietary components.

Conclusion

The value of the “building a vector database from scratch” test lies not in producing a perfect vector database, but in translating the capability boundaries of Agentic Coding into measurable engineering language: models are already reliable enough in the automated closed-loop of planning, coding, and testing, while humans should guard the checkpoints of standard setting, proposal evaluation, and correctness verification. Solidifying this methodology of “testing models with tasks and evaluating through tests” into a team asset is the only way to secure your own certainty in the wave of Agentic Coding.