Designing Memory Systems for AI Agents: A Layered Architecture and Engineering Practices

Most developers who have built AI agents have run into the same problem: memory. Large language models are stateless by nature. Every conversation starts from zero. The preferences a user mentioned yesterday, the decisions made last week, the pitfalls encountered last month—none of it carries over. How you solve this problem in engineering terms directly determines whether an agent product is a toy or a real tool.

The mainstream memory solutions on the market fall into two broad categories. The first is to stuff the entire conversation history into the context window until you hit the token limit. This approach is expensive: token consumption grows linearly with the number of turns, making costs hard to control. Worse, once the context gets long, the model’s attention gets diluted—constraints mentioned earlier are forgotten later, producing the classic “long-context drift” phenomenon. The second category is retrieval-augmented generation using a vector database, recalling historical fragments by semantic similarity. The problem here is unstable retrieval quality—semantic similarity does not equal task relevance, and recalled content often misses the point. Vector databases also introduce additional operational overhead: index construction, embedding model version management, and recall threshold tuning are all ongoing costs.

After trying both paths, you realize their shared flaw: they treat “memory” as a single storage problem without distinguishing the fundamental differences in lifecycle and purpose across different layers of information. This article breaks down a lighter-weight, more stable layered memory architecture from an engineering practice perspective, and discusses its design principles, implementation details, and common pitfalls.

1. Core Design Philosophy: Separating “Streams” from “Sediment”

Human memory is naturally layered. You probably cannot remember what you had for lunch yesterday, but you know clearly that you dislike cilantro and are allergic to seafood. The former is stream information that decays quickly over time; the latter is distilled, long-term, stable sediment information. If these two types of information are stored together, they interfere with each other during retrieval—you look for “today’s to-dos” and pull up a pile of year-old preferences; you try to confirm “user restrictions” and get buried in trivial conversation logs.

The core of a layered memory architecture is to thoroughly separate these two types of information across three stages: storage, writing, and reading:

• Stream layer (short-term memory): Organized by time, append-only, recording events that happened today, decisions executed, and tasks completed. Fine-grained, time-sensitive, and rapidly losing value over time.
• Sediment layer (long-term memory): Distilled from the stream layer, storing user preferences, high-frequency context, key decision conclusions, and verified lessons learned. Coarse-grained, less time-sensitive, and highly reusable.

The two layers have clear responsibilities and do not contaminate each other. They are combined on demand during reads. This approach is so simple it barely feels like an “architecture,” but it is precisely this simplicity that delivers engineering maintainability.

2. Implementation Details for the Stream Layer

The stream layer is best implemented as a plain-text append-only log, organized into files by date—for example, one Markdown file per day. The rationale for choosing Markdown over a structured database is practical: human-readable, easy to debug, naturally version-controllable, and free of additional runtime dependencies. For individual-level or small-team agent products, this is far lighter than introducing a database.

There are several key decisions for the write strategy:

First, append only, never modify. Records generated during the day are appended, not updated in place. This keeps the write path minimal, eliminates concurrent write conflicts, and preserves a complete timeline for later review and distillation. If a correction is needed, append a correction record.

Second, write content for “future retrieval,” not “present recording.” The stream log is not a diary for yourself—its consumers are the downstream memory distillation process and context injection process. Each record should therefore contain structured elements: timestamp, event type (decision/task/fact/anomaly), involved objects, and conclusion or result. A typical record might look like this:

## 2025-03-14
- [Decision] Deployment approach for Project A confirmed as containerized; rationale is that the existing CI pipeline already supports image builds, so migration cost is lower than expected.
- [Task] Completed integration of the log collection module; verified the alert rule trigger chain.
- [Fact] User feedback requests tabular output by default for report generation; confirmed and adopted.
- [Anomaly] Batch tasks experienced queuing delays during peak hours; preliminary assessment is that concurrency is set too high; pending verification.

Third, control daily file size. More is not better for the stream layer. If a single day’s records balloon to thousands of lines, both distillation and injection slow down. In practice, you can do lightweight trimming at the write end: merge repetitive events into a single summary, downgrade or archive pure operation-log records to cold storage, and keep only content with potential value for future decisions.

3. Implementation Details for the Sediment Layer

The sediment layer is typically an independent long-term memory file containing stable knowledge distilled from the stream layer. It is not a simple summary of the stream layer but a “conclusion set” that has been judged and generalized. Typical contents include:

• User preferences and constraints (format preferences, exclusions, commonly used parameters)
• High-frequency business context (common task types, fixed workflows)
• Confirmed key decisions and their rationale (avoiding re-discussion of settled matters)
• Verified lessons learned (which approaches don’t work, which pitfalls have already been encountered)

The distillation process can be scheduled (e.g., triggered as a batch job at the end of each day or after each conversation round) or event-driven (distilled immediately when specific types of records appear in the stream layer). The distillation logic itself can be rule-based or use a model for summarization and categorization, but either way, writes to the sediment layer must be “merge-style”—when new knowledge conflicts with an old entry, update rather than append, avoiding multiple versions of the same preference in the file.

The sediment layer should stay lean. Practical experience suggests keeping it within a few hundred lines; when it exceeds that, actively merge and prune: downweight or remove entries that haven’t been referenced for a long time, and delete old entries superseded by new conclusions. This step is easily overlooked in many implementations—a memory layer that only grows and never shrinks eventually degenerates into a second “full context” and loses the point of layering.

4. Context Injection: How the Two Layers Work Together

At the start of each new session, the system assembles memory context in a fixed order: first read the day’s stream records for recent context, then read the sediment layer for long-term stable background knowledge, then concatenate and inject both into the system prompt.

Several engineering details deserve attention:

Order and weight. Whether the stream layer goes first and the sediment layer second, or vice versa, affects how much attention the model pays to the information. The general recommendation is to place more stable, more important constraints (sediment layer) near the instruction and time-sensitive information (stream layer) before it as background. Different models distribute attention differently over long contexts, so this is worth determining through small-scale experiments.

On-demand trimming. Not every session needs both layers injected in full. For simple single-turn tasks, the sediment layer alone may suffice; for continuation tasks, the day’s stream records need to be loaded. Before injection, filter once based on task type to avoid unnecessary token costs.

Injection format. Use clear section markers to distinguish the two layers—for example, different heading levels or separators—so the model clearly knows which part is “long-term background” and which is “recent activity.” Ambiguous concatenation causes the model to confuse the temporal attributes of information.

An illustrative injection template:

[Long-term memory]
- User preference: Reports default to tabular output
- Known constraint: No third-party dependencies requiring additional payment
- Historical decision: Deployment approach set as containerized
[Recent activity]
- Today's to-dos: Verify alert rule trigger chain
- Today's anomaly: Batch tasks queuing delays during peak hours; concurrency configuration pending investigation

5. Why This Architecture Is More Robust in Engineering

Compared with the full-context and vector-retrieval routes, the layered architecture offers advantages across four dimensions:

Controllable cost. The amount of injected memory has a clear upper bound (single-day stream + fixed sediment layer) and does not grow indefinitely with conversation history, making token costs predictable.

Stable retrieval quality. There is no semantic similarity recall—information is read directly by time and layer, so there is no problem of “recalling irrelevant fragments.” The stream layer ensures recent information is not missed; the sediment layer ensures long-term constraints are not lost. Both have defined responsibilities and predictable results.

Strong debuggability. All memory is human-readable text files. When something goes wrong, you can open them directly to see what was written and what was injected, making diagnosis extremely cheap. In contrast, vector retrieval recall results are opaque, and tuning relies on trial and error.

Low operational burden. No vector database, index, or embedding service to maintain—plain file reads and writes suffice, simplifying deployment and migration.

6. Common Pitfalls and How to Avoid Them

When putting this architecture into practice, several high-frequency issues are worth preempting:

Pitfall 1: Uncontrolled sediment layer growth. Distilling without pruning turns the sediment layer into another full log within a few months, rendering layering meaningless. The fix is to set a size cap and periodically merge and prune, treating “forgetting” as a first-class feature.

Pitfall 2: Overly granular stream records. Writing every operation log into the stream makes distillation expensive and lowers the signal-to-noise ratio. The fix is to define clearly at the write end “what is worth recording,” using “could this affect future decisions” as the criterion.

Pitfall 3: Blurred boundaries between layers. Writing long-term preferences that should be sedimented into the day’s stream, or writing temporary tasks into long-term memory. The fix is to define clear classification rules in the distillation process and reinforce the model’s layer awareness with clear section markers during injection.

Pitfall 4: Ignoring time zones and date boundaries. When splitting files by date, cross-time-zone or midnight-crossing sessions can land in the wrong file, causing “today’s context” to be missing. The fix is to standardize on the server time zone and explicitly determine the “today” date key at session start.

7. Summary

The value of a layered memory architecture lies not in technical complexity but in its accurate modeling of the information lifecycle: the stream handles “what happened,” and the sediment handles “what has been confirmed.” The two are stored separately, maintained separately, and combined on demand for injection—using the most unassuming text files to achieve stable, debuggable, low-cost agent memory. For teams building AI agent products, this approach can be implemented directly and also serves as a solid baseline for more complex memory solutions. Getting the layering right first, then considering smarter generalization capabilities in the sediment distillation stage, is the more reliable evolutionary path.