1. The Problem: Why Your Agent Keeps Forgetting

Developers who use conversational AI agents have almost all encountered the same awkward situation: after an in-depth technical discussion with an agent one day, they reopen the session the next day, and the agent responds like a first-time meeting: “Hello, how can I help you?” It behaves as if the previous interaction never happened.

The root cause is that the memory modules built into most agent frameworks provide only basic vector similarity retrieval. As the number of conversation turns grows and information density increases, this single-dimension recall method quickly degrades in both accuracy and coverage. Users find that the agent “looks like it has memory, but in practice it behaves as if it has none”: either it recalls semantically similar but practically irrelevant fragments, or key information was never written into the memory store at all.

The solution discussed in this article is a production-grade long-term memory plugin. It is not simply storing chat logs as text files. Instead, it builds a complete system of vector retrieval + full-text search + cross-encoder reranking + memory lifecycle management. Its design principle can be summarized in one sentence: get it running first, then enhance it; survive first, then pursue quality.

2. Core Capabilities: What a Memory System Needs

2.1 Hybrid Retrieval: Vector Semantics + Keyword Full-Text Search

The limitation of single vector retrieval is that it is not friendly to exact-match scenarios. For example, if a user previously mentioned a specific error code such as “ERR_CONN_REFUSED_502,” the vector model may generalize it into a semantic representation of “network connection error,” making it impossible to retrieve the exact code later. Conversely, pure keyword retrieval cannot handle synonymous expressions and semantic generalization.

A production-grade approach therefore uses a hybrid retrieval strategy: vector semantic search recalls semantically relevant memory fragments, while BM25 full-text search handles exact matching of keywords and entity names. The two result sets are merged and scored through RRF (Reciprocal Rank Fusion). The advantage of RRF is that it does not need to normalize scores from the two retrieval paths. It fuses results directly based on rank positions, which makes the engineering implementation simple and robust.

# RRF fusion scoring example
def rrf_fusion(vector_results, bm25_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

2.2 Cross-Encoder Reranking: Making Recall Results More Context-Aware

Hybrid retrieval solves the “recall rate” problem, but the recalled result list may still be poorly ordered. Cross-encoder reranking refines the initial recall results by scoring them more precisely: the query and candidate memory fragment are concatenated and fed into the model, which directly outputs a relevance score.

In engineering practice, a weighted fusion strategy is commonly used: final score = 60% cross-encoder score + 40% original fusion score. At the same time, a fallback mechanism is needed: when the reranking service API is unavailable or times out, the system automatically falls back to the original fusion score so that overall system availability is not affected.

For the reranking service, you can connect to common vector reranking API services, or choose a provider with a free tier for early validation. The key design point is to abstract a unified Provider interface to facilitate later switching and extension.

2.3 Intelligent Extraction: Let the Agent Decide What to Remember

Early approaches required developers to call a memory write API manually, which is highly unrealistic in actual use. Users cannot manually save a memory entry every few messages. Therefore, an LLM-driven automatic extraction mechanism is needed.

The specific approach is to send the conversation content into a lightweight extraction pipeline after each dialogue turn. The LLM determines whether it contains information worth keeping long term and classifies it into a predefined category system. Common categories include:

• Profile: Static information such as user identity, role, and technology stack
• Preferences: Preference settings, style tendencies, and tool choices
• Entities: Project names, service addresses, and key identifiers
• Events: Important operations, changes, and decisions that occurred
• Cases: Problem scenarios and corresponding solutions
• Patterns: Recurring interaction patterns or habits

Extracted memory entries are automatically written into the memory store without manual developer intervention. The key design point here is noise interception: not all conversation content is worth remembering. The extraction pipeline must be able to judge “what is not worth remembering” so that the memory store is not flooded with low-value information.

2.4 Memory Lifecycle: Important Memories Get Stronger with Use, Unused Ones Fade

Human memory is not static: frequently used memories are reinforced, while long-unused ones gradually blur. A production-grade memory system also needs lifecycle management.

A feasible engineering implementation uses a Weibull decay engine together with a three-tier promotion system:

• Peripheral: Newly written memories enter this tier first and decay relatively quickly
• Working: Memories that have been recalled and confirmed useful are promoted here and decay at a medium rate
• Core: Memories with high recall frequency and high confidence are promoted here and decay very slowly

Each time a memory is recalled and successfully used in an answer, its “strength” increases, and its decay curve shifts accordingly. Conversely, memories that have not been recalled for a long time are gradually demoted until they are cleaned up. This mechanism ensures that the signal-to-noise ratio of the memory store rises rather than falls over time.

2.5 Multi-Scope Isolation: Different Agents, Users, and Projects Do Not Interfere

In actual deployment, a memory system often needs to serve multiple agent instances, multiple users, or multiple projects. If all memories are mixed together, serious cross-contamination occurs: a question from project A may recall a solution from project B.

Therefore, a scope isolation mechanism is needed. It supports separate memory spaces divided by dimensions such as Agent ID, user ID, and project ID. By default, retrieval is performed only within the current scope, and explicit cross-scope queries are also supported. Scope design needs to balance isolation and flexibility: it should be safe by default while still allowing reasonable sharing scenarios.

3. Deployment Prerequisites: What You Need Before Starting

Before deployment, confirm the following basic conditions:

1. A running agent framework instance. This article assumes that readers already have basic agent building and running capabilities. If you have not completed this step yet, it is recommended to finish setting up the basic environment first.
2. Node.js v18 or later. Run node --version in the terminal to confirm the version. Versions below v18 need to be upgraded first.
3. A usable vector database or built-in vector storage module. Most agent frameworks provide built-in vector storage capabilities, which can be reused directly at the beginning. If there are higher requirements for performance and scale, consider replacing it with a standalone vector database service.

4. Phased Implementation Path: From Minimum Viable to Production-Grade

Stage 1: Minimum Viable Deployment (Running Within 30 Minutes)

Goal: get the memory system running first and validate the basic read/write and recall path.

Steps:

1. Install memory plugin dependencies
2. Initialize with the default configuration (built-in vector storage + basic vector retrieval)
3. Perform a test memory write
4. Issue a retrieval request and confirm that the content just written can be recalled

At this stage, do not enable reranking, intelligent extraction, lifecycle management, or other enhanced features. First confirm that the basic path works, then add capabilities step by step.

Stage 2: Enable Hybrid Retrieval and Reranking

After the basic path is running stably, enable the following in sequence:

1. BM25 full-text index: Build a full-text index for memory entries to run in parallel with vector retrieval
2. RRF fusion: Configure the fusion strategy for the two retrieval result sets
3. Cross-encoder reranking: Connect the reranking service Provider and configure weight ratios and fallback strategy

It is recommended to validate recall quality changes with a fixed set of test queries after enabling each capability, and continue only after confirming a positive benefit.

Stage 3: Enable Intelligent Extraction and Lifecycle Management

1. Configure the LLM extraction pipeline and define the category system and extraction granularity
2. Enable noise interception rules to filter low-value conversation content
3. Enable Weibull decay and three-tier promotion, and set decay parameters and promotion thresholds for each tier
4. Configure multi-scope isolation policies and divide memory spaces according to actual deployment dimensions

5. Progressive Enhancement Strategy and Considerations

Do not tune parameters all at once. The effectiveness of a memory system depends heavily on the specific scenario: a technical documentation agent needs a stronger keyword-matching weight, while a customer service agent needs a more aggressive automatic extraction strategy. It is recommended to run with default parameters for a period of time, collect actual recall logs, and then adjust in a targeted way.

Monitor memory store health. Pay attention to three core metrics: memory entry growth rate, recall hit rate, and cross-scope false recall rate. If memory entries grow quickly but the hit rate does not rise, the extraction granularity is too fine or noise interception is insufficient. If the cross-scope false recall rate is high, check the scope division logic.

A fallback path must be available. Reranking services and LLM extraction pipelines may both become unavailable. Ensure that the system can automatically fall back to basic retrieval mode when enhanced components fail, rather than having the entire memory function break down.

Hot-swappable design reduces trial-and-error cost. The memory plugin should support dynamically enabling and disabling capabilities without restarting the agent process, which facilitates A/B testing and canary releases.

6. Conclusion

The core of building long-term memory for AI agents is not “what is stored,” but “remembering what should be remembered, forgetting what should be forgotten, and recalling it when needed.” The five capabilities discussed in this article—hybrid retrieval, cross-encoder reranking, LLM-driven extraction, lifecycle management, and multi-scope isolation—together form the complete picture of a production-grade memory system.

For the implementation path, always adhere to the principle of getting the minimum path running first and then enhancing item by item. Memory system tuning is a continuous iterative process. There is no one-time parameter configuration, only progressive optimization that increasingly fits actual usage scenarios. When your agent starts to “remember” the technical solution discussed last time, user preferences, and project context, you will truly feel the experience leap brought by long-term memory.