Introduction

One of the biggest limitations of modern AI agents is memory. Most LLM-based agents can handle short-term context within a conversation, but they struggle with long-term memory: tracking facts, adapting to user preferences, and maintaining consistency across sessions.

Check out the repo here

Memory-X is an open-source project that directly addresses this problem. It implements a bi-temporal memory architecture and multi-layered memory management system, giving AI agents the ability to store, retrieve, and reason over past experiences—in a way that resembles human cognitive processes.

Core Concepts

1. Bi-Temporal Memory Model

Every piece of memory in Memory-X carries two timestamps:

  • valid_from: when the fact or memory became true in the system.
  • commit_ts: when it was stored or updated in the database.

This dual-time design enables time-travel queries, versioning, and fine-grained reasoning over changing states. For example, if a user changes their address or medication, the system can distinguish between historical facts and current truth.

2. Multi-Layered Memory

Memory-X structures memory into multiple layers:

  • Short-Term Memory: recent dialogue context.
  • Working Memory: transient knowledge relevant to ongoing tasks.
  • Long-Term Memory: facts, events, and user profiles that persist across sessions.

This separation allows agents to balance context relevance with storage efficiency.

3. Cognitive Components

Several built-in modules make Memory-X more than just a database:

  • EntityRecognizer – extracts structured entities from user input.
  • IntentDetector – identifies user goals and task types.
  • ImportanceEvaluator – scores memories based on significance, helping filter noise from relevant information.
  • MemoryQueryEngine – supports semantic and temporal queries across the memory store.

System Architecture

At the heart of Memory-X is the MemoryManager, orchestrating how memories are created, indexed, and retrieved.

The key models include:

  • FactMemory – stores factual knowledge with version control.
  • ConversationMemory – maintains dialogue logs.
  • EntityMemory – keeps recognized entities for cross-session recall.
  • UserProfile – manages user-specific preferences and attributes.

All components are modular and extensible, making the system flexible for different agent designs.

Usage and Integration

Installation

Memory-X can run locally or in Docker:

# Local setup
git clone https://github.com/ylouis83/memory-x.git
cd memory-x
pip install -r requirements.txt
python init_db.py

# Docker
docker-compose up -d

Python Example

from memory_x.manager import MemoryManager

manager = MemoryManager(user_id="user_001")

manager.add_conversation(
    user_message="My name is Alice",
    ai_response="Nice to meet you, Alice!",
    entities={"PERSON": [("Alice", 0, 5)]},
    intent="INTRODUCE",
    importance=3
)

results = manager.get_relevant_memories("Alice")
print(results)

RESTful API

The system also exposes a REST API, making it easy to plug into existing agent frameworks.

Why It Matters

Most AI frameworks still rely on ad-hoc context windows or vector databases for memory. Memory-X introduces:

  • Temporal consistency: facts evolve over time without losing history.
  • Cognitive alignment: mimics how humans handle short- vs long-term memory.
  • Plug-and-play extensibility: modules can be customized for domain-specific needs (e.g., medical records, customer support, personal assistants).