Skip to content
Merged
90 changes: 90 additions & 0 deletions src/memos/dream/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,93 @@ Dream is designed as a community-building surface. Good places to contribute:
Projects can ship their own plugin with the same logical name (`dream`). When
multiple providers expose `dream`, the plugin manager keeps the implementation
with the highest priority.


## Opt-in lifecycle maintenance (PostgreSQL)

Dream retention is explicit and disabled by default. The initial atomic backend is
PostgreSQL with pgvector. Existing providers remain usable with lifecycle disabled;
enabling it on a backend without the capability raises an error. No schema, public
HTTP route, request model or new core dependency is required.

Supply lifecycle options in the component context before initializing the Dream
plugin (the ordinary server bootstrap owns this context):

```python
context["configs"]["dream_lifecycle"] = {
"enabled": True,
"policy": {
"half_life_seconds": 604800,
"never_hit_grace_seconds": 604800,
"event_max_age_seconds": 86400,
"receipt_capacity": 128,
},
"cube_ids": ["authorized-cube"],
"checkpoint_directory": "/var/lib/memos/dream-cursors",
"interval_seconds": 0, # manual; positive enables a cooperative worker
"limits": {"page_size": 100, "max_nodes": 10000, "max_seconds": 60},
}
plugin.init_components(context)
report = plugin.lifecycle.run_maintenance("authorized-cube", dry_run=True)
print(report.model_dump_json(indent=2))
```

Only explicitly registered cubes are visited, including quiet cubes with no new
Dream signals. Additional scopes require `plugin.lifecycle.register_cube(cube_id)`.
The caller authorizes those scopes; the runner does not enumerate tenants or add
permissions. Maintenance reads bounded metadata pages, revalidates under row locks
and never hard-deletes memories or changes stored content/embeddings/confidence.
It derives effective strength from fixed event-time anchors, not repeated decay.
Retrieval counts reinforce retention but do not constitute positive usefulness.

The decorated API search path records only final returned IDs after result pipes.
Native successful feedback updates notify exact affected source IDs: superseded
sources are invalidated; in-place corrections contribute a negative outcome without
invalidating corrected current content. General answer sentiment does not invalidate
all retrieved memories. Notifications use detached snapshots and fail open for the
original operation. `plugin.lifecycle.observer.delivery_snapshot()` exposes failed
delivery counters. These notifications are not a durable outbox or a transaction
with the search/feedback operation, and the SDK search path is not automatically
covered. Use explicit scoped events for SDK or application feedback:

```python
from datetime import datetime, timezone
from memos.dream.lifecycle import EventKind, LifecycleEvent

service = plugin.lifecycle.service
event = LifecycleEvent(
event_id="stable-producer-event-42", memory_id="returned-memory-id",
cube_id="authorized-cube", at=datetime.now(timezone.utc),
kind=EventKind.POSITIVE,
)
service.observe([event], now=datetime.now(timezone.utc))
```

Retain the same event ID, timestamp and payload across retries. Identical replay is
a no-op within the accepted-event-age horizon; conflicting IDs, expired events and
live receipt-capacity exhaustion fail rather than silently losing deduplication.
Capacity is deliberately bounded. This is not unlimited exactly-once processing.
Policy fingerprints prevent silently reinterpreting existing evidence after a
configuration change. Malformed metadata is quarantined, not treated as low quality.

`service.inspect(ids, cube_id=..., now=...)` explains decisions without initializing
state or modifying receipts. `service.set_pinned` exempts passive archive decisions;
explicit invalidation still wins. `service.restore(memory_id, cube_id=..., reason=...,
now=...)` restores lifecycle-owned archives, requiring `override_invalidation=True`
for explicitly invalidated evidence. It does not reconstruct lost content.

A checkpoint records backend/scope/policy identity and last committed page decisions
with evidence revisions. Atomic replacement follows database commit; a crash in
between can replay a page safely. A failed page rolls back and retains its previous
cursor, reporting the bounded retry IDs. Reports cap detailed decisions and count
omissions, retaining aggregate reasons. Dry runs write neither lifecycle state nor
checkpoint/lock files. Time/cancellation limits are cooperative between pages, not
hard deadlines for arbitrary backend calls. Plugin shutdown unregisters owned hooks
and acknowledges outstanding work; a timed-out backend operation is not claimed
to have stopped.

For native tests, provide `MEMOS_TEST_PG_DSN` for a disposable PostgreSQL+pgvector
database and run `pytest tests/dream -q`. Each test owns/drops its own schema. With
no DSN, backend tests skip explicitly. Install the existing tree/scheduler extras
and the backend's documented psycopg2-binary requirement; no model calls are needed.
Synthetic retention/throughput tests validate mechanics, not real-agent QA gains.
5 changes: 5 additions & 0 deletions src/memos/dream/contextualization.py
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,11 @@ def _persist_group(self, *, graph_db, group: DreamContextGroup, context_node, cu
summary_strategy=summary_strategy,
)

lifecycle = self.context.get("shared", {}).get("dream_lifecycle")
if lifecycle is not None:
metadata = lifecycle.metadata_for_new(
context_id, cube_id, metadata, now=datetime.now(timezone.utc)
)
graph_db.add_node(context_id, summary, metadata, user_name=cube_id)
return _build_context_event(
context_id=context_id,
Expand Down
32 changes: 32 additions & 0 deletions src/memos/dream/lifecycle/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""Opt-in lifecycle evidence and reversible Dream retention policy."""

from memos.dream.lifecycle.contracts import (
LIFECYCLE_METADATA_KEY,
EventKind,
LifecycleConfig,
LifecycleError,
LifecycleEvent,
LifecycleState,
MaintenanceDecision,
)
from memos.dream.lifecycle.policy import (
apply_event,
decide_maintenance,
initialize_state,
restore_state,
)


__all__ = [
"LIFECYCLE_METADATA_KEY",
"EventKind",
"LifecycleConfig",
"LifecycleError",
"LifecycleEvent",
"LifecycleState",
"MaintenanceDecision",
"apply_event",
"decide_maintenance",
"initialize_state",
"restore_state",
]
217 changes: 217 additions & 0 deletions src/memos/dream/lifecycle/contracts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
"""Validated metadata and event contracts for opt-in Dream maintenance."""

from __future__ import annotations

import hashlib
import json

from copy import deepcopy
from datetime import datetime, timezone
from enum import Enum
from typing import Any

from pydantic import BaseModel, ConfigDict, Field, field_validator

from memos.exceptions import MemoryError as MemOSMemoryError
from memos.graph_dbs.base import MANAGED_LIFECYCLE_KEY, MANAGED_LIFECYCLE_VERSION


LIFECYCLE_METADATA_KEY = MANAGED_LIFECYCLE_KEY


class LifecycleError(MemOSMemoryError):
"""An observation or transition cannot safely change lifecycle evidence."""


class LifecycleMetadataError(LifecycleError):
"""Stored evidence is invalid or incompatible and must remain quarantined."""


class LifecycleCapacityError(LifecycleError):
"""Live deduplication receipts cannot be evicted to admit another event."""


def detach_lifecycle_identity(metadata: dict[str, Any], *, source_memory_id: str) -> dict[str, Any]:
"""Prepare an explicit new identity without inheriting observations or receipts.

Source/provenance fields remain intact. The small Dream-origin marker allows
the lifecycle service to initialize fresh evidence even for a system-sourced
diary that was originally recognized only by its reserved envelope. This
function changes a new copy, never the source node's managed state.
"""
fresh = deepcopy(metadata)
if LIFECYCLE_METADATA_KEY not in fresh:
return fresh
fresh.pop(LIFECYCLE_METADATA_KEY)
internal = fresh.get("internal_info") or {}
if not isinstance(internal, dict):
raise LifecycleError("Cannot preserve malformed derived-memory provenance")
dream = internal.get("dream") or {}
if not isinstance(dream, dict):
raise LifecycleError("Cannot preserve malformed Dream-origin provenance")
dream.setdefault("kind", "derived_memory")
dream["lifecycle_origin_memory_id"] = source_memory_id
internal["dream"] = dream
fresh["internal_info"] = internal
return fresh


def utc_time(value: datetime | str, *, legacy: bool = False) -> datetime:
"""Normalize a time; only legacy stored times may omit a timezone."""
if isinstance(value, str):
try:
value = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
raise LifecycleError("Invalid lifecycle timestamp") from exc
if not isinstance(value, datetime):
raise LifecycleError("A lifecycle timestamp must be a datetime")
if value.tzinfo is None or value.utcoffset() is None:
if not legacy:
raise LifecycleError("New lifecycle timestamps require a timezone")
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)


class EventKind(str, Enum):
RETRIEVED = "retrieved"
POSITIVE = "positive"
NEGATIVE = "negative"
INVALIDATED = "invalidated"


class LifecycleConfig(BaseModel):
"""Retention parameters; their fingerprint prevents silent reinterpretation."""

model_config = ConfigDict(frozen=True, extra="forbid", allow_inf_nan=False)

half_life_seconds: float = Field(default=604800, gt=0, strict=True)
never_hit_grace_seconds: float = Field(default=604800, ge=0, strict=True)
archive_strength_threshold: float = Field(default=0.1, ge=0, le=1, strict=True)
retrieval_reinforcement: float = Field(default=0.1, ge=0, strict=True)
strength_cap: float = Field(default=2.0, gt=0, strict=True)
low_usefulness_threshold: float = Field(default=0.2, ge=0, le=1, strict=True)
min_outcome_count: int = Field(default=3, ge=1, strict=True)
event_max_age_seconds: float = Field(default=86400, gt=0, strict=True)
future_tolerance_seconds: float = Field(default=60, ge=0, strict=True)
receipt_capacity: int = Field(default=128, ge=1, le=100000, strict=True)

def fingerprint(self) -> str:
payload = json.dumps(self.model_dump(), sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode()).hexdigest()


class LifecycleEvent(BaseModel):
"""An immutable, scoped producer event; IDs must not be recycled."""

model_config = ConfigDict(frozen=True, extra="forbid")

event_id: str = Field(min_length=1, max_length=512, strict=True)
memory_id: str = Field(min_length=1, max_length=1024, strict=True)
cube_id: str = Field(min_length=1, max_length=1024, strict=True)
at: datetime
kind: EventKind
reason: str = Field(default="", max_length=1024, strict=True)

@field_validator("at", mode="before")
@classmethod
def aware_event_time(cls, value: datetime | str) -> datetime:
try:
return utc_time(value)
except LifecycleError as exc:
raise ValueError(str(exc)) from exc

def digest(self) -> str:
payload = json.dumps(self.model_dump(mode="json"), sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode()).hexdigest()


class EventReceipt(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")

identity: str = Field(pattern=r"^[0-9a-f]{64}$")
payload: str = Field(pattern=r"^[0-9a-f]{64}$")
expires_at: datetime

@field_validator("expires_at", mode="before")
@classmethod
def aware_expiry(cls, value: datetime | str) -> datetime:
return utc_time(value)


class LifecycleState(BaseModel):
"""The reserved graph envelope; ordinary writers cannot replace it."""

model_config = ConfigDict(frozen=True, extra="forbid", allow_inf_nan=False)

version: int = Field(
default=MANAGED_LIFECYCLE_VERSION,
ge=MANAGED_LIFECYCLE_VERSION,
le=MANAGED_LIFECYCLE_VERSION,
strict=True,
)
memory_id: str = Field(min_length=1, strict=True)
cube_id: str = Field(min_length=1, strict=True)
policy_fingerprint: str = Field(pattern=r"^[0-9a-f]{64}$")
source_dream_id: str = ""
created_at: datetime
base_strength: float = Field(default=1.0, ge=0, strict=True)
pinned: bool = Field(default=False, strict=True)
last_hit_at: datetime | None = None
hit_count: int = Field(default=0, ge=0, strict=True)
positive_mass: float = Field(default=0.0, ge=0, strict=True)
negative_mass: float = Field(default=0.0, ge=0, strict=True)
outcome_count: int = Field(default=0, ge=0, strict=True)
mass_anchor_at: datetime | None = None
invalidated_by_feedback: bool = Field(default=False, strict=True)
status: str = Field(default="active", pattern=r"^(active|archived)$")
protect_archived_status: bool = Field(default=False, strict=True)
revision: int = Field(default=0, ge=0, strict=True)
archived_at: datetime | None = None
archive_reason: str = ""
restored_at: datetime | None = None
restore_reason: str = ""
receipts: tuple[EventReceipt, ...] = ()

@field_validator(
"created_at", "last_hit_at", "mass_anchor_at", "archived_at", "restored_at", mode="before"
)
@classmethod
def stored_time(cls, value: Any) -> datetime | None:
return None if value is None else utc_time(value, legacy=True)


class MaintenanceDecision(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid", allow_inf_nan=False)

action: str = Field(pattern=r"^(keep|archive|skip)$")
reason: str
strength: float = Field(ge=0)
usefulness: float = Field(ge=0, le=1)


def checked_state(state: LifecycleState, config: LifecycleConfig) -> LifecycleState:
"""Revalidate even model_copy inputs before a storage reducer trusts them."""
try:
state = LifecycleState.model_validate(state.model_dump(mode="python"))
except ValueError as exc:
raise LifecycleError("Invalid stored lifecycle envelope") from exc
if state.protect_archived_status != (state.status == "archived"):
raise LifecycleError("Lifecycle archive ownership/status mismatch")
if state.status == "archived" and (
state.archived_at is None or not state.archive_reason.strip()
):
raise LifecycleError("Lifecycle archive lacks transition provenance")
if state.invalidated_by_feedback and not state.protect_archived_status:
raise LifecycleError("Invalidated lifecycle evidence must remain archived")
if (state.hit_count == 0) != (state.last_hit_at is None):
raise LifecycleError("Lifecycle retrieval count/time mismatch")
if state.outcome_count == 0 and (state.positive_mass != 0 or state.negative_mass != 0):
raise LifecycleError("Lifecycle outcome mass lacks observed outcomes")
if state.policy_fingerprint != config.fingerprint():
raise LifecycleError("Lifecycle policy mismatch; explicit migration is required")
if len(state.receipts) > config.receipt_capacity:
raise LifecycleError("Stored receipt capacity exceeds lifecycle policy")
identities = [receipt.identity for receipt in state.receipts]
if len(set(identities)) != len(identities):
raise LifecycleError("Duplicate stored lifecycle receipt identity")
return state
Loading
Loading