Add the agent memory lifecycle: brain, holistic recall, pre/post-turn, belief builds

AGENTS.md

@@ -17,7 +17,8 @@ package it holds. There are exactly three, one per part of the memory layer:
   metadata, namespaces, errors, and (feature `conformance`) the suite every
   engine must pass. No I/O.
 - `crates/tinymemory-tools` — the **agent tool spec**: `MemoryTools` over any
-  engine, and the `context.md` compiler.
+  engine, holistic recall and the `context.md` compiler, and the agent memory
+  lifecycle (`AgentMemory`, `Brain`, `MemoryLayout`, background jobs).
 - `crates/tinymemory-integrations` — the **integrations**: the CortexDB engine
   and its registry, documents, sources, safety and the legacy v1 import, each
   a module behind a feature.

Cargo.toml

@@ -5,8 +5,8 @@ resolver = "2"
 # - `tinymemory-api`: the core contract (`MemoryEngine`, items, metadata,
 #   namespaces, errors) and, behind `conformance`, the suite every engine must
 #   pass.
-# - `tinymemory-tools`: the agent-facing tool spec over any engine, and the
-#   `context.md` compiler.
+# - `tinymemory-tools`: the agent-facing tool spec over any engine, holistic
+#   recall and the `context.md` compiler, and the agent memory lifecycle.
 # - `tinymemory-integrations`: everything that talks to the outside world —
 #   the CortexDB engine and its registry, document conversion, source readers,
 #   safety scrubbing and the legacy v1 import — each behind a feature.

README.md

@@ -16,6 +16,8 @@ namespace tree that keeps one tenant's or agent's memory apart from another's.
 
 - **Specified behaviour:** [`docs/specs/memory-v2.md`](docs/specs/memory-v2.md)
   is the source of truth for what the system does and why.
+- **Integrating an agent:** [`docs/integration.md`](docs/integration.md) is
+  the guide for a host wiring its agent loop to the memory API.
 - **How it is built:** [`docs/architecture/README.md`](docs/architecture/README.md)
   has one page per concern: the contract, operations, namespaces, the CortexDB
   engine, the agent tools, the integrations and the test strategy.
@@ -33,7 +35,9 @@ crates/
 │                             an in-memory reference engine
 ├── tinymemory-tools/         the agent surface over any engine: `MemoryTools` (seven
 │                             model-callable tools with JSON Schemas and host-fixed
-│                             scoping) and the `context.md` compiler
+│                             scoping), holistic recall and the `context.md` compiler,
+│                             and the agent lifecycle (`AgentMemory`, `Brain`,
+│                             `MemoryLayout`, background jobs)
 └── tinymemory-integrations/  everything that touches the outside world, one module
                               per feature: `cortex` (+ `registry`, `config`),
                               `documents`, `sources`, `safety`, `import`
@@ -58,11 +62,12 @@ else on request, so a host pays only for what it uses.
 | `cortex` (default) | `cortex`, `registry`, `config` | `CortexEngine` over both wires, `list_engines`, `build_engine`, `EngineCredential`, `MemoryConfig` |
 | `documents` | `documents` | Format sniffing and conversion to markdown, producing `StoreItem::Document` |
 | `documents-office` | `documents::OfficeConverter` | PDF, DOCX, PPTX and XLSX to markdown (implies `documents`) |
+| `brain` | `brain` | Files into `tinymemory_tools::BrainDocument`s, by the source type their format implies (implies `documents`) |
 | `sources` | `sources` | Folder, file and conversation readers, Composio normalisers (implies `documents`) |
 | `sources-network` | `sources::fetch` and the network readers | GitHub, RSS and web-page readers and `fetch_url`, behind the SSRF guard (implies `sources`) |
 | `safety` | `safety` | Secret and PII scrubbing of a `StoreItem` |
 | `legacy-import` | `import` | Migrating a v1 (embedded TinyCortex) workspace into any engine |
-| `full` | all of the above | `cortex`, `documents-office`, `sources-network`, `safety`, `legacy-import` |
+| `full` | all of the above | `cortex`, `documents-office`, `brain`, `sources-network`, `safety`, `legacy-import` |
 
 Dependency weight per feature is tabulated in
 [`crates/tinymemory-integrations/README.md`](crates/tinymemory-integrations/README.md).
@@ -136,6 +141,28 @@ credential, and a credentialed cleartext endpoint that is not loopback.
 To compile a `context.md` for the start of a session, call
 `tinymemory_tools::context::compile(&*engine, &ContextSpec::default())`.
 
+### The agent lifecycle
+
+For memory around every turn, use `tinymemory_tools::AgentMemory`. It
+implements a standard layout: a global brain of documents by source type,
+each agent's conversations, and shared learnings. Call it at each point of
+the agent loop:
+
+```rust,ignore
+let memory = AgentMemory::new(engine.clone(), MemoryLayout::default(), "support-01")?;
+let turn = memory.pre_turn(PreTurn::new("thread-1", 0, user_text)).await?;   // log + recall, no indexing wait
+let reply = llm.generate(&turn.pack.markdown, user_text).await;
+let report = memory.post_turn(PostTurn::new("thread-1", 1, reply)).await?;   // log
+for job in report.jobs { queue.push(job) }                                    // belief builds, off the turn
+```
+
+`Brain` ingests documents (with `tinymemory_integrations::brain::brain_document`
+converting PDFs and markdown first). `start_session` and
+`recall_for_compaction` cover session resume and prompt truncation. Run
+`cargo run -p tinymemory-tools --example agent_loop` for the whole loop
+offline, and see
+[`docs/architecture/lifecycle.md`](docs/architecture/lifecycle.md).
+
 ## Engines
 
 | Id | What | Fetch modes |

crates/tinymemory-api/src/conformance/mod.rs

@@ -33,5 +33,5 @@ pub mod reference;
 mod suite;
 
 pub use error::{Error, Result};
-pub use reference::{REFERENCE_ENGINE_ID, ReferenceEngine};
+pub use reference::{CONSOLIDATED_TAG, REFERENCE_ENGINE_ID, ReferenceEngine};
 pub use suite::run;

crates/tinymemory-api/src/conformance/reference/distil.rs

@@ -0,0 +1,88 @@
+//! The reference engine's consolidation: a toy belief per source item.
+//!
+//! Real engines distil beliefs with a model. The reference engine needs only
+//! something deterministic and obvious: each document or conversation an
+//! [`ConsolidateRequest`] admits yields one [`LearningKind::Fact`] — the
+//! first sentence of the document's prose, or of the conversation's first
+//! user turn — stored at the source item's own node with the source's metadata, a
+//! `consolidated` tag, and the source's id as evidence. Consolidating twice is
+//! a replay, so it never duplicates a belief.
+
+use crate::{ConsolidateRequest, DocumentBody, ItemKind, LearningKind, Role, StoreItem};
+
+/// Tag on every belief the reference engine distils.
+pub const CONSOLIDATED_TAG: &str = "consolidated";
+
+/// Confidence of a distilled belief: a first sentence is a weak signal.
+const BELIEF_CONFIDENCE: f32 = 0.5;
+
+/// Longest statement a belief keeps, in characters.
+const MAX_STATEMENT_CHARS: usize = 240;
+
+/// The beliefs `request` distils from `items`, in item order.
+pub(super) fn distil(items: &[StoreItem], request: &ConsolidateRequest) -> Vec<StoreItem> {
+    let kinds = request.admitted_kinds();
+    items
+        .iter()
+        .filter(|item| item.kind() != ItemKind::Learning && kinds.contains(&item.kind()))
+        .filter(|item| request.reach.admits(&item.meta().namespace))
+        .filter_map(|item| {
+            let statement = first_sentence(&source_text(item)?)?;
+            let mut meta = item.meta().clone();
+            if !meta.tags.iter().any(|tag| tag == CONSOLIDATED_TAG) {
+                meta.tags.push(CONSOLIDATED_TAG.to_string());
+            }
+            Some(StoreItem::Learning {
+                text: statement,
+                kind: LearningKind::Fact,
+                confidence: BELIEF_CONFIDENCE,
+                evidence: Some(item.fingerprint()),
+                meta,
+            })
+        })
+        .collect()
+}
+
+/// The text a belief is drawn from: a document's body, or a conversation's
+/// first user turn.
+fn source_text(item: &StoreItem) -> Option<String> {
+    match item {
+        StoreItem::Document {
+            body: DocumentBody::Text(text),
+            ..
+        } => Some(text.clone()),
+        StoreItem::Conversation { turns, .. } => turns
+            .iter()
+            .find(|turn| turn.role == Role::User)
+            .map(|turn| turn.text.clone()),
+        _ => None,
+    }
+}
+
+/// The first sentence of `text`'s prose, whitespace collapsed; markdown
+/// headings are skipped unless they are all there is. `None` when nothing is
+/// left.
+fn first_sentence(text: &str) -> Option<String> {
+    let (headings, prose): (Vec<&str>, Vec<&str>) = text
+        .lines()
+        .map(str::trim)
+        .filter(|line| !line.is_empty())
+        .partition(|line| line.starts_with('#'));
+    let lines = if prose.is_empty() { headings } else { prose };
+    let collapsed = lines
+        .iter()
+        .map(|line| line.trim_start_matches('#'))
+        .flat_map(|line| line.split_whitespace())
+        .collect::<Vec<_>>()
+        .join(" ");
+    let end = collapsed
+        .char_indices()
+        .find(|(_, c)| matches!(c, '.' | '!' | '?'))
+        .map_or(collapsed.len(), |(index, c)| index + c.len_utf8());
+    let sentence: String = collapsed[..end].chars().take(MAX_STATEMENT_CHARS).collect();
+    (!sentence.is_empty()).then_some(sentence)
+}
+
+#[cfg(test)]
+#[path = "distil_tests.rs"]
+mod tests;

crates/tinymemory-api/src/conformance/reference/distil_tests.rs

@@ -0,0 +1,73 @@
+//! The reference engine's toy belief distillation.
+
+use super::*;
+use crate::{MemoryMeta, Namespace, Reach, Turn};
+
+fn at(namespace: Namespace) -> MemoryMeta {
+    MemoryMeta {
+        namespace,
+        ..MemoryMeta::default()
+    }
+}
+
+#[test]
+fn distils_the_first_sentence_of_documents_and_user_turns() {
+    let items = vec![
+        StoreItem::document(
+            "# Refunds\n\nRefunds take five days. Ask support.",
+            at(Namespace::source("markdown")),
+        ),
+        StoreItem::Conversation {
+            turns: vec![
+                Turn::new(Role::Assistant, "Hello!"),
+                Turn::new(Role::User, "I live in Lagos. What is the weather?"),
+            ],
+            meta: at(Namespace::agent("support")),
+        },
+        StoreItem::learning(
+            "already a belief",
+            LearningKind::Fact,
+            0.9,
+            at(Namespace::ROOT),
+        ),
+    ];
+    let beliefs = distil(
+        &items,
+        &ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)),
+    );
+    let texts: Vec<String> = beliefs.iter().map(StoreItem::render_text).collect();
+    assert_eq!(texts, ["Refunds take five days.", "I live in Lagos."]);
+    let StoreItem::Learning { evidence, meta, .. } = &beliefs[1] else {
+        panic!("a belief is a learning");
+    };
+    assert_eq!(evidence.as_deref(), Some(items[1].fingerprint().as_str()));
+    assert_eq!(meta.namespace, Namespace::agent("support"));
+    assert_eq!(meta.tags, [CONSOLIDATED_TAG]);
+}
+
+#[test]
+fn honours_the_reach_and_the_kinds() {
+    let items = vec![
+        StoreItem::document("Pdf fact.", at(Namespace::source("pdf"))),
+        StoreItem::document("Notion fact.", at(Namespace::source("notion"))),
+    ];
+    let pdf_only = ConsolidateRequest::new(Reach::exact(Namespace::source("pdf")));
+    assert_eq!(distil(&items, &pdf_only).len(), 1);
+    let conversations_only =
+        ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)).kinds([ItemKind::Conversation]);
+    assert!(distil(&items, &conversations_only).is_empty());
+}
+
+#[test]
+fn skips_blank_text_and_caps_long_sentences() {
+    assert_eq!(first_sentence("  \n#  \n"), None);
+    assert_eq!(
+        first_sentence("# Only a title").as_deref(),
+        Some("Only a title")
+    );
+    let long = "word ".repeat(200);
+    assert_eq!(
+        first_sentence(&long).map(|s| s.chars().count()),
+        Some(MAX_STATEMENT_CHARS)
+    );
+}

crates/tinymemory-api/src/conformance/reference/mod.rs

@@ -4,16 +4,22 @@
 //! It is the suite's calibration subject: a failure against it means the
 //! assertion is wrong, not the engine. It serves every fetch mode, using a
 //! trivial keyword scorer and a deterministic toy vector (see `score`), and
-//! answers recall by quoting its best hybrid hits.
+//! answers recall by quoting its best hybrid hits. It consolidates on demand,
+//! distilling one toy belief per source item (see `distil`), so a host's
+//! whole memory lifecycle runs offline against it.
 
+mod distil;
 mod score;
 
 use std::sync::Mutex;
 
+pub use distil::CONSOLIDATED_TAG;
+
 use crate::{
-    Citation, EngineDescriptor, EngineHealth, Error, FetchMode, FetchPage, FetchRequest,
-    ForgetReport, ForgetTarget, Hit, ItemId, ListPage, ListRequest, MemoryEngine, MetaFilter,
-    RecallAnswer, RecallRequest, Result, StoreItem, StoreReceipt,
+    Citation, ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus, Consolidation,
+    EngineDescriptor, EngineHealth, Error, FetchMode, FetchPage, FetchRequest, ForgetReport,
+    ForgetTarget, Hit, ItemId, ListPage, ListRequest, MemoryEngine, MetaFilter, RecallAnswer,
+    RecallRequest, Result, StoreItem, StoreReceipt,
 };
 use async_trait::async_trait;
 
@@ -47,6 +53,7 @@ impl ReferenceEngine {
                 needs_key: false,
                 default_endpoint: None,
                 fetch_modes: FetchMode::ALL.to_vec(),
+                consolidation: Consolidation::OnDemand,
             },
             items: Mutex::new(Vec::new()),
         }
@@ -163,7 +170,11 @@ impl MemoryEngine for ReferenceEngine {
         req.validate()?;
         let hits = self.ranked(&req.query, req.mode, &req.filter)?;
         let (hits, next_cursor) = page(hits, req.cursor.as_deref(), req.limit)?;
-        Ok(FetchPage { hits, next_cursor })
+        Ok(FetchPage {
+            hits,
+            next_cursor,
+            beliefs: Vec::new(),
+        })
     }
 
     async fn store(&self, item: StoreItem) -> Result<StoreReceipt> {
@@ -194,6 +205,34 @@ impl MemoryEngine for ReferenceEngine {
         })
     }
 
+    /// Distils one belief per admitted document or conversation, at once:
+    /// the build is [`ConsolidateStatus::Completed`] on return.
+    async fn consolidate(&self, req: ConsolidateRequest) -> Result<ConsolidateReceipt> {
+        req.validate()?;
+        let mut items = self.items()?;
+        let beliefs = distil::distil(&items, &req);
+        let mut nodes: Vec<&crate::Namespace> = Vec::new();
+        for belief in &beliefs {
+            if !nodes.contains(&&belief.meta().namespace) {
+                nodes.push(&belief.meta().namespace);
+            }
+        }
+        let scopes = nodes.len() * req.admitted_kinds().len();
+        let built = beliefs.len();
+        for belief in beliefs {
+            let id = belief.fingerprint();
+            if !items.iter().any(|held| held.fingerprint() == id) {
+                items.push(belief);
+            }
+        }
+        Ok(ConsolidateReceipt {
+            status: ConsolidateStatus::Completed,
+            jobs: Vec::new(),
+            scopes,
+            built: Some(built),
+        })
+    }
+
     async fn list(&self, req: ListRequest) -> Result<ListPage> {
         req.validate()?;
         let matching: Vec<Hit> = self

crates/tinymemory-api/src/conformance/suite/checks.rs

@@ -18,13 +18,15 @@ pub(super) async fn all(ctx: &Ctx<'_>) -> Result<()> {
     super::explore::explore(ctx).await?;
     super::explore::get(ctx).await?;
     super::bulk::store_many(ctx).await?;
+    super::lifecycle::store_with(ctx).await?;
     fetch_filters(ctx).await?;
     unsupported_modes(ctx).await?;
     super::namespaces::namespaces(ctx).await?;
     empty_forget(ctx).await?;
     forget_by_id(ctx).await?;
     forget_by_filter(ctx).await?;
-    recall(ctx).await
+    recall(ctx).await?;
+    super::lifecycle::consolidate(ctx).await
 }
 
 /// Forgets everything the run stored and checks it is gone.