diff --git a/AGENTS.md b/AGENTS.md index 21b15a1f..c758179a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,8 @@ package it holds. There are exactly three, one per part of the memory layer: metadata, namespaces, errors, and (feature `conformance`) the suite every engine must pass. No I/O. - `crates/tinymemory-tools` — the **agent tool spec**: `MemoryTools` over any - engine, and the `context.md` compiler. + engine, holistic recall and the `context.md` compiler, and the agent memory + lifecycle (`AgentMemory`, `Brain`, `MemoryLayout`, background jobs). - `crates/tinymemory-integrations` — the **integrations**: the CortexDB engine and its registry, documents, sources, safety and the legacy v1 import, each a module behind a feature. diff --git a/Cargo.lock b/Cargo.lock index 66564ce3..583d3035 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1641,6 +1641,7 @@ version = "1.22.4" dependencies = [ "async-trait", "chrono", + "futures", "log", "serde", "serde_json", diff --git a/Cargo.toml b/Cargo.toml index 8cccb10a..72fe3f5c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -5,8 +5,8 @@ resolver = "2" # - `tinymemory-api`: the core contract (`MemoryEngine`, items, metadata, # namespaces, errors) and, behind `conformance`, the suite every engine must # pass. -# - `tinymemory-tools`: the agent-facing tool spec over any engine, and the -# `context.md` compiler. +# - `tinymemory-tools`: the agent-facing tool spec over any engine, holistic +# recall and the `context.md` compiler, and the agent memory lifecycle. # - `tinymemory-integrations`: everything that talks to the outside world — # the CortexDB engine and its registry, document conversion, source readers, # safety scrubbing and the legacy v1 import — each behind a feature. diff --git a/README.md b/README.md index f9aa59c1..3b6ab845 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,8 @@ namespace tree that keeps one tenant's or agent's memory apart from another's. - **Specified behaviour:** [`docs/specs/memory-v2.md`](docs/specs/memory-v2.md) is the source of truth for what the system does and why. +- **Integrating an agent:** [`docs/integration.md`](docs/integration.md) is + the guide for a host wiring its agent loop to the memory API. - **How it is built:** [`docs/architecture/README.md`](docs/architecture/README.md) has one page per concern: the contract, operations, namespaces, the CortexDB engine, the agent tools, the integrations and the test strategy. @@ -33,7 +35,9 @@ crates/ │ an in-memory reference engine ├── tinymemory-tools/ the agent surface over any engine: `MemoryTools` (seven │ model-callable tools with JSON Schemas and host-fixed -│ scoping) and the `context.md` compiler +│ scoping), holistic recall and the `context.md` compiler, +│ and the agent lifecycle (`AgentMemory`, `Brain`, +│ `MemoryLayout`, background jobs) └── tinymemory-integrations/ everything that touches the outside world, one module per feature: `cortex` (+ `registry`, `config`), `documents`, `sources`, `safety`, `import` @@ -58,11 +62,12 @@ else on request, so a host pays only for what it uses. | `cortex` (default) | `cortex`, `registry`, `config` | `CortexEngine` over both wires, `list_engines`, `build_engine`, `EngineCredential`, `MemoryConfig` | | `documents` | `documents` | Format sniffing and conversion to markdown, producing `StoreItem::Document` | | `documents-office` | `documents::OfficeConverter` | PDF, DOCX, PPTX and XLSX to markdown (implies `documents`) | +| `brain` | `brain` | Files into `tinymemory_tools::BrainDocument`s, by the source type their format implies (implies `documents`) | | `sources` | `sources` | Folder, file and conversation readers, Composio normalisers (implies `documents`) | | `sources-network` | `sources::fetch` and the network readers | GitHub, RSS and web-page readers and `fetch_url`, behind the SSRF guard (implies `sources`) | | `safety` | `safety` | Secret and PII scrubbing of a `StoreItem` | | `legacy-import` | `import` | Migrating a v1 (embedded TinyCortex) workspace into any engine | -| `full` | all of the above | `cortex`, `documents-office`, `sources-network`, `safety`, `legacy-import` | +| `full` | all of the above | `cortex`, `documents-office`, `brain`, `sources-network`, `safety`, `legacy-import` | Dependency weight per feature is tabulated in [`crates/tinymemory-integrations/README.md`](crates/tinymemory-integrations/README.md). @@ -136,6 +141,28 @@ credential, and a credentialed cleartext endpoint that is not loopback. To compile a `context.md` for the start of a session, call `tinymemory_tools::context::compile(&*engine, &ContextSpec::default())`. +### The agent lifecycle + +For memory around every turn, use `tinymemory_tools::AgentMemory`. It +implements a standard layout: a global brain of documents by source type, +each agent's conversations, and shared learnings. Call it at each point of +the agent loop: + +```rust,ignore +let memory = AgentMemory::new(engine.clone(), MemoryLayout::default(), "support-01")?; +let turn = memory.pre_turn(PreTurn::new("thread-1", 0, user_text)).await?; // log + recall, no indexing wait +let reply = llm.generate(&turn.pack.markdown, user_text).await; +let report = memory.post_turn(PostTurn::new("thread-1", 1, reply)).await?; // log +for job in report.jobs { queue.push(job) } // belief builds, off the turn +``` + +`Brain` ingests documents (with `tinymemory_integrations::brain::brain_document` +converting PDFs and markdown first). `start_session` and +`recall_for_compaction` cover session resume and prompt truncation. Run +`cargo run -p tinymemory-tools --example agent_loop` for the whole loop +offline, and see +[`docs/architecture/lifecycle.md`](docs/architecture/lifecycle.md). + ## Engines | Id | What | Fetch modes | diff --git a/crates/tinymemory-api/src/conformance/mod.rs b/crates/tinymemory-api/src/conformance/mod.rs index f562e353..2e03888c 100644 --- a/crates/tinymemory-api/src/conformance/mod.rs +++ b/crates/tinymemory-api/src/conformance/mod.rs @@ -33,5 +33,5 @@ pub mod reference; mod suite; pub use error::{Error, Result}; -pub use reference::{REFERENCE_ENGINE_ID, ReferenceEngine}; +pub use reference::{CONSOLIDATED_TAG, REFERENCE_ENGINE_ID, ReferenceEngine}; pub use suite::run; diff --git a/crates/tinymemory-api/src/conformance/reference/distil.rs b/crates/tinymemory-api/src/conformance/reference/distil.rs new file mode 100644 index 00000000..c433d4ab --- /dev/null +++ b/crates/tinymemory-api/src/conformance/reference/distil.rs @@ -0,0 +1,88 @@ +//! The reference engine's consolidation: a toy belief per source item. +//! +//! Real engines distil beliefs with a model. The reference engine needs only +//! something deterministic and obvious: each document or conversation an +//! [`ConsolidateRequest`] admits yields one [`LearningKind::Fact`] — the +//! first sentence of the document's prose, or of the conversation's first +//! user turn — stored at the source item's own node with the source's metadata, a +//! `consolidated` tag, and the source's id as evidence. Consolidating twice is +//! a replay, so it never duplicates a belief. + +use crate::{ConsolidateRequest, DocumentBody, ItemKind, LearningKind, Role, StoreItem}; + +/// Tag on every belief the reference engine distils. +pub const CONSOLIDATED_TAG: &str = "consolidated"; + +/// Confidence of a distilled belief: a first sentence is a weak signal. +const BELIEF_CONFIDENCE: f32 = 0.5; + +/// Longest statement a belief keeps, in characters. +const MAX_STATEMENT_CHARS: usize = 240; + +/// The beliefs `request` distils from `items`, in item order. +pub(super) fn distil(items: &[StoreItem], request: &ConsolidateRequest) -> Vec { + let kinds = request.admitted_kinds(); + items + .iter() + .filter(|item| item.kind() != ItemKind::Learning && kinds.contains(&item.kind())) + .filter(|item| request.reach.admits(&item.meta().namespace)) + .filter_map(|item| { + let statement = first_sentence(&source_text(item)?)?; + let mut meta = item.meta().clone(); + if !meta.tags.iter().any(|tag| tag == CONSOLIDATED_TAG) { + meta.tags.push(CONSOLIDATED_TAG.to_string()); + } + Some(StoreItem::Learning { + text: statement, + kind: LearningKind::Fact, + confidence: BELIEF_CONFIDENCE, + evidence: Some(item.fingerprint()), + meta, + }) + }) + .collect() +} + +/// The text a belief is drawn from: a document's body, or a conversation's +/// first user turn. +fn source_text(item: &StoreItem) -> Option { + match item { + StoreItem::Document { + body: DocumentBody::Text(text), + .. + } => Some(text.clone()), + StoreItem::Conversation { turns, .. } => turns + .iter() + .find(|turn| turn.role == Role::User) + .map(|turn| turn.text.clone()), + _ => None, + } +} + +/// The first sentence of `text`'s prose, whitespace collapsed; markdown +/// headings are skipped unless they are all there is. `None` when nothing is +/// left. +fn first_sentence(text: &str) -> Option { + let (headings, prose): (Vec<&str>, Vec<&str>) = text + .lines() + .map(str::trim) + .filter(|line| !line.is_empty()) + .partition(|line| line.starts_with('#')); + let lines = if prose.is_empty() { headings } else { prose }; + let collapsed = lines + .iter() + .map(|line| line.trim_start_matches('#')) + .flat_map(|line| line.split_whitespace()) + .collect::>() + .join(" "); + let end = collapsed + .char_indices() + .find(|(_, c)| matches!(c, '.' | '!' | '?')) + .map_or(collapsed.len(), |(index, c)| index + c.len_utf8()); + let sentence: String = collapsed[..end].chars().take(MAX_STATEMENT_CHARS).collect(); + (!sentence.is_empty()).then_some(sentence) +} + +#[cfg(test)] +#[path = "distil_tests.rs"] +mod tests; diff --git a/crates/tinymemory-api/src/conformance/reference/distil_tests.rs b/crates/tinymemory-api/src/conformance/reference/distil_tests.rs new file mode 100644 index 00000000..788ae1e8 --- /dev/null +++ b/crates/tinymemory-api/src/conformance/reference/distil_tests.rs @@ -0,0 +1,73 @@ +//! The reference engine's toy belief distillation. + +use super::*; +use crate::{MemoryMeta, Namespace, Reach, Turn}; + +fn at(namespace: Namespace) -> MemoryMeta { + MemoryMeta { + namespace, + ..MemoryMeta::default() + } +} + +#[test] +fn distils_the_first_sentence_of_documents_and_user_turns() { + let items = vec![ + StoreItem::document( + "# Refunds\n\nRefunds take five days. Ask support.", + at(Namespace::source("markdown")), + ), + StoreItem::Conversation { + turns: vec![ + Turn::new(Role::Assistant, "Hello!"), + Turn::new(Role::User, "I live in Lagos. What is the weather?"), + ], + meta: at(Namespace::agent("support")), + }, + StoreItem::learning( + "already a belief", + LearningKind::Fact, + 0.9, + at(Namespace::ROOT), + ), + ]; + let beliefs = distil( + &items, + &ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)), + ); + let texts: Vec = beliefs.iter().map(StoreItem::render_text).collect(); + assert_eq!(texts, ["Refunds take five days.", "I live in Lagos."]); + let StoreItem::Learning { evidence, meta, .. } = &beliefs[1] else { + panic!("a belief is a learning"); + }; + assert_eq!(evidence.as_deref(), Some(items[1].fingerprint().as_str())); + assert_eq!(meta.namespace, Namespace::agent("support")); + assert_eq!(meta.tags, [CONSOLIDATED_TAG]); +} + +#[test] +fn honours_the_reach_and_the_kinds() { + let items = vec![ + StoreItem::document("Pdf fact.", at(Namespace::source("pdf"))), + StoreItem::document("Notion fact.", at(Namespace::source("notion"))), + ]; + let pdf_only = ConsolidateRequest::new(Reach::exact(Namespace::source("pdf"))); + assert_eq!(distil(&items, &pdf_only).len(), 1); + let conversations_only = + ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)).kinds([ItemKind::Conversation]); + assert!(distil(&items, &conversations_only).is_empty()); +} + +#[test] +fn skips_blank_text_and_caps_long_sentences() { + assert_eq!(first_sentence(" \n# \n"), None); + assert_eq!( + first_sentence("# Only a title").as_deref(), + Some("Only a title") + ); + let long = "word ".repeat(200); + assert_eq!( + first_sentence(&long).map(|s| s.chars().count()), + Some(MAX_STATEMENT_CHARS) + ); +} diff --git a/crates/tinymemory-api/src/conformance/reference/mod.rs b/crates/tinymemory-api/src/conformance/reference/mod.rs index 68957e6f..d7700e81 100644 --- a/crates/tinymemory-api/src/conformance/reference/mod.rs +++ b/crates/tinymemory-api/src/conformance/reference/mod.rs @@ -4,16 +4,22 @@ //! It is the suite's calibration subject: a failure against it means the //! assertion is wrong, not the engine. It serves every fetch mode, using a //! trivial keyword scorer and a deterministic toy vector (see `score`), and -//! answers recall by quoting its best hybrid hits. +//! answers recall by quoting its best hybrid hits. It consolidates on demand, +//! distilling one toy belief per source item (see `distil`), so a host's +//! whole memory lifecycle runs offline against it. +mod distil; mod score; use std::sync::Mutex; +pub use distil::CONSOLIDATED_TAG; + use crate::{ - Citation, EngineDescriptor, EngineHealth, Error, FetchMode, FetchPage, FetchRequest, - ForgetReport, ForgetTarget, Hit, ItemId, ListPage, ListRequest, MemoryEngine, MetaFilter, - RecallAnswer, RecallRequest, Result, StoreItem, StoreReceipt, + Citation, ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus, Consolidation, + EngineDescriptor, EngineHealth, Error, FetchMode, FetchPage, FetchRequest, ForgetReport, + ForgetTarget, Hit, ItemId, ListPage, ListRequest, MemoryEngine, MetaFilter, RecallAnswer, + RecallRequest, Result, StoreItem, StoreReceipt, }; use async_trait::async_trait; @@ -47,6 +53,7 @@ impl ReferenceEngine { needs_key: false, default_endpoint: None, fetch_modes: FetchMode::ALL.to_vec(), + consolidation: Consolidation::OnDemand, }, items: Mutex::new(Vec::new()), } @@ -163,7 +170,11 @@ impl MemoryEngine for ReferenceEngine { req.validate()?; let hits = self.ranked(&req.query, req.mode, &req.filter)?; let (hits, next_cursor) = page(hits, req.cursor.as_deref(), req.limit)?; - Ok(FetchPage { hits, next_cursor }) + Ok(FetchPage { + hits, + next_cursor, + beliefs: Vec::new(), + }) } async fn store(&self, item: StoreItem) -> Result { @@ -194,6 +205,34 @@ impl MemoryEngine for ReferenceEngine { }) } + /// Distils one belief per admitted document or conversation, at once: + /// the build is [`ConsolidateStatus::Completed`] on return. + async fn consolidate(&self, req: ConsolidateRequest) -> Result { + req.validate()?; + let mut items = self.items()?; + let beliefs = distil::distil(&items, &req); + let mut nodes: Vec<&crate::Namespace> = Vec::new(); + for belief in &beliefs { + if !nodes.contains(&&belief.meta().namespace) { + nodes.push(&belief.meta().namespace); + } + } + let scopes = nodes.len() * req.admitted_kinds().len(); + let built = beliefs.len(); + for belief in beliefs { + let id = belief.fingerprint(); + if !items.iter().any(|held| held.fingerprint() == id) { + items.push(belief); + } + } + Ok(ConsolidateReceipt { + status: ConsolidateStatus::Completed, + jobs: Vec::new(), + scopes, + built: Some(built), + }) + } + async fn list(&self, req: ListRequest) -> Result { req.validate()?; let matching: Vec = self diff --git a/crates/tinymemory-api/src/conformance/suite/checks.rs b/crates/tinymemory-api/src/conformance/suite/checks.rs index b83667d5..9c0f7914 100644 --- a/crates/tinymemory-api/src/conformance/suite/checks.rs +++ b/crates/tinymemory-api/src/conformance/suite/checks.rs @@ -18,13 +18,15 @@ pub(super) async fn all(ctx: &Ctx<'_>) -> Result<()> { super::explore::explore(ctx).await?; super::explore::get(ctx).await?; super::bulk::store_many(ctx).await?; + super::lifecycle::store_with(ctx).await?; fetch_filters(ctx).await?; unsupported_modes(ctx).await?; super::namespaces::namespaces(ctx).await?; empty_forget(ctx).await?; forget_by_id(ctx).await?; forget_by_filter(ctx).await?; - recall(ctx).await + recall(ctx).await?; + super::lifecycle::consolidate(ctx).await } /// Forgets everything the run stored and checks it is gone. diff --git a/crates/tinymemory-api/src/conformance/suite/lifecycle.rs b/crates/tinymemory-api/src/conformance/suite/lifecycle.rs new file mode 100644 index 00000000..94ece066 --- /dev/null +++ b/crates/tinymemory-api/src/conformance/suite/lifecycle.rs @@ -0,0 +1,176 @@ +//! The lifecycle checks: the hot-path write and consolidation. +//! +//! - `store_with` — a [`WaitFor::Visible`] store is listed on return, exactly +//! as `store`; an [`WaitFor::Accepted`] store answers with the item's own +//! id; and a repeat of a visible item is a replay. +//! - `consolidate` — an invalid request is refused, and a valid one answers +//! as the descriptor promises: [`Consolidation::None`] refuses with +//! `Unsupported`, [`Consolidation::OnDemand`] starts or completes a build, +//! and [`Consolidation::Scheduled`] acknowledges without one. Then +//! `beliefs` refuses a zero limit, and every belief it returns is a +//! learning tagged [`BELIEF_TAG`] within the reach asked for. + +use crate::{ + BELIEF_TAG, BeliefsRequest, ConsolidateRequest, ConsolidateStatus, Consolidation, + Error as ApiError, ItemKind, Namespace, Reach, WaitFor, WriteOptions, +}; + +use super::{Ctx, ensure}; +use crate::conformance::error::{Error, Result}; + +pub(super) async fn store_with(ctx: &Ctx<'_>) -> Result<()> { + const CHECK: &str = "store_with"; + let visible = ctx.run.document("store-with-visible", ctx.run.meta()); + let receipt = ctx + .call( + CHECK, + ctx.engine + .store_with(visible.clone(), WriteOptions::visible()), + ) + .await?; + ensure(CHECK, receipt.id.as_str() == visible.fingerprint(), || { + format!( + "a visible store answered `{}` for an item fingerprinted `{}`", + receipt.id.as_str(), + visible.fingerprint() + ) + })?; + let listed = ctx.list_all(CHECK, &ctx.run.filter()).await?; + ensure(CHECK, listed.iter().any(|hit| hit.id == receipt.id), || { + "a store waiting for visibility was not listed on return".to_string() + })?; + let again = ctx + .call( + CHECK, + ctx.engine.store_with(visible, WriteOptions::visible()), + ) + .await?; + ensure(CHECK, again.replayed, || { + "storing a visible item again was not a replay".to_string() + })?; + + let accepted = ctx.run.document("store-with-accepted", ctx.run.meta()); + let receipt = ctx + .call( + CHECK, + ctx.engine.store_with( + accepted.clone(), + WriteOptions { + wait: WaitFor::Accepted, + }, + ), + ) + .await?; + ensure(CHECK, receipt.id.as_str() == accepted.fingerprint(), || { + format!( + "an accepted store answered `{}` for an item fingerprinted `{}`", + receipt.id.as_str(), + accepted.fingerprint() + ) + }) +} + +pub(super) async fn consolidate(ctx: &Ctx<'_>) -> Result<()> { + const CHECK: &str = "consolidate"; + let node: Namespace = format!("agent:{}-beliefs", ctx.run.marker) + .parse() + .map_err(|source| Error::Engine { + check: CHECK, + source, + })?; + let mut meta = ctx.run.meta(); + meta.namespace = node.clone(); + ctx.call( + CHECK, + ctx.engine + .store(ctx.run.document("consolidate. A fact to build on.", meta)), + ) + .await?; + + let invalid = ConsolidateRequest::new(Reach::exact(node.clone())) + .kinds([ItemKind::Document, ItemKind::Document]); + let refused = ctx.engine.consolidate(invalid).await; + ensure( + CHECK, + matches!(refused, Err(ApiError::InvalidRequest(_))), + || format!("a request naming a kind twice was not refused: {refused:?}"), + )?; + + let request = ConsolidateRequest::new(Reach::exact(node.clone())).kinds([ItemKind::Document]); + let outcome = ctx.engine.consolidate(request).await; + let promised = ctx.engine.descriptor().consolidation; + answers_as_promised(promised, outcome)?; + beliefs(ctx, node).await +} + +/// A consolidation's answer against what the descriptor promised. +fn answers_as_promised( + promised: Consolidation, + outcome: std::result::Result, +) -> Result<()> { + const CHECK: &str = "consolidate"; + match (promised, outcome) { + (Consolidation::None, Err(ApiError::Unsupported(_))) => Ok(()), + (Consolidation::None, other) => Err(Error::Check { + check: CHECK, + detail: format!("an engine declaring no consolidation answered {other:?}"), + }), + (_, Err(source)) => Err(Error::Engine { + check: CHECK, + source, + }), + (Consolidation::OnDemand, Ok(receipt)) => ensure( + CHECK, + matches!( + receipt.status, + ConsolidateStatus::Started | ConsolidateStatus::Completed + ), + || format!("an on-demand engine answered {:?}", receipt.status), + ), + (Consolidation::Scheduled, Ok(receipt)) => ensure( + CHECK, + receipt.status == ConsolidateStatus::Scheduled, + || format!("a scheduled engine answered {:?}", receipt.status), + ), + } +} + +/// Beliefs read back after a build: a malformed request is refused, and +/// every belief is a tagged learning within the reach asked for. +async fn beliefs(ctx: &Ctx<'_>, node: Namespace) -> Result<()> { + const CHECK: &str = "consolidate"; + let reach = Reach::exact(node); + let refused = ctx + .engine + .beliefs(BeliefsRequest::new(reach.clone(), 0)) + .await; + ensure( + CHECK, + matches!(refused, Err(ApiError::InvalidRequest(_))), + || format!("a beliefs read with a zero limit was not refused: {refused:?}"), + )?; + let held = ctx + .call( + CHECK, + ctx.engine + .beliefs(BeliefsRequest::new(reach.clone(), 10).query("A fact to build on")), + ) + .await?; + ensure(CHECK, held.len() <= 10, || { + format!("a beliefs read for 10 answered {}", held.len()) + })?; + for belief in held { + ensure( + CHECK, + belief.kind == ItemKind::Learning + && belief.meta.tags.iter().any(|tag| tag == BELIEF_TAG) + && reach.admits(&belief.meta.namespace), + || { + format!( + "a belief must be a learning tagged `{BELIEF_TAG}` within its reach: {belief:?}" + ) + }, + )?; + } + Ok(()) +} diff --git a/crates/tinymemory-api/src/conformance/suite/mod.rs b/crates/tinymemory-api/src/conformance/suite/mod.rs index a79a99d0..2cd268fa 100644 --- a/crates/tinymemory-api/src/conformance/suite/mod.rs +++ b/crates/tinymemory-api/src/conformance/suite/mod.rs @@ -14,19 +14,24 @@ //! their listing, with an unknown id left out. //! 6. `store_many` — a batch stores in order, every item is listed on //! return, a repeat is all replays, and an empty batch is refused. -//! 7. `fetch_filters` — for every declared fetch mode, a filter on each +//! 7. `store_with` — a store waiting for visibility is listed on return and +//! replays on a repeat; one only waiting for acceptance answers with the +//! item's own id. +//! 8. `fetch_filters` — for every declared fetch mode, a filter on each //! metadata field selects exactly the item carrying it (and `list` agrees). -//! 8. `unsupported_modes` — every undeclared fetch mode fails `Unsupported`. -//! 9. `namespaces` — items at the root, two sibling agents and a sub-agent: -//! each reach (own and inherited, exact, subtree) lists exactly its nodes, -//! never a sibling's; `get` and `fetch` honour the reach; the same text in -//! two namespaces is two items; the namespace facet counts each node; and -//! a forget scoped to one node removes only it. -//! 10. `empty_forget` — a forget with no ids or an empty filter is refused and +//! 9. `unsupported_modes` — every undeclared fetch mode fails `Unsupported`. +//! 10. `namespaces` — items at the root, two sibling agents and a sub-agent: +//! each reach (own and inherited, exact, subtree) lists exactly its nodes, +//! never a sibling's; `get` and `fetch` honour the reach; the same text in +//! two namespaces is two items; the namespace facet counts each node; and +//! a forget scoped to one node removes only it. +//! 11. `empty_forget` — a forget with no ids or an empty filter is refused and //! removes nothing. -//! 11. `forget_by_id` and `forget_by_filter` — forgotten items stop listing and +//! 12. `forget_by_id` and `forget_by_filter` — forgotten items stop listing and //! are counted; others stay. -//! 12. `recall` — an answer cites items that resolve through `list`. +//! 13. `recall` — an answer cites items that resolve through `list`. +//! 14. `consolidate` — a malformed request is refused, and a valid one is +//! answered as the descriptor's `consolidation` promises. //! //! Finally the run's items are forgotten by filter and must be gone. @@ -34,6 +39,7 @@ mod bulk; mod checks; mod explore; mod fixtures; +mod lifecycle; mod namespaces; use std::collections::HashSet; diff --git a/crates/tinymemory-api/src/consolidate/mod.rs b/crates/tinymemory-api/src/consolidate/mod.rs new file mode 100644 index 00000000..44f8bc56 --- /dev/null +++ b/crates/tinymemory-api/src/consolidate/mod.rs @@ -0,0 +1,214 @@ +//! Consolidation: asking an engine to distil what it holds into beliefs. +//! +//! Engines that keep a cognitive layer (CortexDB's beliefs and facts) build +//! it from raw events — documents, conversation turns — with a model, which +//! takes seconds to minutes. That work never belongs on an agent's live turn, +//! so the contract only lets a host *ask* for it: +//! [`MemoryEngine::consolidate`](crate::MemoryEngine::consolidate) names a +//! [`ConsolidateRequest`] (which part of the tree, which kinds) and returns as +//! soon as the engine has taken the job, or once a quick build is done. What +//! it builds comes back through the engine's reads (on CortexDB, the derived +//! layers its answer route reads). +//! +//! An engine says how it consolidates in +//! [`EngineDescriptor::consolidation`](crate::EngineDescriptor::consolidation): +//! +//! - [`Consolidation::None`] — it does not; `consolidate` fails +//! [`Error::Unsupported`]. +//! - [`Consolidation::OnDemand`] — `consolidate` starts (or runs) a build and +//! answers [`ConsolidateStatus::Started`] or +//! [`ConsolidateStatus::Completed`]. +//! - [`Consolidation::Scheduled`] — the engine consolidates on its own +//! schedule; `consolidate` acknowledges with +//! [`ConsolidateStatus::Scheduled`] and does nothing more. +//! +//! **Reading beliefs.** An engine that keeps what it builds apart from its +//! stored items (CortexDB's belief layer) serves it through +//! [`MemoryEngine::beliefs`](crate::MemoryEngine::beliefs): a +//! [`BeliefsRequest`] names a reach, an optional query and a limit, and the +//! answer is learning hits tagged [`BELIEF_TAG`]. They are not stored items, +//! so they cannot be listed, fetched or forgotten by id; forgetting the items +//! they were built from removes them. An engine whose beliefs are ordinary +//! learning items (the reference engine) has nothing to add and keeps the +//! default, which holds none. + +use serde::{Deserialize, Serialize}; + +use crate::error::{Error, Result}; +use crate::item::ItemKind; +use crate::namespace::Reach; + +/// How an engine turns raw memory into beliefs. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Consolidation { + /// It does not consolidate. + #[default] + None, + /// [`crate::MemoryEngine::consolidate`] starts a build. + OnDemand, + /// The engine consolidates on its own schedule. + Scheduled, +} + +/// What to consolidate. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct ConsolidateRequest { + /// The part of the namespace tree to consolidate. A subtree reach + /// consolidates every node below `at`. + pub reach: Reach, + /// The item kinds whose items feed the build; empty means every kind. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub kinds: Vec, +} + +impl ConsolidateRequest { + /// Consolidates everything `reach` admits. + #[must_use] + pub fn new(reach: Reach) -> Self { + Self { + reach, + kinds: Vec::new(), + } + } + + /// Restricts the build to items of `kinds`. + #[must_use] + pub fn kinds(mut self, kinds: impl IntoIterator) -> Self { + self.kinds = kinds.into_iter().collect(); + self + } + + /// The kinds the build reads: [`ItemKind::ALL`] when none were named. + #[must_use] + pub fn admitted_kinds(&self) -> Vec { + if self.kinds.is_empty() { + ItemKind::ALL.to_vec() + } else { + self.kinds.clone() + } + } + + /// Checks the request. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] when a kind is named twice. + pub fn validate(&self) -> Result<()> { + let mut seen = Vec::with_capacity(self.kinds.len()); + for kind in &self.kinds { + if seen.contains(kind) { + return Err(Error::InvalidRequest(format!( + "consolidate names the kind `{}` twice", + kind.as_str() + ))); + } + seen.push(*kind); + } + Ok(()) + } +} + +/// The tag on every hit [`MemoryEngine::beliefs`](crate::MemoryEngine::beliefs) +/// returns. +pub const BELIEF_TAG: &str = "belief"; + +/// Which beliefs to read (see [`MemoryEngine::beliefs`](crate::MemoryEngine::beliefs)). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct BeliefsRequest { + /// The part of the namespace tree whose beliefs to read. + pub reach: Reach, + /// What the beliefs should be relevant to. Without one, the most + /// confident come first, then the newest. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub query: Option, + /// The most beliefs to return. + pub limit: usize, +} + +impl BeliefsRequest { + /// At most `limit` beliefs within `reach`, most confident first. + #[must_use] + pub fn new(reach: Reach, limit: usize) -> Self { + Self { + reach, + query: None, + limit, + } + } + + /// Ranks the beliefs for `query`. + #[must_use] + pub fn query(mut self, query: impl Into) -> Self { + self.query = Some(query.into()); + self + } + + /// Checks the request. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a zero limit or a blank query. + pub fn validate(&self) -> Result<()> { + if self.limit == 0 { + return Err(Error::InvalidRequest( + "a beliefs limit must be positive".to_string(), + )); + } + if self + .query + .as_deref() + .is_some_and(|query| query.trim().is_empty()) + { + return Err(Error::InvalidRequest( + "a beliefs query must not be blank".to_string(), + )); + } + Ok(()) + } +} + +/// How far a consolidation got before the call returned. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ConsolidateStatus { + /// A build was started and runs in the background. + Started, + /// The engine consolidates on its own schedule; nothing was started. + Scheduled, + /// The build ran to completion within the call. + Completed, +} + +/// What [`crate::MemoryEngine::consolidate`] did. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ConsolidateReceipt { + /// How far it got. + pub status: ConsolidateStatus, + /// The engine's handles for the builds it started, when it names them. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub jobs: Vec, + /// How many engine-side scopes (nodes × kinds) the request covered. + pub scopes: usize, + /// How many beliefs the build produced, when it ran within the call and + /// the engine reports the count. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub built: Option, +} + +impl ConsolidateReceipt { + /// A receipt for an engine that consolidates on its own schedule. + #[must_use] + pub fn scheduled() -> Self { + Self { + status: ConsolidateStatus::Scheduled, + jobs: Vec::new(), + scopes: 0, + built: None, + } + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-api/src/consolidate/mod_tests.rs b/crates/tinymemory-api/src/consolidate/mod_tests.rs new file mode 100644 index 00000000..53c99c20 --- /dev/null +++ b/crates/tinymemory-api/src/consolidate/mod_tests.rs @@ -0,0 +1,53 @@ +//! Consolidation requests and receipts. + +use super::*; +use crate::namespace::Namespace; + +#[test] +fn a_beliefs_request_needs_a_limit_and_a_real_query() { + let request = BeliefsRequest::new(Reach::subtree(Namespace::ROOT), 5); + request.validate().unwrap(); + request.clone().query("refunds").validate().unwrap(); + assert!(matches!( + BeliefsRequest::new(Reach::default(), 0).validate(), + Err(Error::InvalidRequest(_)) + )); + assert!(matches!( + request.query(" ").validate(), + Err(Error::InvalidRequest(_)) + )); +} + +#[test] +fn an_unnamed_kind_list_admits_every_kind() { + let request = ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)); + assert_eq!(request.admitted_kinds(), ItemKind::ALL.to_vec()); + let narrowed = request.kinds([ItemKind::Conversation]); + assert_eq!(narrowed.admitted_kinds(), vec![ItemKind::Conversation]); + narrowed.validate().unwrap(); +} + +#[test] +fn rejects_a_kind_named_twice() { + let request = + ConsolidateRequest::new(Reach::default()).kinds([ItemKind::Document, ItemKind::Document]); + assert!(matches!(request.validate(), Err(Error::InvalidRequest(_)))); +} + +#[test] +fn round_trips_through_json() { + let request = ConsolidateRequest::new(Reach::subtree(Namespace::source("pdf"))) + .kinds([ItemKind::Document]); + let json = serde_json::to_value(&request).unwrap(); + assert_eq!(json["reach"]["at"], "source:pdf"); + assert_eq!(json["kinds"], serde_json::json!(["document"])); + let back: ConsolidateRequest = serde_json::from_value(json).unwrap(); + assert_eq!(back, request); + + let receipt = ConsolidateReceipt::scheduled(); + assert_eq!( + serde_json::to_value(&receipt).unwrap(), + serde_json::json!({ "status": "scheduled", "scopes": 0 }) + ); + assert_eq!(Consolidation::default(), Consolidation::None); +} diff --git a/crates/tinymemory-api/src/engine/mod.rs b/crates/tinymemory-api/src/engine/mod.rs index 254d2cf8..6fa51509 100644 --- a/crates/tinymemory-api/src/engine/mod.rs +++ b/crates/tinymemory-api/src/engine/mod.rs @@ -4,6 +4,7 @@ use async_trait::async_trait; use serde::Serialize; +use crate::consolidate::{BeliefsRequest, ConsolidateReceipt, ConsolidateRequest, Consolidation}; use crate::error::{Error, Result}; use crate::explore::{ExplorePage, ExploreRequest, GetRequest, explore_by_listing, get_by_listing}; use crate::item::{StoreItem, StoreReceipt}; @@ -11,6 +12,7 @@ use crate::query::{ FetchMode, FetchPage, FetchRequest, ForgetReport, ForgetTarget, Hit, ListPage, ListRequest, RecallAnswer, RecallRequest, }; +use crate::write::WriteOptions; /// A memory engine: recall, fetch, store, forget and list over typed items. /// @@ -48,6 +50,21 @@ pub trait MemoryEngine: Send + Sync { /// Invalid items, and the engine's own failures. async fn store(&self, item: StoreItem) -> Result; + /// Stores one item, returning as soon as `options` allows (see + /// [`crate::write`]). With [`crate::WaitFor::Visible`] this is exactly + /// [`MemoryEngine::store`]; with [`crate::WaitFor::Accepted`] an engine + /// may return once the item is durably accepted, before it is readable. + /// + /// The default serves every option as `store`, which is always correct. + /// + /// # Errors + /// + /// As [`MemoryEngine::store`]. + async fn store_with(&self, item: StoreItem, options: WriteOptions) -> Result { + let _ = options; + self.store(item).await + } + /// Stores several items, in order: bulk ingestion (imports, backfills, /// source syncs). /// @@ -115,6 +132,45 @@ pub trait MemoryEngine: Send + Sync { async fn get(&self, req: GetRequest) -> Result> { get_by_listing(self, req).await } + + /// Asks the engine to distil what `req` covers into beliefs, returning + /// once the job is taken, not done (see [`crate::consolidate`]). What it + /// builds surfaces through ordinary reads. + /// + /// The default refuses: an engine declaring + /// [`Consolidation::OnDemand`] or [`Consolidation::Scheduled`] overrides + /// it. + /// + /// # Errors + /// + /// [`Error::Unsupported`] when the engine does not consolidate, invalid + /// requests, and the engine's own failures. + async fn consolidate(&self, req: ConsolidateRequest) -> Result { + req.validate()?; + Err(Error::Unsupported(format!( + "engine `{}` does not consolidate memory", + self.descriptor().id + ))) + } + + /// The beliefs the engine built and keeps apart from its stored items, + /// within `req.reach`: ranked for `req.query`, or most confident and + /// then newest without one (see [`crate::consolidate`]). + /// + /// Each is a [`crate::ItemKind::Learning`] hit tagged + /// [`crate::BELIEF_TAG`], at the node its sources live at. It is not a + /// stored item: it cannot be listed, fetched or forgotten by id. + /// + /// The default holds none: an engine whose beliefs are ordinary + /// learning items, or that builds none, keeps it. + /// + /// # Errors + /// + /// Invalid requests, and the engine's own failures. + async fn beliefs(&self, req: BeliefsRequest) -> Result> { + req.validate()?; + Ok(Vec::new()) + } } /// Most items one [`MemoryEngine::store_many`] call may take. @@ -155,6 +211,9 @@ pub struct EngineDescriptor { pub default_endpoint: Option<&'static str>, /// The fetch modes the engine serves. pub fetch_modes: Vec, + /// How the engine turns raw memory into beliefs (see + /// [`MemoryEngine::consolidate`]). + pub consolidation: Consolidation, } impl EngineDescriptor { diff --git a/crates/tinymemory-api/src/engine/mod_tests.rs b/crates/tinymemory-api/src/engine/mod_tests.rs index b2a9224b..2df56ba6 100644 --- a/crates/tinymemory-api/src/engine/mod_tests.rs +++ b/crates/tinymemory-api/src/engine/mod_tests.rs @@ -12,6 +12,7 @@ fn descriptor(modes: Vec) -> EngineDescriptor { needs_key: false, default_endpoint: None, fetch_modes: modes, + consolidation: Consolidation::None, } } diff --git a/crates/tinymemory-api/src/explore/mod_tests.rs b/crates/tinymemory-api/src/explore/mod_tests.rs index b207bc3c..19720cec 100644 --- a/crates/tinymemory-api/src/explore/mod_tests.rs +++ b/crates/tinymemory-api/src/explore/mod_tests.rs @@ -7,6 +7,7 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use async_trait::async_trait; +use crate::consolidate::Consolidation; use crate::engine::{EngineDescriptor, EngineHealth}; use crate::item::{StoreItem, StoreReceipt}; use crate::meta::{SourceRef, ToolCallRef}; @@ -33,6 +34,7 @@ impl Paging { needs_key: false, default_endpoint: None, fetch_modes: Vec::new(), + consolidation: Consolidation::None, }, hits, pages: AtomicUsize::new(0), diff --git a/crates/tinymemory-api/src/lib.rs b/crates/tinymemory-api/src/lib.rs index 6f023e5b..4d59cc9e 100644 --- a/crates/tinymemory-api/src/lib.rs +++ b/crates/tinymemory-api/src/lib.rs @@ -14,6 +14,11 @@ //! [`MemoryEngine::get`] for explorers: counts of stored items per metadata //! [`Facet`], and items read whole by id ([`explore`]). //! +//! A live agent turn stores with [`MemoryEngine::store_with`] and +//! [`WaitFor::Accepted`] so it never waits on indexing, and a host asks an +//! engine to distil beliefs off the hot path with +//! [`MemoryEngine::consolidate`] ([`consolidate`]). +//! //! Every item lives at one [`Namespace`] node (the root, an agent, a team, a //! nested sub-agent); a [`Reach`] in the filter says which nodes a read sees //! ([`namespace`]). @@ -45,6 +50,7 @@ #[cfg(feature = "conformance")] pub mod conformance; +pub mod consolidate; pub mod engine; pub mod error; pub mod explore; @@ -52,7 +58,12 @@ pub mod item; pub mod meta; pub mod namespace; pub mod query; +pub mod write; +pub use consolidate::{ + BELIEF_TAG, BeliefsRequest, ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus, + Consolidation, +}; pub use engine::{EngineDescriptor, EngineHealth, MAX_STORE_MANY, MemoryEngine, validate_many}; pub use error::{Error, Result}; pub use explore::{ @@ -66,6 +77,7 @@ pub use query::{ Citation, FetchMode, FetchPage, FetchRequest, ForgetReport, ForgetTarget, Hit, ListPage, ListRequest, RecallAnswer, RecallRequest, }; +pub use write::{WaitFor, WriteOptions}; /// Re-exported so engines and hosts name the same `async_trait` and `chrono` /// the contract was compiled with. diff --git a/crates/tinymemory-api/src/namespace/mod.rs b/crates/tinymemory-api/src/namespace/mod.rs index 1f66c7e1..6c10e77e 100644 --- a/crates/tinymemory-api/src/namespace/mod.rs +++ b/crates/tinymemory-api/src/namespace/mod.rs @@ -47,10 +47,14 @@ pub enum SegmentKind { Workspace, /// A project. Project, + /// A source of knowledge (`source:pdf`, `source:notion`): where a shared + /// document came from, so a brain can hold each source type apart. + Source, } impl SegmentKind { - /// The stable wire prefix (`agent`, `team`, `user`, `ws`, `project`). + /// The stable wire prefix (`agent`, `team`, `user`, `ws`, `project`, + /// `source`). #[must_use] pub fn as_str(self) -> &'static str { match self { @@ -59,6 +63,7 @@ impl SegmentKind { Self::User => "user", Self::Workspace => "ws", Self::Project => "project", + Self::Source => "source", } } @@ -69,6 +74,7 @@ impl SegmentKind { "user" => Ok(Self::User), "ws" => Ok(Self::Workspace), "project" => Ok(Self::Project), + "source" => Ok(Self::Source), _ => Err(Error::InvalidRequest(format!( "`{value}` is not a namespace segment kind" ))), @@ -190,6 +196,25 @@ impl Namespace { Self(vec![Segment::sanitized(SegmentKind::Agent, id)]) } + /// The node for one knowledge source directly under the root, its id + /// sanitized ([`Segment::sanitized`]). + #[must_use] + pub fn source(id: &str) -> Self { + Self(vec![Segment::sanitized(SegmentKind::Source, id)]) + } + + /// This node with `segment` appended: its child. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] when the child would nest deeper than 8 + /// segments. + pub fn child(&self, segment: Segment) -> Result { + let mut segments = self.0.clone(); + segments.push(segment); + Self::new(segments) + } + /// Whether this is the root. #[must_use] pub fn is_root(&self) -> bool { diff --git a/crates/tinymemory-api/src/namespace/mod_tests.rs b/crates/tinymemory-api/src/namespace/mod_tests.rs index 7f0dbd6c..0b0ae85f 100644 --- a/crates/tinymemory-api/src/namespace/mod_tests.rs +++ b/crates/tinymemory-api/src/namespace/mod_tests.rs @@ -14,7 +14,7 @@ fn parses_and_prints_paths() { assert_eq!(writer.segments()[0].kind(), SegmentKind::Team); assert_eq!(writer.segments()[1].id(), "writer"); assert_eq!(ns("ws:shared").to_string(), "ws:shared"); - for kind in ["agent", "team", "user", "ws", "project"] { + for kind in ["agent", "team", "user", "ws", "project", "source"] { let segment = ns(&format!("{kind}:x")).segments()[0].clone(); assert_eq!(segment.kind().as_str(), kind); } @@ -108,3 +108,32 @@ fn serializes_as_strings() { assert_eq!(back, Reach::of(ns("agent:x"))); assert!(serde_json::from_value::(serde_json::json!("nope")).is_err()); } + +#[test] +fn builds_source_and_child_nodes() { + let pdf = Namespace::source("pdf"); + assert_eq!(pdf.to_string(), "source:pdf"); + assert_eq!(pdf.segments()[0].kind(), SegmentKind::Source); + assert_eq!(ns("source:pdf"), pdf); + + let team = ns("team:acme"); + let child = team + .child(Segment::sanitized(SegmentKind::Source, "notion export")) + .unwrap(); + assert_eq!(child.depth(), 2); + assert!( + child + .to_string() + .starts_with("team:acme/source:notion-export-") + ); + assert!(Reach::subtree(team).admits(&child)); +} + +#[test] +fn refuses_a_child_past_the_depth_limit() { + let deep = ns(&["agent:a"; MAX_DEPTH].join("/")); + let error = deep + .child(Segment::sanitized(SegmentKind::Agent, "b")) + .unwrap_err(); + assert!(matches!(error, Error::InvalidRequest(_)), "{error}"); +} diff --git a/crates/tinymemory-api/src/query/mod.rs b/crates/tinymemory-api/src/query/mod.rs index 28211914..8dadd54b 100644 --- a/crates/tinymemory-api/src/query/mod.rs +++ b/crates/tinymemory-api/src/query/mod.rs @@ -127,6 +127,18 @@ pub struct FetchRequest { /// Continue from a previous page. #[serde(default, skip_serializing_if = "Option::is_none")] pub cursor: Option, + /// Also return up to this many beliefs from what the fetch read, in + /// [`FetchPage::beliefs`]: the beliefs an engine keeps apart from its + /// items (see [`crate::MemoryEngine::beliefs`]), ranked for the same + /// query, from the same reads. `0`, the default, asks for none; an + /// engine that keeps no beliefs apart returns none. + #[serde(default, skip_serializing_if = "is_zero")] + pub beliefs: usize, +} + +/// Whether `n` is zero (serde's skip test). +fn is_zero(n: &usize) -> bool { + *n == 0 } impl FetchRequest { @@ -139,6 +151,7 @@ impl FetchRequest { filter: MetaFilter::default(), limit, cursor: None, + beliefs: 0, } } @@ -162,6 +175,11 @@ pub struct FetchPage { /// end. #[serde(default, skip_serializing_if = "Option::is_none")] pub next_cursor: Option, + /// The beliefs [`FetchRequest::beliefs`] asked for, best first: learning + /// hits tagged [`crate::BELIEF_TAG`] within the filter's reach. Not + /// items of the page, and not paged. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub beliefs: Vec, } /// One stored item as a read returns it. diff --git a/crates/tinymemory-api/src/write/mod.rs b/crates/tinymemory-api/src/write/mod.rs new file mode 100644 index 00000000..b73e0c00 --- /dev/null +++ b/crates/tinymemory-api/src/write/mod.rs @@ -0,0 +1,74 @@ +//! How long a store waits before it returns: [`WriteOptions`]. +//! +//! [`MemoryEngine::store`](crate::MemoryEngine::store) returns only once the +//! item is readable, which is what imports, tools and tests want. An agent's +//! live turn wants the opposite: the write must be durable, but the turn must +//! not wait for indexing. [`MemoryEngine::store_with`](crate::MemoryEngine::store_with) +//! takes a [`WaitFor`] to choose: +//! +//! - [`WaitFor::Visible`] (the default) — exactly `store`: listed, and ranked +//! by fetch and recall, on return. +//! - [`WaitFor::Accepted`] — the engine has durably accepted the item; it +//! becomes listable and ranked shortly after. Replay detection still runs +//! against what is already readable, so two identical `Accepted` stores in +//! quick succession may both write. +//! +//! An engine without a cheaper acknowledgement serves `Accepted` as +//! `Visible`, which is always correct, only slower. +//! +//! # Example +//! +//! ``` +//! use tinymemory_api::{MemoryEngine, MemoryMeta, StoreItem, WaitFor, WriteOptions}; +//! use tinymemory_api::conformance::ReferenceEngine; +//! +//! # let runtime = tokio::runtime::Builder::new_current_thread().build()?; +//! # runtime.block_on(async { +//! let engine = ReferenceEngine::new(); +//! let item = StoreItem::document("logged without waiting", MemoryMeta::default()); +//! let receipt = engine.store_with(item, WriteOptions::accepted()).await?; +//! assert!(!receipt.replayed); +//! assert_eq!(WriteOptions::default().wait, WaitFor::Visible); +//! # Ok::<(), tinymemory_api::Error>(()) +//! # })?; +//! # Ok::<(), Box>(()) +//! ``` + +use serde::{Deserialize, Serialize}; + +/// How far a store goes before it returns. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum WaitFor { + /// The engine durably accepted the item; reads may lag a moment behind. + Accepted, + /// The item is listed and ranked on return, as [`crate::MemoryEngine::store`]. + #[default] + Visible, +} + +/// Options for [`crate::MemoryEngine::store_with`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct WriteOptions { + /// How far the store goes before it returns. + #[serde(default)] + pub wait: WaitFor, +} + +impl WriteOptions { + /// Return once the engine accepted the item: the hot-path write. + #[must_use] + pub fn accepted() -> Self { + Self { + wait: WaitFor::Accepted, + } + } + + /// Return once the item is readable: what `store` does. + #[must_use] + pub fn visible() -> Self { + Self { + wait: WaitFor::Visible, + } + } +} diff --git a/crates/tinymemory-api/tests/conformance_reference.rs b/crates/tinymemory-api/tests/conformance_reference.rs index c0f454b2..53e804ae 100644 --- a/crates/tinymemory-api/tests/conformance_reference.rs +++ b/crates/tinymemory-api/tests/conformance_reference.rs @@ -6,9 +6,11 @@ use async_trait::async_trait; use tinymemory_api::conformance::{Error, ReferenceEngine, run}; use tinymemory_api::{ - EngineDescriptor, EngineHealth, ExplorePage, ExploreRequest, FetchMode, FetchPage, - FetchRequest, ForgetReport, ForgetTarget, GetRequest, Hit, ListPage, ListRequest, MemoryEngine, - MetaFilter, RecallAnswer, RecallRequest, Result, StoreItem, StoreReceipt, + BELIEF_TAG, BeliefsRequest, ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus, + Consolidation, EngineDescriptor, EngineHealth, ExplorePage, ExploreRequest, FetchMode, + FetchPage, FetchRequest, ForgetReport, ForgetTarget, GetRequest, Hit, ItemId, ItemKind, + ListPage, ListRequest, MemoryEngine, MemoryMeta, MetaFilter, Namespace, RecallAnswer, + RecallRequest, Result, StoreItem, StoreReceipt, WaitFor, WriteOptions, }; #[tokio::test] @@ -50,6 +52,18 @@ enum Fault { ListIgnoresReach, /// Reads ids by `get` whatever the reach. GetIgnoresReach, + /// Answers an accepted store with an id other than the item's. + AcceptedWrongId, + /// Declares on-demand consolidation but only acknowledges a schedule. + ConsolidateOffPromise, + /// Declares no consolidation yet answers a request. + ConsolidateUndeclared, + /// Builds without validating the request. + ConsolidateUnvalidated, + /// Returns beliefs that are not tagged as beliefs. + BeliefsUntagged, + /// Returns beliefs outside the reach asked for. + BeliefsOutOfReach, } struct Faulty { @@ -65,6 +79,9 @@ impl Faulty { if matches!(fault, Fault::ClaimEveryMode) { descriptor.fetch_modes = vec![FetchMode::Hybrid]; } + if matches!(fault, Fault::ConsolidateUndeclared) { + descriptor.consolidation = Consolidation::None; + } Self { inner, fault, @@ -114,6 +131,56 @@ impl MemoryEngine for Faulty { Ok(receipt) } + async fn store_with(&self, item: StoreItem, options: WriteOptions) -> Result { + let accepted = options.wait == WaitFor::Accepted; + let mut receipt = self.inner.store_with(item, options).await?; + if accepted && matches!(self.fault, Fault::AcceptedWrongId) { + receipt.id = "not-the-item".into(); + } + Ok(receipt) + } + + async fn consolidate(&self, req: ConsolidateRequest) -> Result { + match self.fault { + Fault::ConsolidateOffPromise => { + req.validate()?; + Ok(ConsolidateReceipt::scheduled()) + } + Fault::ConsolidateUnvalidated => Ok(ConsolidateReceipt { + status: ConsolidateStatus::Completed, + jobs: Vec::new(), + scopes: 1, + built: Some(1), + }), + _ => self.inner.consolidate(req).await, + } + } + + async fn beliefs(&self, req: BeliefsRequest) -> Result> { + req.validate()?; + let namespace = match self.fault { + Fault::BeliefsUntagged => req.reach.at.clone(), + Fault::BeliefsOutOfReach => Namespace::agent("somebody-else"), + _ => return self.inner.beliefs(req).await, + }; + let tags = match self.fault { + Fault::BeliefsUntagged => Vec::new(), + _ => vec![BELIEF_TAG.to_string()], + }; + Ok(vec![Hit { + id: ItemId::new("belief-1"), + kind: ItemKind::Learning, + text: "a belief".to_string(), + meta: MemoryMeta { + namespace, + tags, + ..MemoryMeta::default() + }, + score: 1.0, + confidence: Some(0.9), + }]) + } + async fn forget(&self, target: ForgetTarget) -> Result { if matches!(self.fault, Fault::AcceptEmptyForget) && target.validate().is_err() { return Ok(ForgetReport::default()); @@ -175,6 +242,12 @@ async fn each_fault_is_caught_by_its_check() { (Fault::StoreManyUnordered, "store_many"), (Fault::ListIgnoresReach, "namespaces"), (Fault::GetIgnoresReach, "namespaces"), + (Fault::AcceptedWrongId, "store_with"), + (Fault::ConsolidateOffPromise, "consolidate"), + (Fault::ConsolidateUndeclared, "consolidate"), + (Fault::ConsolidateUnvalidated, "consolidate"), + (Fault::BeliefsUntagged, "consolidate"), + (Fault::BeliefsOutOfReach, "consolidate"), ]; for (fault, expected) in cases { let error = run(&Faulty::new(fault)) @@ -186,3 +259,46 @@ async fn each_fault_is_caught_by_its_check() { assert_eq!(*check, expected, "{fault:?}: {error}"); } } + +#[tokio::test] +async fn an_engine_without_consolidation_passes_by_refusing_it() { + /// The reference engine, minus consolidation: the trait's default. + struct NoBeliefs { + inner: ReferenceEngine, + descriptor: EngineDescriptor, + } + + #[async_trait] + impl MemoryEngine for NoBeliefs { + fn descriptor(&self) -> &EngineDescriptor { + &self.descriptor + } + async fn health(&self) -> EngineHealth { + self.inner.health().await + } + async fn recall(&self, req: RecallRequest) -> Result { + self.inner.recall(req).await + } + async fn fetch(&self, req: FetchRequest) -> Result { + self.inner.fetch(req).await + } + async fn store(&self, item: StoreItem) -> Result { + self.inner.store(item).await + } + async fn forget(&self, target: ForgetTarget) -> Result { + self.inner.forget(target).await + } + async fn list(&self, req: ListRequest) -> Result { + self.inner.list(req).await + } + } + + let inner = ReferenceEngine::new(); + let descriptor = EngineDescriptor { + consolidation: Consolidation::None, + ..inner.descriptor().clone() + }; + let engine = NoBeliefs { inner, descriptor }; + run(&engine).await.expect("refusing consolidation conforms"); + assert!(engine.inner.is_empty()); +} diff --git a/crates/tinymemory-integrations/Cargo.toml b/crates/tinymemory-integrations/Cargo.toml index 70327fd4..62c4c7a3 100644 --- a/crates/tinymemory-integrations/Cargo.toml +++ b/crates/tinymemory-integrations/Cargo.toml @@ -27,6 +27,11 @@ thiserror = { version = "2", optional = true } # decision. log = { version = "0.4", optional = true } +# --- brain --- +# `brain::brain_document` builds the `BrainDocument` the tools crate's +# `Brain` ingests; the brain layout itself lives there. +tinymemory-tools = { path = "../tinymemory-tools", optional = true } + # --- cortex --- # CortexDB speaks HTTP/JSON. `stream` is for `bytes_stream()`: response bodies # are read against a byte cap rather than buffered whole, because the endpoint @@ -81,13 +86,18 @@ rusqlite = { version = "0.40", features = ["bundled"], optional = true } # The behavioural suite every engine must pass, run over both CortexDB wires' # doubles, and the reference engine the import driver is tested against. tinymemory-api = { path = "../tinymemory-api", features = ["conformance"] } -# `tests/live_cortexdb.rs` compiles `context.md` from a live server. +# `tests/live_cortexdb.rs` compiles `context.md` from a live server, and the +# agent lifecycle runs over the CortexDB doubles. tinymemory-tools = { path = "../tinymemory-tools" } # The CortexDB and TinyHumans doubles are real HTTP servers on loopback. axum = "0.8" tokio = { version = "1", features = ["macros", "rt", "rt-multi-thread", "net", "io-util", "time"] } # Reader and import tests build throwaway folders and v1 workspaces. tempfile = "3" +# The memory_eval example timestamps its scripted turns and writes its report. +chrono = { version = "0.4", default-features = false, features = ["clock"] } +serde = { version = "1", features = ["derive"] } +serde_json = "1" # The format and office tests build their OOXML fixtures in code. zip = { version = "8", default-features = false, features = ["deflate"] } # `MemoryConfig` round-trips through the TOML a host stores it in. @@ -107,6 +117,9 @@ documents = ["dep:async-trait", "dep:serde", "dep:serde_json", "dep:thiserror"] # host to prepend to its `ConverterChain`. Off by default because a PDF parser # and a spreadsheet reader are real weight. documents-office = ["documents", "dep:pdf-extract", "dep:calamine", "dep:quick-xml", "dep:zip"] +# `brain`: converting raw files into `tinymemory_tools::BrainDocument`s +# placed under the source type their format implies. +brain = ["documents", "dep:tinymemory-tools"] # `sources`: readers that turn folders, files and conversations into # `StoreItem`s, and the Composio payload normalisers. Links no HTTP stack. sources = ["documents", "dep:schemars", "dep:regex", "dep:walkdir", "dep:chrono", "dep:log", "dep:tracing"] @@ -119,16 +132,29 @@ safety = ["dep:regex", "dep:serde_json", "dep:log"] # into any engine, resumably. legacy-import = ["dep:rusqlite", "dep:serde", "dep:serde_json", "dep:thiserror"] # Every integration. -full = ["cortex", "documents-office", "sources-network", "safety", "legacy-import"] +full = ["cortex", "documents-office", "brain", "sources-network", "safety", "legacy-import"] [[example]] name = "basic" required-features = ["cortex"] +[[example]] +name = "memory_eval" +path = "examples/memory_eval/main.rs" +required-features = ["cortex", "brain"] + +[[example]] +name = "cortex_agent" +required-features = ["cortex", "brain", "documents-office"] + [[test]] name = "live_cortexdb" required-features = ["cortex"] +[[test]] +name = "live_cortex_lifecycle" +required-features = ["cortex", "brain"] + [[test]] name = "office_live" required-features = ["documents-office", "cortex"] diff --git a/crates/tinymemory-integrations/README.md b/crates/tinymemory-integrations/README.md index 2f0f88ed..6d02b39a 100644 --- a/crates/tinymemory-integrations/README.md +++ b/crates/tinymemory-integrations/README.md @@ -37,6 +37,7 @@ already needs): | `cortex` | `reqwest` (rustls TLS, streaming bodies), `tokio` (`time` only), `futures`, `sha2` | | `documents` | nothing beyond the small set above | | `documents-office` | `pdf-extract`, `calamine`, `quick-xml`, `zip` (all pure Rust, no system libraries) | +| `brain` | `tinymemory-tools` (for `BrainDocument`; it adds only `futures`, `chrono`, `log`) | | `sources` | `schemars`, `regex`, `walkdir`, `chrono`, `log`, `tracing` | | `sources-network` | `reqwest`, `futures`, `tokio` with `process`, `io-util` and `net` (the GitHub reader runs `gh` and `git`; the SSRF resolver does DNS) | | `safety` | `regex`, `serde_json`, `log` | @@ -107,6 +108,12 @@ the integration tests: `reader_dispatch` (sources), `legacy_import`, `TINYMEMORY_LIVE_CORTEXDB_URL` names a CortexDB server; `scripts/cortexdb-live.sh` boots the pinned harness in `integration/cortexdb/` and runs them against it. +`examples/memory_eval/` is the agent memory eval: a scripted agent plays nine +scenarios through the lifecycle and scores every pack. +`scripts/memory-eval.sh` runs it against a throwaway CortexDB, with mock or +real models; [`docs/evals/`](../../docs/evals/README.md) has the method and +the recorded results. + ## Architecture [`docs/architecture/integrations.md`](../../docs/architecture/integrations.md) diff --git a/crates/tinymemory-integrations/examples/cortex_agent.rs b/crates/tinymemory-integrations/examples/cortex_agent.rs new file mode 100644 index 00000000..d3c7850c --- /dev/null +++ b/crates/tinymemory-integrations/examples/cortex_agent.rs @@ -0,0 +1,178 @@ +//! The agent memory lifecycle against a real CortexDB server, timing every +//! step: files converted into the brain (a markdown handbook and a PDF), +//! turns logged and recalled for two agents, a session resume, compaction, +//! and the belief builds (`v1/beliefs/build`) the turns hand back. +//! +//! Run against the local harness (`integration/cortexdb/`, see its README) +//! or any CortexDB: +//! +//! ```sh +//! CORTEX_DB_URL=http://localhost:3141 CORTEX_DB_KEY=tinymemory-cortex-test \ +//! cargo run -p tinymemory-integrations --features full --example cortex_agent +//! ``` +//! +//! Without `CORTEX_DB_URL` it explains itself and exits, so CI only compiles +//! it. Everything is written below a node unique to the run and forgotten at +//! the end; set `CORTEX_DB_KEEP=1` to keep it for inspection. + +use std::path::Path; +use std::sync::Arc; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; + +use tinymemory_api::{ForgetTarget, MemoryEngine, MemoryMeta, Role, Turn}; +use tinymemory_integrations::brain::brain_document; +use tinymemory_integrations::cortex::{CortexCredential, CortexEngine}; +use tinymemory_integrations::documents::{ + ConverterChain, NativeConverter, OfficeConverter, RawDocument, +}; +use tinymemory_tools::{ + AgentMemory, BackgroundJob, Brain, Compaction, MemoryLayout, PostTurn, PreTurn, RecallPolicy, + SessionStart, +}; + +type Error = Box; + +/// Prints how long `label` took. +fn took(label: &str, started: Instant) { + println!( + " {label:<34} {:>7.1} ms", + started.elapsed().as_secs_f64() * 1e3 + ); +} + +/// Stands in for the model. +fn generate(context: &str) -> String { + let facts = context + .lines() + .filter(|line| line.starts_with("- ")) + .count(); + format!("(an answer grounded in {facts} remembered lines)") +} + +#[tokio::main] +async fn main() -> Result<(), Error> { + let Ok(url) = std::env::var("CORTEX_DB_URL") else { + println!( + "Set CORTEX_DB_URL (and CORTEX_DB_KEY) to run against CortexDB, e.g. the harness in \ + integration/cortexdb/: CORTEX_DB_URL=http://localhost:3141 \ + CORTEX_DB_KEY=tinymemory-cortex-test" + ); + return Ok(()); + }; + let key = std::env::var("CORTEX_DB_KEY").unwrap_or_else(|_| "tinymemory-cortex-test".into()); + let engine: Arc = + Arc::new(CortexEngine::direct(&url, CortexCredential::api_key(key))?); + println!( + "engine: {} at {url} ({:?})", + engine.descriptor().id, + engine.health().await + ); + + // Everything below one node per run, so runs never see each other. + let run = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs(); + let layout = MemoryLayout::new(format!("project:example-{run}").parse()?)?; + let mut jobs: Vec = Vec::new(); + + println!("\nbrain"); + let fixtures = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/fixtures"); + let converters = + ConverterChain::new(vec![Box::new(OfficeConverter), Box::new(NativeConverter)]); + let brain = Brain::new(engine.clone(), layout.clone()); + for file in ["onboarding.md", "refund-policy.pdf"] { + let raw = RawDocument::new(std::fs::read(fixtures.join(file))?).with_filename(file); + let started = Instant::now(); + let document = brain_document(&converters, &raw, None, MemoryMeta::default()).await?; + let source = document.source.clone(); + let ingested = brain.ingest(document).await?; + took(&format!("ingest {file} -> source:{source}"), started); + jobs.push(ingested.job); + } + + let policy = RecallPolicy { + build_beliefs_every: Some(2), + ..RecallPolicy::default() + }; + let support = + AgentMemory::new(engine.clone(), layout.clone(), "support-01")?.with_policy(policy.clone()); + let coder = AgentMemory::new(engine.clone(), layout.clone(), "coder-42")?.with_policy(policy); + + for (memory, thread, index, user) in [ + (&support, "s-1", 0, "How long do refunds take to settle?"), + (&coder, "c-1", 0, "The refund webhook deploy failed again."), + (&support, "s-1", 2, "Who handles billing disputes?"), + ] { + println!("\n{} turn: {user}", memory.agent_id()); + let started = Instant::now(); + let context = memory.pre_turn(PreTurn::new(thread, index, user)).await?; + took("pre_turn (log + recall)", started); + if let Some(error) = &context.log_error { + println!(" (turn not logged: {error})"); + } + println!("{}", indent(&context.pack.markdown)); + let reply = generate(&context.pack.markdown); + let started = Instant::now(); + let report = memory + .post_turn(PostTurn::new(thread, index + 1, reply)) + .await?; + took("post_turn (log)", started); + jobs.extend(report.jobs); + } + + println!("\nsession resume"); + let started = Instant::now(); + let resumed = support + .start_session(SessionStart { + thread_id: Some("s-1".into()), + focus: None, + }) + .await?; + took("start_session", started); + println!("{}", indent(&resumed.markdown)); + + println!("\ncompaction"); + let started = Instant::now(); + let carried = support + .recall_for_compaction(Compaction { + thread_id: "s-1".into(), + dropped: vec![Turn::new(Role::User, "How long do refunds take to settle?")], + focus: None, + }) + .await?; + took("recall_for_compaction", started); + println!("{}", indent(&carried.markdown)); + + println!("\nbackground"); + for job in jobs { + let started = Instant::now(); + let report = support.run_background(job).await?; + took( + &format!( + "{} ({} scopes)", + report.job, + report.consolidation.map_or(0, |receipt| receipt.scopes) + ), + started, + ); + println!(" -> {:?}", report.outcome); + } + + if std::env::var("CORTEX_DB_KEEP").is_err() { + let forgotten = engine + .forget(ForgetTarget::Filter(layout.holistic_filter())) + .await?; + println!( + "\ncleaned up {} items under {}", + forgotten.forgotten, + layout.root() + ); + } + Ok(()) +} + +fn indent(markdown: &str) -> String { + markdown + .lines() + .map(|line| format!(" {line}")) + .collect::>() + .join("\n") +} diff --git a/crates/tinymemory-integrations/examples/fixtures/onboarding.md b/crates/tinymemory-integrations/examples/fixtures/onboarding.md new file mode 100644 index 00000000..cf545937 --- /dev/null +++ b/crates/tinymemory-integrations/examples/fixtures/onboarding.md @@ -0,0 +1,5 @@ +# Support onboarding + +Every ticket gets a first reply within four hours. + +Escalate billing disputes to the finance channel, never to engineering. diff --git a/crates/tinymemory-integrations/examples/fixtures/refund-policy.pdf b/crates/tinymemory-integrations/examples/fixtures/refund-policy.pdf new file mode 100644 index 00000000..8d6f5d8c Binary files /dev/null and b/crates/tinymemory-integrations/examples/fixtures/refund-policy.pdf differ diff --git a/crates/tinymemory-integrations/examples/memory_eval/agent.rs b/crates/tinymemory-integrations/examples/memory_eval/agent.rs new file mode 100644 index 00000000..3fe57d18 --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/agent.rs @@ -0,0 +1,191 @@ +//! A deliberately tiny scripted agent. +//! +//! It has no model. Each user turn runs the real lifecycle calls, so +//! everything the memory layer does is the same as for a real agent: +//! +//! 1. `pre_turn` logs the user's text and recalls a context pack, leaving +//! out the turns still in its prompt window. +//! 2. The agent "runs" the scripted tool calls and copies their results into +//! its reply. Memory keeps only a call's name and id, so a result the +//! reply leaves out is lost. +//! 3. It answers a question with the pack line that shares the most words +//! with it ([`answer`]), and otherwise acknowledges. +//! 4. `post_turn` logs the reply with its tool calls. +//! +//! The extractive answer is a stand-in for a model reading the pack. It +//! scores what a model would see, not how well some model reasons. + +use std::time::Instant; + +use chrono::{DateTime, Duration, Utc}; +use tinymemory_api::ToolCallRef; +use tinymemory_tools::{AgentMemory, BackgroundJob, ContextPack, PostTurn, PreTurn}; + +/// A scripted tool call and the result the "tool" returns. +#[derive(Debug, Clone, Copy)] +pub(crate) struct ToolStep { + /// The tool's name. + pub(crate) name: &'static str, + /// What it returned. + pub(crate) result: &'static str, +} + +/// What one scripted turn did and how long each step took. +#[derive(Debug, Clone)] +pub(crate) struct TurnRecord { + /// `pre_turn` latency, in milliseconds. + pub(crate) pre_ms: f64, + /// `post_turn` latency, in milliseconds. + pub(crate) post_ms: f64, + /// Whether the user turn was logged. + pub(crate) logged: bool, + /// The jobs `post_turn` handed back. + pub(crate) jobs: Vec, + /// How many tool calls the reply made. + pub(crate) tool_calls: usize, +} + +/// One conversation thread driven by the script. +pub(crate) struct ScriptedAgent { + memory: AgentMemory, + thread: String, + next: u32, + /// How many of the thread's turns stay in the prompt. + window: u32, + clock: Option>, +} + +impl ScriptedAgent { + /// A new thread for `memory`, keeping the last `window` turns in the + /// prompt. + pub(crate) fn new(memory: AgentMemory, thread: impl Into, window: u32) -> Self { + Self { + memory, + thread: thread.into(), + next: 0, + window, + clock: None, + } + } + + /// Timestamps the thread's turns from `at`, a minute apart. + pub(crate) fn at(mut self, at: DateTime) -> Self { + self.clock = Some(at); + self + } + + /// The first turn index still in the prompt. + fn in_prompt_from(&self) -> u32 { + self.next.saturating_sub(self.window) + } + + /// The timestamp of the next turn, if the thread is timed. + fn tick(&mut self) -> Option> { + let at = self.clock?; + self.clock = Some(at + Duration::minutes(1)); + Some(at) + } + + /// One exchange: the user says `text`, the agent calls `tools` and + /// replies. + pub(crate) async fn user( + &mut self, + text: &str, + tools: &[ToolStep], + ) -> Result { + let mut pre = PreTurn::new(&self.thread, self.next, text); + pre.in_prompt_from = self.in_prompt_from(); + pre.at = self.tick(); + let started = Instant::now(); + let context = self.memory.pre_turn(pre).await?; + let pre_ms = ms(started); + + let reply = reply(&context.pack, text, tools); + let mut post = PostTurn::new(&self.thread, self.next + 1, reply); + post.tool_calls = tools + .iter() + .enumerate() + .map(|(index, step)| ToolCallRef { + name: step.name.to_string(), + id: Some(format!("{}-{}-{index}", self.thread, self.next)), + }) + .collect(); + post.at = self.tick(); + let started = Instant::now(); + let report = self.memory.post_turn(post).await?; + let post_ms = ms(started); + self.next += 2; + Ok(TurnRecord { + pre_ms, + post_ms, + logged: context.logged.is_some(), + jobs: report.jobs, + tool_calls: tools.len(), + }) + } +} + +/// The agent's reply: tool results first, then an answer or an +/// acknowledgement. +fn reply(pack: &ContextPack, text: &str, tools: &[ToolStep]) -> String { + let mut lines: Vec = tools + .iter() + .map(|step| format!("{} returned: {}", step.name, step.result)) + .collect(); + if text.trim_end().ends_with('?') { + lines.push(match answer(&pack.markdown, text) { + Some(line) => format!("Going by memory: {line}"), + None => NOT_IN_MEMORY.to_string(), + }); + } else if lines.is_empty() { + lines.push("Noted.".to_string()); + } + lines.join("\n") +} + +/// The agent's reply when nothing in the pack answers. +const NOT_IN_MEMORY: &str = "I don't have that in memory."; + +/// Words too common to tell two lines apart. +const STOPWORDS: [&str; 42] = [ + "a", "an", "the", "is", "are", "was", "were", "do", "does", "did", "of", "to", "in", "on", + "at", "for", "and", "or", "our", "we", "i", "my", "me", "you", "your", "what", "which", "who", + "when", "where", "how", "it", "that", "this", "with", "be", "should", "can", "get", "have", + "has", "from", +]; + +/// The lowercase content words of `text`. +fn words(text: &str) -> Vec { + text.split(|c: char| !c.is_alphanumeric() && c != '-') + .map(str::to_lowercase) + .filter(|word| word.len() > 1 && !STOPWORDS.contains(&word.as_str())) + .collect() +} + +/// The pack line the agent would answer `question` with: the bullet sharing +/// the most content words with it (earlier sections win ties), skipping +/// lines that only repeat a question and ignoring the agent's own +/// non-answers. +pub(crate) fn answer(markdown: &str, question: &str) -> Option { + let wanted = words(question); + let mut best: Option<(usize, String)> = None; + for line in markdown.lines().filter_map(|line| line.strip_prefix("- ")) { + let body = line.trim(); + if body.ends_with('?') { + continue; + } + let body = body.replace(NOT_IN_MEMORY, ""); + let body = body.trim(); + let held = words(body); + let overlap = wanted.iter().filter(|word| held.contains(word)).count(); + if overlap > 0 && best.as_ref().is_none_or(|(score, _)| overlap > *score) { + best = Some((overlap, body.to_string())); + } + } + best.map(|(_, line)| line) +} + +/// Milliseconds since `started`. +pub(crate) fn ms(started: Instant) -> f64 { + started.elapsed().as_secs_f64() * 1e3 +} diff --git a/crates/tinymemory-integrations/examples/memory_eval/inspect.rs b/crates/tinymemory-integrations/examples/memory_eval/inspect.rs new file mode 100644 index 00000000..9f1d2f1c --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/inspect.rs @@ -0,0 +1,253 @@ +//! What CortexDB synthesised, read straight off its wire. +//! +//! The memory API returns items, not the facts, beliefs and conflicts +//! CortexDB derives from them, so the eval reads those itself: `v1/recall` +//! for what a query would find, the list routes (`v1/facts`, `v1/beliefs`, +//! `v1/conflicts`) for everything captured, and `v1/admin/usage` for what +//! the models cost. That is the only way to tell whether memory captured an +//! event even when a pack does not show it. + +use serde::Serialize; +use serde_json::{Value, json}; + +/// The derived layers of one scope. +#[derive(Debug, Clone, Default, Serialize)] +pub(crate) struct Derived { + pub(crate) scope: String, + pub(crate) facts: usize, + pub(crate) beliefs: usize, + /// Each belief as "subject predicate object (stance, confidence)". + pub(crate) claims: Vec, +} + +/// Reads CortexDB's derived layers below a layout root. +pub(crate) struct Inspector { + client: reqwest::Client, + url: String, + key: String, +} + +impl Inspector { + /// An inspector of the server at `url`. + pub(crate) fn new(url: &str, key: &str) -> Self { + Self { + client: reqwest::Client::new(), + url: url.trim_end_matches('/').to_string(), + key: key.to_string(), + } + } + + async fn post(&self, path: &str, body: &Value) -> Result { + self.client + .post(format!("{}/{path}", self.url)) + .bearer_auth(&self.key) + .json(body) + .send() + .await? + .error_for_status()? + .json() + .await + } + + /// Every registered scope whose path holds `node` (a layout root such as + /// `project:eval-1-main`). + pub(crate) async fn scopes(&self, node: &str) -> Result, reqwest::Error> { + let listed: Value = self + .client + .get(format!("{}/v1/scopes/list", self.url)) + .query(&[("prefix", "app:tinymemory")]) + .bearer_auth(&self.key) + .send() + .await? + .error_for_status()? + .json() + .await?; + Ok(listed["items"] + .as_array() + .into_iter() + .flatten() + .filter_map(|item| item["path"].as_str()) + .filter(|path| path.split('/').any(|segment| segment == node)) + .map(str::to_owned) + .collect()) + } + + /// The facts and beliefs of `scope` relevant to `query`. + pub(crate) async fn derived( + &self, + scope: &str, + query: &str, + ) -> Result { + let pack = self + .post( + "v1/recall", + &json!({ + "scope": scope, + "query": query, + "budgets": { "per_layer_limits": { + "events": 1, "facts": 50, "beliefs": 50, + "episodes": 0, "understanding": 0, + }}, + }), + ) + .await?; + let layer = |name: &str| { + pack.pointer(&format!("/layers/{name}")) + .and_then(Value::as_array) + .cloned() + .unwrap_or_default() + }; + let beliefs = layer("beliefs"); + Ok(Derived { + scope: scope.to_string(), + facts: layer("facts").len(), + beliefs: beliefs.len(), + claims: beliefs.iter().map(claim).collect(), + }) + } +} + +/// A belief as one readable line. +fn claim(belief: &Value) -> String { + format!( + "{} {} {} ({}, {:.2})", + part(belief.pointer("/claim/subject")), + part(belief.pointer("/claim/predicate")), + part(belief.pointer("/claim/object")), + belief["stance"].as_str().unwrap_or("?"), + belief["confidence"].as_f64().unwrap_or_default(), + ) +} + +/// Everything CortexDB derived below a layout root, as readable lines. +#[derive(Debug, Clone, Default, Serialize)] +pub(crate) struct Captured { + pub(crate) facts: Vec, + pub(crate) beliefs: Vec, + /// Each open or resolved conflict as "kind status: subject predicate + /// [values]". + pub(crate) conflicts: Vec, +} + +impl Captured { + /// Whether any fact or belief mentions `needle`. + pub(crate) fn mentions(&self, needle: &str) -> bool { + let needle = needle.to_lowercase(); + self.facts + .iter() + .chain(&self.beliefs) + .any(|line| line.to_lowercase().contains(&needle)) + } +} + +/// The models' usage CortexDB accounts for, as its routers price it. +#[derive(Debug, Clone, Copy, Default, Serialize)] +pub(crate) struct Usage { + pub(crate) calls: u64, + pub(crate) tokens: u64, + pub(crate) cost_usd: f64, +} + +impl Usage { + /// What was spent between `before` and `self`. + pub(crate) fn since(self, before: Self) -> Self { + Self { + calls: self.calls.saturating_sub(before.calls), + tokens: self.tokens.saturating_sub(before.tokens), + cost_usd: self.cost_usd - before.cost_usd, + } + } +} + +/// A claim's part (`subject`, `predicate` or `object`) as text. +fn part(value: Option<&Value>) -> String { + value + .and_then(|v| v.get("name").or_else(|| v.get("value")).or(Some(v))) + .map(|v| v.as_str().map_or_else(|| v.to_string(), str::to_owned)) + .unwrap_or_default() +} + +/// The array of a list answer, whatever it is called. +fn listed(answer: &Value, field: &str) -> Vec { + answer + .get(field) + .or_else(|| answer.get("items")) + .and_then(Value::as_array) + .cloned() + .unwrap_or_default() +} + +impl Inspector { + async fn get(&self, path: &str, query: &[(&str, &str)]) -> Result { + self.client + .get(format!("{}/{path}", self.url)) + .query(query) + .bearer_auth(&self.key) + .send() + .await? + .error_for_status()? + .json() + .await + } + + /// Every fact, belief and conflict CortexDB holds in `scopes`. + pub(crate) async fn captured(&self, scopes: &[String]) -> Result { + let mut captured = Captured::default(); + for scope in scopes { + let page = [("scope", scope.as_str()), ("limit", "200")]; + for fact in listed(&self.get("v1/facts", &page).await?, "facts") { + captured.facts.push(format!( + "{} {} {}", + part(fact.pointer("/subject")), + part(fact.pointer("/predicate")), + part(fact.pointer("/object")), + )); + } + for belief in listed(&self.get("v1/beliefs", &page).await?, "beliefs") { + captured.beliefs.push(claim(&belief)); + } + for conflict in listed(&self.get("v1/conflicts", &page).await?, "conflicts") { + let values: Vec = conflict["records"] + .as_array() + .into_iter() + .flatten() + .map(|record| record["object"].as_str().unwrap_or("?").to_string()) + .collect(); + captured.conflicts.push(format!( + "{} {}: {} {} [{}]", + conflict["kind"].as_str().unwrap_or("?"), + conflict["status"].as_str().unwrap_or("?"), + conflict["subject"].as_str().unwrap_or("?"), + conflict["predicate"].as_str().unwrap_or("?"), + values.join(" | "), + )); + } + } + Ok(captured) + } + + /// The enrichment queue (fact extraction): jobs not yet done, and jobs + /// ever queued. + pub(crate) async fn enrichment(&self) -> Result<(u64, u64), reqwest::Error> { + let report = self.get("v1/admin/usage", &[]).await?; + let field = |name: &str| { + report + .pointer(&format!("/enrichment_backlog/{name}")) + .and_then(Value::as_u64) + .unwrap_or_default() + }; + Ok((field("jobs_pending"), field("jobs_queued_total"))) + } + + /// The models' usage so far. + pub(crate) async fn usage(&self) -> Result { + let report = self.get("v1/admin/usage", &[]).await?; + let total = &report["total"]; + Ok(Usage { + calls: total["calls"].as_u64().unwrap_or_default(), + tokens: total["tokens_total"].as_u64().unwrap_or_default() + + total["tokens_unsplit"].as_u64().unwrap_or_default(), + cost_usd: total["cost_usd"].as_f64().unwrap_or_default(), + }) + } +} diff --git a/crates/tinymemory-integrations/examples/memory_eval/llm.rs b/crates/tinymemory-integrations/examples/memory_eval/llm.rs new file mode 100644 index 00000000..07e345e4 --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/llm.rs @@ -0,0 +1,90 @@ +//! An optional model that answers each probe from its pack (`--llm`). +//! +//! The scripted agent's extractive answer only finds lines that share words +//! with the question, so it cannot answer a paraphrase even when the pack +//! holds the fact. A model reading the same pack shows whether the pack is +//! usable, which is what a host cares about. +//! +//! Any OpenAI-compatible chat endpoint works: +//! +//! - `EVAL_LLM_URL`: default `https://openrouter.ai/api/v1`. +//! - `EVAL_LLM_KEY`: default `OPENROUTER_API_KEY`. +//! - `EVAL_LLM_MODEL`: default `openai/gpt-4.1-mini`. +//! +//! It answers at temperature 0 and is scored like the extractive answer. + +use serde_json::{Value, json}; + +/// What the model is told. +const SYSTEM: &str = "You are an assistant with a long-term memory. The user's message \ + starts with what your memory recalled, then their question. Answer the question in one \ + short sentence, using only the memory. If the memory does not say, answer \"unknown\"."; + +/// A chat model. +pub(crate) struct Llm { + client: reqwest::Client, + url: String, + key: String, + /// The model's id. + pub(crate) model: String, +} + +impl Llm { + /// The model the environment names. + /// + /// # Errors + /// + /// When neither `EVAL_LLM_KEY` nor `OPENROUTER_API_KEY` is set. + pub(crate) fn from_env() -> Result { + let key = std::env::var("EVAL_LLM_KEY") + .or_else(|_| std::env::var("OPENROUTER_API_KEY")) + .map_err(|_| "--llm needs EVAL_LLM_KEY or OPENROUTER_API_KEY".to_string())?; + Ok(Self { + client: reqwest::Client::new(), + url: std::env::var("EVAL_LLM_URL") + .unwrap_or_else(|_| "https://openrouter.ai/api/v1".to_string()) + .trim_end_matches('/') + .to_string(), + key, + model: std::env::var("EVAL_LLM_MODEL") + .unwrap_or_else(|_| "openai/gpt-4.1-mini".to_string()), + }) + } + + /// The model's answer to `question` given `pack`. + /// + /// # Errors + /// + /// A transport failure, or an answer without text. + pub(crate) async fn answer(&self, pack: &str, question: &str) -> Result { + let body = json!({ + "model": self.model, + "temperature": 0, + // Room for a reasoning model's thinking (some, such as GLM 5.3 + // Flash, cannot turn it off) as well as the one-sentence answer. + "max_tokens": 2000, + "reasoning": { "effort": "low" }, + "messages": [ + { "role": "system", "content": SYSTEM }, + { "role": "user", "content": format!("{pack}\n\nQuestion: {question}") }, + ], + }); + let answer: Value = self + .client + .post(format!("{}/chat/completions", self.url)) + .bearer_auth(&self.key) + .json(&body) + .send() + .await + .and_then(reqwest::Response::error_for_status) + .map_err(|error| error.to_string())? + .json() + .await + .map_err(|error| error.to_string())?; + answer + .pointer("/choices/0/message/content") + .and_then(Value::as_str) + .map(|text| text.trim().to_string()) + .ok_or_else(|| format!("no answer text in {answer}")) + } +} diff --git a/crates/tinymemory-integrations/examples/memory_eval/main.rs b/crates/tinymemory-integrations/examples/memory_eval/main.rs new file mode 100644 index 00000000..a10d1231 --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/main.rs @@ -0,0 +1,706 @@ +//! An accuracy and latency eval of the agent memory lifecycle. +//! +//! A scripted agent (`agent`) plays nine scenarios (`scenarios`) through the +//! real lifecycle calls: brain lookups, a restart, contradicting facts, a +//! tool-heavy incident, a team handoff, compaction, tenant isolation, a +//! needle in noise, and explicit learnings. After each scenario's writes +//! settle, its probes are scored (`score`). Every scenario then runs a +//! belief build over its whole tree and is probed again, so the effect of +//! synthesis shows up as a second phase. +//! +//! ```sh +//! # Offline, against the reference engine: +//! cargo run -p tinymemory-integrations --features full --example memory_eval +//! +//! # Against a CortexDB (see integration/cortexdb/ and docs/evals/): +//! CORTEX_DB_URL=http://127.0.0.1:3142 CORTEX_DB_KEY=tinymemory-cortex-test \ +//! cargo run -p tinymemory-integrations --features full --example memory_eval -- \ +//! --label mock --json target/memory-eval/mock.json +//! ``` +//! +//! Flags: +//! +//! - `--engine reference|cortex`: the default is `cortex` when +//! `CORTEX_DB_URL` is set, and `reference` otherwise. +//! - `--only [,…]`: run only these scenarios. +//! - `--enrich-wait `: the longest to wait for CortexDB's enrichment +//! queue (fact extraction) to drain before the belief build (default 600 +//! against CortexDB, 0 otherwise). With a real model it takes minutes. +//! - `--json `: write every probe, pack included, as JSON. +//! - `--label `: name the run in the report. +//! - `--llm`: also have a model answer every probe from its pack (see +//! `llm`). +//! +//! Everything is written below roots unique to the run and forgotten at the +//! end, unless `CORTEX_DB_KEEP` is set. + +mod agent; +mod inspect; +mod llm; +mod scenarios; +mod score; + +use std::collections::BTreeMap; +use std::sync::Arc; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use chrono::{TimeZone, Utc}; +use serde::Serialize; +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_api::{ + ConsolidateRequest, ForgetTarget, ListRequest, MemoryEngine, MemoryMeta, Reach, Role, + StoreItem, Turn, +}; +use tinymemory_integrations::cortex::{CortexCredential, CortexEngine}; +use tinymemory_tools::context::{self, Brief, ContextSpec}; +use tinymemory_tools::{ + AgentMemory, BackgroundJob, Brain, BrainDocument, Compaction, ContextPack, JobOutcome, + MemoryLayout, PreTurn, RecallPolicy, SessionStart, +}; + +use agent::{ScriptedAgent, ms}; +use inspect::{Captured, Derived, Inspector}; +use llm::Llm; +use scenarios::{MAIN, Probe, Scenario, Step, Via}; +use score::{Latency, ProbeResult, Totals, grade, score}; + +type Error = Box; + +/// Turns of a thread the scripted agent keeps in its prompt. +const WINDOW: u32 = 8; + +/// The longest a scenario's writes may take to become visible. +const SETTLE_TIMEOUT: Duration = Duration::from_secs(90); + +/// Consecutive empty, quiet polls of the enrichment queue (3 s apart) that +/// count as drained. +const DRAINED_POLLS: u32 = 4; + +/// The command line. +struct Args { + engine: String, + only: Option, + enrich_wait: Option, + json: Option, + label: String, + llm: bool, +} + +fn args() -> Result { + let mut parsed = Args { + engine: if std::env::var("CORTEX_DB_URL").is_ok() { + "cortex".into() + } else { + "reference".into() + }, + only: None, + enrich_wait: None, + json: None, + label: String::new(), + llm: false, + }; + let mut raw = std::env::args().skip(1); + while let Some(flag) = raw.next() { + let mut value = || raw.next().ok_or(format!("{flag} needs a value")); + match flag.as_str() { + "--engine" => parsed.engine = value()?, + "--only" => parsed.only = Some(value()?), + "--enrich-wait" => parsed.enrich_wait = Some(value()?.parse()?), + "--json" => parsed.json = Some(value()?), + "--label" => parsed.label = value()?, + "--llm" => parsed.llm = true, + other => return Err(format!("unknown flag {other}").into()), + } + } + if parsed.label.is_empty() { + parsed.label = parsed.engine.clone(); + } + Ok(parsed) +} + +/// Latency samples by step. +#[derive(Default, Serialize)] +struct Timings(BTreeMap>); + +impl Timings { + fn add(&mut self, step: &str, ms: f64) { + self.0.entry(step.to_string()).or_default().push(ms); + } +} + +/// What the synthesis step did for a scenario. +#[derive(Debug, Default, Serialize)] +struct Synthesis { + jobs: usize, + outcomes: BTreeMap, + scopes: usize, + /// Beliefs the builds reported building. + built: usize, + ms: f64, + derived: Vec, + /// Every fact, belief and conflict CortexDB holds for the scenario. + captured: Captured, +} + +/// One scenario's results. +#[derive(Serialize)] +struct ScenarioReport { + name: &'static str, + about: &'static str, + writes: usize, + tool_calls: usize, + settle_ms: f64, + synthesis: Synthesis, + probes: Vec, +} + +#[tokio::main] +async fn main() -> Result<(), Error> { + let args = args()?; + let url = std::env::var("CORTEX_DB_URL").unwrap_or_default(); + let key = std::env::var("CORTEX_DB_KEY").unwrap_or_else(|_| "tinymemory-cortex-test".into()); + let (engine, inspector): (Arc, Option) = match args.engine.as_str() + { + "reference" => (Arc::new(ReferenceEngine::new()), None), + "cortex" if !url.is_empty() => ( + Arc::new(CortexEngine::direct(&url, CortexCredential::api_key(&key))?), + Some(Inspector::new(&url, &key)), + ), + "cortex" => return Err("--engine cortex needs CORTEX_DB_URL".into()), + other => return Err(format!("unknown engine {other}").into()), + }; + let enrich_wait = args + .enrich_wait + .unwrap_or(if inspector.is_some() { 600 } else { 0 }); + let llm = if args.llm { + Some(Llm::from_env()?) + } else { + None + }; + let run = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs(); + println!( + "memory eval `{}`: engine {} ({:?}), answer model {}, run {run}\n", + args.label, + engine.descriptor().id, + engine.health().await, + llm.as_ref().map_or("none", |llm| llm.model.as_str()), + ); + + let eval = Eval { + engine: engine.clone(), + inspector, + llm, + run, + enrich_wait, + policy: RecallPolicy { + build_beliefs_every: Some(4), + ..RecallPolicy::default() + }, + }; + let usage_before = match &eval.inspector { + Some(inspector) => Some(inspector.usage().await?), + None => None, + }; + let mut timings = Timings::default(); + let mut reports = Vec::new(); + for scenario in scenarios::all() { + if args + .only + .as_deref() + .is_some_and(|only| !only.split(',').any(|name| name == scenario.name)) + { + continue; + } + println!("== {}: {}", scenario.name, scenario.about); + let report = eval.scenario(&scenario, &mut timings).await?; + for phase in ["recall", "synthesis"] { + let totals = Totals::of(report.probes.iter().filter(|p| p.phase == phase)); + println!( + " {phase:<9} hits {:<12} answers {:<12} MRR {:.2}", + Totals::pct(totals.hits, totals.scored), + Totals::pct(totals.answers_ok, totals.scored), + totals.mrr, + ); + } + reports.push(report); + } + + print_summary(&args.label, &reports, &timings); + let usage = match (&eval.inspector, usage_before) { + (Some(inspector), Some(before)) => Some(inspector.usage().await?.since(before)), + _ => None, + }; + if let Some(usage) = usage { + println!( + "\n## Model usage\n\nCortexDB's models: {} calls, {} tokens, ${:.2} as its routers \ + price them (excludes the `--llm` answers).", + usage.calls, usage.tokens, usage.cost_usd + ); + } + if let Some(path) = &args.json { + if let Some(dir) = std::path::Path::new(path).parent() { + std::fs::create_dir_all(dir)?; + } + let out = serde_json::json!({ + "label": args.label, + "engine": engine.descriptor().id, + "run": run, + "scenarios": reports, + "timings": timings, + "usage": usage, + }); + std::fs::write(path, serde_json::to_string_pretty(&out)?)?; + println!("\nwrote {path}"); + } + Ok(()) +} + +/// The layout of `tenant` in `scenario` for this run. +fn layout(run: u64, scenario: &str, tenant: &str) -> Result { + let root = format!("project:eval-{run}-{}-{tenant}", scenario.replace('_', "-")); + Ok(MemoryLayout::new(root.parse()?)?) +} + +/// The tenants a scenario touches. +fn tenants(scenario: &Scenario) -> Vec<&'static str> { + let mut tenants: Vec<&'static str> = scenario + .steps + .iter() + .map(|step| match step { + Step::Doc { tenant, .. } | Step::Chat { tenant, .. } => *tenant, + Step::Learning { .. } => MAIN, + }) + .chain(scenario.probes.iter().map(|probe| probe.tenant)) + .collect(); + tenants.sort_unstable(); + tenants.dedup(); + tenants +} + +/// One eval run: the engine, the optional helpers, and the settings every +/// scenario shares. +struct Eval { + engine: Arc, + inspector: Option, + llm: Option, + run: u64, + enrich_wait: u64, + policy: RecallPolicy, +} + +impl Eval { + /// Writes `scenario`, probes it, synthesises, and probes it again. + async fn scenario( + &self, + scenario: &Scenario, + timings: &mut Timings, + ) -> Result { + let (engine, run, policy) = (&self.engine, self.run, &self.policy); + let memory = |tenant: &str, agent: &str| -> Result { + Ok( + AgentMemory::new(engine.clone(), layout(run, scenario.name, tenant)?, agent)? + .with_policy(policy.clone()), + ) + }; + let epoch = Utc + .with_ymd_and_hms(2026, 9, 1, 9, 0, 0) + .single() + .ok_or("a valid epoch")?; + + // Writes. + let mut jobs: Vec = Vec::new(); + let mut writes: BTreeMap<&'static str, usize> = BTreeMap::new(); + let mut tool_calls = 0; + for step in &scenario.steps { + match step { + Step::Doc { + tenant, + source, + title, + text, + } => { + let brain = Brain::new(engine.clone(), layout(run, scenario.name, tenant)?); + let started = Instant::now(); + let ingested = brain + .ingest(BrainDocument::new(source.clone(), *text).titled(*title)) + .await?; + timings.add("brain ingest (visible)", ms(started)); + jobs.push(ingested.job); + *writes.entry(tenant).or_default() += 1; + } + Step::Learning { + kind, + text, + confidence, + } => { + let layout = layout(run, scenario.name, MAIN)?; + let meta = MemoryMeta { + namespace: layout.learnings().clone(), + ..MemoryMeta::default() + }; + engine + .store(StoreItem::learning(*text, *kind, *confidence, meta)) + .await?; + *writes.entry(MAIN).or_default() += 1; + } + Step::Chat { + tenant, + agent, + thread, + day, + turns, + } => { + let mut scripted = ScriptedAgent::new(memory(tenant, agent)?, thread, WINDOW) + .at(epoch + chrono::Duration::days(*day)); + for (text, tools) in turns { + let record = scripted.user(text, tools).await?; + timings.add("pre_turn (log + recall)", record.pre_ms); + timings.add("post_turn (log)", record.post_ms); + if !record.logged { + println!(" ! a turn of {thread} was not logged"); + } + tool_calls += record.tool_calls; + jobs.extend(record.jobs); + *writes.entry(tenant).or_default() += 2; + } + } + } + } + + // Settle: wait until every write is listed. + let started = Instant::now(); + for (tenant, expected) in &writes { + self.settle(&layout(run, scenario.name, tenant)?, *expected) + .await?; + } + let settle_ms = ms(started); + timings.add("settle (all writes listed)", settle_ms); + + let mut probes = Vec::new(); + for probe in &scenario.probes { + probes.push(self.probe(scenario, probe, "recall", timings).await?); + } + + // Synthesis: the jobs the writes handed back, then one build per tenant + // over its whole tree. + if let Some(inspector) = &self.inspector { + let started = Instant::now(); + let cap = Duration::from_secs(self.enrich_wait); + // Drained means empty with nothing newly queued for a few polls + // in a row: the queue empties between batches while later + // writes are still being taken in. + let mut quiet = 0; + let mut last_queued = None; + while quiet < DRAINED_POLLS && started.elapsed() < cap { + tokio::time::sleep(Duration::from_secs(3)).await; + let (pending, queued) = inspector.enrichment().await?; + let settled = pending == 0 && last_queued == Some(queued); + quiet = if settled { quiet + 1 } else { 0 }; + last_queued = Some(queued); + } + let waited = ms(started); + timings.add("enrichment (queue drained)", waited); + println!(" enrichment drained in {:.0} s", waited / 1e3); + } + for tenant in tenants(scenario) { + let root = layout(run, scenario.name, tenant)?.root().clone(); + jobs.push(BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::subtree(root)), + }); + } + let runner = memory(MAIN, "eval")?.background(); + let mut synthesis = Synthesis { + jobs: jobs.len(), + ..Synthesis::default() + }; + let started = Instant::now(); + for job in jobs { + let report = runner.run(job).await?; + let outcome = match &report.outcome { + JobOutcome::Done => "done", + JobOutcome::Started => "started", + JobOutcome::Scheduled => "scheduled", + JobOutcome::Skipped { .. } => "skipped", + }; + *synthesis.outcomes.entry(outcome.to_string()).or_default() += 1; + if let Some(receipt) = report.consolidation { + synthesis.scopes += receipt.scopes; + synthesis.built += receipt.built.unwrap_or_default(); + } + } + synthesis.ms = ms(started); + timings.add("synthesis (all builds)", synthesis.ms); + let questions: Vec<&str> = scenario.probes.iter().map(|p| p.question).collect(); + let questions = questions.join(" "); + if let Some(inspector) = &self.inspector { + for tenant in tenants(scenario) { + let layout = layout(run, scenario.name, tenant)?; + let node = layout.root().to_string(); + let scopes = inspector.scopes(&node).await?; + for scope in &scopes { + synthesis + .derived + .push(inspector.derived(scope, &questions).await?); + } + let captured = inspector.captured(&scopes).await?; + synthesis.captured.facts.extend(captured.facts); + synthesis.captured.beliefs.extend(captured.beliefs); + synthesis.captured.conflicts.extend(captured.conflicts); + } + } + let beliefs: usize = synthesis.derived.iter().map(|d| d.beliefs).sum(); + let facts: usize = synthesis.derived.iter().map(|d| d.facts).sum(); + println!( + " synthesis {:?} over {} scopes in {:.0} ms: {} beliefs built; \ + recall finds {facts} facts, {beliefs} beliefs", + synthesis.outcomes, synthesis.scopes, synthesis.ms, synthesis.built + ); + if self.inspector.is_some() { + let held = &synthesis.captured; + println!( + " captured {} facts, {} beliefs, {} conflicts", + held.facts.len(), + held.beliefs.len(), + held.conflicts.len() + ); + for conflict in &held.conflicts { + println!(" conflict {conflict}"); + } + } + + for probe in &scenario.probes { + let mut result = self.probe(scenario, probe, "synthesis", timings).await?; + if self.inspector.is_some() && !probe.expect.is_empty() { + let held = &synthesis.captured; + result.captured = Some( + probe.expect.iter().all(|needle| held.mentions(needle)) + || probe.accept.iter().any(|needle| held.mentions(needle)), + ); + } + probes.push(result); + } + + if std::env::var("CORTEX_DB_KEEP").is_err() { + for tenant in tenants(scenario) { + engine + .forget(ForgetTarget::Filter( + layout(run, scenario.name, tenant)?.holistic_filter(), + )) + .await?; + } + } + Ok(ScenarioReport { + name: scenario.name, + about: scenario.about, + writes: writes.values().sum(), + tool_calls, + settle_ms, + synthesis, + probes, + }) + } + + /// Waits until `layout` lists at least `expected` items. + async fn settle(&self, layout: &MemoryLayout, expected: usize) -> Result<(), Error> { + let started = Instant::now(); + loop { + let mut listed = 0; + let mut cursor = None; + loop { + let mut req = ListRequest::new(layout.holistic_filter(), 100); + req.cursor = cursor; + let page = self.engine.list(req).await?; + listed += page.items.len(); + cursor = page.next_cursor; + if cursor.is_none() { + break; + } + } + if listed >= expected { + return Ok(()); + } + if started.elapsed() > SETTLE_TIMEOUT { + return Err(format!( + "only {listed} of {expected} writes visible under {}", + layout.root() + ) + .into()); + } + tokio::time::sleep(Duration::from_millis(200)).await; + } + } + + /// Reads `probe` the way it says and scores the pack. + async fn probe( + &self, + scenario: &Scenario, + probe: &Probe, + phase: &'static str, + timings: &mut Timings, + ) -> Result { + let (engine, run, policy, llm) = (&self.engine, self.run, &self.policy, self.llm.as_ref()); + let memory = AgentMemory::new( + engine.clone(), + layout(run, scenario.name, probe.tenant)?, + probe.agent, + )? + .with_policy(policy.clone()); + let started = Instant::now(); + let (markdown, tokens) = match &probe.via { + Via::Ask => { + let thread = format!("probe-{}", probe.id); + pack( + memory + .pre_turn(PreTurn::new(thread, 0, probe.question)) + .await? + .pack, + ) + } + Via::Resume { thread, focus } => pack( + memory + .start_session(SessionStart { + thread_id: thread.map(str::to_owned), + focus: focus.map(str::to_owned), + }) + .await?, + ), + Via::Compact { thread, dropped } => pack( + memory + .recall_for_compaction(Compaction { + thread_id: (*thread).to_string(), + dropped: dropped + .iter() + .map(|text| Turn::new(Role::User, text.as_str())) + .collect(), + focus: None, + }) + .await?, + ), + Via::Continue { + thread, + turn_index, + in_prompt_from, + } => { + let mut pre = PreTurn::new(*thread, *turn_index, probe.question); + pre.in_prompt_from = *in_prompt_from; + pack(memory.pre_turn(pre).await?.pack) + } + Via::ContextDoc { heading } => { + let root = layout(run, scenario.name, probe.tenant)?.root().clone(); + let spec = ContextSpec { + briefs: vec![Brief::new(*heading, probe.question)], + reach: Some(Reach::subtree(root)), + ..ContextSpec::default() + }; + let doc = context::compile(engine.as_ref(), &spec).await?; + (doc.markdown, doc.tokens) + } + }; + let elapsed = ms(started); + let mut result = score(scenario.name, phase, probe, &markdown, tokens, elapsed); + if let Some(llm) = llm { + let answer = llm.answer(&markdown, probe.question).await?; + result.llm_ok = grade(probe, Some(&answer)); + result.llm_answer = Some(answer); + } + timings.add(&format!("probe {}", result.via), elapsed); + Ok(result) + } +} + +/// A pack's markdown and token count. +fn pack(pack: ContextPack) -> (String, usize) { + (pack.markdown, pack.tokens) +} + +/// One accuracy row. +fn row(name: &str, phase: &str, t: &Totals) -> String { + format!( + "| {name} | {phase} | {} | {} | {} | {} | {:.2} | {} | {} |", + Totals::pct(t.hits, t.scored), + Totals::pct(t.answers_ok, t.scored), + Totals::pct(t.llm_ok, t.llm_scored), + Totals::pct(t.captured, t.captured_checked), + t.mrr, + Totals::pct(t.fresh_first, t.contradictions), + if t.leak_checks == 0 { + "–".to_string() + } else { + format!("{}/{}", t.leaks, t.leak_checks) + }, + ) +} + +fn print_summary(label: &str, reports: &[ScenarioReport], timings: &Timings) { + let phases = ["recall", "synthesis"]; + println!("\n## Accuracy (`{label}`)\n"); + println!( + "| Scenario | Phase | Pack hit | Extractive answer | Model answer | Captured | MRR | Fresh first | Leaks |" + ); + println!("| --- | --- | --- | --- | --- | --- | --- | --- | --- |"); + let all: Vec<&ProbeResult> = reports.iter().flat_map(|r| &r.probes).collect(); + for report in reports { + for phase in phases { + let t = Totals::of(report.probes.iter().filter(|p| p.phase == phase)); + println!("{}", row(report.name, phase, &t)); + } + } + for phase in phases { + let t = Totals::of(all.iter().copied().filter(|p| p.phase == phase)); + println!("{}", row("**all**", phase, &t)); + } + + println!("\n## By question style\n"); + println!( + "| Style | Phase | Pack hit | Extractive answer | Model answer | Captured | MRR | Fresh first | Leaks |" + ); + println!("| --- | --- | --- | --- | --- | --- | --- | --- | --- |"); + for style in ["lexical", "paraphrase"] { + for phase in phases { + let t = Totals::of( + all.iter() + .copied() + .filter(|p| p.phase == phase && p.style == style), + ); + println!("{}", row(style, phase, &t)); + } + } + + println!("\n## Latency (ms)\n"); + println!("| Step | n | p50 | p95 | max |"); + println!("| --- | --- | --- | --- | --- |"); + for (step, samples) in &timings.0 { + let l = Latency::of(samples); + println!( + "| {step} | {} | {:.1} | {:.1} | {:.1} |", + l.n, l.p50, l.p95, l.max + ); + } + + println!("\n## Misses\n"); + for result in all.iter().filter(|p| { + p.hit == Some(false) + || p.answer_ok == Some(false) + || p.llm_ok == Some(false) + || p.leak + || p.stale_first + }) { + println!( + "- {} {}/{} ({}, {}): hit {:?}, rank {:?}, stale first {}, leak {}; extractive {:?}; model {:?}", + result.phase, + result.scenario, + result.id, + result.via, + result.style, + result.hit, + result.rank, + result.stale_first, + result.leak, + result + .answer + .as_deref() + .map(|a| a.chars().take(90).collect::()), + result.llm_answer, + ); + } +} diff --git a/crates/tinymemory-integrations/examples/memory_eval/scenarios.rs b/crates/tinymemory-integrations/examples/memory_eval/scenarios.rs new file mode 100644 index 00000000..826cee81 --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/scenarios.rs @@ -0,0 +1,1152 @@ +//! The scenarios: what gets written, then what gets asked. +//! +//! Each scenario runs below its own layout root, so scenarios never see each +//! other. A `tenant` other than [`MAIN`] gets a sibling root, to check that +//! one root never sees another's memory. +//! +//! Every probe states what a correct pack contains (`expect`), what it may +//! hold but must not prefer (`stale`, superseded values), and what it must +//! never hold (`forbidden`, another tenant's data or turns still in the +//! prompt). Probes are tagged `Lexical` when the question shares its key +//! words with the stored text, and `Paraphrase` when it does not, so keyword +//! retrieval and semantic retrieval can be told apart. + +use tinymemory_api::LearningKind; +use tinymemory_tools::BrainSource; + +use crate::agent::ToolStep; + +/// The default tenant. +pub(crate) const MAIN: &str = "main"; + +/// Something written before the probes run. +pub(crate) enum Step { + /// A brain document. + Doc { + tenant: &'static str, + source: BrainSource, + title: &'static str, + text: &'static str, + }, + /// A learning stored at the root. + Learning { + kind: LearningKind, + text: &'static str, + confidence: f32, + }, + /// A thread of user turns, each with the tool calls the agent makes. + Chat { + tenant: &'static str, + agent: &'static str, + thread: String, + /// Days after the run's epoch the thread starts: orders threads in + /// time. + day: i64, + turns: Vec<(String, Vec)>, + }, +} + +/// How a probe reads memory. +#[derive(Debug, Clone)] +pub(crate) enum Via { + /// A new thread's first `pre_turn`. + Ask, + /// `start_session`, resuming `thread` (or none) for `focus` (or none). + Resume { + thread: Option<&'static str>, + focus: Option<&'static str>, + }, + /// `recall_for_compaction` of `thread`, dropping `dropped`. + Compact { + thread: &'static str, + dropped: Vec, + }, + /// The next `pre_turn` of `thread` at `turn_index`, with the turns from + /// `in_prompt_from` still in the prompt. + Continue { + thread: &'static str, + turn_index: u32, + in_prompt_from: u32, + }, + /// A `context.md` with one brief, `heading`, answering the probe's + /// question over the whole layout: the initial context a host compiles + /// once per session. + ContextDoc { heading: &'static str }, +} + +/// Whether a question shares its key words with the stored text. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Style { + /// It does. + Lexical, + /// It does not: only meaning connects them. + Paraphrase, +} + +/// One question and what a correct pack holds. +#[derive(Debug, Clone)] +pub(crate) struct Probe { + pub(crate) id: &'static str, + pub(crate) tenant: &'static str, + pub(crate) agent: &'static str, + pub(crate) via: Via, + pub(crate) question: &'static str, + pub(crate) style: Style, + pub(crate) expect: Vec<&'static str>, + pub(crate) stale: Vec<&'static str>, + pub(crate) forbidden: Vec<&'static str>, + /// Other wordings a correct answer may use instead of every `expect` + /// string ("Thursday" for "Thursdays"). They grade answers, not packs. + pub(crate) accept: Vec<&'static str>, +} + +impl Probe { + fn new(id: &'static str, agent: &'static str, question: &'static str, style: Style) -> Self { + Self { + id, + tenant: MAIN, + agent, + via: Via::Ask, + question, + style, + expect: Vec::new(), + stale: Vec::new(), + forbidden: Vec::new(), + accept: Vec::new(), + } + } + + fn expect(mut self, expect: &[&'static str]) -> Self { + self.expect = expect.to_vec(); + self + } + + fn stale(mut self, stale: &[&'static str]) -> Self { + self.stale = stale.to_vec(); + self + } + + fn accept(mut self, accept: &[&'static str]) -> Self { + self.accept = accept.to_vec(); + self + } + + fn forbid(mut self, forbidden: &[&'static str]) -> Self { + self.forbidden = forbidden.to_vec(); + self + } + + fn via(mut self, via: Via) -> Self { + self.via = via; + self + } + + fn tenant(mut self, tenant: &'static str) -> Self { + self.tenant = tenant; + self + } +} + +/// A named scenario. +pub(crate) struct Scenario { + pub(crate) name: &'static str, + pub(crate) about: &'static str, + pub(crate) steps: Vec, + pub(crate) probes: Vec, +} + +/// Every scenario, in run order. +pub(crate) fn all() -> Vec { + vec![ + brain_lookup(), + restart_recall(), + contradictions(), + tool_heavy(), + team_handoff(), + compaction(), + isolation(), + needle_in_noise(), + learnings(), + learning_from_feedback(), + surprise(), + conflicts(), + ] +} + +/// User turns with no tool calls. +fn said(lines: &[&str]) -> Vec<(String, Vec)> { + lines + .iter() + .map(|line| ((*line).to_string(), Vec::new())) + .collect() +} + +/// A tool step. +const fn tool(name: &'static str, result: &'static str) -> ToolStep { + ToolStep { name, result } +} + +fn doc(source: BrainSource, title: &'static str, text: &'static str) -> Step { + Step::Doc { + tenant: MAIN, + source, + title, + text, + } +} + +fn chat( + agent: &'static str, + thread: &'static str, + day: i64, + turns: Vec<(String, Vec)>, +) -> Step { + Step::Chat { + tenant: MAIN, + agent, + thread: thread.to_string(), + day, + turns, + } +} + +fn brain_lookup() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "brain_lookup", + about: "Company documents from six sources, asked about by an agent that never saw them", + steps: vec![ + doc( + BrainSource::Pdf, + "Refund policy", + "Refunds settle within five business days of approval. Enterprise customers \ + get a dedicated support channel.", + ), + doc( + BrainSource::Markdown, + "On-call handbook", + "The on-call rotation hands over every Monday at 09:00 UTC. Pages that are not \ + acknowledged within 15 minutes escalate to the engineering manager.", + ), + doc( + BrainSource::Notion, + "Pricing FAQ", + "The Team plan costs 40 dollars per seat per month. Annual billing gets two \ + months free.", + ), + doc( + BrainSource::Github, + "deploy/README.md", + "Production deploys run from the release branch through the ship-it workflow. \ + Rollbacks use make rollback ENV=prod.", + ), + doc( + BrainSource::Web, + "Status page", + "Scheduled maintenance windows are on Sundays between 02:00 and 04:00 UTC.", + ), + doc( + BrainSource::Markdown, + "Security policy", + "Customer data must never leave the eu-central-1 region. Access keys rotate \ + every 90 days.", + ), + ], + probes: vec![ + Probe::new( + "refund-days", + "support-01", + "How many business days do refunds take to settle?", + Lexical, + ) + .expect(&["five business days"]) + .accept(&["5 business days"]), + Probe::new( + "refund-paraphrase", + "support-01", + "If we give money back to a customer, when does it land?", + Paraphrase, + ) + .expect(&["five business days"]) + .accept(&["5 business days"]), + Probe::new( + "handover", + "support-01", + "When does the on-call rotation hand over?", + Lexical, + ) + .expect(&["Monday at 09:00"]) + .accept(&["Monday"]), + Probe::new( + "page-escalation", + "support-01", + "Who gets woken up if nobody answers an alert?", + Paraphrase, + ) + .expect(&["engineering manager"]), + Probe::new( + "seat-price", + "support-01", + "How much does the Team plan cost per seat?", + Lexical, + ) + .expect(&["40 dollars"]) + .accept(&["$40", "40 USD"]), + Probe::new( + "undo-release", + "support-01", + "How do I undo a bad release?", + Paraphrase, + ) + .expect(&["make rollback"]), + Probe::new( + "maintenance", + "support-01", + "When are the scheduled maintenance windows?", + Lexical, + ) + .expect(&["Sundays"]) + .accept(&["Sunday"]), + Probe::new( + "credential-rotation", + "support-01", + "How often must we change our credentials?", + Paraphrase, + ) + .expect(&["90 days"]), + ], + } +} + +fn restart_recall() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "restart_recall", + about: "A user introduces themselves; the agent restarts and must remember them", + steps: vec![chat( + "assistant-01", + "intro", + 0, + said(&[ + "Hi, I'm Dana and I lead the payments team.", + "I'm based in Lisbon, so my timezone is WET.", + "I prefer answers as short bullet points, no long essays.", + "Our project codename is Bluefin.", + "We ship to production on Thursdays.", + "Thanks, that's all for now.", + ]), + )], + probes: vec![ + Probe::new( + "resume-cold", + "assistant-01", + "What is the project codename?", + Lexical, + ) + .via(Via::Resume { + thread: None, + focus: None, + }) + .expect(&["Bluefin"]), + Probe::new( + "resume-focused", + "assistant-01", + "What is the user's timezone?", + Lexical, + ) + .via(Via::Resume { + thread: None, + focus: Some("the user's timezone and location"), + }) + .expect(&["WET"]) + .accept(&["Western European", "Lisbon"]), + Probe::new( + "name-team", + "assistant-01", + "What's my name, and which team do I lead?", + Lexical, + ) + .expect(&["Dana", "payments"]), + Probe::new( + "timezone", + "assistant-01", + "Which time zone should meetings with me be scheduled in?", + Paraphrase, + ) + .expect(&["WET"]) + .accept(&["Western European", "Lisbon"]), + Probe::new( + "codename", + "assistant-01", + "What's our project codename?", + Lexical, + ) + .expect(&["Bluefin"]), + Probe::new( + "release-day", + "assistant-01", + "Which weekday do our releases go out?", + Paraphrase, + ) + .expect(&["Thursdays"]) + .accept(&["Thursday"]), + Probe::new( + "format", + "assistant-01", + "How should you format replies for me?", + Paraphrase, + ) + .expect(&["bullet points"]) + .accept(&["bullet"]), + ], + } +} + +fn contradictions() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "contradictions", + about: "Facts change across three sessions; the newest value must win", + steps: vec![ + chat( + "ops-01", + "week-1", + 0, + said(&[ + "Our production region is us-east-1.", + "The monthly cloud budget is 5000 dollars.", + "Standup is on Tuesdays at 10:00.", + "The main database is Postgres 14.", + ]), + ), + chat( + "ops-01", + "week-2", + 7, + said(&[ + "Update: we migrated the production region to eu-west-2 last night.", + "Finance raised the monthly cloud budget to 8000 dollars.", + ]), + ), + chat( + "ops-01", + "week-3", + 14, + said(&[ + "Standup moved to Thursdays at 10:00.", + "Correction: the monthly cloud budget got cut back to 6500 dollars.", + ]), + ), + ], + probes: vec![ + Probe::new( + "region", + "ops-01", + "Which production region are we in?", + Lexical, + ) + .expect(&["eu-west-2"]) + .stale(&["us-east-1"]), + Probe::new( + "budget", + "ops-01", + "What is the monthly cloud budget?", + Lexical, + ) + .expect(&["6500"]) + .accept(&["6,500"]) + .stale(&["5000", "8000"]), + Probe::new("standup", "ops-01", "When is standup?", Lexical) + .expect(&["Thursdays"]) + .accept(&["Thursday"]) + .stale(&["Tuesdays"]), + Probe::new( + "spend-limit", + "ops-01", + "How much can we spend on infrastructure each month?", + Paraphrase, + ) + .expect(&["6500"]) + .accept(&["6,500"]) + .stale(&["5000", "8000"]), + Probe::new("database", "ops-01", "Which database do we run?", Lexical) + .expect(&["Postgres 14"]), + Probe::new( + "resume-latest", + "ops-01", + "What is the monthly cloud budget?", + Lexical, + ) + .via(Via::Resume { + thread: None, + focus: None, + }) + .expect(&["6500"]) + .accept(&["6,500"]) + .stale(&["5000", "8000"]), + ], + } +} + +fn tool_heavy() -> Scenario { + use Style::{Lexical, Paraphrase}; + let turns = vec![ + ( + "The checkout service is returning 500s, can you look?".to_string(), + vec![ + tool( + "search_logs", + "error code E4031: connection pool exhausted in checkout-db", + ), + tool( + "get_deploy_status", + "checkout version 2.14.3 deployed 40 minutes ago by ci-bot", + ), + tool("get_metrics", "checkout p99 latency 2.8s, up from 180ms"), + tool( + "list_alerts", + "2 firing: CheckoutErrorRate, DbPoolSaturation", + ), + ], + ), + ( + "Which tests cover the connection pool?".to_string(), + vec![ + tool( + "grep_repo", + "pool_size=10 set in services/checkout/config/db.toml", + ), + tool( + "run_tests", + "checkout::db: 3 passed, 1 failed: test_pool_saturation", + ), + tool("read_file", "db.toml: max_overflow=0, timeout=5s"), + ], + ), + ( + "Roll it back please.".to_string(), + vec![ + tool("rollback", "checkout rolled back to 2.14.2"), + tool("get_metrics", "checkout p99 back to 190ms"), + tool("create_ticket", "ticket OPS-7781 opened for pool sizing"), + tool( + "notify_channel", + "posted incident summary to #checkout-oncall", + ), + ], + ), + ( + "Find the commit that caused it.".to_string(), + vec![ + tool("git_log", "commit 9f2c1ab lowered pool_size from 50 to 10"), + tool("git_blame", "change authored by jmiller in PR 4412"), + tool("get_pr", "PR 4412 approved by one reviewer, merged Friday"), + ], + ), + ("Thanks, that's it.".to_string(), Vec::new()), + ]; + Scenario { + name: "tool_heavy", + about: "An incident debugged through 17 tool calls; the facts live in tool results", + steps: vec![ + chat("coder-42", "incident-1", 0, turns), + chat( + "coder-42", + "chores", + 1, + said(&[ + "Bump the lint config to the new rules.", + "Rename the billing module to invoicing.", + ]), + ), + ], + probes: vec![ + Probe::new( + "error-code", + "coder-42", + "What error code did the checkout logs show?", + Lexical, + ) + .expect(&["E4031"]), + Probe::new("failing-test", "coder-42", "Which test failed?", Lexical) + .expect(&["test_pool_saturation"]), + Probe::new( + "rollback-version", + "coder-42", + "Which version did checkout roll back to?", + Lexical, + ) + .expect(&["2.14.2"]), + Probe::new( + "ticket", + "coder-42", + "What ticket tracks the pool sizing?", + Lexical, + ) + .expect(&["OPS-7781"]), + Probe::new( + "culprit-commit", + "coder-42", + "Which change caused the outage?", + Paraphrase, + ) + .expect(&["9f2c1ab"]), + Probe::new( + "who-to-ask", + "coder-42", + "Who should I talk to about the regression?", + Paraphrase, + ) + .expect(&["jmiller"]), + ], + } +} + +fn team_handoff() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "team_handoff", + about: "Support learns of a bug; a coding agent must see it without being told", + steps: vec![chat( + "support-01", + "ticket-9", + 0, + vec![ + ( + "Customer Acme Corp reports duplicated invoices since Monday.".to_string(), + vec![tool( + "lookup_account", + "Acme Corp is on the Enterprise plan, account id ACC-2209", + )], + ), + ( + "They were charged twice for September.".to_string(), + Vec::new(), + ), + ], + )], + probes: vec![ + Probe::new( + "duplicate-invoices", + "coder-42", + "Is any customer reporting duplicated invoices?", + Lexical, + ) + .expect(&["Acme"]), + Probe::new( + "account-id", + "coder-42", + "What is Acme Corp's account id?", + Lexical, + ) + .expect(&["ACC-2209"]), + Probe::new( + "billed-twice", + "coder-42", + "Has anyone been billed two times for the same month?", + Paraphrase, + ) + .expect(&["charged twice"]) + .accept(&["twice"]), + ], + } +} + +/// The 16 filler turns of the compaction scenario, after its four facts. +fn agenda() -> Vec { + (1..=16) + .map(|n| { + format!("Next, agenda item {n}: assign an owner and a deadline for workstream {n}.") + }) + .collect() +} + +fn compaction() -> Scenario { + use Style::Lexical; + let facts = [ + "The offsite is in Porto on 12 May.", + "The offsite budget is 20000 euros.", + "Catering is booked with Taberna Azul.", + "We need a vegetarian option for 9 people.", + ]; + let mut turns = said(&facts); + turns.extend(agenda().into_iter().map(|line| (line, Vec::new()))); + // 20 exchanges: turns 0..40. The first 4 exchanges are the facts. + let dropped: Vec = facts + .iter() + .map(|fact| (*fact).to_string()) + .chain(agenda().into_iter().take(8)) + .collect(); + Scenario { + name: "compaction", + about: "A 20-exchange planning thread whose early facts have left the prompt", + steps: vec![chat("planner-07", "offsite", 0, turns)], + probes: vec![ + Probe::new( + "carry-over", + "planner-07", + "Where is the offsite and who caters it?", + Lexical, + ) + .via(Via::Compact { + thread: "offsite", + dropped, + }) + .expect(&["Porto", "Taberna Azul"]), + Probe::new( + "out-of-window", + "planner-07", + "How many vegetarian meals do we need?", + Lexical, + ) + .via(Via::Continue { + thread: "offsite", + turn_index: 40, + in_prompt_from: 32, + }) + .expect(&["9 people"]) + .accept(&["9 vegetarian", "nine vegetarian", "9 meals"]) + // Turns 32..40 are agenda items 13 to 16: still in the prompt. + .forbid(&[ + "agenda item 13", + "agenda item 14", + "agenda item 15", + "agenda item 16", + ]), + ], + } +} + +fn isolation() -> Scenario { + use Style::Lexical; + Scenario { + name: "isolation", + about: "Two tenants side by side: neither may see the other's brain or turns", + steps: vec![ + Step::Doc { + tenant: "acme", + source: BrainSource::Markdown, + title: "M&A memo", + text: "The acquisition target is Zephyr Labs, under the code name Kestrel.", + }, + Step::Chat { + tenant: "acme", + agent: "cfo-bot", + thread: "q3".to_string(), + day: 0, + turns: said(&["Our Q3 revenue was 4.2 million dollars."]), + }, + Step::Doc { + tenant: "globex", + source: BrainSource::Markdown, + title: "Office handbook", + text: "Lunch is served at noon in the atrium.", + }, + Step::Chat { + tenant: "globex", + agent: "cfo-bot", + thread: "q3".to_string(), + day: 0, + turns: said(&["Our Q3 revenue was strong this year."]), + }, + ], + probes: vec![ + Probe::new( + "own-tenant", + "cfo-bot", + "What is the acquisition target?", + Lexical, + ) + .tenant("acme") + .expect(&["Zephyr"]), + Probe::new( + "other-tenant-brain", + "cfo-bot", + "What is the acquisition target?", + Lexical, + ) + .tenant("globex") + .forbid(&["Zephyr", "Kestrel"]), + Probe::new( + "other-tenant-turns", + "cfo-bot", + "What was our Q3 revenue?", + Lexical, + ) + .tenant("globex") + .expect(&["strong"]) + .forbid(&["4.2 million"]), + ], + } +} + +fn needle_in_noise() -> Scenario { + use Style::{Lexical, Paraphrase}; + let topics = [ + "resetting a password", + "exporting invoices to CSV", + "changing the billing email", + "adding a teammate", + "enabling two-factor login", + "downgrading a plan", + ]; + let mut steps: Vec = (0..24) + .map(|n| { + let topic = topics[n % topics.len()]; + Step::Chat { + tenant: MAIN, + agent: "support-02", + thread: format!("noise-{n}"), + day: 0, + turns: vec![( + format!( + "Customer {} asked about {topic}; I sent the help article.", + 100 + n + ), + Vec::new(), + )], + } + }) + .collect(); + steps.insert( + 12, + chat( + "support-02", + "vip", + 0, + said(&["Heads up: the VIP account Orion Freight must always be routed to Priya."]), + ), + ); + Scenario { + name: "needle_in_noise", + about: "One routing rule hidden among 24 routine support threads", + steps, + probes: vec![ + Probe::new( + "needle", + "support-02", + "Who handles the Orion Freight account?", + Lexical, + ) + .expect(&["Priya"]), + Probe::new( + "needle-paraphrase", + "support-02", + "Which teammate gets our big logistics client?", + Paraphrase, + ) + .expect(&["Priya"]), + ], + } +} + +fn learnings() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "learnings", + about: "Explicit learnings must lead the pack, ahead of raw history", + steps: vec![ + Step::Learning { + kind: LearningKind::Procedure, + text: "Always confirm the customer's plan before quoting a price.", + confidence: 0.9, + }, + Step::Learning { + kind: LearningKind::Preference, + text: "The team prefers metric units in every report.", + confidence: 0.8, + }, + chat( + "sales-03", + "quote-1", + 0, + said(&["I quoted Initech 12 seats at the list price."]), + ), + ], + probes: vec![ + Probe::new( + "quote-procedure", + "sales-03", + "Can you quote a price for 12 seats?", + Lexical, + ) + .expect(&["confirm the customer's plan"]) + .accept(&[ + "confirm your plan", + "confirm the plan", + "confirm which plan", + "confirm the customer", + ]), + Probe::new( + "units", + "sales-03", + "Should the report use miles or kilometres?", + Paraphrase, + ) + .expect(&["metric units"]) + .accept(&["kilomet", "metric"]), + ], + } +} + +/// The `context.md` brief that should carry standing instructions. +const PREFERENCES: Via = Via::ContextDoc { + heading: "Preferences and standing instructions", +}; + +/// The `context.md` brief that should carry surprises. +const EVENTS: Via = Via::ContextDoc { + heading: "Recent important events", +}; + +/// A fresh session's initial context: `start_session` with nothing to go on. +const COLD_START: Via = Via::Resume { + thread: None, + focus: None, +}; + +fn learning_from_feedback() -> Scenario { + use Style::{Lexical, Paraphrase}; + let turns = vec![ + ( + "Install the dependencies for the web app.".to_string(), + vec![tool( + "run_shell", + "npm install failed: this repo uses pnpm workspaces (found pnpm-lock.yaml)", + )], + ), + ( + "Right, in this repo always use pnpm, never npm.".to_string(), + Vec::new(), + ), + ( + "Now deploy the staging build.".to_string(), + vec![tool( + "run_shell", + "kubectl apply -f k8s/staging refused: not logged in to the cluster", + )], + ), + ( + "Don't run kubectl yourself. We always deploy with make ship ENV=staging.".to_string(), + Vec::new(), + ), + ( + "Also: open pull requests ready for review, never as drafts.".to_string(), + Vec::new(), + ), + ]; + Scenario { + name: "learning_from_feedback", + about: "The agent makes mistakes, the user corrects it; the corrections must stick", + steps: vec![ + chat("coder-42", "setup", 0, turns), + chat( + "coder-42", + "chores", + 1, + said(&["Bump the eslint config.", "Rename utils.ts to format.ts."]), + ), + ], + probes: vec![ + Probe::new( + "install-cold", + "coder-42", + "Which package manager should I use in this repo?", + Paraphrase, + ) + .via(COLD_START) + .expect(&["pnpm"]), + Probe::new( + "install-context", + "coder-42", + "What preferences and standing instructions has the user given?", + Lexical, + ) + .via(PREFERENCES) + .expect(&["pnpm", "make ship"]) + .accept(&["pnpm"]), + Probe::new( + "install", + "coder-42", + "How do I install the dependencies?", + Lexical, + ) + .expect(&["pnpm"]), + Probe::new( + "deploy", + "coder-42", + "What's the command to deploy staging?", + Lexical, + ) + .expect(&["make ship"]) + .stale(&["kubectl apply"]), + Probe::new( + "draft-pr", + "coder-42", + "Should my new pull request be a draft?", + Paraphrase, + ) + .expect(&["ready for review"]) + .accept(&[ + "not as a draft", + "never as a draft", + "never as drafts", + "not a draft", + ]), + ], + } +} + +fn surprise() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "surprise", + about: "A long-stable routine breaks and a metric spikes; the surprise must surface", + steps: vec![ + chat( + "ops-01", + "baseline", + 0, + said(&[ + "The nightly backup has succeeded every night for over a year.", + "Checkout's error rate always sits under 0.1%.", + ]), + ), + chat( + "ops-01", + "morning-check", + 5, + vec![ + ( + "Check last night's backup.".to_string(), + vec![tool( + "backup_status", + "FAILED: disk full on backup-02, the first failure in 412 days", + )], + ), + ( + "That's a surprise. Keep an eye on backup-02.".to_string(), + Vec::new(), + ), + ( + "How is checkout doing?".to_string(), + vec![tool( + "get_metrics", + "checkout error rate 4.7% since 02:00, about 47 times its usual level", + )], + ), + ], + ), + ], + probes: vec![ + Probe::new( + "unusual-cold", + "ops-01", + "Is anything unusual going on?", + Paraphrase, + ) + .via(COLD_START) + .expect(&["backup-02"]) + .accept(&["disk full", "backup failed", "4.7%"]), + Probe::new( + "events-context", + "ops-01", + "What important events happened recently?", + Paraphrase, + ) + .via(EVENTS) + .expect(&["backup-02"]) + .accept(&["disk full", "backup failed", "4.7%"]), + Probe::new( + "backup", + "ops-01", + "Did last night's backup succeed?", + Lexical, + ) + .expect(&["FAILED"]) + .accept(&["fail", "did not succeed", "didn't succeed"]), + Probe::new( + "checkout", + "ops-01", + "Is anything off with checkout?", + Paraphrase, + ) + .expect(&["4.7%"]) + .accept(&["error rate"]), + Probe::new( + "surprising", + "ops-01", + "What surprised us recently?", + Paraphrase, + ) + .expect(&["backup-02"]) + .accept(&["disk full", "backup failed", "4.7%"]), + ], + } +} + +fn conflicts() -> Scenario { + use Style::{Lexical, Paraphrase}; + Scenario { + name: "conflicts", + about: "Sources disagree (a policy, a colleague, an agent); the disagreement must show", + steps: vec![ + doc( + BrainSource::Pdf, + "Refund policy", + "Refunds settle within five business days of approval.", + ), + chat( + "support-01", + "finance-sync", + 0, + said(&["Finance told me refunds now take ten business days."]), + ), + chat( + "sales-02", + "acme-call", + 1, + said(&[ + "I promised Acme their refund lands in seven days.", + "Acme is on the Enterprise plan.", + ]), + ), + chat( + "support-01", + "acme-ticket", + 2, + said(&["Acme downgraded to the Team plan last week."]), + ), + ], + probes: vec![ + Probe::new( + "refund-days", + "support-01", + "How long do refunds take?", + Lexical, + ) + .expect(&["five", "ten"]) + .accept(&["conflict", "inconsistent", "disagree", "differ"]), + Probe::new( + "refund-cold", + "support-01", + "How long do refunds take?", + Lexical, + ) + .via(COLD_START) + .expect(&["five", "ten"]) + .accept(&["conflict", "inconsistent", "disagree", "differ"]), + Probe::new("acme-plan", "support-01", "Which plan is Acme on?", Lexical) + .expect(&["Team plan"]) + .stale(&["Enterprise"]), + Probe::new( + "promise", + "support-01", + "Did anyone promise Acme a refund timeline that disagrees with policy?", + Paraphrase, + ) + .expect(&["seven days"]) + .accept(&["7 days"]), + ], + } +} diff --git a/crates/tinymemory-integrations/examples/memory_eval/score.rs b/crates/tinymemory-integrations/examples/memory_eval/score.rs new file mode 100644 index 00000000..ecca88a4 --- /dev/null +++ b/crates/tinymemory-integrations/examples/memory_eval/score.rs @@ -0,0 +1,249 @@ +//! Scoring a pack against a probe, and summing the scores up. +//! +//! A pack is cut into **units**: each bullet, and each prose paragraph (an +//! answered section), in the order the model reads them. Every check is a +//! case-insensitive substring match: +//! +//! - **hit**: every `expect` string is somewhere in the pack. +//! - **rank**: the 1-based unit holding the first `expect` string. Its +//! reciprocal averages to the MRR. +//! - **stale first**: a superseded value comes in an earlier unit than the +//! current one, or the current one is missing. That is the error a reader +//! is most likely to repeat. **Fresh first** is its complement, counted +//! over the probes that name superseded values. +//! - **leak**: a `forbidden` string is in the pack. +//! - **answer**: the scripted agent's extractive answer (see `agent`) holds +//! every `expect` string and no stale one. With `--llm`, a model's answer +//! from the same pack is graded the same way. + +use serde::Serialize; + +use crate::agent::answer; +use crate::scenarios::{Probe, Style, Via}; + +/// One probe's outcome in one phase. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct ProbeResult { + pub(crate) scenario: &'static str, + pub(crate) phase: &'static str, + pub(crate) id: &'static str, + pub(crate) via: &'static str, + pub(crate) style: &'static str, + /// `None` when the probe expects nothing (a pure leak check). + pub(crate) hit: Option, + pub(crate) rank: Option, + /// The heading of the section holding the first expected string. + pub(crate) section: Option, + /// Whether the probe names superseded values. + pub(crate) contradiction: bool, + pub(crate) stale_present: bool, + pub(crate) stale_first: bool, + /// Whether the probe names forbidden strings. + pub(crate) leak_checked: bool, + pub(crate) leak: bool, + pub(crate) answer: Option, + pub(crate) answer_ok: Option, + /// The `--llm` model's answer from the same pack. + pub(crate) llm_answer: Option, + pub(crate) llm_ok: Option, + /// Synthesis phase on CortexDB: whether a fact or belief CortexDB + /// derived holds the expected answer, whether or not the pack shows it. + pub(crate) captured: Option, + pub(crate) ms: f64, + pub(crate) tokens: usize, + pub(crate) units: usize, + pub(crate) markdown: String, +} + +/// Which lifecycle call a probe used. +pub(crate) fn via_name(via: &Via) -> &'static str { + match via { + Via::Ask => "pre_turn", + Via::Resume { .. } => "start_session", + Via::Compact { .. } => "recall_for_compaction", + Via::Continue { .. } => "pre_turn (in thread)", + Via::ContextDoc { .. } => "context.md", + } +} + +/// The units of `markdown`, each with its section heading. +fn units(markdown: &str) -> Vec<(String, String)> { + let mut heading = String::new(); + let mut out = Vec::new(); + let mut in_frontmatter = false; + for line in markdown.lines() { + let line = line.trim(); + if line == "---" { + in_frontmatter = !in_frontmatter; + continue; + } + if in_frontmatter || line.is_empty() || line.starts_with("# ") { + continue; + } + if let Some(title) = line.strip_prefix("## ") { + heading = title.to_string(); + continue; + } + let body = line.strip_prefix("- ").unwrap_or(line); + out.push((heading.clone(), body.to_lowercase())); + } + out +} + +/// Scores one pack. +pub(crate) fn score( + scenario: &'static str, + phase: &'static str, + probe: &Probe, + markdown: &str, + tokens: usize, + ms: f64, +) -> ProbeResult { + let lower = markdown.to_lowercase(); + let units = units(markdown); + let first = |needle: &str| { + let needle = needle.to_lowercase(); + units.iter().position(|(_, unit)| unit.contains(&needle)) + }; + let expected = probe.expect.first().and_then(|needle| first(needle)); + let stale_at = probe.stale.iter().filter_map(|needle| first(needle)).min(); + let has = |needle: &&str| lower.contains(&needle.to_lowercase()); + let answered = answer(markdown, probe.question); + let answer_ok = grade(probe, answered.as_deref()); + ProbeResult { + scenario, + phase, + id: probe.id, + via: via_name(&probe.via), + style: match probe.style { + Style::Lexical => "lexical", + Style::Paraphrase => "paraphrase", + }, + hit: (!probe.expect.is_empty()).then(|| probe.expect.iter().all(has)), + rank: expected.map(|at| at + 1), + section: expected.map(|at| units[at].0.clone()), + contradiction: !probe.stale.is_empty(), + stale_present: probe.stale.iter().any(has), + stale_first: match (stale_at, expected) { + (Some(stale), Some(fresh)) => stale < fresh, + (Some(_), None) => true, + _ => false, + }, + leak_checked: !probe.forbidden.is_empty(), + leak: probe.forbidden.iter().any(has), + answer: answered, + answer_ok, + llm_answer: None, + llm_ok: None, + captured: None, + ms, + tokens, + units: units.len(), + markdown: markdown.to_string(), + } +} + +/// Whether `answer` is right for `probe`: every expected string (or one of +/// its accepted wordings) and no stale one. `None` for a probe that expects nothing. +pub(crate) fn grade(probe: &Probe, answer: Option<&str>) -> Option { + (!probe.expect.is_empty()).then(|| { + answer.is_some_and(|text| { + let text = text.to_lowercase(); + let has = |needle: &&str| text.contains(&needle.to_lowercase()); + (probe.expect.iter().all(has) || probe.accept.iter().any(has)) + && !probe.stale.iter().any(has) + }) + }) +} + +/// Totals over a set of results. +#[derive(Debug, Clone, Default, Serialize)] +pub(crate) struct Totals { + pub(crate) probes: usize, + pub(crate) scored: usize, + pub(crate) hits: usize, + pub(crate) mrr: f64, + pub(crate) answers_ok: usize, + pub(crate) llm_scored: usize, + pub(crate) llm_ok: usize, + pub(crate) captured_checked: usize, + pub(crate) captured: usize, + pub(crate) contradictions: usize, + pub(crate) fresh_first: usize, + pub(crate) leak_checks: usize, + pub(crate) leaks: usize, +} + +impl Totals { + /// Sums `results`. + pub(crate) fn of<'a>(results: impl IntoIterator) -> Self { + let mut totals = Self::default(); + let mut reciprocal = 0.0; + for result in results { + totals.probes += 1; + if let Some(hit) = result.hit { + totals.scored += 1; + totals.hits += usize::from(hit); + reciprocal += result.rank.map_or(0.0, |rank| 1.0 / rank as f64); + totals.answers_ok += usize::from(result.answer_ok == Some(true)); + if let Some(held) = result.captured { + totals.captured_checked += 1; + totals.captured += usize::from(held); + } + if let Some(ok) = result.llm_ok { + totals.llm_scored += 1; + totals.llm_ok += usize::from(ok); + } + } + if result.contradiction { + totals.contradictions += 1; + totals.fresh_first += usize::from(result.hit == Some(true) && !result.stale_first); + } + totals.leak_checks += usize::from(result.leak_checked); + totals.leaks += usize::from(result.leak); + } + if totals.scored > 0 { + totals.mrr = reciprocal / totals.scored as f64; + } + totals + } + + /// `part` of `whole` as a percentage cell. + pub(crate) fn pct(part: usize, whole: usize) -> String { + if whole == 0 { + "–".to_string() + } else { + format!( + "{:.0}% ({part}/{whole})", + 100.0 * part as f64 / whole as f64 + ) + } + } +} + +/// Latency percentiles of a set of samples, in milliseconds. +#[derive(Debug, Clone, Default, Serialize)] +pub(crate) struct Latency { + pub(crate) n: usize, + pub(crate) p50: f64, + pub(crate) p95: f64, + pub(crate) max: f64, +} + +impl Latency { + /// Percentiles of `samples`. + pub(crate) fn of(samples: &[f64]) -> Self { + if samples.is_empty() { + return Self::default(); + } + let mut sorted = samples.to_vec(); + sorted.sort_by(f64::total_cmp); + let at = |q: f64| sorted[((sorted.len() - 1) as f64 * q).round() as usize]; + Self { + n: sorted.len(), + p50: at(0.5), + p95: at(0.95), + max: sorted[sorted.len() - 1], + } + } +} diff --git a/crates/tinymemory-integrations/src/brain/mod.rs b/crates/tinymemory-integrations/src/brain/mod.rs new file mode 100644 index 00000000..8d40dc3a --- /dev/null +++ b/crates/tinymemory-integrations/src/brain/mod.rs @@ -0,0 +1,102 @@ +//! Files into the brain: conversion to a [`BrainDocument`] of the right +//! source type. +//! +//! [`tinymemory_tools::Brain`] stores text; this module is the step before +//! it. [`brain_document`] runs a [`RawDocument`] through a +//! [`DocumentConverter`] (the [`crate::documents`] pipeline) and places the +//! markdown under a [`BrainSource`] — the one the caller names, or the one +//! its detected format implies ([`source_for`]): a PDF lands in +//! `source:pdf`, markdown and plain text in `source:markdown`, HTML in +//! `source:web`. +//! +//! # Example +//! +//! ``` +//! use std::sync::Arc; +//! use tinymemory_api::MemoryMeta; +//! use tinymemory_api::conformance::ReferenceEngine; +//! use tinymemory_integrations::brain::brain_document; +//! use tinymemory_integrations::documents::{ConverterChain, RawDocument}; +//! use tinymemory_tools::{Brain, BrainSource, MemoryLayout}; +//! +//! # let runtime = tokio::runtime::Builder::new_current_thread().build()?; +//! # runtime.block_on(async { +//! let file = RawDocument::new("# Refunds\n\nRefunds take five business days.\n") +//! .with_filename("handbook/refunds.md"); +//! let document = brain_document(&ConverterChain::default(), &file, None, MemoryMeta::default()) +//! .await?; +//! assert_eq!(document.source, BrainSource::Markdown); +//! assert_eq!(document.title.as_deref(), Some("Refunds")); +//! +//! let brain = Brain::new(Arc::new(ReferenceEngine::new()), MemoryLayout::default()); +//! brain.ingest(document).await?; +//! # Ok::<(), Box>(()) +//! # })?; +//! # Ok::<(), Box>(()) +//! ``` + +use tinymemory_api::{DocumentBody, MemoryMeta, StoreItem}; +use tinymemory_tools::{BrainDocument, BrainSource}; + +use crate::documents::{ + DocumentConverter, DocumentFormat, Error, RawDocument, Result, converted_item, +}; + +/// The brain source a document of `format` belongs to when the caller names +/// none: PDFs to `pdf`, markdown and plain text to `markdown`, HTML to `web`, +/// and each other format to a source of its own name (`docx`, `xlsx`, +/// `pptx`, `code`, `other`). +#[must_use] +pub fn source_for(format: DocumentFormat) -> BrainSource { + match format { + DocumentFormat::Pdf => BrainSource::Pdf, + DocumentFormat::Markdown | DocumentFormat::PlainText => BrainSource::Markdown, + DocumentFormat::Html => BrainSource::Web, + DocumentFormat::Docx => BrainSource::Other("docx".to_string()), + DocumentFormat::Xlsx => BrainSource::Other("xlsx".to_string()), + DocumentFormat::Pptx => BrainSource::Other("pptx".to_string()), + DocumentFormat::Code => BrainSource::Other("code".to_string()), + DocumentFormat::Unknown => BrainSource::Other("other".to_string()), + } +} + +/// Converts `document` through `converter` into a brain document of +/// `source` (or the source its format implies, see [`source_for`]), +/// carrying `meta`. The title and MIME type come from the conversion, and +/// `meta.language` is filled as [`crate::documents::document_item`] fills +/// it. +/// +/// # Errors +/// +/// Whatever the converter returns (see +/// [`crate::documents::document_item`]). +pub async fn brain_document( + converter: &dyn DocumentConverter, + document: &RawDocument, + source: Option, + meta: MemoryMeta, +) -> Result { + let converted = converter.convert(document).await?; + let source = source.unwrap_or_else(|| source_for(converted.format)); + match converted_item(converted, document, meta) { + StoreItem::Document { + title, + body: DocumentBody::Text(text), + mime, + meta, + } => Ok(BrainDocument { + source, + title, + text, + mime, + meta, + }), + _ => Err(Error::Invalid( + "conversion did not produce a text document".to_string(), + )), + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-integrations/src/brain/mod_tests.rs b/crates/tinymemory-integrations/src/brain/mod_tests.rs new file mode 100644 index 00000000..7bf29aee --- /dev/null +++ b/crates/tinymemory-integrations/src/brain/mod_tests.rs @@ -0,0 +1,69 @@ +//! Converting raw files into brain documents. + +use super::*; +use crate::documents::ConverterChain; + +#[test] +fn every_format_has_a_source() { + assert_eq!(source_for(DocumentFormat::Pdf), BrainSource::Pdf); + assert_eq!(source_for(DocumentFormat::PlainText), BrainSource::Markdown); + assert_eq!(source_for(DocumentFormat::Html), BrainSource::Web); + assert_eq!( + source_for(DocumentFormat::Xlsx), + BrainSource::Other("xlsx".into()) + ); + assert_eq!(source_for(DocumentFormat::Unknown).to_string(), "other"); +} + +#[tokio::test] +async fn html_lands_on_the_web_unless_the_caller_says_otherwise() { + let page = RawDocument::new( + "Pricing

Pro is $20.

", + ) + .with_filename("pricing.html"); + let chain = ConverterChain::default(); + let document = brain_document(&chain, &page, None, MemoryMeta::default()) + .await + .unwrap(); + assert_eq!(document.source, BrainSource::Web); + assert_eq!(document.mime.as_deref(), Some("text/html")); + assert!(document.text.contains("Pro is $20.")); + + let notion = brain_document( + &chain, + &page, + Some(BrainSource::Notion), + MemoryMeta::default(), + ) + .await + .unwrap(); + assert_eq!(notion.source, BrainSource::Notion); +} + +#[tokio::test] +async fn the_caller_s_metadata_is_kept_and_language_filled() { + let file = RawDocument::new("fn main() {}\n").with_filename("src/main.rs"); + let meta = MemoryMeta { + repo: Some("acme/app".into()), + ..MemoryMeta::default() + }; + let document = brain_document(&ConverterChain::default(), &file, None, meta) + .await + .unwrap(); + assert_eq!(document.source, BrainSource::Other("code".into())); + assert_eq!(document.meta.repo.as_deref(), Some("acme/app")); + assert_eq!(document.meta.language.as_deref(), Some("rust")); +} + +#[tokio::test] +async fn an_empty_file_is_refused() { + let error = brain_document( + &ConverterChain::default(), + &RawDocument::new("").with_filename("empty.md"), + None, + MemoryMeta::default(), + ) + .await + .unwrap_err(); + assert!(matches!(error, Error::Invalid(_)), "{error:?}"); +} diff --git a/crates/tinymemory-integrations/src/cortex/README.md b/crates/tinymemory-integrations/src/cortex/README.md index 1dd95807..582b3732 100644 --- a/crates/tinymemory-integrations/src/cortex/README.md +++ b/crates/tinymemory-integrations/src/cortex/README.md @@ -39,6 +39,15 @@ From `tinymemory_integrations::cortex`: - `Error`/`Result` (the contract's own `tinymemory_api::Error`), `error_code`, `is_insufficient_credits` +Beyond the contract's reads and writes, the engine consolidates: Direct +posts `v1/beliefs/build` once per held scope a `ConsolidateRequest` admits +(`engine/consolidate.rs`, declared `Consolidation::OnDemand`) and reports the +beliefs built, since the server builds within the request; hosted +declares `Consolidation::Scheduled` and sends nothing. What was built is +read back by `beliefs` (`engine/beliefs.rs`): a `beliefs`-only recall per +held scope for a query, or the `v1/beliefs` listing without one, each belief +a `Learning` hit tagged `belief`. Fetch and list are unchanged. + A host usually goes through the registry instead of naming the engine: `tinymemory_integrations::{MemoryConfig, EngineCredential, build_engine, list_engines}` (modules `config` and `registry`). @@ -130,7 +139,9 @@ as prefixes, so they cannot be labelled and are filtered only client-side. earlier store failed part-way), only the missing turns are written. Direct writes `v1/experience?wait=indexed`, or `v1/experience/bulk?wait=indexed` with `ordering: strict_temporal` when an item has two or more events due. - Hosted writes one event at a time, in order. Every write uses a fresh + Hosted writes one event at a time, in order. `store_with` with + `WaitFor::Accepted` drops `?wait=indexed` and skips the waits below: the + agent lifecycle's live turns return once CortexDB captured them. Every write uses a fresh `idempotency_key`, never a content-derived one, because CortexDB keeps a forgotten event's key and would swallow a re-store. Then one listing wait per scope written (for its last event) and one ranked-recall wait (best-effort) diff --git a/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs b/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs index 17e8eb36..561a03cc 100644 --- a/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/descriptor/mod.rs @@ -16,8 +16,16 @@ //! promise a ranking the wire cannot ask for, so both descriptors list //! `Hybrid` alone and the other modes fail with //! [`tinymemory_api::Error::Unsupported`]. +//! +//! # Consolidation +//! +//! CortexDB builds beliefs on demand at `v1/beliefs/build`, one scope per +//! call, so the direct descriptor declares [`Consolidation::OnDemand`]. The +//! TinyHumans backend exposes no build route; CortexDB's own scheduler +//! consolidates behind it, so the hosted descriptor declares +//! [`Consolidation::Scheduled`]. -use tinymemory_api::{EngineDescriptor, FetchMode}; +use tinymemory_api::{Consolidation, EngineDescriptor, FetchMode}; /// Configuration id of CortexDB reached directly. pub const CORTEXDB_ENGINE_ID: &str = "cortexdb"; @@ -36,7 +44,7 @@ const FETCH_MODES: [FetchMode; 1] = [FetchMode::Hybrid]; /// The descriptor of CortexDB reached directly: not hosted by a third party, /// an endpoint is optional ([`CORTEX_API_ENDPOINT`] by default), an API key -/// is required, and fetch is hybrid only. +/// is required, fetch is hybrid only, and beliefs build on demand. #[must_use] pub fn cortexdb_descriptor() -> EngineDescriptor { EngineDescriptor { @@ -49,12 +57,14 @@ pub fn cortexdb_descriptor() -> EngineDescriptor { needs_key: true, default_endpoint: Some(CORTEX_API_ENDPOINT), fetch_modes: FETCH_MODES.to_vec(), + consolidation: Consolidation::OnDemand, } } /// The descriptor of CortexDB behind the TinyHumans backend: hosted, the /// endpoint defaults to [`TINYHUMANS_API_ENDPOINT`], a bearer (session JWT or -/// API key) is required, and fetch is hybrid only. +/// API key) is required, fetch is hybrid only, and beliefs build on the +/// server's schedule. #[must_use] pub fn tinyhumans_descriptor() -> EngineDescriptor { EngineDescriptor { @@ -66,6 +76,7 @@ pub fn tinyhumans_descriptor() -> EngineDescriptor { needs_key: true, default_endpoint: Some(TINYHUMANS_API_ENDPOINT), fetch_modes: FETCH_MODES.to_vec(), + consolidation: Consolidation::Scheduled, } } @@ -103,12 +114,19 @@ impl CortexWire { (Self::Direct, Route::Answer) => "v1/answer", (Self::Direct, Route::Health) => "v1/admin/health", (Self::Direct, Route::Scopes) => "v1/scopes/list", + (Self::Direct, Route::BuildBeliefs) => "v1/beliefs/build", + (Self::Direct, Route::Beliefs) => "v1/beliefs", (Self::TinyHumans, Route::Experience | Route::Bulk) => "memory/experience", (Self::TinyHumans, Route::Events) => "memory/events", (Self::TinyHumans, Route::Recall) => "memory/recall", (Self::TinyHumans, Route::Forget) => "memory/forget", (Self::TinyHumans, Route::Answer) => "memory/answer", (Self::TinyHumans, Route::Health | Route::Scopes) => "memory/scopes", + // Never sent: the hosted descriptor declares scheduled + // consolidation, so `consolidate` makes no request there. + (Self::TinyHumans, Route::BuildBeliefs) => "memory/beliefs/build", + // Never sent: hosted beliefs are read through recall only. + (Self::TinyHumans, Route::Beliefs) => "memory/beliefs", } } } @@ -132,6 +150,10 @@ pub(crate) enum Route { Health, /// List the caller's registered scopes under a prefix. Scopes, + /// Build one scope's beliefs on demand (Direct only). + BuildBeliefs, + /// List one scope's beliefs (Direct only). + Beliefs, } #[cfg(test)] diff --git a/crates/tinymemory-integrations/src/cortex/engine/beliefs.rs b/crates/tinymemory-integrations/src/cortex/engine/beliefs.rs new file mode 100644 index 00000000..8b17c390 --- /dev/null +++ b/crates/tinymemory-integrations/src/cortex/engine/beliefs.rs @@ -0,0 +1,203 @@ +//! Beliefs: the claims CortexDB builds (`v1/beliefs/build`), read as +//! learning hits. +//! +//! CortexDB keeps beliefs in a derived layer beside the events this crate +//! writes, so fetch and list (which decode events) never return them. This +//! read serves [`tinymemory_api::MemoryEngine::beliefs`] instead: +//! +//! - **Which scopes.** Every scope CortexDB holds in the request's reach, +//! whatever its kind (`scopes::held`): a build writes its beliefs into +//! the scope it read, so a conversation's beliefs sit with the +//! conversation. +//! - **With a query**, one recall pack per scope asks for the `beliefs` +//! layer only (no events), a few scopes at once, and the scopes' rankings +//! are interleaved rank by rank. +//! - **Without one**, the Direct wire lists each scope's beliefs +//! (`GET v1/beliefs`) and orders them most confident, then newest. The +//! hosted wire has no listing, so it answers none. +//! +//! Only `supported` and `contested` beliefs are read; a contested one says +//! so in its text. A belief is rendered as a sentence, `subject predicate +//! object` with the predicate's underscores as spaces, and becomes a +//! [`LearningKind::Fact`] at the node of its scope, tagged [`BELIEF_TAG`], +//! with the belief's confidence and `valid_from` as its time. The same +//! sentence from two scopes is kept once. + +use std::collections::HashSet; + +use futures::{StreamExt, TryStreamExt, stream}; +use reqwest::Method; +use serde_json::{Value, json}; +use tinymemory_api::chrono::{DateTime, Utc}; +use tinymemory_api::{ + BELIEF_TAG, BeliefsRequest, Hit, ItemKind, LearningKind, MemoryMeta, MetaFilter, StoreItem, +}; + +use super::CortexEngine; +use super::fetch::recall_body; +use super::items::hit; +use crate::cortex::descriptor::{CortexWire, Route}; +use crate::cortex::envelope::parse_scope; +use crate::cortex::error::{Error, Result}; +use crate::cortex::transport::{Attempts, urlencode}; + +/// Scopes read at once. +const SCOPES_AT_ONCE: usize = 4; + +/// The stances worth showing a reader. +const SHOWN_STANCES: [&str; 2] = ["supported", "contested"]; + +/// The text of a claim part: an entity's name, a literal's value, or the +/// part itself when it is a bare string. +fn part(value: Option<&Value>) -> Option { + let value = value?; + let text = value + .get("name") + .or_else(|| value.get("value")) + .unwrap_or(value); + let text = match text { + Value::String(text) => text.clone(), + Value::Null | Value::Object(_) | Value::Array(_) => return None, + other => other.to_string(), + }; + let text = text.trim(); + (!text.is_empty()).then(|| text.to_string()) +} + +/// One belief as a learning hit at `rank`; `None` for a belief that is not +/// shown (a stance other than [`SHOWN_STANCES`], no claim, or a scope this +/// crate did not write). +pub(super) fn belief_hit(belief: &Value, rank: usize) -> Option { + let id = belief.get("id").and_then(Value::as_str)?; + let stance = belief + .get("stance") + .and_then(Value::as_str) + .unwrap_or("supported"); + if !SHOWN_STANCES.contains(&stance) { + return None; + } + let (namespace, _) = parse_scope(belief.get("scope").and_then(Value::as_str)?)?; + let claim = belief.get("claim")?; + let predicate = part(claim.get("predicate"))?.replace('_', " "); + let object = part(claim.get("object"))?; + let mut text = match part(claim.get("subject")) { + Some(subject) => format!("{subject} {predicate} {object}"), + None => format!("{predicate} {object}"), + }; + if stance == "contested" { + text.push_str(" (contested)"); + } + let confidence = belief + .get("confidence") + .and_then(Value::as_f64) + .map_or(0.5, |c| c.clamp(0.0, 1.0)) as f32; + let observed_at = ["valid_from", "updated_at", "created_at"] + .iter() + .find_map(|field| belief.get(*field)?.as_str()?.parse::>().ok()); + let meta = MemoryMeta { + namespace, + tags: vec![BELIEF_TAG.to_string()], + observed_at, + ..MemoryMeta::default() + }; + let item = StoreItem::learning(text, LearningKind::Fact, confidence, meta); + Some(hit(id, &item, 1.0 / (1.0 + rank as f32))) +} + +/// The beliefs of one recall pack or listing. +pub(super) fn beliefs_in(answer: &Value, pointer: &str) -> Vec { + answer + .pointer(pointer) + .and_then(Value::as_array) + .into_iter() + .flatten() + .enumerate() + .filter_map(|(rank, belief)| belief_hit(belief, rank)) + .collect() +} + +/// `lists` merged rank by rank, each sentence once, at most `limit`. +pub(super) fn merge(lists: Vec>, limit: usize) -> Vec { + let longest = lists.iter().map(Vec::len).max().unwrap_or(0); + let mut seen = HashSet::new(); + let mut out = Vec::new(); + for rank in 0..longest { + for list in &lists { + if let Some(belief) = list.get(rank) + && seen.insert(belief.text.to_lowercase()) + { + out.push(belief.clone()); + } + } + } + out.truncate(limit); + out +} + +impl CortexEngine { + /// See the module docs. + pub(super) async fn read_beliefs(&self, req: BeliefsRequest) -> Result> { + req.validate()?; + let query = req.query.as_deref(); + if query.is_none() && self.wire() == CortexWire::TinyHumans { + return Ok(Vec::new()); + } + let scopes = self.held(&req.reach, &ItemKind::ALL).await?; + let lists: Vec> = stream::iter(scopes) + .map(|scope| async move { + match query { + Some(query) => self.ranked_beliefs(&scope.path, query, req.limit).await, + None => self.listed_beliefs(&scope.path, req.limit).await, + } + }) + .buffered(SCOPES_AT_ONCE) + .try_collect() + .await?; + if query.is_some() { + return Ok(merge(lists, req.limit)); + } + let mut all = merge(lists, usize::MAX); + all.sort_by(|a, b| { + b.confidence + .unwrap_or_default() + .total_cmp(&a.confidence.unwrap_or_default()) + .then_with(|| b.meta.observed_at.cmp(&a.meta.observed_at)) + }); + all.truncate(req.limit); + Ok(all) + } + + /// The beliefs of `scope` a recall for `query` ranks, at most `limit`. + async fn ranked_beliefs(&self, scope: &str, query: &str, limit: usize) -> Result> { + let mut body = recall_body(scope, query, 0, &MetaFilter::default()); + body["budgets"]["per_layer_limits"] = json!({ + "events": 0, "facts": 0, "episodes": 0, "understanding": 0, "beliefs": limit, + }); + let pack = self.log.recall(&body).await?; + Ok(beliefs_in(&pack, "/layers/beliefs")) + } + + /// The beliefs CortexDB lists for `scope`, at most `limit`. + async fn listed_beliefs(&self, scope: &str, limit: usize) -> Result> { + let path = format!( + "{base}?scope={scope}&limit={limit}", + base = self.wire().path(Route::Beliefs), + scope = urlencode(scope), + ); + let listed = match self + .log + .client + .json(Method::GET, &path, None, Attempts::RetryTransient) + .await + { + Ok(listed) => listed, + Err(Error::NotFound(_)) => return Ok(Vec::new()), + Err(error) => return Err(error), + }; + Ok(beliefs_in(&listed, "/items")) + } +} + +#[cfg(test)] +#[path = "beliefs_tests.rs"] +mod tests; diff --git a/crates/tinymemory-integrations/src/cortex/engine/beliefs_tests.rs b/crates/tinymemory-integrations/src/cortex/engine/beliefs_tests.rs new file mode 100644 index 00000000..193b5d4b --- /dev/null +++ b/crates/tinymemory-integrations/src/cortex/engine/beliefs_tests.rs @@ -0,0 +1,193 @@ +//! Beliefs: decoding CortexDB's belief layer, and reading it from a scope's +//! recall pack or listing. + +use super::*; +use crate::cortex::testing::{both, direct_double, direct_engine}; +use tinymemory_api::{MemoryEngine, Namespace, Reach}; + +fn belief(scope: &str, subject: &str, predicate: &str, object: &str, stance: &str) -> Value { + json!({ + "id": format!("belief_{object}"), + "scope": scope, + "claim": { + "subject": { "type": "entity", "id": "ent_x", "name": subject }, + "predicate": predicate, + "object": { "type": "literal", "datatype": "string", "value": object }, + }, + "stance": stance, + "confidence": 0.93, + "valid_from": "2026-09-01T09:00:00Z", + }) +} + +const SCOPE: &str = "app:tinymemory/agent:coder-42/app:conversations"; + +#[test] +fn a_belief_reads_as_a_tagged_learning_sentence_at_its_node() { + let hit = belief_hit( + &belief(SCOPE, "user", "prefers_using", "pnpm over npm", "supported"), + 0, + ) + .unwrap(); + assert_eq!(hit.kind, ItemKind::Learning); + assert_eq!(hit.text, "user prefers using pnpm over npm"); + assert_eq!(hit.meta.namespace, Namespace::agent("coder-42")); + assert_eq!(hit.meta.tags, [BELIEF_TAG]); + assert_eq!(hit.confidence, Some(0.93)); + assert!(hit.meta.observed_at.is_some()); + assert_eq!(hit.id.as_str(), "belief_pnpm over npm"); +} + +#[test] +fn a_contested_belief_says_so_and_a_retired_one_is_left_out() { + let contested = belief_hit(&belief(SCOPE, "Acme", "has_plan", "Team", "contested"), 0); + assert_eq!(contested.unwrap().text, "Acme has plan Team (contested)"); + assert!( + belief_hit( + &belief(SCOPE, "backup", "succeeded", "true", "deprecated"), + 0 + ) + .is_none() + ); + assert!(belief_hit(&belief("org:elsewhere/x", "a", "b", "c", "supported"), 0).is_none()); + assert!(belief_hit(&json!({ "id": "b", "scope": SCOPE }), 0).is_none()); +} + +#[test] +fn merging_keeps_each_sentence_once_rank_by_rank() { + let a = belief_hit(&belief(SCOPE, "user", "likes", "tea", "supported"), 0).unwrap(); + let b = belief_hit(&belief(SCOPE, "user", "likes", "coffee", "supported"), 1).unwrap(); + let merged = merge(vec![vec![a.clone(), b.clone()], vec![a.clone()]], 10); + assert_eq!(merged, [a.clone(), b]); + assert_eq!(merge(vec![vec![a.clone()]], 0), Vec::::new()); +} + +#[tokio::test] +async fn a_built_scope_s_beliefs_are_read_with_and_without_a_query() { + let (endpoint, _state) = direct_double().await; + let engine = direct_engine(&endpoint); + let node = Namespace::agent("coder-42"); + engine + .store(StoreItem::document( + "In this repo always use pnpm.", + MemoryMeta { + namespace: node.clone(), + ..MemoryMeta::default() + }, + )) + .await + .unwrap(); + let none = engine + .beliefs(BeliefsRequest::new(Reach::subtree(Namespace::ROOT), 5)) + .await + .unwrap(); + assert!(none.is_empty(), "nothing is built yet: {none:?}"); + + engine + .consolidate(tinymemory_api::ConsolidateRequest::new(Reach::exact( + node.clone(), + ))) + .await + .unwrap(); + let ranked = engine + .beliefs(BeliefsRequest::new(Reach::subtree(Namespace::ROOT), 5).query("pnpm")) + .await + .unwrap(); + assert_eq!(ranked.len(), 1, "{ranked:?}"); + assert_eq!(ranked[0].text, "user said In this repo always use pnpm."); + assert_eq!(ranked[0].meta.namespace, node); + let listed = engine + .beliefs(BeliefsRequest::new(Reach::exact(node), 5)) + .await + .unwrap(); + let texts = |hits: &[Hit]| hits.iter().map(|h| h.text.clone()).collect::>(); + assert_eq!( + texts(&listed), + texts(&ranked), + "the listing holds the same belief" + ); + let elsewhere = engine + .beliefs(BeliefsRequest::new(Reach::exact(Namespace::agent("other")), 5).query("pnpm")) + .await + .unwrap(); + assert!( + elsewhere.is_empty(), + "a sibling's beliefs stay out of reach" + ); +} + +#[tokio::test] +async fn the_hosted_wire_reads_beliefs_only_for_a_query() { + for (engine, state) in both().await { + let before = state.requests().len(); + let listed = engine + .beliefs(BeliefsRequest::new(Reach::subtree(Namespace::ROOT), 5)) + .await + .unwrap(); + assert!(listed.is_empty()); + if engine.wire() == CortexWire::TinyHumans { + assert_eq!(state.requests().len(), before, "no listing is sent"); + } + } +} + +#[tokio::test] +async fn a_malformed_request_is_refused() { + let (endpoint, state) = direct_double().await; + let error = direct_engine(&endpoint) + .beliefs(BeliefsRequest::new(Reach::default(), 0)) + .await + .unwrap_err(); + assert!(matches!(error, Error::InvalidRequest(_)), "{error:?}"); + assert!(state.requests().is_empty()); +} + +#[tokio::test] +async fn a_fetch_reads_beliefs_from_its_own_recall_packs() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let node = Namespace::agent("coder-42"); + engine + .store(StoreItem::document( + "In this repo always use pnpm.", + MemoryMeta { + namespace: node.clone(), + ..MemoryMeta::default() + }, + )) + .await + .unwrap(); + engine + .consolidate(tinymemory_api::ConsolidateRequest::new(Reach::exact( + node.clone(), + ))) + .await + .unwrap(); + let before = state.seen.lock().unwrap().recalls.len(); + let mut request = + tinymemory_api::FetchRequest::new("pnpm", tinymemory_api::FetchMode::Hybrid, 5); + request.filter.reach = Some(Reach::exact(node.clone())); + request.filter.kinds = vec![ItemKind::Document]; + request.beliefs = 3; + let page = engine.fetch(request).await.unwrap(); + assert_eq!(page.hits.len(), 1, "the document itself"); + assert_eq!(page.beliefs.len(), 1, "{:?}", page.beliefs); + assert_eq!( + page.beliefs[0].text, + "user said In this repo always use pnpm." + ); + let recalls = state.seen.lock().unwrap().recalls[before..].to_vec(); + assert_eq!( + recalls.len(), + 1, + "one pack for the one scope, beliefs included" + ); + assert_eq!(recalls[0]["budgets"]["per_layer_limits"]["beliefs"], 3); + + let mut plain = tinymemory_api::FetchRequest::new("pnpm", tinymemory_api::FetchMode::Hybrid, 5); + plain.filter.reach = Some(Reach::exact(node)); + let page = engine.fetch(plain).await.unwrap(); + assert!(page.beliefs.is_empty(), "no beliefs unless asked"); + let last = state.seen.lock().unwrap().recalls.last().cloned().unwrap(); + assert!(last["budgets"]["per_layer_limits"].get("beliefs").is_none()); +} diff --git a/crates/tinymemory-integrations/src/cortex/engine/consolidate.rs b/crates/tinymemory-integrations/src/cortex/engine/consolidate.rs new file mode 100644 index 00000000..ab1440e7 --- /dev/null +++ b/crates/tinymemory-integrations/src/cortex/engine/consolidate.rs @@ -0,0 +1,104 @@ +//! Consolidate: CortexDB's on-demand belief build. +//! +//! **Direct.** CortexDB builds a scope's beliefs at `v1/beliefs/build`, one +//! scope per request (`{"scope": ""}`). A [`ConsolidateRequest`] covers +//! a reach and some kinds, so the engine first finds the kind scopes in reach +//! that CortexDB actually holds (`v1/scopes/list`, see `scopes::held`) and +//! asks for each, in order. +//! +//! CortexDB v0.10 builds within the request, from the facts it has already +//! extracted, and answers with the count (`{"built": 2, "items": [...]}`): +//! the receipt is then [`ConsolidateStatus::Completed`] with the summed +//! count. An answer that names a job instead (`job_id`, `build_id` or `id`) +//! means the build was queued, and the receipt is +//! [`ConsolidateStatus::Started`] with the handles. What gets built lands in +//! recall's derived layers (`facts`, `beliefs`); the answer route reads them, +//! fetch does not (it ranks stored items only). +//! +//! Every build is sent once: a build is not idempotent work to repeat on a +//! timeout, and the host can always ask again. A failure part way leaves the +//! earlier scopes' builds queued. +//! +//! **TinyHumans.** The backend has no build route and CortexDB consolidates +//! behind it on its own schedule, so the receipt is +//! [`ConsolidateStatus::Scheduled`] and nothing is sent. + +use reqwest::Method; +use serde_json::{Value, json}; +use tinymemory_api::{ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus}; + +use super::CortexEngine; +use crate::cortex::descriptor::{CortexWire, Route}; +use crate::cortex::error::Result; +use crate::cortex::transport::Attempts; + +/// The fields a build answer may name its job by, in preference order. +const JOB_FIELDS: [&str; 3] = ["job_id", "build_id", "id"]; + +/// The job handle a build answer carries, if any. +pub(super) fn job_id(answer: &Value) -> Option { + JOB_FIELDS + .iter() + .find_map(|field| match answer.get(field)? { + Value::String(id) if !id.is_empty() => Some(id.clone()), + Value::Number(id) => Some(id.to_string()), + _ => None, + }) +} + +/// The receipt for the answers of builds over `scopes` scopes: completed +/// when every answer reports what it built, started when any was queued. +pub(super) fn receipt(answers: &[Value], scopes: usize) -> ConsolidateReceipt { + let jobs: Vec = answers.iter().filter_map(job_id).collect(); + let counts: Vec = answers + .iter() + .filter_map(|answer| answer.get("built")?.as_u64()) + .filter_map(|built| usize::try_from(built).ok()) + .collect(); + let completed = jobs.is_empty() && counts.len() == answers.len(); + ConsolidateReceipt { + status: if completed { + ConsolidateStatus::Completed + } else { + ConsolidateStatus::Started + }, + jobs, + scopes, + built: completed.then(|| counts.iter().sum()), + } +} + +impl CortexEngine { + /// See the module docs. + pub(super) async fn build_beliefs( + &self, + req: ConsolidateRequest, + ) -> Result { + req.validate()?; + let wire = self.wire(); + if wire == CortexWire::TinyHumans { + return Ok(ConsolidateReceipt::scheduled()); + } + let scopes = self.held(&req.reach, &req.admitted_kinds()).await?; + let mut answers = Vec::new(); + for scope in &scopes { + let body = json!({ "scope": scope.path }); + let answer = self + .log + .client + .json( + Method::POST, + wire.path(Route::BuildBeliefs), + Some(&body), + Attempts::Once, + ) + .await?; + answers.push(answer); + } + Ok(receipt(&answers, scopes.len())) + } +} + +#[cfg(test)] +#[path = "consolidate_tests.rs"] +mod tests; diff --git a/crates/tinymemory-integrations/src/cortex/engine/consolidate_tests.rs b/crates/tinymemory-integrations/src/cortex/engine/consolidate_tests.rs new file mode 100644 index 00000000..311cd35e --- /dev/null +++ b/crates/tinymemory-integrations/src/cortex/engine/consolidate_tests.rs @@ -0,0 +1,161 @@ +//! Belief builds: one `v1/beliefs/build` per held scope on Direct, none on +//! the hosted wire. + +use super::*; +use crate::cortex::testing::{both, direct_double, direct_engine, sample_items}; +use tinymemory_api::{ + ConsolidateRequest, ItemKind, MemoryEngine, MemoryMeta, Namespace, Reach, StoreItem, +}; + +fn at(namespace: Namespace) -> MemoryMeta { + MemoryMeta { + namespace, + ..MemoryMeta::default() + } +} + +#[test] +fn reads_the_job_handle_by_any_known_field() { + assert_eq!(job_id(&json!({ "job_id": "j1" })).as_deref(), Some("j1")); + assert_eq!(job_id(&json!({ "build_id": 7 })).as_deref(), Some("7")); + assert_eq!( + job_id(&json!({ "id": "x", "build_id": "b" })).as_deref(), + Some("b") + ); + assert_eq!(job_id(&json!({ "job_id": "" })), None); + assert_eq!(job_id(&json!({ "status": "queued" })), None); +} + +#[test] +fn a_build_that_reports_its_count_completed() { + let answers = [json!({ "built": 2, "items": [] }), json!({ "built": 0 })]; + assert_eq!( + receipt(&answers, 2), + ConsolidateReceipt { + status: ConsolidateStatus::Completed, + jobs: Vec::new(), + scopes: 2, + built: Some(2), + } + ); +} + +#[test] +fn a_queued_build_started_with_its_handles() { + let answers = [ + json!({ "built": 1 }), + json!({ "status": "queued", "job_id": "j9" }), + ]; + let queued = receipt(&answers, 2); + assert_eq!(queued.status, ConsolidateStatus::Started); + assert_eq!(queued.jobs, ["j9"]); + assert_eq!(queued.built, None, "a count is only reported once all ran"); + + let silent = receipt(&[json!({ "status": "ok" })], 1); + assert_eq!(silent.status, ConsolidateStatus::Started); +} + +#[test] +fn no_scope_to_build_is_already_complete() { + let empty = receipt(&[], 0); + assert_eq!(empty.status, ConsolidateStatus::Completed); + assert_eq!(empty.built, Some(0)); +} + +#[tokio::test] +async fn builds_every_held_scope_in_reach_and_nothing_else() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + for item in [ + StoreItem::document("Refunds take five days.", at(Namespace::source("pdf"))), + StoreItem::document("Deploys run on Fridays.", at(Namespace::source("notion"))), + StoreItem::document("Agent scratch note.", at(Namespace::agent("support"))), + ] { + engine.store(item).await.unwrap(); + } + + let receipt = engine + .consolidate(ConsolidateRequest::new(Reach::exact(Namespace::source( + "pdf", + )))) + .await + .unwrap(); + assert_eq!(receipt.status, ConsolidateStatus::Completed); + assert_eq!(receipt.scopes, 1, "only the pdf documents scope is held"); + assert_eq!(receipt.built, Some(1)); + assert!(receipt.jobs.is_empty()); + let builds = state.seen.lock().unwrap().builds.clone(); + assert_eq!( + builds, + [json!({ "scope": "app:tinymemory/source:pdf/app:documents" })] + ); + + let whole = engine + .consolidate( + ConsolidateRequest::new(Reach::subtree(Namespace::ROOT)).kinds([ItemKind::Document]), + ) + .await + .unwrap(); + assert_eq!(whole.scopes, 3, "every document scope below the root"); + assert_eq!( + whole.built, + Some(2), + "the counts of every scope, summed; the pdf scope was built already" + ); + assert_eq!(state.seen.lock().unwrap().builds.len(), 4); +} + +#[tokio::test] +async fn an_empty_reach_builds_nothing() { + let (endpoint, state) = direct_double().await; + let receipt = direct_engine(&endpoint) + .consolidate(ConsolidateRequest::new(Reach::exact(Namespace::agent( + "nobody", + )))) + .await + .unwrap(); + assert_eq!(receipt.scopes, 0); + assert!(receipt.jobs.is_empty()); + assert!(state.seen.lock().unwrap().builds.is_empty()); +} + +#[tokio::test] +async fn the_hosted_wire_acknowledges_a_schedule_without_a_request() { + for (engine, state) in both().await { + for item in sample_items() { + engine.store(item).await.unwrap(); + } + let before = state.requests().len(); + let receipt = engine + .consolidate(ConsolidateRequest::new(Reach::subtree(Namespace::ROOT))) + .await + .unwrap(); + match engine.wire() { + CortexWire::TinyHumans => { + assert_eq!(receipt, ConsolidateReceipt::scheduled()); + assert_eq!(state.requests().len(), before, "no request is sent"); + } + CortexWire::Direct => { + assert_eq!(receipt.status, ConsolidateStatus::Completed); + assert!(state.count("POST /v1/beliefs/build") > 0); + } + } + } +} + +#[tokio::test] +async fn a_malformed_request_is_refused_before_any_request() { + let (endpoint, state) = direct_double().await; + let error = direct_engine(&endpoint) + .consolidate( + ConsolidateRequest::new(Reach::default()) + .kinds([ItemKind::Learning, ItemKind::Learning]), + ) + .await + .unwrap_err(); + assert!( + matches!(error, crate::cortex::Error::InvalidRequest(_)), + "{error:?}" + ); + assert!(state.requests().is_empty()); +} diff --git a/crates/tinymemory-integrations/src/cortex/engine/fetch.rs b/crates/tinymemory-integrations/src/cortex/engine/fetch.rs index d24c27bc..b9295863 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/fetch.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/fetch.rs @@ -8,10 +8,17 @@ //! when the [`tinymemory_api::MetaFilter`] has a labelled field. The events //! are decoded back to items, the full filter is applied, repeats of an item //! are dropped keeping its best rank, and the scopes are interleaved rank by -//! rank. CortexDB reports no per-hit score, so the score is the rank's, +//! rank. The scopes are read a few at a time, in order. CortexDB reports no per-hit score, so the score is the rank's, //! `1 / (1 + rank)`. A conversation hit carries the whole conversation's //! text, assembled from all its turns. //! +//! **Beliefs.** When the request asks for beliefs +//! ([`FetchRequest::beliefs`]), each scope's pack also budgets the `beliefs` +//! layer, so the same pack (one query embedding per scope) ranks both. The +//! beliefs are decoded as in `beliefs`, kept within the filter's reach, +//! merged rank by rank across scopes, each sentence once, and returned on +//! the first page only. +//! //! **Cursor.** Recall is a ranking, not a log, so it has no cursor of its //! own. The fetch cursor is an offset into the merged ranking; the next page //! asks again with a budget large enough to reach past it. A page ends the @@ -19,14 +26,16 @@ use std::collections::HashSet; +use futures::{StreamExt, TryStreamExt, stream}; use serde_json::{Value, json}; use tinymemory_api::{FetchPage, FetchRequest, Hit, ItemKind, MetaFilter}; use super::CortexEngine; +use super::beliefs::{beliefs_in, merge}; use super::cursor::{self, FetchCursor}; use super::items::{hit, keeps}; use crate::cortex::envelope::{Envelope, decode_event, labels, rebuild}; -use crate::cortex::error::Result; +use crate::cortex::error::{Error, Result}; /// The cursor tag of a fetch. const TAG: char = 'f'; @@ -34,6 +43,11 @@ const TAG: char = 'f'; /// Events one recall pack may hold. Bounds how deep fetch pages can go. const MAX_PACK_EVENTS: usize = 1000; +/// Recall packs read at once when a filter spans several scopes. Each is one +/// query embedding and one ranking on the server; reading them one after +/// the other made a turn's latency grow with the number of scopes. +const PACKS_AT_ONCE: usize = 4; + /// Raw events asked for per wanted hit: a conversation contributes several /// turns, and the client-side filter drops some. const EVENTS_PER_HIT: usize = 3; @@ -80,12 +94,36 @@ impl CortexEngine { .saturating_add(1) .saturating_mul(EVENTS_PER_HIT) .min(MAX_PACK_EVENTS); - let mut per_scope = Vec::new(); - for scope in self.scopes_for(&req.filter).await? { - let body = recall_body(&scope.path, &req.query, events, &req.filter); - let pack = self.log.recall(&body).await?; - per_scope.push(ranked(&pack, Some(scope.kind), &req.filter)); - } + let scopes = self.scopes_for(&req.filter).await?; + let wanted_beliefs = if offset == 0 { req.beliefs } else { 0 }; + let req = &req; + let packs: Vec<(Vec, Vec)> = stream::iter(scopes) + .map(|scope| async move { + let mut body = recall_body(&scope.path, &req.query, events, &req.filter); + if wanted_beliefs > 0 { + body["budgets"]["per_layer_limits"]["beliefs"] = json!(wanted_beliefs); + } + let pack = self.log.recall(&body).await?; + let beliefs = if wanted_beliefs > 0 { + beliefs_in(&pack, "/layers/beliefs") + } else { + Vec::new() + }; + Ok::<_, Error>((ranked(&pack, Some(scope.kind), &req.filter), beliefs)) + }) + .buffered(PACKS_AT_ONCE) + .try_collect() + .await?; + let (per_scope, beliefs): (Vec>, Vec>) = packs.into_iter().unzip(); + let beliefs: Vec = merge(beliefs, wanted_beliefs) + .into_iter() + .filter(|belief| { + req.filter + .reach + .as_ref() + .is_none_or(|reach| reach.admits(&belief.meta.namespace)) + }) + .collect(); let merged = interleave(per_scope); let more = merged.len() > end; let page: Vec<(usize, Envelope)> = merged @@ -119,7 +157,11 @@ impl CortexEngine { } else { None }; - Ok(FetchPage { hits, next_cursor }) + Ok(FetchPage { + hits, + next_cursor, + beliefs, + }) } } diff --git a/crates/tinymemory-integrations/src/cortex/engine/mod.rs b/crates/tinymemory-integrations/src/cortex/engine/mod.rs index dd66634e..f14253ff 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/mod.rs @@ -9,8 +9,11 @@ //! - `list` — a cursor over the kind scopes' listings, each item once; //! - `fetch` — hybrid retrieval through recall packs, ranked by the engine; //! - `recall` — one pack, one answer, citations from the pack; -//! - `forget` — look the items' events up, remove them by `memory_ids`. +//! - `forget` — look the items' events up, remove them by `memory_ids`; +//! - `consolidate` — one `v1/beliefs/build` per held scope in reach. +mod beliefs; +mod consolidate; mod cursor; mod fetch; mod forget; @@ -24,9 +27,9 @@ use std::sync::Arc; use async_trait::async_trait; use tinymemory_api::{ - EngineDescriptor, EngineHealth, FetchPage, FetchRequest, ForgetReport, ForgetTarget, - GetRequest, Hit, ListPage, ListRequest, MemoryEngine, RecallAnswer, RecallRequest, StoreItem, - StoreReceipt, + BeliefsRequest, ConsolidateReceipt, ConsolidateRequest, EngineDescriptor, EngineHealth, + FetchPage, FetchRequest, ForgetReport, ForgetTarget, GetRequest, Hit, ListPage, ListRequest, + MemoryEngine, RecallAnswer, RecallRequest, StoreItem, StoreReceipt, WaitFor, WriteOptions, }; use crate::cortex::credential::{BearerSource, CortexCredential}; @@ -147,7 +150,13 @@ impl MemoryEngine for CortexEngine { /// A batch of one (see `store`): listed and ranked on return. async fn store(&self, item: StoreItem) -> Result { - self.store_items(vec![item]) + self.store_with(item, WriteOptions::visible()).await + } + + /// [`WaitFor::Accepted`] returns once CortexDB captured the events, + /// without `?wait=indexed` and without the visibility waits. + async fn store_with(&self, item: StoreItem, options: WriteOptions) -> Result { + self.store_items(vec![item], options.wait) .await? .pop() .ok_or_else(|| Error::Engine("a store of one item returned no receipt".to_string())) @@ -155,7 +164,7 @@ impl MemoryEngine for CortexEngine { /// Ranked recall is awaited for the last item only (see `store`). async fn store_many(&self, items: Vec) -> Result> { - self.store_items(items).await + self.store_items(items, WaitFor::Visible).await } async fn forget(&self, target: ForgetTarget) -> Result { @@ -170,6 +179,16 @@ impl MemoryEngine for CortexEngine { async fn get(&self, req: GetRequest) -> Result> { self.get_items(req).await } + + /// Direct: one `v1/beliefs/build` per held scope in reach. Hosted: + /// acknowledged as scheduled, with no request. + async fn consolidate(&self, req: ConsolidateRequest) -> Result { + self.build_beliefs(req).await + } + + async fn beliefs(&self, req: BeliefsRequest) -> Result> { + self.read_beliefs(req).await + } } #[cfg(test)] diff --git a/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs b/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs index 34de9de2..f82004ae 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/mod_direct_tests.rs @@ -207,3 +207,39 @@ async fn a_rejected_key_is_unauthorized() { assert!(matches!(error, Error::Unauthorized(_)), "{error:?}"); assert!(error.to_string().contains("API key"), "{error}"); } + +#[tokio::test] +async fn an_accepted_write_neither_asks_for_indexing_nor_waits_to_be_listed() { + let (endpoint, state) = direct_double().await; + let engine = direct_engine(&endpoint); + let before = state.count("GET /v1/events"); + // Listings never show the write: a visible store would time out here. + state.hide_listing_for.store(usize::MAX, Ordering::SeqCst); + let mut items = sample_items(); + let conversation = items.remove(1); + let document = items.remove(0); + for item in [document, conversation] { + let receipt = engine + .clone() + .with_test_timing(std::time::Duration::from_millis(50)) + .store_with(item.clone(), tinymemory_api::WriteOptions::accepted()) + .await + .unwrap(); + assert_eq!(receipt.id.as_str(), item.fingerprint()); + } + let requests = state.requests(); + assert!( + requests.iter().any(|r| r == "POST /v1/experience"), + "{requests:?}" + ); + assert!( + requests.iter().any(|r| r == "POST /v1/experience/bulk"), + "{requests:?}" + ); + assert!(requests.iter().all(|r| !r.contains("wait=indexed"))); + assert_eq!( + state.count("GET /v1/events") - before, + 2, + "only the replay lookup, one per item: no visibility polling" + ); +} diff --git a/crates/tinymemory-integrations/src/cortex/engine/scopes.rs b/crates/tinymemory-integrations/src/cortex/engine/scopes.rs index b9cba4ee..ebcc961a 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/scopes.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/scopes.rs @@ -89,6 +89,23 @@ impl CortexEngine { Ok(known(base, &kinds)) } + /// The scopes of `kinds` in `reach` that CortexDB has registered — only + /// those with something written — kind first then namespace. Unlike a + /// read, nothing is assumed to exist: a build of an empty scope would be + /// wasted model time. + pub(super) async fn held(&self, reach: &Reach, kinds: &[ItemKind]) -> Result> { + let mut found = BTreeSet::new(); + for path in self.log.scopes(ROOT_SCOPE).await? { + let Some((namespace, kind)) = parse_scope(&path) else { + continue; + }; + if reach.admits(&namespace) && kinds.contains(&kind) { + found.insert(KindScope::new(namespace, kind)); + } + } + Ok(found.into_iter().collect()) + } + /// Every scope of `kinds` the engine holds, in reach. async fn discovered( &self, diff --git a/crates/tinymemory-integrations/src/cortex/engine/store.rs b/crates/tinymemory-integrations/src/cortex/engine/store.rs index 3aa1f874..f8e75d1e 100644 --- a/crates/tinymemory-integrations/src/cortex/engine/store.rs +++ b/crates/tinymemory-integrations/src/cortex/engine/store.rs @@ -21,7 +21,7 @@ use std::collections::{BTreeMap, HashMap, HashSet}; -use tinymemory_api::{ItemId, StoreItem, StoreReceipt, validate_many}; +use tinymemory_api::{ItemId, StoreItem, StoreReceipt, WaitFor, validate_many}; use super::CortexEngine; use super::scopes::KindScope; @@ -40,7 +40,14 @@ impl CortexEngine { /// ones are), and ranked recall for the batch's final event only. /// /// An item repeated inside the batch is a replay of its first copy. - pub(super) async fn store_items(&self, items: Vec) -> Result> { + /// + /// With [`WaitFor::Accepted`] the writes ask for no indexing and the + /// waits are skipped: the call returns once CortexDB captured every event. + pub(super) async fn store_items( + &self, + items: Vec, + wait: WaitFor, + ) -> Result> { validate_many(&items)?; let ids: Vec = items.iter().map(StoreItem::fingerprint).collect(); let mut held: HashMap>> = HashMap::new(); @@ -76,7 +83,7 @@ impl CortexEngine { } } let replayed = requests.is_empty(); - if let Some(written) = self.log.write(&requests).await? { + if let Some(written) = self.log.write(&requests, wait).await? { last_per_scope.retain(|w| w.scope != written.scope); last_per_scope.push(written); } @@ -86,6 +93,9 @@ impl CortexEngine { replayed, }); } + if wait == WaitFor::Accepted { + return Ok(receipts); + } let final_index = last_per_scope.len().saturating_sub(1); for (index, written) in last_per_scope.iter().enumerate() { self.log diff --git a/crates/tinymemory-integrations/src/cortex/envelope/mod.rs b/crates/tinymemory-integrations/src/cortex/envelope/mod.rs index 01aad4cc..4459bbab 100644 --- a/crates/tinymemory-integrations/src/cortex/envelope/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/envelope/mod.rs @@ -14,8 +14,9 @@ //! So within every node, learnings, documents and conversations are separate //! scopes, and CortexDB can recall, retain and erase each on its own. The //! namespace segments map onto CortexDB's built-in scope types (`agent`, -//! `team`, `user`, `ws`, `project`), which every shipped deployment preset -//! allows. The hosted backend additionally re-roots every scope under the +//! `team`, `user`, `ws`, `project`, `source`), which every shipped +//! deployment preset allows. A brain's per-source documents therefore live in +//! `app:tinymemory/source:pdf/app:documents`. The hosted backend additionally re-roots every scope under the //! caller's tenant, which is invisible here. A //! [`tinymemory_api::MetaFilter`]'s `kinds` and `reach` pick which scopes are //! read (see `engine::scopes`). diff --git a/crates/tinymemory-integrations/src/cortex/envelope/mod_tests.rs b/crates/tinymemory-integrations/src/cortex/envelope/mod_tests.rs index fa08a953..128b7606 100644 --- a/crates/tinymemory-integrations/src/cortex/envelope/mod_tests.rs +++ b/crates/tinymemory-integrations/src/cortex/envelope/mod_tests.rs @@ -166,6 +166,14 @@ fn scopes_nest_kinds_under_their_namespace_node() { } } +#[test] +fn a_brain_source_is_a_cortex_source_scope() { + let pdf: Namespace = "team:acme/source:pdf".parse().unwrap(); + let path = scope_path(&pdf, ItemKind::Document); + assert_eq!(path, "app:tinymemory/team:acme/source:pdf/app:documents"); + assert_eq!(parse_scope(&path), Some((pdf, ItemKind::Document))); +} + #[test] fn an_item_is_written_to_its_namespace_scope() { let meta = MemoryMeta { diff --git a/crates/tinymemory-integrations/src/cortex/lifecycle_tests.rs b/crates/tinymemory-integrations/src/cortex/lifecycle_tests.rs new file mode 100644 index 00000000..68974466 --- /dev/null +++ b/crates/tinymemory-integrations/src/cortex/lifecycle_tests.rs @@ -0,0 +1,143 @@ +//! The agent memory lifecycle (`tinymemory_tools`) end to end over both +//! wires' doubles: the same host code against either CortexDB surface. + +use std::sync::Arc; + +use tinymemory_api::MemoryEngine; +use tinymemory_tools::{ + AgentMemory, Brain, BrainDocument, BrainSource, JobOutcome, MemoryLayout, PostTurn, PreTurn, + RecallPolicy, SessionStart, +}; + +use crate::cortex::CortexWire; +use crate::cortex::testing::both; + +#[tokio::test] +async fn an_agent_loop_runs_the_same_on_either_wire() { + for (engine, state) in both().await { + let wire = engine.wire(); + let engine: Arc = Arc::new(engine); + let layout = MemoryLayout::default(); + + let brain = Brain::new(engine.clone(), layout.clone()); + let ingested = brain + .ingest(BrainDocument::new( + BrainSource::Pdf, + "Refunds settle within five business days.", + )) + .await + .unwrap(); + let events = state.log.lock().unwrap().events.clone(); + assert!( + events.iter().any(|e| e["scope"] + .as_str() + .unwrap() + .ends_with("source:pdf/app:documents")), + "{wire:?}: the pdf lands in its source scope" + ); + + let support = AgentMemory::new(engine.clone(), layout.clone(), "support-01") + .unwrap() + .with_policy(RecallPolicy { + build_beliefs_every: Some(2), + ..RecallPolicy::default() + }); + let turn = support + .pre_turn(PreTurn::new("t1", 0, "when do refunds settle")) + .await + .unwrap(); + assert!(turn.log_error.is_none(), "{wire:?}: {:?}", turn.log_error); + assert!( + turn.pack.markdown.contains("## Brain"), + "{wire:?}: {}", + turn.pack.markdown + ); + assert!(turn.pack.markdown.contains("five business days")); + let report = support + .post_turn(PostTurn::new( + "t1", + 1, + "Refunds settle in five business days.", + )) + .await + .unwrap(); + assert_eq!(report.jobs.len(), 1, "{wire:?}: the policy's build is due"); + + if wire == CortexWire::Direct { + let requests = state.requests(); + let unwaited = requests + .iter() + .filter(|r| *r == "POST /v1/experience") + .count(); + assert_eq!( + unwaited, 2, + "both turns logged without waiting: {requests:?}" + ); + assert_eq!( + requests + .iter() + .filter(|r| *r == "POST /v1/experience?wait=indexed") + .count(), + 1, + "only the brain ingest waits to be indexed" + ); + } + + let resumed = support + .start_session(SessionStart { + thread_id: Some("t1".into()), + focus: Some("refunds".into()), + }) + .await + .unwrap(); + assert!( + resumed.markdown.contains("## Earlier in this thread"), + "{wire:?}: {}", + resumed.markdown + ); + + for job in report.jobs.into_iter().chain([ingested.job]) { + let ran = support.run_background(job).await.unwrap(); + match wire { + CortexWire::Direct => assert_eq!(ran.outcome, JobOutcome::Done), + CortexWire::TinyHumans => assert_eq!(ran.outcome, JobOutcome::Scheduled), + } + } + let builds = state.seen.lock().unwrap().builds.clone(); + match wire { + CortexWire::Direct => assert_eq!( + builds, + [ + serde_json::json!({ "scope": "app:tinymemory/agent:support-01/app:conversations" }), + serde_json::json!({ "scope": "app:tinymemory/source:pdf/app:documents" }), + ] + ), + CortexWire::TinyHumans => assert!(builds.is_empty()), + } + + // The built beliefs come back as learnings, in a turn's pack and in + // a cold session start. The hosted wire built none. + let later = support + .pre_turn(PreTurn::new("t2", 0, "refunds settle")) + .await + .unwrap(); + let cold = support + .start_session(SessionStart::default()) + .await + .unwrap(); + for pack in [&later.pack.markdown, &cold.markdown] { + let learnings = pack + .split("## Learnings") + .nth(1) + .map(|rest| rest.split("\n## ").next().unwrap_or_default()); + match wire { + CortexWire::Direct => assert!( + learnings.is_some_and(|section| section + .contains("- user said Refunds settle within five business days.")), + "{wire:?}: {pack}" + ), + CortexWire::TinyHumans => assert!(learnings.is_none(), "{wire:?}: {pack}"), + } + } + } +} diff --git a/crates/tinymemory-integrations/src/cortex/log/write.rs b/crates/tinymemory-integrations/src/cortex/log/write.rs index 4d5f0dbc..91f6b54d 100644 --- a/crates/tinymemory-integrations/src/cortex/log/write.rs +++ b/crates/tinymemory-integrations/src/cortex/log/write.rs @@ -2,7 +2,9 @@ //! //! **Direct** sends one experience (`v1/experience?wait=indexed`) or, for a //! conversation, one ordered batch (`v1/experience/bulk?wait=indexed` with -//! `ordering: strict_temporal`), once: a timeout on a write leaves whether it +//! `ordering: strict_temporal`), once — without `?wait=indexed` when the +//! caller only waits for acceptance ([`WaitFor::Accepted`]), so the server +//! answers on capture: a timeout on a write leaves whether it //! applied unknown, and the store's replay check makes a retry by the caller //! safe. //! @@ -27,6 +29,7 @@ //! Either way the write then waits for its last event to be readable. use serde_json::{Value, json}; +use tinymemory_api::WaitFor; use super::{Log, PAGE_SIZE}; use crate::cortex::descriptor::{CortexWire, Route}; @@ -73,12 +76,15 @@ impl Log { /// [`Log::await_written`], or for a later one in the same scope, which /// implies it: the log is ordered, so the last event being listed implies /// the earlier ones are. - pub(crate) async fn write(&self, requests: &[Value]) -> Result> { + /// + /// `wait` decides only whether a Direct write asks the server to index + /// before answering; waiting for readability is the caller's step. + pub(crate) async fn write(&self, requests: &[Value], wait: WaitFor) -> Result> { let Some(last) = requests.last() else { return Ok(None); }; let event_id = match self.client.wire() { - CortexWire::Direct => self.append_direct(requests).await?, + CortexWire::Direct => self.append_direct(requests, wait).await?, CortexWire::TinyHumans => { let mut event_id = String::new(); for request in requests { @@ -114,17 +120,21 @@ impl Log { /// One Direct write of one event or one ordered batch; the last event's /// id. - async fn append_direct(&self, requests: &[Value]) -> Result { + async fn append_direct(&self, requests: &[Value], wait: WaitFor) -> Result { let wire = self.client.wire(); + let query = match wait { + WaitFor::Visible => "?wait=indexed", + WaitFor::Accepted => "", + }; if let [single] = requests { - let path = format!("{}?wait=indexed", wire.path(Route::Experience)); + let path = format!("{}{query}", wire.path(Route::Experience)); let answer = self .client .json(reqwest::Method::POST, &path, Some(single), Attempts::Once) .await?; return receipt(&answer); } - let path = format!("{}?wait=indexed", wire.path(Route::Bulk)); + let path = format!("{}{query}", wire.path(Route::Bulk)); let body = json!({ "items": requests, "ordering": "strict_temporal" }); let answer = self .client diff --git a/crates/tinymemory-integrations/src/cortex/mod.rs b/crates/tinymemory-integrations/src/cortex/mod.rs index f0a2aa79..0be8dd9f 100644 --- a/crates/tinymemory-integrations/src/cortex/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/mod.rs @@ -68,6 +68,10 @@ mod testing; #[path = "conformance_tests.rs"] mod conformance_tests; +#[cfg(test)] +#[path = "lifecycle_tests.rs"] +mod lifecycle_tests; + pub use credential::{BearerSource, CortexCredential, StaticBearer}; pub use descriptor::{ CORTEX_API_ENDPOINT, CORTEXDB_ENGINE_ID, CortexWire, TINYHUMANS_API_ENDPOINT, diff --git a/crates/tinymemory-integrations/src/cortex/testing/log.rs b/crates/tinymemory-integrations/src/cortex/testing/log.rs index c1805778..17d44b5a 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/log.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/log.rs @@ -12,7 +12,10 @@ //! `confirm_all` is refused, and a selector with `confirm_all` is refused //! as ambiguous; //! - recall renders text for a reader (`[role] {...}`), honours `view: -//! "descend"`, metadata label filters and the events budget. +//! "descend"`, metadata label filters and the events budget; +//! - a belief build turns each of a scope's events into one supported +//! belief (`user said `), which recall returns in `layers.beliefs` +//! when its budget asks for them, and `GET /v1/beliefs` lists. use std::collections::BTreeMap; @@ -28,6 +31,8 @@ pub(crate) struct CortexLog { next_id: u64, /// Every event id a forget removed. pub(crate) forgotten: Vec, + /// Beliefs built, oldest first. + pub(crate) beliefs: Vec, } /// Whether `event` carries any one of `wanted` (an empty list keeps all). @@ -234,6 +239,69 @@ impl CortexLog { hit }) .collect(); - json!({ "pack_id": "pack_test", "layers": { "events": events } }) + let wanted = body + .pointer("/budgets/per_layer_limits/beliefs") + .and_then(Value::as_u64) + .map_or(0, |b| usize::try_from(b).unwrap_or(usize::MAX)); + let beliefs: Vec = self + .beliefs + .iter() + .rev() + .filter(|belief| in_scope(belief)) + .filter(|belief| { + let text = belief.to_string().to_lowercase(); + words.is_empty() || words.iter().any(|w| text.contains(w.as_str())) + }) + .take(wanted) + .cloned() + .collect(); + json!({ "pack_id": "pack_test", "layers": { "events": events, "beliefs": beliefs } }) + } + + /// `POST /v1/beliefs/build`: one belief per event in `scope` that has + /// none yet; how many were built. + pub(crate) fn build(&mut self, scope: &str) -> usize { + let fresh: Vec = self + .events + .iter() + .filter(|event| str_of(event, "/scope") == scope) + .filter(|event| { + let id = str_of(event, "/id"); + !self.beliefs.iter().any(|b| str_of(b, "/source") == id) + }) + .map(|event| { + let raw = str_of(event, "/content/text"); + let said = serde_json::from_str::(raw) + .ok() + .and_then(|envelope| envelope["text"].as_str().map(str::to_owned)) + .unwrap_or_else(|| raw.to_string()); + json!({ + "id": format!("belief_{}", str_of(event, "/id")), + "source": str_of(event, "/id"), + "scope": scope, + "claim": { + "subject": { "type": "entity", "id": "ent_user", "name": "user" }, + "predicate": "said", + "object": { "type": "literal", "datatype": "string", "value": said }, + }, + "stance": "supported", + "confidence": 0.9, + "valid_from": "2026-09-01T09:00:00Z", + }) + }) + .collect(); + let built = fresh.len(); + self.beliefs.extend(fresh); + built + } + + /// `GET /v1/beliefs?scope=`: the scope's beliefs, newest first. + pub(crate) fn list_beliefs(&self, scope: &str) -> Vec { + self.beliefs + .iter() + .rev() + .filter(|belief| str_of(belief, "/scope") == scope) + .cloned() + .collect() } } diff --git a/crates/tinymemory-integrations/src/cortex/testing/mod.rs b/crates/tinymemory-integrations/src/cortex/testing/mod.rs index 4c38586d..d80a39be 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/mod.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/mod.rs @@ -42,6 +42,8 @@ pub(crate) struct Seen { pub(crate) answers: Vec, /// Every forget body. pub(crate) forgets: Vec, + /// Every beliefs build body. + pub(crate) builds: Vec, } /// One double's state and knobs. diff --git a/crates/tinymemory-integrations/src/cortex/testing/routes.rs b/crates/tinymemory-integrations/src/cortex/testing/routes.rs index 188fcdfa..74d0675f 100644 --- a/crates/tinymemory-integrations/src/cortex/testing/routes.rs +++ b/crates/tinymemory-integrations/src/cortex/testing/routes.rs @@ -339,6 +339,48 @@ async fn scopes( } } +async fn build_beliefs( + State(state): State, + uri: Uri, + headers: HeaderMap, + Json(body): Json, +) -> Reply { + if let Some(early) = gate(&state, "POST", &uri, &headers) { + return early; + } + let Some(scope) = body["scope"].as_str().filter(|scope| !scope.is_empty()) else { + return fail(&state, 422, "VALIDATION_ERROR"); + }; + if let Some(refused) = refuse_scope(&state, scope) { + return refused; + } + state.seen.lock().unwrap().builds.push(body.clone()); + // CortexDB v0.10 builds within the request and reports the count. + let built = state.log.lock().unwrap().build(scope); + ok( + &state, + 200, + json!({ "built": built, "items": [], "facts_scanned": built, "events_scanned": built }), + ) +} + +async fn beliefs( + State(state): State, + uri: Uri, + headers: HeaderMap, + Query(query): Query>, +) -> Reply { + if let Some(early) = gate(&state, "GET", &uri, &headers) { + return early; + } + let scope = query.get("scope").cloned().unwrap_or_default(); + if let Some(refused) = refuse_scope(&state, &scope) { + return refused; + } + let items = state.log.lock().unwrap().list_beliefs(&scope); + ok(&state, 200, json!({ "items": items, "has_more": false })) +} + /// CortexDB's own routes. pub(super) fn direct(state: Shared) -> Router { Router::new() @@ -350,6 +392,8 @@ pub(super) fn direct(state: Shared) -> Router { .route("/v1/answer", post(answer)) .route("/v1/admin/health", get(health)) .route("/v1/scopes/list", get(scopes)) + .route("/v1/beliefs/build", post(build_beliefs)) + .route("/v1/beliefs", get(beliefs)) .with_state(state) } diff --git a/crates/tinymemory-integrations/src/lib.rs b/crates/tinymemory-integrations/src/lib.rs index f5a2111a..f320a470 100644 --- a/crates/tinymemory-integrations/src/lib.rs +++ b/crates/tinymemory-integrations/src/lib.rs @@ -7,6 +7,7 @@ //! | --- | --- | --- | //! | [`cortex`], [`registry`], [`config`] | `cortex` (default) | The CortexDB engine over its two wires, and building one from configuration | //! | `documents` | `documents`, `documents-office` | Format sniffing and conversion to markdown, emitting `StoreItem::Document` | +//! | `brain` | `brain` | Converting files into the brain documents `tinymemory_tools::Brain` ingests, by source type | //! | `sources` | `sources`, `sources-network` | Readers turning folders, files, links, GitHub, RSS, Composio payloads and conversations into `StoreItem`s | //! | `safety` | `safety` | Secret and PII scrubbing for a `StoreItem` before it is stored | //! | `import` | `legacy-import` | Migrating a legacy v1 (embedded TinyCortex) workspace into any engine | @@ -43,6 +44,8 @@ pub mod cortex; #[cfg(feature = "cortex")] pub mod registry; +#[cfg(feature = "brain")] +pub mod brain; #[cfg(feature = "documents")] pub mod documents; #[cfg(feature = "legacy-import")] diff --git a/crates/tinymemory-integrations/tests/live_cortex_lifecycle.rs b/crates/tinymemory-integrations/tests/live_cortex_lifecycle.rs new file mode 100644 index 00000000..9899326e --- /dev/null +++ b/crates/tinymemory-integrations/tests/live_cortex_lifecycle.rs @@ -0,0 +1,143 @@ +//! The agent memory lifecycle (`tinymemory_tools`) against a real CortexDB +//! server. +//! +//! Skipped unless `TINYMEMORY_LIVE_CORTEXDB_URL` names one (see +//! `integration/cortexdb/`); the key defaults to the harness's +//! (`TINYMEMORY_TEST_CORTEX_KEY`). Everything is written below a node unique +//! to the run and forgotten at the end. +//! +//! It proves the hot path on the real wire: a brain document converted and +//! ingested into its source scope, turns logged without waiting for +//! indexing, and pre-turn packs that carry the brain, the agent's history +//! and the team's turns once CortexDB has indexed them. Belief builds are +//! requested through `v1/beliefs/build`. + +// The helpers outside `#[test]` fns fail the test by panicking, like the tests. +#![allow(clippy::expect_used)] + +use std::sync::Arc; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use tinymemory_api::{ForgetTarget, MemoryEngine, MemoryMeta}; +use tinymemory_integrations::brain::brain_document; +use tinymemory_integrations::cortex::{CortexCredential, CortexEngine}; +use tinymemory_integrations::documents::{ConverterChain, RawDocument}; +use tinymemory_tools::{AgentMemory, Brain, ContextPack, MemoryLayout, PostTurn, PreTurn}; + +const DEFAULT_KEY: &str = "tinymemory-cortex-test"; + +/// How long an accepted write may take to be ranked by recall. +const VISIBILITY: Duration = Duration::from_secs(60); + +fn live_engine() -> Option> { + let url = std::env::var("TINYMEMORY_LIVE_CORTEXDB_URL").ok()?; + let key = std::env::var("TINYMEMORY_TEST_CORTEX_KEY").unwrap_or_else(|_| DEFAULT_KEY.into()); + Some(Arc::new( + CortexEngine::direct(&url, CortexCredential::api_key(key)).expect("a valid live endpoint"), + )) +} + +/// Recalls `query` until the pack contains every one of `wanted` or +/// [`VISIBILITY`] runs out. +async fn recall_until(memory: &AgentMemory, query: &str, wanted: &[&str]) -> ContextPack { + let deadline = Instant::now() + VISIBILITY; + loop { + let pack = memory.recall(query).await.expect("recall"); + if wanted.iter().all(|text| pack.markdown.contains(text)) || Instant::now() >= deadline { + return pack; + } + tokio::time::sleep(Duration::from_millis(500)).await; + } +} + +#[tokio::test] +async fn live_an_agent_loop_runs_against_cortexdb() { + let Some(engine) = live_engine() else { + eprintln!("TINYMEMORY_LIVE_CORTEXDB_URL unset; skipping"); + return; + }; + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("clock after the epoch") + .as_nanos(); + let layout = MemoryLayout::new( + format!("project:live-{nanos}") + .parse() + .expect("a valid root"), + ) + .expect("a valid layout"); + + let handbook = RawDocument::new( + "# Billing\n\nBilling disputes go to the finance channel within one day.\n", + ) + .with_filename("billing.md"); + let document = brain_document( + &ConverterChain::default(), + &handbook, + None, + MemoryMeta::default(), + ) + .await + .expect("convert"); + let ingested = Brain::new(engine.clone(), layout.clone()) + .ingest(document) + .await + .expect("ingest"); + + let support = AgentMemory::new(engine.clone(), layout.clone(), "support-01").expect("agent"); + let coder = AgentMemory::new(engine.clone(), layout.clone(), "coder-42").expect("agent"); + + let started = Instant::now(); + let turn = support + .pre_turn(PreTurn::new("s-1", 0, "where do billing disputes go")) + .await + .expect("pre_turn"); + let pre_turn = started.elapsed(); + assert!(turn.log_error.is_none(), "{:?}", turn.log_error); + support + .post_turn(PostTurn::new("s-1", 1, "Billing disputes go to finance.")) + .await + .expect("post_turn"); + coder + .pre_turn(PreTurn::new( + "c-1", + 0, + "billing service migration is blocked", + )) + .await + .expect("pre_turn"); + eprintln!("pre_turn took {pre_turn:?}"); + + let pack = recall_until( + &support, + "billing disputes", + &[ + "finance channel", + "Billing disputes go to finance", + "migration is blocked", + ], + ) + .await; + let md = &pack.markdown; + assert!( + md.contains("## Brain") && md.contains("finance channel"), + "{md}" + ); + assert!(md.contains("## This agent's history"), "{md}"); + assert!( + md.contains("## Team conversations") && md.contains("migration is blocked"), + "{md}" + ); + + let built = support + .run_background(ingested.job) + .await + .expect("a belief build is accepted"); + eprintln!("belief build: {:?}", built.outcome); + + let forgotten = engine + .forget(ForgetTarget::Filter(layout.holistic_filter())) + .await + .expect("forget"); + assert!(forgotten.forgotten >= 4, "{forgotten:?}"); +} diff --git a/crates/tinymemory-tools/Cargo.toml b/crates/tinymemory-tools/Cargo.toml index b77a22ed..493623a1 100644 --- a/crates/tinymemory-tools/Cargo.toml +++ b/crates/tinymemory-tools/Cargo.toml @@ -24,6 +24,9 @@ chrono = { version = "0.4", default-features = false, features = ["clock", "std" log = "0.4" # `context::Error`. thiserror = "2" +# A holistic recall reads its sections concurrently (`join_all`). Runtime +# agnostic, so the crate still links no executor. +futures = { version = "0.3", default-features = false, features = ["alloc"] } [dev-dependencies] # Every tool and the context compiler run against the reference engine. diff --git a/crates/tinymemory-tools/README.md b/crates/tinymemory-tools/README.md index 0ca92865..4d03b462 100644 --- a/crates/tinymemory-tools/README.md +++ b/crates/tinymemory-tools/README.md @@ -7,6 +7,15 @@ The agent-facing side of TinyMemory, over any `tinymemory_api::MemoryEngine`: entry point that runs them and returns compact JSON. - **`context`**: the `context.md` compiler, a token-budgeted brief a host injects at the start of a session. +- **`recall`**: holistic recall, the one read across several scopes, + rendered as one budgeted `ContextPack`. `context.md` is one preset of it. +- **`layout`**, **`brain`**, **`lifecycle`**, **`background`**: the agent + memory lifecycle. `MemoryLayout` is the standard tree: brain documents by + source, per-agent conversations, and learnings. `Brain` ingests documents. + `AgentMemory` serves session start, pre-turn context, post-turn logging and + compaction. `BackgroundJob` carries the belief builds those calls hand back. + See [`docs/architecture/lifecycle.md`](../../docs/architecture/lifecycle.md) + and the runnable `examples/agent_loop.rs` and `examples/brain.rs`. The crate has no tool-runtime dependency (no MCP, no `tinytools`). A host adapts `ToolSpec` to whatever runtime it uses; see diff --git a/crates/tinymemory-tools/examples/agent_loop.rs b/crates/tinymemory-tools/examples/agent_loop.rs new file mode 100644 index 00000000..84c9293b --- /dev/null +++ b/crates/tinymemory-tools/examples/agent_loop.rs @@ -0,0 +1,179 @@ +//! A whole agent memory lifecycle, offline, against the in-memory reference +//! engine: brain ingestion, two agents' turns with pre-turn context +//! injection, cross-agent recall, a session resume, compaction, and the +//! background belief builds the turns hand back. +//! +//! Run with: +//! +//! ```sh +//! cargo run -p tinymemory-tools --example agent_loop +//! ``` +//! +//! Swap `ReferenceEngine::new()` for any other `MemoryEngine` (the CortexDB +//! one is in `tinymemory-integrations`, see its `cortex_agent` example) and +//! nothing else changes. + +use std::sync::Arc; + +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_api::{MemoryEngine, Role, Turn}; +use tinymemory_tools::{ + AgentMemory, BackgroundJob, Brain, BrainDocument, BrainSource, Compaction, MemoryLayout, + PostTurn, PreTurn, RecallPolicy, SessionStart, +}; + +/// Stands in for the model: answers with the injected line sharing the most +/// words with the question, never quoting its own earlier answers. +fn generate(context: &str, user: &str) -> String { + let words: Vec = user + .split_whitespace() + .map(|word| { + word.trim_matches(|c: char| !c.is_alphanumeric()) + .to_lowercase() + }) + .filter(|word| word.len() > 4) + .collect(); + let overlap = |line: &str| { + let line = line.to_lowercase(); + words + .iter() + .filter(|word| line.contains(word.as_str())) + .count() + }; + let best = context + .lines() + .filter_map(|line| line.strip_prefix("- ")) + .filter(|line| !line.starts_with("assistant:")) + .max_by_key(|line| overlap(line)) + .filter(|line| overlap(line) > 0); + match best { + Some(fact) => format!("Going by memory ({fact})."), + None => "I have nothing on that yet; noting it.".to_string(), + } +} + +/// One turn of the agent loop: pre-turn context, generation, post-turn log. +async fn turn( + memory: &AgentMemory, + thread: &str, + index: u32, + user: &str, + jobs: &mut Vec, +) -> Result> { + let context = memory.pre_turn(PreTurn::new(thread, index, user)).await?; + let reply = generate(&context.pack.markdown, user); + let report = memory + .post_turn(PostTurn::new(thread, index + 1, reply.clone())) + .await?; + jobs.extend(report.jobs); + println!( + "[{}] user: {user}\n[{}] context: {} tokens, {} refs\n[{}] reply: {reply}\n", + memory.agent_id(), + memory.agent_id(), + context.pack.tokens, + context.pack.refs.len(), + memory.agent_id(), + ); + Ok(reply) +} + +#[tokio::main(flavor = "current_thread")] +async fn main() -> Result<(), Box> { + let engine: Arc = Arc::new(ReferenceEngine::new()); + let layout = MemoryLayout::default(); + let mut jobs: Vec = Vec::new(); + + // 1. The brain: company documents, global to every agent, by source. + let brain = Brain::new(engine.clone(), layout.clone()); + let batch = brain + .ingest_many(vec![ + BrainDocument::new( + BrainSource::Markdown, + "Every support ticket gets a first reply within four hours.", + ) + .titled("onboarding.md"), + BrainDocument::new( + BrainSource::Notion, + "Refunds settle within five business days of approval.", + ) + .titled("Refund policy"), + ]) + .await?; + println!( + "brain: {} documents stored, {} belief builds queued\n", + batch.receipts.len(), + batch.jobs.len() + ); + jobs.extend(batch.jobs); + + // 2. Two agents, each logging to its own scope and reading the whole tree. + let policy = RecallPolicy { + build_beliefs_every: Some(4), + ..RecallPolicy::default() + }; + let support = + AgentMemory::new(engine.clone(), layout.clone(), "support-01")?.with_policy(policy.clone()); + let coder = AgentMemory::new(engine.clone(), layout.clone(), "coder-42")?.with_policy(policy); + + turn( + &support, + "s-1", + 0, + "How long do refunds take to settle?", + &mut jobs, + ) + .await?; + turn( + &coder, + "c-1", + 0, + "The refund webhook deploy failed on Friday.", + &mut jobs, + ) + .await?; + turn( + &support, + "s-1", + 2, + "My customer says the refund webhook is broken.", + &mut jobs, + ) + .await?; + + // 3. Cross-agent recall: the coder's turn shows under the team. + let team = support.recall("refund webhook").await?; + assert!(team.markdown.contains("## Team conversations")); + println!("support's view of the team:\n{}\n", team.markdown); + + // 4. A new session resumes the thread from memory alone. + let resumed = support + .start_session(SessionStart { + thread_id: Some("s-1".into()), + focus: None, + }) + .await?; + println!("session resume:\n{}\n", resumed.markdown); + + // 5. The prompt overflows: carry the dropped turns forward as a summary. + let carried = support + .recall_for_compaction(Compaction { + thread_id: "s-1".into(), + dropped: vec![ + Turn::new(Role::User, "How long do refunds take to settle?"), + Turn::new(Role::User, "My customer says the refund webhook is broken."), + ], + focus: Some("refund webhook".into()), + }) + .await?; + println!("compaction carry-over:\n{}\n", carried.markdown); + + // 6. Off the hot path: run every queued job. + for job in jobs { + let report = support.run_background(job).await?; + println!("background {}: {:?}", report.job, report.outcome); + } + let after = support.recall("refunds settle").await?; + assert!(after.markdown.contains("## Learnings")); + println!("\nafter the belief builds:\n{}", after.markdown); + Ok(()) +} diff --git a/crates/tinymemory-tools/examples/brain.rs b/crates/tinymemory-tools/examples/brain.rs new file mode 100644 index 00000000..311b6496 --- /dev/null +++ b/crates/tinymemory-tools/examples/brain.rs @@ -0,0 +1,77 @@ +//! The brain on its own: documents by source type, searched per source or +//! across all of them, and erased one source at a time. +//! +//! Run with: +//! +//! ```sh +//! cargo run -p tinymemory-tools --example brain +//! ``` + +use std::sync::Arc; + +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_tools::{Brain, BrainDocument, BrainSource, MemoryLayout}; + +#[tokio::main(flavor = "current_thread")] +async fn main() -> Result<(), Box> { + // A layout rooted below a team node keeps one tenant's brain apart. + let layout = MemoryLayout::new("team:acme".parse()?)?; + let brain = Brain::new(Arc::new(ReferenceEngine::new()), layout.clone()); + + for (source, title, text) in [ + ( + BrainSource::Pdf, + "pricing.pdf", + "The Pro plan costs 20 dollars a month.", + ), + ( + BrainSource::Markdown, + "deploys.md", + "Deploys run on weekdays only.", + ), + ( + BrainSource::Notion, + "Pricing FAQ", + "Annual Pro plans get two months free.", + ), + ( + BrainSource::Github, + "README", + "Run cargo test before every pull request.", + ), + ] { + let ingested = brain + .ingest(BrainDocument::new(source.clone(), text).titled(title)) + .await?; + println!( + "{:<9} -> {} (then: {})", + source.to_string(), + layout.brain(&source)?, + ingested.job.name() + ); + } + + let everywhere = brain.search("pro plan pricing", None, 5).await?; + println!( + "\n'pro plan pricing' across the brain: {} hits", + everywhere.len() + ); + for hit in &everywhere { + println!(" {} | {}", hit.meta.namespace, hit.text.replace('\n', " ")); + } + + let notion_only = brain + .search("pro plan pricing", Some(&BrainSource::Notion), 5) + .await?; + println!("in notion only: {} hit", notion_only.len()); + + let forgotten = brain.forget(&BrainSource::Pdf).await?; + println!("\nforgot {} pdf document(s)", forgotten.forgotten); + let left = brain.search("pro plan pricing", None, 5).await?; + assert!( + left.iter() + .all(|hit| !hit.meta.namespace.to_string().contains("pdf")) + ); + println!("{} hits remain across the brain", left.len()); + Ok(()) +} diff --git a/crates/tinymemory-tools/src/background/mod.rs b/crates/tinymemory-tools/src/background/mod.rs new file mode 100644 index 00000000..43a08eb7 --- /dev/null +++ b/crates/tinymemory-tools/src/background/mod.rs @@ -0,0 +1,165 @@ +//! Background work: what a host runs off the live turn, and when it likes. +//! +//! Lifecycle calls never block on slow work. Instead they hand back +//! [`BackgroundJob`]s — plain, serializable values a host can queue, persist, +//! compare to drop duplicates, and run later on whatever executor it owns. +//! This crate spawns nothing. +//! +//! - [`BackgroundJob::BuildBeliefs`] asks the engine to consolidate a scope +//! ([`tinymemory_api::MemoryEngine::consolidate`]). +//! - [`BackgroundJob::IngestBrain`] stores brain documents the host chose +//! not to ingest inline; running it yields the belief builds that follow. +//! +//! [`BackgroundRunner::run`] executes one job. An engine that does not +//! consolidate turns a build into [`JobOutcome::Skipped`], not an error, so +//! the same host code runs against any engine. + +use std::sync::Arc; + +use serde::{Deserialize, Serialize}; +use tinymemory_api::{ + ConsolidateReceipt, ConsolidateRequest, ConsolidateStatus, Error, MemoryEngine, Result, + StoreReceipt, +}; + +use crate::brain::{Brain, BrainDocument}; +use crate::layout::MemoryLayout; + +/// One unit of deferred work. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "job", rename_all = "snake_case")] +pub enum BackgroundJob { + /// Build beliefs from a scope. + BuildBeliefs { + /// What to consolidate. + request: ConsolidateRequest, + }, + /// Store brain documents. + IngestBrain { + /// The documents, in order. + documents: Vec, + }, +} + +impl BackgroundJob { + /// The job's name: `build_beliefs` or `ingest_brain`. + #[must_use] + pub fn name(&self) -> &'static str { + match self { + Self::BuildBeliefs { .. } => "build_beliefs", + Self::IngestBrain { .. } => "ingest_brain", + } + } +} + +/// How a job ended. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "outcome", rename_all = "snake_case")] +pub enum JobOutcome { + /// The work is done. + Done, + /// The engine took the work and is doing it in the background. + Started, + /// The engine does this on its own schedule; nothing was started. + Scheduled, + /// The engine cannot do this; nothing happened. + Skipped { + /// Why. + reason: String, + }, +} + +/// What running one job did. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct JobReport { + /// The job's name ([`BackgroundJob::name`]). + pub job: &'static str, + /// How it ended. + pub outcome: JobOutcome, + /// The engine's consolidation receipt, for a build it accepted. + #[serde(skip_serializing_if = "Option::is_none")] + pub consolidation: Option, + /// Receipts of the documents an ingest stored. + #[serde(skip_serializing_if = "Vec::is_empty")] + pub stored: Vec, + /// Jobs this one gave rise to (an ingest's belief builds). + #[serde(skip_serializing_if = "Vec::is_empty")] + pub follow_ups: Vec, +} + +/// Runs [`BackgroundJob`]s against one engine and layout. Cheap to clone. +#[derive(Clone)] +pub struct BackgroundRunner { + engine: Arc, + layout: MemoryLayout, +} + +impl std::fmt::Debug for BackgroundRunner { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("BackgroundRunner") + .field("engine", &self.engine.descriptor().id) + .finish_non_exhaustive() + } +} + +impl BackgroundRunner { + /// A runner for `layout` on `engine`. + #[must_use] + pub fn new(engine: Arc, layout: MemoryLayout) -> Self { + Self { engine, layout } + } + + /// Runs `job` to the point the engine takes it. + /// + /// # Errors + /// + /// Invalid jobs, and the engine's failures other than + /// [`Error::Unsupported`] (which is [`JobOutcome::Skipped`]). + pub async fn run(&self, job: BackgroundJob) -> Result { + let name = job.name(); + match job { + BackgroundJob::BuildBeliefs { request } => self.build(name, request).await, + BackgroundJob::IngestBrain { documents } => { + let batch = Brain::new(self.engine.clone(), self.layout.clone()) + .ingest_many(documents) + .await?; + Ok(JobReport { + job: name, + outcome: JobOutcome::Done, + consolidation: None, + stored: batch.receipts, + follow_ups: batch.jobs, + }) + } + } + } + + async fn build(&self, name: &'static str, request: ConsolidateRequest) -> Result { + let report = |outcome, consolidation| JobReport { + job: name, + outcome, + consolidation, + stored: Vec::new(), + follow_ups: Vec::new(), + }; + match self.engine.consolidate(request).await { + Ok(receipt) => { + let outcome = match receipt.status { + ConsolidateStatus::Started => JobOutcome::Started, + ConsolidateStatus::Scheduled => JobOutcome::Scheduled, + ConsolidateStatus::Completed => JobOutcome::Done, + }; + Ok(report(outcome, Some(receipt))) + } + Err(Error::Unsupported(reason)) => { + log::debug!("[background] belief build skipped reason={reason}"); + Ok(report(JobOutcome::Skipped { reason }, None)) + } + Err(error) => Err(error), + } + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/background/mod_tests.rs b/crates/tinymemory-tools/src/background/mod_tests.rs new file mode 100644 index 00000000..fdc6e18f --- /dev/null +++ b/crates/tinymemory-tools/src/background/mod_tests.rs @@ -0,0 +1,124 @@ +//! Running jobs: builds against engines that do and do not consolidate, and +//! deferred brain ingestion. + +use async_trait::async_trait; +use tinymemory_api::conformance::{CONSOLIDATED_TAG, ReferenceEngine}; +use tinymemory_api::{ + EngineDescriptor, EngineHealth, FetchPage, FetchRequest, ForgetReport, ForgetTarget, ItemKind, + ListPage, ListRequest, MemoryMeta, MetaFilter, Namespace, Reach, RecallAnswer, RecallRequest, + StoreItem, +}; + +use super::*; +use crate::layout::BrainSource; + +fn runner(engine: Arc) -> BackgroundRunner { + BackgroundRunner::new(engine, MemoryLayout::default()) +} + +#[tokio::test] +async fn a_build_on_a_consolidating_engine_reports_its_receipt() { + let engine = Arc::new(ReferenceEngine::new()); + engine + .store(StoreItem::document( + "Refunds take five days.", + MemoryMeta { + namespace: Namespace::source("pdf"), + ..MemoryMeta::default() + }, + )) + .await + .unwrap(); + let job = BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::exact(Namespace::source("pdf"))), + }; + let report = runner(engine.clone()).run(job).await.unwrap(); + assert_eq!(report.job, "build_beliefs"); + assert_eq!(report.outcome, JobOutcome::Done); + let receipt = report.consolidation.expect("a build reports its receipt"); + assert_eq!(receipt.status, ConsolidateStatus::Completed); + assert_eq!(receipt.built, Some(1), "one belief from one document"); + let learnings = engine + .list(ListRequest::new( + MetaFilter::kinds([ItemKind::Learning]), + 10, + )) + .await + .unwrap(); + assert_eq!(learnings.items.len(), 1); + assert_eq!(learnings.items[0].meta.tags, [CONSOLIDATED_TAG]); +} + +/// The reference engine without consolidation: the trait's default refusal. +struct Plain(ReferenceEngine); + +#[async_trait] +impl MemoryEngine for Plain { + fn descriptor(&self) -> &EngineDescriptor { + self.0.descriptor() + } + async fn health(&self) -> EngineHealth { + EngineHealth::Ok + } + async fn recall(&self, req: RecallRequest) -> Result { + self.0.recall(req).await + } + async fn fetch(&self, req: FetchRequest) -> Result { + self.0.fetch(req).await + } + async fn store(&self, item: StoreItem) -> Result { + self.0.store(item).await + } + async fn forget(&self, target: ForgetTarget) -> Result { + self.0.forget(target).await + } + async fn list(&self, req: ListRequest) -> Result { + self.0.list(req).await + } +} + +#[tokio::test] +async fn a_build_on_an_engine_that_cannot_is_skipped_not_failed() { + let report = runner(Arc::new(Plain(ReferenceEngine::new()))) + .run(BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::default()), + }) + .await + .unwrap(); + assert!( + matches!(report.outcome, JobOutcome::Skipped { .. }), + "{report:?}" + ); + assert!(report.consolidation.is_none()); +} + +#[tokio::test] +async fn an_invalid_build_is_an_error() { + let error = runner(Arc::new(ReferenceEngine::new())) + .run(BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::default()) + .kinds([ItemKind::Document, ItemKind::Document]), + }) + .await + .unwrap_err(); + assert!(matches!(error, Error::InvalidRequest(_)), "{error:?}"); +} + +#[tokio::test] +async fn a_deferred_ingest_stores_and_hands_back_its_builds() { + let engine = Arc::new(ReferenceEngine::new()); + let job = BackgroundJob::IngestBrain { + documents: vec![ + BrainDocument::new(BrainSource::Pdf, "one"), + BrainDocument::new(BrainSource::Pdf, "two"), + ], + }; + let json = serde_json::to_value(&job).unwrap(); + assert_eq!(json["job"], "ingest_brain"); + let job: BackgroundJob = serde_json::from_value(json).unwrap(); + let report = runner(engine.clone()).run(job).await.unwrap(); + assert_eq!(report.outcome, JobOutcome::Done); + assert_eq!(report.stored.len(), 2); + assert_eq!(report.follow_ups.len(), 1); + assert_eq!(engine.len(), 2); +} diff --git a/crates/tinymemory-tools/src/brain/mod.rs b/crates/tinymemory-tools/src/brain/mod.rs new file mode 100644 index 00000000..c5e3486d --- /dev/null +++ b/crates/tinymemory-tools/src/brain/mod.rs @@ -0,0 +1,193 @@ +//! The brain: an agent-independent store of documents, kept apart by source +//! type. +//! +//! Company knowledge — PDFs, markdown, Notion exports, GitHub — belongs to no +//! agent, so a brain document carries no agent id and lives at its source's +//! node in the [`MemoryLayout`] (`source:pdf`, `source:notion`). Every agent +//! reads it through the holistic recall. +//! +//! [`Brain::ingest`] stores one document (by default waiting until it is +//! readable, since ingestion is not on a live turn) and returns the +//! [`BackgroundJob`] that would build beliefs from its source — the host runs +//! it whenever suits, through [`crate::BackgroundRunner`]. Converting a +//! file's bytes to text is the integrations crate's job; the brain takes +//! text. +//! +//! # Example +//! +//! ``` +//! use std::sync::Arc; +//! use tinymemory_api::conformance::ReferenceEngine; +//! use tinymemory_tools::{Brain, BrainDocument, BrainSource, MemoryLayout}; +//! +//! # let runtime = tokio::runtime::Builder::new_current_thread().build()?; +//! # runtime.block_on(async { +//! let brain = Brain::new(Arc::new(ReferenceEngine::new()), MemoryLayout::default()); +//! let ingested = brain +//! .ingest(BrainDocument::new(BrainSource::Markdown, "Refunds take five business days.") +//! .titled("refunds.md")) +//! .await?; +//! assert!(!ingested.receipt.replayed); +//! +//! let hits = brain.search("refunds", Some(&BrainSource::Markdown), 5).await?; +//! assert_eq!(hits.len(), 1); +//! assert!(brain.search("refunds", Some(&BrainSource::Pdf), 5).await?.is_empty()); +//! # Ok::<(), tinymemory_api::Error>(()) +//! # })?; +//! # Ok::<(), Box>(()) +//! ``` + +mod types; + +use std::collections::BTreeSet; +use std::sync::Arc; + +use tinymemory_api::{ + ConsolidateRequest, FetchMode, FetchRequest, ForgetReport, ForgetTarget, Hit, ItemKind, + MAX_STORE_MANY, MemoryEngine, Reach, Result, StoreItem, WriteOptions, +}; + +use crate::background::BackgroundJob; +use crate::layout::{BrainSource, MemoryLayout}; + +pub use types::{BrainBatch, BrainDocument, Ingested}; + +/// The brain over one engine and layout. Cheap to clone. +#[derive(Clone)] +pub struct Brain { + engine: Arc, + layout: MemoryLayout, +} + +impl std::fmt::Debug for Brain { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Brain") + .field("engine", &self.engine.descriptor().id) + .field("root", &self.layout.root().to_string()) + .finish() + } +} + +impl Brain { + /// The brain of `layout` on `engine`. + #[must_use] + pub fn new(engine: Arc, layout: MemoryLayout) -> Self { + Self { engine, layout } + } + + /// The layout the brain writes to. + #[must_use] + pub fn layout(&self) -> &MemoryLayout { + &self.layout + } + + /// Stores one document at its source's node, returning once it is + /// readable. + /// + /// # Errors + /// + /// An invalid document, and the engine's failures. + pub async fn ingest(&self, document: BrainDocument) -> Result { + self.ingest_with(document, WriteOptions::visible()).await + } + + /// [`Brain::ingest`], returning as soon as `options` allows. + /// + /// # Errors + /// + /// As [`Brain::ingest`]. + pub async fn ingest_with( + &self, + document: BrainDocument, + options: WriteOptions, + ) -> Result { + let source = document.source.clone(); + let item = document.into_item(&self.layout)?; + let receipt = self.engine.store_with(item, options).await?; + Ok(Ingested { + receipt, + job: self.build_job(&source)?, + }) + } + + /// Stores many documents, in order, in batches of at most + /// [`MAX_STORE_MANY`]; returns every receipt and one belief build per + /// source touched. + /// + /// # Errors + /// + /// An invalid document (nothing is stored), and the engine's failures + /// (earlier batches stay stored; storing again replays them). + pub async fn ingest_many(&self, documents: Vec) -> Result { + let sources: BTreeSet = + documents.iter().map(|doc| doc.source.clone()).collect(); + let items = documents + .into_iter() + .map(|document| document.into_item(&self.layout)) + .collect::>>()?; + let mut receipts = Vec::with_capacity(items.len()); + let mut rest = items; + while !rest.is_empty() { + let tail = rest.split_off(rest.len().min(MAX_STORE_MANY)); + receipts.extend(self.engine.store_many(rest).await?); + rest = tail; + } + let jobs = sources + .iter() + .map(|source| self.build_job(source)) + .collect::>>()?; + Ok(BrainBatch { receipts, jobs }) + } + + /// Ranked documents for `query`: one source's, or the whole brain's. + /// + /// # Errors + /// + /// An invalid query, an engine with no fetch mode, and the engine's + /// failures. + pub async fn search( + &self, + query: &str, + source: Option<&BrainSource>, + limit: usize, + ) -> Result> { + let descriptor = self.engine.descriptor(); + let mode = if descriptor.supports(FetchMode::Hybrid) { + FetchMode::Hybrid + } else { + descriptor.fetch_modes.first().copied().ok_or_else(|| { + tinymemory_api::Error::Unsupported(format!( + "engine `{}` offers no fetch mode", + descriptor.id + )) + })? + }; + let mut request = FetchRequest::new(query, mode, limit); + request.filter = self.layout.brain_filter(source); + Ok(self.engine.fetch(request).await?.hits) + } + + /// Erases one source's documents (and the beliefs an engine built in + /// that source's scope are the engine's to drop). + /// + /// # Errors + /// + /// The engine's failures. + pub async fn forget(&self, source: &BrainSource) -> Result { + let mut filter = self.layout.brain_filter(Some(source)); + filter.reach = Some(Reach::exact(self.layout.brain(source)?)); + self.engine.forget(ForgetTarget::Filter(filter)).await + } + + /// The belief build for `source`'s documents. + fn build_job(&self, source: &BrainSource) -> Result { + Ok(BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::exact(self.layout.brain(source)?)) + .kinds([ItemKind::Document]), + }) + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/brain/mod_tests.rs b/crates/tinymemory-tools/src/brain/mod_tests.rs new file mode 100644 index 00000000..7cc1f27c --- /dev/null +++ b/crates/tinymemory-tools/src/brain/mod_tests.rs @@ -0,0 +1,139 @@ +//! Brain ingestion, search, forgetting and the belief builds it hands back. + +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_api::{ + ConsolidateRequest, Error, ListRequest, MemoryMeta, Namespace, SourceKind, SourceRef, +}; + +use super::*; + +fn brain() -> (Arc, Brain) { + let engine = Arc::new(ReferenceEngine::new()); + (engine.clone(), Brain::new(engine, MemoryLayout::default())) +} + +#[tokio::test] +async fn a_document_lands_at_its_source_node_without_an_agent() { + let (engine, brain) = brain(); + let meta = MemoryMeta { + namespace: Namespace::agent("sneaky"), + agent_id: Some("sneaky".into()), + file_path: Some("/docs/refunds.pdf".into()), + ..MemoryMeta::default() + }; + let ingested = brain + .ingest( + BrainDocument::new(BrainSource::Pdf, "Refunds take five days.") + .titled("refunds.pdf") + .with_meta(meta), + ) + .await + .unwrap(); + assert_eq!( + ingested.job, + BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::exact(Namespace::source("pdf"))) + .kinds([ItemKind::Document]), + } + ); + let listed = engine + .list(ListRequest::new(Default::default(), 10)) + .await + .unwrap(); + let meta = &listed.items[0].meta; + assert_eq!(meta.namespace, Namespace::source("pdf")); + assert_eq!(meta.agent_id, None); + assert_eq!(meta.file_path.as_deref(), Some("/docs/refunds.pdf")); + assert_eq!(meta.source.kind, SourceKind::File); +} + +#[tokio::test] +async fn a_reader_s_source_is_kept() { + let (engine, brain) = brain(); + let meta = MemoryMeta { + source: SourceRef { + kind: SourceKind::Folder, + id: Some("handbook".into()), + }, + ..MemoryMeta::default() + }; + brain + .ingest(BrainDocument::new(BrainSource::Markdown, "Be kind.").with_meta(meta)) + .await + .unwrap(); + let listed = engine + .list(ListRequest::new(Default::default(), 10)) + .await + .unwrap(); + assert_eq!(listed.items[0].meta.source.kind, SourceKind::Folder); +} + +#[tokio::test] +async fn ingest_many_batches_and_builds_once_per_source() { + let (engine, brain) = brain(); + let documents: Vec = (0..MAX_STORE_MANY + 5) + .map(|i| { + let source = if i % 2 == 0 { + BrainSource::Notion + } else { + BrainSource::Github + }; + BrainDocument::new(source, format!("page {i}")) + }) + .collect(); + let batch = brain.ingest_many(documents).await.unwrap(); + assert_eq!(batch.receipts.len(), MAX_STORE_MANY + 5); + assert_eq!(engine.len(), MAX_STORE_MANY + 5); + assert_eq!(batch.jobs.len(), 2); + let again = brain + .ingest_many(vec![BrainDocument::new(BrainSource::Notion, "page 0")]) + .await + .unwrap(); + assert!(again.receipts[0].replayed); +} + +#[tokio::test] +async fn a_blank_document_is_refused_and_nothing_is_stored() { + let (engine, brain) = brain(); + let error = brain + .ingest_many(vec![ + BrainDocument::new(BrainSource::Web, "fine"), + BrainDocument::new(BrainSource::Web, " "), + ]) + .await + .unwrap_err(); + assert!(matches!(error, Error::InvalidRequest(_)), "{error:?}"); + assert!(engine.is_empty()); +} + +#[tokio::test] +async fn search_and_forget_stay_inside_one_source() { + let (engine, brain) = brain(); + for (source, text) in [ + (BrainSource::Pdf, "refund policy pdf"), + (BrainSource::Notion, "refund policy notion"), + ] { + brain + .ingest(BrainDocument::new(source, text)) + .await + .unwrap(); + } + assert_eq!(brain.search("refund", None, 10).await.unwrap().len(), 2); + let notion = brain + .search("refund", Some(&BrainSource::Notion), 10) + .await + .unwrap(); + assert_eq!(notion.len(), 1); + assert!(notion[0].text.contains("notion")); + + let report = brain.forget(&BrainSource::Pdf).await.unwrap(); + assert_eq!(report.forgotten, 1); + assert_eq!(engine.len(), 1); + assert!( + brain + .search("refund", Some(&BrainSource::Pdf), 10) + .await + .unwrap() + .is_empty() + ); +} diff --git a/crates/tinymemory-tools/src/brain/types.rs b/crates/tinymemory-tools/src/brain/types.rs new file mode 100644 index 00000000..ce13ae2c --- /dev/null +++ b/crates/tinymemory-tools/src/brain/types.rs @@ -0,0 +1,104 @@ +//! The brain's document and receipt types. + +use serde::{Deserialize, Serialize}; +use tinymemory_api::{ + DocumentBody, Error, MemoryMeta, Result, SourceKind, StoreItem, StoreReceipt, +}; + +use crate::background::BackgroundJob; +use crate::layout::{BrainSource, MemoryLayout}; + +/// One document for the brain, as text. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct BrainDocument { + /// The source type; decides the document's node. + pub source: BrainSource, + /// Title, when the source has one (a file name, a page title). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub title: Option, + /// The text, normally markdown. + pub text: String, + /// The text's MIME type, when known. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub mime: Option, + /// Metadata. Its namespace and agent id are overwritten: a brain + /// document lives at its source's node and belongs to no agent. + #[serde(default)] + pub meta: MemoryMeta, +} + +impl BrainDocument { + /// A document of `source` with no title, MIME type or metadata. + #[must_use] + pub fn new(source: BrainSource, text: impl Into) -> Self { + Self { + source, + title: None, + text: text.into(), + mime: None, + meta: MemoryMeta::default(), + } + } + + /// The document with `title`. + #[must_use] + pub fn titled(mut self, title: impl Into) -> Self { + self.title = Some(title.into()); + self + } + + /// The document with `meta` (namespace and agent id are still + /// overwritten on ingest). + #[must_use] + pub fn with_meta(mut self, meta: MemoryMeta) -> Self { + self.meta = meta; + self + } + + /// The item this document is stored as, placed in `layout`. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for blank text, and an invalid item. + pub fn into_item(self, layout: &MemoryLayout) -> Result { + if self.text.trim().is_empty() { + return Err(Error::InvalidRequest(format!( + "a {} brain document has no text", + self.source + ))); + } + let mut meta = self.meta; + meta.namespace = layout.brain(&self.source)?; + meta.agent_id = None; + if meta.source.kind == SourceKind::default() && meta.source.id.is_none() { + meta.source.kind = self.source.source_kind(); + } + let item = StoreItem::Document { + title: self.title, + body: DocumentBody::Text(self.text), + mime: self.mime, + meta, + }; + item.validate()?; + Ok(item) + } +} + +/// What [`crate::Brain::ingest`] did. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct Ingested { + /// The engine's receipt. + pub receipt: StoreReceipt, + /// The belief build for the document's source, for the host to run when + /// it suits. + pub job: BackgroundJob, +} + +/// What [`crate::Brain::ingest_many`] did. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct BrainBatch { + /// One receipt per document, in order. + pub receipts: Vec, + /// One belief build per source touched. + pub jobs: Vec, +} diff --git a/crates/tinymemory-tools/src/context/compile/mod.rs b/crates/tinymemory-tools/src/context/compile/mod.rs index 9baa3bb7..50633e7e 100644 --- a/crates/tinymemory-tools/src/context/compile/mod.rs +++ b/crates/tinymemory-tools/src/context/compile/mod.rs @@ -1,33 +1,30 @@ -//! [`ContextCompiler`]: gathers briefs and learnings from an engine and -//! renders them into a budgeted `context.md`. +//! [`ContextCompiler`]: `context.md` as a holistic recall. //! -//! Gathering is one recall per brief, then a listing of learnings. A brief -//! whose recall fails, or that cites nothing, is skipped (a failure is logged); -//! a failed learnings listing leaves the learnings out. Neither fails the -//! document, so an engine that holds nothing yields an empty document. - -mod render; +//! The document is [`crate::recall`] with a fixed shape: one answered +//! section per brief (in order), then the latest learnings as a list, under +//! `# Context` with frontmatter. A brief whose recall fails, or that cites +//! nothing, is skipped (a failure is logged); a failed learnings listing +//! leaves the learnings out. Neither fails the document, so an engine that +//! holds nothing yields an empty document. use chrono::{DateTime, Utc}; use serde::Serialize; -use tinymemory_api::{ - Hit, ItemId, ItemKind, ListRequest, MemoryEngine, MetaFilter, Reach, RecallRequest, -}; +use tinymemory_api::{ItemId, ItemKind, MemoryEngine, MetaFilter}; -use crate::context::error::Result; +use crate::context::error::{Error, Result}; use crate::context::spec::ContextSpec; -use render::{BriefSection, LearningLine, Sections}; +use crate::recall::{self, Frontmatter, HolisticRecall, ScopeSection, SectionQuery}; -pub use render::estimate_tokens; +pub use crate::recall::estimate_tokens; /// Most citations one brief's recall gathers. const BRIEF_CITATIONS: usize = 8; -/// Page size of the learnings listing. -const LEARNINGS_PAGE: usize = 100; +/// The document's `#` heading. +const TITLE: &str = "Context"; -/// Most learnings pages read before sorting; a ceiling, not a target. -const LEARNINGS_MAX_PAGES: usize = 50; +/// The learnings section's heading. +const LEARNINGS_HEADING: &str = "Learnings"; /// Instructions sent with every brief's recall. const BRIEF_INSTRUCTIONS: &str = "Answer briefly, as markdown bullet points suitable for a \ @@ -84,22 +81,24 @@ impl ContextCompiler { spec.validate()?; let generated_at = self.at.unwrap_or_else(Utc::now); let engine_id = engine.descriptor().id; - let sections = Sections { - briefs: gather_briefs(engine, spec).await, - learnings: gather_learnings(engine, spec.learnings_limit, spec.reach.as_ref()).await, + let frontmatter = Frontmatter { + engine: engine_id, + generated_at, }; - let rendered = render::render(sections, spec.budget_tokens, engine_id, generated_at); + let pack = recall::run(engine, &holistic(spec), Some(frontmatter)) + .await + .map_err(|error| Error::InvalidSpec(error.to_string()))?; log::debug!( "[context] compiled engine={engine_id} tokens={} refs={}", - rendered.tokens, - rendered.refs.len() + pack.tokens, + pack.refs.len() ); Ok(ContextDoc { - markdown: rendered.markdown, - tokens: rendered.tokens, + markdown: pack.markdown, + tokens: pack.tokens, generated_at, engine: engine_id.to_string(), - refs: rendered.refs, + refs: pack.refs, }) } } @@ -113,98 +112,44 @@ pub async fn compile(engine: &dyn MemoryEngine, spec: &ContextSpec) -> Result Vec { - let mut sections = Vec::with_capacity(spec.briefs.len()); - for brief in &spec.briefs { - let mut filter = brief.filter.clone(); - if spec.reach.is_some() { - filter.reach = spec.reach.clone(); - } - let request = RecallRequest { - question: brief.question.clone(), - filter, - limit: BRIEF_CITATIONS, - instructions: Some(BRIEF_INSTRUCTIONS.to_string()), - }; - match engine.recall(request).await { - Ok(answer) if answer.citations.is_empty() || answer.answer.trim().is_empty() => { - log::debug!( - "[context] brief skipped heading={:?} reason=nothing_cited", - brief.heading - ); +/// The holistic recall `spec` describes: one answered section per brief, +/// then the latest learnings, every read confined to `spec.reach`. +fn holistic(spec: &ContextSpec) -> HolisticRecall { + let mut sections: Vec = spec + .briefs + .iter() + .map(|brief| { + let mut filter = brief.filter.clone(); + if spec.reach.is_some() { + filter.reach = spec.reach.clone(); } - Ok(answer) => sections.push(BriefSection { + ScopeSection { heading: brief.heading.clone(), - body: answer.answer.trim().to_string(), - refs: answer.citations.into_iter().map(|c| c.id).collect(), - }), - Err(error) => { - log::warn!( - "[context] brief skipped heading={:?} error={error}", - brief.heading - ); + filter, + limit: BRIEF_CITATIONS, + query: SectionQuery::Answer { + question: brief.question.clone(), + instructions: Some(BRIEF_INSTRUCTIONS.to_string()), + fallback_to_fetch: false, + }, } - } + }) + .collect(); + if spec.learnings_limit > 0 { + sections.push(ScopeSection::latest( + LEARNINGS_HEADING, + MetaFilter { + reach: spec.reach.clone(), + ..MetaFilter::kinds([ItemKind::Learning]) + }, + spec.learnings_limit, + )); } - sections -} - -async fn gather_learnings( - engine: &dyn MemoryEngine, - limit: usize, - reach: Option<&Reach>, -) -> Vec { - if limit == 0 { - return Vec::new(); + HolisticRecall { + budget_tokens: spec.budget_tokens, + title: TITLE.to_string(), + ..HolisticRecall::new(None, sections) } - let mut all: Vec = Vec::new(); - let mut cursor: Option = None; - for _ in 0..LEARNINGS_MAX_PAGES { - let filter = MetaFilter { - reach: reach.cloned(), - ..MetaFilter::kinds([ItemKind::Learning]) - }; - let mut request = ListRequest::new(filter, LEARNINGS_PAGE); - request.cursor = cursor.take(); - match engine.list(request).await { - Ok(page) => { - all.extend( - page.items - .into_iter() - .filter(|hit| hit.kind == ItemKind::Learning), - ); - match page.next_cursor { - Some(next) => cursor = Some(next), - None => break, - } - } - Err(error) => { - log::warn!("[context] learnings skipped error={error}"); - return Vec::new(); - } - } - } - rank_learnings(all) - .into_iter() - .take(limit) - .map(|hit| LearningLine { - id: hit.id, - text: hit.text, - }) - .collect() -} - -/// Newest first, then most confident; undated learnings after dated ones, -/// and ties keep the engine's listing order. -fn rank_learnings(mut hits: Vec) -> Vec { - hits.sort_by(|a, b| { - b.meta.observed_at.cmp(&a.meta.observed_at).then_with(|| { - b.confidence - .unwrap_or(0.0) - .total_cmp(&a.confidence.unwrap_or(0.0)) - }) - }); - hits } #[cfg(test)] diff --git a/crates/tinymemory-tools/src/context/compile/mod_tests.rs b/crates/tinymemory-tools/src/context/compile/mod_tests.rs index d7485933..1f56e4f5 100644 --- a/crates/tinymemory-tools/src/context/compile/mod_tests.rs +++ b/crates/tinymemory-tools/src/context/compile/mod_tests.rs @@ -5,7 +5,8 @@ use chrono::TimeZone; use tinymemory_api::conformance::ReferenceEngine; use tinymemory_api::{ EngineDescriptor, EngineHealth, Error as ApiError, FetchPage, FetchRequest, ForgetReport, - ForgetTarget, LearningKind, ListPage, MemoryMeta, RecallAnswer, StoreItem, StoreReceipt, + ForgetTarget, LearningKind, ListPage, ListRequest, MemoryMeta, RecallAnswer, RecallRequest, + StoreItem, StoreReceipt, }; use super::*; diff --git a/crates/tinymemory-tools/src/context/compile/render.rs b/crates/tinymemory-tools/src/context/compile/render.rs deleted file mode 100644 index 25e1c7b6..00000000 --- a/crates/tinymemory-tools/src/context/compile/render.rs +++ /dev/null @@ -1,176 +0,0 @@ -//! Rendering gathered sections into budgeted markdown. -//! -//! Pure: given the sections and the budget, the output is fixed. Trimming -//! order is the spec's: learnings go first, one line at a time from the end; -//! then the last remaining brief is shortened, and dropped once too little of -//! it is left, so earlier briefs keep their text longest and every brief keeps -//! its place. - -use chrono::{DateTime, SecondsFormat, Utc}; -use tinymemory_api::ItemId; - -/// Characters per estimated token. -const CHARS_PER_TOKEN: usize = 4; - -/// A shortened brief shorter than this is dropped rather than kept as a stub. -const MIN_BRIEF_CHARS: usize = 40; - -/// Marks a shortened brief. -const ELLIPSIS: char = '…'; - -/// The estimated token count of `text`: four characters per token, rounded -/// up — the estimate every context budget uses. -pub fn estimate_tokens(text: &str) -> usize { - text.chars().count().div_ceil(CHARS_PER_TOKEN) -} - -/// One answered brief. -#[derive(Debug, Clone, PartialEq)] -pub(crate) struct BriefSection { - pub(crate) heading: String, - pub(crate) body: String, - pub(crate) refs: Vec, -} - -/// One learning line. -#[derive(Debug, Clone, PartialEq)] -pub(crate) struct LearningLine { - pub(crate) id: ItemId, - pub(crate) text: String, -} - -/// What the document is rendered from. -#[derive(Debug, Clone)] -pub(crate) struct Sections { - pub(crate) briefs: Vec, - pub(crate) learnings: Vec, -} - -/// The rendered document. -#[derive(Debug, Clone, PartialEq)] -pub(crate) struct Rendered { - pub(crate) markdown: String, - pub(crate) tokens: usize, - pub(crate) refs: Vec, -} - -/// Renders `sections` within `budget_tokens`, trimming learnings first. -pub(crate) fn render( - mut sections: Sections, - budget_tokens: usize, - engine: &str, - generated_at: DateTime, -) -> Rendered { - loop { - let rendered = compose(§ions, engine, generated_at); - if rendered.tokens <= budget_tokens { - return rendered; - } - if sections.learnings.pop().is_some() { - continue; - } - let Some(last) = sections.briefs.last_mut() else { - // Nothing left to trim: an empty document always fits. - return compose(§ions, engine, generated_at); - }; - let overflow_chars = (rendered.tokens - budget_tokens) * CHARS_PER_TOKEN; - let keep = last - .body - .chars() - .count() - .saturating_sub(overflow_chars + ELLIPSIS.len_utf8()); - if keep < MIN_BRIEF_CHARS { - sections.briefs.pop(); - } else { - last.body = shorten(&last.body, keep); - } - } -} - -/// The first `keep` characters of `text`, cut back to a word boundary when -/// one is near, with an ellipsis. -fn shorten(text: &str, keep: usize) -> String { - let cut: String = text.chars().take(keep).collect(); - let trimmed = match cut.rfind(char::is_whitespace) { - Some(space) if space * 2 > cut.len() => &cut[..space], - _ => cut.as_str(), - }; - format!("{}{ELLIPSIS}", trimmed.trim_end()) -} - -/// Composes the full document, frontmatter included, without trimming. -fn compose(sections: &Sections, engine: &str, generated_at: DateTime) -> Rendered { - if sections.briefs.is_empty() && sections.learnings.is_empty() { - return Rendered { - markdown: String::new(), - tokens: 0, - refs: Vec::new(), - }; - } - let mut refs: Vec = Vec::new(); - let mut body = String::from("# Context\n"); - for brief in §ions.briefs { - body.push_str(&format!( - "\n## {}\n\n{}\n", - brief.heading, - brief.body.trim() - )); - push_unique(&mut refs, &brief.refs); - } - if !sections.learnings.is_empty() { - body.push_str("\n## Learnings\n\n"); - for line in §ions.learnings { - body.push_str(&format!("- {}\n", single_line(&line.text))); - push_unique(&mut refs, std::slice::from_ref(&line.id)); - } - } - // The frontmatter's own token count is part of the document, so estimate - // the count it reports from a draft carrying a same-width placeholder, - // then fix the number point so the report is exact. - let draft = frontmatter(engine, generated_at, 0, &refs) + "\n" + &body; - let mut tokens = estimate_tokens(&draft); - let mut markdown = draft; - for _ in 0..8 { - markdown = frontmatter(engine, generated_at, tokens, &refs) + "\n" + &body; - let settled = estimate_tokens(&markdown); - if settled == tokens { - break; - } - tokens = settled; - } - Rendered { - tokens: estimate_tokens(&markdown), - markdown, - refs, - } -} - -fn frontmatter( - engine: &str, - generated_at: DateTime, - tokens: usize, - refs: &[ItemId], -) -> String { - let refs: Vec<&str> = refs.iter().map(ItemId::as_str).collect(); - format!( - "---\ngenerated_at: {}\nengine: {engine}\ntokens: {tokens}\nrefs: [{}]\n---\n", - generated_at.to_rfc3339_opts(SecondsFormat::Secs, true), - refs.join(", ") - ) -} - -fn single_line(text: &str) -> String { - text.split_whitespace().collect::>().join(" ") -} - -fn push_unique(refs: &mut Vec, more: &[ItemId]) { - for id in more { - if !refs.contains(id) { - refs.push(id.clone()); - } - } -} - -#[cfg(test)] -#[path = "render_tests.rs"] -mod tests; diff --git a/crates/tinymemory-tools/src/context/compile/render_tests.rs b/crates/tinymemory-tools/src/context/compile/render_tests.rs deleted file mode 100644 index 9f843cfc..00000000 --- a/crates/tinymemory-tools/src/context/compile/render_tests.rs +++ /dev/null @@ -1,141 +0,0 @@ -//! Budgeting, trimming order and frontmatter. - -use chrono::TimeZone; - -use super::*; - -fn at() -> DateTime { - Utc.with_ymd_and_hms(2026, 10, 2, 12, 0, 0).unwrap() -} - -fn brief(heading: &str, body: &str, refs: &[&str]) -> BriefSection { - BriefSection { - heading: heading.to_string(), - body: body.to_string(), - refs: refs.iter().map(|id| ItemId::from(*id)).collect(), - } -} - -fn learning(id: &str, text: &str) -> LearningLine { - LearningLine { - id: ItemId::from(id), - text: text.to_string(), - } -} - -fn sections() -> Sections { - Sections { - briefs: vec![ - brief( - "About the user", - &"Steven builds memory systems. ".repeat(4), - &["a", "b"], - ), - brief( - "Active work", - &"Working on memory v2 in Rust. ".repeat(4), - &["b", "c"], - ), - ], - learnings: (0..10) - .map(|i| learning(&format!("l{i}"), &format!("learning number {i}\nwrapped"))) - .collect(), - } -} - -#[test] -fn tokens_are_four_characters_rounded_up() { - assert_eq!(estimate_tokens(""), 0); - assert_eq!(estimate_tokens("abcd"), 1); - assert_eq!(estimate_tokens("abcde"), 2); - assert_eq!(estimate_tokens("ééééé"), 2); -} - -#[test] -fn an_ample_budget_keeps_everything_with_frontmatter() { - let rendered = render(sections(), 10_000, "reference", at()); - let md = &rendered.markdown; - assert!(md.starts_with("---\ngenerated_at: 2026-10-02T12:00:00Z\nengine: reference\n")); - assert!(md.contains(&format!("tokens: {}\n", rendered.tokens))); - assert!(md.contains("refs: [a, b, c, l0, l1")); - let about = md.find("## About the user").unwrap(); - let active = md.find("## Active work").unwrap(); - let learnings = md.find("## Learnings").unwrap(); - assert!(about < active && active < learnings); - assert!(md.contains("- learning number 0 wrapped\n")); - assert_eq!(rendered.tokens, estimate_tokens(md)); - assert_eq!(rendered.refs.len(), 13); -} - -#[test] -fn learnings_are_trimmed_before_any_brief() { - let full = render(sections(), 10_000, "reference", at()); - let budget = full.tokens - 20; - let rendered = render(sections(), budget, "reference", at()); - assert!(rendered.tokens <= budget); - assert!( - rendered.markdown.contains( - &"Steven builds memory systems. " - .repeat(4) - .trim() - .to_string() - ) - ); - assert!( - rendered.markdown.contains( - &"Working on memory v2 in Rust. " - .repeat(4) - .trim() - .to_string() - ) - ); - assert!(!rendered.markdown.contains("learning number 9")); - assert!(rendered.markdown.contains("learning number 0")); - assert!(!rendered.refs.contains(&ItemId::from("l9"))); -} - -#[test] -fn once_learnings_are_gone_the_last_brief_shrinks_then_drops() { - let without_learnings = Sections { - learnings: Vec::new(), - ..sections() - }; - let full = render(without_learnings.clone(), 10_000, "reference", at()); - let shrunk = render( - without_learnings.clone(), - full.tokens - 10, - "reference", - at(), - ); - assert!(shrunk.tokens <= full.tokens - 10); - assert!(shrunk.markdown.contains("## Active work")); - assert!(shrunk.markdown.contains('…')); - assert!( - shrunk.markdown.contains( - &"Steven builds memory systems. " - .repeat(4) - .trim() - .to_string() - ) - ); - - let tight = render(without_learnings, full.tokens - 45, "reference", at()); - assert!(tight.tokens <= full.tokens - 45); - assert!(tight.markdown.contains("## About the user")); - assert!(!tight.markdown.contains("## Active work")); - assert!(!tight.refs.contains(&ItemId::from("c"))); -} - -#[test] -fn nothing_to_say_or_no_room_is_an_empty_document() { - let empty = Sections { - briefs: Vec::new(), - learnings: Vec::new(), - }; - let rendered = render(empty, 100, "reference", at()); - assert_eq!(rendered.markdown, ""); - assert_eq!(rendered.tokens, 0); - let squeezed = render(sections(), 5, "reference", at()); - assert_eq!(squeezed.markdown, ""); - assert!(squeezed.refs.is_empty()); -} diff --git a/crates/tinymemory-tools/src/layout/mod.rs b/crates/tinymemory-tools/src/layout/mod.rs new file mode 100644 index 00000000..29b84144 --- /dev/null +++ b/crates/tinymemory-tools/src/layout/mod.rs @@ -0,0 +1,160 @@ +//! The standard memory layout: where the brain, each agent's conversations +//! and the learnings live, as namespaces and filters. +//! +//! Every host that adopts the layout reads and writes the same tree, so an +//! engine can be swapped without moving anything: +//! +//! ```text +//! root core — holistic recall reads it all +//! ├── source:pdf documents core/brain/pdf +//! ├── source:markdown documents core/brain/markdown +//! ├── source:notion documents core/brain/notion +//! ├── agent:support-01 conversations core/conversations/support-01 +//! ├── agent:coder-42 conversations core/conversations/coder-42 +//! └── (root itself) learnings core/learnings +//! ``` +//! +//! - **Brain** — documents, global to every agent: no agent id, one +//! `source:` node per [`BrainSource`]. +//! - **Conversations** — each agent's turns at its own `agent:` node. +//! - **Learnings** — beliefs and facts. Shared ones live at the root; an +//! engine that consolidates writes its beliefs into the scope it built +//! them from, and the learnings scope reads the whole tree. +//! +//! The root is [`Namespace::ROOT`] unless a host scopes the whole layout +//! below a node of its own (`team:acme`), which keeps tenants apart on one +//! engine. +//! +//! # Example +//! +//! ``` +//! use tinymemory_api::{ItemKind, Namespace}; +//! use tinymemory_tools::{BrainSource, MemoryLayout}; +//! +//! let layout = MemoryLayout::default(); +//! assert_eq!(layout.brain(&BrainSource::Pdf)?.to_string(), "source:pdf"); +//! assert_eq!(layout.conversations("support-01")?.to_string(), "agent:support-01"); +//! assert_eq!(layout.learnings(), &Namespace::ROOT); +//! +//! let team = MemoryLayout::new("team:acme".parse()?)?; +//! assert_eq!(team.brain(&BrainSource::Notion)?.to_string(), "team:acme/source:notion"); +//! assert_eq!(team.brain_filter(None).kinds, [ItemKind::Document]); +//! # Ok::<(), tinymemory_api::Error>(()) +//! ``` + +mod source; + +use tinymemory_api::{Error, ItemKind, MetaFilter, Namespace, Reach, Result, Segment, SegmentKind}; + +pub use source::BrainSource; + +/// Deepest a layout root may be: one level must remain for the brain's and +/// the agents' nodes. +const MAX_ROOT_DEPTH: usize = 7; + +/// Where every part of an agent's memory lives. See the module docs. +#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)] +pub struct MemoryLayout { + root: Namespace, +} + +impl MemoryLayout { + /// A layout below `root`. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] when `root` is so deep no node fits below + /// it. + pub fn new(root: Namespace) -> Result { + if root.depth() > MAX_ROOT_DEPTH { + return Err(Error::InvalidRequest(format!( + "a memory layout root nests at most {MAX_ROOT_DEPTH} deep, `{root}` is deeper" + ))); + } + Ok(Self { root }) + } + + /// The layout's root: `core`. + #[must_use] + pub fn root(&self) -> &Namespace { + &self.root + } + + /// The node `source`'s documents live at. + /// + /// # Errors + /// + /// Never for a layout built by [`MemoryLayout::new`]; the depth check is + /// the namespace's own. + pub fn brain(&self, source: &BrainSource) -> Result { + self.root + .child(Segment::sanitized(SegmentKind::Source, source.id())) + } + + /// The node `agent_id`'s conversations live at. The id is sanitized + /// ([`Segment::sanitized`]). + /// + /// # Errors + /// + /// As [`MemoryLayout::brain`]. + pub fn conversations(&self, agent_id: &str) -> Result { + self.root + .child(Segment::sanitized(SegmentKind::Agent, agent_id)) + } + + /// The node shared learnings are written to: the root. + #[must_use] + pub fn learnings(&self) -> &Namespace { + &self.root + } + + /// The brain's documents: one source's, or every source's. + /// + /// With no source this reads every document in the layout, including any + /// an agent stored at its own node. + #[must_use] + pub fn brain_filter(&self, source: Option<&BrainSource>) -> MetaFilter { + let at = source + .and_then(|source| self.brain(source).ok()) + .unwrap_or_else(|| self.root.clone()); + MetaFilter { + reach: Some(Reach::subtree(at)), + ..MetaFilter::kinds([ItemKind::Document]) + } + } + + /// Conversations: one agent's (and its sub-agents'), or every agent's. + #[must_use] + pub fn conversations_filter(&self, agent_id: Option<&str>) -> MetaFilter { + let at = agent_id + .and_then(|agent| self.conversations(agent).ok()) + .unwrap_or_else(|| self.root.clone()); + MetaFilter { + reach: Some(Reach::subtree(at)), + ..MetaFilter::kinds([ItemKind::Conversation]) + } + } + + /// Every learning in the layout: shared ones at the root and the beliefs + /// an engine built at any node below. + #[must_use] + pub fn learnings_filter(&self) -> MetaFilter { + MetaFilter { + reach: Some(Reach::subtree(self.root.clone())), + ..MetaFilter::kinds([ItemKind::Learning]) + } + } + + /// Everything in the layout: the holistic scope. + #[must_use] + pub fn holistic_filter(&self) -> MetaFilter { + MetaFilter { + reach: Some(Reach::subtree(self.root.clone())), + ..MetaFilter::default() + } + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/layout/mod_tests.rs b/crates/tinymemory-tools/src/layout/mod_tests.rs new file mode 100644 index 00000000..c8e5cb68 --- /dev/null +++ b/crates/tinymemory-tools/src/layout/mod_tests.rs @@ -0,0 +1,77 @@ +//! Layout nodes, filters and brain source ids. + +use super::*; +use tinymemory_api::SourceKind; + +#[test] +fn places_every_part_below_the_root() { + let layout = MemoryLayout::new("team:acme".parse().unwrap()).unwrap(); + assert_eq!( + layout.brain(&BrainSource::Pdf).unwrap().to_string(), + "team:acme/source:pdf" + ); + assert_eq!( + layout.conversations("coder-42").unwrap().to_string(), + "team:acme/agent:coder-42" + ); + assert_eq!(layout.learnings().to_string(), "team:acme"); + let odd = layout.conversations("agent 7").unwrap(); + assert!(odd.to_string().starts_with("team:acme/agent:agent-7-")); +} + +#[test] +fn filters_read_their_scope_only() { + let layout = MemoryLayout::default(); + let pdf = layout.brain_filter(Some(&BrainSource::Pdf)); + let pdf_reach = pdf.reach.clone().unwrap(); + assert!(pdf_reach.admits(&Namespace::source("pdf"))); + assert!(!pdf_reach.admits(&Namespace::source("notion"))); + assert!(!pdf_reach.admits(&Namespace::ROOT)); + assert_eq!(pdf.kinds, [ItemKind::Document]); + + let brain = layout.brain_filter(None).reach.unwrap(); + assert!(brain.admits(&Namespace::source("notion"))); + + let support = layout.conversations_filter(Some("support")).reach.unwrap(); + assert!(support.admits(&Namespace::agent("support"))); + assert!(!support.admits(&Namespace::agent("coder"))); + let team = layout.conversations_filter(None); + assert!(team.reach.unwrap().admits(&Namespace::agent("coder"))); + assert_eq!(team.kinds, [ItemKind::Conversation]); + + assert_eq!(layout.learnings_filter().kinds, [ItemKind::Learning]); + assert!(layout.holistic_filter().kinds.is_empty()); +} + +#[test] +fn refuses_a_root_with_no_room_below() { + let deep: Namespace = ["agent:a"; 8].join("/").parse().unwrap(); + assert!(matches!( + MemoryLayout::new(deep), + Err(Error::InvalidRequest(_)) + )); + let deepest_allowed: Namespace = ["agent:a"; 7].join("/").parse().unwrap(); + let layout = MemoryLayout::new(deepest_allowed).unwrap(); + assert_eq!(layout.brain(&BrainSource::Web).unwrap().depth(), 8); +} + +#[test] +fn source_ids_round_trip() { + for source in [ + BrainSource::Pdf, + BrainSource::Markdown, + BrainSource::Notion, + BrainSource::Github, + BrainSource::Web, + BrainSource::Other("confluence".into()), + ] { + let id = source.to_string(); + assert_eq!(id.parse::().unwrap(), source); + let json = serde_json::to_value(&source).unwrap(); + assert_eq!(json, serde_json::json!(id)); + assert_eq!(serde_json::from_value::(json).unwrap(), source); + } + assert_eq!("MD".parse::().unwrap(), BrainSource::Markdown); + assert!(" ".parse::().is_err()); + assert_eq!(BrainSource::Github.source_kind(), SourceKind::Github); +} diff --git a/crates/tinymemory-tools/src/layout/source.rs b/crates/tinymemory-tools/src/layout/source.rs new file mode 100644 index 00000000..26202385 --- /dev/null +++ b/crates/tinymemory-tools/src/layout/source.rs @@ -0,0 +1,100 @@ +//! [`BrainSource`]: the type of source a brain document came from. + +use std::fmt; +use std::str::FromStr; + +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use tinymemory_api::{Error, Result, SourceKind}; + +/// Where a brain document came from. Each source type is its own scope +/// (`source:`), so the brain can be read, rebuilt or erased one source at +/// a time. +/// +/// On the wire a source is its id: `pdf`, `markdown`, `notion`, `github`, +/// `web`, or any other `[A-Za-z0-9_-]` id for [`BrainSource::Other`]. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum BrainSource { + /// PDF files. + Pdf, + /// Markdown and plain-text files. + Markdown, + /// Notion pages. + Notion, + /// GitHub repositories, issues and pull requests. + Github, + /// Web pages. + Web, + /// Any other source type, by id. + Other(String), +} + +impl BrainSource { + /// The source's id: its `source:` segment. + #[must_use] + pub fn id(&self) -> &str { + match self { + Self::Pdf => "pdf", + Self::Markdown => "markdown", + Self::Notion => "notion", + Self::Github => "github", + Self::Web => "web", + Self::Other(id) => id, + } + } + + /// The contract's [`SourceKind`] a document of this source carries when + /// its reader named none. + #[must_use] + pub fn source_kind(&self) -> SourceKind { + match self { + Self::Pdf | Self::Markdown => SourceKind::File, + Self::Notion => SourceKind::Composio, + Self::Github => SourceKind::Github, + Self::Web => SourceKind::Link, + Self::Other(_) => SourceKind::Import, + } + } +} + +impl fmt::Display for BrainSource { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.id()) + } +} + +impl FromStr for BrainSource { + type Err = Error; + + /// Parses a source id; the five known ids map to their variants + /// (`md` is `markdown`), anything else is [`BrainSource::Other`]. + fn from_str(value: &str) -> Result { + let value = value.trim(); + Ok(match value.to_ascii_lowercase().as_str() { + "" => { + return Err(Error::InvalidRequest( + "a brain source id must not be blank".to_string(), + )); + } + "pdf" => Self::Pdf, + "markdown" | "md" => Self::Markdown, + "notion" => Self::Notion, + "github" => Self::Github, + "web" => Self::Web, + _ => Self::Other(value.to_string()), + }) + } +} + +impl Serialize for BrainSource { + fn serialize(&self, serializer: S) -> std::result::Result { + serializer.serialize_str(self.id()) + } +} + +impl<'de> Deserialize<'de> for BrainSource { + fn deserialize>(deserializer: D) -> std::result::Result { + String::deserialize(deserializer)? + .parse() + .map_err(serde::de::Error::custom) + } +} diff --git a/crates/tinymemory-tools/src/lib.rs b/crates/tinymemory-tools/src/lib.rs index e259646d..831db703 100644 --- a/crates/tinymemory-tools/src/lib.rs +++ b/crates/tinymemory-tools/src/lib.rs @@ -8,6 +8,20 @@ //! argument can name either. //! - [`context`] compiles `context.md`, a token-budgeted brief a host injects //! at the start of a session. +//! - [`recall`] is the one read every lifecycle step is built from: a +//! holistic recall across several scopes, rendered as one budgeted +//! [`ContextPack`]. `context.md` is one preset of it. +//! - [`layout`] is the standard memory tree — a global **brain** of +//! documents by source type, each agent's **conversations**, and shared +//! **learnings** — as namespaces and filters ([`MemoryLayout`]). +//! - [`lifecycle`] runs one agent's memory through the agent loop +//! ([`AgentMemory`]): session start, pre-turn context, post-turn logging, +//! compaction recall. [`brain`] ingests and searches documents +//! ([`Brain`]), and [`background`] runs the slow work both hand back +//! ([`BackgroundJob`]). +//! +//! Everything is written against [`tinymemory_api::MemoryEngine`] alone, so a +//! host moves between engines by changing the one it constructs. //! //! # Example //! @@ -48,9 +62,22 @@ //! # Ok::<(), Box>(()) //! ``` +pub mod background; +pub mod brain; pub mod context; +pub mod layout; +pub mod lifecycle; +pub mod recall; pub mod tools; +pub use background::{BackgroundJob, BackgroundRunner, JobOutcome, JobReport}; +pub use brain::{Brain, BrainBatch, BrainDocument, Ingested}; +pub use layout::{BrainSource, MemoryLayout}; +pub use lifecycle::{ + AgentMemory, Compaction, PostTurn, PostTurnReport, PreTurn, RecallPolicy, SessionStart, + TurnContext, +}; +pub use recall::{ContextPack, HolisticRecall, ScopeSection, SectionQuery, holistic_recall}; pub use tools::{ MEMORY_EXPLORE, MEMORY_FETCH, MEMORY_FORGET, MEMORY_GET, MEMORY_LIST, MEMORY_RECALL, MEMORY_STORE, MemoryTools, TOOL_NAMES, ToolScope, ToolSpec, WRITE_TOOL_NAMES, diff --git a/crates/tinymemory-tools/src/lifecycle/mod.rs b/crates/tinymemory-tools/src/lifecycle/mod.rs new file mode 100644 index 00000000..41327129 --- /dev/null +++ b/crates/tinymemory-tools/src/lifecycle/mod.rs @@ -0,0 +1,519 @@ +//! [`AgentMemory`]: one agent's memory lifecycle over any engine. +//! +//! A host calls it at fixed points of the agent loop. Every read is a +//! [`crate::recall`] over the [`MemoryLayout`]'s scopes; every write is one +//! turn at this agent's node; every slow step is a [`BackgroundJob`] handed +//! back for the host to run. +//! +//! | When | Call | Engine work | +//! | --- | --- | --- | +//! | session start or resume | [`AgentMemory::start_session`] | reads only | +//! | user turn, before the model | [`AgentMemory::pre_turn`] | logs the turn (accepted, not indexed) while fetching the pack | +//! | after the reply | [`AgentMemory::post_turn`] | logs the reply; may return a belief build | +//! | prompt truncated | [`AgentMemory::recall_for_compaction`] | an answered summary of the thread, plus related memory | +//! | any time | [`AgentMemory::recall`] | a pre-turn pack without logging | +//! | off the turn | [`AgentMemory::run_background`] | the job | +//! +//! The hot path — `pre_turn` and `post_turn` — never waits for indexing and +//! never runs a model: writes use [`tinymemory_api::WaitFor::Accepted`] and the pack is +//! ranked retrieval ([`SectionQuery::Fetch`]). Logging runs concurrently +//! with the read, and the pack never contains the turn being logged or the +//! part of the thread still in the prompt. +//! +//! A pack's sections, highest priority first (budget trimming takes from the +//! last): **Learnings**, **Brain**, **This agent's history**, and **Team +//! conversations** (other agents' turns). An item appears once, in the first +//! section that found it. +//! +//! Each turn is stored as its own one-turn conversation item carrying the +//! thread id, the turn's index, the agent id and the `conversation` source, +//! so a retried call with the same input is a replay, not a duplicate. +//! +//! # Example +//! +//! ``` +//! use std::sync::Arc; +//! use tinymemory_api::conformance::ReferenceEngine; +//! use tinymemory_tools::{ +//! AgentMemory, Brain, BrainDocument, BrainSource, MemoryLayout, PostTurn, PreTurn, +//! }; +//! +//! # let runtime = tokio::runtime::Builder::new_current_thread().build()?; +//! # runtime.block_on(async { +//! let engine = Arc::new(ReferenceEngine::new()); +//! let layout = MemoryLayout::default(); +//! Brain::new(engine.clone(), layout.clone()) +//! .ingest(BrainDocument::new(BrainSource::Markdown, "Refunds take five business days.")) +//! .await?; +//! +//! let memory = AgentMemory::new(engine, layout, "support-01")?; +//! let turn = memory +//! .pre_turn(PreTurn::new("thread-1", 0, "How long do refunds take?")) +//! .await?; +//! assert!(turn.pack.markdown.contains("Refunds take five business days.")); +//! assert!(turn.logged.is_some()); +//! +//! let report = memory +//! .post_turn(PostTurn::new("thread-1", 1, "Five business days.")) +//! .await?; +//! assert!(!report.receipt.replayed); +//! # Ok::<(), tinymemory_api::Error>(()) +//! # })?; +//! # Ok::<(), Box>(()) +//! ``` + +mod types; + +use std::sync::Arc; + +use futures::future::join; +use tinymemory_api::{ + ConsolidateRequest, Error, ItemKind, MemoryEngine, MemoryMeta, MetaFilter, Namespace, Reach, + Result, Role, SourceKind, SourceRef, StoreItem, Turn, TurnRange, WriteOptions, +}; + +use crate::background::{BackgroundJob, BackgroundRunner, JobReport}; +use crate::brain::Brain; +use crate::layout::MemoryLayout; +use crate::recall::{ + ContextPack, HolisticRecall, ScopeSection, SectionQuery, ThreadWindow, holistic_recall, +}; +use crate::tools::MemoryTools; + +pub use types::{ + Compaction, DEFAULT_TURN_BUDGET_TOKENS, PostTurn, PostTurnReport, PreTurn, RecallPolicy, + SessionStart, TurnContext, +}; + +/// Heading of the learnings section. +pub const LEARNINGS_HEADING: &str = "Learnings"; +/// Heading of the brain section. +pub const BRAIN_HEADING: &str = "Brain"; +/// Heading of this agent's own past conversations. +pub const HISTORY_HEADING: &str = "This agent's history"; +/// Heading of the other agents' conversations. +pub const TEAM_HEADING: &str = "Team conversations"; +/// Heading of a resumed thread's own turns. +pub const THREAD_HEADING: &str = "Earlier in this thread"; +/// Heading of a compaction's summary. +pub const SUMMARY_HEADING: &str = "Earlier in this conversation"; + +/// Title of every pack. +const PACK_TITLE: &str = "Memory"; + +/// Longest the gist of dropped turns used as a query may be, in characters. +const MAX_GIST_CHARS: usize = 600; + +/// The fewest characters of each dropped turn the gist keeps. +const MIN_GIST_SHARE: usize = 40; + +/// The most turns a compaction's summary reads. +const MAX_SUMMARY_TURNS: usize = 24; + +/// The question a compaction's summary answers. +const SUMMARY_QUESTION: &str = "What was discussed, decided and left open earlier in this \ + conversation?"; + +/// Guidance for a compaction's summary. +const SUMMARY_INSTRUCTIONS: &str = "Summarise briefly as markdown bullets: facts the user \ + gave, decisions made, and open questions. State only what the stored turns support."; + +/// One agent's memory: its layout, its node, its engine and its policy. +/// Cheap to clone. +#[derive(Clone)] +pub struct AgentMemory { + engine: Arc, + layout: MemoryLayout, + agent_id: String, + node: Namespace, + policy: RecallPolicy, +} + +impl std::fmt::Debug for AgentMemory { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("AgentMemory") + .field("engine", &self.engine.descriptor().id) + .field("agent_id", &self.agent_id) + .field("node", &self.node.to_string()) + .field("policy", &self.policy) + .finish() + } +} + +impl AgentMemory { + /// The memory of `agent_id` in `layout` on `engine`, with the default + /// [`RecallPolicy`]. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a blank agent id. + pub fn new( + engine: Arc, + layout: MemoryLayout, + agent_id: &str, + ) -> Result { + let agent_id = agent_id.trim(); + if agent_id.is_empty() { + return Err(Error::InvalidRequest( + "an agent's memory needs an agent id".to_string(), + )); + } + Ok(Self { + node: layout.conversations(agent_id)?, + engine, + layout, + agent_id: agent_id.to_string(), + policy: RecallPolicy::default(), + }) + } + + /// The same memory under `policy`. + #[must_use] + pub fn with_policy(mut self, policy: RecallPolicy) -> Self { + self.policy = policy; + self + } + + /// The agent's id. + #[must_use] + pub fn agent_id(&self) -> &str { + &self.agent_id + } + + /// The node the agent's turns are stored at. + #[must_use] + pub fn namespace(&self) -> &Namespace { + &self.node + } + + /// The layout. + #[must_use] + pub fn layout(&self) -> &MemoryLayout { + &self.layout + } + + /// The policy. + #[must_use] + pub fn policy(&self) -> &RecallPolicy { + &self.policy + } + + /// The engine. + #[must_use] + pub fn engine(&self) -> &Arc { + &self.engine + } + + /// The layout's brain, on the same engine. + #[must_use] + pub fn brain(&self) -> Brain { + Brain::new(self.engine.clone(), self.layout.clone()) + } + + /// A runner for the jobs this memory hands back. + #[must_use] + pub fn background(&self) -> BackgroundRunner { + BackgroundRunner::new(self.engine.clone(), self.layout.clone()) + } + + /// The model-facing memory tools for this agent: writes land at its node, + /// reads see its node and the shared root ([`Reach::of`]). + #[must_use] + pub fn tools(&self) -> MemoryTools { + MemoryTools::new(self.engine.clone()).placed_at(self.node.clone()) + } + + /// Context for a session starting or resuming: a resumed thread's own + /// turns first, then the standard sections, ranked for `focus` or newest + /// first without one. Reads only. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a blank thread id or a policy with no + /// budget. Engine failures leave sections out instead (see + /// [`crate::recall`]). + pub async fn start_session(&self, start: SessionStart) -> Result { + let mut sections = Vec::new(); + if let Some(thread_id) = &start.thread_id { + let thread_id = non_blank(thread_id, "thread id")?; + sections.push(ScopeSection::latest( + THREAD_HEADING, + MetaFilter { + thread_id: Some(thread_id.to_string()), + ..self.layout.conversations_filter(Some(&self.agent_id)) + }, + self.policy.history_limit.max(1), + )); + } + sections.extend(self.standard_sections()); + self.read(start.focus, sections).await + } + + /// Logs the user's turn and, concurrently, recalls the context to inject + /// before the model runs. + /// + /// Logging waits only for the engine to accept the turn. A failed log + /// does not fail the call: the pack is still returned, with + /// [`TurnContext::log_error`] set. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a blank thread id or text. Engine + /// failures never fail the call. + pub async fn pre_turn(&self, turn: PreTurn) -> Result { + let thread_id = non_blank(&turn.thread_id, "thread id")?; + let text = non_blank(&turn.user_text, "user text")?; + let item = self.turn_item( + thread_id, + turn.turn_index, + Turn { + at: turn.at, + ..Turn::new(Role::User, text) + }, + ); + let id = tinymemory_api::ItemId::new(item.fingerprint()); + let window = ThreadWindow { + thread_id: thread_id.to_string(), + from_turn: turn.in_prompt_from, + }; + let mut request = self.request(Some(text.to_string()), self.standard_sections()); + request.exclude_ids = vec![id]; + request.exclude_thread = Some(window); + let (logged, pack) = join( + self.engine.store_with(item, WriteOptions::accepted()), + holistic_recall(self.engine.as_ref(), &request), + ) + .await; + let pack = pack?; + let (logged, log_error) = match logged { + Ok(receipt) => (Some(receipt), None), + Err(error) => { + log::warn!( + "[lifecycle] user turn not logged agent={} thread={thread_id} error={error}", + self.agent_id + ); + (None, Some(error.to_string())) + } + }; + Ok(TurnContext { + pack, + logged, + log_error, + }) + } + + /// Logs the assistant's reply, and returns the belief build the policy + /// asks for at this turn, if any. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a blank thread id or text, and the + /// engine's failure to accept the turn. + pub async fn post_turn(&self, turn: PostTurn) -> Result { + let thread_id = non_blank(&turn.thread_id, "thread id")?; + let text = non_blank(&turn.assistant_text, "assistant text")?; + let item = self.turn_item( + thread_id, + turn.turn_index, + Turn { + at: turn.at, + tool_calls: turn.tool_calls.clone(), + ..Turn::new(Role::Assistant, text) + }, + ); + let receipt = self + .engine + .store_with(item, WriteOptions::accepted()) + .await?; + let due = self + .policy + .build_beliefs_every + .is_some_and(|every| every > 0 && (turn.turn_index + 1).is_multiple_of(every)); + let jobs = if due { + vec![self.history_build()] + } else { + Vec::new() + }; + Ok(PostTurnReport { receipt, jobs }) + } + + /// Context to carry across a compaction: an answered summary of the + /// thread so far (falling back to its most relevant turns when the engine + /// cannot answer), then the standard sections, ranked for `focus` or the + /// dropped turns' gist. + /// + /// Runs a model on engines that answer with one; call it off the hot + /// path. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a blank thread id. Engine failures leave + /// sections out instead. + pub async fn recall_for_compaction(&self, compaction: Compaction) -> Result { + let thread_id = non_blank(&compaction.thread_id, "thread id")?; + let gist = gist(&compaction.dropped); + let query = compaction + .focus + .filter(|focus| !focus.trim().is_empty()) + .or(gist); + let summary = ScopeSection { + heading: SUMMARY_HEADING.to_string(), + filter: MetaFilter { + thread_id: Some(thread_id.to_string()), + ..self.layout.conversations_filter(Some(&self.agent_id)) + }, + // Every dropped turn is a candidate, not only the policy's few. + limit: self + .policy + .history_limit + .max(compaction.dropped.len()) + .clamp(1, MAX_SUMMARY_TURNS), + query: SectionQuery::Answer { + question: SUMMARY_QUESTION.to_string(), + instructions: Some(SUMMARY_INSTRUCTIONS.to_string()), + fallback_to_fetch: true, + }, + }; + let mut sections = vec![summary]; + sections.extend(self.standard_sections()); + self.read(query, sections).await + } + + /// A pre-turn pack for `query`, without logging anything: for a tool, a + /// sub-agent, or a mid-turn refresh. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a policy with no budget. + pub async fn recall(&self, query: &str) -> Result { + let query = (!query.trim().is_empty()).then(|| query.to_string()); + self.read(query, self.standard_sections()).await + } + + /// Runs one job this memory (or its brain) handed back. + /// + /// # Errors + /// + /// As [`BackgroundRunner::run`]. + pub async fn run_background(&self, job: BackgroundJob) -> Result { + self.background().run(job).await + } + + /// A belief build of this agent's conversations. + #[must_use] + pub fn history_build(&self) -> BackgroundJob { + BackgroundJob::BuildBeliefs { + request: ConsolidateRequest::new(Reach::exact(self.node.clone())) + .kinds([ItemKind::Conversation]), + } + } + + /// Learnings, brain, this agent's history, then the team's, each filled + /// by fetch; a zero limit leaves its section out. + fn standard_sections(&self) -> Vec { + let policy = &self.policy; + [ + ( + LEARNINGS_HEADING, + self.layout.learnings_filter(), + policy.learnings_limit, + ), + ( + BRAIN_HEADING, + self.layout.brain_filter(None), + policy.brain_limit, + ), + ( + HISTORY_HEADING, + self.layout.conversations_filter(Some(&self.agent_id)), + policy.history_limit, + ), + ( + TEAM_HEADING, + self.layout.conversations_filter(None), + policy.team_limit, + ), + ] + .into_iter() + .filter(|(_, _, limit)| *limit > 0) + .map(|(heading, filter, limit)| ScopeSection::fetch(heading, filter, limit)) + .collect() + } + + fn request(&self, query: Option, sections: Vec) -> HolisticRecall { + HolisticRecall { + budget_tokens: self.policy.budget_tokens, + title: PACK_TITLE.to_string(), + ..HolisticRecall::new(query, sections) + } + } + + async fn read( + &self, + query: Option, + sections: Vec, + ) -> Result { + holistic_recall(self.engine.as_ref(), &self.request(query, sections)).await + } + + /// One turn of `thread_id` as a one-turn conversation at this agent's + /// node. + fn turn_item(&self, thread_id: &str, index: u32, turn: Turn) -> StoreItem { + StoreItem::Conversation { + meta: MemoryMeta { + namespace: self.node.clone(), + thread_id: Some(thread_id.to_string()), + turns: Some(TurnRange { + first: index, + last: index, + }), + agent_id: Some(self.agent_id.clone()), + source: SourceRef { + kind: SourceKind::Conversation, + id: Some(thread_id.to_string()), + }, + observed_at: turn.at, + ..MemoryMeta::default() + }, + turns: vec![turn], + } + } +} + +/// `value` trimmed, refused when blank. +fn non_blank<'a>(value: &'a str, what: &str) -> Result<&'a str> { + let value = value.trim(); + if value.is_empty() { + Err(Error::InvalidRequest(format!( + "the {what} must not be blank" + ))) + } else { + Ok(value) + } +} + +/// The dropped turns' text within [`MAX_GIST_CHARS`], oldest first: each +/// turn keeps the start of its text, an equal share of the budget (at least +/// [`MIN_GIST_SHARE`]), so a fact stated early in a long thread still steers +/// the query. `None` when the turns say nothing. +fn gist(turns: &[Turn]) -> Option { + let texts: Vec = turns + .iter() + .map(|turn| turn.text.split_whitespace().collect::>().join(" ")) + .filter(|text| !text.is_empty()) + .collect(); + if texts.is_empty() { + return None; + } + let share = (MAX_GIST_CHARS / texts.len()).max(MIN_GIST_SHARE); + let joined = texts + .iter() + .map(|text| text.chars().take(share).collect::()) + .collect::>() + .join(" "); + Some(joined.chars().take(MAX_GIST_CHARS).collect()) +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/lifecycle/mod_tests.rs b/crates/tinymemory-tools/src/lifecycle/mod_tests.rs new file mode 100644 index 00000000..edd337ba --- /dev/null +++ b/crates/tinymemory-tools/src/lifecycle/mod_tests.rs @@ -0,0 +1,409 @@ +//! The agent lifecycle against the reference engine and a write-broken one. + +use async_trait::async_trait; +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_api::{ + EngineDescriptor, EngineHealth, FetchPage, FetchRequest, ForgetReport, ForgetTarget, + LearningKind, ListPage, ListRequest, RecallAnswer, RecallRequest, StoreReceipt, ToolCallRef, +}; + +use super::*; +use crate::brain::BrainDocument; +use crate::layout::BrainSource; + +fn memory(engine: &Arc, agent: &str) -> AgentMemory { + AgentMemory::new(engine.clone(), MemoryLayout::default(), agent).unwrap() +} + +async fn with_brain() -> Arc { + let engine = Arc::new(ReferenceEngine::new()); + let brain = Brain::new(engine.clone(), MemoryLayout::default()); + brain + .ingest(BrainDocument::new( + BrainSource::Pdf, + "Refunds take five business days.", + )) + .await + .unwrap(); + engine + .store(StoreItem::learning( + "Customers want refund updates by email", + LearningKind::Fact, + 0.9, + MemoryMeta::default(), + )) + .await + .unwrap(); + engine +} + +#[tokio::test] +async fn pre_turn_logs_the_turn_and_recalls_without_it() { + let engine = with_brain().await; + let support = memory(&engine, "support-01"); + let context = support + .pre_turn(PreTurn::new("t1", 0, "how long do refunds take")) + .await + .unwrap(); + let md = &context.pack.markdown; + assert!(md.starts_with("# Memory\n"), "{md}"); + assert!(md.contains("## Learnings\n\n- Customers want refund updates by email")); + assert!(md.contains("## Brain\n\n- Refunds take five business days.")); + assert!( + !md.contains("how long do refunds take"), + "the live turn is left out" + ); + let receipt = context.logged.unwrap(); + assert!(context.log_error.is_none()); + + let listed = engine + .list(ListRequest::new( + MetaFilter::kinds([ItemKind::Conversation]), + 10, + )) + .await + .unwrap(); + let logged = &listed.items[0]; + assert_eq!(logged.id, receipt.id); + assert_eq!(logged.meta.namespace, Namespace::agent("support-01")); + assert_eq!(logged.meta.agent_id.as_deref(), Some("support-01")); + assert_eq!(logged.meta.thread_id.as_deref(), Some("t1")); + assert_eq!(logged.meta.turns, Some(TurnRange { first: 0, last: 0 })); + assert_eq!(logged.meta.source.kind, SourceKind::Conversation); +} + +#[tokio::test] +async fn the_thread_in_the_prompt_is_left_out_until_it_is_compacted_away() { + let engine = with_brain().await; + let support = memory(&engine, "support-01"); + support + .pre_turn(PreTurn::new( + "t1", + 0, + "my order number is 4417 for the refund", + )) + .await + .unwrap(); + support + .post_turn(PostTurn::new( + "t1", + 1, + "Thanks, refund for order 4417 noted.", + )) + .await + .unwrap(); + + let in_prompt = support + .pre_turn(PreTurn::new("t1", 2, "what was my refund order number")) + .await + .unwrap(); + assert!(!in_prompt.pack.markdown.contains("4417")); + + let compacted = support + .pre_turn(PreTurn { + in_prompt_from: 2, + ..PreTurn::new("t1", 3, "what was my refund order number again") + }) + .await + .unwrap(); + assert!( + compacted.pack.markdown.contains("## This agent's history"), + "{}", + compacted.pack.markdown + ); + assert!(compacted.pack.markdown.contains("4417")); +} + +#[tokio::test] +async fn other_agents_turns_appear_once_under_the_team() { + let engine = with_brain().await; + let coder = memory(&engine, "coder-42"); + let support = memory(&engine, "support-01"); + coder + .pre_turn(PreTurn::new("c1", 0, "the refund service deploy failed")) + .await + .unwrap(); + support + .pre_turn(PreTurn::new("s1", 0, "refund delayed for a customer")) + .await + .unwrap(); + + let pack = support.recall("refund").await.unwrap(); + let md = &pack.markdown; + let history = md.find("## This agent's history").unwrap(); + let team = md.find("## Team conversations").unwrap(); + assert!(md[history..team].contains("refund delayed")); + assert!(md[team..].contains("deploy failed")); + assert_eq!(md.matches("refund delayed").count(), 1, "shown once: {md}"); +} + +#[tokio::test] +async fn post_turn_asks_for_a_belief_build_on_the_policy_s_cadence() { + let engine = Arc::new(ReferenceEngine::new()); + let support = memory(&engine, "support-01").with_policy(RecallPolicy { + build_beliefs_every: Some(2), + ..RecallPolicy::default() + }); + let first = support + .post_turn(PostTurn { + tool_calls: vec![ToolCallRef { + name: "lookup_order".into(), + id: Some("call-1".into()), + }], + ..PostTurn::new("t1", 0, "Looking that up.") + }) + .await + .unwrap(); + assert!(first.jobs.is_empty()); + let second = support + .post_turn(PostTurn::new("t1", 1, "Found it.")) + .await + .unwrap(); + assert_eq!(second.jobs, [support.history_build()]); + + let listed = engine + .list(ListRequest::new(MetaFilter::default(), 10)) + .await + .unwrap(); + assert!( + listed + .items + .iter() + .any(|hit| hit.text.contains("lookup_order")) + ); + + let never = memory(&engine, "quiet").with_policy(RecallPolicy { + build_beliefs_every: None, + ..RecallPolicy::default() + }); + for index in 0..4 { + let report = never + .post_turn(PostTurn::new("t2", index, format!("reply {index}"))) + .await + .unwrap(); + assert!(report.jobs.is_empty()); + } +} + +#[tokio::test] +async fn a_retried_turn_is_a_replay() { + let engine = Arc::new(ReferenceEngine::new()); + let support = memory(&engine, "support-01"); + let first = support + .post_turn(PostTurn::new("t1", 1, "Five days.")) + .await + .unwrap(); + let again = support + .post_turn(PostTurn::new("t1", 1, "Five days.")) + .await + .unwrap(); + assert_eq!(first.receipt.id, again.receipt.id); + assert!(again.receipt.replayed); +} + +#[tokio::test] +async fn a_belief_build_turns_history_into_learnings() { + let engine = Arc::new(ReferenceEngine::new()); + let support = memory(&engine, "support-01"); + support + .pre_turn(PreTurn::new( + "t1", + 0, + "I prefer refunds to my original card.", + )) + .await + .unwrap(); + let report = support + .run_background(support.history_build()) + .await + .unwrap(); + assert_eq!(report.outcome, crate::background::JobOutcome::Done); + let pack = support.recall("refunds card").await.unwrap(); + let learnings = pack.markdown.find("## Learnings").unwrap(); + assert!(pack.markdown[learnings..].contains("I prefer refunds to my original card.")); +} + +#[tokio::test] +async fn start_session_resumes_the_thread_first() { + let engine = with_brain().await; + let support = memory(&engine, "support-01"); + support + .pre_turn(PreTurn::new("t1", 0, "order 4417 refund")) + .await + .unwrap(); + support + .post_turn(PostTurn::new("t1", 1, "Refund for 4417 is on its way.")) + .await + .unwrap(); + + let resumed = support + .start_session(SessionStart { + thread_id: Some("t1".into()), + focus: None, + }) + .await + .unwrap(); + let md = &resumed.markdown; + let thread = md.find("## Earlier in this thread").unwrap(); + assert!(thread < md.find("## Learnings").unwrap()); + assert!(md[thread..].starts_with("## Earlier in this thread\n\n- assistant: Refund for 4417")); + + let fresh = support + .start_session(SessionStart::default()) + .await + .unwrap(); + assert!(!fresh.markdown.contains("## Earlier in this thread")); + assert!(fresh.markdown.contains("## Brain")); +} + +#[tokio::test] +async fn compaction_summarises_the_thread_then_adds_related_memory() { + let engine = with_brain().await; + let support = memory(&engine, "support-01"); + support + .pre_turn(PreTurn::new("t1", 0, "my refund for order 4417 is late")) + .await + .unwrap(); + let pack = support + .recall_for_compaction(Compaction { + thread_id: "t1".into(), + dropped: vec![Turn::new(Role::User, "my refund for order 4417 is late")], + focus: None, + }) + .await + .unwrap(); + let md = &pack.markdown; + let summary = md.find("## Earlier in this conversation").unwrap(); + assert!(md[summary..].contains("4417")); + assert!(pack.sections[0].answer.is_some()); + assert!(md.contains("## Brain")); +} + +#[tokio::test] +async fn blank_inputs_are_refused() { + let engine = Arc::new(ReferenceEngine::new()); + assert!(AgentMemory::new(engine.clone(), MemoryLayout::default(), " ").is_err()); + let support = memory(&engine, "support-01"); + let refusals = [ + support.pre_turn(PreTurn::new(" ", 0, "hi")).await.err(), + support.pre_turn(PreTurn::new("t", 0, " ")).await.err(), + support.post_turn(PostTurn::new("t", 0, "")).await.err(), + support + .start_session(SessionStart { + thread_id: Some(String::new()), + focus: None, + }) + .await + .err(), + support + .recall_for_compaction(Compaction { + thread_id: " ".into(), + dropped: Vec::new(), + focus: None, + }) + .await + .err(), + ]; + for refusal in refusals { + assert!( + matches!(refusal, Some(Error::InvalidRequest(_))), + "{refusal:?}" + ); + } + assert!(engine.is_empty()); +} + +/// The reference engine refusing every write. +struct ReadOnly(ReferenceEngine); + +#[async_trait] +impl MemoryEngine for ReadOnly { + fn descriptor(&self) -> &EngineDescriptor { + self.0.descriptor() + } + async fn health(&self) -> EngineHealth { + EngineHealth::Ok + } + async fn recall(&self, req: RecallRequest) -> Result { + self.0.recall(req).await + } + async fn fetch(&self, req: FetchRequest) -> Result { + self.0.fetch(req).await + } + async fn store(&self, _item: StoreItem) -> Result { + Err(Error::Unavailable("writes are down".into())) + } + async fn forget(&self, target: ForgetTarget) -> Result { + self.0.forget(target).await + } + async fn list(&self, req: ListRequest) -> Result { + self.0.list(req).await + } +} + +#[tokio::test] +async fn a_failed_log_still_returns_the_pack() { + let inner = ReferenceEngine::new(); + inner + .store(StoreItem::document( + "Refunds take five days.", + MemoryMeta::default(), + )) + .await + .unwrap(); + let support = + AgentMemory::new(Arc::new(ReadOnly(inner)), MemoryLayout::default(), "s").unwrap(); + let context = support + .pre_turn(PreTurn::new("t1", 0, "refunds")) + .await + .unwrap(); + assert!(context.logged.is_none()); + assert!(context.log_error.unwrap().contains("writes are down")); + assert!(context.pack.markdown.contains("Refunds take five days.")); + + let error = support + .post_turn(PostTurn::new("t1", 1, "Five days.")) + .await + .unwrap_err(); + assert!(matches!(error, Error::Unavailable(_))); +} + +#[test] +fn the_gist_samples_every_dropped_turn() { + assert_eq!(gist(&[]), None); + assert_eq!(gist(&[Turn::new(Role::User, " ")]), None); + let long = vec![ + Turn::new(Role::User, "The offsite is in Porto."), + Turn::new(Role::Assistant, "a".repeat(MAX_GIST_CHARS)), + Turn::new(Role::User, "the end"), + ]; + let gist = gist(&long).unwrap(); + assert!(gist.chars().count() <= MAX_GIST_CHARS); + assert!(gist.starts_with("The offsite is in Porto."), "{gist}"); + assert!(gist.ends_with("the end"), "{gist}"); + + let many: Vec = (0..100) + .map(|n| Turn::new(Role::User, format!("turn {n} {}", "x".repeat(80)))) + .collect(); + let gist = super::gist(&many).unwrap(); + assert_eq!(gist.chars().count(), MAX_GIST_CHARS); + assert!(gist.starts_with("turn 0 "), "the oldest turns come first"); +} + +#[test] +fn a_zero_limit_leaves_its_section_out() { + let engine = Arc::new(ReferenceEngine::new()); + let support = memory(&engine, "s").with_policy(RecallPolicy { + team_limit: 0, + ..RecallPolicy::default() + }); + let headings: Vec = support + .standard_sections() + .into_iter() + .map(|section| section.heading) + .collect(); + assert_eq!( + headings, + [LEARNINGS_HEADING, BRAIN_HEADING, HISTORY_HEADING] + ); +} diff --git a/crates/tinymemory-tools/src/lifecycle/types.rs b/crates/tinymemory-tools/src/lifecycle/types.rs new file mode 100644 index 00000000..a39e2b9e --- /dev/null +++ b/crates/tinymemory-tools/src/lifecycle/types.rs @@ -0,0 +1,170 @@ +//! The inputs and outputs of each lifecycle step, and the recall policy. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use tinymemory_api::{StoreReceipt, ToolCallRef, Turn}; + +use crate::background::BackgroundJob; +use crate::recall::ContextPack; + +/// Default token budget of a turn's context pack. +pub const DEFAULT_TURN_BUDGET_TOKENS: usize = 1_200; + +/// How an [`crate::AgentMemory`] fills its packs and when it asks for +/// belief builds. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(default)] +pub struct RecallPolicy { + /// The most tokens a pack's block may take. + pub budget_tokens: usize, + /// The most learnings a pack shows. + pub learnings_limit: usize, + /// The most brain documents a pack shows. + pub brain_limit: usize, + /// The most of this agent's past turns a pack shows. + pub history_limit: usize, + /// The most other agents' turns a pack shows; `0` leaves the team's + /// conversations out. + pub team_limit: usize, + /// Ask for a belief build of this agent's conversations after every this + /// many turns (by `turn_index + 1`); `None` never asks. + pub build_beliefs_every: Option, +} + +impl Default for RecallPolicy { + fn default() -> Self { + Self { + budget_tokens: DEFAULT_TURN_BUDGET_TOKENS, + learnings_limit: 8, + brain_limit: 6, + history_limit: 6, + team_limit: 3, + build_beliefs_every: Some(10), + } + } +} + +/// A session starting or resuming. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SessionStart { + /// The thread being resumed, if any: its own past turns come first in + /// this agent's history. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub thread_id: Option, + /// What the session is about, when known: ranks every section. Without + /// one, each section shows its newest items. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub focus: Option, +} + +/// A user turn about to be answered. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct PreTurn { + /// The conversation thread. + pub thread_id: String, + /// This turn's position in the thread, from `0`. + pub turn_index: u32, + /// What the user said. + pub user_text: String, + /// The first turn of this thread still in the host's prompt; turns from + /// here on are left out of the pack, since the model already sees them. + /// `0` (the default) leaves the whole thread out. + #[serde(default)] + pub in_prompt_from: u32, + /// When the user spoke, if the host knows: orders history newest first, + /// and leads the turn's bullet in later packs, so a reader can tell a + /// corrected value from its correction. Pass the message's own time (not + /// the time of the call), so a retried turn stays a replay. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub at: Option>, +} + +impl PreTurn { + /// A turn of `thread_id` at `turn_index` saying `user_text`, with the + /// whole thread in the prompt. + #[must_use] + pub fn new( + thread_id: impl Into, + turn_index: u32, + user_text: impl Into, + ) -> Self { + Self { + thread_id: thread_id.into(), + turn_index, + user_text: user_text.into(), + in_prompt_from: 0, + at: None, + } + } +} + +/// What [`crate::AgentMemory::pre_turn`] produced. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct TurnContext { + /// The context to inject before the model runs. + pub pack: ContextPack, + /// The logged user turn's receipt; `None` when logging failed. + #[serde(skip_serializing_if = "Option::is_none")] + pub logged: Option, + /// Why logging failed, when it did. The pack is still good. + #[serde(skip_serializing_if = "Option::is_none")] + pub log_error: Option, +} + +/// A reply the model just gave. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct PostTurn { + /// The conversation thread. + pub thread_id: String, + /// This reply's position in the thread. + pub turn_index: u32, + /// What the assistant said. + pub assistant_text: String, + /// The tool calls the reply made. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub tool_calls: Vec, + /// When the reply was given, if the host knows. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub at: Option>, +} + +impl PostTurn { + /// A reply of `thread_id` at `turn_index` saying `assistant_text`, with + /// no tool calls. + #[must_use] + pub fn new( + thread_id: impl Into, + turn_index: u32, + assistant_text: impl Into, + ) -> Self { + Self { + thread_id: thread_id.into(), + turn_index, + assistant_text: assistant_text.into(), + tool_calls: Vec::new(), + at: None, + } + } +} + +/// What [`crate::AgentMemory::post_turn`] did. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct PostTurnReport { + /// The logged reply's receipt. + pub receipt: StoreReceipt, + /// Work to run off the turn (a belief build, per the policy). + pub jobs: Vec, +} + +/// Turns about to leave the host's prompt. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Compaction { + /// The conversation thread. + pub thread_id: String, + /// The turns being dropped, oldest first. They are already stored; they + /// steer what is recalled. + pub dropped: Vec, + /// What the conversation is about now, when known. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub focus: Option, +} diff --git a/crates/tinymemory-tools/src/recall/gather.rs b/crates/tinymemory-tools/src/recall/gather.rs new file mode 100644 index 00000000..033c3cc1 --- /dev/null +++ b/crates/tinymemory-tools/src/recall/gather.rs @@ -0,0 +1,453 @@ +//! Filling one section from the engine, and turning what it found into a +//! renderable section. +//! +//! Every section runs on its own and none can fail the pack: an engine error +//! or an empty result becomes a [`SkippedSection`], logged and reported. +//! [`section`] reads; [`settle`] then applies the request's exclusions and +//! the items earlier sections already list, in section order, so an item is +//! listed once — in its highest-priority section. +//! +//! **Beliefs.** To the reader a belief the engine built is a learning like +//! any other, so a pack with a learnings section (a fetched or latest +//! section that admits learnings) gathers the engine's beliefs into it: +//! +//! - every ranked section's fetch asks for beliefs too +//! ([`FetchRequest::beliefs`]), so they come from the reads the pack makes +//! anyway (on CortexDB, the same recall packs) — the brain, history and +//! team sections between them read every scope a belief can be built in; +//! - a learnings section with no query (a cold session start) lists them +//! instead ([`MemoryEngine::beliefs`] without a query), alongside its own +//! read; +//! - once every section is read, [`fold_beliefs`] merges what they returned, +//! each belief once, and interleaves it with each learnings section's +//! stored learnings rank by rank, stored learnings first, keeping only the +//! beliefs the section's filter admits. +//! +//! A belief read that fails leaves the learnings to the stored ones. +//! Answered sections ask for none: the engine's answer draws on its beliefs +//! itself. + +use std::collections::HashSet; + +use futures::future::join; +use tinymemory_api::{ + BeliefsRequest, FetchMode, FetchRequest, Hit, ItemId, ItemKind, ListRequest, MemoryEngine, + MetaFilter, Namespace, Reach, RecallRequest, +}; + +use super::render::{Body, Line, Section, shorten, single_line}; +use super::types::{HolisticRecall, ScopeSection, SectionHits, SectionQuery, SkippedSection}; + +/// Longest a single bullet may be, in characters, before it is shortened: one +/// long document must not crowd out a section. +const MAX_LINE_CHARS: usize = 600; + +/// Page size of a [`SectionQuery::Latest`] listing. +const LATEST_PAGE: usize = 100; + +/// Most listing pages read before ranking; a ceiling, not a target. +const LATEST_MAX_PAGES: usize = 50; + +/// What one section read. +pub(super) enum Gathered { + /// An answer and its citations. + Answered(Section, SectionHits), + /// Ranked or latest hits, before exclusions, and the beliefs the read + /// returned beside them. + Hits { + /// The section's own hits. + hits: Vec, + /// Beliefs for the pack's learnings, whichever section read them. + beliefs: Vec, + }, + /// Nothing, and why. + Skipped(SkippedSection), +} + +/// What one section contributes once settled. +pub(super) enum Settled { + /// Something to render, and what it was drawn from. + Filled(Section, SectionHits), + /// Nothing, and why. + Skipped(SkippedSection), +} + +/// Fills `section` for `request`, asking a ranked read for up to +/// `beliefs` beliefs as well. +pub(super) async fn section( + engine: &dyn MemoryEngine, + request: &HolisticRecall, + section: &ScopeSection, + beliefs: usize, +) -> Gathered { + let want = wanted(request, section); + let outcome = match §ion.query { + SectionQuery::Answer { + question, + instructions, + fallback_to_fetch, + } => match answer(engine, section, question, instructions.clone()).await { + Ok(Some(filled)) => return filled, + Ok(None) => Ok((Vec::new(), Vec::new())), + Err(error) if *fallback_to_fetch => { + log::debug!( + "[recall] answer failed, fetching instead heading={:?} error={error}", + section.heading + ); + fetch(engine, §ion.filter, question, want, 0).await + } + Err(error) => Err(error), + }, + SectionQuery::Fetch { query } => { + match query + .as_deref() + .or(request.query.as_deref()) + .filter(|query| !query.trim().is_empty()) + { + Some(query) => fetch(engine, §ion.filter, query, want, beliefs).await, + None => with_listed_beliefs(engine, section, want).await, + } + } + SectionQuery::Latest => with_listed_beliefs(engine, section, want).await, + }; + match outcome { + Ok((hits, beliefs)) => Gathered::Hits { hits, beliefs }, + Err(error) => { + log::warn!( + "[recall] section skipped heading={:?} error={error}", + section.heading + ); + skipped(section, error.to_string()) + } + } +} + +/// How many hits to ask for so that `section.limit` survive the request's +/// exclusions and the items earlier sections already show: one more per +/// excluded id and per item an earlier section may hold, and double when a +/// whole thread window may be left out. +fn wanted(request: &HolisticRecall, section: &ScopeSection) -> usize { + let window = if request.exclude_thread.is_some() { + section.limit + } else { + 0 + }; + let earlier: usize = request + .sections + .iter() + .take_while(|other| !std::ptr::eq(*other, section)) + .map(|other| other.limit) + .sum(); + section.limit + request.exclude_ids.len() + window + earlier +} + +fn skipped(section: &ScopeSection, reason: String) -> Gathered { + Gathered::Skipped(skipped_section(section, reason)) +} + +fn skipped_section(section: &ScopeSection, reason: String) -> SkippedSection { + SkippedSection { + heading: section.heading.clone(), + reason, + } +} + +/// One recall; `None` when it cited nothing or answered blank. +async fn answer( + engine: &dyn MemoryEngine, + section: &ScopeSection, + question: &str, + instructions: Option, +) -> tinymemory_api::Result> { + let answer = engine + .recall(RecallRequest { + question: question.to_string(), + filter: section.filter.clone(), + limit: section.limit, + instructions, + }) + .await?; + if answer.citations.is_empty() || answer.answer.trim().is_empty() { + return Ok(None); + } + let text = answer.answer.trim().to_string(); + let hits: Vec = answer + .citations + .into_iter() + .map(|citation| Hit { + id: citation.id, + kind: citation.kind, + text: citation.snippet, + meta: citation.meta, + score: citation.score.unwrap_or_default(), + confidence: None, + }) + .collect(); + let refs = hits.iter().map(|hit| hit.id.clone()).collect(); + Ok(Some(Gathered::Answered( + Section { + heading: section.heading.clone(), + body: Body::Prose { + text: text.clone(), + refs, + }, + }, + SectionHits { + heading: section.heading.clone(), + answer: Some(text), + hits, + }, + ))) +} + +/// The fetch mode a section ranks with: hybrid when the engine serves it, +/// else the first it declares. +fn preferred_mode(engine: &dyn MemoryEngine) -> Option { + let modes = &engine.descriptor().fetch_modes; + if modes.contains(&FetchMode::Hybrid) { + Some(FetchMode::Hybrid) + } else { + modes.first().copied() + } +} + +/// Whether `section` is a learnings section: one ranked or latest read +/// that admits learnings, where the pack's beliefs go. +pub(super) fn reads_learnings(section: &ScopeSection) -> bool { + !matches!(section.query, SectionQuery::Answer { .. }) + && section.filter.admits_kind(ItemKind::Learning) +} + +/// How many beliefs each ranked read asks for: what the first learnings +/// section wants, or none when the pack has no learnings section. +pub(super) fn belief_budget(request: &HolisticRecall) -> usize { + request + .sections + .iter() + .find(|section| reads_learnings(section)) + .map_or(0, |section| wanted(request, section)) +} + +/// The newest hits of `section`, and, for a learnings section, the beliefs +/// the engine lists in its reach (read concurrently; a failed listing +/// leaves none). +async fn with_listed_beliefs( + engine: &dyn MemoryEngine, + section: &ScopeSection, + want: usize, +) -> tinymemory_api::Result<(Vec, Vec)> { + if !reads_learnings(section) { + return Ok((latest(engine, §ion.filter, want).await?, Vec::new())); + } + let reach = section + .filter + .reach + .clone() + .unwrap_or_else(|| Reach::subtree(Namespace::ROOT)); + let (hits, beliefs) = join( + latest(engine, §ion.filter, want), + engine.beliefs(BeliefsRequest::new(reach, want)), + ) + .await; + let beliefs = beliefs.unwrap_or_else(|error| { + log::warn!( + "[recall] beliefs unavailable heading={:?} error={error}", + section.heading + ); + Vec::new() + }); + Ok((hits?, beliefs)) +} + +/// Merges the beliefs every section returned (each once, in section order, +/// rank by rank) into each learnings section's hits; see the module docs. +pub(super) fn fold_beliefs(request: &HolisticRecall, gathered: &mut [Gathered]) { + let lists: Vec> = gathered + .iter_mut() + .filter_map(|outcome| match outcome { + Gathered::Hits { beliefs, .. } => Some(std::mem::take(beliefs)), + _ => None, + }) + .collect(); + let longest = lists.iter().map(Vec::len).max().unwrap_or(0); + let mut seen = HashSet::new(); + let mut beliefs = Vec::new(); + for rank in 0..longest { + for list in &lists { + if let Some(belief) = list.get(rank) + && seen.insert(belief.id.clone()) + { + beliefs.push(belief.clone()); + } + } + } + if beliefs.is_empty() { + return; + } + for (section, outcome) in request.sections.iter().zip(gathered.iter_mut()) { + if !reads_learnings(section) { + continue; + } + if let Gathered::Hits { hits, .. } = outcome { + let admitted = beliefs + .iter() + .filter(|belief| section.filter.matches(ItemKind::Learning, &belief.meta)) + .cloned(); + *hits = interleave(std::mem::take(hits), admitted); + } + } +} + +/// `first` and `second` merged rank by rank, `first` leading at each rank. +fn interleave(first: Vec, second: impl IntoIterator) -> Vec { + let mut first = first.into_iter(); + let mut second = second.into_iter(); + let mut out = Vec::new(); + loop { + match (first.next(), second.next()) { + (None, None) => return out, + (a, b) => out.extend(a.into_iter().chain(b)), + } + } +} + +/// One page of ranked hits; an engine that declares no fetch mode is read +/// newest first instead. +async fn fetch( + engine: &dyn MemoryEngine, + filter: &MetaFilter, + query: &str, + limit: usize, + beliefs: usize, +) -> tinymemory_api::Result<(Vec, Vec)> { + let Some(mode) = preferred_mode(engine) else { + return Ok((latest(engine, filter, limit).await?, Vec::new())); + }; + let mut request = FetchRequest::new(query, mode, limit); + request.filter = filter.clone(); + request.beliefs = beliefs; + let page = engine.fetch(request).await?; + Ok((page.hits, page.beliefs)) +} + +/// The newest hits, then the most confident, then the latest turn; ties +/// keep the engine's order. +async fn latest( + engine: &dyn MemoryEngine, + filter: &MetaFilter, + limit: usize, +) -> tinymemory_api::Result> { + let mut all: Vec = Vec::new(); + let mut cursor: Option = None; + for _ in 0..LATEST_MAX_PAGES { + let mut request = ListRequest::new(filter.clone(), LATEST_PAGE); + request.cursor = cursor.take(); + let page = engine.list(request).await?; + all.extend(page.items); + match page.next_cursor { + Some(next) => cursor = Some(next), + None => break, + } + } + all.sort_by(|a, b| { + b.meta + .observed_at + .cmp(&a.meta.observed_at) + .then_with(|| { + b.confidence + .unwrap_or(0.0) + .total_cmp(&a.confidence.unwrap_or(0.0)) + }) + .then_with(|| { + let last = |hit: &Hit| hit.meta.turns.as_ref().map(|turns| turns.last); + last(b).cmp(&last(a)) + }) + }); + all.truncate(limit); + Ok(all) +} + +/// The text a hit's bullet shows: +/// +/// - a titled document as `title: body` rather than its `# title` heading run +/// into the body, and without the body's own copy of that heading when +/// converted markdown repeats it; +/// - a conversation led by when it was said (`[2026-09-15 09:01] user: …`), +/// when the turn carries a time, so a reader can tell a value that was +/// later corrected from the correction. +fn bullet_text(hit: &Hit) -> String { + if hit.kind == ItemKind::Conversation + && let Some(at) = hit.meta.observed_at + { + return format!("[{}] {}", at.format("%Y-%m-%d %H:%M"), hit.text); + } + if hit.kind == ItemKind::Document + && let Some(rest) = hit.text.strip_prefix("# ") + && let Some((title, body)) = rest.split_once("\n\n") + && !title.contains('\n') + { + let title = title.trim(); + let body = body.trim_start(); + let body = body + .strip_prefix("# ") + .and_then(|heading| heading.strip_prefix(title)) + .filter(|after| after.is_empty() || after.starts_with('\n')) + .map_or(body, str::trim_start); + return format!("{title}: {body}"); + } + hit.text.clone() +} + +/// Settles one gathered section: answers pass through untouched (an answer +/// citing an item does not hide its bullet elsewhere); hits lose the +/// request's exclusions and any item an earlier section already lists, are +/// cut to the section's limit, and join `shown`. +pub(super) fn settle( + request: &HolisticRecall, + section: &ScopeSection, + gathered: Gathered, + shown: &mut HashSet, +) -> Settled { + let hits = match gathered { + Gathered::Skipped(reason) => return Settled::Skipped(reason), + Gathered::Answered(rendered, hits) => return Settled::Filled(rendered, hits), + Gathered::Hits { hits, .. } => hits, + }; + let kinds = §ion.filter.kinds; + let hits: Vec = hits + .into_iter() + .filter(|hit| kinds.is_empty() || kinds.contains(&hit.kind)) + .filter(|hit| !request.excludes(hit) && !shown.contains(&hit.id)) + .take(section.limit) + .collect(); + if hits.is_empty() { + return Settled::Skipped(skipped_section(section, "empty".to_string())); + } + shown.extend(hits.iter().map(|hit| hit.id.clone())); + let lines = hits + .iter() + .map(|hit| { + let text = single_line(&bullet_text(hit)); + let text = if text.chars().count() > MAX_LINE_CHARS { + shorten(&text, MAX_LINE_CHARS) + } else { + text + }; + Line { + id: hit.id.clone(), + text, + } + }) + .collect(); + Settled::Filled( + Section { + heading: section.heading.clone(), + body: Body::Lines(lines), + }, + SectionHits { + heading: section.heading.clone(), + answer: None, + hits, + }, + ) +} diff --git a/crates/tinymemory-tools/src/recall/mod.rs b/crates/tinymemory-tools/src/recall/mod.rs new file mode 100644 index 00000000..5a7920ac --- /dev/null +++ b/crates/tinymemory-tools/src/recall/mod.rs @@ -0,0 +1,151 @@ +//! Holistic recall: one read across several scopes, rendered as one +//! budgeted markdown block — the [`ContextPack`] a host injects into a +//! prompt. +//! +//! A [`HolisticRecall`] lists [`ScopeSection`]s, highest priority first. Each +//! names a scope (a [`tinymemory_api::MetaFilter`]: its reach and kinds) and +//! how to fill it ([`SectionQuery`]): +//! +//! - **Fetch** — ranked hits for the pack's query; no model runs, so this is +//! what a live turn uses. +//! - **Answer** — a synthesised answer to a fixed question; an engine may run +//! a model, so this is for session start and compaction. +//! - **Latest** — the newest, most confident items, with no query. +//! +//! The sections are read concurrently. An item is listed once, in the first +//! section that lists it, so overlapping scopes (one agent's history inside +//! the team's) never repeat a bullet; an answer citing an item does not +//! hide it. Then they are rendered under one `#` title, +//! one `##` heading per section that found something. The block fits +//! `budget_tokens` (four characters per token): bullets are trimmed from the +//! last section first, then answers shorten (see `render`). +//! +//! Nothing an engine does fails a pack. A section whose read fails, or finds +//! nothing, is left out and reported in [`ContextPack::skipped`]. The only +//! error is an invalid request. +//! +//! Every lifecycle read is a preset of this: `context.md` +//! ([`crate::context`]) is answered briefs plus the latest learnings with +//! frontmatter, and [`crate::AgentMemory`]'s session start, pre-turn and +//! compaction packs are the layout's scopes filled for their moment. +//! +//! # Example +//! +//! ``` +//! use tinymemory_api::conformance::ReferenceEngine; +//! use tinymemory_api::{ItemKind, LearningKind, MemoryEngine, MemoryMeta, MetaFilter, StoreItem}; +//! use tinymemory_tools::recall::{HolisticRecall, ScopeSection, holistic_recall}; +//! +//! # let runtime = tokio::runtime::Builder::new_current_thread().build()?; +//! # runtime.block_on(async { +//! let engine = ReferenceEngine::new(); +//! engine +//! .store(StoreItem::document("Refunds take five business days.", MemoryMeta::default())) +//! .await?; +//! engine +//! .store(StoreItem::learning("Customers prefer email", LearningKind::Fact, 0.8, MemoryMeta::default())) +//! .await?; +//! +//! let request = HolisticRecall::new( +//! Some("how long do refunds take".into()), +//! vec![ +//! ScopeSection::fetch("Documents", MetaFilter::kinds([ItemKind::Document]), 5), +//! ScopeSection::latest("Learnings", MetaFilter::kinds([ItemKind::Learning]), 5), +//! ], +//! ); +//! let pack = holistic_recall(&engine, &request).await?; +//! assert!(pack.markdown.starts_with("# Memory\n")); +//! assert!(pack.markdown.contains("## Documents\n\n- Refunds take five business days.")); +//! assert!(pack.markdown.contains("## Learnings\n\n- Customers prefer email")); +//! # Ok::<(), tinymemory_api::Error>(()) +//! # })?; +//! # Ok::<(), Box>(()) +//! ``` + +mod gather; +pub(crate) mod render; +mod types; + +use std::collections::HashSet; + +use futures::future::join_all; +use tinymemory_api::{MemoryEngine, Result}; + +use gather::Settled; +pub(crate) use render::Frontmatter; +pub use render::estimate_tokens; +pub use types::{ + ContextPack, DEFAULT_PACK_BUDGET_TOKENS, DEFAULT_PACK_TITLE, HolisticRecall, ScopeSection, + SectionHits, SectionQuery, SkippedSection, ThreadWindow, +}; + +/// Reads every section of `request` from `engine`, concurrently, and renders +/// the pack. +/// +/// # Errors +/// +/// [`tinymemory_api::Error::InvalidRequest`] for an invalid request +/// ([`HolisticRecall::validate`]). Engine failures are not errors: see the +/// module docs. +pub async fn holistic_recall( + engine: &dyn MemoryEngine, + request: &HolisticRecall, +) -> Result { + run(engine, request, None).await +} + +/// [`holistic_recall`], optionally with `context.md` frontmatter. +pub(crate) async fn run( + engine: &dyn MemoryEngine, + request: &HolisticRecall, + frontmatter: Option>, +) -> Result { + request.validate()?; + let beliefs = gather::belief_budget(request); + let mut gathered = join_all( + request + .sections + .iter() + .map(|section| gather::section(engine, request, section, beliefs)), + ) + .await; + gather::fold_beliefs(request, &mut gathered); + let mut sections = Vec::new(); + let mut rendered_from = Vec::new(); + let mut skipped = Vec::new(); + let mut shown = HashSet::new(); + for (section, outcome) in request.sections.iter().zip(gathered) { + match gather::settle(request, section, outcome, &mut shown) { + Settled::Filled(rendered, hits) => { + rendered_from.push(rendered); + sections.push(hits); + } + Settled::Skipped(reason) => skipped.push(reason), + } + } + let rendered = render::render( + rendered_from, + request.budget_tokens, + &request.title, + frontmatter, + ); + let engine_id = engine.descriptor().id; + log::debug!( + "[recall] pack engine={engine_id} sections={} skipped={} tokens={}", + sections.len(), + skipped.len(), + rendered.tokens + ); + Ok(ContextPack { + markdown: rendered.markdown, + tokens: rendered.tokens, + refs: rendered.refs, + sections, + skipped, + engine: engine_id.to_string(), + }) +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/recall/mod_tests.rs b/crates/tinymemory-tools/src/recall/mod_tests.rs new file mode 100644 index 00000000..4da9295f --- /dev/null +++ b/crates/tinymemory-tools/src/recall/mod_tests.rs @@ -0,0 +1,617 @@ +//! Holistic recall against the reference engine and a half-broken one. + +use async_trait::async_trait; +use tinymemory_api::conformance::ReferenceEngine; +use tinymemory_api::{ + EngineDescriptor, EngineHealth, Error, FetchPage, FetchRequest, ForgetReport, ForgetTarget, + ItemKind, LearningKind, ListPage, ListRequest, MemoryMeta, MetaFilter, Namespace, Reach, + RecallAnswer, RecallRequest, Role, StoreItem, StoreReceipt, Turn, TurnRange, +}; + +use super::*; + +fn turn(thread: &str, index: u32, text: &str) -> StoreItem { + StoreItem::Conversation { + turns: vec![Turn::new(Role::User, text)], + meta: MemoryMeta { + thread_id: Some(thread.to_string()), + turns: Some(TurnRange { + first: index, + last: index, + }), + ..MemoryMeta::default() + }, + } +} + +async fn seeded() -> ReferenceEngine { + let engine = ReferenceEngine::new(); + for item in [ + StoreItem::document("Refunds take five business days.", MemoryMeta::default()), + StoreItem::document("Deploys happen on Fridays.", MemoryMeta::default()), + StoreItem::learning( + "Customers prefer refunds by email", + LearningKind::Fact, + 0.9, + MemoryMeta::default(), + ), + turn("t1", 0, "asked about refunds yesterday"), + turn("t2", 4, "refunds question in this very thread"), + ] { + engine.store(item).await.unwrap(); + } + engine +} + +fn docs() -> MetaFilter { + MetaFilter::kinds([ItemKind::Document]) +} + +#[tokio::test] +async fn fetch_sections_rank_for_the_pack_query() { + let engine = seeded().await; + let request = HolisticRecall::new( + Some("refunds".into()), + vec![ScopeSection::fetch("Docs", docs(), 1)], + ); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert_eq!( + pack.markdown, + "# Memory\n\n## Docs\n\n- Refunds take five business days.\n" + ); + assert_eq!(pack.sections.len(), 1); + assert_eq!(pack.sections[0].hits.len(), 1); + assert_eq!(pack.refs.len(), 1); + assert_eq!(pack.engine, "reference"); + assert!(pack.skipped.is_empty()); +} + +#[tokio::test] +async fn a_fetch_section_without_any_query_reads_the_latest() { + let engine = seeded().await; + let request = HolisticRecall::new(None, vec![ScopeSection::fetch("Docs", docs(), 5)]); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert_eq!(pack.sections[0].hits.len(), 2); +} + +#[tokio::test] +async fn a_section_s_own_query_overrides_the_pack_s() { + let engine = seeded().await; + let mut section = ScopeSection::fetch("Docs", docs(), 1); + section.query = SectionQuery::Fetch { + query: Some("deploys fridays".into()), + }; + let pack = holistic_recall( + &engine, + &HolisticRecall::new(Some("refunds".into()), vec![section]), + ) + .await + .unwrap(); + assert!(pack.markdown.contains("Deploys happen on Fridays.")); +} + +#[tokio::test] +async fn answered_sections_are_prose_with_citations() { + let engine = seeded().await; + let request = HolisticRecall::new( + None, + vec![ScopeSection::answer( + "Refunds", + "how long do refunds take", + docs(), + 3, + )], + ); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert!( + pack.markdown.contains("## Refunds\n\nFrom "), + "{}", + pack.markdown + ); + assert!(pack.sections[0].answer.is_some()); + assert!(!pack.refs.is_empty()); +} + +#[tokio::test] +async fn excluded_ids_and_the_live_thread_window_are_left_out() { + let engine = seeded().await; + let conversations = MetaFilter::kinds([ItemKind::Conversation]); + let mut request = HolisticRecall::new( + Some("refunds".into()), + vec![ScopeSection::fetch("History", conversations, 5)], + ); + request.exclude_thread = Some(ThreadWindow { + thread_id: "t2".into(), + from_turn: 3, + }); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert!(pack.markdown.contains("asked about refunds yesterday")); + assert!(!pack.markdown.contains("this very thread")); + + request.exclude_thread = Some(ThreadWindow { + thread_id: "t2".into(), + from_turn: 5, + }); + let older = holistic_recall(&engine, &request).await.unwrap(); + assert!( + older.markdown.contains("this very thread"), + "a turn before the window is no longer in the prompt" + ); + + request.exclude_thread = None; + request.exclude_ids = older.refs.clone(); + let none = holistic_recall(&engine, &request).await.unwrap(); + assert!(none.is_empty()); + assert_eq!(none.skipped[0].reason, "empty"); +} + +/// The reference engine with recall (and optionally fetch) broken. +struct Broken { + inner: ReferenceEngine, + fetch_too: bool, +} + +#[async_trait] +impl tinymemory_api::MemoryEngine for Broken { + fn descriptor(&self) -> &EngineDescriptor { + self.inner.descriptor() + } + async fn health(&self) -> EngineHealth { + EngineHealth::Ok + } + async fn recall(&self, _req: RecallRequest) -> tinymemory_api::Result { + Err(Error::Unavailable("recall is down".into())) + } + async fn fetch(&self, req: FetchRequest) -> tinymemory_api::Result { + if self.fetch_too { + return Err(Error::Unavailable("fetch is down".into())); + } + self.inner.fetch(req).await + } + async fn store(&self, item: StoreItem) -> tinymemory_api::Result { + self.inner.store(item).await + } + async fn forget(&self, target: ForgetTarget) -> tinymemory_api::Result { + self.inner.forget(target).await + } + async fn list(&self, req: ListRequest) -> tinymemory_api::Result { + self.inner.list(req).await + } +} + +#[tokio::test] +async fn a_failed_answer_falls_back_to_fetch_only_when_asked() { + let engine = Broken { + inner: seeded().await, + fetch_too: false, + }; + let mut section = ScopeSection::answer("Refunds", "refunds", docs(), 3); + let plain = holistic_recall(&engine, &HolisticRecall::new(None, vec![section.clone()])) + .await + .unwrap(); + assert!(plain.is_empty()); + assert!(plain.skipped[0].reason.contains("recall is down")); + + if let SectionQuery::Answer { + fallback_to_fetch, .. + } = &mut section.query + { + *fallback_to_fetch = true; + } + let fallen = holistic_recall(&engine, &HolisticRecall::new(None, vec![section])) + .await + .unwrap(); + assert!( + fallen + .markdown + .contains("- Refunds take five business days.") + ); +} + +#[tokio::test] +async fn a_failing_section_never_fails_the_pack() { + let engine = Broken { + inner: seeded().await, + fetch_too: true, + }; + let request = HolisticRecall::new( + Some("refunds".into()), + vec![ + ScopeSection::fetch("Docs", docs(), 3), + ScopeSection::latest("Learnings", MetaFilter::kinds([ItemKind::Learning]), 3), + ], + ); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert_eq!(pack.skipped.len(), 1); + assert_eq!(pack.skipped[0].heading, "Docs"); + assert!(pack.markdown.contains("## Learnings")); +} + +#[tokio::test] +async fn sections_read_only_their_scope() { + let engine = ReferenceEngine::new(); + for (text, namespace) in [ + ("pdf fact about refunds", Namespace::source("pdf")), + ("agent note about refunds", Namespace::agent("a")), + ] { + engine + .store(StoreItem::document( + text, + MemoryMeta { + namespace, + ..MemoryMeta::default() + }, + )) + .await + .unwrap(); + } + let pdf_only = MetaFilter { + reach: Some(Reach::subtree(Namespace::source("pdf"))), + ..docs() + }; + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + Some("refunds".into()), + vec![ScopeSection::fetch("Pdf", pdf_only, 5)], + ), + ) + .await + .unwrap(); + assert!(pack.markdown.contains("pdf fact")); + assert!(!pack.markdown.contains("agent note")); +} + +#[tokio::test] +async fn an_invalid_request_is_refused() { + let engine = ReferenceEngine::new(); + let cases = [ + HolisticRecall { + budget_tokens: 0, + ..HolisticRecall::new(None, Vec::new()) + }, + HolisticRecall { + title: " ".into(), + ..HolisticRecall::new(None, Vec::new()) + }, + HolisticRecall::new(None, vec![ScopeSection::fetch(" ", docs(), 1)]), + HolisticRecall::new(None, vec![ScopeSection::fetch("Docs", docs(), 0)]), + HolisticRecall::new(None, vec![ScopeSection::answer("Docs", " ", docs(), 1)]), + ]; + for request in cases { + let error = holistic_recall(&engine, &request).await.unwrap_err(); + assert!(matches!(error, Error::InvalidRequest(_)), "{error:?}"); + } +} + +#[tokio::test] +async fn an_item_is_listed_once_in_its_first_section() { + let engine = seeded().await; + let request = HolisticRecall::new( + Some("refunds".into()), + vec![ + ScopeSection::fetch("Docs", docs(), 1), + ScopeSection::fetch("Everything", MetaFilter::default(), 10), + ], + ); + let pack = holistic_recall(&engine, &request).await.unwrap(); + assert_eq!( + pack.markdown + .matches("Refunds take five business days.") + .count(), + 1, + "{}", + pack.markdown + ); + assert!( + pack.sections[1] + .hits + .iter() + .all(|hit| hit.id != pack.sections[0].hits[0].id) + ); +} + +#[tokio::test] +async fn a_titled_document_is_one_readable_bullet() { + let engine = ReferenceEngine::new(); + for (title, body) in [ + ("Onboarding", "# Onboarding\n\nReply within four hours."), + ("Refunds", "Refunds take five days."), + ("Billing", "# Billing disputes\n\nGo to finance."), + ] { + engine + .store(StoreItem::Document { + title: Some(title.into()), + body: tinymemory_api::DocumentBody::Text(body.into()), + mime: None, + meta: MemoryMeta::default(), + }) + .await + .unwrap(); + } + let pack = holistic_recall( + &engine, + &HolisticRecall::new(None, vec![ScopeSection::latest("Docs", docs(), 5)]), + ) + .await + .unwrap(); + let md = &pack.markdown; + assert!( + md.contains("- Onboarding: Reply within four hours.\n"), + "{md}" + ); + assert!(md.contains("- Refunds: Refunds take five days.\n"), "{md}"); + assert!( + md.contains("- Billing: # Billing disputes Go to finance.\n"), + "{md}" + ); +} + +#[tokio::test] +async fn a_timed_turn_is_led_by_when_it_was_said() { + let engine = ReferenceEngine::new(); + let at = "2026-09-15T09:01:30Z".parse().unwrap(); + for item in [ + StoreItem::Conversation { + turns: vec![Turn::new(Role::User, "The budget is now 6500 dollars.")], + meta: MemoryMeta { + observed_at: Some(at), + ..MemoryMeta::default() + }, + }, + turn("t1", 0, "The budget is 5000 dollars."), + ] { + engine.store(item).await.unwrap(); + } + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + None, + vec![ScopeSection::latest( + "History", + MetaFilter::kinds([ItemKind::Conversation]), + 5, + )], + ), + ) + .await + .unwrap(); + let md = &pack.markdown; + assert!( + md.contains("- [2026-09-15 09:01] user: The budget is now 6500 dollars.\n"), + "{md}" + ); + assert!( + md.contains("- user: The budget is 5000 dollars.\n"), + "an untimed turn has no date: {md}" + ); +} + +/// The reference engine plus beliefs kept apart from its items, or a belief +/// read that fails. Fetch returns them when asked; it records every belief +/// listing and every fetch's belief budget. +struct Believer { + inner: ReferenceEngine, + beliefs: Option>, + asked: std::sync::Mutex>, + budgets: std::sync::Mutex>, +} + +impl Believer { + async fn new(beliefs: Option>) -> Self { + Self { + inner: seeded().await, + beliefs, + asked: std::sync::Mutex::new(Vec::new()), + budgets: std::sync::Mutex::new(Vec::new()), + } + } + + fn held(&self, limit: usize) -> tinymemory_api::Result> { + let Some(texts) = &self.beliefs else { + return Err(Error::Unavailable("beliefs are down".into())); + }; + Ok(texts + .iter() + .map(|text| tinymemory_api::Hit { + id: tinymemory_api::ItemId::new(format!("belief:{text}")), + kind: ItemKind::Learning, + text: (*text).to_string(), + meta: MemoryMeta { + tags: vec![tinymemory_api::BELIEF_TAG.to_string()], + ..MemoryMeta::default() + }, + score: 0.0, + confidence: Some(0.9), + }) + .take(limit) + .collect()) + } +} + +#[async_trait] +impl tinymemory_api::MemoryEngine for Believer { + fn descriptor(&self) -> &EngineDescriptor { + self.inner.descriptor() + } + async fn health(&self) -> EngineHealth { + EngineHealth::Ok + } + async fn recall(&self, req: RecallRequest) -> tinymemory_api::Result { + self.inner.recall(req).await + } + async fn fetch(&self, req: FetchRequest) -> tinymemory_api::Result { + self.budgets.lock().unwrap().push(req.beliefs); + let wanted = req.beliefs; + let mut page = self.inner.fetch(req).await?; + if wanted > 0 { + page.beliefs = self.held(wanted).unwrap_or_default(); + } + Ok(page) + } + async fn store(&self, item: StoreItem) -> tinymemory_api::Result { + self.inner.store(item).await + } + async fn forget(&self, target: ForgetTarget) -> tinymemory_api::Result { + self.inner.forget(target).await + } + async fn list(&self, req: ListRequest) -> tinymemory_api::Result { + self.inner.list(req).await + } + async fn beliefs( + &self, + req: tinymemory_api::BeliefsRequest, + ) -> tinymemory_api::Result> { + self.asked.lock().unwrap().push(req.clone()); + self.held(req.limit) + } +} + +fn learnings_section(heading: &str) -> ScopeSection { + ScopeSection::fetch(heading, MetaFilter::kinds([ItemKind::Learning]), 5) +} + +#[tokio::test] +async fn a_learnings_section_merges_the_engine_s_beliefs() { + let engine = Believer::new(Some(vec!["user prefers pnpm over npm"])).await; + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + Some("refunds".into()), + vec![ + learnings_section("Learnings"), + ScopeSection::fetch("Docs", docs(), 5), + ], + ), + ) + .await + .unwrap(); + let md = &pack.markdown; + assert!(md.contains("- Customers prefer refunds by email\n"), "{md}"); + assert!(md.contains("- user prefers pnpm over npm\n"), "{md}"); + assert!( + md.find("Customers prefer").unwrap() < md.find("user prefers pnpm").unwrap(), + "stored learnings lead at each rank: {md}" + ); + assert!( + engine.asked.lock().unwrap().is_empty(), + "with a query, beliefs come from the fetches, not a separate read" + ); + assert_eq!( + *engine.budgets.lock().unwrap(), + [5, 5], + "every fetch asks for what the learnings section wants" + ); +} + +#[tokio::test] +async fn beliefs_another_section_read_land_in_the_learnings() { + let engine = Believer::new(Some(vec!["user prefers pnpm over npm"])).await; + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + Some("refunds".into()), + vec![ + ScopeSection::fetch("Docs", docs(), 5), + learnings_section("Learnings"), + ], + ), + ) + .await + .unwrap(); + let learnings = pack + .sections + .iter() + .find(|section| section.heading == "Learnings") + .unwrap(); + assert_eq!( + learnings + .hits + .iter() + .filter(|hit| hit.text == "user prefers pnpm over npm") + .count(), + 1, + "each belief once, in the learnings: {}", + pack.markdown + ); + assert!( + !pack.sections[0] + .hits + .iter() + .any(|hit| hit.text.contains("pnpm")), + "beliefs never land in a section that does not read learnings" + ); +} + +#[tokio::test] +async fn a_pack_without_learnings_asks_for_no_beliefs() { + let engine = Believer::new(Some(vec!["user prefers pnpm over npm"])).await; + holistic_recall( + &engine, + &HolisticRecall::new( + Some("refunds".into()), + vec![ScopeSection::fetch("Docs", docs(), 5)], + ), + ) + .await + .unwrap(); + assert_eq!(*engine.budgets.lock().unwrap(), [0]); + assert!(engine.asked.lock().unwrap().is_empty()); +} + +#[tokio::test] +async fn a_latest_learnings_section_reads_beliefs_without_a_query() { + let engine = Believer::new(Some(vec!["user prefers pnpm over npm"])).await; + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + None, + vec![ScopeSection::latest( + "Learnings", + MetaFilter::kinds([ItemKind::Learning]), + 5, + )], + ), + ) + .await + .unwrap(); + assert!(pack.markdown.contains("- user prefers pnpm over npm\n")); + assert_eq!(engine.asked.lock().unwrap()[0].query, None); +} + +#[tokio::test] +async fn a_failed_belief_listing_leaves_the_stored_learnings() { + let engine = Believer::new(None).await; + let pack = holistic_recall( + &engine, + &HolisticRecall::new( + None, + vec![ScopeSection::latest( + "Learnings", + MetaFilter::kinds([ItemKind::Learning]), + 5, + )], + ), + ) + .await + .unwrap(); + assert!( + pack.markdown + .contains("- Customers prefer refunds by email\n") + ); + assert!(pack.skipped.is_empty(), "{:?}", pack.skipped); +} + +#[tokio::test] +async fn a_belief_the_filter_rules_out_is_left_out() { + let engine = Believer::new(Some(vec!["user prefers pnpm over npm"])).await; + let mut section = learnings_section("Learnings"); + section.filter.thread_id = Some("t1".into()); + let pack = holistic_recall( + &engine, + &HolisticRecall::new(Some("refunds".into()), vec![section]), + ) + .await + .unwrap(); + assert!(!pack.markdown.contains("pnpm"), "{}", pack.markdown); +} diff --git a/crates/tinymemory-tools/src/recall/render.rs b/crates/tinymemory-tools/src/recall/render.rs new file mode 100644 index 00000000..4db3def8 --- /dev/null +++ b/crates/tinymemory-tools/src/recall/render.rs @@ -0,0 +1,228 @@ +//! Rendering gathered sections into budgeted markdown. +//! +//! Pure: given the sections and the budget, the output is fixed. A section is +//! either **prose** (an answer) or **lines** (one bullet per hit). Sections +//! come highest priority first, and trimming takes from the end: lines go +//! first, one at a time from the last section that has any, and a section +//! left with none is dropped; then the last prose section is shortened, and +//! dropped once too little of it is left. So earlier sections keep their text +//! longest and every section keeps its place. +//! +//! `context.md` is this renderer with frontmatter: its briefs are prose and +//! its learnings one lines section at the end, so learnings trim first. + +use chrono::{DateTime, SecondsFormat, Utc}; +use tinymemory_api::ItemId; + +/// Characters per estimated token. +const CHARS_PER_TOKEN: usize = 4; + +/// A shortened prose section shorter than this is dropped rather than kept +/// as a stub. +const MIN_PROSE_CHARS: usize = 40; + +/// Marks a shortened text. +const ELLIPSIS: char = '…'; + +/// The estimated token count of `text`: four characters per token, rounded +/// up — the estimate every context budget uses. +pub fn estimate_tokens(text: &str) -> usize { + text.chars().count().div_ceil(CHARS_PER_TOKEN) +} + +/// One bullet: an item and the text shown for it. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct Line { + pub(crate) id: ItemId, + pub(crate) text: String, +} + +/// What a section shows. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum Body { + /// An answer, citing `refs`. + Prose { text: String, refs: Vec }, + /// One bullet per item, best first. + Lines(Vec), +} + +/// One gathered section. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct Section { + pub(crate) heading: String, + pub(crate) body: Body, +} + +impl Section { + fn is_empty(&self) -> bool { + match &self.body { + Body::Prose { text, .. } => text.trim().is_empty(), + Body::Lines(lines) => lines.is_empty(), + } + } +} + +/// The frontmatter `context.md` carries. +#[derive(Debug, Clone, Copy)] +pub(crate) struct Frontmatter<'a> { + pub(crate) engine: &'a str, + pub(crate) generated_at: DateTime, +} + +/// The rendered block. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct Rendered { + pub(crate) markdown: String, + pub(crate) tokens: usize, + pub(crate) refs: Vec, +} + +/// Renders `sections` under `title` within `budget_tokens`. +pub(crate) fn render( + mut sections: Vec
, + budget_tokens: usize, + title: &str, + frontmatter: Option>, +) -> Rendered { + sections.retain(|section| !section.is_empty()); + loop { + let rendered = compose(§ions, title, frontmatter); + if rendered.tokens <= budget_tokens { + return rendered; + } + if pop_line(&mut sections) { + continue; + } + let Some(index) = sections + .iter() + .rposition(|section| matches!(section.body, Body::Prose { .. })) + else { + // Nothing left to trim: an empty document always fits. + return compose(§ions, title, frontmatter); + }; + let overflow_chars = (rendered.tokens - budget_tokens) * CHARS_PER_TOKEN; + if let Body::Prose { text, .. } = &mut sections[index].body { + let keep = text + .chars() + .count() + .saturating_sub(overflow_chars + ELLIPSIS.len_utf8()); + if keep < MIN_PROSE_CHARS { + sections.remove(index); + } else { + *text = shorten(text, keep); + } + } + } +} + +/// Drops the last line of the last section that has lines, and the section +/// with it once it is empty. `false` when no section has lines. +fn pop_line(sections: &mut Vec
) -> bool { + let Some(index) = sections + .iter() + .rposition(|section| matches!(section.body, Body::Lines(_))) + else { + return false; + }; + if let Body::Lines(lines) = &mut sections[index].body { + lines.pop(); + if lines.is_empty() { + sections.remove(index); + } + } + true +} + +/// The first `keep` characters of `text`, cut back to a word boundary when +/// one is near, with an ellipsis. +pub(crate) fn shorten(text: &str, keep: usize) -> String { + let cut: String = text.chars().take(keep).collect(); + let trimmed = match cut.rfind(char::is_whitespace) { + Some(space) if space * 2 > cut.len() => &cut[..space], + _ => cut.as_str(), + }; + format!("{}{ELLIPSIS}", trimmed.trim_end()) +} + +/// Composes the full block, frontmatter included, without trimming. +fn compose(sections: &[Section], title: &str, frontmatter: Option>) -> Rendered { + if sections.is_empty() { + return Rendered { + markdown: String::new(), + tokens: 0, + refs: Vec::new(), + }; + } + let mut refs: Vec = Vec::new(); + let mut body = format!("# {title}\n"); + for section in sections { + match §ion.body { + Body::Prose { text, refs: cited } => { + body.push_str(&format!("\n## {}\n\n{}\n", section.heading, text.trim())); + push_unique(&mut refs, cited); + } + Body::Lines(lines) => { + body.push_str(&format!("\n## {}\n\n", section.heading)); + for line in lines { + body.push_str(&format!("- {}\n", single_line(&line.text))); + push_unique(&mut refs, std::slice::from_ref(&line.id)); + } + } + } + } + let Some(frontmatter) = frontmatter else { + return Rendered { + tokens: estimate_tokens(&body), + markdown: body, + refs, + }; + }; + // The frontmatter's own token count is part of the document, so estimate + // the count it reports from a draft carrying a same-width placeholder, + // then fix the number point so the report is exact. + let draft = header(frontmatter, 0, &refs) + "\n" + &body; + let mut tokens = estimate_tokens(&draft); + let mut markdown = draft; + for _ in 0..8 { + markdown = header(frontmatter, tokens, &refs) + "\n" + &body; + let settled = estimate_tokens(&markdown); + if settled == tokens { + break; + } + tokens = settled; + } + Rendered { + tokens: estimate_tokens(&markdown), + markdown, + refs, + } +} + +fn header(frontmatter: Frontmatter<'_>, tokens: usize, refs: &[ItemId]) -> String { + let refs: Vec<&str> = refs.iter().map(ItemId::as_str).collect(); + format!( + "---\ngenerated_at: {}\nengine: {}\ntokens: {tokens}\nrefs: [{}]\n---\n", + frontmatter + .generated_at + .to_rfc3339_opts(SecondsFormat::Secs, true), + frontmatter.engine, + refs.join(", ") + ) +} + +/// `text` on one line, whitespace collapsed. +pub(crate) fn single_line(text: &str) -> String { + text.split_whitespace().collect::>().join(" ") +} + +fn push_unique(refs: &mut Vec, more: &[ItemId]) { + for id in more { + if !refs.contains(id) { + refs.push(id.clone()); + } + } +} + +#[cfg(test)] +#[path = "render_tests.rs"] +mod tests; diff --git a/crates/tinymemory-tools/src/recall/render_tests.rs b/crates/tinymemory-tools/src/recall/render_tests.rs new file mode 100644 index 00000000..1024466f --- /dev/null +++ b/crates/tinymemory-tools/src/recall/render_tests.rs @@ -0,0 +1,259 @@ +//! Budgeting, trimming order and frontmatter, through the `context.md` +//! shape (prose briefs, then a learnings list) and through plain packs. + +use chrono::TimeZone; + +use super::*; + +/// One answered brief, as `context.md` has them. +#[derive(Debug, Clone)] +struct BriefSection { + heading: String, + body: String, + refs: Vec, +} + +/// One learning line. +type LearningLine = Line; + +/// The `context.md` shape: prose briefs, then one learnings list. +#[derive(Debug, Clone)] +struct Sections { + briefs: Vec, + learnings: Vec, +} + +/// Renders the `context.md` shape with frontmatter, as the compiler does. +fn render_doc( + sections: Sections, + budget_tokens: usize, + engine: &str, + generated_at: DateTime, +) -> Rendered { + let mut all: Vec
= sections + .briefs + .into_iter() + .map(|brief| Section { + heading: brief.heading, + body: Body::Prose { + text: brief.body, + refs: brief.refs, + }, + }) + .collect(); + all.push(Section { + heading: "Learnings".to_string(), + body: Body::Lines(sections.learnings), + }); + render( + all, + budget_tokens, + "Context", + Some(Frontmatter { + engine, + generated_at, + }), + ) +} + +fn at() -> DateTime { + Utc.with_ymd_and_hms(2026, 10, 2, 12, 0, 0).unwrap() +} + +fn brief(heading: &str, body: &str, refs: &[&str]) -> BriefSection { + BriefSection { + heading: heading.to_string(), + body: body.to_string(), + refs: refs.iter().map(|id| ItemId::from(*id)).collect(), + } +} + +fn learning(id: &str, text: &str) -> LearningLine { + LearningLine { + id: ItemId::from(id), + text: text.to_string(), + } +} + +fn sections() -> Sections { + Sections { + briefs: vec![ + brief( + "About the user", + &"Steven builds memory systems. ".repeat(4), + &["a", "b"], + ), + brief( + "Active work", + &"Working on memory v2 in Rust. ".repeat(4), + &["b", "c"], + ), + ], + learnings: (0..10) + .map(|i| learning(&format!("l{i}"), &format!("learning number {i}\nwrapped"))) + .collect(), + } +} + +#[test] +fn tokens_are_four_characters_rounded_up() { + assert_eq!(estimate_tokens(""), 0); + assert_eq!(estimate_tokens("abcd"), 1); + assert_eq!(estimate_tokens("abcde"), 2); + assert_eq!(estimate_tokens("ééééé"), 2); +} + +#[test] +fn an_ample_budget_keeps_everything_with_frontmatter() { + let rendered = render_doc(sections(), 10_000, "reference", at()); + let md = &rendered.markdown; + assert!(md.starts_with("---\ngenerated_at: 2026-10-02T12:00:00Z\nengine: reference\n")); + assert!(md.contains(&format!("tokens: {}\n", rendered.tokens))); + assert!(md.contains("refs: [a, b, c, l0, l1")); + let about = md.find("## About the user").unwrap(); + let active = md.find("## Active work").unwrap(); + let learnings = md.find("## Learnings").unwrap(); + assert!(about < active && active < learnings); + assert!(md.contains("- learning number 0 wrapped\n")); + assert_eq!(rendered.tokens, estimate_tokens(md)); + assert_eq!(rendered.refs.len(), 13); +} + +#[test] +fn learnings_are_trimmed_before_any_brief() { + let full = render_doc(sections(), 10_000, "reference", at()); + let budget = full.tokens - 20; + let rendered = render_doc(sections(), budget, "reference", at()); + assert!(rendered.tokens <= budget); + assert!( + rendered.markdown.contains( + &"Steven builds memory systems. " + .repeat(4) + .trim() + .to_string() + ) + ); + assert!( + rendered.markdown.contains( + &"Working on memory v2 in Rust. " + .repeat(4) + .trim() + .to_string() + ) + ); + assert!(!rendered.markdown.contains("learning number 9")); + assert!(rendered.markdown.contains("learning number 0")); + assert!(!rendered.refs.contains(&ItemId::from("l9"))); +} + +#[test] +fn once_learnings_are_gone_the_last_brief_shrinks_then_drops() { + let without_learnings = Sections { + learnings: Vec::new(), + ..sections() + }; + let full = render_doc(without_learnings.clone(), 10_000, "reference", at()); + let shrunk = render_doc( + without_learnings.clone(), + full.tokens - 10, + "reference", + at(), + ); + assert!(shrunk.tokens <= full.tokens - 10); + assert!(shrunk.markdown.contains("## Active work")); + assert!(shrunk.markdown.contains('…')); + assert!( + shrunk.markdown.contains( + &"Steven builds memory systems. " + .repeat(4) + .trim() + .to_string() + ) + ); + + let tight = render_doc(without_learnings, full.tokens - 45, "reference", at()); + assert!(tight.tokens <= full.tokens - 45); + assert!(tight.markdown.contains("## About the user")); + assert!(!tight.markdown.contains("## Active work")); + assert!(!tight.refs.contains(&ItemId::from("c"))); +} + +#[test] +fn nothing_to_say_or_no_room_is_an_empty_document() { + let empty = Sections { + briefs: Vec::new(), + learnings: Vec::new(), + }; + let rendered = render_doc(empty, 100, "reference", at()); + assert_eq!(rendered.markdown, ""); + assert_eq!(rendered.tokens, 0); + let squeezed = render_doc(sections(), 5, "reference", at()); + assert_eq!(squeezed.markdown, ""); + assert!(squeezed.refs.is_empty()); +} + +fn lines(heading: &str, ids: &[&str]) -> Section { + Section { + heading: heading.to_string(), + body: Body::Lines( + ids.iter() + .map(|id| Line { + id: ItemId::from(*id), + text: format!("hit {id} with some words in it"), + }) + .collect(), + ), + } +} + +#[test] +fn a_pack_without_frontmatter_is_just_the_sections() { + let rendered = render( + vec![lines("Brain", &["b1"]), lines("Learnings", &["l1"])], + 10_000, + "Memory", + None, + ); + assert_eq!( + rendered.markdown, + "# Memory\n\n## Brain\n\n- hit b1 with some words in it\n\n## Learnings\n\n- hit l1 with some words in it\n" + ); + assert_eq!(rendered.tokens, estimate_tokens(&rendered.markdown)); + assert_eq!(rendered.refs, [ItemId::from("b1"), ItemId::from("l1")]); +} + +#[test] +fn the_last_lines_section_is_trimmed_first_and_dropped_when_empty() { + let all = vec![lines("First", &["a1", "a2"]), lines("Last", &["z1", "z2"])]; + let full = render(all.clone(), 10_000, "Memory", None); + let rendered = render(all.clone(), full.tokens - 5, "Memory", None); + assert!(rendered.markdown.contains("z1") && !rendered.markdown.contains("z2")); + assert!(rendered.markdown.contains("a2")); + let tighter = render(all, full.tokens - 25, "Memory", None); + assert!( + !tighter.markdown.contains("## Last"), + "{}", + tighter.markdown + ); + assert!(tighter.markdown.contains("## First")); +} + +#[test] +fn empty_sections_never_render_a_heading() { + let rendered = render( + vec![ + lines("Nothing", &[]), + Section { + heading: "Blank".to_string(), + body: Body::Prose { + text: " ".to_string(), + refs: Vec::new(), + }, + }, + ], + 10_000, + "Memory", + None, + ); + assert_eq!(rendered.markdown, ""); +} diff --git a/crates/tinymemory-tools/src/recall/types.rs b/crates/tinymemory-tools/src/recall/types.rs new file mode 100644 index 00000000..5e38cda8 --- /dev/null +++ b/crates/tinymemory-tools/src/recall/types.rs @@ -0,0 +1,263 @@ +//! The request and result types of a holistic recall. + +use serde::{Deserialize, Serialize}; +use tinymemory_api::{Error, Hit, ItemId, MetaFilter, Result}; + +/// Default token budget of a context pack. +pub const DEFAULT_PACK_BUDGET_TOKENS: usize = 1_200; + +/// Default heading of a context pack's markdown. +pub const DEFAULT_PACK_TITLE: &str = "Memory"; + +/// How one section is filled. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "by", rename_all = "snake_case")] +pub enum SectionQuery { + /// Ranked retrieval ([`tinymemory_api::MemoryEngine::fetch`]) for the + /// pack's query, or for `query` when the section names its own. With no + /// query at all the section reads [`SectionQuery::Latest`] instead. The + /// hot-path choice: no model runs. + Fetch { + /// This section's own query, overriding the pack's. + #[serde(default, skip_serializing_if = "Option::is_none")] + query: Option, + }, + /// A synthesised answer ([`tinymemory_api::MemoryEngine::recall`]) to a + /// fixed question. Slower: an engine may run a model. + Answer { + /// The question. + question: String, + /// Extra guidance for the answer. + #[serde(default, skip_serializing_if = "Option::is_none")] + instructions: Option, + /// When the answer fails, fetch for the question instead of skipping + /// the section. + #[serde(default)] + fallback_to_fetch: bool, + }, + /// The newest items, then the most confident, with no query: what a + /// learnings list or a recent-history section wants. + Latest, +} + +/// One section of a context pack: a heading, the scope it reads, and how it +/// is filled. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ScopeSection { + /// The section's `##` heading. + pub heading: String, + /// Which items the section reads: its reach and kinds are its scope. + #[serde(default)] + pub filter: MetaFilter, + /// The most hits (or citations) the section gathers. + pub limit: usize, + /// How the section is filled. + pub query: SectionQuery, +} + +impl ScopeSection { + /// A section of ranked hits for the pack's query. + #[must_use] + pub fn fetch(heading: impl Into, filter: MetaFilter, limit: usize) -> Self { + Self { + heading: heading.into(), + filter, + limit, + query: SectionQuery::Fetch { query: None }, + } + } + + /// A section answering `question`. + #[must_use] + pub fn answer( + heading: impl Into, + question: impl Into, + filter: MetaFilter, + limit: usize, + ) -> Self { + Self { + heading: heading.into(), + filter, + limit, + query: SectionQuery::Answer { + question: question.into(), + instructions: None, + fallback_to_fetch: false, + }, + } + } + + /// A section of the newest items. + #[must_use] + pub fn latest(heading: impl Into, filter: MetaFilter, limit: usize) -> Self { + Self { + heading: heading.into(), + filter, + limit, + query: SectionQuery::Latest, + } + } + + fn validate(&self) -> Result<()> { + if self.heading.trim().is_empty() { + return Err(Error::InvalidRequest( + "every recall section needs a heading".to_string(), + )); + } + if self.limit == 0 { + return Err(Error::InvalidRequest(format!( + "recall section `{}` has a zero limit", + self.heading + ))); + } + if let SectionQuery::Answer { question, .. } = &self.query + && question.trim().is_empty() + { + return Err(Error::InvalidRequest(format!( + "recall section `{}` answers a blank question", + self.heading + ))); + } + Ok(()) + } +} + +/// Part of a conversation thread a pack leaves out because the host's prompt +/// already holds it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ThreadWindow { + /// The thread. + pub thread_id: String, + /// The first turn index still in the prompt; turns from here on are + /// left out. + #[serde(default)] + pub from_turn: u32, +} + +impl ThreadWindow { + /// Whether `hit` lies inside the window. + #[must_use] + pub fn covers(&self, hit: &Hit) -> bool { + hit.meta.thread_id.as_deref() == Some(self.thread_id.as_str()) + && hit + .meta + .turns + .as_ref() + .is_none_or(|turns| turns.last >= self.from_turn) + } +} + +/// A recall across several scopes at once, rendered into one budgeted +/// markdown block: the single read every lifecycle step is built from. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct HolisticRecall { + /// The query [`SectionQuery::Fetch`] sections rank for. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub query: Option, + /// The sections, highest priority first: budget trimming takes from the + /// last. + pub sections: Vec, + /// The most tokens the rendered block may take, four characters per + /// token. + pub budget_tokens: usize, + /// The block's `#` heading. + pub title: String, + /// Items never included (the turn just logged, say). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub exclude_ids: Vec, + /// Conversation turns never included because the prompt holds them. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub exclude_thread: Option, +} + +impl HolisticRecall { + /// A recall of `sections` for `query`, with the default budget and title. + #[must_use] + pub fn new(query: Option, sections: Vec) -> Self { + Self { + query, + sections, + budget_tokens: DEFAULT_PACK_BUDGET_TOKENS, + title: DEFAULT_PACK_TITLE.to_string(), + exclude_ids: Vec::new(), + exclude_thread: None, + } + } + + /// Checks the request. + /// + /// # Errors + /// + /// [`Error::InvalidRequest`] for a zero budget, a blank title, or a + /// section with a blank heading, a zero limit or a blank question. + pub fn validate(&self) -> Result<()> { + if self.budget_tokens == 0 { + return Err(Error::InvalidRequest( + "a recall budget must be positive".to_string(), + )); + } + if self.title.trim().is_empty() { + return Err(Error::InvalidRequest( + "a recall block needs a title".to_string(), + )); + } + self.sections.iter().try_for_each(ScopeSection::validate) + } + + /// Whether `hit` is left out by [`HolisticRecall::exclude_ids`] or + /// [`HolisticRecall::exclude_thread`]. + pub(crate) fn excludes(&self, hit: &Hit) -> bool { + self.exclude_ids.contains(&hit.id) + || self + .exclude_thread + .as_ref() + .is_some_and(|window| window.covers(hit)) + } +} + +/// What one section gathered, before budget trimming. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct SectionHits { + /// The section's heading. + pub heading: String, + /// The synthesised answer, for an answered section. + #[serde(skip_serializing_if = "Option::is_none")] + pub answer: Option, + /// The hits, best first (an answered section's citations as hits). + pub hits: Vec, +} + +/// A section that contributed nothing, and why. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct SkippedSection { + /// The section's heading. + pub heading: String, + /// Why: `empty`, or the engine's error. + pub reason: String, +} + +/// A rendered holistic recall. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct ContextPack { + /// The block to inject into a prompt; empty when nothing was found. + pub markdown: String, + /// Its estimated tokens. + pub tokens: usize, + /// Every item the block cites, in order of first citation. + pub refs: Vec, + /// Everything each section gathered, in section order, including what + /// the budget trimmed from the block. + pub sections: Vec, + /// Sections that contributed nothing. + pub skipped: Vec, + /// The id of the engine read. + pub engine: String, +} + +impl ContextPack { + /// Whether the block is empty. + #[must_use] + pub fn is_empty(&self) -> bool { + self.markdown.is_empty() + } +} diff --git a/crates/tinymemory-tools/src/tools/read/mod.rs b/crates/tinymemory-tools/src/tools/read/mod.rs index dc7187c8..1cb02456 100644 --- a/crates/tinymemory-tools/src/tools/read/mod.rs +++ b/crates/tinymemory-tools/src/tools/read/mod.rs @@ -77,6 +77,7 @@ pub(crate) async fn fetch( filter, limit: args.count("limit", DEFAULT_LIMIT, MAX_LIMIT)?, cursor: args.string("cursor")?, + beliefs: 0, }; Ok(render::fetch(&engine.fetch(request).await?)) } diff --git a/crates/tinymemory-tools/src/tools/render/mod_tests.rs b/crates/tinymemory-tools/src/tools/render/mod_tests.rs index 8d205be7..31272ef7 100644 --- a/crates/tinymemory-tools/src/tools/render/mod_tests.rs +++ b/crates/tinymemory-tools/src/tools/render/mod_tests.rs @@ -67,6 +67,7 @@ fn pages_carry_a_cursor_only_when_there_is_more() { let more = FetchPage { hits: vec![sample_hit()], next_cursor: Some("1".into()), + beliefs: Vec::new(), }; assert_eq!(fetch(&more)["next_cursor"], json!("1")); let end = ListPage { diff --git a/docs/README.md b/docs/README.md index 4508902a..89328b03 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,12 +10,17 @@ where it cannot drift. ```text docs/ ├── README.md # this index +├── integration.md # how an agent host wires in the memory API ├── architecture/ # how the code is built, one document per concern ├── specs/ # behavior and architecture specifications ├── plans/ # implementation plans derived from approved specs +├── evals/ # accuracy and latency evals, with recorded runs └── adr/ # architecture decision records, numbered and immutable ``` +- **[`integration.md`](integration.md)** — for an agent host (OpenHuman or + any other): which calls to make around each turn, what they cost, the + rules that keep recall accurate, and how to bring your own engine. - **[`architecture/`](architecture/README.md)** — the shape of the three crates: the core contract, operation semantics, namespaces, the CortexDB engine, the tools, the integrations and the test strategy. @@ -24,6 +29,9 @@ docs/ - **[`plans/`](plans/README.md)** — implementation-ordered, test-first steps for delivering an approved specification. Plans name exact files and verification commands, and are updated as the work progresses. +- **[`evals/`](evals/README.md)** — how well the memory serves an agent, + measured: each eval's method, how to rerun it, and the results of recorded + runs. - **`adr/`** — a dated record per significant decision. Use [`adr/0001-record-architecture-decisions.md`](adr/0001-record-architecture-decisions.md) as the template. An accepted ADR is not edited; it is superseded by a later diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 109925b3..411c6dc3 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,7 +1,8 @@ # Architecture How TinyMemory is built, one document per concern. The accepted behaviour (the -"what and why") is [`specs/memory-v2.md`](../specs/memory-v2.md); these pages +"what and why") is [`specs/memory-v2.md`](../specs/memory-v2.md) and, for the +agent lifecycle, [`specs/agent-memory.md`](../specs/agent-memory.md); these pages describe the shape of the code that delivers it. Item-level reference lives in rustdoc next to the code. @@ -15,6 +16,7 @@ rustdoc next to the code. | [cortex.md](cortex.md) | The CortexDB engine: wires, scopes, envelopes, recall | | [cortex-wire.md](cortex-wire.md), [cortex-flows.md](cortex-flows.md) | The CortexDB wire formats and the step-by-step request flows | | [tools.md](tools.md) | `tinymemory-tools`: the seven agent tools, host-fixed scoping, `context.md` | +| [lifecycle.md](lifecycle.md) | The agent memory lifecycle: the standard layout (brain, conversations, learnings), holistic recall, pre- and post-turn, compaction, background belief builds | | [integrations.md](integrations.md) | `tinymemory-integrations`: registry and config, documents, sources, safety, legacy import | | [testing.md](testing.md) | The conformance suite, the reference engine and the test layout | diff --git a/docs/architecture/cortex-wire.md b/docs/architecture/cortex-wire.md index be3baf45..6ef71f00 100644 --- a/docs/architecture/cortex-wire.md +++ b/docs/architecture/cortex-wire.md @@ -44,6 +44,7 @@ for. Both fail with `Error::Unsupported` before any request. | Answer | `v1/answer` | `memory/answer` | POST | | Health | `v1/admin/health` | `memory/scopes` | GET | | Scopes (registered scopes under a prefix) | `v1/scopes/list` | `memory/scopes` | GET | +| Build beliefs (one scope) | `v1/beliefs/build` | none (never sent) | POST | | Whoami (Direct only) | `v1/auth/whoami` | | GET | The endpoint is joined with the route, so a base URL with a path prefix keeps @@ -85,7 +86,11 @@ Response: `{"event_id": "..."}` (Direct answers `202`, with `status` and `replayed_from_idempotency` the engine does not read). A response without `event_id` is `Error::Engine`. -Direct appends with `?wait=indexed`. A single event goes to `v1/experience`; +Direct appends with `?wait=indexed`, except for a store that waits only for +acceptance (`store_with` with `WaitFor::Accepted`, which the agent lifecycle +uses for live turns). That store omits the parameter and also skips the +visibility waits, so the call returns once CortexDB has captured the event. +A single event goes to `v1/experience`; **two or more** go to `v1/experience/bulk` with ```json @@ -186,6 +191,64 @@ The reader accepts either `{"items": [{"path": "..."}]}` (Direct) or object with `path`. A `404` means "no scope listing" and is treated as no scopes. +### Build beliefs: `v1/beliefs/build` (Direct only) + +`consolidate` resolves its reach and kinds to the kind scopes that CortexDB +has registered. It reads them through `v1/scopes/list` under +`app:tinymemory`, keeping only the scopes the reach admits. It then posts one +request per scope, in order: + +```json +{ "scope": "app:tinymemory/source:pdf/app:documents" } +``` + +CortexDB v0.10 builds within the request, from the facts its enrichment has +already extracted, and answers with what it built: + +```json +{ "built": 2, "items": [ … ], "facts_scanned": 4, "events_scanned": 4, + "reasons": { "no_subject_or_predicate": 2 } } +``` + +When every answer carries `built`, the receipt is `Completed` with the counts +summed in `built`. An answer naming a job instead (`job_id`, `build_id` or +`id`) means the build was queued: the receipt is `Started` with the handles. +A build takes seconds per scope with a real model (8–30 s for a whole +scenario in [the eval](../evals/agent-memory.md)), so it belongs off the turn. +Each build is sent once and never retried: a host can always ask again. + +The beliefs land in a derived layer, read two ways: + +- **Inside a fetch** (`FetchRequest::beliefs > 0`): each scope's recall + pack also carries `"beliefs": N` in `per_layer_limits`. One pack, and one + query embedding, serves both the events and the beliefs. This is how + holistic recall reads them. +- **`beliefs` with a query** (`engine/beliefs.rs`), for a caller that wants + beliefs alone: one recall per scope held in reach, with a budget for the + `beliefs` layer only: + + ```json + { "scope": "…", "query": "…", + "budgets": { "per_layer_limits": { "events": 0, "facts": 0, "episodes": 0, + "understanding": 0, "beliefs": 8 } } } + ``` + +- **Without one:** `GET v1/beliefs?scope=…&limit=…` per scope, ordered most + confident and then newest. The hosted wire has no listing and answers + none. +- **Each belief** (`{id, scope, claim: {subject, predicate, object}, + stance, confidence, valid_from}`) becomes a `Learning` hit: + - its text is `subject predicate object`, with the predicate's + underscores as spaces; + - it is tagged `belief`, at its scope's node; + - only `supported` and `contested` stances are read, and a contested + belief says so. + +The answer route reads the same layer, so a `SectionQuery::Answer` section +(a compaction summary, a `context.md` brief) uses beliefs as well. Fetch +and list are unchanged: they decode only this crate's events. The TinyHumans backend has no such route; its descriptor declares +`Consolidation::Scheduled`, and `consolidate` sends nothing. + ### Health Direct: `GET v1/admin/health`. TinyHumans has no health route, so it lists one @@ -229,6 +292,7 @@ scope segment of the same text, using the contract's prefixes: | User | `user` | | Workspace | `ws` | | Project | `project` | +| Source | `source` | | TinyMemory root and each kind leaf | `app` | These are CortexDB's built-in types, chosen on purpose. From CortexDB v0.10 a diff --git a/docs/architecture/lifecycle.md b/docs/architecture/lifecycle.md new file mode 100644 index 00000000..16b2a212 --- /dev/null +++ b/docs/architecture/lifecycle.md @@ -0,0 +1,154 @@ +# The agent memory lifecycle + +How `tinymemory-tools` turns the contract into the calls a host makes around +every agent turn. The accepted behaviour is in +[`specs/agent-memory.md`](../specs/agent-memory.md). + +## The pieces + +| Module | Type | Role | +| --- | --- | --- | +| `layout` | `MemoryLayout`, `BrainSource` | The standard tree: brain sources, agent conversations, learnings, below one root | +| `recall` | `HolisticRecall` → `ContextPack` | The one read: several scopes at once, deduplicated, rendered within a budget | +| `brain` | `Brain`, `BrainDocument` | Global documents by source type: ingest, search, forget | +| `lifecycle` | `AgentMemory`, `RecallPolicy` | One agent's session start, pre-turn, post-turn and compaction | +| `background` | `BackgroundJob`, `BackgroundRunner` | The slow work the others hand back | +| `context` | `ContextCompiler` | `context.md`: a holistic recall preset with frontmatter | + +Every module takes an `Arc`. None of them knows which engine +is behind it, so swapping engines is a one-line change in the host. + +## The tree + +```text + (Namespace::ROOT, or a host node like team:acme) +├── source:pdf documents — brain, no agent id +├── source:markdown documents +├── source:notion documents +├── agent:support-01 conversations (one item per turn) +├── agent:coder-42 conversations +└── itself learnings (shared); built beliefs land in each scope +``` + +On CortexDB every node becomes a scope family under `app:tinymemory/…` (see +[cortex-wire.md](cortex-wire.md#scope-layout)). For example, +`source:pdf/app:documents` and `agent:coder-42/app:conversations`. + +## One turn + +```text +host AgentMemory engine + │ pre_turn(thread, i, text) ──▶│ + │ ├─ store_with(turn, Accepted) ─▶ (≈ capture, no indexing wait) + │ ├─ holistic recall, concurrently: + │ │ Learnings fetch ──────────▶ + │ │ Brain fetch ──────────▶ + │ │ History fetch ──────────▶ + │ │ Team fetch ──────────▶ + │ ├─ drop: the logged turn, the thread from in_prompt_from + │ ├─ dedupe across sections, render within budget + │◀── TurnContext { pack, logged | log_error } + │ model.generate(system + pack.markdown + text) + │ post_turn(thread, i+1, reply) ▶ store_with(reply, Accepted) ─▶ + │◀── PostTurnReport { receipt, jobs: [BuildBeliefs?] } + │ queue jobs; later: run_background(job) ──▶ consolidate ─────▶ v1/beliefs/build +``` + +The hot path runs no model. `fetch` is ranked retrieval. On CortexDB it is +one recall pack per scope, with no `/answer`. Against the pinned harness a +warm `pre_turn` measured about 40 ms and a `post_turn` about 2 ms; see +`examples/cortex_agent.rs`. + +`pre_turn` returns a pack even when logging fails. The failure is in +`log_error`, so a write outage degrades memory but never blocks a turn. + +## Holistic recall in detail + +1. **Validate.** The only error a pack can return is an invalid request. +2. **Gather** (`recall/gather.rs`) every section concurrently with + `join_all`, which links no runtime. Each section asks for more than its + limit, to leave room for the exclusions and for items earlier sections + may already list: + - `Fetch` uses the section's query, else the pack's, else `Latest`. + - `Answer` uses recall, falling back to fetch when asked. + - `Latest` pages the listing, then sorts by `observed_at`, confidence + and the turn's index. +3. **Settle** the sections in order: + - Answers pass through untouched. + - Hit lists lose the kinds the section does not admit, the excluded ids, + the thread window, and anything an earlier section listed. Each list + is then cut to its limit. + - A section left empty is reported as skipped (`reason: "empty"`), as is + a failed one (its error). +4. **Render** (`recall/render.rs`). A section is prose (an answer) or lines + (one bullet per hit). A titled document is shown as `title: body`, and a + turn that carries a time is led by it (`[2026-09-15 09:01] user: …`), so + a reader can tell a corrected value from its correction. When + the block overflows its budget, the last line is dropped first, from the + last section; once no lines are left, the last answer shortens and is + then dropped. `context.md` adds frontmatter whose token count includes + itself. + +## Background work + +`BackgroundJob` is plain, serializable data, so the host decides what +happens to it: run it inline, push it to a queue, persist it, or coalesce +duplicates. + +- `Brain::ingest` returns a `BuildBeliefs` job for the source's scope. +- `post_turn` returns one every `build_beliefs_every` turns, for the agent's + conversations. +- `BackgroundJob::IngestBrain` defers a whole ingestion. Its report hands + back the follow-up builds. + +What a build does depends on the engine's `consolidation`: + +| Engine | `consolidation` | `run_background(BuildBeliefs)` | +| --- | --- | --- | +| Reference | `OnDemand` | distils one `Fact` per item at once → `Done` | +| CortexDB direct | `OnDemand` | `POST v1/beliefs/build` per held scope, built within the request → `Done` | +| CortexDB via TinyHumans | `Scheduled` | nothing sent → `Scheduled` | +| an engine without it | `None` | `Skipped { reason }` | + +Built beliefs reach the Learnings section of every pack: + +- On the reference engine they are `Learning` items, so the Learnings + section lists them like any other. +- On CortexDB they live in a derived layer, which fetch and list (stored + items only) never return. Holistic recall therefore asks + for them whenever a pack has a learnings section, and interleaves them + with the stored learnings: + - with the pack's query, every fetched section's fetch asks for beliefs + (`FetchRequest::beliefs`). CortexDB adds the `beliefs` layer to the + recall pack it builds for each scope anyway, so a scope's query is + embedded once. Brain, history and team between them read every scope + a belief can be built in; + - without one (a cold `start_session`, `context.md`), the learnings + section lists them (`MemoryEngine::beliefs`, the `v1/beliefs` listing). + + Answered sections, such as a compaction summary or a `context.md` brief, + also draw on beliefs through CortexDB's answer route. + +## Compaction and session start + +- **`start_session`** puts a resumed thread's newest turns first, under + "Earlier in this thread", followed by the standard sections. Without a + `focus`, each section shows its newest items. +- **`recall_for_compaction`** answers "what was discussed, decided and left + open" over the thread, using recall (a model on engines that have one), + from as many of the thread's turns as were dropped (at least the policy's + history limit, at most 24). When the engine cannot answer, it fetches + instead. The standard sections follow, ranked for the focus or for the + dropped turns' gist: the start of every dropped turn, oldest first, within + 600 characters, so a fact stated early still steers the query. + +Neither call is on the hot path. + +## Integrations + +- `tinymemory_integrations::brain::brain_document` (feature `brain`) runs a + file through the `documents` converters and picks its `BrainSource` from + the detected format: PDF to `pdf`, markdown and text to `markdown`, HTML + to `web`. +- `cortex/lifecycle_tests.rs` runs one agent loop over both wires' doubles. +- `tests/live_cortex_lifecycle.rs` runs one against the harness. diff --git a/docs/architecture/namespaces.md b/docs/architecture/namespaces.md index a32b71da..03fa6579 100644 --- a/docs/architecture/namespaces.md +++ b/docs/architecture/namespaces.md @@ -37,6 +37,7 @@ team:acme/agent:writer/agent:helper | `User` | `user` | A human user | | `Workspace` | `ws` | A shared workspace | | `Project` | `project` | A project | +| `Source` | `source` | A knowledge source type (`source:pdf`): where the brain keeps each source's documents (see [lifecycle.md](lifecycle.md)) | `SegmentKind::as_str()` gives the path prefix (`ws` for `Workspace`). Note that the enum's own serde form is `snake_case` of the variant, so diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index e5e62f00..94e6916e 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -20,7 +20,7 @@ to depend on. | Crate | Role | Why it is separate | | --- | --- | --- | | `tinymemory-api` | The **contract**: `MemoryEngine`, items, metadata, filters, namespaces, errors. With the `conformance` feature, the suite every engine must pass and an in-memory reference engine. | It performs no I/O and links no runtime, HTTP stack or storage engine, so an engine, a tool layer or a host can depend on it without inheriting anything else. | -| `tinymemory-tools` | The **agent surface**: `MemoryTools` (seven model-callable tools with JSON Schemas and host-fixed scoping) and the `context.md` compiler. | It works over any `MemoryEngine`, has no tool-runtime dependency, and is where "a model must never choose whose memory it touches" is enforced. | +| `tinymemory-tools` | The **agent surface**: `MemoryTools` (seven model-callable tools with JSON Schemas and host-fixed scoping), holistic recall and the `context.md` compiler, and the agent lifecycle — `AgentMemory`, `Brain`, `MemoryLayout`, background jobs (see [lifecycle.md](lifecycle.md)). | It works over any `MemoryEngine`, has no tool-runtime dependency, and is where "a model must never choose whose memory it touches" is enforced. | | `tinymemory-integrations` | Everything that touches the **outside world**: the CortexDB engine, the engine registry and `MemoryConfig`, document conversion, source readers, safety scrubbing and the legacy v1 import. | Each integration is a Cargo feature, so a host pays only for the ones it uses. | ## Dependency graph @@ -51,7 +51,7 @@ alone, so the contract is the only coupling point. | Crate | Feature | Enables | | --- | --- | --- | | `tinymemory-api` | `conformance` | `conformance::run`, `conformance::ReferenceEngine`. No extra dependency. | -| `tinymemory-tools` | (none) | `tools` and `context` are always built. | +| `tinymemory-tools` | (none) | Every module (`tools`, `context`, `recall`, `layout`, `brain`, `lifecycle`, `background`) is always built. | | `tinymemory-integrations` | `cortex` (default) | `cortex::CortexEngine` (both wires), `registry` (`list_engines`, `build_engine`), `config` (`MemoryConfig`) | | | `documents` | Format sniffing and conversion to markdown | | | `documents-office` | PDF, DOCX, PPTX, XLSX conversion (implies `documents`) | diff --git a/docs/architecture/testing.md b/docs/architecture/testing.md index df4f0fc1..63ce23f4 100644 --- a/docs/architecture/testing.md +++ b/docs/architecture/testing.md @@ -306,7 +306,9 @@ TINYMEMORY_LIVE_CORTEXDB_URL=http://127.0.0.1:3141 \ The harness runs `mock-inference`, a deterministic OpenAI-compatible double for CortexDB's embeddings, extraction and answer models, so it needs no -credential; it is a wiring fixture, not a quality benchmark. Live tests name +credential; it is a wiring fixture, not a quality benchmark. Quality is +measured by the agent memory eval instead (`./scripts/memory-eval.sh`, with +`MODELS=openrouter` for real models); see [`docs/evals/`](../evals/README.md). Live tests name their file `live_*` (or `*_live`) so they are easy to exclude, and write only under a workspace unique to the run. diff --git a/docs/architecture/tools.md b/docs/architecture/tools.md index 38cae757..10f61108 100644 --- a/docs/architecture/tools.md +++ b/docs/architecture/tools.md @@ -303,7 +303,10 @@ let output = match tools.call(&call.name, call.arguments).await { `context.md` is a token-budgeted brief a host injects at the start of a session so the agent begins with what memory knows about its user. The -compiler works over any engine and is stateless. +compiler works over any engine and is stateless. It is a preset of the +holistic recall in `recall`: one answered section per brief, then the +latest learnings, rendered with frontmatter. The agent lifecycle's packs are +other presets of the same read (see [lifecycle.md](lifecycle.md)). ### ContextSpec @@ -404,7 +407,12 @@ crates/tinymemory-tools/src/ │ ├── read/ # recall, fetch, list, get, explore │ ├── write/ # store, forget │ └── render/ # compact result JSON -└── context/ # ContextSpec, Brief, compile, render, Error +├── context/ # ContextSpec, Brief, compile, Error +├── recall/ # HolisticRecall, ContextPack: gather, settle, render +├── layout/ # MemoryLayout, BrainSource +├── brain/ # Brain, BrainDocument +├── lifecycle/ # AgentMemory, RecallPolicy, turn types +└── background/ # BackgroundJob, BackgroundRunner ``` Tests: `tests/tools_roundtrip.rs` runs every tool against the reference diff --git a/docs/evals/README.md b/docs/evals/README.md new file mode 100644 index 00000000..14ea85bd --- /dev/null +++ b/docs/evals/README.md @@ -0,0 +1,113 @@ +# Evals + +Accuracy and latency measurements of TinyMemory against real engines, with +the method and the results of each recorded run. Tests prove behaviour; evals +measure how well that behaviour serves an agent. + +| Eval | What it measures | Latest run | +| --- | --- | --- | +| [Agent memory](agent-memory.md) | The lifecycle (`pre_turn`, `post_turn`, `start_session`, compaction, belief builds) across nine scenarios | 2026-10-04, CortexDB v0.10.4 | + +## The agent memory eval + +The harness is the `memory_eval` example in +`crates/tinymemory-integrations/examples/memory_eval/`: + +| File | Role | +| --- | --- | +| `main.rs` | Runs each scenario: writes, settle, probe, synthesise, probe again, forget | +| `agent.rs` | The scripted agent: real lifecycle calls, scripted tool calls, an extractive "model" | +| `scenarios.rs` | The nine scenarios and their probes | +| `score.rs` | Scoring a pack against a probe, totals and latency percentiles | +| `inspect.rs` | Reads CortexDB's derived layers (facts, beliefs) straight off the wire | +| `llm.rs` | The optional model that answers each probe from its pack (`--llm`) | + +### Running it + +```sh +# Offline, on the reference engine: seconds, no network. +cargo run -p tinymemory-integrations --features full --example memory_eval + +# A throwaway CortexDB with deterministic mock models (needs Docker). +./scripts/memory-eval.sh + +# The same with real models through OpenRouter (needs OPENROUTER_API_KEY), +# and a model answering every probe. +MODELS=openrouter ./scripts/memory-eval.sh --llm +``` + +**Cost.** Real models are not cheap here. CortexDB runs its extraction +model over every stored event and again during belief builds. The report +ends with the usage CortexDB accounted for (calls, tokens and dollars as its +router prices them). Earlier runs did not record it, so their cost is +unknown. Run one scenario first (`--only `) to gauge it. When the key +hits its spending limit, every embedding fails (`403 Key limit exceeded`) +and the run stops at the settle step with "only 0 of N writes visible". + +The script prints the report and writes `target/memory-eval/