Simplify TinyMemory into api, tools and integrations crates - #193
Merged
Merged
Conversation
Moved the source code from several standalone crates (tinymemory-conformance, tinymemory-cortex, tinymemory-documents, tinymemory-import, tinymemory-safety, tinymemory-sources, tinymemory-context, and parts of tinymemory) into the tinymemory-integrations crate, removing the now-empty source files from the original crates. This consolidation reduces the number of crates and simplifies the build graph by colocating related integration code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The workspace contained several crates that were no longer depended on by any consumer, including the top-level `tinymemory` facade crate and its associated sub-crates for conformance, context, cortex, documents, import, safety, and sources. These crates were removed along with the facade's `lib.rs` to simplify the workspace and eliminate dead code that was not being built or tested. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Move shared package fields (version, edition, rust-version, license, repository, publish) and lint tables into `[workspace.package]` and `[workspace.lints]` so the three crates inherit them uniformly, and add the new `tinymemory-tools` crate to the workspace. The per-crate `Cargo.toml` files now use `workspace = true` for these fields and opt into the workspace lint table, eliminating duplication and ensuring a single version bump applies to all crates. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the thiserror crate as a dependency to support the `context::Error` type, which is needed for error handling in the tools crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…modules This change introduces the tinymemory-integrations crate with a comprehensive set of modules for document processing, source ingestion, safety policies, and Cortex integration. The crate provides the foundational architecture for handling various document formats, importing data from multiple sources, and managing memory storage operations. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…mentation Introduce a conformance testing framework for the tinymemory API, including a reference implementation, test suites for bulk operations, checks, exploration, fixtures, and namespaces, along with error handling and scoring modules. This provides a structured way to validate API behavior against a known reference. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce the tinymemory-api crate with a basic Cargo.toml and lib.rs to establish the foundation for the memory API module. This provides the necessary scaffolding for future development of the API layer. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The error module was re-exporting the `Error` type from an incorrect path, causing compilation failures in dependent crates. This change updates the re-export to point to the correct location within the crate's module structure. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Renamed the `office` feature flag to `documents-office` and the `network` feature flag to `sources-network` across the tinymemory-integrations crate to avoid ambiguity and align with the crate's module naming conventions. The Cargo.lock was also updated to reflect the removal of several unused dependencies and version adjustments. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the unused PII safety module and its re-export to clean up the codebase and eliminate dead code that was no longer referenced anywhere. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Replace nested `if let` blocks with the more concise let-chain syntax (`&& let`) throughout the composio sources, readers, safety, and registry modules. This change also adds a new `pattern` module to the safety crate and imports its `literal` function for use in PII redaction, improving code readability and reducing unnecessary nesting. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Replaced the separate filter and map calls with a single filter_map that directly converts ASCII digits to their numeric values, removing an unnecessary expect call and making the code more idiomatic. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Re-export the `BearerSource` and `StaticBearer` types from the `cortex` module under the `cortex` feature flag, making them available to downstream consumers of the crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reorganised import statements to follow a consistent convention of grouping external crate imports before internal crate imports, and reformatted several multi-line expressions and chained method calls to improve readability without changing any runtime behaviour. This includes adjusting the layout of assert macros, regex pattern definitions, and conditional let-else blocks to reduce line lengths and align with project formatting standards. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…ture gates Register a basic example and three integration tests, each gated behind the appropriate Cargo feature so they are only built when the required dependencies are available. This ensures the example and tests are exercised as part of the feature-specific compilation rather than unconditionally. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The test module now requires both the `documents-office` and `cortex` features to be enabled, preventing compilation failures when only one of them is available. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the doc comment in the conformance module to use an intra-doc link (`crate::MemoryEngine`) instead of a fully qualified path, improving documentation consistency and making the link work correctly within the crate's documentation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…ce-level versioning The CI workflow was updated to test individual packages (tinymemory-integrations, tinymemory-api, tinymemory-tools) instead of the monolithic tinymemory crate, adding new matrix entries for api and tools packages while removing the context and conformance entries that are now covered by the api conformance test. The release workflow was changed to bump the workspace-level version in Cargo.toml rather than the facade crate's version, and now verifies the version update took effect across all workspace members. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce new modules for processing composio email sources, including email cleaning functionality and markdown conversion. These additions enable structured handling of email content within the composio integration, with corresponding test modules to ensure correctness. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the GitHub token is not set in the environment, the reader now returns an appropriate error instead of panicking or silently failing. This ensures clear feedback to users about missing configuration. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new composio source module for tinymemory integrations, enabling data ingestion from composio services. This includes the source implementation and its registration in the sources module, along with corresponding test updates for the GitHub reader to validate the integration. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the AGENTS.md file to reflect the current workspace structure, which now has three crates instead of the previous facade-based layout. The new descriptions clarify the role of each crate and the rule against adding a fourth one, and the release workflow section is updated to point to the workspace-level version field. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the description of how `unsafe` is forbidden to reflect the current workspace-level lint configuration, where the restriction is set in the root `Cargo.toml` via `[workspace.lints]` and inherited by each crate, rather than being configured per-crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
… and Notion sources Removed several utility functions that were no longer called anywhere in the codebase, including `now_ms()`, `extract_user_id()`, `extract_workspace_ids()` from the ClickUp source, `extract_viewer()`, `extract_viewer_id()`, `extract_pagination_cursor()`, and `now_ms()` from the Linear source, and `extract_notion_cursor()` and `now_ms()` from the Notion source. These functions were left over from earlier iterations of the integration and their removal reduces dead code and maintenance burden. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the `extract_user_login` and `now_ms` functions from the GitHub source module, as they were no longer used anywhere in the codebase. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Replaced deprecated or incorrect assertion macros in the ClickUp, GitHub, Linear, and Notion test files with the appropriate equivalents to ensure tests compile and run correctly. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The spec module import path was incorrect, causing a compilation error when building the tinymemory-tools crate. This change updates the path to point to the correct location within the crate's module hierarchy. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…sing The clickup and linear source implementations now use optional field access with `and_then` and `unwrap_or_default` to gracefully handle missing or null fields in API responses, preventing panics when optional metadata like assignee names or status values are absent. This change also adds a test for the linear source to verify correct parsing of partial data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…d parsing Remove several public methods and constants that are no longer needed outside the crate, and replace the `SegmentKind::ALL` array lookup with a direct match expression for parsing. The `child`, `parent`, and `shared_ancestor` methods were unused externally, while `ancestors_and_self` and `is_within` are now crate-internal. The `MAX_DEPTH`, `MAX_SEGMENT_ID`, and `ROOT_LABEL` constants are also restricted to `pub(crate)` visibility. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a URL with an empty host is passed to the SSRF protection, the validation now correctly rejects it instead of panicking. This ensures that malformed URLs are handled gracefully and do not cause unexpected crashes. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new document explaining the architecture and usage of namespaces in the project, providing clarity for developers on how namespaces are structured and intended to be used. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Two doc comments in the context module were updated to accurately describe their scope: the token estimate function now refers to "every context budget" instead of "every budget in this crate", and the result alias is now described as belonging to the context module rather than the entire crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the context is empty, the memory tools now return an appropriate response instead of panicking or producing undefined behavior. This ensures robust handling of edge cases where no context has been provided. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the architecture documentation for the cortex component to provide a clearer explanation of its responsibilities and interactions with other system components, improving developer understanding of the system's design. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the memory-v2 specification document to the docs directory and update the tinymemory-tools README to reference the new specification, providing a complete reference for the updated memory model. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added README documentation for the import and sources modules within the tinymemory-integrations crate to clarify their purpose and usage for developers working with external data ingestion. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a paragraph to the safety module's README clarifying that no engine automatically scrubs items, so the host must invoke the scrub step explicitly in the write pipeline. This helps callers understand the integration contract without having to dig into the architecture docs. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a link to the new architecture documentation directory in the docs README index, including a brief description of the topics covered in that section. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a README file to the tinymemory-integrations crate to document how to use the integration examples and provide context for developers working with the crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a table row linking to the new cortex-wire.md and cortex-flows.md documents, which describe the CortexDB wire formats and step-by-step request flows, to complete the architecture documentation index. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a README file to the tinymemory-integrations crate to document how to use the library with various external systems and frameworks, providing users with clear guidance on integration patterns and usage examples. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new document covering the testing strategy and architecture for the project, providing guidance on test structure, tools, and best practices for contributors. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…tools The README now describes the workspace as three crates split by dependency concerns, introduces the agent tools that let a model use memory without choosing whose memory it touches, and replaces the old feature table with one scoped to the integrations crate. The quickstart example is updated to show tool registration and invocation, and the migration section is restored with a working example. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the v1 migration documentation to better describe how the `migrate` function works, explaining that it takes a checkpoint from a previous run and returns the new position, and clarify that `migrate_with` provides checkpoints for the host to persist. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a README file to document the cortex module within the tinymemory-integrations crate, providing users with an overview of its purpose and usage. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add default values for optional configuration fields in the Cortex integration to prevent panics when the configuration file omits certain keys. This change ensures that the system continues to operate with sensible defaults instead of failing on startup. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the module-level documentation to remove an outdated reference to the re-exported error type, and reworded the `Result` type alias doc comment to accurately describe its scope as module-level rather than crate-wide. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The README now reflects the new crate and module paths after the `tinymemory-cortex` crate was renamed to `tinymemory-integrations` and its internal structure was reorganised, keeping the documentation accurate for developers running the live test harness. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a new architecture documentation file that describes the tools used in the project, providing clarity on the technology stack and development environment for contributors. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The documentation for the facet-count endpoint now explains that `total` refers to items admitted by the filter, `more_buckets` indicates values beyond the limit, and `truncated` means the engine stopped its scan early so counts are a lower bound. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…ts to integrations crate Register the new office_live test in the tinymemory-integrations crate and update the live test script to run both live_cortexdb and office_live tests from the integrations crate instead of their previous locations, consolidating all live integration tests in one place. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added a new documentation file covering the architecture of system integrations, providing a reference for how different components connect and interact within the system. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The table row for XLSX format contained unescaped pipe characters that were being interpreted as table delimiters, breaking the markdown table rendering. The pipes are now properly escaped with backslashes so they display as literal characters in the cell content. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a conversation has no messages, the reader now returns an empty result instead of panicking or producing undefined behavior. This ensures robustness when processing incomplete or empty conversation data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new architecture document describing the integration sources used in the system, covering their purpose and how they connect to the overall integration framework. This provides developers with a clear reference for understanding and extending the integration layer. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Tiny Sweeper review
|
|
Warning Review limit reached
This review includes 286 billable files and costs up to $71.50. View limit detailsReview configuration: ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (286)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduces TinyMemory to three crates, one for each part of the memory layer:
tinymemory-api: the core contract. The conformance suite and reference engine move in behind aconformancefeature.tinymemory-tools(new): the agent-facing tool spec,MemoryTools, which works over any engine. Thecontext.mdcompiler moves here too.tinymemory-integrations: everything that talks to the outside world, each part a module behind its own feature:cortex(default), which also pulls inregistryandconfigdocumentsanddocuments-officesourcesandsources-networksafetylegacy-importThe
tinymemoryfacade and seven separate crates are gone:tinymemory-cortextinymemory-documentstinymemory-sourcestinymemory-safetytinymemory-contexttinymemory-importtinymemory-conformanceCode that nothing called is deleted, duplicated paths are merged, and the whole engine is documented in a new
docs/architecture/.Rust lines go from 38,750 to 39,426. About 3.6k lines of dead or duplicated code were removed. About 3.2k were added for the new tool layer, roughly half of that tests.
Related issue
None.
API or behavior changes
Breaking. No host uses the current layout yet: openhuman and opencompany still vendor the v1.22 tree. Both will need a migration when they bump.
Facade removed. Hosts now depend on the three crates directly:
tinymemory::{build_engine, list_engines, MemoryConfig, EngineCredential, BearerSource, StaticBearer}tinymemory_integrations::…(the same names)tinymemory::cortex/documents/sources/safety/importtinymemory_integrations::…tinymemory::conformancetinymemory_api::conformance, behind theconformancefeaturetinymemory::contexttinymemory_tools::contextFeature renames.
officeis nowdocuments-officeandnetworkis nowsources-network. Hosts can still use thelegacy-importfeature, so the v1 TinyCortex → CortexDB path stays available for backwards compatibility.Versioning. The version lives in the root
[workspace.package], andrelease.ymlbumps it there. Lints come from[workspace.lints].New:
tinymemory_tools::MemoryTools. It defines seven JSON-schema tools:memory_recall,memory_fetch,memory_list,memory_get,memory_explore,memory_storeandmemory_forget. Their names and schemas are frozen intests/fixtures/tool_contracts.json.namespaceandreach. If the model passes either one anywhere in its arguments, the call is refused.getcan see under the reach.New:
import::migrate/migrate_with. They migrate a v1 workspace into any engine instore_manybatches. A checkpoint is recorded after each batch, and on error the last committed checkpoint is returned, so a re-run replays instead of duplicating.Removed with no callers:
sources.tomleditor (SourceRegistry,reconcile,validation,MemorySourcePatch);raw_kind; the GitHub raw-archive helpers; the Composio email markdown/clean modules; and unused extractors andnow_mscopies.Namespace::{parent, child, shared_ancestor},Facet::ALLandSegmentKind::ALL. Internal helpers became private, andMAX_BUCKETS,MAX_GET_IDSandMAX_SCAN_LIMITare now exported from the crate root.CortexEngine::with_request_timeout.INSUFFICIENT_CREDITS_CODEis now crate-private.Simplified:
storenow runs through thestore_manypath. The old single path already waited until readable, so its guarantees are unchanged.build_enginerelies on the engine's own endpoint security check, and its duplicated branches are gone.fetch_url, so the SSRF guard lives in one place with one IP classifier. They reusedocuments::htmlfor titles and entities, and return typed errors instead of strings.policy/,sanitize/,pii/,markers/anditem/, with no behaviour change.Small behaviour changes in sources:
198.18.0.0/15. The old code only blocked a/16, although its comment said/15.<title>falls back to the page URL.fetch_urlsends the readers' user agent.Validation
Commands run in the worktree, with their outcome:
cargo fmt --all -- --check: cleancargo clippy --all-targets --all-features -- -D warnings: cleancargo build --all-targets --all-features: okcargo test --all-features: 932 passed, 0 failedcargo test(default features): 273 passed, 0 failedcargo clippy -p tinymemory-integrations --no-default-features --features <each of: none, cortex, documents, documents-office, sources, sources-network, safety, legacy-import> --all-targets -- -D warnings: all cleanRUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features: cleancargo run -p tinymemory-integrations --example basic: lists both engines, buildstinyhumans, refusescortexdbwithout a credentialcargo tree -p tinymemory-api -e normal,build: no rusqlite, reqwest, regex or tokioThe live CortexDB tests (
live_*) were not run, because they need a server.scripts/cortexdb-live.shnow points at-p tinymemory-integrations.Tests
BLESS_TOOL_CONTRACTS=1).namespacein the arguments, reading a sibling node, forgetting a sibling's id.migratetests on a 250-item v1 fixture: a full run, an idempotent re-run, resuming from a checkpoint, the checkpoint callback, a failure on the first and on the Nth batch, an empty workspace, and a check that the future isSend.Documentation
docs/architecture/with these pages:README,overview,api,api-items,operations,namespacescortex,cortex-wire,cortex-flowstools,integrations,integrations-sources,testingdocs/specs/memory-v2.mdis updated for the three-crate layout, with a new Tools section andmigrate.README.mdis rewritten, and AGENTS.md's structure and release sections are updated.tinymemory-api,tinymemory-toolsandtinymemory-integrations. The cortex, documents, sources, safety and import module READMEs are refreshed.Points for reviewers (found during the audit, not changed here)
Reach::of(place),memory_forgetcan delete memory the agent shares with its team or the root. Confining forgets toReach::exact(place)is a one-line change, if we want it.Segment::sanitized("")andSegment::sanitized("_")both produce_, which contradicts "distinct ids stay distinct".ForgetTarget::Filterthat sets onlyreachis not treated as empty, so it forgets the whole reach. The tools guard against this; raw engine callers do not.openhumanuser agent.Checklist
#[allow(...)],#[ignore], or relaxed lints. There is one scoped#[allow(clippy::expect_used, reason = …)]on the safety regex helper, plus per-helperunwrap_usedallows in tools integration-test helpers..envcontents. The safety test fixtures are fake credential-shaped strings that were already onmain.