diff --git a/.sdd/graph/2026/09/06-113911-s-cpt-ikx.md b/.sdd/graph/2026/09/06-113911-s-cpt-ikx.md new file mode 100644 index 00000000..30a11dcc --- /dev/null +++ b/.sdd/graph/2026/09/06-113911-s-cpt-ikx.md @@ -0,0 +1,35 @@ +--- +type: signal +layer: conceptual +kind: insight +refs: + - id: 20260422-122317-d-stg-qlt + kind: grounded-in + - id: 20260905-111056-s-cpt-ne0 + kind: related + - id: 20260603-172628-d-cpt-fbi + kind: related + - id: 20260617-182152-s-cpt-yve + kind: grounded-in + - id: 20260727-224047-d-cpt-u8o + kind: related +participants: + - Christopher +confidence: low +topics: + - collaboration/coherence + - collaboration/external-systems +summary: 'An extensible project event stream is proposed as a conceptual insight: occurrences (CI runs, deployments, graph entry creation) could be recorded automatically, while consumers connect selected events with graph context for durable capture, notifications, reporting, and decisions-governed action. The idea is grounded in the shared-context aspiration (d-stg-qlt) and the reasoning-record framing (s-cpt-yve), complements the graph-change subscription gap (s-cpt-ne0), and raises an uninterpreted-occurrence boundary with the intake model (d-cpt-fbi). It deliberately does not revise the graph-as-only-durable-record directive (d-cpt-u8o), and remains low-confidence, uncommitted, and awaiting concrete integration scenarios for validation.' +--- + +An extensible project event stream could complement the SDD graph by recording occurrences automatically, while consumers connect selected events with graph context to support durable capture, notifications, reporting, and action governed by recorded decisions. + +Christopher raised the idea from a need for better visibility across participants, serving the shared-context aspiration (d-stg-qlt). Git-synchronized WIP markers convey limited activity, while focus decisions express commitments of attention. The proposed stream reaches beyond work claims: SDD itself, working sessions, and external integrations could publish events such as exploration starting, CI running, deployments completing, or graph entries being created. Events record what happened without confirmation, mostly through tooling. + +Sessions could query the stream for current context. Background consumers could review changes periodically, for example hourly, combine them with graph context, and synthesize what merits durable capture. Events need not become entries individually. Autonomous action would draw its authority from durable decisions, potentially expressed as procedures; an event supplies the occasion to act, not permission. Entry creation could itself publish an event, connecting the graph back to consumers. + +Consumers could also turn fine-grained events into notifications and reports. Procedures could determine how events are combined with relevant graph material, which people can act on the resulting information, and how to make any need for action explicit. IT infrastructure monitoring and software development are examples; the intended applicability extends to businesses and projects generally. How to select, combine, and route events without overwhelming participants remains open. + +This suggests a broader integration foundation alongside the graph-change subscription gap (s-cpt-ne0). It also raises a boundary with the intake direction (d-cpt-fbi): incoming occurrences could remain useful before any consumer interprets them into graph material. The distinction between a reasoning record and external execution records already raised in s-cpt-yve helps frame that boundary. A persistent, replayable project history would need reconciliation with d-cpt-u8o's declaration that the graph is the only durable record; this insight does not revise that directive. + +The idea remains unvalidated and deliberately uncommitted. Replay could help reconstruct project activity, but its scope, retention, event semantics, consumer execution, notification and reporting surfaces, and the limits of autonomous authority remain unsettled. Concrete integration scenarios should test whether the stream improves shared awareness and graph context without duplicating reasoning or expanding SDD into a general automation platform or ERP. diff --git a/.sdd/graph/2026/09/06-121218-d-tac-ccm.md b/.sdd/graph/2026/09/06-121218-d-tac-ccm.md new file mode 100644 index 00000000..ca1f4713 --- /dev/null +++ b/.sdd/graph/2026/09/06-121218-d-tac-ccm.md @@ -0,0 +1,51 @@ +--- +type: decision +layer: tactical +kind: plan +refs: + - id: 20260905-112352-d-tac-nhu + kind: builds-on + - id: 20260902-154750-d-tac-o1s + kind: builds-on + - id: 20260413-142536-d-cpt-ah1 + kind: grounded-in + - id: 20260905-111056-s-cpt-ne0 + kind: related +participants: + - Christopher +confidence: high +topics: + - search/index + - portability/runtime + - llm/providers + - llm/observability + - reliability/testing +summary: 'This tactical plan commits to incremental entry indexing: entry publication, provider transport, and search coverage are separated so that independently retryable entry versions make durable progress while concurrent callers share bounded provider batches and search retrieves against a fixed snapshot target, avoiding whole-project embedding runs that can exceed job deadlines without retaining progress. It extends the independent reconciliation operation (20260905-112352-d-tac-nhu) and the per-request batching composition (20260902-154750-d-tac-o1s), applies the mandatory CQRS decomposition contract (20260413-142536-d-cpt-ah1), and explicitly leaves the durable graph-change subscription gap (20260905-111056-s-cpt-ne0) open.' +--- + +Indexing makes durable progress through independently retryable entry versions while concurrent callers share bounded provider batches and search reports coverage of a fixed snapshot target. + +Whole-project embedding followed by one publication can exceed a job deadline without retaining useful progress. This extends the independent reconciliation operation in 20260905-112352-d-tac-nhu and the per-request batching composition in 20260902-154750-d-tac-o1s by separating entry publication, provider transport and search coverage. Increasing project deadlines does not remove the failure mode. + +The [design record](./06-121218-d-tac-ccm/entry-indexing-design.md) carries the source-lifetime contract, public capability proposals, publication semantics, provisional settings and validation exercise. Following 20260413-142536-d-cpt-ah1, commands and handlers own indexing writes, queries and finders own discovery and coverage reads, pure computation stays below orchestration, and the application composes these boundaries. The shared SDD application selects authorized home and dependency snapshots, lets consumer preparation run against that fixed target, then derives actual coverage from published entry versions and retrieves against the same snapshots. Consumers own scheduling, queue state, retries and waiting policy. Preparation returns normally or fails explicitly; only SDD determines completeness and renders coverage metadata and the readable notice. + +## Acceptance criteria + +- [ ] Discovery yields entry requirements incrementally without preparing all chunks, and releases control on early stop, cancellation or error. +- [ ] A revision-bound cursor resumes in stable entry-ID order without rehashing earlier entries; replayed enqueue groups preserve logical work identity. +- [ ] An indexing attempt uses its exact retained entry and attachment source, with project, full entry hash and index configuration distinguishing publication identities. +- [ ] Complete-entry publication, including zero-chunk entries, is atomic for completion queries and retrieval; unpublished chunks are unsearchable. +- [ ] Retrying a published version skips embedding, and concurrent attempts converge without one version suppressing another. +- [ ] Interrupted or failing entries preserve completed siblings, and an entry larger than a provider batch can eventually publish in full. +- [ ] Shared batching bounds buffering, payloads and provider concurrency, routes vectors correctly, and flushes partial batches without further arrivals. +- [ ] Oversized requests make incremental admission progress; cancellation and shutdown release callers without stranding other participants. +- [ ] Each actual provider call records usage once with batch-to-caller correlation, while query embeddings avoid the document backlog. +- [ ] Search selects authorized home and dependency targets before preparation, observes a revision including the caller's successful write, and retrieves against those same targets. +- [ ] After consumer preparation, SDD alone derives complete/incomplete coverage from published entry versions and renders result metadata and the readable notice; ordinary consumer wait expiry makes no completeness claim, incomplete zero-match differs from complete no-match, and preparation failures remain explicit errors. +- [ ] Executable external-consumer and local examples demonstrate one preparation composition point through the existing application and MCP search path before its public signature is finalized. +- [ ] Existing routing and deadline composition are assessed for query separation and provider timeouts, keeping those policies outside the batcher where sufficient. +- [ ] Manifest-read amplification and unpublished-candidate filtering costs are measured and addressed without weakening atomic visibility or hiding published results. +- [ ] A persistent multi-entry interruption exercise demonstrates progress across attempts, skipped completed versions and eventual completion; provider concurrency and latency measurements justify recommended settings. +- [ ] Local execution remains supported, repository and example-module checks pass, and delivery identifies the commit, release status, validation evidence and external consumer adoption steps. + +The useful primitives in checkpoint 0af4e1d7 remain the starting point. Write-triggered integration starts from the existing mutation finalizer and graph-write adapter; the consumer chooses to trigger indexing and guarantees durable scheduling through its write/recovery protocol. No within-entry checkpoints, cross-process batch aggregation, queue implementation, parallel prepare/execute search API or new event subsystem is included. The broader subscription gap 20260905-111056-s-cpt-ne0 stays open. Completion is recorded through the implementation move with a done signal confirmed by the user. diff --git a/.sdd/graph/2026/09/06-121218-d-tac-ccm/entry-indexing-design.md b/.sdd/graph/2026/09/06-121218-d-tac-ccm/entry-indexing-design.md new file mode 100644 index 00000000..60e0ad93 --- /dev/null +++ b/.sdd/graph/2026/09/06-121218-d-tac-ccm/entry-indexing-design.md @@ -0,0 +1,162 @@ +# Entry indexing, provider batching and search coverage + +## Why separate the units + +Whole-project reconciliation currently derives every missing chunk, embeds the aggregate, then persists. A workload that exceeds a job deadline can repeatedly consume provider work without publishing any progress. Raising that deadline leaves the failure mode intact. + +An entry version is the unit of durable scheduling, retry and atomic publication. A provider batch is a bounded transport unit shared by compatible concurrent callers. A search target is the fixed set of entry versions required by selected snapshots. None of these units requires a project job that waits for every child. + +## Source and discovery + +Reuse the public Snapshot type and GraphStore attachment paging. Add a read capability only where existing acquisition cannot express exact retained source or causal freshness. A snapshot lease pairs the graph with immutable attachment bytes and a release operation. The host retains source revisions until jobs finish and may share cached source objects across leases. An indexing job must reopen its exact revision; missing retained input is an explicit failure. It must never substitute a current branch. Search freshness instead selects a revision that includes the successful write and may include later changes. + +Discovery returns a Go iterator over eligible entry requirements and their published status. It hashes one entry at a time and does not prepare a project-wide chunk slice. Cancellation, early termination and source errors stop iteration. Snapshot-based graph loading remains supported; this is not a constant-memory graph claim. + +A cursor contains the source revision, index configuration and last entry ID. Stable ID ordering skips earlier entries before attachment reads or hashing. The host commits cursor advancement with bounded durable enqueue groups. Replaying a group is safe through version-key uniqueness. Queue keys use canonical project ID, entry ID, full entry-state hash and index configuration fingerprint. Revision locates source but does not prevent identical versions across branches sharing publication. Fingerprints must distinguish applicable embedding and derivation configurations. + +## Entry publication + +The application command indexes one explicit entry version. It checks published completion before loading source or embedding, verifies reproducible input, derives all chunks, embeds through the supplied client and publishes the complete version. The store capability atomically governs both completion and retrieval. Unpublished chunks cannot be returned by search. A zero-chunk entry still receives a completion record. Publication is idempotent under concurrent attempts, and a retry after publication but before acknowledgement skips embedding. + +An oversized entry spans multiple bounded provider batches and publishes once. An interrupted entry may repeat its unfinished embedding. Completed siblings remain usable, and a failed entry does not prevent independently scheduled siblings from progressing. No within-entry checkpoint mechanism is included. + +Local stores must validate entire writes before mutation. Disk serialization must not leave partially written live documents after process interruption. Completion is published only after all entry rows exist; retrieval excludes rows absent from published completion. Compatibility reconciliation remains available for existing stores, with its limits documented. The stronger public operation requires explicit publication support. + +## Shared embedding batches + +A process-local stateful decorator wraps one fixed provider configuration. It combines document chunks across callers, flushes on a size bound or the oldest queued chunk's waiting limit, and routes vectors to input positions. Later arrivals do not reset the timer. An underfilled tail flushes without another caller. Requests larger than the input buffer are admitted incrementally. + +Active entry-job concurrency belongs to the consumer. The batcher owns buffering, cross-caller batching, size/window flushing, bounded dispatch, cancellation isolation and vector distribution. Byte and provider-specific token/payload bounds supplement chunk count. A single input that violates a provider constraint fails explicitly. + +Reuse existing composition tools for adjacent policies. Inspection found that embed.Bounded already applies a per-call deadline and EmbedderFunc can route by Purpose. Place Bounded inside the batcher so the timeout applies to each actual provider call. Route document requests to the batcher and query requests to a separately composed client through EmbedderFunc. Both routes must use the same compatible vector space. Distinct rate limiters or concurrency limits belong to consumer composition if needed; a shared limiter must not accidentally restore a document backlog ahead of queries. The checkpoint's built-in query slot and timeout option should be removed if the composition examples prove these existing tools sufficient. + +A caller can stop waiting without canceling a provider batch serving others. The batcher owns shared request lifetime, and the inner deadline decorator bounds provider calls. Provider failure reaches affected callers without an opaque durable retry layer. Shutdown releases queued and waiting callers with explicit outcomes. Test the composed timeout and shutdown behavior, not only the batcher in isolation. + +Observe provider calls inside the decorator. Each actual batch records usage once. Batch identity and participating caller identifiers correlate jobs and attempts without attributing the whole batch to an arbitrary caller or multiplying totals. Job logs separately carry project, attempt, committed-entry progress, elapsed time and failure cause. + +The starting values discussed were 32 chunks, a 10 ms window, two document provider calls and 128 buffered chunks; eight host entry workers, groups of 100 enqueues and a two-second coverage wait were integration starting points. These are provisional, not universal library defaults. A real exercise must establish provider concurrency and latency before recommending them. + +## Consumer preparation and SDD-owned coverage + +The shared SDD application is the sole authority for search coverage. It authorizes and selects the home and requested dependency snapshots first, establishes their required entry versions, and retains the same snapshot and attachment authorities through preparation, coverage reads and retrieval. Later writes do not move this target. Required descriptors are graph-derived values, not caller assertions. The consumer cannot replace target snapshots or mark entries covered. + +Provide one preparation extension point in the existing application search path. The proposed narrow shape is an optional ApplicationOptions.PrepareSearch callback: + +```go +PrepareSearch func(context.Context, SearchTarget) error +``` + +SearchTarget is a read-only view of the fixed target. Its proposed Entries method yields SearchEntryDescriptor values for the target's required versions. It exposes the selected project/revision identities and requested synchronization scope as read-only data needed by composition. The callback cannot mutate the target or write result metadata. Exact method names and the final signature must be demonstrated by executable external-consumer and local examples before they are finalized. Do not add a separate public prepare/execute search API alongside the callback. + +Preparation may schedule durable work, index synchronously, wait under consumer policy, or return immediately. Returning nil means only that preparation returned normally. In particular, an ordinary consumer waiting-budget expiry returns nil without claiming completeness. A preparation failure remains an error. A consumer must distinguish its own elapsed waiting budget from external cancellation and provider/storage failures; catching every context deadline as ordinary expiry would hide failures. + +After preparation, SDD reads published entry versions and derives actual complete/incomplete coverage for the fixed target. It retrieves against those same selected snapshots and owns structured result metadata and the readable incomplete-results notice. Incomplete zero-match differs from complete no-match. Query embedding remains necessary for semantic search, and no text fallback is added. Completion and retrieval both obey the store's atomic publication boundary. + +Scheduling, queue states, retries, waiting budgets and interpreting worker failures stay in consumer composition. No callback result or queue status proves coverage. A local preparation callback can synchronously index required versions. An immediate-return callback still produces SDD-derived incomplete metadata when publication is missing. + +Existing SearchSyncNone/Local/All behavior must have one deliberate preparation path. With no custom callback, the compatibility preparation honors the requested scope. With an explicitly configured callback, the callback is the preparation path and there is no second implicit reconciliation afterwards. Document that distinction and test it through both Application.Search and the MCP search tool; changing MCP to none alone is not the external consumer integration. + +## Concrete composition proposals + +These snippets demonstrate ownership and call flow using proposed target/callback names. They are design examples, not APIs already present in the checkpoint. Turn them into executable examples using the real application and local adapters before finalizing the signature. Application access, sessions and blob stores remain the existing composition inputs. + +An external consumer composition durably submits required descriptors in bounded groups, then optionally waits for consumer-owned notifications: + +```go +options.PrepareSearch = func(ctx context.Context, target sdd.SearchTarget) error { + // Consumer method: commits enqueue groups and returns a request-scoped waiter. + pending, err := scheduler.Enqueue(ctx, target.Entries()) + if err != nil { + return err + } + defer pending.Close() + timer := time.NewTimer(waitBudget) + defer timer.Stop() + select { + case err := <-pending.Result(): + return err + case <-timer.C: + return ctx.Err() + case <-ctx.Done(): + return ctx.Err() + } +} +application, err := sdd.NewApplication(options) +// Handle err before composing the server. +server, err := mcpapp.New(mcpapp.Options{ + Application: application, + SearchSyncMode: sdd.SearchSyncNone, +}) +``` + +Here scheduler and pending are consumer-owned example types, not proposed SDD ports. A nil notification result means preparation is finished, not that the search is complete. The timer branch returns nil only while the parent context remains live. The waiter must have bounded resources and release them when preparation returns. The enqueue and waiting phases both respect request cancellation; the consumer reserves time for subsequent query embedding and retrieval. Durable jobs outlive this request and retain their source independently of the search lease. Integration tests must show that even a premature successful notification cannot make SDD report false completion. + +A local composition uses the same callback and the existing entry-indexing operation: + +```go +options.PrepareSearch = func(ctx context.Context, target sdd.SearchTarget) error { + for entry := range target.Entries() { + runtime, ok := runtimes[entry.Version.Namespace.Project] + if !ok { + return fmt.Errorf("selected project runtime unavailable") + } + if err := runtime.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{ + Entry: entry, + }); err != nil { + return err + } + } + return nil +} +application, err := sdd.NewApplication(options) +// The ordinary Application.Search and MCP search paths use this preparation. +``` + +This local example chooses synchronous preparation for all selected projects. The runtime map belongs to consumer composition and must contain only the authorized target's selected runtimes. Exact-source reads reuse retained snapshot authority, including attachments; they must not reopen a moving branch. Demonstrate this with real local adapters, including zero-chunk entries and cancellation, before accepting the signature. A missing exact-source capability is an integration failure to resolve in the shared implementation, not a reason to work around entry indexing in consumer code. + +An embedding composition can use the existing boundary without adding query routing to the batcher: + +```go +documents, err := embed.NewBatcher(lifetime, + embed.Observed(embed.Bounded(provider, providerTimeout), sink), batchOptions) +// Handle err and arrange batcher shutdown at the consumer lifecycle boundary. +queries := embed.Observed(embed.Bounded(provider, queryTimeout), sink) +routed := embed.EmbedderFunc{ + Space: provider.Fingerprint(), + Run: func(ctx context.Context, req embed.Request) (embed.Result, error) { + switch req.Purpose { + case embed.PurposeDocument: + return documents.Embed(ctx, req) + case embed.PurposeQuery: + return queries.Embed(ctx, req) + default: + return embed.Result{}, fmt.Errorf("unsupported embedding purpose") + } + }, +} +``` + +Provider concurrency safety is a prerequisite of sharing that provider instance. Otherwise compose distinct compatible clients. Measure provider concurrency and query latency with document work active; a separate route is not evidence of latency isolation by itself. + +## Write-triggered integration + +Start from the existing MutationFinalizer and GraphStore write adapter. The consumer decides whether a successful write triggers indexing. Its graph-write/recovery protocol must retain a durable scheduling intent across the interval between canonical commit and finalizer execution. The finalizer can idempotently deliver that intent; background recovery must finish delivery without requiring the originating MCP session to resume. The graph adapter must provide post-write freshness so later search observes a revision containing the write. + +Use the consumer's transaction, outbox or recoverable write protocol to guarantee that delivery. An in-memory finalizer callback alone is insufficient. Periodic reconciliation is a repair path. No new event or subscription subsystem is requested. + +## Checkpoint costs to resolve + +Retain the useful primitives in 0af4e1d7, but assess two costs before completion. Per-entry EntryPublished checks repeatedly load the local manifest. Measure discovery and worker costs as entry count grows, and reuse a generation-validated manifest view or bounded bulk read where that removes repeated work without accepting stale completeness after publication. A view used before preparation must be refreshed for the authoritative post-preparation coverage read. Prefer existing cache and locking boundaries; justify any extra public capability with the measured need. + +Persistent retrieval currently requests all candidates before discarding unpublished rows. Preserve invisibility while evaluating a publication-filtered cached read view or bounded candidate expansion. Compare query memory, candidate count and latency with both many committed rows and interrupted unpublished rows. Do not let unpublished rows fill the candidate limit and hide healthy published results. Neither ignoring unpublished rows for speed nor trusting queue completion is acceptable. + +## Code responsibilities + +Public types live under pkg/application and its public types package. Command and query structs retain the existing CQRS vocabulary. Handlers own embedding and publication side effects. Finders perform discovery and coverage reads. Pure identity, ordering, eligibility and coverage comparison belong below orchestration. The application composes ports and selected snapshots. CLI and MCP shells translate requests and present results. The embedding decorator owns transport batching; existing decorators and consumer routing supply adjacent policies. Only the shared SDD application derives search coverage. + +## Validation and delivery + +Implement and run the concrete external-consumer and local composition examples before finalizing the preparation callback signature. Exercise public behavior: lazy discovery and resumed scans; exact sources, attachments and configuration drift; concurrent idempotent publication and zero-chunk completion; invisible unpublished rows; invalid vectors and storage errors; oversized requests, tail flushes, cancellation and shutdown; usage counted per actual call; fixed dependency coverage and immediate write-then-search; normal preparation wait expiry, preparation errors and false consumer completion notifications; manifest read amplification and unpublished-candidate filtering costs. + +Run a realistic persistent multi-entry, multi-batch workload across interrupted attempts. Show durable progress, skipped completed versions on restart, eventual completion and isolation of a failing entry. Run repository-required checks including the separate example module. Return the commit, actual release status, evidence and precise external consumer adoption checklist. Closing completion is recorded through the implementation move as a done signal whose text the user confirms. + diff --git a/.sdd/graph/wip/20260906-121310-christopher.md b/.sdd/graph/wip/20260906-121310-christopher.md new file mode 100644 index 00000000..e448d62f --- /dev/null +++ b/.sdd/graph/wip/20260906-121310-christopher.md @@ -0,0 +1,7 @@ +--- +entry: 20260906-121218-d-tac-ccm +participant: Christopher +exclusive: true +--- + +Implement entry-version indexing, shared batches and consumer search preparation with SDD-owned coverage. diff --git a/cmd/sdd/serve.go b/cmd/sdd/serve.go index fe186154..bb6a1b59 100644 --- a/cmd/sdd/serve.go +++ b/cmd/sdd/serve.go @@ -428,7 +428,7 @@ func newLocalMutationTargets(project sdd.ProjectID, serverCheckout string) (*loc return nil, nil, nil, fmt.Errorf("mutation target checkout %q does not contain project %s", checkout, project) } targetGraphDir := meta.ResolveGraphDir(checkout, targetCfg) - targetGraph, graphErr := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: project, GraphDir: targetGraphDir}) + targetGraph, graphErr := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: project, GraphDir: targetGraphDir, Branch: target.Branch}) if graphErr != nil { return nil, nil, nil, graphErr } diff --git a/internal/chunking/chunking.go b/internal/chunking/chunking.go index dc13cbda..6491149a 100644 --- a/internal/chunking/chunking.go +++ b/internal/chunking/chunking.go @@ -53,7 +53,7 @@ func IncludeEntry(entry *model.Entry, excludeEmbedded bool) bool { // or non-markdown text is out of scope for v1. // // entryHash is the entry's state hash (from EntryStateHash); new writes mint -// version-qualified chunk IDs (entryID#v-#…) so a changed entry adds a +// version-qualified chunk IDs (entryID#v-#…) so a changed entry adds a // version to the shared store rather than overwriting the old one. Both write // paths (CLI indexer, application vector search) pass the same hash, so the // derived IDs match across paths. @@ -106,6 +106,8 @@ func DeriveChunks(ctx context.Context, entry *model.Entry, entryHash string, spl // definition CanonicalChunk.EntryHash and the CLI manifest hash share. func EntryStateHash(ctx context.Context, entry *model.Entry, attachments AttachmentReader) (string, error) { hh := sha256.New() + // Bump when fixed derivation rules change so publication cannot reuse old chunks. + hh.Write([]byte("sdd-entry-derivation-v1\n")) hh.Write([]byte(entry.Content)) hh.Write([]byte("\n--summary--\n")) hh.Write([]byte(entry.Summary)) @@ -131,3 +133,28 @@ type DiskAttachmentReader struct { func (r DiskAttachmentReader) ReadAttachment(_ context.Context, _ *model.Entry, relPath string) ([]byte, error) { return os.ReadFile(filepath.Join(r.GraphDir, relPath)) } + +// CachedAttachments keeps hashing and derivation on the same bytes within one +// entry operation. Its lifetime is bounded by that entry, not the whole graph. +type CachedAttachments struct { + Reader AttachmentReader + content map[string][]byte +} + +func (r *CachedAttachments) ReadAttachment(ctx context.Context, entry *model.Entry, path string) ([]byte, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + if content, ok := r.content[path]; ok { + return content, nil + } + content, err := r.Reader.ReadAttachment(ctx, entry, path) + if err != nil { + return nil, err + } + if r.content == nil { + r.content = map[string][]byte{} + } + r.content[path] = content + return content, nil +} diff --git a/internal/command/index_search_entry.go b/internal/command/index_search_entry.go new file mode 100644 index 00000000..281709bb --- /dev/null +++ b/internal/command/index_search_entry.go @@ -0,0 +1,5 @@ +package command + +import "github.com/networkteam/sdd/pkg/application/types" + +type IndexSearchEntryCmd = types.IndexSearchEntryCmd diff --git a/internal/finders/search_entries.go b/internal/finders/search_entries.go new file mode 100644 index 00000000..7c569e80 --- /dev/null +++ b/internal/finders/search_entries.go @@ -0,0 +1,76 @@ +package finders + +import ( + "context" + "fmt" + "iter" + + "github.com/networkteam/sdd/internal/chunking" + "github.com/networkteam/sdd/internal/model" + "github.com/networkteam/sdd/internal/query" + "github.com/networkteam/sdd/pkg/application/types" +) + +type EntryPublicationReader interface { + EntryPublished(context.Context, types.SearchEntryVersion) (bool, error) +} + +type SearchEntriesFinder struct { + Graph *model.Graph + Revision string + Namespace types.IndexNamespace + Attachments chunking.AttachmentReader + Store EntryPublicationReader + ExcludeEmbedded bool + EntryIDs []string +} + +func (f SearchEntriesFinder) Discover(ctx context.Context, cursor query.SearchDiscoveryCursor) iter.Seq2[query.SearchEntryRequirement, error] { + return func(yield func(query.SearchEntryRequirement, error) bool) { + fail := func(err error) { yield(query.SearchEntryRequirement{}, err) } + if err := ctx.Err(); err != nil { + fail(err) + return + } + ids, selection, err := model.NormalizeEntrySelection(f.EntryIDs) + if err != nil { + fail(err) + return + } + if cursor != (query.SearchDiscoveryCursor{}) && (cursor.Revision != f.Revision || cursor.Namespace != f.Namespace || cursor.Selection != selection) { + fail(fmt.Errorf("sdd: discovery cursor does not match snapshot and index configuration")) + return + } + if err := f.Graph.EntrySelectionLoadError(ids, cursor.AfterEntryID); err != nil { + fail(err) + return + } + for _, entry := range f.Graph.SelectedEntriesAfter(ids, cursor.AfterEntryID) { + if err := ctx.Err(); err != nil { + fail(err) + return + } + if !chunking.IncludeEntry(entry, f.ExcludeEmbedded) { + continue + } + hash, err := chunking.EntryStateHash(ctx, entry, f.Attachments) + if err != nil { + fail(err) + return + } + version := types.SearchEntryVersion{Namespace: f.Namespace, EntryID: entry.ID, EntryHash: hash} + published, err := f.Store.EntryPublished(ctx, version) + if err != nil { + fail(err) + return + } + item := query.SearchEntryRequirement{ + Entry: types.SearchEntryDescriptor{Version: version, SourceRevision: f.Revision}, Published: published, + Cursor: query.SearchDiscoveryCursor{Revision: f.Revision, Namespace: f.Namespace, AfterEntryID: entry.ID, Selection: selection}, + } + if !yield(item, nil) { + return + } + } + } +} diff --git a/internal/finders/search_entries_selection_test.go b/internal/finders/search_entries_selection_test.go new file mode 100644 index 00000000..1a5203cd --- /dev/null +++ b/internal/finders/search_entries_selection_test.go @@ -0,0 +1,28 @@ +package finders_test + +import ( + "context" + "testing" + + "github.com/networkteam/sdd/internal/finders" + "github.com/networkteam/sdd/internal/model" + "github.com/networkteam/sdd/internal/query" + "github.com/networkteam/sdd/pkg/application/types" +) + +type forbiddenPublicationRead struct{ t *testing.T } + +func (r forbiddenPublicationRead) EntryPublished(context.Context, types.SearchEntryVersion) (bool, error) { + r.t.Fatal("ineligible entry reached publication lookup") + return false, nil +} + +func TestDiscoverySelectionSharesEligibility(t *testing.T) { + const id = "20260101-100000-s-tac-aaa" + for _, ids := range [][]string{nil, {id}} { + finder := finders.SearchEntriesFinder{Graph: model.NewGraph([]*model.Entry{{ID: id, Embedded: true}}), EntryIDs: ids, ExcludeEmbedded: true, Store: forbiddenPublicationRead{t: t}} + for _, err := range finder.Discover(t.Context(), query.SearchDiscoveryCursor{}) { + t.Fatalf("ineligible entry yielded: %v", err) + } + } +} diff --git a/internal/handlers/index_search_entry.go b/internal/handlers/index_search_entry.go new file mode 100644 index 00000000..ead95839 --- /dev/null +++ b/internal/handlers/index_search_entry.go @@ -0,0 +1,77 @@ +package handlers + +import ( + "context" + "fmt" + + "github.com/networkteam/sdd/internal/chunking" + "github.com/networkteam/sdd/internal/command" + "github.com/networkteam/sdd/internal/model" + "github.com/networkteam/sdd/internal/textsplitter" + "github.com/networkteam/sdd/pkg/application/types" + "github.com/networkteam/sdd/pkg/llm/embed" + "github.com/networkteam/slogutils" +) + +type EntryPublisher interface { + EntryPublished(context.Context, types.SearchEntryVersion) (bool, error) + PublishEntry(context.Context, types.SearchEntryVersion, []types.IndexedChunk) error +} + +type SearchEntryHandler struct { + Store EntryPublisher + Embedder embed.Embedder + Entry *model.Entry + Attachments chunking.AttachmentReader +} + +func (h SearchEntryHandler) Index(ctx context.Context, cmd command.IndexSearchEntryCmd) error { + if err := ctx.Err(); err != nil { + return err + } + key := cmd.Entry.Version + published, err := h.Store.EntryPublished(ctx, key) + if err != nil || published { + return err + } + attachments := &chunking.CachedAttachments{Reader: h.Attachments} + hash, err := chunking.EntryStateHash(ctx, h.Entry, attachments) + if err != nil { + return err + } + if hash != key.EntryHash { + return fmt.Errorf("sdd: pinned entry content does not match descriptor") + } + chunks, err := chunking.DeriveChunks(ctx, h.Entry, hash, textsplitter.NewSplitter(), attachments) + if err != nil { + return err + } + rows := make([]types.IndexedChunk, len(chunks)) + if len(chunks) > 0 { + texts := make([]string, len(chunks)) + for i, chunk := range chunks { + texts[i] = chunk.Chunk.Text + } + result, err := h.Embedder.Embed(ctx, embed.Request{Purpose: embed.PurposeDocument, Texts: texts}) + if err != nil { + return err + } + if len(result.Vectors) != len(chunks) { + return fmt.Errorf("sdd: embedder returned %d vectors for %d chunks", len(result.Vectors), len(chunks)) + } + for i, chunk := range chunks { + rows[i] = types.IndexedChunk{Chunk: chunking.CanonicalChunk(h.Entry.ID, hash, chunk), Vector: result.Vectors[i]} + } + } + if err := types.ValidateEntryPublication(key, rows); err != nil { + return err + } + if err := h.Store.PublishEntry(ctx, key, rows); err != nil { + return err + } + slogutils.FromContext(ctx).DebugContext(ctx, "published search entry", "project", key.Namespace.Project, "entry", key.EntryID, "hash", key.EntryHash, "chunks", len(rows)) + if cmd.OnPublished != nil { + cmd.OnPublished(key.EntryID, len(rows)) + } + return nil +} diff --git a/internal/handlers/search_index.go b/internal/handlers/search_index.go index b10da66c..4654ba30 100644 --- a/internal/handlers/search_index.go +++ b/internal/handlers/search_index.go @@ -36,9 +36,39 @@ type SearchIndexHandler struct { type versionKey struct{ entryID, entryHash string } func (h *SearchIndexHandler) complete(ctx context.Context, entries []*model.Entry, hashes map[string]string, skip func(types.CanonicalChunk) bool) error { - if err := h.embedEntries(ctx, h.Namespace, entries, hashes, skip); err != nil { - return err + if store, ok := h.Store.(EntryPublisher); ok { + for _, entry := range entries { + hash := hashes[entry.ID] + if hash == "" { + var err error + hash, err = chunking.EntryStateHash(ctx, entry, h.Attachments) + if err != nil { + return err + } + } + handler := SearchEntryHandler{Store: store, Embedder: h.Embedder, Entry: entry, Attachments: h.Attachments} + cmd := command.IndexSearchEntryCmd{ + Entry: types.SearchEntryDescriptor{Version: types.SearchEntryVersion{Namespace: h.Namespace, EntryID: entry.ID, EntryHash: hash}, SourceRevision: h.Revision}, + OnPublished: func(id string, count int) { + h.entries++ + h.chunks += count + if h.cmd.OnEntryIndexed != nil { + h.cmd.OnEntryIndexed(id, count) + } + }, + } + if err := handler.Index(ctx, cmd); err != nil { + return err + } + } + } else { + for _, entry := range entries { + if err := h.embedEntries(ctx, h.Namespace, []*model.Entry{entry}, hashes, skip); err != nil { + return err + } + } } + if h.cmd.OnComplete != nil { h.cmd.OnComplete(h.Revision, h.entries, h.chunks) } diff --git a/internal/index/ids.go b/internal/index/ids.go index f3fe351f..1298f39a 100644 --- a/internal/index/ids.go +++ b/internal/index/ids.go @@ -44,21 +44,12 @@ func AttachmentChunkID(entryID, attachmentPath string, n int) string { return fmt.Sprintf("%s#attach-%s-%d", entryID, short, n) } -// VersionSegment derives the short version tag embedded in a versioned chunk ID -// from an entry-state hash — the first 8 hex chars, enough to distinguish an -// entry's stored versions while keeping IDs bounded. It is an identity tag -// only: the FULL entry hash lives in the row's entry_hash metadata and in the -// manifest, and that is what read-time freshness compares against. A hash -// shorter than 8 chars (only in tests) is used whole. -func VersionSegment(entryHash string) string { - if len(entryHash) >= 8 { - return entryHash[:8] - } - return entryHash -} +// VersionSegment preserves the full hash so different published versions +// cannot overwrite one another through a truncated chunk identity. +func VersionSegment(entryHash string) string { return entryHash } // SummaryChunkIDVersioned is the version-qualified summary chunk ID: -// entryID#v-#summary. New writes mint versioned IDs so a changed entry +// entryID#v-#summary. New writes mint versioned IDs so a changed entry // adds a version rather than overwriting the old one — two branches holding // different versions of one entry each own their own rows in the shared store. func SummaryChunkIDVersioned(entryID, entryHash string) string { @@ -66,13 +57,13 @@ func SummaryChunkIDVersioned(entryID, entryHash string) string { } // BodyChunkIDVersioned is the version-qualified n-th body chunk ID: -// entryID#v-#body-N. +// entryID#v-#body-N. func BodyChunkIDVersioned(entryID, entryHash string, n int) string { return fmt.Sprintf("%s#v-%s#body-%d", entryID, VersionSegment(entryHash), n) } // AttachmentChunkIDVersioned is the version-qualified n-th attachment chunk ID: -// entryID#v-#attach--N. +// entryID#v-#attach--N. func AttachmentChunkIDVersioned(entryID, entryHash, attachmentPath string, n int) string { h := sha256.Sum256([]byte(attachmentPath)) short := hex.EncodeToString(h[:3]) diff --git a/internal/index/index.go b/internal/index/index.go index 0c4abb74..1674cab3 100644 --- a/internal/index/index.go +++ b/internal/index/index.go @@ -14,6 +14,7 @@ import ( "context" "errors" "fmt" + "math" "os" "path/filepath" "strconv" @@ -87,6 +88,8 @@ type Hit struct { type Index struct { db *chromem.DB coll *chromem.Collection + published *chromem.Collection + manifest *Manifest indexDir string // root of the index storage tree (passed to Open) chromemDir string // sub-directory chromem-go writes its gob files into // dirty records whether a mutation touched the collection during this @@ -140,7 +143,11 @@ func loadStore(indexDir string) (*Index, error) { if err != nil { return nil, fmt.Errorf("get/create collection %q: %w", CollectionName, err) } - return &Index{db: db, coll: coll, indexDir: indexDir, chromemDir: chromemDir}, nil + manifest, err := LoadManifest(indexDir) + if err != nil { + return nil, err + } + return &Index{db: db, coll: coll, indexDir: indexDir, chromemDir: chromemDir, manifest: manifest}, nil } // ensureStoreDir creates the store directory tree so the advisory lock file @@ -187,6 +194,25 @@ func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []s if entryID == "" { return errors.New("entryID is required") } + dims := 0 + if i.indexDir != "" { + manifest, err := LoadManifest(i.indexDir) + if err != nil { + return err + } + for _, state := range manifest.Entries { + ids := state.AllChunkIDs() + if len(ids) == 0 { + continue + } + doc, err := i.coll.GetByID(ctx, ids[0]) + if err != nil { + return err + } + dims = len(doc.Embedding) + break + } + } for j, r := range rows { if r.EntryID != entryID { return fmt.Errorf("row %d: entry id %q does not match %q", j, r.EntryID, entryID) @@ -194,6 +220,18 @@ func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []s if len(r.Embedding) == 0 { return fmt.Errorf("row %d (chunk %s): embedding is empty", j, r.ChunkID) } + if dims == 0 { + dims = len(r.Embedding) + } + if len(r.Embedding) != dims { + return fmt.Errorf("inconsistent vector dimensions") + } + for _, v := range r.Embedding { + if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) { + return fmt.Errorf("non-finite vector") + } + } + } if len(oldChunkIDs) > 0 { @@ -201,6 +239,7 @@ func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []s return fmt.Errorf("delete old chunks for %s: %w", entryID, err) } i.dirty = true + i.published = nil } if len(rows) == 0 { @@ -216,10 +255,11 @@ func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []s Content: r.Text, }) } - if err := i.coll.AddDocuments(ctx, docs, 1); err != nil { + if err := i.addDocuments(ctx, docs); err != nil { return fmt.Errorf("add chunks for %s: %w", entryID, err) } i.dirty = true + i.published = nil return nil } @@ -233,6 +273,7 @@ func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error { return err } i.dirty = true + i.published = nil return nil } @@ -243,24 +284,27 @@ func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([ if len(embedding) == 0 { return nil, errors.New("query embedding is empty") } - count := i.coll.Count() + collection, err := i.publishedCollection(ctx) + if err != nil { + return nil, err + } + count := collection.Count() if count == 0 { return nil, nil } - if nResults > count { - nResults = count - } + nResults = min(nResults, count) if nResults <= 0 { - nResults = 0 + return nil, nil } - results, err := i.coll.QueryEmbedding(ctx, embedding, nResults, nil, nil) + results, err := collection.QueryEmbedding(ctx, embedding, nResults, nil, nil) if err != nil { return nil, fmt.Errorf("vector query: %w", err) } hits := make([]Hit, 0, len(results)) - for _, r := range results { - hits = append(hits, hitFromResult(r)) + for _, result := range results { + hits = append(hits, hitFromResult(result)) } + return hits, nil } diff --git a/internal/index/index_test.go b/internal/index/index_test.go index e3cd9b8f..a7c79741 100644 --- a/internal/index/index_test.go +++ b/internal/index/index_test.go @@ -181,6 +181,11 @@ func TestIndex_PersistRoundTrip(t *testing.T) { if err := idx.UpsertEntry(ctx, "entry-P", nil, rows); err != nil { t.Fatal(err) } + manifest := &Manifest{Version: 2, Entries: map[string]EntryState{}} + manifest.AddVersion("entry-P", EntryVersion{Hash: "published", ChunkIDs: []string{"entry-P#summary"}}) + if err := manifest.Save(dir); err != nil { + t.Fatal(err) + } // Verify chromem subdirectory exists at the expected place. if _, err := os.Stat(filepath.Join(dir, "chromem")); err != nil { t.Errorf("chromem dir not created: %v", err) diff --git a/internal/index/manifest.go b/internal/index/manifest.go index 9e8adfcc..074c0709 100644 --- a/internal/index/manifest.go +++ b/internal/index/manifest.go @@ -2,6 +2,7 @@ package index import ( "encoding/json" + "errors" "fmt" "os" "path/filepath" @@ -170,13 +171,18 @@ func (m *Manifest) Save(indexDir string) error { if err != nil { return fmt.Errorf("marshal manifest: %w", err) } - if err := os.WriteFile(tmp, data, 0o644); err != nil { + file, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { return fmt.Errorf("write manifest tmp: %w", err) } + _, writeErr := file.Write(data) + if err := errors.Join(writeErr, file.Sync(), file.Close()); err != nil { + return err + } if err := os.Rename(tmp, final); err != nil { return fmt.Errorf("rename manifest: %w", err) } - return nil + return syncPublicationDirectory(indexDir) } // AddVersion records a version for an entry (monotonic accumulation). A version diff --git a/internal/index/manifest_cache.go b/internal/index/manifest_cache.go new file mode 100644 index 00000000..6d96f21b --- /dev/null +++ b/internal/index/manifest_cache.go @@ -0,0 +1,48 @@ +package index + +import ( + "encoding/json" + "errors" + "io/fs" + "os" +) + +// ManifestCache reads one immutable publication file per identity change. +// Callers serialize access. Atomic rename makes file identity the freshness +// token even when a writer dies before updating the index generation marker. +type ManifestCache struct { + identity fs.FileInfo + manifest *Manifest + loads int +} + +func (c *ManifestCache) Read(dir string) (_ *Manifest, err error) { + file, err := os.Open(manifestPath(dir)) + if errors.Is(err, fs.ErrNotExist) { + c.identity, c.manifest = nil, nil + return &Manifest{Entries: map[string]EntryState{}}, nil + } + if err != nil { + return nil, err + } + defer func() { err = errors.Join(err, file.Close()) }() + identity, err := file.Stat() + if err != nil { + return nil, err + } + if c.identity != nil && os.SameFile(c.identity, identity) && c.identity.ModTime().Equal(identity.ModTime()) && c.identity.Size() == identity.Size() { + return c.manifest, nil + } + var manifest Manifest + if err := json.NewDecoder(file).Decode(&manifest); err != nil { + return nil, err + } + if manifest.Entries == nil { + manifest.Entries = map[string]EntryState{} + } + c.identity, c.manifest = identity, &manifest + c.loads++ + return c.manifest, nil +} + +func (c *ManifestCache) Loads() int { return c.loads } diff --git a/internal/index/manifest_cache_test.go b/internal/index/manifest_cache_test.go new file mode 100644 index 00000000..f49d2a85 --- /dev/null +++ b/internal/index/manifest_cache_test.go @@ -0,0 +1,78 @@ +package index_test + +import ( + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/networkteam/sdd/internal/index" +) + +func cacheManifest(n int) *index.Manifest { + m := &index.Manifest{Version: 1, Entries: map[string]index.EntryState{}} + for i := range n { + m.AddVersion(fmt.Sprintf("entry-%d", i), index.EntryVersion{Hash: "hash", Fingerprint: "space"}) + } + return m +} + +func TestManifestCacheRefreshAndReadAmplification(t *testing.T) { + dir := t.TempDir() + m := cacheManifest(10000) + if err := m.Save(dir); err != nil { + t.Fatal(err) + } + cache := &index.ManifestCache{} + for range 10000 { + loaded, err := cache.Read(dir) + if err != nil || len(loaded.Entries) != 10000 { + t.Fatalf("read=%v", err) + } + } + if cache.Loads() != 1 { + t.Fatalf("10000 presence reads decoded manifest %d times", cache.Loads()) + } + m.AddVersion("new", index.EntryVersion{Hash: "new", Fingerprint: "space"}) + if err := m.Save(dir); err != nil { + t.Fatal(err) + } + loaded, err := cache.Read(dir) + if err != nil || len(loaded.Entries) != 10001 || cache.Loads() != 2 { + t.Fatalf("publication refresh=%v loads=%d", err, cache.Loads()) + } + if err := os.WriteFile(filepath.Join(dir, "manifest.json"), []byte("broken"), 0644); err != nil { + t.Fatal(err) + } + if _, err := cache.Read(dir); err == nil { + t.Fatal("storage corruption hidden by cache") + } +} + +func BenchmarkManifestRead(b *testing.B) { + dir := b.TempDir() + if err := cacheManifest(10000).Save(dir); err != nil { + b.Fatal(err) + } + b.Run("decode-every-check", func(b *testing.B) { + b.ReportAllocs() + for b.Loop() { + if _, err := index.LoadManifest(dir); err != nil { + b.Fatal(err) + } + } + }) + b.Run("cached-presence-check", func(b *testing.B) { + cache := &index.ManifestCache{} + if _, err := cache.Read(dir); err != nil { + b.Fatal(err) + } + b.ReportAllocs() + b.ResetTimer() + for b.Loop() { + if _, err := cache.Read(dir); err != nil { + b.Fatal(err) + } + } + }) +} diff --git a/internal/index/publication.go b/internal/index/publication.go new file mode 100644 index 00000000..c32536fd --- /dev/null +++ b/internal/index/publication.go @@ -0,0 +1,122 @@ +package index + +import ( + "context" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + + chromem "github.com/philippgille/chromem-go" +) + +// Stage provider-owned serialization outside the live tree: chromem writes +// files in place, so a killed writer must not leave a truncated live gob. +func (i *Index) addDocuments(ctx context.Context, docs []chromem.Document) (err error) { + if i.indexDir == "" { + return i.coll.AddDocuments(ctx, docs, 1) + } + stage, err := os.MkdirTemp(i.indexDir, ".entry-") + if err != nil { + return err + } + defer func() { err = errors.Join(err, os.RemoveAll(stage)) }() + staged, err := loadStore(stage) + if err != nil { + return err + } + if err := staged.coll.AddDocuments(ctx, docs, 1); err != nil { + return err + } + err = filepath.WalkDir(staged.chromemDir, func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if err := ctx.Err(); err != nil { + return err + } + relative, err := filepath.Rel(staged.chromemDir, path) + if err != nil { + return err + } + target := filepath.Join(i.chromemDir, relative) + if entry.IsDir() { + return os.MkdirAll(target, 0o700) + } + file, err := os.Open(path) + if err != nil { + return err + } + syncErr := file.Sync() + if err := errors.Join(syncErr, file.Close()); err != nil { + return err + } + return os.Rename(path, target) + }) + if err != nil { + return err + } + if err := filepath.WalkDir(i.chromemDir, func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() { + return syncPublicationDirectory(path) + } + return nil + }); err != nil { + return err + } + loaded, err := loadStore(i.indexDir) + if err != nil { + return err + } + i.db, i.coll = loaded.db, loaded.coll + return nil +} + +// Build the published read view once per store snapshot, before nearest-N +// selection, so unpublished rows neither fill the hit limit nor require an +// all-candidate result allocation on each query. +func (i *Index) publishedCollection(ctx context.Context) (*chromem.Collection, error) { + if i.indexDir == "" { + return i.coll, nil + } + if i.published != nil { + return i.published, nil + } + manifest := i.manifest + db := chromem.NewDB() + collection, err := db.GetOrCreateCollection(CollectionName, nil, embedFuncStub) + if err != nil { + return nil, err + } + for _, state := range manifest.Entries { + for _, version := range state.Versions { + for _, id := range version.ChunkIDs { + doc, err := i.coll.GetByID(ctx, id) + if err != nil { + return nil, err + } + if doc.Metadata[MetaEntryHash] != "" && doc.Metadata[MetaEntryHash] != version.Hash { + return nil, fmt.Errorf("index: published chunk version mismatch") + } + doc.Metadata[MetaEntryHash] = version.Hash + if err := collection.AddDocument(ctx, doc); err != nil { + return nil, err + } + } + } + } + i.published = collection + return collection, nil +} + +func syncPublicationDirectory(path string) error { + directory, err := os.Open(path) + if err != nil { + return err + } + return errors.Join(directory.Sync(), directory.Close()) +} diff --git a/internal/index/store.go b/internal/index/store.go index e61ff9d1..9b20a031 100644 --- a/internal/index/store.go +++ b/internal/index/store.go @@ -255,20 +255,18 @@ func readGenerationMarker(indexDir string) (value uint64, present bool, err erro return n, true, nil } -// storeGeneration returns a token that changes whenever the store is written. -// It prefers the explicit marker (bumped by every mutating WriteStore); for a -// legacy store that has not been written since the upgrade it falls back to the -// manifest sidecar's identity (size+mtime), which the atomic manifest save -// always changes. An empty store (no marker, no manifest) is generation 0. -// -// The fallback means an unchanged legacy store still caches — any write, which -// rewrites the manifest, invalidates it — and the first local write then -// creates the marker and takes over. A store with neither marker nor manifest -// reloads until the first write, never wrongly reusing a stale snapshot. +// storeGeneration includes publication changes that precede the marker update. func storeGeneration(indexDir string) (uint64, error) { if n, present, err := readGenerationMarker(indexDir); err != nil { return 0, err } else if present { + info, err := os.Stat(manifestPath(indexDir)) + if err != nil && !os.IsNotExist(err) { + return 0, err + } + if err == nil { + return n ^ uint64(info.ModTime().UnixNano()) ^ (uint64(info.Size()) << 20), nil + } return n, nil } info, err := os.Stat(manifestPath(indexDir)) @@ -286,9 +284,10 @@ func storeGeneration(indexDir string) (uint64, error) { // A caller shares one per store directory and guards it with its own mutex — // SnapshotCache carries no locking of its own. type SnapshotCache struct { - index *Index - gen uint64 - valid bool + index *Index + gen uint64 + valid bool + manifestIdentity os.FileInfo } // ReadCached runs fn against a store snapshot under the shared lock, reusing @@ -315,12 +314,20 @@ func ReadCached(ctx context.Context, indexDir string, cache *SnapshotCache, fn f if err != nil { return false, err } - if !cache.valid || cache.index == nil || cache.gen != gen { + identity, statErr := os.Stat(manifestPath(indexDir)) + if statErr != nil && !os.IsNotExist(statErr) { + return false, statErr + } + sameManifest := identity == nil && cache.manifestIdentity == nil + if identity != nil && cache.manifestIdentity != nil { + sameManifest = os.SameFile(identity, cache.manifestIdentity) && identity.ModTime().Equal(cache.manifestIdentity.ModTime()) && identity.Size() == cache.manifestIdentity.Size() + } + if !cache.valid || cache.index == nil || cache.gen != gen || !sameManifest { store, err := loadStore(indexDir) if err != nil { return false, err } - cache.index, cache.gen, cache.valid = store, gen, true + cache.index, cache.gen, cache.valid, cache.manifestIdentity = store, gen, true, identity reloaded = true } return reloaded, fn(cache.index) diff --git a/internal/index/store_test.go b/internal/index/store_test.go index e3a3539b..973b328d 100644 --- a/internal/index/store_test.go +++ b/internal/index/store_test.go @@ -168,20 +168,29 @@ func TestStoreGeneration(t *testing.T) { t.Errorf("legacy generation not stable across reads: %d vs %d", again, legacyGen) } - // The explicit marker takes precedence and increments per write, so a store - // that has been written since upgrade no longer depends on mtime resolution. if err := bumpGeneration(dir); err != nil { t.Fatal(err) } - if g, _ := storeGeneration(dir); g != 1 { - t.Errorf("generation after first bump = %d, want 1 (marker wins over identity fallback)", g) + first, err := storeGeneration(dir) + if err != nil || first == legacyGen { + t.Fatalf("generation did not change: %v", err) } if err := bumpGeneration(dir); err != nil { t.Fatal(err) } - if g, _ := storeGeneration(dir); g != 2 { - t.Errorf("generation after second bump = %d, want 2", g) + second, err := storeGeneration(dir) + if err != nil || second == first { + t.Fatalf("generation did not change: %v", err) } + m.AddVersion("another", EntryVersion{Hash: "h2"}) + if err := m.Save(dir); err != nil { + t.Fatal(err) + } + afterManifest, err := storeGeneration(dir) + if err != nil || afterManifest == second { + t.Fatalf("manifest publication without generation bump was missed: %v", err) + } + } func TestMigrateDir_EmptyManifestIsNoop(t *testing.T) { diff --git a/internal/model/entry_order.go b/internal/model/entry_order.go new file mode 100644 index 00000000..a3bc4923 --- /dev/null +++ b/internal/model/entry_order.go @@ -0,0 +1,17 @@ +package model + +import ( + "slices" + "strings" +) + +// EntriesAfter returns stable ID order without changing the graph's entry order. +func (g *Graph) EntriesAfter(id string) []*Entry { + entries := slices.Clone(g.Entries) + slices.SortFunc(entries, func(a, b *Entry) int { return strings.Compare(a.ID, b.ID) }) + start, found := slices.BinarySearchFunc(entries, id, func(entry *Entry, id string) int { return strings.Compare(entry.ID, id) }) + if found { + start++ + } + return entries[start:] +} diff --git a/internal/model/entry_selection.go b/internal/model/entry_selection.go new file mode 100644 index 00000000..4feb09c6 --- /dev/null +++ b/internal/model/entry_selection.go @@ -0,0 +1,118 @@ +package model + +import ( + "crypto/sha256" + "fmt" + "io/fs" + "regexp" + "slices" + "strings" +) + +var fullEntryIDShape = regexp.MustCompile(`^[0-9]{8}-[0-9]{6}-[a-z]-[a-z]{3}-[a-z0-9]+$`) + +func ValidateEntryID(id string) error { + parts, err := ParseID(id) + if err != nil || !fullEntryIDShape.MatchString(id) { + return fmt.Errorf("invalid full entry ID %q", id) + } + if _, ok := TypeFromAbbrev[parts.TypeCode]; !ok { + return fmt.Errorf("invalid entry type in %q", id) + } + if _, ok := LayerFromAbbrev[parts.LayerCode]; !ok { + return fmt.Errorf("invalid entry layer in %q", id) + } + return nil +} + +// NormalizeEntrySelection preserves nil as whole-graph scope. +func NormalizeEntrySelection(ids []string) ([]string, string, error) { + if ids == nil { + return nil, "", nil + } + if len(ids) == 0 { + return nil, "", fmt.Errorf("entry selection must not be empty") + } + normalized := slices.Clone(ids) + for _, id := range normalized { + if err := ValidateEntryID(id); err != nil { + return nil, "", err + } + } + slices.Sort(normalized) + normalized = slices.Compact(normalized) + digest := sha256.Sum256([]byte(strings.Join(normalized, "\n"))) + return normalized, fmt.Sprintf("sha256:%x", digest), nil +} + +// EntryIDForArtifactPath resolves an entry document or its attachment owner. +// Non-entry artifacts have no owner; malformed graph paths fail explicitly. +func EntryIDForArtifactPath(logicalPath string) (string, error) { + if !fs.ValidPath(logicalPath) || strings.Contains(logicalPath, `\`) { + return "", fmt.Errorf("invalid artifact path %q", logicalPath) + } + parts := strings.Split(logicalPath, "/") + if len(parts[0]) != 4 || strings.Trim(parts[0], "0123456789") != "" { + return "", nil + } + if len(parts) < 3 { + return "", fmt.Errorf("invalid entry artifact path %q", logicalPath) + } + name := parts[2] + if len(parts) == 3 { + if !strings.HasSuffix(name, ".md") { + return "", fmt.Errorf("invalid entry document path %q", logicalPath) + } + name = strings.TrimSuffix(name, ".md") + } + id := parts[0] + parts[1] + name + if err := ValidateEntryID(id); err != nil { + return "", err + } + canonical, err := IDToRelPath(id) + if err != nil { + return "", err + } + if strings.ReplaceAll(canonical, `\`, "/") != parts[0]+"/"+parts[1]+"/"+name+".md" { + return "", fmt.Errorf("noncanonical entry path %q", logicalPath) + } + return id, nil +} + +func (g *Graph) SelectedEntriesAfter(ids []string, after string) []*Entry { + if ids == nil { + return g.EntriesAfter(after) + } + start, found := slices.BinarySearch(ids, after) + if found { + start++ + } + entries := make([]*Entry, 0, len(ids)-start) + for _, id := range ids[start:] { + if entry := g.ByID[id]; entry != nil { + entries = append(entries, entry) + } + } + return entries +} + +func (g *Graph) EntrySelectionLoadError(ids []string, after string) error { + for _, issue := range g.LoadIssues { + id := issue.Ref + if ValidateEntryID(id) != nil { + var err error + id, err = EntryIDForArtifactPath(issue.Ref) + if err != nil || id == "" { + if ids == nil { + return fmt.Errorf("read entry %s: %s", issue.Ref, issue.Message) + } + continue + } + } + if id <= after || (ids != nil && !slices.Contains(ids, id)) { + continue + } + return fmt.Errorf("read entry %s: %s", issue.Ref, issue.Message) + } + return nil +} diff --git a/internal/proctest/proctest.go b/internal/proctest/proctest.go index 16cbe180..7c376ac4 100644 --- a/internal/proctest/proctest.go +++ b/internal/proctest/proctest.go @@ -232,7 +232,7 @@ func NewWorld(t *testing.T, opts ...Option) *World { WriteEntry(t, cfg.graphDir, entry) } - graph, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "proctest", GraphDir: cfg.graphDir}) + graph, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "proctest", GraphDir: cfg.graphDir, Branch: "main"}) if err != nil { t.Fatal(err) } @@ -252,7 +252,7 @@ func NewWorld(t *testing.T, opts ...Option) *World { if len(cfg.branchDirs) > 0 { targets := branchTargets{fallback: graph, graphs: map[string]sdd.GraphStore{"main": graph}} for branch, dir := range cfg.branchDirs { - store, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "proctest", GraphDir: dir}) + store, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "proctest", GraphDir: dir, Branch: branch}) if err != nil { t.Fatal(err) } diff --git a/internal/query/discover_search_entries.go b/internal/query/discover_search_entries.go new file mode 100644 index 00000000..d0afb3dc --- /dev/null +++ b/internal/query/discover_search_entries.go @@ -0,0 +1,6 @@ +package query + +import "github.com/networkteam/sdd/pkg/application/types" + +type SearchDiscoveryCursor = types.SearchDiscoveryCursor +type SearchEntryRequirement = types.SearchEntryRequirement diff --git a/pkg/application/application.go b/pkg/application/application.go index 8032102b..e95a0aac 100644 --- a/pkg/application/application.go +++ b/pkg/application/application.go @@ -37,6 +37,10 @@ type ApplicationOptions struct { Sessions SessionStore StagedBlobs StagedBlobStore Clock Clock + // PrepareSearch runs once after authorized semantic-search snapshots are + // selected. nil preserves SyncMode preparation. Returning nil claims no + // coverage; SDD reads publication afterwards. Callback failures propagate. + PrepareSearch func(context.Context, SearchTarget) error } // Application resolves current access and dispatches protocol-neutral SDD @@ -44,10 +48,11 @@ type ApplicationOptions struct { // session-addressed method resolves the session's home project from the // session's own record. type Application struct { - access AccessResolver - sessions SessionStore - blobs StagedBlobStore - clock Clock + access AccessResolver + sessions SessionStore + blobs StagedBlobStore + clock Clock + prepareSearch func(context.Context, SearchTarget) error } func NewApplication(options ApplicationOptions) (*Application, error) { @@ -65,10 +70,11 @@ func NewApplication(options ApplicationOptions) (*Application, error) { clock = ClockFunc(time.Now) } return &Application{ - access: options.Access, - sessions: legacyEndStore{options.Sessions}, - blobs: options.StagedBlobs, - clock: clock, + access: options.Access, + sessions: legacyEndStore{options.Sessions}, + blobs: options.StagedBlobs, + clock: clock, + prepareSearch: options.PrepareSearch, }, nil } @@ -304,7 +310,7 @@ func (a *Application) Search(ctx context.Context, identity RequestIdentity, proj if err != nil { return SearchResult{}, err } - selected, err := acquireSnapshotForReadBranch(ctx, runtime, request.Branch) + selected, err := acquireSnapshotForSearch(ctx, runtime, request.Branch, request.IncludesRevision) if err != nil { return SearchResult{}, withSessionBindingTargetError(request.Branch, request.BranchFromSession, err) } @@ -319,10 +325,9 @@ func (a *Application) Search(ctx context.Context, identity RequestIdentity, proj Terms: request.Terms, Phrase: request.Phrase, Filter: filter, IncludeSuperseded: request.IncludeSuperseded, Limit: request.Limit, MaxCitationsPerEntry: request.MaxCitations, } - searchResult, err := runtime.searchSnapshot(ctx, snapshot, selected.store, q) - if err != nil { - return SearchResult{}, err - } + members := []*searchTargetMember{{runtime: runtime, selected: selected}} + target := SearchTarget{state: &searchTargetState{members: members, mode: request.SyncMode}} + defer target.state.closed.Store(true) repos, err := a.selectedDependencies(request.Repos, request.AllRepos, runtime.options.Dependencies) if err != nil { return SearchResult{}, err @@ -332,30 +337,35 @@ func (a *Application) Search(ctx context.Context, identity RequestIdentity, proj if err != nil { return SearchResult{}, err } - member, err := dependency.options.Graph.Current(ctx) - if err != nil { - return SearchResult{}, dependencyUnavailable() - } - memberQuery := q - if request.SyncMode == SearchSyncLocal { - memberQuery.SyncMode = SearchSyncNone - } - memberResult, err := dependency.searchSnapshot(ctx, member, dependency.options.Graph, memberQuery) + member, err := acquireSnapshotForSearch(ctx, dependency, "", "") if err != nil { return SearchResult{}, err } - for i := range memberResult.Entries { - memberResult.Entries[i].RepoID = repoID - } - searchResult.Entries = append(searchResult.Entries, memberResult.Entries...) + defer member.releaseInto(&err) + target.state.members = append(target.state.members, &searchTargetMember{runtime: dependency, selected: member, repoID: repoID}) } + searchResult, coverage, err := a.searchTarget(ctx, target, q) + if err != nil { + return SearchResult{}, err + } + sort.SliceStable(searchResult.Entries, func(i, j int) bool { return searchResult.Entries[i].Score > searchResult.Entries[j].Score }) if limit := q.EffectiveLimit(); len(searchResult.Entries) > limit { searchResult.Entries = searchResult.Entries[:limit] } var rendered bytes.Buffer presenters.RenderSearch(&rendered, searchResult, snapshot.graph) - search := SearchResult{Project: runtime.options.Project, Results: strings.TrimRight(rendered.String(), "\n")} + search := SearchResult{Project: runtime.options.Project, Results: strings.TrimRight(rendered.String(), "\n"), Coverage: coverage} + for _, member := range coverage { + if !member.Complete { + search.Notice = "Search indexing is incomplete for the selected snapshots. Available matches are shown; retry later." + if search.Results != "" { + search.Results += "\n\n" + } + search.Results += search.Notice + break + } + } for _, entry := range searchResult.Entries { if entry.Entry != nil { search.EntryIDs = append(search.EntryIDs, entry.DisplayID()) diff --git a/pkg/application/doc.go b/pkg/application/doc.go index 8ec951bd..073d349d 100644 --- a/pkg/application/doc.go +++ b/pkg/application/doc.go @@ -1,16 +1,55 @@ // Package application owns SDD's protocol-neutral runtime, public request and -// result types, and the infrastructure ports implemented by embedding hosts. -// -// SearchRequest.SyncMode is required: SearchSyncNone reads the existing vector -// index, SearchSyncLocal first reconciles the selected project branch snapshot, -// and SearchSyncAll also reconciles searched dependencies. Text-only searches -// require a mode but do not need index maintenance. Existing synchronous callers -// should pass SearchSyncAll. -// No-sync search verifies only eligible returned candidates against the selected -// snapshot, reading their attachments as needed; stale versions are omitted. -// -// Hosts can call ProjectRuntime.ReconcileSearchIndex independently to warm its -// current graph index, with optional callbacks after persistence. The host owns -// authorization and scheduling. Reconciliation adds missing entry versions; -// it does not remove old ones or watch for subsequent graph changes. +// result types, and the infrastructure ports implemented by consumers. +// +// SearchRequest.SyncMode is required. Without ApplicationOptions.PrepareSearch, +// SearchSyncNone skips maintenance, SearchSyncLocal reconciles the selected +// home snapshot, and SearchSyncAll also reconciles searched dependencies. +// A supplied PrepareSearch callback owns preparation policy for the complete +// authorized SearchTarget. Its error-only result never asserts coverage. +// See the SearchTarget examples for synchronous and external composition. +// +// Semantic search derives coverage from published entry versions after +// preparation. This hashes the target's eligible entries and attachments even +// with SearchSyncNone. Retrieval verifies returned candidates against those +// same snapshots. Legacy adapters without SearchIndexEntryStore retain +// candidate-only verification without coverage metadata or custom preparation. +// Text-only search requires a mode but skips preparation and embedding coverage. +// +// ProjectRuntime.DiscoverSearchEntries streams revision-bound requirements; +// ProjectRuntime.IndexSearchEntry publishes one exact-source version atomically. +// ProjectRuntime.ReconcileSearchIndex remains a synchronous convenience. +// Consumers own authorization, durable source retention, scheduling and retries. +// Reconciliation adds versions; it does not watch for subsequent graph changes. +// +// # Consumer adoption +// +// Compose authorized project runtimes through the existing access resolver and +// register PrepareSearch once. MCP uses the same application. Every selected +// project needs SnapshotReader and SearchIndexEntryStore for custom preparation. +// Preserve SDD's Coverage and readable Notice in the consumer's search response. +// +// In the mutation finalizer or graph-write/recovery adapter, call +// AppliedMutation.AffectedEntryIDs. An empty result means no discovery job. +// Before enqueueing, durably retain a reproducible source and its attachments. +// This may be the finalized Git revision, rather than AppliedMutation.Revision +// from an earlier workspace apply. AffectedEntryIDs establishes no such guarantee. +// The consumer's write/recovery protocol must close any crash gap between commit, +// finalization and durable scheduling; a best-effort finalizer alone is insufficient. +// +// Queue selected IDs for write-triggered discovery and nil for cold search, +// periodic reconciliation or configuration changes. Acquire the exact retained +// source, then call DiscoverSearchEntries for either scope. Persist each cursor +// atomically with durable enqueueing or the record that published work needs no +// enqueue. Deduplicate indexing by full SearchEntryVersion, and run IndexSearchEntry +// with source retention through retries. Queue state never establishes coverage. +// +// Share one document batcher per embedding configuration and process. Compose +// query routing separately and provider deadlines and observation inside it. +// Configure explicit limits and measure provider/query latency in the consumer's +// workload; cross-process limits belong to the consumer. See embed.Batcher. +// +// Deploy publication-aware retrieval before asynchronous writers. The derivation +// schema participates in entry hashes, so prior rows can remain stored while +// current entries require fresh publication. Embedding configuration changes +// must change the fingerprint. Existing retention/rebuild tools own old-row cleanup. package application diff --git a/pkg/application/mutation_entries.go b/pkg/application/mutation_entries.go new file mode 100644 index 00000000..33df92b8 --- /dev/null +++ b/pkg/application/mutation_entries.go @@ -0,0 +1,50 @@ +package application + +import ( + "fmt" + "slices" + + "github.com/networkteam/sdd/internal/model" +) + +// AffectedEntryIDs returns entry selections for post-write discovery, including +// deleted documents and attachment owners. It does not establish durable source: +// queue discovery only after the consumer can reacquire its exact finalized +// revision and attachments. An empty result means no discovery job. +func (m AppliedMutation) AffectedEntryIDs() ([]string, error) { + var ids []string + add := func(path string) error { + id, err := model.EntryIDForArtifactPath(path) + if err != nil { + return err + } + if id != "" { + ids = append(ids, id) + } + return nil + } + for _, change := range m.Batch.Changes { + if change.Document != nil { + if change.Document.LogicalPath != change.LogicalPath { + return nil, fmt.Errorf("sdd: inconsistent mutation document paths") + } + id, err := model.EntryIDForArtifactPath(change.LogicalPath) + if err != nil { + return nil, err + } + if id == "" { + return nil, fmt.Errorf("sdd: structured document has no entry identity") + } + } + if err := add(change.LogicalPath); err != nil { + return nil, err + } + } + for _, attachment := range m.Batch.Attachments { + if err := add(attachment.LogicalPath); err != nil { + return nil, err + } + } + slices.Sort(ids) + return slices.Compact(ids), nil +} diff --git a/pkg/application/mutation_entries_example_test.go b/pkg/application/mutation_entries_example_test.go new file mode 100644 index 00000000..06968f02 --- /dev/null +++ b/pkg/application/mutation_entries_example_test.go @@ -0,0 +1,51 @@ +package application_test + +import ( + "context" + "fmt" + + sdd "github.com/networkteam/sdd/pkg/application" +) + +type indexingDiscoveryFinalizer struct { + retainFinalizedSource func(context.Context, sdd.AppliedMutation) (string, error) + enqueueDiscovery func(context.Context, sdd.ProjectID, string, []string) error +} + +func (indexingDiscoveryFinalizer) Name() string { return "index-discovery" } +func (f indexingDiscoveryFinalizer) Finalize(ctx context.Context, mutation sdd.AppliedMutation) error { + ids, err := mutation.AffectedEntryIDs() + if err != nil { + return err + } + if len(ids) == 0 { + return nil + } + revision, err := f.retainFinalizedSource(ctx, mutation) + if err != nil { + return err + } + if revision == "" { + return fmt.Errorf("finalized source revision is required") + } + return f.enqueueDiscovery(ctx, mutation.Project, revision, ids) +} + +func ExampleAppliedMutation_AffectedEntryIDs() { + mutation := sdd.AppliedMutation{Project: "example", Revision: "workspace-revision", Batch: sdd.MutationBatch{ + Attachments: []sdd.AttachmentMaterialization{{LogicalPath: "2026/01/01-100000-s-tac-aaa/evidence.md"}}, + }} + // The consumer's durable write/recovery protocol makes these effects + // idempotent and retries a crash between source finalization and enqueueing. + finalizer := indexingDiscoveryFinalizer{ + retainFinalizedSource: func(context.Context, sdd.AppliedMutation) (string, error) { return "retained-git-revision", nil }, + enqueueDiscovery: func(_ context.Context, project sdd.ProjectID, revision string, ids []string) error { + fmt.Println(project, revision, ids) + return nil + }, + } + if err := finalizer.Finalize(context.Background(), mutation); err != nil { + panic(err) + } + // Output: example retained-git-revision [20260101-100000-s-tac-aaa] +} diff --git a/pkg/application/mutation_entries_test.go b/pkg/application/mutation_entries_test.go new file mode 100644 index 00000000..d1ee4c9d --- /dev/null +++ b/pkg/application/mutation_entries_test.go @@ -0,0 +1,63 @@ +package application_test + +import ( + "context" + "errors" + "reflect" + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" +) + +func TestAppliedMutationAffectedEntryIDs(t *testing.T) { + const first = "20260101-100000-s-tac-aaa" + const second = "20260101-100000-s-tac-bbb" + const path = "2026/01/01-100000-s-tac-aaa.md" + tests := []struct { + name string + batch sdd.MutationBatch + want []string + fails bool + }{ + {name: "structured and raw duplicates", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: path, Document: &sdd.EntryDocument{LogicalPath: path}}, {LogicalPath: path}}}, want: []string{first}}, + {name: "deletion without document", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: path, Delete: true}}}, want: []string{first}}, + {name: "attachment only", batch: sdd.MutationBatch{Attachments: []sdd.AttachmentMaterialization{{LogicalPath: "2026/01/01-100000-s-tac-aaa/evidence.md"}}}, want: []string{first}}, + {name: "attachment deletion and other owner", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: "2026/01/01-100000-s-tac-bbb/note.txt", Delete: true}, {LogicalPath: path}}}, want: []string{first, second}}, + {name: "no entries", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: "wip/marker.md"}}}}, + {name: "empty"}, + {name: "invalid graph path", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: "2026/01/broken.md"}}}, fails: true}, + {name: "invalid attachment owner", batch: sdd.MutationBatch{Attachments: []sdd.AttachmentMaterialization{{LogicalPath: "2026/01/broken/note.md"}}}, fails: true}, + {name: "path traversal", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: "../outside.md"}}}, fails: true}, + {name: "inconsistent structured path", batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: path, Document: &sdd.EntryDocument{LogicalPath: "other.md"}}}}, fails: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ids, err := (sdd.AppliedMutation{Project: "base", Revision: "workspace-only", Batch: tt.batch}).AffectedEntryIDs() + if (err != nil) != tt.fails { + t.Fatalf("error=%v", err) + } + if !tt.fails && !reflect.DeepEqual(ids, tt.want) { + t.Fatalf("ids=%v want=%v", ids, tt.want) + } + }) + } +} + +func TestDiscoveryFinalizerRequiresDurableSource(t *testing.T) { + sentinel := errors.New("source not yet durable") + called := false + finalizer := indexingDiscoveryFinalizer{ + retainFinalizedSource: func(context.Context, sdd.AppliedMutation) (string, error) { called = true; return "", sentinel }, + enqueueDiscovery: func(context.Context, sdd.ProjectID, string, []string) error { + t.Fatal("queued without durable source") + return nil + }, + } + if err := finalizer.Finalize(t.Context(), sdd.AppliedMutation{}); err != nil || called { + t.Fatalf("empty mutation finalized source: called=%v error=%v", called, err) + } + mutation := sdd.AppliedMutation{Revision: "workspace", Batch: sdd.MutationBatch{Changes: []sdd.DocumentChange{{LogicalPath: "2026/01/01-100000-s-tac-aaa.md"}}}} + if err := finalizer.Finalize(t.Context(), mutation); !errors.Is(err, sentinel) { + t.Fatalf("error=%v", err) + } +} diff --git a/pkg/application/read_api.go b/pkg/application/read_api.go index c91249a5..82d126dd 100644 --- a/pkg/application/read_api.go +++ b/pkg/application/read_api.go @@ -71,6 +71,8 @@ type ShowResult struct { } type SearchRequest struct { + // IncludesRevision asks the source to observe a revision containing this write. + IncludesRevision string // SyncMode is required, including for text-only requests. SyncMode SearchSyncMode Terms []string @@ -88,6 +90,8 @@ type SearchRequest struct { } type SearchResult struct { + Coverage []SearchCoverage + Notice string Project ProjectRef Results string EntryIDs []string diff --git a/pkg/application/read_snapshot.go b/pkg/application/read_snapshot.go new file mode 100644 index 00000000..c15bb551 --- /dev/null +++ b/pkg/application/read_snapshot.go @@ -0,0 +1,105 @@ +package application + +import ( + "context" + "errors" + "fmt" +) + +// AttachmentPageReader is the read portion of GraphStore used by indexing. +type AttachmentPageReader interface { + ReadAttachmentPage(context.Context, string, string, int64, int) (AttachmentPage, error) +} + +// SnapshotReadQuery separates exact job input from causal search freshness. +// ExactRevision selects precisely that revision. IncludesRevision selects a +// branch revision containing the write, which may be newer. They are exclusive. +type SnapshotReadQuery struct { + Branch string + ExactRevision string + IncludesRevision string +} + +// AcquiredSnapshot reuses the canonical snapshot and attachment paging types. +// Attachments must remain fixed at Snapshot.Revision until Release. Release +// is mandatory; the acquirer may share retained objects across many leases. +type AcquiredSnapshot struct { + Snapshot *Snapshot + Attachments AttachmentPageReader + Release func() error +} + +// SnapshotReader is an optional GraphStore capability for pinned reads. Hosts +// retain exact revisions independently of lease lifetime for durable jobs. +// IncludesRevision is a causal guarantee, never lexical revision comparison. +// Readers must honor Branch or reject it; an empty branch selects their current +// authority. Target-scoped readers must validate nonempty branch requests. +type SnapshotReader interface { + AcquireSnapshot(context.Context, SnapshotReadQuery) (*AcquiredSnapshot, error) +} + +func validateAcquiredSnapshot(source *AcquiredSnapshot, project ProjectID, exact string) error { + if source == nil || source.Snapshot == nil || source.Attachments == nil || source.Release == nil { + return fmt.Errorf("sdd: incomplete acquired snapshot") + } + if source.Snapshot.Project() != project || (exact != "" && source.Snapshot.Revision() != exact) { + return fmt.Errorf("sdd: acquired snapshot does not match requested source") + } + return nil +} + +func acquireReadSnapshot(ctx context.Context, graph GraphStore, project ProjectID, q SnapshotReadQuery) (*AcquiredSnapshot, error) { + if q.ExactRevision != "" && q.IncludesRevision != "" { + return nil, fmt.Errorf("sdd: exact and including revisions are exclusive") + } + reader, ok := graph.(SnapshotReader) + if !ok { + return nil, fmt.Errorf("sdd: graph store does not support pinned snapshot reads") + } + source, err := reader.AcquireSnapshot(ctx, q) + if err == nil { + err = validateAcquiredSnapshot(source, project, q.ExactRevision) + } + if err != nil { + if source != nil && source.Release != nil { + err = errors.Join(err, source.Release()) + } + return nil, err + } + return source, nil +} + +type pinnedGraphStore struct { + GraphStore + source *AcquiredSnapshot +} + +func (s pinnedGraphStore) Current(context.Context) (*Snapshot, error) { return s.source.Snapshot, nil } +func (s pinnedGraphStore) ReadAttachmentPage(ctx context.Context, entry, name string, offset int64, limit int) (AttachmentPage, error) { + return s.source.Attachments.ReadAttachmentPage(ctx, entry, name, offset, limit) +} + +func acquireSnapshotForSearch(ctx context.Context, runtime *ProjectRuntime, branch, includes string) (*readSnapshotSelection, error) { + selected, err := acquireSnapshotForReadBranch(ctx, runtime, branch) + if err != nil { + return nil, err + } + if _, ok := selected.store.(SnapshotReader); !ok { + if includes == "" { + return selected, nil + } + err := fmt.Errorf("sdd: source cannot establish read-your-writes freshness") + selected.releaseInto(&err) + return nil, err + } + source, err := acquireReadSnapshot(ctx, selected.store, runtime.Project().ID, SnapshotReadQuery{Branch: branch, IncludesRevision: includes}) + if err != nil { + selected.releaseInto(&err) + return nil, err + } + return &readSnapshotSelection{snapshot: source.Snapshot, store: pinnedGraphStore{GraphStore: selected.store, source: source}, branch: branch, release: func() error { + err := source.Release() + selected.releaseInto(&err) + return err + }}, nil +} diff --git a/pkg/application/search_branch_snapshot_test.go b/pkg/application/search_branch_snapshot_test.go new file mode 100644 index 00000000..794648cb --- /dev/null +++ b/pkg/application/search_branch_snapshot_test.go @@ -0,0 +1,89 @@ +package application_test + +import ( + "context" + "slices" + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/llm" + "github.com/networkteam/sdd/pkg/llm/embed" + "github.com/networkteam/sdd/pkg/local" +) + +type multiBranchSnapshotReader struct { + sdd.GraphStore + selected sdd.SnapshotReader + queries []sdd.SnapshotReadQuery +} + +func (r *multiBranchSnapshotReader) AcquireSnapshot(ctx context.Context, q sdd.SnapshotReadQuery) (*sdd.AcquiredSnapshot, error) { + r.queries = append(r.queries, q) + if q.Branch == "work" { + q.Branch = "" + q.IncludesRevision = "" + return r.selected.AcquireSnapshot(ctx, q) + } + return r.GraphStore.(sdd.SnapshotReader).AcquireSnapshot(ctx, q) +} + +func TestSearchPinnedReadPreservesAuthorizedBranch(t *testing.T) { + _, base, _ := preparedRuntime(t, "base") + _, work, workDir := preparedRuntime(t, "base") + putSearchEntry(t, workDir, "bbb", "Only on selected branch") + reader := &multiBranchSnapshotReader{GraphStore: base, selected: work} + options := sdd.ProjectRuntimeOptions{ + Project: sdd.ProjectRef{ID: "base"}, Graph: reader, SearchIndex: local.NewMemorySearchIndexStore(), + Targets: sdd.TargetAcquirerFunc(func(_ context.Context, target sdd.MutationTarget) (*sdd.AcquiredTarget, error) { + return &sdd.AcquiredTarget{Target: target, Graph: reader, Release: func() error { return nil }}, nil + }), + LLM: llm.RunnerFunc(func(context.Context, llm.Request) (llm.Result, error) { return llm.Result{}, nil }), + Embedder: embed.EmbedderFunc{Space: "test-space", Run: func(_ context.Context, req embed.Request) (embed.Result, error) { + result := embed.Result{Vectors: make([][]float32, len(req.Texts))} + for i := range result.Vectors { + result.Vectors[i] = []float32{1, 1} + } + return result, nil + }}, ExcludeEmbeddedFromIndex: true, + } + runtime, err := sdd.NewProjectRuntime(options) + if err != nil { + t.Fatal(err) + } + options.Graph = work + options.Targets = nil + workRuntime, err := sdd.NewProjectRuntime(options) + if err != nil { + t.Fatal(err) + } + var targetRevision string + app := preparationApp(t, runtime, nil, func(ctx context.Context, target sdd.SearchTarget) error { + count := 0 + for item, err := range target.Entries(ctx) { + if err != nil { + return err + } + count++ + targetRevision = item.Entry.SourceRevision + } + if count != 2 { + t.Errorf("preparation selected %d entries, want work's two", count) + } + // A separate worker uses exact source retained by the selected graph. + return workRuntime.ReconcileSearchIndex(ctx, sdd.ReconcileSearchIndexCmd{}) + }) + result, err := app.Search(t.Context(), sdd.RequestIdentity{Subject: "reader"}, "base", sdd.SearchRequest{Branch: "work", IncludesRevision: "containing-write", SyncMode: sdd.SearchSyncNone, Phrase: "alpha", Terms: []string{"Only"}}) + if !slices.Contains(result.EntryIDs, "20260101-100000-s-tac-bbb") { + t.Fatalf("retrieval=%v", result.EntryIDs) + } + // The source adapter owns the causal guarantee; this fixture records it. + if err != nil { + t.Fatal(err) + } + if len(reader.queries) != 1 || reader.queries[0].Branch != "work" || reader.queries[0].IncludesRevision != "containing-write" { + t.Fatalf("queries=%+v", reader.queries) + } + if len(result.Coverage) != 1 || result.Coverage[0].Revision != targetRevision || result.Coverage[0].Required != 2 || !result.Coverage[0].Complete { + t.Fatalf("coverage=%+v", result.Coverage) + } +} diff --git a/pkg/application/search_entries.go b/pkg/application/search_entries.go new file mode 100644 index 00000000..52258934 --- /dev/null +++ b/pkg/application/search_entries.go @@ -0,0 +1,105 @@ +package application + +import ( + "context" + "errors" + "fmt" + "iter" + + "github.com/networkteam/sdd/internal/chunking" + "github.com/networkteam/sdd/internal/command" + "github.com/networkteam/sdd/internal/finders" + "github.com/networkteam/sdd/internal/handlers" + "github.com/networkteam/sdd/internal/query" +) + +type IndexSearchEntryCmd = command.IndexSearchEntryCmd +type SearchDiscoveryCursor = query.SearchDiscoveryCursor +type SearchEntryRequirement = query.SearchEntryRequirement + +type DiscoverSearchEntriesQuery struct { + // The caller owns the lease and releases it after consuming the iterator. + Source *AcquiredSnapshot + Cursor SearchDiscoveryCursor + // EntryIDs selects canonical full IDs. Nil means all; nonnil empty is invalid. + EntryIDs []string +} + +// DiscoverSearchEntries hashes at most the current entry and never prepares +// chunks. Stop iteration to stop I/O. A returned error ends the sequence. +// Save each cursor atomically with enqueueing its missing descriptor. Cursors +// bind revision, index namespace and the canonical selected ID set. Reordered +// or duplicated IDs preserve scope; nil differs from every explicit selection. +// Malformed IDs fail; valid absent or ineligible IDs yield no requirement. +// Missing attachment bytes and other source/read failures remain errors. +func (r *ProjectRuntime) DiscoverSearchEntries(ctx context.Context, q DiscoverSearchEntriesQuery) iter.Seq2[SearchEntryRequirement, error] { + return func(yield func(SearchEntryRequirement, error) bool) { + if err := validateAcquiredSnapshot(q.Source, r.options.Project.ID, ""); err != nil { + yield(SearchEntryRequirement{}, err) + return + } + ns, err := r.indexNamespace() + if err != nil { + yield(SearchEntryRequirement{}, err) + return + } + store, ok := r.options.SearchIndex.(SearchIndexEntryStore) + if !ok { + yield(SearchEntryRequirement{}, fmt.Errorf("sdd: entry publication capability is required")) + return + } + finder := finders.SearchEntriesFinder{Graph: q.Source.Snapshot.graph, Revision: q.Source.Snapshot.Revision(), Namespace: ns, Attachments: graphStoreAttachmentReader{store: q.Source.Attachments}, Store: store, EntryIDs: q.EntryIDs, ExcludeEmbedded: r.options.ExcludeEmbeddedFromIndex} + finder.Discover(ctx, q.Cursor)(yield) + } +} + +// IndexSearchEntry indexes exact retained source, never the current branch. +// Hosts authorize calls and retry failures. Already published work requires +// neither a source lease nor embedding, including after a lost acknowledgement. +func (r *ProjectRuntime) IndexSearchEntry(ctx context.Context, cmd IndexSearchEntryCmd) (err error) { + store, err := r.entryStore(cmd.Entry) + if err != nil { + return err + } + published, err := store.EntryPublished(ctx, cmd.Entry.Version) + if err != nil || published { + return err + } + source, err := acquireReadSnapshot(ctx, r.options.Graph, r.options.Project.ID, SnapshotReadQuery{ExactRevision: cmd.Entry.SourceRevision}) + if err != nil { + return err + } + defer func() { err = errors.Join(err, source.Release()) }() + return r.indexSearchEntrySource(ctx, source, cmd) +} + +func (r *ProjectRuntime) entryStore(entry SearchEntryDescriptor) (SearchIndexEntryStore, error) { + ns, err := r.indexNamespace() + if err != nil { + return nil, err + } + if entry.SourceRevision == "" || entry.Version.Namespace != ns || entry.Version.EntryID == "" || entry.Version.EntryHash == "" { + return nil, fmt.Errorf("sdd: entry descriptor does not match runtime configuration") + } + store, ok := r.options.SearchIndex.(SearchIndexEntryStore) + if !ok { + return nil, fmt.Errorf("sdd: entry publication capability is required") + } + return store, nil +} + +func (r *ProjectRuntime) indexSearchEntrySource(ctx context.Context, source *AcquiredSnapshot, cmd IndexSearchEntryCmd) error { + store, err := r.entryStore(cmd.Entry) + if err != nil { + return err + } + if err := validateAcquiredSnapshot(source, r.options.Project.ID, cmd.Entry.SourceRevision); err != nil { + return err + } + entry := source.Snapshot.graph.ByID[cmd.Entry.Version.EntryID] + if entry == nil || !chunking.IncludeEntry(entry, r.options.ExcludeEmbeddedFromIndex) { + return fmt.Errorf("sdd: descriptor entry is absent or ineligible") + } + h := handlers.SearchEntryHandler{Store: store, Embedder: r.options.Embedder, Entry: entry, Attachments: graphStoreAttachmentReader{store: source.Attachments}} + return h.Index(ctx, cmd) +} diff --git a/pkg/application/search_entries_test.go b/pkg/application/search_entries_test.go new file mode 100644 index 00000000..0c63f993 --- /dev/null +++ b/pkg/application/search_entries_test.go @@ -0,0 +1,179 @@ +package application_test + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync" + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" +) + +type countingAttachmentPages struct { + sdd.AttachmentPageReader + reads map[string]int + fail error +} + +func (c *countingAttachmentPages) ReadAttachmentPage(ctx context.Context, id, name string, offset int64, limit int) (sdd.AttachmentPage, error) { + c.reads[id]++ + if c.fail != nil { + return sdd.AttachmentPage{}, c.fail + } + return c.AttachmentPageReader.ReadAttachmentPage(ctx, id, name, offset, limit) +} + +func TestEntryDiscoveryIsLazyAndCursorSkipsEarlierHashes(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + putSearchEntry(t, dir, "bbb", "Second") + for _, id := range []string{"aaa", "bbb"} { + att := filepath.Join(dir, "2026", "01", "01-100000-s-tac-"+id, "note.md") + if err := os.MkdirAll(filepath.Dir(att), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(att, []byte("Attached content"), 0644); err != nil { + t.Fatal(err) + } + } + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + pages := &countingAttachmentPages{AttachmentPageReader: source.Attachments, reads: map[string]int{}} + source.Attachments = pages + seq := runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) + if len(pages.reads) != 0 { + t.Fatal("eager discovery") + } + var cursor sdd.SearchDiscoveryCursor + for item, err := range seq { + if err != nil { + t.Fatal(err) + } + cursor = item.Cursor + break + } + if len(pages.reads) != 1 || pages.reads["20260101-100000-s-tac-aaa"] != 1 { + t.Fatalf("reads=%v", pages.reads) + } + count := 0 + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, Cursor: cursor}) { + if err != nil { + t.Fatal(err) + } + count++ + if item.Entry.Version.EntryID != "20260101-100000-s-tac-bbb" { + t.Fatal(item) + } + } + if count != 1 || pages.reads["20260101-100000-s-tac-aaa"] != 1 { + t.Fatalf("resume rehashed earlier entry: %v", pages.reads) + } + cursor.Revision = "other" + for _, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, Cursor: cursor}) { + if err == nil { + t.Fatal("wrong cursor accepted") + } + break + } + canceled, cancel := context.WithCancel(t.Context()) + cancel() + for _, err := range runtime.DiscoverSearchEntries(canceled, sdd.DiscoverSearchEntriesQuery{Source: source}) { + if !errors.Is(err, context.Canceled) { + t.Fatalf("cancel=%v", err) + } + break + } + pages.fail = errors.New("attachment unavailable") + for _, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) { + if !errors.Is(err, pages.fail) { + t.Fatalf("source error=%v", err) + } + break + } +} + +func TestEntryIndexingIdempotencyAndExactSource(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + var descriptor sdd.SearchEntryDescriptor + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) { + if err != nil { + t.Fatal(err) + } + descriptor = item.Entry + break + } + putSearchEntry(t, dir, "aaa", "Changed body") + var wg sync.WaitGroup + for range 8 { + wg.Go(func() { + if err := runtime.IndexSearchEntry(t.Context(), sdd.IndexSearchEntryCmd{Entry: descriptor}); err != nil { + t.Error(err) + } + }) + } + wg.Wait() + if err := source.Release(); err != nil { + t.Fatal(err) + } + // Publication skips source acquisition even after that exact source expires. + if err := runtime.IndexSearchEntry(t.Context(), sdd.IndexSearchEntryCmd{Entry: descriptor}); err != nil { + t.Fatal(err) + } + descriptor.Version.EntryHash = "different" + if err := runtime.IndexSearchEntry(t.Context(), sdd.IndexSearchEntryCmd{Entry: descriptor}); err == nil { + t.Fatal("moving branch substituted for exact source") + } + descriptor.Version.Namespace.Fingerprint = "different-config" + if err := runtime.IndexSearchEntry(t.Context(), sdd.IndexSearchEntryCmd{Entry: descriptor}); err == nil { + t.Fatal("configuration mismatch accepted") + } +} + +func TestZeroChunkEntryPublishes(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + path := filepath.Join(dir, "2026", "01", "01-100000-s-tac-aaa.md") + if err := os.WriteFile(path, []byte("---\ntype: signal\nkind: gap\nlayer: tactical\n---\n"), 0644); err != nil { + t.Fatal(err) + } + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) { + if err != nil { + t.Fatal(err) + } + called := false + err = runtime.IndexSearchEntry(t.Context(), sdd.IndexSearchEntryCmd{Entry: item.Entry, OnPublished: func(_ string, n int) { + called = true + if n != 0 { + t.Fatalf("chunks=%d", n) + } + }}) + if err != nil || !called { + t.Fatalf("publication=%v called=%v", err, called) + } + } + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) { + if err != nil || !item.Published { + t.Fatalf("zero completion=%v %v", item, err) + } + } +} diff --git a/pkg/application/search_index_exercise_test.go b/pkg/application/search_index_exercise_test.go new file mode 100644 index 00000000..6c90f3d5 --- /dev/null +++ b/pkg/application/search_index_exercise_test.go @@ -0,0 +1,285 @@ +package application_test + +import ( + "bufio" + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + provider "github.com/networkteam/sdd/internal/llm/embed" + "github.com/networkteam/sdd/internal/model" + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/llm" + "github.com/networkteam/sdd/pkg/llm/embed" + "github.com/networkteam/sdd/pkg/local" +) + +type exerciseStats struct { + mu sync.Mutex + Calls int `json:"calls"` + Tokens int `json:"tokens"` + Shared int `json:"shared_batches"` + MaxItems int `json:"max_items"` +} + +func (s *exerciseStats) RecordCall(ctx context.Context, stat llm.CallStat) { + s.mu.Lock() + defer s.mu.Unlock() + s.Calls++ + s.Tokens += stat.Usage.InputTokens + s.MaxItems = max(s.MaxItems, stat.Items) + if len(embed.Attribution(ctx).Callers) > 1 { + s.Shared++ + } +} + +type exerciseStore struct { + *local.PersistentSearchIndexStore + fail bool +} + +func (s exerciseStore) PublishEntry(ctx context.Context, v sdd.SearchEntryVersion, rows []sdd.IndexedChunk) error { + if s.fail && strings.HasSuffix(v.EntryID, "bad") { + return errors.New("injected entry publication failure") + } + return s.PersistentSearchIndexStore.PublishEntry(ctx, v, rows) +} + +type exerciseReport struct { + Published int `json:"published"` + Skipped int `json:"skipped"` + Failed int `json:"failed"` + Calls int `json:"calls"` + Tokens int `json:"tokens"` + Shared int `json:"shared_batches"` + MaxItems int `json:"max_items"` + Peak int32 `json:"peak_provider_calls"` + Elapsed time.Duration `json:"elapsed"` + QueryLatency time.Duration `json:"query_latency"` +} + +func TestEntryIndexingExerciseWorker(t *testing.T) { + root := os.Getenv("SDD_INDEX_EXERCISE_ROOT") + if root == "" { + t.Skip("subprocess helper") + } + concurrency, _ := strconv.Atoi(os.Getenv("SDD_INDEX_EXERCISE_CONCURRENCY")) + graph, err := local.NewFilesystemGraphStore(local.FilesystemGraphStoreOptions{Project: "exercise", GraphDir: filepath.Join(root, "graph")}) + if err != nil { + t.Fatal(err) + } + var inner embed.Embedder + liveModel := os.Getenv("SDD_LIVE_EMBED_MODEL") + if liveModel != "" { + inner, err = provider.New(model.EmbeddingConfig{Provider: "ollama", Model: liveModel, OllamaEndpoint: "http://127.0.0.1:11434"}) + if err != nil { + t.Fatal(err) + } + } else { + inner = embed.EmbedderFunc{Space: "scripted", Run: func(ctx context.Context, req embed.Request) (embed.Result, error) { + timer := time.NewTimer(4 * time.Millisecond) + defer timer.Stop() + select { + case <-ctx.Done(): + return embed.Result{}, ctx.Err() + case <-timer.C: + } + r := embed.Result{Vectors: make([][]float32, len(req.Texts)), Usage: llm.Usage{InputTokens: len(req.Texts)}} + for i := range r.Vectors { + r.Vectors[i] = []float32{1, 1} + } + return r, nil + }} + } + stats := &exerciseStats{} + var active, peak atomic.Int32 + tracked := embed.EmbedderFunc{Space: inner.Fingerprint(), Run: func(ctx context.Context, req embed.Request) (embed.Result, error) { + n := active.Add(1) + defer active.Add(-1) + for { + old := peak.Load() + if n <= old || peak.CompareAndSwap(old, n) { + break + } + } + return inner.Embed(ctx, req) + }} + observed := embed.Observed(embed.Bounded(tracked, 2*time.Minute), stats) + batcher, err := embed.NewBatcher(t.Context(), observed, embed.BatchOptions{MaxItems: 32, MaxBytes: 128 * 1024, BufferItems: 4, Window: 10 * time.Millisecond, Concurrency: concurrency}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := batcher.Close(t.Context()); err != nil { + t.Error(err) + } + }() + store := exerciseStore{PersistentSearchIndexStore: local.NewPersistentSearchIndexStore("exercise", filepath.Join(root, "index"), "exercise"), fail: os.Getenv("SDD_INDEX_EXERCISE_FAIL") == "1"} + runtime, err := sdd.NewProjectRuntime(sdd.ProjectRuntimeOptions{Project: sdd.ProjectRef{ID: "exercise"}, Graph: graph, SearchIndex: store, Embedder: batcher, ExcludeEmbeddedFromIndex: true, LLM: llm.RunnerFunc(func(context.Context, llm.Request) (llm.Result, error) { return llm.Result{}, nil })}) + if err != nil { + t.Fatal(err) + } + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + jobs := make(chan sdd.SearchEntryDescriptor, 4) + var mu sync.Mutex + report := exerciseReport{} + started := time.Now() + var workers sync.WaitGroup + for range 8 { + workers.Go(func() { + for entry := range jobs { + ctx := embed.WithCaller(t.Context(), entry.Version.EntryID) + err := runtime.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: entry, OnPublished: func(id string, _ int) { mu.Lock(); report.Published++; fmt.Println("COMMITTED", id); mu.Unlock() }}) + if err != nil { + mu.Lock() + report.Failed++ + mu.Unlock() + if !strings.Contains(err.Error(), "injected entry publication failure") { + t.Error(err) + } + } + } + }) + } + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source}) { + if err != nil { + t.Fatal(err) + } + if item.Published { + report.Skipped++ + continue + } + jobs <- item.Entry + } + close(jobs) + queryStart := time.Now() + if _, err := observed.Embed(t.Context(), embed.Request{Purpose: embed.PurposeQuery, Texts: []string{"How does durable indexing preserve progress?"}}); err != nil { + t.Fatal(err) + } + report.QueryLatency = time.Since(queryStart) + workers.Wait() + report.Elapsed = time.Since(started) + stats.mu.Lock() + report.Calls = stats.Calls + report.Tokens = stats.Tokens + report.Shared = stats.Shared + report.MaxItems = stats.MaxItems + stats.mu.Unlock() + report.Peak = peak.Load() + raw, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "report.json"), raw, 0644); err != nil { + t.Fatal(err) + } +} + +func TestEntryIndexingInterruptionExercise(t *testing.T) { + if testing.Short() { + t.Skip("subprocess interruption exercise") + } + for _, concurrency := range []int{1, 2} { + t.Run(fmt.Sprintf("provider-concurrency-%d", concurrency), func(t *testing.T) { + root := t.TempDir() + dir := filepath.Join(root, "graph") + for i := range 16 { + suffix := fmt.Sprintf("e%02d", i) + sections := 6 + if i == 0 { + sections = 40 + } + var body strings.Builder + for section := range sections { + fmt.Fprintf(&body, "\n## Topic %d\n\n", section) + body.WriteString(strings.Repeat("Independent entry indexing preserves published progress while other entries retry. A search target fixes the versions selected before preparation. ", 12)) + } + putSearchEntry(t, dir, suffix, body.String()) + } + putSearchEntry(t, dir, "bad", "This entry has an injected storage failure.") + run := func(interrupt, fail bool) exerciseReport { + cmd := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestEntryIndexingExerciseWorker$", "-test.timeout=5m") + flag := "0" + if fail { + flag = "1" + } + cmd.Env = append(os.Environ(), "SDD_INDEX_EXERCISE_ROOT="+root, "SDD_INDEX_EXERCISE_CONCURRENCY="+strconv.Itoa(concurrency), "SDD_INDEX_EXERCISE_FAIL="+flag) + output, err := cmd.StdoutPipe() + if err != nil { + t.Fatal(err) + } + var stderr strings.Builder + cmd.Stderr = &stderr + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + scanner := bufio.NewScanner(output) + committed := 0 + var transcript strings.Builder + for scanner.Scan() { + line := scanner.Text() + transcript.WriteString(line + "\n") + if strings.HasPrefix(line, "COMMITTED ") { + committed++ + if interrupt && committed == 2 { + if err := cmd.Process.Kill(); err != nil { + t.Fatal(err) + } + } + } + } + err = cmd.Wait() + if !interrupt && err != nil { + t.Fatalf("worker: %v\n%s\n%s", err, transcript.String(), stderr.String()) + } + if interrupt { + if committed < 2 { + t.Fatalf("no durable progress before interruption: %s %s", transcript.String(), stderr.String()) + } + return exerciseReport{Published: committed} + } + raw, err := os.ReadFile(filepath.Join(root, "report.json")) + if err != nil { + t.Fatal(err) + } + var report exerciseReport + if err := json.Unmarshal(raw, &report); err != nil { + t.Fatal(err) + } + return report + } + interrupted := run(true, true) + resumed := run(false, true) + if resumed.Skipped < 2 || resumed.Published+resumed.Skipped != 16 || resumed.Failed != 1 { + t.Fatalf("restart lost progress or sibling isolation: %+v", resumed) + } + completed := run(false, false) + if completed.Skipped != 16 || completed.Published != 1 || completed.Failed != 0 { + t.Fatalf("final convergence: %+v", completed) + } + noop := run(false, false) + if noop.Skipped != 17 || noop.Published != 0 || noop.Calls != 1 { + t.Fatalf("published versions re-embedded: %+v", noop) + } + t.Logf("model=%q interrupted=%+v resumed=%+v completed=%+v noop=%+v", os.Getenv("SDD_LIVE_EMBED_MODEL"), interrupted, resumed, completed, noop) + }) + } +} diff --git a/pkg/application/search_ports.go b/pkg/application/search_ports.go index 3a6ddbf5..3f51a166 100644 --- a/pkg/application/search_ports.go +++ b/pkg/application/search_ports.go @@ -27,3 +27,16 @@ type SearchIndexStore interface { type SearchIndexEntryManifest interface { IndexedEntries(context.Context, IndexNamespace) ([]StoredEntryRef, error) } + +type SearchEntryVersion = types.SearchEntryVersion +type SearchEntryDescriptor = types.SearchEntryDescriptor + +// SearchIndexEntryStore publishes complete entry versions, including versions +// with no chunks. Unpublished chunks must be invisible to both EntryPublished +// and Nearest. Publication is atomic, durable on success, and idempotent under +// concurrent calls. A returned error may have committed; callers recheck. +// Implementations validate all identities and vectors before publication. +type SearchIndexEntryStore interface { + EntryPublished(context.Context, SearchEntryVersion) (bool, error) + PublishEntry(context.Context, SearchEntryVersion, []IndexedChunk) error +} diff --git a/pkg/application/search_preparation.go b/pkg/application/search_preparation.go new file mode 100644 index 00000000..651b8b98 --- /dev/null +++ b/pkg/application/search_preparation.go @@ -0,0 +1,183 @@ +package application + +import ( + "context" + "fmt" + "iter" + "sync/atomic" + + "github.com/networkteam/sdd/internal/query" +) + +// SearchTarget is the fixed, authorized selection passed to PrepareSearch. +// Its read capabilities expire when Search returns; retain descriptors, not +// this value, for durable jobs. Source retention remains the consumer's duty. +type SearchTarget struct{ state *searchTargetState } + +type searchTargetState struct { + members []*searchTargetMember + mode SearchSyncMode + closed atomic.Bool +} + +type searchTargetMember struct { + runtime *ProjectRuntime + selected *readSnapshotSelection + repoID string +} + +type SearchTargetProject struct { + Project ProjectID + Revision string +} + +func (t SearchTarget) Projects() []SearchTargetProject { + if t.state == nil { + return nil + } + projects := make([]SearchTargetProject, len(t.state.members)) + for i, member := range t.state.members { + projects[i] = SearchTargetProject{Project: member.runtime.Project().ID, Revision: member.selected.snapshot.Revision()} + } + return projects +} + +func (t SearchTarget) SyncMode() SearchSyncMode { + if t.state == nil { + return "" + } + return t.state.mode +} + +// Entries lazily derives required versions, including attachments, and reads +// published presence. Errors terminate iteration and must be propagated by +// preparation callbacks. A Published hint may advance after it was yielded; +// SDD always reads publication again after preparation. Partial iteration does +// not narrow the target whose coverage SDD will check. +func (t SearchTarget) Entries(ctx context.Context) iter.Seq2[SearchEntryRequirement, error] { + return func(yield func(SearchEntryRequirement, error) bool) { + if t.state == nil || t.state.closed.Load() { + yield(SearchEntryRequirement{}, fmt.Errorf("sdd: search target is no longer available")) + return + } + for _, member := range t.state.members { + source := &AcquiredSnapshot{Snapshot: member.selected.snapshot, Attachments: member.selected.store, Release: func() error { return nil }} + for item, err := range member.runtime.DiscoverSearchEntries(ctx, DiscoverSearchEntriesQuery{Source: source}) { + if !yield(item, err) || err != nil { + return + } + } + } + } +} + +// SearchCoverage reports SDD's post-preparation publication read for one fixed +// project snapshot. It contains no consumer queue or retry state. +type SearchCoverage struct { + Project ProjectID `json:"project"` + Revision string `json:"revision"` + Required int `json:"required"` + Published int `json:"published"` + Complete bool `json:"complete"` +} + +func (t SearchTarget) coverage(ctx context.Context) ([]SearchCoverage, error) { + coverage := make([]SearchCoverage, len(t.state.members)) + for i, member := range t.state.members { + coverage[i] = SearchCoverage{Project: member.runtime.Project().ID, Revision: member.selected.snapshot.Revision(), Complete: true} + } + for item, err := range t.Entries(ctx) { + if err != nil { + return nil, err + } + for i := range coverage { + if coverage[i].Project != item.Entry.Version.Namespace.Project { + continue + } + coverage[i].Required++ + if item.Published { + coverage[i].Published++ + } else { + coverage[i].Complete = false + } + break + } + } + return coverage, nil +} + +func (a *Application) prepareSearchTarget(ctx context.Context, target SearchTarget) error { + if a.prepareSearch != nil { + return a.prepareSearch(ctx, target) + } + for i, member := range target.state.members { + if target.SyncMode() == SearchSyncNone || (i > 0 && target.SyncMode() == SearchSyncLocal) { + continue + } + ns, err := member.runtime.indexNamespace() + if err != nil { + return err + } + if err := member.runtime.reconcileSearchSnapshot(ctx, member.selected.snapshot, member.selected.store, ns, nil, ReconcileSearchIndexCmd{}); err != nil { + return err + } + } + return nil +} + +func supportsSearchCoverage(members []*searchTargetMember) bool { + for _, member := range members { + if _, ok := member.runtime.options.SearchIndex.(SearchIndexEntryStore); !ok { + return false + } + } + return true +} + +// Text-only search has no embedding coverage to await or report. +func (a *Application) searchTarget(ctx context.Context, target SearchTarget, request query.SearchQuery) (*query.SearchResult, []SearchCoverage, error) { + semantic := request.Phrase != "" + var coverage []SearchCoverage + if semantic { + if a.prepareSearch != nil && !supportsSearchCoverage(target.state.members) { + return nil, nil, fmt.Errorf("sdd: search preparation requires entry publication support for every selected project") + } + if a.prepareSearch != nil { + for _, member := range target.state.members { + if _, ok := member.selected.store.(pinnedGraphStore); !ok { + return nil, nil, fmt.Errorf("sdd: search preparation requires pinned snapshot reads for every selected project") + } + } + } + if err := a.prepareSearchTarget(ctx, target); err != nil { + return nil, nil, err + } + if err := ctx.Err(); err != nil { + return nil, nil, err + } + if supportsSearchCoverage(target.state.members) { + var err error + coverage, err = target.coverage(ctx) + if err != nil { + return nil, nil, err + } + } + } + var combined *query.SearchResult + request.SyncMode = SearchSyncNone + for _, member := range target.state.members { + result, err := member.runtime.searchSnapshot(ctx, member.selected.snapshot, member.selected.store, request) + if err != nil { + return nil, nil, err + } + for i := range result.Entries { + result.Entries[i].RepoID = member.repoID + } + if combined == nil { + combined = result + } else { + combined.Entries = append(combined.Entries, result.Entries...) + } + } + return combined, coverage, nil +} diff --git a/pkg/application/search_preparation_example_test.go b/pkg/application/search_preparation_example_test.go new file mode 100644 index 00000000..c81b3db5 --- /dev/null +++ b/pkg/application/search_preparation_example_test.go @@ -0,0 +1,55 @@ +package application_test + +import ( + "context" + + sdd "github.com/networkteam/sdd/pkg/application" +) + +func ExampleSearchTarget_local() { + // The composition root supplies runtimes for the projects it authorizes. + var runtimes map[sdd.ProjectID]*sdd.ProjectRuntime + options := sdd.ApplicationOptions{ + PrepareSearch: func(ctx context.Context, target sdd.SearchTarget) error { + for requirement, err := range target.Entries(ctx) { + if err != nil { + return err + } + if requirement.Published { + continue + } + runtime := runtimes[requirement.Entry.Version.Namespace.Project] + if err := runtime.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: requirement.Entry}); err != nil { + return err + } + } + return nil + }, + } + _ = options // Pass to NewApplication with the other required capabilities. +} + +func ExampleSearchTarget_externalConsumer() { + // These functions belong to the consumer's durable scheduling protocol. + var enqueue func(context.Context, sdd.SearchEntryDescriptor, sdd.SearchDiscoveryCursor) error + var waitWithinBudget func(context.Context, []sdd.SearchTargetProject) error + options := sdd.ApplicationOptions{ + PrepareSearch: func(ctx context.Context, target sdd.SearchTarget) error { + for requirement, err := range target.Entries(ctx) { + if err != nil { + return err + } + if requirement.Published { + continue + } + if err := enqueue(ctx, requirement.Entry, requirement.Cursor); err != nil { + return err + } + } + // Ordinary budget expiry returns nil. Parent cancellation and preparation + // failures return errors. SDD determines coverage after this returns. + return waitWithinBudget(ctx, target.Projects()) + }, + } + _ = options +} diff --git a/pkg/application/search_preparation_test.go b/pkg/application/search_preparation_test.go new file mode 100644 index 00000000..b9fa2464 --- /dev/null +++ b/pkg/application/search_preparation_test.go @@ -0,0 +1,235 @@ +package application_test + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/llm" + "github.com/networkteam/sdd/pkg/llm/embed" + "github.com/networkteam/sdd/pkg/local" +) + +func preparedRuntime(t *testing.T, project sdd.ProjectID, dependencies ...string) (*sdd.ProjectRuntime, *local.FilesystemGraphStore, string) { + t.Helper() + dir := t.TempDir() + putSearchEntry(t, dir, "aaa", "Alpha searchable body.") + graph, err := local.NewFilesystemGraphStore(local.FilesystemGraphStoreOptions{Project: project, GraphDir: dir}) + if err != nil { + t.Fatal(err) + } + runtime, err := sdd.NewProjectRuntime(sdd.ProjectRuntimeOptions{ + Project: sdd.ProjectRef{ID: project}, Graph: graph, Dependencies: dependencies, ExcludeEmbeddedFromIndex: true, + LLM: llm.RunnerFunc(func(context.Context, llm.Request) (llm.Result, error) { return llm.Result{}, nil }), + SearchIndex: local.NewMemorySearchIndexStore(), Embedder: embed.EmbedderFunc{Space: "test-space", Run: func(_ context.Context, req embed.Request) (embed.Result, error) { + result := embed.Result{Vectors: make([][]float32, len(req.Texts))} + for i := range result.Vectors { + result.Vectors[i] = []float32{1, 1} + } + return result, nil + }}, + }) + if err != nil { + t.Fatal(err) + } + return runtime, graph, dir +} + +func putSearchEntry(t *testing.T, dir, suffix, body string) { + t.Helper() + name := filepath.Join(dir, "2026", "01", "01-100000-s-tac-"+suffix+".md") + if err := os.MkdirAll(filepath.Dir(name), 0755); err != nil { + t.Fatal(err) + } + content := "---\ntype: signal\nkind: gap\nlayer: tactical\nsummary: Alpha summary\n---\n\n" + body + if err := os.WriteFile(name, []byte(content), 0644); err != nil { + t.Fatal(err) + } +} + +func preparationApp(t *testing.T, base, dep *sdd.ProjectRuntime, prepare func(context.Context, sdd.SearchTarget) error) *sdd.Application { + t.Helper() + app, err := sdd.NewApplication(sdd.ApplicationOptions{Access: &multiAccessResolver{base: base, dependency: dep}, Sessions: noSessionStore{}, StagedBlobs: noBlobStore{}, PrepareSearch: prepare}) + if err != nil { + t.Fatal(err) + } + return app +} + +func searchPrepared(t *testing.T, app *sdd.Application, includes string) (sdd.SearchResult, error) { + t.Helper() + return app.Search(t.Context(), sdd.RequestIdentity{Subject: "reader"}, "base", sdd.SearchRequest{SyncMode: sdd.SearchSyncNone, Phrase: "alpha", AllRepos: true, IncludesRevision: includes}) +} + +func TestLocalSearchPreparationComposition(t *testing.T) { + base, _, baseDir := preparedRuntime(t, "base", "dep") + dep, _, depDir := preparedRuntime(t, "dep") + runtimes := map[sdd.ProjectID]*sdd.ProjectRuntime{"base": base, "dep": dep} + calls := 0 + var retained sdd.SearchTarget + app := preparationApp(t, base, dep, func(ctx context.Context, target sdd.SearchTarget) error { + calls++ + retained = target + if len(target.Projects()) != 2 { + return fmt.Errorf("dependencies not selected before preparation") + } + // Both selected sources must survive writes while preparation runs. + putSearchEntry(t, baseDir, "bbb", "Later home entry") + putSearchEntry(t, depDir, "bbb", "Later dependency entry") + for item, err := range target.Entries(ctx) { + if err != nil { + return err + } + if err := runtimes[item.Entry.Version.Namespace.Project].IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: item.Entry}); err != nil { + return err + } + } + return nil + }) + result, err := searchPrepared(t, app, "") + if err != nil { + t.Fatal(err) + } + if calls != 1 || len(result.Coverage) != 2 || len(result.EntryIDs) != 2 { + t.Fatalf("result=%+v calls=%d", result, calls) + } + for _, coverage := range result.Coverage { + if !coverage.Complete || coverage.Required != 1 { + t.Fatalf("moving target: %+v", coverage) + } + } + for _, err := range retained.Entries(t.Context()) { + if err == nil { + t.Fatal("expired target remained usable") + } + break + } +} + +func TestExternalConsumerPreparationDoesNotClaimCoverage(t *testing.T) { + base, _, _ := preparedRuntime(t, "base") + var queued []sdd.SearchEntryDescriptor + app := preparationApp(t, base, nil, func(ctx context.Context, target sdd.SearchTarget) error { + for item, err := range target.Entries(ctx) { + if err != nil { + return err + } + queued = append(queued, item.Entry) + } + timer := time.NewTimer(time.Millisecond) + defer timer.Stop() + select { + case <-timer.C: + return ctx.Err() + case <-ctx.Done(): + return ctx.Err() + } + }) + result, err := searchPrepared(t, app, "") + if err != nil { + t.Fatal(err) + } + if len(queued) != 1 || len(result.EntryIDs) != 0 || len(result.Coverage) != 1 || result.Coverage[0].Complete || result.Notice == "" { + t.Fatalf("false completion: %+v", result) + } +} + +func TestPreparationErrorsAndLazyIterationErrorsPropagate(t *testing.T) { + base, _, _ := preparedRuntime(t, "base") + failure := errors.New("consumer preparation failed") + app := preparationApp(t, base, nil, func(context.Context, sdd.SearchTarget) error { return failure }) + if _, err := searchPrepared(t, app, ""); !errors.Is(err, failure) { + t.Fatalf("error=%v", err) + } + app = preparationApp(t, base, nil, func(ctx context.Context, target sdd.SearchTarget) error { + canceled, cancel := context.WithCancel(ctx) + cancel() + for _, err := range target.Entries(canceled) { + if err != nil { + return err + } + } + return nil + }) + if _, err := searchPrepared(t, app, ""); !errors.Is(err, context.Canceled) { + t.Fatalf("iteration error=%v", err) + } +} + +func TestSearchReadYourWritesAndLocalSourceLifetime(t *testing.T) { + base, graph, _ := preparedRuntime(t, "base") + initial, err := graph.Current(t.Context()) + if err != nil { + t.Fatal(err) + } + document := sdd.EntryDocument{LogicalPath: "2026/01/01-100000-s-tac-new.md", Frontmatter: map[string]any{"type": "signal", "kind": "gap", "layer": "tactical", "summary": "New"}, Body: "New write"} + apply := func(id string, doc sdd.EntryDocument) string { + before, err := graph.Current(t.Context()) + if err != nil { + t.Fatal(err) + } + batch := sdd.MutationBatch{ID: id, Changes: []sdd.DocumentChange{{LogicalPath: doc.LogicalPath, Document: &doc, CanonicalBytes: []byte("---\ntype: signal\nkind: gap\nlayer: tactical\nsummary: New\n---\n" + doc.Body)}}} + batch.Digest, err = sdd.MutationBatchDigest(batch) + if err != nil { + t.Fatal(err) + } + result, err := graph.Apply(t.Context(), before.Revision(), batch, nil) + if err != nil { + t.Fatal(err) + } + return result.Revision + } + written := apply("write-one", document) + document.LogicalPath = "2026/01/01-100000-s-tac-two.md" + document.Body = "Second write" + latest := apply("write-two", document) + app := preparationApp(t, base, nil, func(ctx context.Context, target sdd.SearchTarget) error { + for item, err := range target.Entries(ctx) { + if err != nil { + return err + } + if err := base.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: item.Entry}); err != nil { + return err + } + } + return nil + }) + result, err := searchPrepared(t, app, written) + if err != nil { + t.Fatal(err) + } + if len(result.Coverage) != 1 || result.Coverage[0].Revision != latest || result.Coverage[0].Required != 3 || !result.Coverage[0].Complete { + t.Fatalf("freshness=%+v initial=%s", result, initial.Revision()) + } + if _, err := searchPrepared(t, app, "unknown-write"); err == nil || !strings.Contains(err.Error(), "include") { + t.Fatalf("unproved freshness=%v", err) + } +} + +func TestPreparedCompleteNoMatchHasNoIncompleteNotice(t *testing.T) { + runtime, _, _ := preparedRuntime(t, "base") + app := preparationApp(t, runtime, nil, func(ctx context.Context, target sdd.SearchTarget) error { + for requirement, err := range target.Entries(ctx) { + if err != nil { + return err + } + if err := runtime.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: requirement.Entry}); err != nil { + return err + } + } + return nil + }) + result, err := app.Search(t.Context(), sdd.RequestIdentity{Subject: "reader"}, "base", sdd.SearchRequest{SyncMode: sdd.SearchSyncNone, Phrase: "alpha", Type: "decision"}) + if err != nil { + t.Fatal(err) + } + if len(result.EntryIDs) != 0 || result.Notice != "" || len(result.Coverage) != 1 || !result.Coverage[0].Complete { + t.Fatalf("result=%+v", result) + } +} diff --git a/pkg/application/search_selection_test.go b/pkg/application/search_selection_test.go new file mode 100644 index 00000000..2030da41 --- /dev/null +++ b/pkg/application/search_selection_test.go @@ -0,0 +1,214 @@ +package application_test + +import ( + "errors" + "os" + "path/filepath" + "reflect" + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" +) + +func TestSearchDiscoverySelectionAndCursor(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + putSearchEntry(t, dir, "bbb", "Second") + putSearchEntry(t, dir, "ccc", "Third") + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + const a = "20260101-100000-s-tac-aaa" + const b = "20260101-100000-s-tac-bbb" + const c = "20260101-100000-s-tac-ccc" + const absent = "20260101-100000-s-tac-zzz" + collect := func(selection []string, cursor sdd.SearchDiscoveryCursor) ([]string, sdd.SearchDiscoveryCursor, error) { + var ids []string + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: selection, Cursor: cursor}) { + if err != nil { + return nil, cursor, err + } + ids = append(ids, item.Entry.Version.EntryID) + cursor = item.Cursor + } + return ids, cursor, nil + } + ids, _, err := collect(nil, sdd.SearchDiscoveryCursor{}) + if err != nil || !reflect.DeepEqual(ids, []string{a, b, c}) { + t.Fatalf("all=%v %v", ids, err) + } + ids, _, err = collect([]string{c, a, c, absent}, sdd.SearchDiscoveryCursor{}) + if err != nil || !reflect.DeepEqual(ids, []string{a, c}) { + t.Fatalf("selection=%v %v", ids, err) + } + for _, selection := range [][]string{{}, {""}, {"s-tac-aaa"}, {"20260101-100000-x-tac-aaa"}, {"20260101-100000-s-tac-../aaa"}, {"20261301-100000-s-tac-aaa"}} { + if _, _, err := collect(selection, sdd.SearchDiscoveryCursor{}); err == nil { + t.Errorf("invalid selection accepted: %v", selection) + } + } + ids, _, err = collect([]string{absent}, sdd.SearchDiscoveryCursor{}) + if err != nil || len(ids) != 0 { + t.Fatalf("absent=%v %v", ids, err) + } + var cursor sdd.SearchDiscoveryCursor + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: []string{c, a}}) { + if err != nil { + t.Fatal(err) + } + cursor = item.Cursor + break + } + ids, _, err = collect([]string{a, c, a}, cursor) + if err != nil || !reflect.DeepEqual(ids, []string{c}) { + t.Fatalf("resume=%v %v", ids, err) + } + for _, selection := range [][]string{nil, {a, b}, {a, c, absent}} { + if _, _, err := collect(selection, cursor); err == nil { + t.Errorf("incompatible selection accepted: %v", selection) + } + } + _, allCursor, err := collect(nil, sdd.SearchDiscoveryCursor{}) + if err != nil { + t.Fatal(err) + } + if _, _, err := collect([]string{a, b, c}, allCursor); err == nil { + t.Fatal("whole-project cursor accepted for explicit selection") + } + wrong := cursor + wrong.Revision = "changed" + if _, _, err := collect([]string{a, c}, wrong); err == nil { + t.Fatal("changed revision accepted") + } + wrong = cursor + wrong.Namespace.Fingerprint = "changed" + if _, _, err := collect([]string{a, c}, wrong); err == nil { + t.Fatal("changed config accepted") + } +} + +func TestScopedDiscoveryPropagatesReadFailure(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + addDiscoveryAttachment(t, dir) + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + sentinel := errors.New("retained attachment unavailable") + source.Attachments = &countingAttachmentPages{AttachmentPageReader: source.Attachments, reads: map[string]int{}, fail: sentinel} + seen := false + for _, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: []string{"20260101-100000-s-tac-aaa"}}) { + seen = true + if !errors.Is(err, sentinel) { + t.Fatalf("error=%v", err) + } + } + if !seen { + t.Fatal("source failure treated as absent") + } +} + +func addDiscoveryAttachment(t *testing.T, dir string) { + t.Helper() + path := filepath.Join(dir, "2026/01/01-100000-s-tac-aaa/note.md") + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("evidence"), 0644); err != nil { + t.Fatal(err) + } +} + +func TestScopedDiscoveryUnreadableDocumentIsNotAbsent(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + if err := os.WriteFile(filepath.Join(dir, "2026/01/01-100000-s-tac-bbb.md"), []byte("---\nbroken: [\n---\n"), 0644); err != nil { + t.Fatal(err) + } + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + for _, selection := range [][]string{nil, {"20260101-100000-s-tac-bbb"}} { + failed := false + for _, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: selection}) { + if err != nil { + failed = true + } + } + if !failed { + t.Fatalf("unreadable entry treated as absent: %v", selection) + } + } + count := 0 + for _, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: []string{"20260101-100000-s-tac-aaa"}}) { + if err != nil { + t.Fatal(err) + } + count++ + } + if count != 1 { + t.Fatalf("unrelated unreadable entry affected selection: count=%d", count) + } +} + +func TestScopedDiscoverySkipsUnselectedAndEarlierAttachmentReads(t *testing.T) { + runtime, graph, dir := preparedRuntime(t, "base") + addDiscoveryAttachment(t, dir) + putSearchEntry(t, dir, "bbb", "Second") + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{}) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := source.Release(); err != nil { + t.Error(err) + } + }() + pages := &countingAttachmentPages{AttachmentPageReader: source.Attachments, reads: map[string]int{}} + source.Attachments = pages + ids := []string{"20260101-100000-s-tac-aaa", "20260101-100000-s-tac-bbb"} + var cursor sdd.SearchDiscoveryCursor + for item, err := range runtime.DiscoverSearchEntries(t.Context(), sdd.DiscoverSearchEntriesQuery{Source: source, EntryIDs: ids}) { + if err != nil { + t.Fatal(err) + } + cursor = item.Cursor + break + } + if pages.reads[ids[0]] != 1 { + t.Fatalf("reads=%v", pages.reads) + } + pages.fail = errors.New("earlier source must not be read") + for _, q := range []sdd.DiscoverSearchEntriesQuery{{Source: source, EntryIDs: ids, Cursor: cursor}, {Source: source, EntryIDs: ids[1:]}} { + count := 0 + for item, err := range runtime.DiscoverSearchEntries(t.Context(), q) { + if err != nil { + t.Fatal(err) + } + if item.Entry.Version.EntryID != ids[1] { + t.Fatal(item) + } + count++ + } + if count != 1 { + t.Fatalf("count=%d", count) + } + } + if pages.reads[ids[0]] != 1 { + t.Fatalf("rehashed skipped entry: %v", pages.reads) + } +} diff --git a/pkg/application/types/search_discovery.go b/pkg/application/types/search_discovery.go new file mode 100644 index 00000000..957e692e --- /dev/null +++ b/pkg/application/types/search_discovery.go @@ -0,0 +1,14 @@ +package types + +type SearchDiscoveryCursor struct { + Revision string + Namespace IndexNamespace + AfterEntryID string + Selection string +} + +type SearchEntryRequirement struct { + Entry SearchEntryDescriptor + Published bool + Cursor SearchDiscoveryCursor +} diff --git a/pkg/application/types/search_entry.go b/pkg/application/types/search_entry.go new file mode 100644 index 00000000..0bac217c --- /dev/null +++ b/pkg/application/types/search_entry.go @@ -0,0 +1,61 @@ +package types + +import ( + "fmt" + "math" +) + +// SearchEntryVersion is the publication and deduplication key. Revision is +// deliberately absent: identical content in different snapshots shares work. +type SearchEntryVersion struct { + Namespace IndexNamespace + EntryID string + EntryHash string +} + +// SearchEntryDescriptor identifies exact retained input, including attachments. +// The host must retain SourceRevision until all work using it is finished. +type SearchEntryDescriptor struct { + Version SearchEntryVersion + SourceRevision string +} + +type IndexSearchEntryCmd struct { + Entry SearchEntryDescriptor + OnPublished func(entryID string, chunks int) +} + +// ValidateEntryPublication validates the entire write before a store changes. +func ValidateEntryPublication(version SearchEntryVersion, chunks []IndexedChunk) error { + if version.Namespace.Project == "" || version.Namespace.Fingerprint == "" || version.Namespace.Metric != "cosine" || version.EntryID == "" || version.EntryHash == "" { + return fmt.Errorf("sdd: incomplete entry version") + } + seen := make(map[string]bool, len(chunks)) + dims := 0 + for _, row := range chunks { + if row.Chunk.EntryID != version.EntryID || row.Chunk.EntryHash != version.EntryHash || row.Chunk.ID == "" || seen[row.Chunk.ID] { + return fmt.Errorf("sdd: invalid or duplicate chunk identity %q", row.Chunk.ID) + } + seen[row.Chunk.ID] = true + if len(row.Vector) == 0 { + return fmt.Errorf("sdd: empty vector for %s", row.Chunk.ID) + } + if dims == 0 { + dims = len(row.Vector) + } + if len(row.Vector) != dims { + return fmt.Errorf("sdd: inconsistent vector dimensions") + } + norm := float64(0) + for _, v := range row.Vector { + if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) { + return fmt.Errorf("sdd: non-finite vector") + } + norm += float64(v) * float64(v) + } + if norm == 0 { + return fmt.Errorf("sdd: zero vector") + } + } + return nil +} diff --git a/pkg/application/vector_search.go b/pkg/application/vector_search.go index e313ac5e..3d3db8b8 100644 --- a/pkg/application/vector_search.go +++ b/pkg/application/vector_search.go @@ -183,7 +183,7 @@ func candidateSet(graph *model.Graph, request query.SearchQuery, excludeEmbedded // so the application derives the same attachment content the CLI reads from // disk. It is a pure read over the canonical graph authority. type graphStoreAttachmentReader struct { - store GraphStore + store AttachmentPageReader } // attachmentPageSize is the page size for assembling an attachment's full diff --git a/pkg/application/vector_search_persistent_test.go b/pkg/application/vector_search_persistent_test.go index a888d75e..fad0330a 100644 --- a/pkg/application/vector_search_persistent_test.go +++ b/pkg/application/vector_search_persistent_test.go @@ -326,7 +326,7 @@ func TestBranchVectorAndHybridSearchUseSelectedAttachmentAuthorityAndRelease(t * if err != nil { t.Fatal(err) } - branch, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: counterProject, GraphDir: branchDir}) + branch, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: counterProject, GraphDir: branchDir, Branch: "work"}) if err != nil { t.Fatal(err) } diff --git a/pkg/llm/embed/batcher.go b/pkg/llm/embed/batcher.go new file mode 100644 index 00000000..be8db7d6 --- /dev/null +++ b/pkg/llm/embed/batcher.go @@ -0,0 +1,298 @@ +package embed + +import ( + "context" + "crypto/rand" + "errors" + "fmt" + "sync" + "sync/atomic" + "time" + + "github.com/networkteam/sdd/pkg/llm" +) + +var ErrBatcherClosed = errors.New("embed: batcher closed") + +// BatchOptions bounds admitted document work separately from provider calls. +// MaxBytes is a UTF-8 input bound, not a provider token or wire-payload bound. +// Measure and MaxUnits optionally impose a provider-specific aggregate limit; +// the adapter must still enforce its exact wire contract. A single text above +// either limit fails explicitly. Entries with many texts are admitted one at +// a time and may span arbitrarily many batches. +type BatchOptions struct { + MaxItems int + MaxBytes int + BufferItems int + Window time.Duration + Concurrency int + MaxUnits int + Measure func(string) (int, error) +} + +// Batcher combines document requests over one fixed embedder configuration. +// Route queries separately and compose Bounded inside the batcher for +// provider deadlines. Shared calls use the batcher's lifetime, never a caller's +// context. Place Observed inside Batcher: document results carry zero Usage +// because usage belongs to provider batches, not to participating callers. +type Batcher struct { + inner Embedder + options BatchOptions + ctx context.Context + cancel context.CancelFunc + items chan batchItem + batches chan []batchItem + done chan struct{} + wg sync.WaitGroup + sequence atomic.Uint64 + instance string +} + +type batchItem struct { + text string + index int + units int + queued time.Time + caller context.Context + reply chan batchReply +} + +type batchReply struct { + index int + vector []float32 + identity llm.Identity + err error +} + +type callerKey struct{} +type attributionKey struct{} + +type BatchAttribution struct { + ID string + Callers []string +} + +// WithCaller attaches an opaque correlation ID, such as a host's job attempt. +// It affects observation only; no queue or retry semantics enter the batcher. +func WithCaller(ctx context.Context, id string) context.Context { + return context.WithValue(ctx, callerKey{}, id) +} + +// Attribution returns the shared-call identity available to an inner StatsSink. +// Callers is a set; usage must be recorded once against ID, never per caller. +func Attribution(ctx context.Context) BatchAttribution { + a, _ := ctx.Value(attributionKey{}).(BatchAttribution) + a.Callers = append([]string(nil), a.Callers...) + return a +} + +func NewBatcher(ctx context.Context, inner Embedder, options BatchOptions) (*Batcher, error) { + if inner == nil || options.MaxItems < 1 || options.MaxBytes < 1 || options.BufferItems < 1 || options.Concurrency < 1 || options.Window <= 0 || (options.Measure == nil) != (options.MaxUnits == 0) || options.MaxUnits < 0 { + return nil, fmt.Errorf("embed: invalid batch options") + } + lifetime, cancel := context.WithCancel(ctx) + b := &Batcher{instance: rand.Text(), inner: inner, options: options, ctx: lifetime, cancel: cancel, items: make(chan batchItem, options.BufferItems), batches: make(chan []batchItem), done: make(chan struct{})} + b.wg.Add(1 + options.Concurrency) + go b.collect() + for range options.Concurrency { + go b.work() + } + go func() { b.wg.Wait(); close(b.done) }() + return b, nil +} + +func (b *Batcher) Fingerprint() string { return b.inner.Fingerprint() } + +func (b *Batcher) Embed(ctx context.Context, req Request) (Result, error) { + if err := ctx.Err(); err != nil { + return Result{}, err + } + if b.ctx.Err() != nil { + return Result{}, ErrBatcherClosed + } + if req.Purpose != PurposeDocument { + return Result{}, fmt.Errorf("embed: unsupported batch purpose %q", req.Purpose) + } + units := make([]int, len(req.Texts)) + for i, text := range req.Texts { + if err := ctx.Err(); err != nil { + return Result{}, err + } + if b.ctx.Err() != nil { + return Result{}, ErrBatcherClosed + } + if len(text) > b.options.MaxBytes { + return Result{}, fmt.Errorf("embed: text exceeds batch byte limit") + } + if b.options.Measure != nil { + value, err := b.options.Measure(text) + if err != nil { + return Result{}, err + } + if value < 0 || value > b.options.MaxUnits { + return Result{}, fmt.Errorf("embed: text exceeds provider unit limit") + } + units[i] = value + } + } + result := Result{Vectors: make([][]float32, len(req.Texts))} + replies := make(chan batchReply, min(len(req.Texts), b.options.MaxItems)) + caller, cancel := context.WithCancel(ctx) + defer cancel() + submitted, received := 0, 0 + var next *batchItem + for received < len(req.Texts) { + var admission chan batchItem + var item batchItem + if submitted < len(req.Texts) { + if next == nil { + text := req.Texts[submitted] + + next = &batchItem{text: text, index: submitted, units: units[submitted], caller: caller, reply: replies, queued: time.Now()} + } + admission, item = b.items, *next + } + select { + case admission <- item: + submitted++ + next = nil + case reply := <-replies: + if reply.err != nil { + return Result{}, reply.err + } + result.Vectors[reply.index] = reply.vector + result.Identity = reply.identity + received++ + case <-ctx.Done(): + return Result{}, ctx.Err() + case <-b.ctx.Done(): + return Result{}, ErrBatcherClosed + } + } + + return result, nil +} + +func (b *Batcher) collect() { + defer b.wg.Done() + var batch []batchItem + var timer *time.Timer + var tick <-chan time.Time + bytes, units := 0, 0 + stop := func() { + if timer != nil { + timer.Stop() + } + timer = nil + tick = nil + } + defer stop() + flush := func() bool { + stop() + if len(batch) == 0 { + return true + } + select { + case b.batches <- batch: + batch = nil + bytes, units = 0, 0 + return true + case <-b.ctx.Done(): + return false + } + } + for { + select { + case <-b.ctx.Done(): + return + case <-tick: + if !flush() { + return + } + case item := <-b.items: + if item.caller.Err() != nil { + continue + } + if len(batch) > 0 && (bytes+len(item.text) > b.options.MaxBytes || (b.options.Measure != nil && units+item.units > b.options.MaxUnits)) { + if !flush() { + return + } + } + if len(batch) == 0 { + timer = time.NewTimer(max(0, time.Until(item.queued.Add(b.options.Window)))) + tick = timer.C + } + batch = append(batch, item) + bytes += len(item.text) + units += item.units + if len(batch) == b.options.MaxItems { + if !flush() { + return + } + } + } + } +} + +func (b *Batcher) work() { + defer b.wg.Done() + for { + select { + case <-b.ctx.Done(): + return + case items := <-b.batches: + b.dispatch(items) + } + } +} + +func (b *Batcher) dispatch(items []batchItem) { + active := items[:0] + texts := make([]string, 0, len(items)) + callers := []string{} + seen := map[string]bool{} + for _, item := range items { + if item.caller.Err() != nil { + continue + } + active = append(active, item) + texts = append(texts, item.text) + id, _ := item.caller.Value(callerKey{}).(string) + if id != "" && !seen[id] { + seen[id] = true + callers = append(callers, id) + } + } + if len(active) == 0 { + return + } + id := fmt.Sprintf("%s-%d", b.instance, b.sequence.Add(1)) + call := context.WithValue(b.ctx, attributionKey{}, BatchAttribution{ID: id, Callers: callers}) + result, err := b.inner.Embed(call, Request{Purpose: PurposeDocument, Texts: texts}) + if err == nil { + err = validateBatchVectors(result.Vectors, len(active)) + } + for i, item := range active { + reply := batchReply{index: item.index, identity: result.Identity, err: err} + if err == nil { + reply.vector = result.Vectors[i] + } + select { + case item.reply <- reply: + case <-item.caller.Done(): + case <-b.ctx.Done(): + } + } +} + +// Close cancels queued and in-flight work. Callers receive explicit failure; +// no durable retry is performed. ctx bounds waiting for providers to stop. +func (b *Batcher) Close(ctx context.Context) error { + b.cancel() + select { + case <-b.done: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} diff --git a/pkg/llm/embed/batcher_test.go b/pkg/llm/embed/batcher_test.go new file mode 100644 index 00000000..2178fde9 --- /dev/null +++ b/pkg/llm/embed/batcher_test.go @@ -0,0 +1,359 @@ +package embed_test + +import ( + "context" + "errors" + "fmt" + "strconv" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/networkteam/sdd/pkg/llm" + "github.com/networkteam/sdd/pkg/llm/embed" +) + +func batchOptions() embed.BatchOptions { + return embed.BatchOptions{MaxItems: 8, MaxBytes: 1024, BufferItems: 2, Window: 15 * time.Millisecond, Concurrency: 2} +} +func newBatcher(t *testing.T, run func(context.Context, embed.Request) (embed.Result, error), options embed.BatchOptions) *embed.Batcher { + t.Helper() + b, err := embed.NewBatcher(t.Context(), embed.EmbedderFunc{Space: "fixed", Run: run}, options) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + if err := b.Close(ctx); err != nil { + t.Error(err) + } + }) + return b +} +func numberedVectors(req embed.Request) embed.Result { + r := embed.Result{Vectors: make([][]float32, len(req.Texts)), Usage: llm.Usage{InputTokens: len(req.Texts)}} + for i, text := range req.Texts { + n, _ := strconv.Atoi(text) + r.Vectors[i] = []float32{float32(n + 1), 1} + } + return r +} + +func TestBatcherCrossCallerRoutingAndOversizedAdmission(t *testing.T) { + var mu sync.Mutex + calls, active, peak := 0, 0, 0 + var attributed []embed.BatchAttribution + b := newBatcher(t, func(ctx context.Context, req embed.Request) (embed.Result, error) { + mu.Lock() + calls++ + active++ + peak = max(peak, active) + attributed = append(attributed, embed.Attribution(ctx)) + mu.Unlock() + if len(req.Texts) > 8 { + t.Error("oversized provider batch") + } + timer := time.NewTimer(2 * time.Millisecond) + defer timer.Stop() + select { + case <-timer.C: + case <-ctx.Done(): + return embed.Result{}, ctx.Err() + } + mu.Lock() + active-- + mu.Unlock() + return numberedVectors(req), nil + }, batchOptions()) + var wg sync.WaitGroup + for caller := range 12 { + wg.Go(func() { + texts := make([]string, 25) + for i := range texts { + texts[i] = strconv.Itoa(caller*25 + i) + } + result, err := b.Embed(embed.WithCaller(t.Context(), fmt.Sprintf("job-%d", caller)), embed.Request{Purpose: embed.PurposeDocument, Texts: texts}) + if err != nil { + t.Error(err) + return + } + if result.Usage.InputTokens != 0 { + t.Error("usage duplicated across callers") + } + for i, v := range result.Vectors { + if v[0] != float32(caller*25+i+1) { + t.Errorf("misrouted vector %d: %v", i, v) + } + } + }) + } + wg.Wait() + if calls >= 300 || peak > 2 || peak < 2 { + t.Fatalf("calls=%d peak=%d", calls, peak) + } + shared := false + for _, a := range attributed { + if a.ID == "" { + t.Error("missing batch ID") + } + shared = shared || len(a.Callers) > 1 + } + if !shared { + t.Fatal("no cross-caller batch") + } +} + +func TestBatcherTailWindowAndPayloadFlushing(t *testing.T) { + options := batchOptions() + options.MaxBytes = 3 + options.Window = 20 * time.Millisecond + var calls atomic.Int32 + b := newBatcher(t, func(_ context.Context, req embed.Request) (embed.Result, error) { + calls.Add(1) + size := 0 + for _, s := range req.Texts { + size += len(s) + } + if size > 3 { + t.Error("payload overflow") + } + return numberedVectors(req), nil + }, options) + started := time.Now() + result, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"11", "22", "33"}}) + if err != nil || len(result.Vectors) != 3 { + t.Fatalf("%v %v", result, err) + } + if calls.Load() != 3 || time.Since(started) > time.Second { + t.Fatalf("tail not flushed: calls=%d elapsed=%s", calls.Load(), time.Since(started)) + } + if _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"too long"}}); err == nil { + t.Fatal("oversized text accepted") + } +} + +func TestBatcherCancellationDoesNotCancelSharedProvider(t *testing.T) { + started := make(chan struct{}) + release := make(chan struct{}) + options := batchOptions() + options.MaxItems = 2 + options.Concurrency = 1 + options.Window = time.Second + b := newBatcher(t, func(ctx context.Context, req embed.Request) (embed.Result, error) { + close(started) + select { + case <-release: + return numberedVectors(req), nil + case <-ctx.Done(): + return embed.Result{}, ctx.Err() + } + }, options) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + first, second := make(chan error, 1), make(chan error, 1) + go func() { + _, err := b.Embed(ctx, embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1"}}) + first <- err + }() + go func() { + _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"2"}}) + second <- err + }() + <-started + cancel() + if err := <-first; !errors.Is(err, context.Canceled) { + t.Fatalf("first=%v", err) + } + close(release) + if err := <-second; err != nil { + t.Fatalf("second=%v", err) + } +} + +func TestBatcherShutdownReleasesBlockedAdmission(t *testing.T) { + started := make(chan struct{}) + options := batchOptions() + options.MaxItems = 1 + options.Concurrency = 1 + options.BufferItems = 1 + b := newBatcher(t, func(ctx context.Context, _ embed.Request) (embed.Result, error) { + close(started) + <-ctx.Done() + return embed.Result{}, ctx.Err() + }, options) + result := make(chan error, 1) + go func() { + _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1", "2", "3", "4", "5"}}) + result <- err + }() + <-started + if err := b.Close(t.Context()); err != nil { + t.Fatal(err) + } + if err := <-result; !errors.Is(err, embed.ErrBatcherClosed) && !errors.Is(err, context.Canceled) { + t.Fatalf("shutdown=%v", err) + } +} + +func TestBatcherPropagatesProviderAndVectorFailures(t *testing.T) { + providerFailure := errors.New("provider failure") + for _, kind := range []string{"provider", "count", "empty", "dimensions", "zero"} { + t.Run(kind, func(t *testing.T) { + b := newBatcher(t, func(_ context.Context, req embed.Request) (embed.Result, error) { + r := numberedVectors(req) + switch kind { + case "provider": + return embed.Result{}, providerFailure + case "count": + r.Vectors = nil + case "empty": + r.Vectors[0] = nil + case "dimensions": + r.Vectors[0] = []float32{1} + case "zero": + r.Vectors[0] = []float32{0, 0} + } + return r, nil + }, batchOptions()) + _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1", "2"}}) + if err == nil { + t.Fatal("failure swallowed") + } + if kind == "provider" && !errors.Is(err, providerFailure) { + t.Fatalf("provider cause lost: %v", err) + } + }) + } +} + +func TestBatcherComposesDeadlineAndQueryRouting(t *testing.T) { + provider := embed.EmbedderFunc{Space: "fixed", Run: func(ctx context.Context, req embed.Request) (embed.Result, error) { + if req.Purpose == embed.PurposeQuery { + return numberedVectors(req), nil + } + <-ctx.Done() + return embed.Result{}, ctx.Err() + }} + documents, err := embed.NewBatcher(t.Context(), embed.Bounded(provider, 30*time.Millisecond), batchOptions()) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := documents.Close(t.Context()); err != nil { + t.Error(err) + } + }() + routed := embed.EmbedderFunc{Space: provider.Fingerprint(), Run: func(ctx context.Context, req embed.Request) (embed.Result, error) { + if req.Purpose == embed.PurposeQuery { + return provider.Embed(ctx, req) + } + return documents.Embed(ctx, req) + }} + docResult := make(chan error, 1) + go func() { + _, err := routed.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1"}}) + docResult <- err + }() + if _, err := routed.Embed(t.Context(), embed.Request{Purpose: embed.PurposeQuery, Texts: []string{"2"}}); err != nil { + t.Fatal(err) + } + if err := <-docResult; !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("deadline=%v", err) + } +} + +func TestBatcherProviderUnitLimit(t *testing.T) { + options := batchOptions() + options.MaxUnits = 3 + options.Measure = func(text string) (int, error) { return len(text), nil } + b := newBatcher(t, func(_ context.Context, req embed.Request) (embed.Result, error) { + units := 0 + for _, text := range req.Texts { + units += len(text) + } + if units > 3 { + t.Errorf("provider units=%d", units) + } + return numberedVectors(req), nil + }, options) + result, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"11", "22", "33"}}) + if err != nil || len(result.Vectors) != 3 { + t.Fatalf("result=%+v error=%v", result, err) + } + if _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1234"}}); err == nil { + t.Fatal("oversized units accepted") + } +} + +func TestBatcherWindowStartsAtOldestItem(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + options := batchOptions() + options.Window = time.Second + called := make(chan int, 2) + b := newBatcher(t, func(_ context.Context, req embed.Request) (embed.Result, error) { + called <- len(req.Texts) + return numberedVectors(req), nil + }, options) + var wg sync.WaitGroup + submit := func(text string) { + wg.Go(func() { + if _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{text}}); err != nil { + t.Error(err) + } + }) + } + submit("1") + synctest.Wait() + time.Sleep(750 * time.Millisecond) + submit("2") + synctest.Wait() + time.Sleep(250 * time.Millisecond) + synctest.Wait() + select { + case count := <-called: + if count != 2 { + t.Fatalf("items=%d", count) + } + default: + t.Fatal("later arrival reset the flush window") + } + wg.Wait() + }) +} + +func TestBatcherRejectsEntireRequestBeforeAdmission(t *testing.T) { + for _, mode := range []string{"bytes", "units", "measurement"} { + t.Run(mode, func(t *testing.T) { + options := batchOptions() + options.MaxItems = 1 + options.MaxBytes = 2 + if mode != "bytes" { + options.MaxBytes = 100 + options.MaxUnits = 2 + options.Measure = func(text string) (int, error) { + if mode == "measurement" && text == "bad" { + return 0, errors.New("cannot measure") + } + return len(text), nil + } + } + var calls atomic.Int32 + b := newBatcher(t, func(_ context.Context, req embed.Request) (embed.Result, error) { + calls.Add(1) + return numberedVectors(req), nil + }, options) + if _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"1", "2", "bad"}}); err == nil { + t.Fatal("invalid trailing text accepted") + } + if _, err := b.Embed(t.Context(), embed.Request{Purpose: embed.PurposeDocument, Texts: []string{"3"}}); err != nil { + t.Fatal(err) + } + if calls.Load() != 1 { + t.Fatalf("invalid request dispatched work: calls=%d", calls.Load()) + } + }) + } +} diff --git a/pkg/llm/embed/vectors.go b/pkg/llm/embed/vectors.go new file mode 100644 index 00000000..fe5fd53b --- /dev/null +++ b/pkg/llm/embed/vectors.go @@ -0,0 +1,35 @@ +package embed + +import ( + "fmt" + "math" +) + +func validateBatchVectors(vectors [][]float32, count int) error { + if len(vectors) != count { + return fmt.Errorf("embed: got %d vectors for %d texts", len(vectors), count) + } + dims := 0 + for _, vector := range vectors { + if len(vector) == 0 { + return fmt.Errorf("embed: empty vector") + } + if dims == 0 { + dims = len(vector) + } + if len(vector) != dims { + return fmt.Errorf("embed: inconsistent vector dimensions") + } + norm := float64(0) + for _, v := range vector { + if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) { + return fmt.Errorf("embed: non-finite vector") + } + norm += float64(v) * float64(v) + } + if norm == 0 { + return fmt.Errorf("embed: zero vector") + } + } + return nil +} diff --git a/pkg/local/entry_publication.go b/pkg/local/entry_publication.go new file mode 100644 index 00000000..d9141c20 --- /dev/null +++ b/pkg/local/entry_publication.go @@ -0,0 +1,92 @@ +package local + +import ( + "context" + "fmt" + + "github.com/networkteam/sdd/internal/index" + app "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/application/types" +) + +func (s *PersistentSearchIndexStore) EntryPublished(ctx context.Context, version app.SearchEntryVersion) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + dir, err := s.storeDir(version.Namespace) + if err != nil { + return false, err + } + manifest, err := s.readManifest(dir) + if err != nil { + return false, err + } + return publishedVersion(manifest, version), nil +} + +func publishedVersion(manifest *index.Manifest, version app.SearchEntryVersion) bool { + state := manifest.Entries[version.EntryID] + for _, stored := range state.Versions { + if stored.Hash == version.EntryHash && stored.Fingerprint == version.Namespace.Fingerprint { + return true + } + } + return false +} + +func (s *PersistentSearchIndexStore) PublishEntry(ctx context.Context, version app.SearchEntryVersion, chunks []app.IndexedChunk) error { + if err := types.ValidateEntryPublication(version, chunks); err != nil { + return err + } + dir, err := s.storeDir(version.Namespace) + if err != nil { + return err + } + return index.WriteStore(ctx, dir, func(idx *index.Index) error { + manifest, err := index.LoadManifest(dir) + if err != nil { + return err + } + if publishedVersion(manifest, version) { + return nil + } + incoming := make(map[string]bool, len(chunks)) + for _, chunk := range chunks { + incoming[chunk.Chunk.ID] = true + } + for entry, state := range manifest.Entries { + for _, stored := range state.Versions { + if entry == version.EntryID && stored.Hash == version.EntryHash { + continue + } + for _, id := range stored.ChunkIDs { + if incoming[id] { + return fmt.Errorf("sdd: chunk ID conflicts with another published version") + } + } + } + } + rows := make([]index.Row, len(chunks)) + ids := make([]string, len(chunks)) + for i, chunk := range chunks { + rows[i] = rowFromChunk(chunk, version.Namespace.Fingerprint) + ids[i] = chunk.Chunk.ID + } + if err := idx.UpsertEntry(ctx, version.EntryID, nil, rows); err != nil { + return err + } + manifest.AddVersion(version.EntryID, index.EntryVersion{Hash: version.EntryHash, Fingerprint: version.Namespace.Fingerprint, ChunkIDs: ids, IndexedAt: s.now()}) + return manifest.Save(dir) + }) +} + +func (s *PersistentSearchIndexStore) readManifest(dir string) (*index.Manifest, error) { + s.mu.Lock() + defer s.mu.Unlock() + cache := s.manifests[dir] + if cache == nil { + cache = &index.ManifestCache{} + s.manifests[dir] = cache + } + return cache.Read(dir) +} diff --git a/pkg/local/entry_publication_test.go b/pkg/local/entry_publication_test.go new file mode 100644 index 00000000..8d6fae51 --- /dev/null +++ b/pkg/local/entry_publication_test.go @@ -0,0 +1,172 @@ +package local_test + +import ( + "context" + "fmt" + "math" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/networkteam/sdd/internal/index" + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/local" +) + +func publicationRows(version sdd.SearchEntryVersion) []sdd.IndexedChunk { + return []sdd.IndexedChunk{{Chunk: sdd.CanonicalChunk{ID: version.EntryID + "-" + version.EntryHash, EntryID: version.EntryID, EntryHash: version.EntryHash, Text: "entry", Body: "entry"}, Vector: []float32{1, 1}}} +} +func TestEntryPublicationConformance(t *testing.T) { + for _, persistent := range []bool{false, true} { + name := "memory" + if persistent { + name = "disk" + } + t.Run(name, func(t *testing.T) { + var store interface { + sdd.SearchIndexStore + sdd.SearchIndexEntryStore + } = local.NewMemorySearchIndexStore() + if persistent { + store = local.NewPersistentSearchIndexStore("project", t.TempDir(), "project") + } + version := sdd.SearchEntryVersion{Namespace: sdd.IndexNamespace{Project: "project", Fingerprint: "space", Metric: "cosine"}, EntryID: "entry", EntryHash: "hash"} + invalid := publicationRows(version) + invalid = append(invalid, sdd.IndexedChunk{Chunk: sdd.CanonicalChunk{ID: "second", EntryID: "entry", EntryHash: "hash", Text: "bad"}, Vector: []float32{float32(math.NaN()), 1}}) + if err := store.PublishEntry(t.Context(), version, invalid); err == nil { + t.Fatal("invalid vectors accepted") + } + if published, err := store.EntryPublished(t.Context(), version); err != nil || published { + t.Fatalf("false completion=%v %v", published, err) + } + hits, err := store.Nearest(t.Context(), []sdd.IndexNamespace{version.Namespace}, []float32{1, 1}, 10) + if err != nil || len(hits) != 0 { + t.Fatalf("partial visibility=%v %v", hits, err) + } + for range 2 { + if err := store.PublishEntry(t.Context(), version, publicationRows(version)); err != nil { + t.Fatal(err) + } + } + changed := version + changed.EntryHash = "hash2" + if err := store.PublishEntry(t.Context(), changed, publicationRows(changed)); err != nil { + t.Fatal(err) + } + other := version + other.Namespace.Fingerprint = "other" + if published, err := store.EntryPublished(t.Context(), other); err != nil || published { + t.Fatalf("config collision=%v %v", published, err) + } + empty := version + empty.EntryID = "empty" + if err := store.PublishEntry(t.Context(), empty, nil); err != nil { + t.Fatal(err) + } + if published, err := store.EntryPublished(t.Context(), empty); err != nil || !published { + t.Fatalf("zero chunks=%v %v", published, err) + } + hits, err = store.Nearest(t.Context(), []sdd.IndexNamespace{version.Namespace}, []float32{1, 1}, 10) + if err != nil || len(hits) != 2 { + t.Fatalf("versions=%v %v", hits, err) + } + }) + } +} + +func TestDiskPublicationFailureLeavesRowsInvisibleAndRetryable(t *testing.T) { + root := t.TempDir() + store := local.NewPersistentSearchIndexStore("project", root, "project") + version := sdd.SearchEntryVersion{Namespace: sdd.IndexNamespace{Project: "project", Fingerprint: "space", Metric: "cosine"}, EntryID: "entry", EntryHash: "hash"} + dir := index.StoreDir(root, "project", "space") + obstruction := filepath.Join(dir, "manifest.json.tmp") + if err := os.MkdirAll(obstruction, 0755); err != nil { + t.Fatal(err) + } + if err := store.PublishEntry(t.Context(), version, publicationRows(version)); err == nil { + t.Fatal("expected storage failure") + } + store = local.NewPersistentSearchIndexStore("project", root, "project") + hits, err := store.Nearest(t.Context(), []sdd.IndexNamespace{version.Namespace}, []float32{1, 1}, 1) + if err != nil || len(hits) != 0 { + t.Fatalf("unpublished rows visible after restart: %v %v", hits, err) + } + if err := os.Remove(obstruction); err != nil { + t.Fatal(err) + } + if err := store.PublishEntry(t.Context(), version, publicationRows(version)); err != nil { + t.Fatal(err) + } + published, err := store.EntryPublished(t.Context(), version) + if err != nil || !published { + t.Fatalf("retry=%v %v", published, err) + } + hits, err = store.Nearest(t.Context(), []sdd.IndexNamespace{version.Namespace}, []float32{1, 1}, 1) + if err != nil || len(hits) != 1 { + t.Fatalf("retry visibility=%v %v", hits, err) + } +} + +func TestFullVersionChunkIdentity(t *testing.T) { + a, b := "deadbeef"+strings.Repeat("a", 56), "deadbeef"+strings.Repeat("b", 56) + if index.BodyChunkIDVersioned("entry", a, 0) == index.BodyChunkIDVersioned("entry", b, 0) { + t.Fatal("full versions collided") + } +} + +func BenchmarkEntryPublicationPresence(b *testing.B) { + store := local.NewPersistentSearchIndexStore("project", b.TempDir(), "project") + version := sdd.SearchEntryVersion{Namespace: sdd.IndexNamespace{Project: "project", Fingerprint: "space", Metric: "cosine"}, EntryID: "entry", EntryHash: "hash"} + if err := store.PublishEntry(context.Background(), version, publicationRows(version)); err != nil { + b.Fatal(err) + } + b.ReportAllocs() + b.ResetTimer() + for b.Loop() { + if _, err := store.EntryPublished(context.Background(), version); err != nil { + b.Fatal(err) + } + } +} + +func BenchmarkPublishedRetrieval(b *testing.B) { + root := b.TempDir() + ns := sdd.IndexNamespace{Project: "project", Fingerprint: "space", Metric: "cosine"} + store := local.NewPersistentSearchIndexStore("project", root, "project") + key := sdd.SearchEntryVersion{Namespace: ns, EntryID: "published", EntryHash: "h"} + rows := make([]sdd.IndexedChunk, 1000) + for i := range rows { + rows[i] = sdd.IndexedChunk{Chunk: sdd.CanonicalChunk{ID: fmt.Sprintf("p-%d", i), EntryID: key.EntryID, EntryHash: key.EntryHash, Text: "published document text", Body: "body"}, Vector: []float32{1, 1}} + } + if err := store.PublishEntry(context.Background(), key, rows); err != nil { + b.Fatal(err) + } + dir := index.StoreDir(root, "project", "space") + if err := index.WriteStore(context.Background(), dir, func(idx *index.Index) error { + unpublished := make([]index.Row, 1000) + for i := range unpublished { + unpublished[i] = index.Row{EntryID: "unpublished", EntryHash: "other", ChunkID: fmt.Sprintf("u-%d", i), Text: "unpublished", Embedding: []float32{1, 1}} + } + return idx.UpsertEntry(context.Background(), "unpublished", nil, unpublished) + }); err != nil { + b.Fatal(err) + } + hits, err := store.Nearest(context.Background(), []sdd.IndexNamespace{ns}, []float32{1, 1}, 10) + if err != nil || len(hits) != 10 { + b.Fatalf("query: %v %v", hits, err) + } + for _, hit := range hits { + if hit.EntryID != "published" { + b.Fatal("unpublished hit") + } + } + b.ReportAllocs() + b.ResetTimer() + for b.Loop() { + hits, err := store.Nearest(context.Background(), []sdd.IndexNamespace{ns}, []float32{1, 1}, 10) + if err != nil || len(hits) != 10 { + b.Fatal(err) + } + } +} diff --git a/pkg/local/local_graphstore.go b/pkg/local/local_graphstore.go index 17284202..27230179 100644 --- a/pkg/local/local_graphstore.go +++ b/pkg/local/local_graphstore.go @@ -23,14 +23,18 @@ import ( type FilesystemGraphStoreOptions struct { Project app.ProjectID GraphDir string + // Branch is the authority assigned by the target acquirer, if branch-scoped. + Branch string } // FilesystemGraphStore is the local canonical graph authority. It owns its // revision cache and never requires callers to invalidate snapshots. type FilesystemGraphStore struct { - project app.ProjectID - dir string - mu sync.Mutex + project app.ProjectID + branch string + dir string + mu sync.Mutex + snapshots map[string]*retainedSnapshot beforeApplyOperation func(int) error beforeRollbackOperation func(int) error @@ -79,6 +83,7 @@ func NewFilesystemGraphStore(options FilesystemGraphStoreOptions) (*FilesystemGr } return &FilesystemGraphStore{ project: options.Project, + branch: options.Branch, dir: options.GraphDir, }, nil } diff --git a/pkg/local/memory_indexstore.go b/pkg/local/memory_indexstore.go index 22a22210..15696106 100644 --- a/pkg/local/memory_indexstore.go +++ b/pkg/local/memory_indexstore.go @@ -8,18 +8,20 @@ import ( "sync" app "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/application/types" ) // MemorySearchIndexStore is a process-local mechanical vector index. It is // useful for embedded compositions that want root-owned lazy reconciliation // without operating a separate persistent index service. type MemorySearchIndexStore struct { - mu sync.RWMutex - chunks map[app.IndexNamespace]map[string]app.IndexedChunk + mu sync.RWMutex + chunks map[app.IndexNamespace]map[string]app.IndexedChunk + published map[app.SearchEntryVersion]bool } func NewMemorySearchIndexStore() *MemorySearchIndexStore { - return &MemorySearchIndexStore{chunks: map[app.IndexNamespace]map[string]app.IndexedChunk{}} + return &MemorySearchIndexStore{chunks: map[app.IndexNamespace]map[string]app.IndexedChunk{}, published: map[app.SearchEntryVersion]bool{}} } func (s *MemorySearchIndexStore) Manifest(_ context.Context, namespace app.IndexNamespace) ([]app.StoredChunkRef, error) { @@ -41,16 +43,10 @@ func (s *MemorySearchIndexStore) Manifest(_ context.Context, namespace app.Index func (s *MemorySearchIndexStore) IndexedEntries(_ context.Context, namespace app.IndexNamespace) ([]app.StoredEntryRef, error) { s.mu.RLock() defer s.mu.RUnlock() - seen := map[app.StoredEntryRef]bool{} var refs []app.StoredEntryRef - for _, item := range s.chunks[namespace] { - if item.Chunk.EntryID == "" { - continue - } - ref := app.StoredEntryRef{EntryID: item.Chunk.EntryID, EntryHash: item.Chunk.EntryHash} - if !seen[ref] { - seen[ref] = true - refs = append(refs, ref) + for version := range s.published { + if version.Namespace == namespace { + refs = append(refs, app.StoredEntryRef{EntryID: version.EntryID, EntryHash: version.EntryHash}) } } sort.Slice(refs, func(i, j int) bool { @@ -62,37 +58,98 @@ func (s *MemorySearchIndexStore) IndexedEntries(_ context.Context, namespace app return refs, nil } -func (s *MemorySearchIndexStore) Reconcile(_ context.Context, namespace app.IndexNamespace, _ string, upserts []app.IndexedChunk, deletes []string) error { +func (s *MemorySearchIndexStore) Reconcile(ctx context.Context, namespace app.IndexNamespace, _ string, upserts []app.IndexedChunk, deletes []string) error { s.mu.Lock() defer s.mu.Unlock() + if err := ctx.Err(); err != nil { + return err + } + if err := s.validateDimensions(namespace, upserts); err != nil { + return err + } if s.chunks[namespace] == nil { s.chunks[namespace] = map[string]app.IndexedChunk{} } for _, id := range deletes { + if item, ok := s.chunks[namespace][id]; ok { + delete(s.published, app.SearchEntryVersion{Namespace: namespace, EntryID: item.Chunk.EntryID, EntryHash: item.Chunk.EntryHash}) + } delete(s.chunks[namespace], id) } - // The namespace fingerprint pins the embedding model, so every vector in - // a namespace must share one dimensionality — the first stored vector - // sets it, a mismatch is a provider bug. + for _, item := range upserts { + s.chunks[namespace][item.Chunk.ID] = cloneIndexedChunk(item) + s.published[app.SearchEntryVersion{Namespace: namespace, EntryID: item.Chunk.EntryID, EntryHash: item.Chunk.EntryHash}] = true + } + return nil +} + +func (s *MemorySearchIndexStore) validateDimensions(namespace app.IndexNamespace, rows []app.IndexedChunk) error { dims := 0 for _, item := range s.chunks[namespace] { dims = len(item.Vector) break } - for _, item := range upserts { - if len(item.Vector) == 0 { - return fmt.Errorf("sdd: vector for %s is empty", item.Chunk.ID) + for _, row := range rows { + if len(row.Vector) == 0 { + return fmt.Errorf("sdd: empty vector") } if dims == 0 { - dims = len(item.Vector) + dims = len(row.Vector) + } + if len(row.Vector) != dims { + return fmt.Errorf("sdd: inconsistent vector dimensions") + } + for _, v := range row.Vector { + if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) { + return fmt.Errorf("sdd: non-finite vector") + } } - if len(item.Vector) != dims { - return fmt.Errorf("sdd: vector for %s has %d dimensions, want %d", item.Chunk.ID, len(item.Vector), dims) + } + return nil +} + +func cloneIndexedChunk(row app.IndexedChunk) app.IndexedChunk { + row.Vector = append([]float32(nil), row.Vector...) + row.Chunk.Breadcrumb = append([]string(nil), row.Chunk.Breadcrumb...) + return row +} + +func (s *MemorySearchIndexStore) EntryPublished(ctx context.Context, version app.SearchEntryVersion) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + s.mu.RLock() + defer s.mu.RUnlock() + return s.published[version], nil +} + +func (s *MemorySearchIndexStore) PublishEntry(ctx context.Context, version app.SearchEntryVersion, rows []app.IndexedChunk) error { + if err := types.ValidateEntryPublication(version, rows); err != nil { + return err + } + s.mu.Lock() + defer s.mu.Unlock() + if err := ctx.Err(); err != nil { + return err + } + if s.published[version] { + return nil + } + for _, row := range rows { + if stored, ok := s.chunks[version.Namespace][row.Chunk.ID]; ok && (stored.Chunk.EntryID != version.EntryID || stored.Chunk.EntryHash != version.EntryHash) { + return fmt.Errorf("sdd: chunk ID conflicts with another entry version") } - copy := item - copy.Vector = append([]float32(nil), item.Vector...) - s.chunks[namespace][item.Chunk.ID] = copy } + if err := s.validateDimensions(version.Namespace, rows); err != nil { + return err + } + if s.chunks[version.Namespace] == nil { + s.chunks[version.Namespace] = map[string]app.IndexedChunk{} + } + for _, row := range rows { + s.chunks[version.Namespace][row.Chunk.ID] = cloneIndexedChunk(row) + } + s.published[version] = true return nil } @@ -102,6 +159,9 @@ func (s *MemorySearchIndexStore) Nearest(_ context.Context, namespaces []app.Ind var result []app.ScoredChunkHit for _, namespace := range namespaces { for _, item := range s.chunks[namespace] { + if !s.published[app.SearchEntryVersion{Namespace: namespace, EntryID: item.Chunk.EntryID, EntryHash: item.Chunk.EntryHash}] { + continue + } if len(vector) != len(item.Vector) { return nil, fmt.Errorf("sdd: query vector has %d dimensions, want %d", len(vector), len(item.Vector)) } diff --git a/pkg/local/persistent_indexstore.go b/pkg/local/persistent_indexstore.go index 65919e38..3b6a56bb 100644 --- a/pkg/local/persistent_indexstore.go +++ b/pkg/local/persistent_indexstore.go @@ -37,8 +37,9 @@ type PersistentSearchIndexStore struct { repoKey string now func() time.Time - mu sync.Mutex - caches map[string]*index.SnapshotCache // per store dir + mu sync.Mutex + manifests map[string]*index.ManifestCache + caches map[string]*index.SnapshotCache // per store dir // reloads counts fresh snapshot loads, for tests that assert the cache // reuses a snapshot until a write bumps the generation. reloads atomic.Int64 @@ -55,6 +56,7 @@ func NewPersistentSearchIndexStore(project app.ProjectID, cacheRoot, repoKey str repoKey: repoKey, now: time.Now, caches: map[string]*index.SnapshotCache{}, + manifests: map[string]*index.ManifestCache{}, } } @@ -86,7 +88,7 @@ func (s *PersistentSearchIndexStore) IndexedEntries(_ context.Context, namespace if err != nil { return nil, err } - manifest, err := index.LoadManifest(dir) + manifest, err := s.readManifest(dir) if err != nil { return nil, err } @@ -107,7 +109,7 @@ func (s *PersistentSearchIndexStore) Manifest(_ context.Context, namespace app.I if err != nil { return nil, err } - manifest, err := index.LoadManifest(dir) + manifest, err := s.readManifest(dir) if err != nil { return nil, err } diff --git a/pkg/local/read_snapshot.go b/pkg/local/read_snapshot.go new file mode 100644 index 00000000..e2bff84f --- /dev/null +++ b/pkg/local/read_snapshot.go @@ -0,0 +1,190 @@ +package local + +import ( + "archive/zip" + "bytes" + "context" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "sync" + + app "github.com/networkteam/sdd/pkg/application" +) + +type retainedSnapshot struct { + snapshot *app.Snapshot + files *zip.Reader + leases int +} + +type snapshotAttachments struct{ files fs.FS } + +func (s snapshotAttachments) ReadAttachmentPage(ctx context.Context, entry, name string, offset int64, limit int) (app.AttachmentPage, error) { + if err := ctx.Err(); err != nil { + return app.AttachmentPage{}, err + } + return app.PageAttachment(s.files, ".", entry, name, offset, limit) +} + +// AcquireSnapshot retains immutable graph and attachment bytes while a lease +// exists. Exact revisions survive concurrent Apply calls, not process restarts; +// durable consumers supply their own revision-backed SnapshotReader. Memory +// scales with all graph and attachment bytes in live revisions. A nonempty +// requested branch must match FilesystemGraphStoreOptions.Branch. +func (s *FilesystemGraphStore) AcquireSnapshot(ctx context.Context, q app.SnapshotReadQuery) (*app.AcquiredSnapshot, error) { + if (q.Branch != "" && q.Branch != s.branch) || (q.ExactRevision != "" && q.IncludesRevision != "") { + return nil, fmt.Errorf("sdd: invalid filesystem snapshot selection") + } + s.mu.Lock() + defer s.mu.Unlock() + if err := ctx.Err(); err != nil { + return nil, err + } + if q.ExactRevision != "" { + if retained := s.snapshots[q.ExactRevision]; retained != nil { + return s.leaseSnapshot(retained), nil + } + } + lock, err := s.lock() + if err != nil { + return nil, err + } + defer unlock(lock) + if err := s.recoverPendingTransactionsLocked(); err != nil { + return nil, err + } + revision, err := graphDirectoryRevision(s.dir) + if err != nil { + return nil, err + } + if q.ExactRevision != "" && q.ExactRevision != revision { + return nil, fmt.Errorf("sdd: exact source revision is no longer retained") + } + if q.IncludesRevision != "" { + ok, err := s.includesRevision(revision, q.IncludesRevision) + if err != nil { + return nil, err + } + if !ok { + return nil, fmt.Errorf("sdd: current revision cannot be shown to include the requested write") + } + } + if retained := s.snapshots[revision]; retained != nil { + return s.leaseSnapshot(retained), nil + } + files, err := freezeGraphFS(ctx, s.dir) + if err != nil { + return nil, err + } + after, err := graphDirectoryRevision(s.dir) + if err != nil { + return nil, err + } + if after != revision { + return nil, fmt.Errorf("sdd: graph changed while acquiring snapshot") + } + snapshot, err := app.LoadSnapshotFS(ctx, s.project, revision, files, ".") + if err != nil { + return nil, err + } + retained := &retainedSnapshot{snapshot: snapshot, files: files} + if s.snapshots == nil { + s.snapshots = map[string]*retainedSnapshot{} + } + s.snapshots[revision] = retained + return s.leaseSnapshot(retained), nil +} + +func (s *FilesystemGraphStore) leaseSnapshot(retained *retainedSnapshot) *app.AcquiredSnapshot { + retained.leases++ + var once sync.Once + return &app.AcquiredSnapshot{Snapshot: retained.snapshot, Attachments: snapshotAttachments{files: retained.files}, Release: func() error { + once.Do(func() { + s.mu.Lock() + defer s.mu.Unlock() + retained.leases-- + if retained.leases == 0 { + delete(s.snapshots, retained.snapshot.Revision()) + } + }) + return nil + }} +} + +func (s *FilesystemGraphStore) includesRevision(current, required string) (bool, error) { + if current == required { + return true, nil + } + names, err := os.ReadDir(filepath.Join(s.dir, ".sdd-runtime", "applied")) + if err != nil { + return false, err + } + parents := map[string][]string{} + for _, name := range names { + if name.IsDir() || filepath.Ext(name.Name()) != ".json" { + continue + } + id := name.Name()[:len(name.Name())-5] + record, found, err := s.loadApplyRecord(id) + if err != nil { + return false, err + } + if found && record.Result.State == app.MutationApplied { + parents[record.Result.Revision] = append(parents[record.Result.Revision], record.ExpectedRevision) + } + } + todo := []string{current} + seen := map[string]bool{} + for len(todo) > 0 { + node := todo[len(todo)-1] + todo = todo[:len(todo)-1] + if node == required { + return true, nil + } + if seen[node] { + continue + } + seen[node] = true + todo = append(todo, parents[node]...) + } + return false, nil +} + +func freezeGraphFS(ctx context.Context, dir string) (*zip.Reader, error) { + var buffer bytes.Buffer + archive := zip.NewWriter(&buffer) + err := fs.WalkDir(os.DirFS(dir), ".", func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if err := ctx.Err(); err != nil { + return err + } + if entry.IsDir() { + if entry.Name() == ".sdd-runtime" { + return fs.SkipDir + } + return nil + } + if entry.Type()&fs.ModeSymlink != 0 { + return fmt.Errorf("sdd: snapshot contains a symbolic link: %s", path) + } + data, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(path))) + if err != nil { + return err + } + file, err := archive.CreateHeader(&zip.FileHeader{Name: path, Method: zip.Store}) + if err != nil { + return err + } + _, err = file.Write(data) + return err + }) + if err := errors.Join(err, archive.Close()); err != nil { + return nil, err + } + return zip.NewReader(bytes.NewReader(buffer.Bytes()), int64(buffer.Len())) +} diff --git a/pkg/local/read_snapshot_branch_test.go b/pkg/local/read_snapshot_branch_test.go new file mode 100644 index 00000000..6a979f03 --- /dev/null +++ b/pkg/local/read_snapshot_branch_test.go @@ -0,0 +1,34 @@ +package local_test + +import ( + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/local" +) + +func TestFilesystemSnapshotBranchScope(t *testing.T) { + graph, err := local.NewFilesystemGraphStore(local.FilesystemGraphStoreOptions{Project: "test", GraphDir: t.TempDir(), Branch: "work"}) + if err != nil { + t.Fatal(err) + } + for _, branch := range []string{"", "work"} { + source, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{Branch: branch}) + if err != nil { + t.Fatal(err) + } + if err := source.Release(); err != nil { + t.Fatal(err) + } + } + if _, err := graph.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{Branch: "main"}); err == nil { + t.Fatal("wrong branch accepted") + } + unscoped, err := local.NewFilesystemGraphStore(local.FilesystemGraphStoreOptions{Project: "test", GraphDir: t.TempDir()}) + if err != nil { + t.Fatal(err) + } + if _, err := unscoped.AcquireSnapshot(t.Context(), sdd.SnapshotReadQuery{Branch: "work"}); err == nil { + t.Fatal("unscoped store accepted branch authority") + } +} diff --git a/pkg/mcpapp/search_preparation_test.go b/pkg/mcpapp/search_preparation_test.go new file mode 100644 index 00000000..74066939 --- /dev/null +++ b/pkg/mcpapp/search_preparation_test.go @@ -0,0 +1,80 @@ +package mcpapp_test + +import ( + "context" + "testing" + + sdd "github.com/networkteam/sdd/pkg/application" + "github.com/networkteam/sdd/pkg/llm" + "github.com/networkteam/sdd/pkg/llm/embed" + "github.com/networkteam/sdd/pkg/local" + mcpserver "github.com/networkteam/sdd/pkg/mcpapp" +) + +func TestMCPUsesApplicationPreparationAndCoverage(t *testing.T) { + for _, synchronous := range []bool{false, true} { + name := "external consumer" + if synchronous { + name = "local" + } + t.Run(name, func(t *testing.T) { + graph, err := local.NewFilesystemGraphStore(local.FilesystemGraphStoreOptions{Project: "root-test", GraphDir: writeFixtureGraph(t)}) + if err != nil { + t.Fatal(err) + } + sessions, err := local.NewFilesystemSessionStoreAt(t.TempDir()) + if err != nil { + t.Fatal(err) + } + blobs, err := local.NewFilesystemStagedBlobStoreAt(t.TempDir()) + if err != nil { + t.Fatal(err) + } + runtime, err := sdd.NewProjectRuntime(sdd.ProjectRuntimeOptions{Project: sdd.ProjectRef{ID: "root-test"}, Graph: graph, ExcludeEmbeddedFromIndex: true, + LLM: llm.RunnerFunc(func(context.Context, llm.Request) (llm.Result, error) { return llm.Result{}, nil }), SearchIndex: local.NewMemorySearchIndexStore(), + Embedder: embed.EmbedderFunc{Space: "fixture", Run: func(_ context.Context, req embed.Request) (embed.Result, error) { + r := embed.Result{Vectors: make([][]float32, len(req.Texts))} + for i := range r.Vectors { + r.Vectors[i] = []float32{1, 1} + } + return r, nil + }}, + }) + if err != nil { + t.Fatal(err) + } + calls := 0 + app, err := sdd.NewApplication(sdd.ApplicationOptions{Access: rootAccess{runtime: runtime}, Sessions: sessions, StagedBlobs: blobs, PrepareSearch: func(ctx context.Context, target sdd.SearchTarget) error { + calls++ + for item, err := range target.Entries(ctx) { + if err != nil { + return err + } + if synchronous { + if err := runtime.IndexSearchEntry(ctx, sdd.IndexSearchEntryCmd{Entry: item.Entry}); err != nil { + return err + } + } + } + return nil + }}) + if err != nil { + t.Fatal(err) + } + server, err := mcpserver.New(mcpserver.Options{Application: app, SearchSyncMode: sdd.SearchSyncNone, LocalClient: true, LocalIdentity: sdd.RequestIdentity{Subject: "tester"}}) + if err != nil { + t.Fatal(err) + } + client := connect(t, server) + opened := openSession(t, client) + var result mcpserver.SearchResult + call(t, client, "search", map[string]any{"session": opened.Session, "query": "oscillation"}, &result) + if calls != 1 || len(result.Coverage) != 1 || result.Coverage[0].Complete != synchronous { + t.Fatalf("calls=%d result=%+v", calls, result) + } + if (result.Notice == "") != synchronous { + t.Fatalf("notice=%q", result.Notice) + } + }) + } +} diff --git a/pkg/mcpapp/server_test.go b/pkg/mcpapp/server_test.go index 41d8676e..d7cc6e33 100644 --- a/pkg/mcpapp/server_test.go +++ b/pkg/mcpapp/server_test.go @@ -768,7 +768,7 @@ Branch-only nebula routing evidence exists exclusively on the bound branch. `), 0o644); err != nil { t.Fatal(err) } - branchGraph, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "test", GraphDir: branchDir}) + branchGraph, err := localadapter.NewFilesystemGraphStore(localadapter.FilesystemGraphStoreOptions{Project: "test", GraphDir: branchDir, Branch: branch}) if err != nil { t.Fatal(err) } @@ -861,7 +861,7 @@ func TestToolContractSnapshot(t *testing.T) { t.Fatal(err) } got := fmt.Sprintf("%x", sha256.Sum256(encoded)) - const want = "62724c76e938ff3fca52aa189b1ee63f5aeef0d63c40849fd6324acbc05ed5f8" + const want = "89d78a41c46e005588bd229cce01bde7bae13e5573eb82f11302704786e4da40" if got != want { t.Fatalf("MCP tool contract changed: got %s, want %s", got, want) } diff --git a/pkg/mcpapp/tools.go b/pkg/mcpapp/tools.go index d5e8f39a..8e0ea7e4 100644 --- a/pkg/mcpapp/tools.go +++ b/pkg/mcpapp/tools.go @@ -200,6 +200,7 @@ type StageAttachmentResult struct { // --- free reads ------------------------------------------------------------- type SearchArgs struct { + IncludesRevision string `json:"includes_revision,omitempty" jsonschema:"require a selected revision containing this successful write"` Session string `json:"session,omitempty" jsonschema:"session handle this connection is attached to (from start_session or resume_session); required — the read runs in that session's project and branch"` Project string `json:"project,omitempty" jsonschema:"project to read in; defaults to the session's home project. Another project must lie in the home project's declared dependency closure and be one the principal can read"` Terms []string `json:"terms,omitempty" jsonschema:"text mode: regex terms combined with AND"` @@ -215,7 +216,9 @@ type SearchArgs struct { } type SearchResult struct { - Results string `json:"results" jsonschema:"matching entries with citations"` + Coverage []sdd.SearchCoverage `json:"coverage,omitempty" jsonschema:"published entry coverage for each fixed search snapshot"` + Notice string `json:"notice,omitempty" jsonschema:"readable incomplete-indexing notice"` + Results string `json:"results" jsonschema:"matching entries with citations"` } type ViewArgs struct { @@ -807,8 +810,8 @@ func (s *Server) search(ctx context.Context, req *mcp.CallToolRequest, args Sear return nil, SearchResult{}, err } result, err := s.app.Search(ctx, s.requestIdentity(req), project, sdd.SearchRequest{ - SyncMode: s.searchSyncMode, - Terms: args.Terms, Phrase: args.Query, Type: args.Type, Layer: args.Layer, Kind: args.Kind, + SyncMode: s.searchSyncMode, IncludesRevision: args.IncludesRevision, + Terms: args.Terms, Phrase: args.Query, Type: args.Type, Layer: args.Layer, Kind: args.Kind, IncludeSuperseded: args.IncludeSuperseded, Limit: limit, MaxCitations: maxCitations, Branch: branch, BranchFromSession: branchFromSession, Repos: args.Repos, AllRepos: args.AllRepos, }) @@ -822,7 +825,7 @@ func (s *Server) search(ctx context.Context, req *mcp.CallToolRequest, args Sear if strings.TrimSpace(out) == "" { out = "(no entries matched — try another phrasing, or proceed if the topic is genuinely new)" } - return nil, SearchResult{Results: out}, nil + return nil, SearchResult{Results: out, Coverage: result.Coverage, Notice: result.Notice}, nil }