From 770048bc396b349cbc82fdb05de6e5a2bbbe49fc Mon Sep 17 00:00:00 2001 From: Richie Mackay Date: Tue, 16 Jun 2026 19:05:10 +0100 Subject: [PATCH 1/4] docs(metabolism): record coreness validation result (gate failed-with-cause) Validated k-core coreness against the real ~1000-note work vault: it does NOT track durable knowledge. The top core is dominated by machine-generated hubs (daily digest, ingest log, action/open-actions registers) plus the recurring dated meeting stream, with human maps-of-content stuck mid-core. An exclude_from_graph experiment showed the metrics only clean up with aggressive curation. Conclusion recorded: coreness is a structural-centrality descriptor, not a durability axis; durability stays deferred to the Phase 3 two-strength signal, and Phase 5 must not gate protect/forget on coreness. Co-Authored-By: Claude Opus 4.8 (1M context) --- METABOLISM.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/METABOLISM.md b/METABOLISM.md index 59d7885..029c70b 100644 --- a/METABOLISM.md +++ b/METABOLISM.md @@ -156,6 +156,26 @@ coreness ranking actually puts your known durable reference notes at high corene mis-ranks them as periphery, the structural axis is wrong (likely a Phase 0 resolution bug) and no later access logging saves it. +**Validation result (2026-06-16, OneVault, 1067 notes): the gate FAILED as framed.** +k-core coreness does not track durable knowledge on a meeting-and-action-heavy work +vault. The maximum core was dominated by machine-generated hubs (the daily digest at +degree 297, the ingest log, the action-review and open-actions registers) plus the +recurring dated sync/meeting stream, with person notes mixed in. The human +maps-of-content (project indexes, dashboards) sat mid-core (k=8 of 13), only 1 of ~25 +reaching the top two core levels. Coreness tracked raw degree closely (19/30 top-set +overlap), so it is essentially "centrality in the operational mesh". + +An `exclude_from_graph` experiment (recompute coreness with scaffolding removed) showed +the effect is real but needs aggressive curation: excluding the 9 automation hubs alone +just promoted person notes (MOCs still 1/16); only after also excluding the recurring +sync/meeting notes (about 10% of the vault) did the MOCs reach the top tier (8/16). + +Conclusion: coreness is a structural-centrality descriptor, not a durability axis, on a +work vault. `exclude_from_graph` is worth shipping to keep the dashboard's graph metrics +(orphans, islands, coreness) meaningful by stripping machine scaffolding, but Phase 5 +must NOT gate protect/forget on coreness. Durability must come from the Phase 3 +two-strength (access + stability) signal, as this plan already sequences. + --- ## Phase 3 — Access log as a silent observer (~20 lines, no scorer) @@ -352,7 +372,7 @@ spent ~20 lines, not a scorer plus a dashboard plus two jobs, finding out. - [x] Phase 1: `core/health.go` + `hebb health` CLI: dangling-link, PARA-drift, oversized detectors - [ ] Phase 1: dedup precision test (throwaway script, top-30 eyeball) before building dedup - [x] Phase 2: orphans / components / k-core in `core/health.go`; worklist-first panel on `hebb serve` -- [ ] Phase 2: validate coreness puts known-durable notes at high coreness +- [x] Phase 2: validate coreness puts known-durable notes at high coreness — FAILED (2026-06-16): coreness tracks operational centrality, not durability, on a work vault (see "Validation result" in Phase 2). `exclude_from_graph` helps only with aggressive curation; durability deferred to Phase 3. - [ ] Phase 3: `access_log` table + writes in the three MCP handlers (post-result), behind a config flag - [ ] Phase 3: capture `return_count` and `followup_count` separately - [ ] EXP: run the vault for two weeks; hand-label 30 durable + 30 transitory From e26e456133d3204799d604b7317207fb5130ed26 Mon Sep 17 00:00:00 2001 From: Richie Mackay Date: Tue, 16 Jun 2026 19:05:10 +0100 Subject: [PATCH 2/4] feat(core,cli): add exclude_from_graph to drop scaffolding from graph metrics Machine-generated hub notes (a daily digest linking to hundreds of notes, the ingest log, action/open-actions registers) dominate the link graph and distort coreness, components, orphans and islands. The new [health] exclude_from_graph config (glob patterns matched against a note's title, basename, and path) drops matching notes and their edges from the graph BEFORE those metrics are computed, so the dashboard reflects human-authored structure. Exclusion is graph-only: the content detectors (dangling_link, ambiguous_link, para_drift, oversized) still run over excluded notes. A --exclude-from-graph CLI flag overrides the config for ad-hoc runs. Verified end-to-end on the real vault: excluding the automation hubs plus the recurring meeting stream drops max k-core 13 -> 7 and lets the human maps-of-content rise into the top core. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONFIG.md | 3 + cli/health.go | 22 +++ cli/health_test.go | 83 ++++++++++ core/graph.go | 61 +++++++- core/graph_test.go | 365 ++++++++++++++++++++++++++++++++++++++++++++ core/health.go | 8 +- core/vaultconfig.go | 36 +++++ 7 files changed, 570 insertions(+), 8 deletions(-) diff --git a/CONFIG.md b/CONFIG.md index 404ac38..04380e4 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -54,6 +54,8 @@ project_stale_days = 180 size_threshold = 1200 report_unresolved_links = false # attachment_extensions defaults to a built-in list when omitted. +# exclude_from_graph defaults to empty (exclude nothing). +# exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"] ``` ## Top-level keys @@ -134,6 +136,7 @@ Obsidian). | `size_threshold` | int | `1200` | Estimated token count (`len(body)/4`) above which a note is checked for multiple sections (oversized candidate). | | `report_unresolved_links` | bool | `false` | List each unresolved wiki-link (a link to a note that does not exist) as a `dangling_link` finding. Obsidian treats these as expected future notes, so they are counted but not listed by default. `hebb health --unresolved` forces listing for one run. | | `attachment_extensions` | list of string | built-in list | File extensions (no leading dot) treated as attachment links and excluded from dangling checks, since hebb does not index non-note files. Empty uses the built-in default (`png`, `jpg`, `pdf`, `pptx`, `canvas`, `excalidraw`, ...). Setting it replaces the default rather than extending it. | +| `exclude_from_graph` | list of string | empty | Glob patterns matched against a note's title, basename without `.md`, and vault-relative path (any match excludes the note). A matched note is removed from the link graph before computing connected components, k-core coreness, orphans, leaves, and islands, so machine-generated scaffolding that links to hundreds of notes does not dominate those metrics. Content detectors (`dangling_link`, `ambiguous_link`, `para_drift`, `oversized`) are unaffected and still run over all notes. Patterns use `filepath.Match` semantics (shell-style globs; `*` does not cross path separators). Default: empty (exclude nothing). The `--exclude-from-graph` flag on `hebb health` overrides this list for a single run and is useful for ad-hoc experiments. Example: `exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"]` | Folder links (a target ending in `/`, or one naming a real directory) are never treated as broken note links. diff --git a/cli/health.go b/cli/health.go index 12076a0..ef421d2 100644 --- a/cli/health.go +++ b/cli/health.go @@ -5,6 +5,7 @@ import ( "fmt" "io" "sort" + "strings" "github.com/cizer/hebb/core" "github.com/spf13/cobra" @@ -13,6 +14,7 @@ import ( func healthCmd() *cobra.Command { var asJSON bool var unresolved bool + var excludeFromGraph string c := &cobra.Command{ Use: "health", Short: "Report vault-health findings (dangling links, PARA drift, oversized notes)", @@ -56,6 +58,20 @@ func healthCmd() *cobra.Command { // default (report_unresolved_links, off unless set). reportUnresolved := unresolved || cfg.Health.ReportUnresolvedLinks + // Effective exclude-from-graph list: the --exclude-from-graph flag + // (comma-separated glob patterns) overrides the config list for this + // run. When the flag is empty, the config list is used unchanged. + if excludeFromGraph != "" { + parts := strings.Split(excludeFromGraph, ",") + trimmed := make([]string, 0, len(parts)) + for _, p := range parts { + if s := strings.TrimSpace(p); s != "" { + trimmed = append(trimmed, s) + } + } + cfg.Health.ExcludeFromGraph = trimmed + } + result, err := core.RunHealthFull(cfg, db, reportUnresolved) if err != nil { return fmt.Errorf("health check failed: %w", err) @@ -89,6 +105,12 @@ func healthCmd() *cobra.Command { } c.Flags().BoolVar(&asJSON, "json", false, "emit findings as a JSON array (for the Phase 2 dashboard)") c.Flags().BoolVar(&unresolved, "unresolved", false, "list unresolved wiki-links (links to non-existent notes), suppressed by default") + c.Flags().StringVar(&excludeFromGraph, "exclude-from-graph", "", + "comma-separated glob patterns to drop from the graph for this run (overrides exclude_from_graph in config.toml). "+ + "Patterns are matched against a note's title, basename without .md, and vault-relative path. "+ + "Excluded notes are removed from graph metrics (coreness, components, orphans, islands) but content "+ + "detectors (dangling_link, oversized, ...) still run over them. "+ + "Example: --exclude-from-graph=\"Vault Daily Digest,Ingest Log,Open Actions*\"") return c } diff --git a/cli/health_test.go b/cli/health_test.go index 8535f89..5d89d52 100644 --- a/cli/health_test.go +++ b/cli/health_test.go @@ -320,3 +320,86 @@ func TestHealthCommandStructuralSummaryLine(t *testing.T) { } } } + +// buildExcludeVaultCLI builds a vault with a high-degree hub note ("Vault Daily +// Digest") and ordinary notes, returning the vault path and the name of the hub. +func buildExcludeVaultCLI(t *testing.T) string { + t.Helper() + vault := t.TempDir() + write := func(rel, content string) { + t.Helper() + p := filepath.Join(vault, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + if err := core.DefaultVaultConfig("test").Save(vault); err != nil { + t.Fatal(err) + } + + write("Notes/A.md", "# A\n\n[[B]] [[C]]\n") + write("Notes/B.md", "# B\n\n[[A]]\n") + write("Notes/C.md", "# C\n\n[[A]]\n") + // Hub links to every other note. + write("Daily/Digest.md", "# Vault Daily Digest\n\n[[A]] [[B]] [[C]]\n") + + cfg, err := core.ResolveVault(vault, "") + if err != nil { + t.Fatal(err) + } + db, err := core.OpenDB(cfg.DBPath) + if err != nil { + t.Fatal(err) + } + if _, err := core.FullReindex(cfg, db); err != nil { + db.Close() + t.Fatal(err) + } + db.Close() + return vault +} + +// TestHealthCommandExcludeFromGraphFlag verifies that passing +// --exclude-from-graph="Vault Daily Digest" drops the hub note from the graph +// stats (node count falls by 1, the hub does not appear in text output's node +// count comparison). +func TestHealthCommandExcludeFromGraphFlag(t *testing.T) { + vault := buildExcludeVaultCLI(t) + + // Without the flag: 4 notes in the graph. + outAll, err := runHealth(t, vault) + if err != nil { + t.Fatalf("hebb health (baseline): %v\n%s", err, outAll) + } + if !strings.Contains(outAll, "4 notes") { + t.Errorf("baseline graph summary should report 4 notes:\n%s", outAll) + } + + // With the flag: 3 notes (Digest excluded). + outExcl, err := runHealth(t, vault, "--exclude-from-graph=Vault Daily Digest") + if err != nil { + t.Fatalf("hebb health --exclude-from-graph: %v\n%s", err, outExcl) + } + if !strings.Contains(outExcl, "3 notes") { + t.Errorf("excluded graph summary should report 3 notes:\n%s", outExcl) + } +} + +// TestHealthCommandExcludeFromGraphFlagMultiple verifies that a +// comma-separated list of patterns in --exclude-from-graph excludes each one. +func TestHealthCommandExcludeFromGraphFlagMultiple(t *testing.T) { + vault := buildExcludeVaultCLI(t) + + // Exclude both "Vault Daily Digest" and "A" (by title/basename). + outExcl, err := runHealth(t, vault, "--exclude-from-graph=Vault Daily Digest,A") + if err != nil { + t.Fatalf("hebb health --exclude-from-graph (multi): %v\n%s", err, outExcl) + } + if !strings.Contains(outExcl, "2 notes") { + t.Errorf("two excluded notes: graph summary should report 2 notes:\n%s", outExcl) + } +} diff --git a/core/graph.go b/core/graph.go index b72e333..f0548c1 100644 --- a/core/graph.go +++ b/core/graph.go @@ -3,6 +3,7 @@ package core import ( "database/sql" "fmt" + "path/filepath" "sort" "strings" "time" @@ -38,23 +39,64 @@ func (g *noteGraph) edgeCount() int { return total / 2 } +// matchesExcludePatterns reports whether a note should be excluded from the +// graph. A note is excluded when any of the supplied glob patterns matches any +// of the three candidates: title, basename-without-.md, or vault-relative path. +// Matching uses filepath.Match semantics (same as shell globs). An empty +// patterns slice always returns false (exclude nothing). +func matchesExcludePatterns(patterns []string, title, path string) bool { + if len(patterns) == 0 { + return false + } + base := strings.TrimSuffix(filepath.Base(path), ".md") + for _, pat := range patterns { + if ok, _ := filepath.Match(pat, title); ok { + return true + } + if ok, _ := filepath.Match(pat, base); ok { + return true + } + if ok, _ := filepath.Match(pat, path); ok { + return true + } + } + return false +} + // buildGraph reads the notes and resolved links tables and constructs the // undirected note graph. It is called by RunHealthFull (once per invocation, // shared across all graph-based detectors) and by GraphHealth (for stats-only // callers such as tests and future tooling). func buildGraph(db *sql.DB) (*noteGraph, error) { - // Load all note paths, ordered for determinism. - rows, err := db.Query("SELECT path FROM notes ORDER BY path") + return buildGraphExcluding(db, nil) +} + +// buildGraphExcluding is the underlying implementation for buildGraph. It +// accepts an optional slice of glob patterns (see matchesExcludePatterns); when +// non-empty, any note whose title, basename-without-.md, or vault-relative path +// matches ANY pattern is excluded from the graph entirely: it becomes neither a +// node nor the endpoint of any edge. Pass nil (or an empty slice) to build the +// full graph without exclusions. +func buildGraphExcluding(db *sql.DB, excludePatterns []string) (*noteGraph, error) { + // Load all note paths and titles, ordered for determinism. + rows, err := db.Query("SELECT path, title FROM notes ORDER BY path") if err != nil { return nil, fmt.Errorf("graph: load notes: %w", err) } var nodes []string + // excluded is the set of paths that are filtered out by excludePatterns. It + // is used when loading edges to drop any edge incident to an excluded node. + excluded := make(map[string]bool) for rows.Next() { - var p string - if err := rows.Scan(&p); err != nil { + var p, title string + if err := rows.Scan(&p, &title); err != nil { rows.Close() return nil, fmt.Errorf("graph: scan note path: %w", err) } + if matchesExcludePatterns(excludePatterns, title, p) { + excluded[p] = true + continue + } nodes = append(nodes, p) } rows.Close() @@ -90,6 +132,10 @@ func buildGraph(db *sql.DB) (*noteGraph, error) { lrows.Close() return nil, fmt.Errorf("graph: scan link: %w", err) } + // Drop any edge incident to an excluded note. + if excluded[src] || excluded[tgt] { + continue + } si, sok := nodeIdx[src] ti, tok := nodeIdx[tgt] if !sok || !tok { @@ -152,13 +198,16 @@ type GraphStats struct { // It returns the full GraphStats summary. GraphHealth is used directly by the // web layer (/api/health calls RunHealthFull, which builds its own graph // internally) and by graph-stats tests. RunHealthFull does not call GraphHealth; -// it calls buildGraph itself and then calls computeComponents and +// it calls buildGraphExcluding itself and then calls computeComponents and // computeCoreness directly so it can reuse the same graph for detectOrphansAndLeaves // and detectIslands without a second DB round-trip. Consolidating the two paths // would require changing GraphHealth's signature to accept a pre-built graph; // left separate to keep the public API stable. +// +// Any notes matched by cfg.Health.GetExcludeFromGraph() are removed from the +// graph before metrics are computed (see matchesExcludePatterns). func GraphHealth(cfg Config, db *sql.DB) (GraphStats, error) { - g, err := buildGraph(db) + g, err := buildGraphExcluding(db, cfg.Health.GetExcludeFromGraph()) if err != nil { return GraphStats{}, err } diff --git a/core/graph_test.go b/core/graph_test.go index 56c98ec..080293b 100644 --- a/core/graph_test.go +++ b/core/graph_test.go @@ -837,3 +837,368 @@ func equalStrSlice(a, b []string) bool { } return true } + +// buildExcludeVault creates a vault topology for exclude_from_graph tests. +// +// Topology: +// +// Notes/Hub.md links to A, B, C, D, E (hub: degree 5) +// Notes/A.md links to Hub, B +// Notes/B.md links to Hub +// Notes/C.md links to Hub +// Notes/D.md links to Hub +// Notes/E.md links to Hub +// 3-Resources/Island1.md links to Island2 (2-note island) +// 3-Resources/Island2.md links to Island1 +// 2-Areas/OldOrphan.md no links, aged 60 days +// Daily/Digest.md links to every note (297-style hub, excluded note under test) +// Daily/OpenActions.md no links (for glob exclusion test) +// +// Notes/Hub.md is placed in Notes/ (expected-orphan, never orphan-flagged). +// Daily/Digest.md is the machine-generated hub we exclude in tests. +func buildExcludeVault(t *testing.T) (Config, *sql.DB) { + t.Helper() + vault := t.TempDir() + + write := func(rel, content string) { + t.Helper() + p := filepath.Join(vault, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + write("Notes/Hub.md", "# Hub\n\n[[A]] [[B]] [[C]] [[D]] [[E]]\n") + write("Notes/A.md", "# A\n\n[[Hub]] [[B]]\n") + write("Notes/B.md", "# B\n\n[[Hub]]\n") + write("Notes/C.md", "# C\n\n[[Hub]]\n") + write("Notes/D.md", "# D\n\n[[Hub]]\n") + write("Notes/E.md", "# E\n\n[[Hub]]\n") + + // 2-note island in 3-Resources. + write("3-Resources/Island1.md", "# Island1\n\n[[Island2]]\n") + write("3-Resources/Island2.md", "# Island2\n\nSee [[Island1]].\n") + + // Old orphan in a connective folder. + write("2-Areas/OldOrphan.md", "# OldOrphan\n\nNo links here.\n") + + // Machine-generated hub: links to every named note, inflating their coreness. + // Title is "Vault Daily Digest"; basename without .md is "Digest". + // Vault-relative path is "Daily/Digest.md". + write("Daily/Digest.md", "# Vault Daily Digest\n\n"+ + "[[Hub]] [[A]] [[B]] [[C]] [[D]] [[E]] [[Island1]] [[Island2]] [[OldOrphan]]\n") + + // Note for glob matching tests: "Open Actions Register" basename "OpenActions". + write("Daily/OpenActions.md", "# Open Actions Register\n\nNo content.\n") + + cfg := Config{ + VaultPath: vault, + DBPath: filepath.Join(vault, ".hebb", "index.db"), + ExcludeDirs: defaultExcludeDirs, + Health: HealthConfig{ + OrphanStaleDays: 30, + }, + } + if err := os.MkdirAll(filepath.Dir(cfg.DBPath), 0o755); err != nil { + t.Fatal(err) + } + db, err := OpenDB(cfg.DBPath) + if err != nil { + t.Fatal(err) + } + if _, err := FullReindex(cfg, db); err != nil { + db.Close() + t.Fatal(err) + } + + // Age OldOrphan to 60 days so the orphan detector fires. + oldTime := timeNowForTest().AddDate(0, 0, -60) + orphanPath := filepath.Join(vault, "2-Areas/OldOrphan.md") + if err := os.Chtimes(orphanPath, oldTime, oldTime); err != nil { + db.Close() + t.Fatal(err) + } + if _, err := FullReindexForce(cfg, db); err != nil { + db.Close() + t.Fatal(err) + } + + return cfg, db +} + +// TestExcludeFromGraph_ExcludedHubAbsentFromCoreness verifies that when the +// machine-generated hub ("Vault Daily Digest") is listed in exclude_from_graph, +// it does not appear in Stats.Coreness and its absence causes the neighbours' +// coreness to be recomputed as if the hub were never in the graph. +func TestExcludeFromGraph_ExcludedHubAbsentFromCoreness(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // With Digest excluded, the graph is just Hub+A+B+C+D+E + Island1+Island2 + + // OldOrphan + OpenActions. Hub is the remaining highest-degree node. + cfg.Health.ExcludeFromGraph = []string{"Vault Daily Digest"} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + // The excluded note must not appear in Coreness at all. + for path := range result.Stats.Coreness { + if strings.Contains(path, "Digest") { + t.Errorf("excluded note Daily/Digest.md appears in Coreness as %q", path) + } + } + + // NodeCount must not include the excluded note. + totalNotes := 11 // Hub A B C D E Island1 Island2 OldOrphan OpenActions + full set minus Digest + // buildExcludeVault has 11 notes total; 1 excluded -> 10 in the graph. + if result.Stats.NodeCount != totalNotes-1 { + t.Errorf("NodeCount = %d, want %d (excluded 1 note)", result.Stats.NodeCount, totalNotes-1) + } +} + +// TestExcludeFromGraph_NeighboursRecomputed verifies that the coreness values +// of nodes that were neighbours of the excluded hub are recalculated as if the +// hub never existed. Without Digest in the graph, Hub, A, B, C, D, E form their +// own cluster, and C/D/E (degree 1 within that cluster) have lower coreness than +// A and B. +func TestExcludeFromGraph_NeighboursRecomputed(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // Baseline: without exclusion, every spoke gets extra coreness from Digest. + cfgWithout := cfg + cfgWithout.Health.ExcludeFromGraph = nil + baseResult, err := RunHealthFull(cfgWithout, db, false) + if err != nil { + t.Fatalf("RunHealthFull (baseline): %v", err) + } + baseHubCoreness := baseResult.Stats.Coreness["Notes/Hub.md"] + + // With Digest excluded, Hub's coreness must be at most the baseline value. + // The point is that the graph without Digest is strictly smaller, so + // coreness values can only be equal to or less than the baseline. + cfg.Health.ExcludeFromGraph = []string{"Vault Daily Digest"} + excludedResult, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull (excluded): %v", err) + } + + if _, present := excludedResult.Stats.Coreness["Daily/Digest.md"]; present { + t.Error("Daily/Digest.md must not appear in Coreness after exclusion") + } + + hubAfter := excludedResult.Stats.Coreness["Notes/Hub.md"] + if hubAfter > baseHubCoreness { + t.Errorf("Hub coreness after exclusion (%d) exceeds baseline (%d), expected <= baseline", + hubAfter, baseHubCoreness) + } +} + +// TestExcludeFromGraph_NotOrphanOrIsland verifies that an excluded note is not +// reported as an orphan or island even when it would qualify (degree 0 in the +// filtered graph). +func TestExcludeFromGraph_NotOrphanOrIsland(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + cfg.Health.ExcludeFromGraph = []string{"Vault Daily Digest"} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for _, f := range result.Findings { + if strings.Contains(f.Path, "Digest") { + if f.Type == "orphan" || f.Type == "island" || f.Type == "leaf" { + t.Errorf("excluded note must not be reported as %s: %+v", f.Type, f) + } + } + } +} + +// TestExcludeFromGraph_MatchByBasename verifies that a pattern matching the +// basename without .md (e.g. "Digest") correctly excludes the note. +func TestExcludeFromGraph_MatchByBasename(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // "Digest" is the basename without .md of Daily/Digest.md. + cfg.Health.ExcludeFromGraph = []string{"Digest"} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for path := range result.Stats.Coreness { + if strings.Contains(path, "Digest") { + t.Errorf("basename-matched note appears in Coreness as %q", path) + } + } +} + +// TestExcludeFromGraph_MatchByPath verifies that a pattern matching the +// vault-relative path (e.g. "Daily/Digest.md") correctly excludes the note. +func TestExcludeFromGraph_MatchByPath(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // "Daily/Digest.md" is the vault-relative path. + cfg.Health.ExcludeFromGraph = []string{"Daily/Digest.md"} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for path := range result.Stats.Coreness { + if strings.Contains(path, "Digest") { + t.Errorf("path-matched note appears in Coreness as %q", path) + } + } +} + +// TestExcludeFromGraph_GlobMatch verifies that a glob pattern (e.g. +// "Open Actions*") correctly excludes notes whose title, basename, or path +// matches the glob. +func TestExcludeFromGraph_GlobMatch(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // "Open Actions*" should match the title "Open Actions Register". + cfg.Health.ExcludeFromGraph = []string{"Open Actions*"} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for path := range result.Stats.Coreness { + if strings.Contains(path, "OpenActions") { + t.Errorf("glob-matched note appears in Coreness as %q", path) + } + } +} + +// TestExcludeFromGraph_ContentDetectorStillRuns verifies that content detectors +// (oversized, para_drift, dangling_link) still run over excluded notes. Exclusion +// is graph-only; it must not hide content findings. +func TestExcludeFromGraph_ContentDetectorStillRuns(t *testing.T) { + vault := t.TempDir() + write := func(rel, content string) { + t.Helper() + p := filepath.Join(vault, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + // Mainland: two connected notes so there is a graph with > 0 edges. + write("Notes/A.md", "# A\n\n[[B]]\n") + write("Notes/B.md", "# B\n\n[[A]]\n") + + // An oversized note that will be excluded from the graph. It must still + // trigger the oversized detector. + bigBody := strings.Builder{} + bigBody.WriteString("# Digest\n\n") + for section := 0; section < 4; section++ { + bigBody.WriteString("## Section\n\n") + for line := 0; line < 40; line++ { + bigBody.WriteString("This is a line of body text to pad the token count beyond 1200.\n") + } + bigBody.WriteString("\n") + } + write("Daily/Digest.md", bigBody.String()) + + cfg := Config{ + VaultPath: vault, + DBPath: filepath.Join(vault, ".hebb", "index.db"), + ExcludeDirs: defaultExcludeDirs, + Health: HealthConfig{ + ExcludeFromGraph: []string{"Digest"}, + }, + } + if err := os.MkdirAll(filepath.Dir(cfg.DBPath), 0o755); err != nil { + t.Fatal(err) + } + db, err := OpenDB(cfg.DBPath) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if _, err := FullReindex(cfg, db); err != nil { + t.Fatal(err) + } + + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + // Digest must not appear in the graph stats. + if _, present := result.Stats.Coreness["Daily/Digest.md"]; present { + t.Error("excluded note Daily/Digest.md must not appear in graph Coreness") + } + + // The oversized detector must still flag Daily/Digest.md. + var found bool + for _, f := range result.Findings { + if f.Type == "oversized" && f.Path == "Daily/Digest.md" { + found = true + } + } + if !found { + t.Errorf("oversized detector must still fire on excluded note Daily/Digest.md; findings: %+v", result.Findings) + } +} + +// TestExcludeFromGraph_EmptyListChangesNothing verifies that an empty +// exclude_from_graph list (the default) leaves all graph metrics unchanged. +func TestExcludeFromGraph_EmptyListChangesNothing(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + // Explicitly empty list. + cfg.Health.ExcludeFromGraph = []string{} + result, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + // All 11 notes must appear in the graph. + if result.Stats.NodeCount != 11 { + t.Errorf("NodeCount = %d, want 11 (empty exclude list must change nothing)", result.Stats.NodeCount) + } + // Digest must appear in Coreness. + if _, present := result.Stats.Coreness["Daily/Digest.md"]; !present { + t.Error("Daily/Digest.md must appear in Coreness when exclude list is empty") + } +} + +// TestGetExcludeFromGraph_Default verifies the accessor returns an empty slice +// when the field is not set. +func TestGetExcludeFromGraph_Default(t *testing.T) { + hc := HealthConfig{} + got := hc.GetExcludeFromGraph() + if got == nil { + t.Error("GetExcludeFromGraph() must return a non-nil slice, got nil") + } + if len(got) != 0 { + t.Errorf("GetExcludeFromGraph() = %v, want empty slice", got) + } +} + +// TestGetExcludeFromGraph_Custom verifies the accessor returns the configured +// list unchanged when it is non-empty. +func TestGetExcludeFromGraph_Custom(t *testing.T) { + patterns := []string{"Vault Daily Digest", "Ingest Log", "Open Actions*"} + hc := HealthConfig{ExcludeFromGraph: patterns} + got := hc.GetExcludeFromGraph() + if !equalStrSlice(got, patterns) { + t.Errorf("GetExcludeFromGraph() = %v, want %v", got, patterns) + } +} diff --git a/core/health.go b/core/health.go index 29e2616..3a06b0a 100644 --- a/core/health.go +++ b/core/health.go @@ -82,8 +82,12 @@ func RunHealthFull(cfg Config, db *sql.DB, reportUnresolved bool) (HealthResult, } all = append(all, os_...) - // Phase 2a: build the graph once and reuse it for all three graph metrics. - g, err := buildGraph(db) + // Phase 2a: build the graph once (with any exclude_from_graph patterns applied) + // and reuse it for all three graph metrics. Content detectors above this point + // (dangling_link, para_drift, oversized) are unaffected: they run over ALL + // notes regardless of exclusion, because exclusion is about graph centrality + // only, not about hiding note content. + g, err := buildGraphExcluding(db, cfg.Health.GetExcludeFromGraph()) if err != nil { return HealthResult{}, fmt.Errorf("graph build: %w", err) } diff --git a/core/vaultconfig.go b/core/vaultconfig.go index e266a3f..64e4bb4 100644 --- a/core/vaultconfig.go +++ b/core/vaultconfig.go @@ -86,6 +86,17 @@ type HealthConfig struct { // cannot judge them broken. When empty the accessor returns the built-in // default list. Setting this replaces the default rather than extending it. AttachmentExtensions []string `toml:"attachment_extensions"` + + // ExcludeFromGraph is an optional list of glob patterns matched against each + // note's title, basename-without-.md, and vault-relative path (any match + // excludes the note). Matching uses filepath.Match semantics. Notes that match + // are removed from the link graph BEFORE computing connected components, + // k-core coreness, orphans, leaves, and islands, so a machine-generated hub + // that would otherwise dominate those metrics is invisible to the graph + // detectors. Content detectors (dangling_link, ambiguous_link, para_drift, + // oversized) are unaffected: they still run over ALL notes, including excluded + // ones. Default: empty (exclude nothing). + ExcludeFromGraph []string `toml:"exclude_from_graph"` } // defaultAttachmentExtensions is the built-in set of file extensions the @@ -175,6 +186,18 @@ func (h HealthConfig) GetArchiveFolders() []string { return h.ArchiveFolders } +// GetExcludeFromGraph returns the configured list of glob patterns used to drop +// notes from the link graph before computing graph metrics (connected +// components, k-core coreness, orphans, leaves, islands). Returns an empty +// (non-nil) slice when the field is not set; the default behaviour is to +// exclude nothing. +func (h HealthConfig) GetExcludeFromGraph() []string { + if h.ExcludeFromGraph == nil { + return []string{} + } + return h.ExcludeFromGraph +} + // IngestConfig is the committed [ingest] block. It records ingest policy that // must travel with the vault, not live in per-user agent memory, so a cloned // or second-machine vault inherits the same behaviour. @@ -451,6 +474,19 @@ func (vc VaultConfig) Save(vaultPath string) error { buf.WriteString("# (hebb does not index non-note files). Empty uses the\n") buf.WriteString("# built-in default (png pdf pptx canvas excalidraw ...);\n") buf.WriteString("# setting it replaces the default rather than extending.\n") + buf.WriteString("# exclude_from_graph - glob patterns matched against a note's title, basename\n") + buf.WriteString("# without .md, and vault-relative path. A note is dropped\n") + buf.WriteString("# from the link graph (and thus from coreness, components,\n") + buf.WriteString("# orphan, leaf, and island metrics) when ANY pattern\n") + buf.WriteString("# matches ANY of those three candidates via filepath.Match.\n") + buf.WriteString("# Content detectors (dangling_link, oversized, ...) are\n") + buf.WriteString("# unaffected and still run over ALL notes. Default: empty\n") + buf.WriteString("# (exclude nothing). Use for machine-generated scaffolding\n") + buf.WriteString("# that would otherwise dominate graph-centrality metrics.\n") + buf.WriteString("# Example:\n") + buf.WriteString("# exclude_from_graph = [\"Vault Daily Digest\", \"Ingest Log\",\n") + buf.WriteString("# \"Action Review\", \"My Open Actions\",\n") + buf.WriteString("# \"Open Actions*\"]\n") buf.WriteString("\n") if err := toml.NewEncoder(&buf).Encode(vc); err != nil { return err From 0c7e9ec04a97f9851b8d3cab5c54d9bcb15383fc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 16 Jun 2026 22:34:49 +0000 Subject: [PATCH 3/4] fix(core): validate exclude_from_graph patterns; use slash-based path.Match Addresses the PR review on exclude_from_graph: - Reject malformed glob patterns up front. matchesExcludePatterns discarded ErrBadPattern, so a bad pattern (e.g. an unclosed "[") silently excluded nothing and graph metrics were then computed over the unfiltered graph, quietly invalidating the experiment. buildGraphExcluding now validates every pattern via validateExcludePatterns and returns an error naming the offending pattern, failing the health run (CLI and the /api/health dashboard path) instead of producing a wrong result. - Match with the slash-based path package, not OS-dependent path/filepath. Note paths are stored vault-relative and slash-separated, so path.Match / path.Base give the documented "globs do not cross /" semantics on every OS. Docs (CONFIG.md, rendered config.toml comments) updated to path.Match. New test TestExcludeFromGraph_MalformedPatternErrors covers both RunHealthFull and the stats-only GraphHealth path. https://claude.ai/code/session_01QJ1WxqGMfCTuBGgCygW1jU --- CONFIG.md | 2 +- core/graph.go | 40 ++++++++++++++++++++++++++++++++-------- core/graph_test.go | 29 +++++++++++++++++++++++++++++ core/vaultconfig.go | 7 +++++-- 4 files changed, 67 insertions(+), 11 deletions(-) diff --git a/CONFIG.md b/CONFIG.md index 04380e4..f85bcdd 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -136,7 +136,7 @@ Obsidian). | `size_threshold` | int | `1200` | Estimated token count (`len(body)/4`) above which a note is checked for multiple sections (oversized candidate). | | `report_unresolved_links` | bool | `false` | List each unresolved wiki-link (a link to a note that does not exist) as a `dangling_link` finding. Obsidian treats these as expected future notes, so they are counted but not listed by default. `hebb health --unresolved` forces listing for one run. | | `attachment_extensions` | list of string | built-in list | File extensions (no leading dot) treated as attachment links and excluded from dangling checks, since hebb does not index non-note files. Empty uses the built-in default (`png`, `jpg`, `pdf`, `pptx`, `canvas`, `excalidraw`, ...). Setting it replaces the default rather than extending it. | -| `exclude_from_graph` | list of string | empty | Glob patterns matched against a note's title, basename without `.md`, and vault-relative path (any match excludes the note). A matched note is removed from the link graph before computing connected components, k-core coreness, orphans, leaves, and islands, so machine-generated scaffolding that links to hundreds of notes does not dominate those metrics. Content detectors (`dangling_link`, `ambiguous_link`, `para_drift`, `oversized`) are unaffected and still run over all notes. Patterns use `filepath.Match` semantics (shell-style globs; `*` does not cross path separators). Default: empty (exclude nothing). The `--exclude-from-graph` flag on `hebb health` overrides this list for a single run and is useful for ad-hoc experiments. Example: `exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"]` | +| `exclude_from_graph` | list of string | empty | Glob patterns matched against a note's title, basename without `.md`, and vault-relative path (any match excludes the note). A matched note is removed from the link graph before computing connected components, k-core coreness, orphans, leaves, and islands, so machine-generated scaffolding that links to hundreds of notes does not dominate those metrics. Content detectors (`dangling_link`, `ambiguous_link`, `para_drift`, `oversized`) are unaffected and still run over all notes. Patterns use `path.Match` semantics (shell-style globs over the `/`-separated vault path; `*` does not cross `/`). A malformed pattern fails the run with a clear error rather than being silently ignored. Default: empty (exclude nothing). The `--exclude-from-graph` flag on `hebb health` overrides this list for a single run and is useful for ad-hoc experiments. Example: `exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"]` | Folder links (a target ending in `/`, or one naming a real directory) are never treated as broken note links. diff --git a/core/graph.go b/core/graph.go index f0548c1..917ce4d 100644 --- a/core/graph.go +++ b/core/graph.go @@ -3,7 +3,7 @@ package core import ( "database/sql" "fmt" - "path/filepath" + "path" "sort" "strings" "time" @@ -39,24 +39,42 @@ func (g *noteGraph) edgeCount() int { return total / 2 } +// validateExcludePatterns checks every glob pattern for validity up front, so a +// malformed pattern (e.g. an unclosed "[") is reported rather than silently +// treated as a non-match. The whole point of the feature is graph-metric +// fidelity, so silently computing the wrong graph from a typo would invalidate +// the result. path.Match reports a bad pattern via ErrBadPattern regardless of +// the candidate string, so an empty candidate is enough to validate. An empty +// patterns slice validates trivially. +func validateExcludePatterns(patterns []string) error { + for _, pat := range patterns { + if _, err := path.Match(pat, ""); err != nil { + return fmt.Errorf("invalid exclude_from_graph pattern %q: %w", pat, err) + } + } + return nil +} + // matchesExcludePatterns reports whether a note should be excluded from the // graph. A note is excluded when any of the supplied glob patterns matches any // of the three candidates: title, basename-without-.md, or vault-relative path. -// Matching uses filepath.Match semantics (same as shell globs). An empty -// patterns slice always returns false (exclude nothing). -func matchesExcludePatterns(patterns []string, title, path string) bool { +// Matching uses path.Match semantics (shell-style globs over the '/'-separated +// vault path, OS-independent: "*" does not cross "/"). Patterns are assumed +// pre-validated by validateExcludePatterns, so a match error here cannot occur. +// An empty patterns slice always returns false (exclude nothing). +func matchesExcludePatterns(patterns []string, title, notePath string) bool { if len(patterns) == 0 { return false } - base := strings.TrimSuffix(filepath.Base(path), ".md") + base := strings.TrimSuffix(path.Base(notePath), ".md") for _, pat := range patterns { - if ok, _ := filepath.Match(pat, title); ok { + if ok, _ := path.Match(pat, title); ok { return true } - if ok, _ := filepath.Match(pat, base); ok { + if ok, _ := path.Match(pat, base); ok { return true } - if ok, _ := filepath.Match(pat, path); ok { + if ok, _ := path.Match(pat, notePath); ok { return true } } @@ -78,6 +96,12 @@ func buildGraph(db *sql.DB) (*noteGraph, error) { // node nor the endpoint of any edge. Pass nil (or an empty slice) to build the // full graph without exclusions. func buildGraphExcluding(db *sql.DB, excludePatterns []string) (*noteGraph, error) { + // Validate the patterns before touching the DB: a malformed glob must fail + // the run with a clear message, not silently exclude nothing and report + // metrics over the unfiltered graph. + if err := validateExcludePatterns(excludePatterns); err != nil { + return nil, err + } // Load all note paths and titles, ordered for determinism. rows, err := db.Query("SELECT path, title FROM notes ORDER BY path") if err != nil { diff --git a/core/graph_test.go b/core/graph_test.go index 080293b..55b9ce3 100644 --- a/core/graph_test.go +++ b/core/graph_test.go @@ -1202,3 +1202,32 @@ func TestGetExcludeFromGraph_Custom(t *testing.T) { t.Errorf("GetExcludeFromGraph() = %v, want %v", got, patterns) } } + +// TestExcludeFromGraph_MalformedPatternErrors verifies that a malformed glob in +// exclude_from_graph fails the health run with an error naming the pattern, +// rather than being silently ignored (which would compute graph metrics over the +// unfiltered graph and quietly invalidate the result). +func TestExcludeFromGraph_MalformedPatternErrors(t *testing.T) { + cfg, db := buildExcludeVault(t) + defer db.Close() + + cfg.Health.ExcludeFromGraph = []string{"[unclosed"} + _, err := RunHealthFull(cfg, db, false) + if err == nil { + t.Fatal("RunHealthFull must fail on a malformed exclude_from_graph pattern") + } + if !strings.Contains(err.Error(), "[unclosed") { + t.Errorf("error should name the offending pattern, got: %v", err) + } + + // The stats-only path (used by the dashboard's /api/health) must reject it too. + if _, err := GraphHealth(cfg, db); err == nil { + t.Fatal("GraphHealth must fail on a malformed exclude_from_graph pattern") + } + + // A valid glob is unaffected: the guard rejects only malformed patterns. + cfg.Health.ExcludeFromGraph = []string{"Open Actions*"} + if _, err := RunHealthFull(cfg, db, false); err != nil { + t.Fatalf("valid pattern must not error: %v", err) + } +} diff --git a/core/vaultconfig.go b/core/vaultconfig.go index 64e4bb4..75c893e 100644 --- a/core/vaultconfig.go +++ b/core/vaultconfig.go @@ -89,7 +89,9 @@ type HealthConfig struct { // ExcludeFromGraph is an optional list of glob patterns matched against each // note's title, basename-without-.md, and vault-relative path (any match - // excludes the note). Matching uses filepath.Match semantics. Notes that match + // excludes the note). Matching uses path.Match semantics (shell-style globs + // over the '/'-separated vault path; a malformed pattern fails the health run + // rather than being silently ignored). Notes that match // are removed from the link graph BEFORE computing connected components, // k-core coreness, orphans, leaves, and islands, so a machine-generated hub // that would otherwise dominate those metrics is invisible to the graph @@ -478,7 +480,8 @@ func (vc VaultConfig) Save(vaultPath string) error { buf.WriteString("# without .md, and vault-relative path. A note is dropped\n") buf.WriteString("# from the link graph (and thus from coreness, components,\n") buf.WriteString("# orphan, leaf, and island metrics) when ANY pattern\n") - buf.WriteString("# matches ANY of those three candidates via filepath.Match.\n") + buf.WriteString("# matches ANY of those three candidates via path.Match\n") + buf.WriteString("# (a malformed glob fails the run, not silently ignored).\n") buf.WriteString("# Content detectors (dangling_link, oversized, ...) are\n") buf.WriteString("# unaffected and still run over ALL notes. Default: empty\n") buf.WriteString("# (exclude nothing). Use for machine-generated scaffolding\n") From 333522b4b7c4ca55c381116a0f29922847a3ccef Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 16 Jun 2026 23:01:39 +0000 Subject: [PATCH 4/4] test(acceptance): assert the global web launchd plist, not a per-vault one The multi-vault refactor made the web launchd service one machine-global job (local.hebb.web), retiring per-vault web plists, but scripts/acceptance.sh still asserted the per-vault glob local.hebb.*.web.plist, which the global plist can never match (install/run.go notes the same). That left the "web launchd plist rendered" check failing on current main and on any branch merge-tested against it. Assert the exact global plist local.hebb.web.plist instead. https://claude.ai/code/session_01QJ1WxqGMfCTuBGgCygW1jU --- scripts/acceptance.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/acceptance.sh b/scripts/acceptance.sh index 5372033..0f4647d 100755 --- a/scripts/acceptance.sh +++ b/scripts/acceptance.sh @@ -99,7 +99,9 @@ sed 's/^/ /' "$WORK/install.out" [ -d "$VAULT/.hebb/memory" ]; report $? "memory dir under .hebb" ls "$HOME_DIR"/.claude/projects/*/memory >/dev/null 2>&1; report $? "memory linked into claude project dir" [ ! -e "$VAULT/.claude/skills" ]; report $? "no project skills dir (the plugin delivers skills)" -ls "$LAUNCHD"/local.hebb.*.web.plist >/dev/null 2>&1; report $? "web launchd plist rendered" +# The web launchd service is one machine-global job (local.hebb.web), not a +# per-vault one, since the multi-vault web refactor; assert that exact plist. +[ -f "$LAUNCHD"/local.hebb.web.plist ]; report $? "web launchd plist rendered" # --- plugin-less wiring (--mcp-json opt-in) ----------------------------------- echo "==> install --mcp-json (opt-in plugin-less wiring)"