diff --git a/CONFIG.md b/CONFIG.md index fb57c59..4731f64 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -51,7 +51,8 @@ url = "" [health] project_stale_days = 180 -size_threshold = 1200 +size_threshold = 4000 +# stub_threshold defaults to 20 when omitted. report_unresolved_links = false # attachment_extensions defaults to a built-in list when omitted. # exclude_from_graph defaults to empty (exclude nothing). @@ -136,10 +137,11 @@ Obsidian). | Key | Type | Default | Meaning | | --- | --- | --- | --- | | `project_stale_days` | int | `180` | Days without modification before a `1-Projects/` note is flagged as PARA drift. | -| `size_threshold` | int | `1200` | Estimated token count (`len(body)/4`) above which a note is checked for multiple sections (oversized candidate). | +| `size_threshold` | int | `4000` | Estimated token count (`len(body)/4`) above which a multi-section note is flagged as an oversized split candidate. The default targets the genuinely bloated top few percent of notes in a typical vault; tune this per vault if your median note size differs significantly. | +| `stub_threshold` | int | `20` | Estimated token count (`len(body)/4`) below which a note is considered near-empty for the stub detector. A note is a stub candidate only when ALL of: token count is below this threshold, it has zero outbound resolved links, and it is not under `expected_orphan_folders`. | | `report_unresolved_links` | bool | `false` | List each unresolved wiki-link (a link to a note that does not exist) as a `dangling_link` finding. Obsidian treats these as expected future notes, so they are counted but not listed by default. `hebb audit --unresolved` forces listing for one run. | | `attachment_extensions` | list of string | built-in list | File extensions (no leading dot) treated as attachment links and excluded from dangling checks, since hebb does not index non-note files. Empty uses the built-in default (`png`, `jpg`, `pdf`, `pptx`, `canvas`, `excalidraw`, ...). Setting it replaces the default rather than extending it. | -| `exclude_from_graph` | list of string | empty | Glob patterns matched against a note's title, basename without `.md`, and vault-relative path (any match excludes the note). A matched note is removed from the link graph before computing connected components, k-core coreness, orphans, leaves, and islands, so machine-generated scaffolding that links to hundreds of notes does not dominate those metrics. Content detectors (`dangling_link`, `ambiguous_link`, `para_drift`, `oversized`) are unaffected and still run over all notes. Patterns use `path.Match` semantics (shell-style globs over the `/`-separated vault path; `*` does not cross `/`). A malformed pattern fails the run with a clear error rather than being silently ignored. Default: empty (exclude nothing). The `--exclude-from-graph` flag on `hebb health` overrides this list for a single run and is useful for ad-hoc experiments. Example: `exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"]` | +| `exclude_from_graph` | list of string | empty | Glob patterns matched against a note's title, basename without `.md`, and vault-relative path (any match excludes the note). A matched note is removed from the link graph before computing connected components, k-core coreness, orphans, leaves, and islands, so machine-generated scaffolding that links to hundreds of notes does not dominate those metrics. Content detectors (`dangling_link`, `ambiguous_link`, `para_drift`, `oversized`, `stub`) are unaffected and still run over all notes. Patterns use `path.Match` semantics (shell-style globs over the `/`-separated vault path; `*` does not cross `/`). A malformed pattern fails the run with a clear error rather than being silently ignored. Default: empty (exclude nothing). The `--exclude-from-graph` flag on `hebb health` overrides this list for a single run and is useful for ad-hoc experiments. Example: `exclude_from_graph = ["Vault Daily Digest", "Ingest Log", "Action Review", "My Open Actions", "Open Actions*"]` | Folder links (a target ending in `/`, or one naming a real directory) are never treated as broken note links. diff --git a/cli/health.go b/cli/health.go index bb565a4..af9800e 100644 --- a/cli/health.go +++ b/cli/health.go @@ -25,7 +25,8 @@ func healthCmd() *cobra.Command { " dangling_link wiki-links with no matching note\n" + " ambiguous_link wiki-links that match more than one note\n" + " para_drift 1-Projects/ notes that are done or stale\n" + - " oversized notes over the token threshold with multiple sections\n\n" + + " oversized notes over the token threshold with multiple sections\n" + + " stub near-empty notes with no outbound links (merge or archive candidate)\n\n" + "Wiki-links are resolved case-insensitively (matching Obsidian), and\n" + "attachment links (.png, .pdf, ...) and folder links are not treated as\n" + "broken note links. Links to notes that do not exist yet (Obsidian\n" + @@ -110,14 +111,14 @@ func healthCmd() *cobra.Command { "comma-separated glob patterns to drop from the graph for this run (overrides exclude_from_graph in config.toml). "+ "Patterns are matched against a note's title, basename without .md, and vault-relative path. "+ "Excluded notes are removed from graph metrics (coreness, components, orphans, islands) but content "+ - "detectors (dangling_link, oversized, ...) still run over them. "+ + "detectors (dangling_link, oversized, stub, ...) still run over them. "+ "Example: --exclude-from-graph=\"Vault Daily Digest,Ingest Log,Open Actions*\"") return c } // typeOrder is the fixed display order for finding types. Types not listed here // appear last in lexicographic order (forward-compatibility with future detectors). -var typeOrder = []string{"dangling_link", "ambiguous_link", "para_drift", "oversized", "orphan", "leaf", "island"} +var typeOrder = []string{"dangling_link", "ambiguous_link", "para_drift", "oversized", "stub", "orphan", "leaf", "island"} // printGraphSummary writes the one-line structural graph summary to cmd's // output writer. It is printed above the findings worklist in text mode. diff --git a/cli/health_test.go b/cli/health_test.go index 5d89d52..feda5f5 100644 --- a/cli/health_test.go +++ b/cli/health_test.go @@ -52,12 +52,12 @@ func buildHealthVaultCLI(t *testing.T) string { // PARA drift: done project. write("1-Projects/Done.md", "---\ntitle: Done\nstatus: done\n---\n\nFinished.\n") - // Oversized: token-heavy body with 4 H2 sections. + // Oversized: token-heavy body with 4 H2 sections, above the 4000-token default threshold. bigBody := strings.Builder{} bigBody.WriteString("# Big Note\n\n") for section := 0; section < 4; section++ { bigBody.WriteString("## Section\n\n") - for line := 0; line < 40; line++ { + for line := 0; line < 160; line++ { bigBody.WriteString("This is a line of body text in the section to pad out the token count.\n") } bigBody.WriteString("\n") diff --git a/core/graph_test.go b/core/graph_test.go index 55b9ce3..1f12b6f 100644 --- a/core/graph_test.go +++ b/core/graph_test.go @@ -1107,8 +1107,8 @@ func TestExcludeFromGraph_ContentDetectorStillRuns(t *testing.T) { bigBody.WriteString("# Digest\n\n") for section := 0; section < 4; section++ { bigBody.WriteString("## Section\n\n") - for line := 0; line < 40; line++ { - bigBody.WriteString("This is a line of body text to pad the token count beyond 1200.\n") + for line := 0; line < 160; line++ { + bigBody.WriteString("This is a line of body text to pad the token count beyond 4000.\n") } bigBody.WriteString("\n") } diff --git a/core/health.go b/core/health.go index 3a06b0a..350d075 100644 --- a/core/health.go +++ b/core/health.go @@ -20,7 +20,7 @@ var timeNowForTest = time.Now // index. type Finding struct { // Type identifies the finding: "dangling_link", "ambiguous_link", - // "para_drift", "oversized". + // "para_drift", "oversized", "stub". Type string `json:"type"` // Path is the vault-relative path of the affected note. Path string `json:"path"` @@ -82,11 +82,17 @@ func RunHealthFull(cfg Config, db *sql.DB, reportUnresolved bool) (HealthResult, } all = append(all, os_...) + sb, err := detectStub(cfg, db) + if err != nil { + return HealthResult{}, fmt.Errorf("stub detector: %w", err) + } + all = append(all, sb...) + // Phase 2a: build the graph once (with any exclude_from_graph patterns applied) - // and reuse it for all three graph metrics. Content detectors above this point - // (dangling_link, para_drift, oversized) are unaffected: they run over ALL - // notes regardless of exclusion, because exclusion is about graph centrality - // only, not about hiding note content. + // and reuse it for all three graph metrics. The content detectors above this + // point (dangling_link, ambiguous_link, para_drift, oversized, stub) are + // unaffected: they run over ALL notes regardless of exclusion, because + // exclusion is about graph centrality only, not about hiding note content. g, err := buildGraphExcluding(db, cfg.Health.GetExcludeFromGraph()) if err != nil { return HealthResult{}, fmt.Errorf("graph build: %w", err) @@ -422,6 +428,114 @@ func detectOversized(cfg Config, db *sql.DB) ([]Finding, error) { return findings, nil } +// detectStub finds notes that are genuine dead stubs: near-empty notes with no +// outbound resolved links that are not under the configured expected-orphan +// folders. All three conditions must hold: +// +// 1. The note's estimated token count (len(body)/4) is below +// cfg.Health.GetStubThreshold() (default 20). +// 2. The note has zero outbound resolved links: no row in links where +// source_path = this note AND target_path IS NOT NULL. A thin note that +// links out is likely an intentional map/index stub, not a dead one. +// 3. The note is NOT under any of the expected-orphan folder prefixes +// (cfg.Health.GetExpectedOrphanFolders(), default ["Journal", "Notes", +// "4-Archives"]), where short sparse notes are entirely normal. +// +// The detector runs over all notes regardless of exclude_from_graph (it +// concerns the note's own content and outbound links, not graph centrality). +func detectStub(cfg Config, db *sql.DB) ([]Finding, error) { + threshold := cfg.Health.GetStubThreshold() + orphanFolders := cfg.Health.GetExpectedOrphanFolders() + + // Select notes whose body token estimate is below the stub threshold. + rows, err := db.Query(` + SELECT path, body + FROM notes + WHERE (length(body) / 4) < ? + ORDER BY path + `, threshold) + if err != nil { + return nil, err + } + type candidate struct { + path string + tokens int + } + var candidates []candidate + scanErr := func() error { + defer rows.Close() + for rows.Next() { + var path, body string + if err := rows.Scan(&path, &body); err != nil { + return err + } + candidates = append(candidates, candidate{path: path, tokens: len(body) / 4}) + } + return rows.Err() + }() + if scanErr != nil { + return nil, scanErr + } + + // The set of notes with at least one resolved outbound link, built in a single + // query rather than a COUNT per candidate, so the stub check does not degrade + // into an N+1 pattern that slows `hebb audit` on large vaults. + linked := map[string]bool{} + lrows, err := db.Query(`SELECT DISTINCT source_path FROM links WHERE target_path IS NOT NULL`) + if err != nil { + return nil, err + } + if scanErr := func() error { + defer lrows.Close() + for lrows.Next() { + var p string + if err := lrows.Scan(&p); err != nil { + return err + } + linked[p] = true + } + return lrows.Err() + }(); scanErr != nil { + return nil, scanErr + } + + var findings []Finding + for _, c := range candidates { + // Condition 3: skip notes under expected-orphan folders. + if isUnderFolders(c.path, orphanFolders) { + continue + } + + // Condition 2: skip notes that have at least one resolved outbound link + // (a thin note that links out is an intentional map or index stub). + if linked[c.path] { + continue + } + + findings = append(findings, Finding{ + Type: "stub", + Path: c.path, + Detail: fmt.Sprintf("near-empty (~%d tokens), links nowhere - merge or archive?", c.tokens), + Severity: "warn", + }) + } + return findings, nil +} + +// isUnderFolders reports whether the vault-relative path sits under any of the +// given folder prefixes. The comparison is prefix-based: a folder "Journal" +// matches "Journal/foo.md" but not "JournalExtra/foo.md". The path separator +// is always '/' in vault-relative paths. +func isUnderFolders(path string, folders []string) bool { + for _, folder := range folders { + prefix := strings.TrimSuffix(folder, "/") + "/" + if strings.HasPrefix(path, prefix) { + return true + } + } + return false +} + // countSubstantialSections counts the number of H2/H3 headings in raw markdown // that have at least one non-blank line of body content following them before // the next H2/H3/H1 heading or end-of-file. This guards against treating a diff --git a/core/health_test.go b/core/health_test.go index be06c2e..e025b13 100644 --- a/core/health_test.go +++ b/core/health_test.go @@ -30,15 +30,15 @@ func buildHealthVault(t *testing.T) (Config, *sql.DB) { // (ii) A 1-Projects note with status: done (PARA drift by status). write("1-Projects/Foo.md", "---\ntitle: Foo\nstatus: done\n---\n\nFinished project.\n") - // (iii) An oversized note: body long enough to exceed the default 1200-token - // threshold (1200 * 4 = 4800 chars) and containing >= 3 H2/H3 sections, + // (iii) An oversized note: body long enough to exceed the default 4000-token + // threshold (4000 * 4 = 16000 chars) and containing >= 3 H2/H3 sections, // each with non-trivial content. bigBody := strings.Builder{} bigBody.WriteString("# Big Note\n\n") for section := 0; section < 4; section++ { bigBody.WriteString("## Section\n\n") // Each section needs enough text to be considered substantial. - for line := 0; line < 40; line++ { + for line := 0; line < 160; line++ { bigBody.WriteString("This is a line of body text in the section to pad out the token count.\n") } bigBody.WriteString("\n") @@ -49,8 +49,9 @@ func buildHealthVault(t *testing.T) (Config, *sql.DB) { // - A resolved link (target exists). write("Notes/Target.md", "# Target\n\nA real note.\n") write("Notes/Resolved.md", "# Resolved\n\nSee [[Target]] for details.\n") - // - A 1-Projects note with an active status. - write("1-Projects/Active.md", "---\ntitle: Active\nstatus: in-progress\n---\n\nStill going.\n") + // - A 1-Projects note with an active status and a link to the target note (so + // the stub detector does not flag it: it has an outbound resolved link). + write("1-Projects/Active.md", "---\ntitle: Active\nstatus: in-progress\n---\n\nStill going. See [[Target]].\n") // - A small note (well under the token threshold). write("Notes/Small.md", "# Small\n\nJust a tiny note.\n") @@ -437,6 +438,7 @@ func TestRunHealthCleanNotesNotFlagged(t *testing.T) { "dangling_link": true, "para_drift": true, "oversized": true, + "stub": true, } clean := []string{"Notes/Target.md", "Notes/Resolved.md", "1-Projects/Active.md", "Notes/Small.md"} for _, path := range clean { @@ -534,8 +536,11 @@ func TestHealthConfigDefaults(t *testing.T) { if hc.GetProjectStaleDays() != 180 { t.Errorf("ProjectStaleDays default = %d, want 180", hc.GetProjectStaleDays()) } - if hc.GetSizeThreshold() != 1200 { - t.Errorf("SizeThreshold default = %d, want 1200", hc.GetSizeThreshold()) + if hc.GetSizeThreshold() != 4000 { + t.Errorf("SizeThreshold default = %d, want 4000", hc.GetSizeThreshold()) + } + if hc.GetStubThreshold() != 20 { + t.Errorf("StubThreshold default = %d, want 20", hc.GetStubThreshold()) } } @@ -548,3 +553,241 @@ func TestHealthConfigCustom(t *testing.T) { t.Errorf("SizeThreshold = %d, want 500", hc.GetSizeThreshold()) } } + +// buildStubVault creates a minimal vault for stub-detector tests and returns +// cfg + an open, reindexed DB. Caller must defer db.Close(). +// +// Notes written: +// - 2-Areas/Stub.md -- 5 words, no outbound links -> should fire +// - 2-Areas/ThinLinker.md -- 5 words but has an outbound link -> should NOT fire +// - Journal/ShortJournal.md -- 5 words, no links, but under Journal/ -> NOT fired +// - 2-Areas/LongNote.md -- many words, no outbound links -> NOT fired (token count too high) +// - 2-Areas/Target.md -- target for the ThinLinker resolved link +// +// Note: "Notes/" is in the default expected_orphan_folders, so stub notes are +// placed under "2-Areas/" which is NOT an expected-orphan folder. +func buildStubVault(t *testing.T) (Config, *sql.DB) { + t.Helper() + vault := t.TempDir() + write := func(rel, content string) { + t.Helper() + p := filepath.Join(vault, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + // A near-empty note with no outbound links and not in an expected-orphan folder: + // should be flagged as a stub. Under 2-Areas/ which is not an orphan folder. + write("2-Areas/Stub.md", "# Stub\n\nhello world tiny note\n") + + // A thin note that has an outbound link: intentional index stub, NOT flagged. + write("2-Areas/Target.md", "# Target\n\nA real note with content.\n") + write("2-Areas/ThinLinker.md", "# Thin\n\nSee [[Target]] here\n") + + // A thin note under Journal/: expected-orphan folder, NOT flagged. + write("Journal/ShortJournal.md", "# Entry\n\nhello world entry\n") + + // A longer note with no outbound links: token count above stub threshold, NOT flagged. + longBody := strings.Builder{} + longBody.WriteString("# Long\n\n") + for i := 0; i < 30; i++ { + longBody.WriteString("This is a longer line of text to push the token estimate above the stub threshold.\n") + } + write("2-Areas/LongNote.md", longBody.String()) + + cfg := Config{ + VaultPath: vault, + DBPath: filepath.Join(vault, ".hebb", "index.db"), + ExcludeDirs: defaultExcludeDirs, + } + if err := os.MkdirAll(filepath.Dir(cfg.DBPath), 0o755); err != nil { + t.Fatal(err) + } + db, err := OpenDB(cfg.DBPath) + if err != nil { + t.Fatal(err) + } + if _, err := FullReindex(cfg, db); err != nil { + db.Close() + t.Fatal(err) + } + return cfg, db +} + +// TestStubDetectorFlagsNearEmptyNote proves a near-empty note (few tokens) with +// no outbound resolved links and not under an expected-orphan folder is flagged +// as a stub finding. +func TestStubDetectorFlagsNearEmptyNote(t *testing.T) { + cfg, db := buildStubVault(t) + defer db.Close() + + report, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + stubs := findingsByType(report.Findings, "stub") + var found bool + for _, f := range stubs { + if f.Path == "2-Areas/Stub.md" { + found = true + if f.Severity != "warn" { + t.Errorf("stub finding severity = %q, want warn", f.Severity) + } + if !strings.Contains(f.Detail, "tokens") { + t.Errorf("stub detail %q should mention tokens", f.Detail) + } + if !strings.Contains(strings.ToLower(f.Detail), "merge") && !strings.Contains(strings.ToLower(f.Detail), "archive") { + t.Errorf("stub detail %q should mention merge or archive", f.Detail) + } + } + } + if !found { + t.Errorf("2-Areas/Stub.md should be flagged as a stub; stubs: %+v, all findings: %+v", stubs, report.Findings) + } +} + +// TestStubDetectorDoesNotFlagNoteWithOutboundLink proves a thin note that has +// at least one resolved outbound link is NOT flagged: it is an intentional +// index/map stub, not a dead stub. +func TestStubDetectorDoesNotFlagNoteWithOutboundLink(t *testing.T) { + cfg, db := buildStubVault(t) + defer db.Close() + + report, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for _, f := range findingsByType(report.Findings, "stub") { + if f.Path == "2-Areas/ThinLinker.md" { + t.Errorf("2-Areas/ThinLinker.md has an outbound link and must NOT be flagged as a stub: %+v", f) + } + } +} + +// TestStubDetectorDoesNotFlagJournalNote proves a short note under Journal/ +// (an expected-orphan folder) is NOT flagged even if it has no outbound links. +func TestStubDetectorDoesNotFlagJournalNote(t *testing.T) { + cfg, db := buildStubVault(t) + defer db.Close() + + report, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for _, f := range findingsByType(report.Findings, "stub") { + if f.Path == "Journal/ShortJournal.md" { + t.Errorf("Journal/ShortJournal.md is in an expected-orphan folder and must NOT be flagged as a stub: %+v", f) + } + } +} + +// TestStubDetectorDoesNotFlagLongNote proves a note whose token estimate +// exceeds the stub threshold is NOT flagged even if it has no outbound links. +func TestStubDetectorDoesNotFlagLongNote(t *testing.T) { + cfg, db := buildStubVault(t) + defer db.Close() + + report, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + for _, f := range findingsByType(report.Findings, "stub") { + if f.Path == "2-Areas/LongNote.md" { + t.Errorf("2-Areas/LongNote.md is long enough and must NOT be flagged as a stub: %+v", f) + } + } +} + +// TestOversizedThresholdRecalibrated proves the default 4000-token threshold +// means a ~1500-token note with sections is NOT flagged, but a ~5000-token +// note with sections IS flagged. +func TestOversizedThresholdRecalibrated(t *testing.T) { + vault := t.TempDir() + write := func(rel, content string) { + t.Helper() + p := filepath.Join(vault, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + // ~1500-token note with 3 substantial H2 sections. + // 1500 tokens * 4 chars/token = 6000 chars of body content. + // We use 1400 tokens worth of body so it is clearly below 4000. + smallSections := strings.Builder{} + smallSections.WriteString("# Moderate Note\n\n") + for section := 0; section < 3; section++ { + smallSections.WriteString("## Section\n\n") + for line := 0; line < 22; line++ { + // ~75 chars per line; 22 lines per section * 3 sections = ~1500 tokens + smallSections.WriteString("This is a line of body text in the section to pad the token count here.\n") + } + smallSections.WriteString("\n") + } + write("Notes/Moderate.md", smallSections.String()) + + // ~5000-token note with 4 substantial H2 sections (above the 4000 threshold). + hugeSections := strings.Builder{} + hugeSections.WriteString("# Huge Note\n\n") + for section := 0; section < 4; section++ { + hugeSections.WriteString("## Section\n\n") + for line := 0; line < 130; line++ { + hugeSections.WriteString("This is a line of body text in the section to pad the token count here.\n") + } + hugeSections.WriteString("\n") + } + write("Notes/Huge.md", hugeSections.String()) + + cfg := Config{ + VaultPath: vault, + DBPath: filepath.Join(vault, ".hebb", "index.db"), + ExcludeDirs: defaultExcludeDirs, + } + if err := os.MkdirAll(filepath.Dir(cfg.DBPath), 0o755); err != nil { + t.Fatal(err) + } + db, err := OpenDB(cfg.DBPath) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if _, err := FullReindex(cfg, db); err != nil { + t.Fatal(err) + } + + report, err := RunHealthFull(cfg, db, false) + if err != nil { + t.Fatalf("RunHealthFull: %v", err) + } + + oversized := findingsByType(report.Findings, "oversized") + + // The ~1500-token note must NOT be flagged. + for _, f := range oversized { + if f.Path == "Notes/Moderate.md" { + t.Errorf("Notes/Moderate.md (~1500 tokens) must NOT be flagged with default threshold 4000; got: %+v", f) + } + } + + // The ~5000-token note MUST be flagged. + var hugeFound bool + for _, f := range oversized { + if f.Path == "Notes/Huge.md" { + hugeFound = true + } + } + if !hugeFound { + t.Errorf("Notes/Huge.md (~5000 tokens, 4 sections) MUST be flagged with default threshold 4000; oversized: %+v", oversized) + } +} diff --git a/core/vaultconfig.go b/core/vaultconfig.go index 86df165..af6c5d1 100644 --- a/core/vaultconfig.go +++ b/core/vaultconfig.go @@ -49,8 +49,13 @@ type HealthConfig struct { // note under 1-Projects/ is flagged as a PARA-drift candidate. Default 180. ProjectStaleDays int `toml:"project_stale_days"` // SizeThreshold is the estimated token count (len(body)/4) above which a - // note is a candidate for the oversized detector. Default 1200. + // multi-section note is a split candidate. Default 4000, which targets + // the genuinely bloated top few percent of notes (p90 of typical vaults) + // rather than flagging the median. Tunable per vault. SizeThreshold int `toml:"size_threshold"` + // StubThreshold is the estimated token count (len(body)/4) below which a + // note is considered near-empty for the stub detector. Default 20. + StubThreshold int `toml:"stub_threshold"` // Phase 2a graph-health fields. @@ -106,7 +111,7 @@ type HealthConfig struct { // k-core coreness, orphans, leaves, and islands, so a machine-generated hub // that would otherwise dominate those metrics is invisible to the graph // detectors. Content detectors (dangling_link, ambiguous_link, para_drift, - // oversized) are unaffected: they still run over ALL notes, including excluded + // oversized, stub) are unaffected: they still run over ALL notes, including excluded // ones. Default: empty (exclude nothing). ExcludeFromGraph []string `toml:"exclude_from_graph"` } @@ -140,15 +145,28 @@ func (h HealthConfig) GetProjectStaleDays() int { return h.ProjectStaleDays } -// GetSizeThreshold returns the configured token-count threshold, defaulting to -// 1200 when the field is zero (section absent or not set). +// GetSizeThreshold returns the configured token-count threshold above which a +// multi-section note is flagged as an oversized split candidate. Defaults to +// 4000 when the field is zero (section absent or not set). That default targets +// the genuinely bloated top few percent of notes in a typical vault, rather +// than near-median notes that would make the worklist untrustworthy. func (h HealthConfig) GetSizeThreshold() int { if h.SizeThreshold <= 0 { - return 1200 + return 4000 } return h.SizeThreshold } +// GetStubThreshold returns the configured token-count threshold below which a +// note is considered near-empty for the stub detector. Defaults to 20 when the +// field is zero (section absent or not set). +func (h HealthConfig) GetStubThreshold() int { + if h.StubThreshold <= 0 { + return 20 + } + return h.StubThreshold +} + // GetConnectiveFolders returns the configured connective-folder prefixes, // defaulting to ["2-Areas", "3-Resources"] when the slice is empty. func (h HealthConfig) GetConnectiveFolders() []string { @@ -463,7 +481,14 @@ func (vc VaultConfig) Save(vaultPath string) error { buf.WriteString("# project_stale_days - days without modification before a 1-Projects/ note\n") buf.WriteString("# is flagged as PARA drift (default 180)\n") buf.WriteString("# size_threshold - estimated token count (len(body)/4) above which a\n") - buf.WriteString("# note is checked for multiple sections (default 1200)\n") + buf.WriteString("# multi-section note is a split candidate (default 4000).\n") + buf.WriteString("# Targets the genuinely bloated top few percent of\n") + buf.WriteString("# notes; tunable per vault to match your median note size.\n") + buf.WriteString("# stub_threshold - estimated token count (len(body)/4) below which a\n") + buf.WriteString("# note is considered near-empty for the stub detector\n") + buf.WriteString("# (default 20). A note is a stub candidate only when\n") + buf.WriteString("# ALL of: token count < stub_threshold, zero outbound\n") + buf.WriteString("# resolved links, and not under expected_orphan_folders.\n") buf.WriteString("# connective_folders - folder prefixes where sparse connectivity is flagged\n") buf.WriteString("# (default [\"2-Areas\", \"3-Resources\"])\n") buf.WriteString("# expected_orphan_folders - folder prefixes where sparse connectivity is normal\n") @@ -492,7 +517,7 @@ func (vc VaultConfig) Save(vaultPath string) error { buf.WriteString("# orphan, leaf, and island metrics) when ANY pattern\n") buf.WriteString("# matches ANY of those three candidates via path.Match\n") buf.WriteString("# (a malformed glob fails the run, not silently ignored).\n") - buf.WriteString("# Content detectors (dangling_link, oversized, ...) are\n") + buf.WriteString("# Content detectors (dangling_link, oversized, stub, ...) are\n") buf.WriteString("# unaffected and still run over ALL notes. Default: empty\n") buf.WriteString("# (exclude nothing). Use for machine-generated scaffolding\n") buf.WriteString("# that would otherwise dominate graph-centrality metrics.\n") diff --git a/plugin/README.md b/plugin/README.md index d040000..5af98e5 100644 --- a/plugin/README.md +++ b/plugin/README.md @@ -7,7 +7,8 @@ whatever vault you open: project via `HEBB_VAULT=${CLAUDE_PROJECT_DIR}`. Gives Claude the tools `search_vault`, `expand_context`, `get_context_for_topic`, `vault_stats`, `reindex_vault`. -- **Skills** (`skills/`): `vault-ingest`, loaded namespaced as `hebb:vault-ingest`. +- **Skills** (`skills/`): `vault-ingest`, `vault-gardener`, `ingest-inbox`, + `ingest-meetings`, each loaded namespaced (e.g. `hebb:vault-gardener`). The plugin is the agent-facing layer only. The `hebb` binary (the engine, CLI, and per-vault data: `install`/`new`/`doctor`, config, index, memory) is separate diff --git a/plugin/skills/vault-gardener/SKILL.md b/plugin/skills/vault-gardener/SKILL.md new file mode 100644 index 0000000..abfbe52 --- /dev/null +++ b/plugin/skills/vault-gardener/SKILL.md @@ -0,0 +1,118 @@ +--- +name: vault-gardener +description: Use this skill to remediate vault-health findings, acting on the worklist that `hebb audit` produces. Triggers on "garden the vault", "clean up the vault", "tidy the vault", "fix vault health", "resolve the audit findings", "work the health worklist", "fix the ambiguous links", "fix the dangling links", "archive the done projects", "deal with the stubs", or any request to act on what `hebb audit` (or the health dashboard) reports. Don't trigger for filing new incoming content (use vault-ingest), for retrieval-only questions, or for edits to a file the user is already working in. +--- + +# Vault Gardener + +Turns the `hebb audit` worklist into reviewed, reversible fixes. This skill +detects nothing itself: it reads the findings the engine already produces and, +one at a time, proposes a concrete edit, applies it only after you confirm, and +always leaves a way back. + +Each vault documents its own conventions (folder names, the archive location, +what "done" means, tag vocabulary) in its `CLAUDE.md`. **Follow that file when it +exists**; the rules below are the generic defaults. + +The hebb MCP (`mcp__hebb__*`) is the retrieval and indexing surface. Prefer it +over directory listing or grep. + +## Core rules (non-negotiable) + +- **Propose, then confirm, then apply.** Show the exact change (a diff, or a + precise before/after) and get explicit approval with `AskUserQuestion` before + writing anything. Never auto-edit. +- **One finding at a time**, or one named class in a batch the user has approved. + Never sweep the whole worklist silently. +- **Move, never delete.** "Remove" means move the note to the archive folder + (default `4-Archives/`) with a frontmatter tombstone (`archived_on`, + `archived_reason`, `prior_path`), preserving the body and backlinks so it can + be restored. The vault is git-backed, so every change is a revertible commit. + Hard `rm` is never used. +- **Preserve history.** Mark deprecations ("Status: closed", "Superseded by + [[...]]") rather than erasing them. +- **Stay in scope.** Fix the finding in front of you; do not refactor unrelated + notes along the way. + +## Workflow + +### 1. Get the worklist + +Run `hebb audit --json` to get the findings as a JSON array. Each finding has +`type`, `path`, `detail`, and `severity`. Group by `type` and tell the user the +counts. Ask which category (or specific finding) to work, or take the one they +named. For the full unresolved-link list, use `hebb audit --json --unresolved`. + +### 2. Work one finding at a time, by type + +**`ambiguous_link`** (a `[[link]]` that matches more than one note) +1. Read the source note around the link (`mcp__hebb__expand_context` or read the + file) to infer the intended target. +2. `mcp__hebb__search_vault` for the link text to see the candidate notes. +3. Propose rewriting the link to an unambiguous form: a path-qualified + `[[folder/Note]]`, or the exact title that resolves to a single note. +4. Confirm, then edit the source note. This is the highest-value, safest class. + +**`para_drift`** (a `1-Projects/` note that is done or long untouched) +1. Confirm it is genuinely finished (frontmatter `status`, or ask). +2. Propose moving it to the archive folder with a tombstone (`archived_on`, + `archived_reason`), keeping all backlinks intact. +3. Confirm, then move (write the note to the archive path; never delete). + +**`stub`** (a near-empty note that links nowhere) +1. `mcp__hebb__search_vault` / `get_context_for_topic` on its title to find a + related note it might belong in. +2. Propose either merging its content into the related note and replacing the + stub with a redirect pointer, or archiving it if it carries nothing. +3. Confirm, then apply. + +**`dangling_link` / unresolved links** (a link to a note that does not exist) +- Most are intentional links to not-yet-written notes; **do not touch them by + default.** Act only when the user asks, or when a link is an obvious typo of an + existing note (search to find the near-match), then propose the correction and + confirm. + +**`oversized`** (a large, multi-section note) — the heaviest case +1. Only if the user wants it split. Read the note and identify the H2/H3 sections + that are genuinely independent ideas. +2. Propose one atomic child note per independent section, each made + self-contained (resolve pronouns, restate the subject) with a `parent:` + backlink, and rewrite the original into a thin map-of-content that wiki-links + the children. +3. Confirm each split; apply; the original survives as the map (no content lost). +- A long but single-topic note (a meeting, a detailed design doc) is not a split + candidate. Leave it. + +### 3. Reindex and report + +After applying approved changes, run `mcp__hebb__reindex_vault` (or `hebb index`) +so the worklist reflects the fix. Close with a short chat summary: what changed, +what was archived and where, and what you left for the user to decide, as +clickable links. + +## What this skill should not do + +- Don't auto-apply anything; every write is confirmed first. +- Don't hard-delete; archive with a tombstone. +- Don't act on dangling or unresolved links wholesale; they are usually + intentional future-note links. +- Don't split a long single-topic note just because it is large. +- Don't override vault conventions; defer to the vault's `CLAUDE.md`. +- For a regulated or compliance vault, never archive or consolidate a note tagged + regulated/compliance without explicit, per-note confirmation; surface it for a + human decision instead. + +## Example + +The user says "clean up the ambiguous links." + +1. `hebb audit --json`; filter to `type == "ambiguous_link"` (say there are 12). +2. Take the first: `2-Areas/Foo.md` contains `[[Sync]]`, which matches three + notes. +3. `expand_context` on `Foo.md` shows it is about the BE VS architecture sync; + `search_vault "Sync"` lists the candidates. +4. Propose: rewrite `[[Sync]]` to `[[2-Areas/BE VS Arch & Eng Sync]]`. Show the + before/after. Confirm. +5. Apply, then move to the next finding. +6. After the batch, `reindex_vault` and report which links were disambiguated and + which were left for the user to decide.