Report the index shortfall the reindex audit was structurally unable to see

[jpr5]

The audit could not report an index that had shrunk

reindex-audit.ts Check 3 only ever fired on db_has_more. The db_has_fewer
direction was in the finding type and was never implemented:

// Check 3 — Count divergence (db_has_more only; db_has_fewer is expected
// when the indexer filters low-semantic-value files like SVGs, base64, etc.)
if (dbCount > diskCount) { ... }

The stated reason is real — the indexer legitimately drops files with no
extractable prose — but "some shortfall is normal" got implemented as "all
shortfall is invisible."

Motivation: three bugs the check watched and never mentioned

1. Wrong crawl scope. The CopilotKit API reference (184 .mdx) lived in a
   sibling directory outside the configured walk root; 183 of 184 pages were
   live on the site and absent from the index.
2. A wedged item. The code source sat in a false error from 2026-09-01
   to 2026-09-11: one file produced a 36 KB chunk and a non-retryable 400, and
   the orchestrator refused to advance past it.
3. Successful extraction of nothing. 130 of 682 prose .mdx files are pure
   JSX stubs whose text lives in the excluded src/content/snippets/**. They
   are walked, matched, read, stripped to empty, and chunk to zero.
   markdown.ts returns [], pipeline.ts writes nothing — both silently. A
   19% shortfall that stood for months.

Honest scope note: this change catches #2 and #3. It does not catch #1 —
when the walk root itself is wrong, the disk side of the comparison is wrong
too, so disk and index agree. #1 needs a config-level scope check, not this one.

What changed

Check 3 reports both directions. The shortfall is measured on the set
difference (disk minus index), not on diskCount - dbCount. The raw delta is
a lossy proxy: a source holding one stale row and missing one real file has
matching counts and reports nothing — the same blindness in miniature. The set
difference fires in every case the count comparison would, plus that one, and
it yields the actual paths, so the finding names the files instead of
saying "the count is off by 130."

Keeping the signal from becoming noise. The finding fires only above a
per-source baseline: unindexed_tolerance, a new optional file-source key,
defaulting to 5% of walked files, with an absolute floor of 3 files so
a small source does not alert on one or two legitimately-empty ones. The floor
is bypassed when a source explicitly sets 0.

What a healthy source looks like after this change: it emits nothing.
A source dropping a handful of empty or content-free files per reindex stays
silent at the default. A source that has been cleaned up sets
unindexed_tolerance: 0 and then hears about the very first regression — that
is the intended end state, and it is why the lever is per-source rather than a
global mute. The failure mode this design is guarding against is an alarm that
fires on every source on every reindex and gets muted within a week, which is
how the direction came to be suppressed outright in the first place.

unindexed_tolerance is deliberately not in the source fingerprint
(source-fingerprint.ts uses an explicit allowlist): it changes reporting, not
what gets indexed, so changing it must not force a reindex.

Dedup key now includes the direction. A source can produce both directions
in one run; source:check alone let one overwrite the other and suppress its
Slack alert.

The two silent drop sites are now audible. Every return [] in
chunkMarkdown warns with the file and a distinct reason — empty file / MDX
stripping left no prose / every split trimmed to empty — because those have
completely different fixes. (There was a third silent path at the bottom of the
function; it got the same treatment.) pipeline.indexItem logs that the index
actually lost the item, prefixed by source and id, greppable next to
next_acquire_reason and quarantined_items:

[chunker] no chunks for docs/quickstart.mdx: MDX stripping left no prose (the file is JSX/imports only — its text may live in a snippet or component that is not indexed); it will not be indexed
[pipeline:docs] docs/quickstart.mdx produced zero chunks; nothing indexed (any previously indexed chunks for it are being cleared)

Should a zero-chunk file be its own audit finding?

No — and it does not need to be, because the shortfall finding already names
it individually. A zero-chunk file is written as replaceChunksForFile(src, id, []), which deletes the row, so it is by construction a disk file absent
from the index and it lands in the shortfall finding's samples by path. A
separate persisted zero-chunk finding would need the indexer to write a second
record of the same fact into a second store, which can go stale against the
chunks table and would report a file as dropped after someone fixed it. The
audit's two sources of truth are disk and the index; keeping it that way means
the finding cannot lie. The per-file reason — which the audit genuinely
cannot know — is what the new chunker warnings supply, at the one place that
does know.

Red-green

RED — 40 .mdx pages, 8 of them pure-JSX stubs, run through the real
chunker and the real pipeline (only the DB and the disk walk are faked), so
the test exercises the actual drop path:

stdout | reports a db_has_fewer finding when pure-JSX stubs chunk to zero
RED PROOF findings = []

× reports a db_has_fewer finding when pure-JSX stubs chunk to zero
  AssertionError: expected undefined to be defined

Preconditions in the same run passed: disk 40 files, index 32. The shrink was
real and the audit said nothing.

GREEN — same test, unchanged:

stderr | [reindex-audit] docs — count_divergence: 8 issues (db_has_fewer): page-0.mdx, page-5.mdx, page-10.mdx, page-15.mdx, page-20.mdx
stdout | GREEN_PROOF [{"source":"docs","check":"count_divergence","count":8,
          "samples":["page-0.mdx","page-5.mdx","page-10.mdx","page-15.mdx","page-20.mdx",
                     "page-25.mdx","page-30.mdx","page-35.mdx"],"direction":"db_has_fewer"}]
✓ reports a db_has_fewer finding when pure-JSX stubs chunk to zero

Negative assertion — a source skipping a normal share (1 of 40, 2.5%)
produces findings === [], and the same shortfall does report once the
source sets unindexed_tolerance: 0:

✓ does NOT report a shortfall for a source skipping a normal share of files
✓ reports that same small shortfall once the source sets unindexed_tolerance: 0

Mutation-tested so the negative assertion is not vacuous — replacing
unindexed.length > budget with > 0 fails it:

× does NOT report a shortfall for a source skipping a normal share of files
  Tests  1 failed | 2 passed (3)

The chunker and pipeline warnings were red-greened the same way (2 chunker
tests and 1 pipeline test failed before the warnings existed), each paired with
a "does not warn on a normal file" assertion.

Local gate

npm run build ✅ · npx tsc --noEmit ✅ · npx tsc --noEmit -p tsconfig.scripts.json ✅ ·
node scripts/check-test-shapes.mjs ✅ · npx prettier --check ✅ ·
npm test ✅ 189 passed | 1 skipped (190 files), 3712 passed | 1 skipped. No pre-existing failures.

What an operator does when this fires

Open the named files. Three outcomes, in order of likelihood:

1. The files are stubs or otherwise have no indexable prose — real but
   legitimate. Either fix the extraction (inline the snippet, widen the source)
   or, if the drop is genuinely correct, raise that source's
   unindexed_tolerance past the measured baseline and leave a note saying why.
2. The files should have indexed and did not — grep the reindex logs for
   [chunker] no chunks for <path> and produced zero chunks; the reason is
   there.
3. No chunker warning for them at all — the source never got that far.
   Check /health for a wedged source and a held state token (Make a reindex actually pick up a change to a source's crawl configuration #160, Unwedge the code source: a URL in a template literal, an 8192-token chunk, and an unbounded retry #161).