docs(kb): record findings 31-36, retract the document_id false alarm, add two diagnostics

[DerrickF]

Documentation only — HANDOFF.md for the managed-KB migration. No code.

Retracts the top open risk

An earlier revision listed a "known unknown": document_id and relevance coming back empty from the facade after promotion, with the inference that the status filter was either not running on the managed path or losing the join key. Both inferences were wrong. Measured against the live dev knowledge base:

Layer	Result
Raw Retrieve	location.customDocumentLocation.id = "DOC-ae5cc5434f2d"
_to_chunk	document_id='DOC-ae5cc5434f2d', relevance=0.4226 / 0.1616
Facade	metadata.document_id correct, distance = −0.4226 / −0.1616

The probe read result["document_id"] and result["relevance"] at the top level. Those keys have never existed — git log -L on the formatted_results.append block shows the same four keys (text, distance, metadata, key) back to the first commit of that function. Reading outside the contract returns None on both backends, so the observation said nothing about the managed path. The filter is live and did join: it matched the document and kept both chunks because that DOC# record reads complete.

Records defect 31 and generalizes it

The missing bedrock:StartIngestionJob grant, fixed in #898's fdf15d21. Promoted to a §3 constraint because it is the third instance of one pattern: a Bedrock grant listing exactly the API the code calls, deploying clean, failing on first real use, because AWS checks an adjacent action name (TagResource, ListTagsForResource, now StartIngestionJob).

Also records why it survived a migration that had "completed end to end": run-kb-migration.py runs under an SSO identity broader than every Lambda role, so it is structurally unable to find this class of defect.

Two new findings, both still open

• 32. Routing exclusivity is enforced only on the consumer's side. The legacy handler has no engine gate in either copy and its S3 notification is still live, so a document added to a promoted knowledge base is indexed twice. Visible in real data on DOC-dc8b65658e29 (chunkCount: 8, vectorStoreId: assistants-index). Not a wrong-answer bug — retrieval reads managed only. Every exclusivity test covers the side that already works.
• 33. _filter_vectors_by_document_status opens with if not doc_ids: return vectors — one fail-open path in a function that fails closed everywhere else.

Housekeeping

• Corrects two stale claims: the worker image is deployed (feat(kb): ship the real kb-migration Lambda image #886), and the tree has no uncommitted work.
• Adds a verified "what works today" table, including that a born-managed knowledge base is not implemented (newDefault has zero readers in backend/src).
• Jest count 611 → 634.
• Fixes a duplicate defect number 24 by renumbering the run to 25–33. All §5.x cross-references are internal to this file and were updated with it; each now resolves to exactly one item.

---

Second commit: four more findings, and the tooling that found them

Records findings 32–36. All but one are fixed in #900; §5.33 is the only one from 2026-08-31 still open.

They share a root cause worth stating once: the two engines were never made exclusive. The consumer stands down for a legacy document; nothing made the legacy pipeline stand down for a managed one, and nothing propagated a deletion to the managed engine at all. Double-indexing (32), a status field with two owners (34), and an orphaned managed corpus (36) are three costumes on one cause.

Item 34 is the one to read. Two writers owned status, so "ready" was decided by whichever finished last. Observed both ways within an hour — a PDF marked complete 65 s before the managed KB could answer, and an image-only PDF marked failed while managed served it correctly, which only came out right because the consumer finished second. The generalisable lesson: any field two components can write needs a stated owner. Nobody chose this race; a writer was added beside an old one and the question was never asked.

Item 35 is not a defect but a measured capability difference: Bedrock's image extraction works. A pure-diagram curriculum flowchart is permanently failed on legacy (do_ocr=False) and retrievable in 94.5 s on managed, with the vision model's own description in the chunks. Also records that the old bundled aws CLI omits mediaExtractionConfiguration from get-data-source — a field can look unset when it isn't.

Item 36 documents the deliberate asymmetry between the two gates, which is the thing most likely to be "cleaned up" later: on ingest an unreadable KB record resolves to legacy; on delete it must fail. Same question, opposite answers, because the DOC# row is what the fail-closed status filter joins against.

Corrects one claim in item 32: an earlier revision said the gate was missing from "both copies" of the handler. The bootstrap copy is a 33-line no-op placeholder that indexes nothing.

Two read-only diagnostics, documented in §2

script	what it answers
kb-doc-timings.py	per-document ingestion timing and which engine did the work, derived from the record rather than guessed. Legacy 30 s vs managed 95 s on the same 132 KB PDF; INDEXED → retrievable 0.9 s, independently confirming the evaluation's 0.75–1.03 s
kb-compare-engines.py	one query through both engines, exploiting the retain window where both indexes still hold the corpus

kb-compare-engines.py marks each chunk against MAX_CONTEXT_CHARS, which is what makes the comparison honest: ~2,000 characters reach the model, so one or two chunks, so precision@1 is nearly the whole game. On CS434, legacy's five chunks sit within 0.056 of each other and its top chunk does not contain the string; managed separates by 0.4988 with the literal match first.

Noted in §2: #900 ends that both-indexes trick. Once the legacy pipeline stands down, new uploads land in the managed index only, so a future A/B needs two assistants or documents predating promotion.

Also adds an engine visibility item to the work queue — nothing logs which engine served a query, so "is the new one working?" is currently only answerable from the KB record. Worth closing before a wide rollout, given this feature's risk profile is silent regressions.