fix(kb): wait for indexing, and stop re-ingesting while it runs

[DerrickF]

A 1.5 MB PDF uploaded to a promoted knowledge base sat at uploading indefinitely, with a fully retrievable copy already in the knowledge base. Three bugs stacked.

1. The poll window was sized against the wrong measurement

The consumer ingested, then polled a retrieval for 30 s. Its own header justifies that with "Bedrock reports INDEXED up to a second before it can be retrieved — measured at 0.75–1.03 s". But the poll starts the instant the ingest call returns, so it actually has to cover:

ingest returns ──► Bedrock parses + indexes ──► INDEXED ──► retrievable
                   37–264 s per §5.1                       the ~1 s the 30 s
                   (5 m 30 s for this file)                 was sized for

The budget was smaller than the documented lower bound for a PDF.

Same shape as §5.30, where verify failed good migrations against that identical 0.75–1.03 s figure. That fix never reached this component, which inherited the constant.

2. Every redelivery re-ingested, restarting the work it was waiting for

IngestKnowledgeBaseDocuments is fire-and-forget and nothing asked Bedrock what it already knew, so the consumer could not distinguish "not indexed yet" from "never submitted". All three deliveries re-submitted. The document reached INDEXED 54 seconds after the final attempt was dead-lettered.

handle_object now probes GetKnowledgeBaseDocuments first and branches on the real DocumentStatus enum (read from the packaged service model, not guessed):

status	behaviour
STARTING / PENDING / IN_PROGRESS	do not re-ingest; wait
FAILED / METADATA_UPDATE_FAILED	terminal — mark failed, no retry
PARTIALLY_INDEXED / METADATA_PARTIALLY_INDEXED	usable; it IS retrievable, so failing it would hide content the user can see
NOT_FOUND (incl. probe failure)	no evidence of prior work → ingest

3. indexedAt was fabricated

indexed_at = _now_iso() ran immediately after the ingest call returned, recording when we asked and presenting it as when indexing finished — minutes apart here. Now Bedrock's own updatedAt.

The pre-existing test asserted only that the key existed and was truthy, which a fabricated value satisfies perfectly. That is why it survived. It also means an earlier claim of mine — that a measured 0.9 s gap "confirmed the 0.75–1.03 s figure" — was wrong: that gap was ingest-return→retrievable, not INDEXED→retrievable.

Why the wait is in-invocation, not more retries

I first raised a RetryPolicy on the EventBridge target. It does nothing, and the construct now says so instead of carrying it: for a Lambda target EventBridge hands the event off, and from there the function's own async retry config governs — retryAttempts: 2, which is Lambda's hard maximum and exactly the 1 + 2 attempts in the logs (verified against the live get-function-event-invoke-config).

Redelivery therefore spans only a few minutes and cannot be extended. So INDEXED_POLL_TIMEOUT_SECONDS = 600: covers the measured tail, leaves 5 minutes under the 15-minute Lambda timeout. Small documents still finish in one invocation — the fast path is unchanged.

Verification

8 new tests, plus a cross-language test that parses the Lambda timeout out of the CDK construct and asserts the poll budget fits inside it with headroom. Those two values live in different languages with no compiler between them, and a wait longer than the timeout is a killed invocation — the exact bug being fixed.

_FakeBackend now models document status, not just ingest calls. A fake that reported instant success is what let a 30 s window look adequate for work that takes minutes — the same failure as §5.28, where the fake modelled clientToken dedup as permanent.

Mutations verified caught, each by a correctly-named test:

mutation	caught by
in-flight guard removed	test_a_document_already_being_indexed_is_not_re_ingested
indexedAt back to local clock	test_indexed_at_is_bedrocks_timestamp_not_our_clock
FAILED treated as retryable	test_a_failed_document_is_marked_failed_and_not_retried
budget raised past the Lambda timeout	test_the_poll_budget_leaves_headroom_under_the_lambda_timeout
budget dropped back to 25 s	test_the_budget_covers_the_measured_indexing_tail

Backend 2,510 passed across the affected areas; infra 640 passed.

Why this surfaced now

Before #900 the legacy pipeline also wrote document status and would have marked this complete regardless, masking it. Single ownership is correct — it just means latent failures in the owner are visible instead of hidden.

Note

DOC-87323a4fa3ef in dev is still parked at uploading. Its content is fine (Bedrock reports INDEXED; 10 chunks retrieved), but nothing will reconcile the record — there is no reprocess endpoint yet (task 14.4). It needs a one-off repair or a re-upload.