Skip to content

fix: fail ingestion when vector persistence fails (#78) - #81

Open
spelech wants to merge 2 commits into
mainfrom
hotfix/issue-78-vector-write-failure
Open

spelech wants to merge 2 commits into
mainfrom
hotfix/issue-78-vector-write-failure

Conversation

@spelech

@spelech spelech commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Fixes #78.

Changes

  • Propagate vector initialization/deletion/upsert failures instead of returning success and positive indexed chunk counts.
  • Reuse existing disk backup/rollback. Invalidate previous relational index metadata before destructive vector replacement so a restored file can be retried rather than falsely considered indexed.
  • Attempt cleanup of partially persisted batches and document the retry policy; no distributed-transaction guarantee.
  • Remove old vectors on a successful empty replacement.

Verification

  • New real SQLite/Qdrant/filesystem/HTTP regressions: 7 passed after reproducing 5 expected failures.
  • Generated requirement inventory: 2 passed; release/link-integrity gate passed.
  • Combined backend regression run: 70 passed, 1 skipped.
  • With the companion PostgreSQL fixes: 2 authenticated full-profile checks passed. A real PostgreSQL trigger rejected vector inserts; uploads returned HTTP 500, original PDF bytes were restored, stale index/cache records were absent and retry succeeded after removing the fault.

Broader storage checks had 21 passes and one unchanged cached-runtime failure: tests/test_local_storage_indexing.py::test_incremental_indexing_code_file expects AST symbols but finds zero. The same assertion was reproduced with unmodified application source from base 8491cc4. The full suite and a fresh source build are not claimed green.

Steve Pelech and others added 2 commits October 28, 2026 10:08
Propagate vector store initialization, delete, and upsert failures through the existing disk rollback. Invalidate index metadata before destructive vector changes and clean up incomplete upserts so failed replacements remain retryable. Preserve successful zero-chunk replacements without stale vectors.

Refs #78.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(storage): fail ingestion when vector persistence fails

1 participant