Skip to content

fix(core): preserve configured blob payloads - #232

Open
dcfocus wants to merge 1 commit into
mainfrom
codex/issue-222-blob-columns
Open

fix(core): preserve configured blob payloads#232
dcfocus wants to merge 1 commit into
mainfrom
codex/issue-222-blob-columns

Conversation

@dcfocus

@dcfocus dcfocus commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep configured context payload columns inline and mark them with neutral schema metadata for lazy default projections
  • recover blob-column configuration on reopen and reject legacy blob-v2 schemas with an actionable error
  • preserve lazy payloads during partial updates and materialize text when lexical retrieval needs it
  • add regressions for binary round trips, WAL merge, reopen, updates, text retrieval, and legacy-schema rejection

Compatibility

Datasets already created with lance-encoding:blob cannot be recovered through MemWAL today: reads return no payload and WAL merge fails. This change fails fast when opening those schemas rather than continuing with silent data loss. New datasets store payloads inline and exclude configured blob columns from default bulk projections.

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace --all-targets
  • cd python && uv run --extra tests pytest (213 passed, 6 skipped, 3 xfailed)
  • cd python && uv run ruff format --check .
  • cd python && uv run ruff check .
  • cd python && uv run --extra dev pyright

Closes #222

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

blob_columns (blob-v2 offload) silently loses data and permanently breaks WAL merge

1 participant