blob_columns (blob-v2 offload) silently loses data and permanently breaks WAL merge

[beinan]

Summary

ContextStore's blob_columns option tags a column with lance-encoding:blob (blob-v2 offload). On this storage path that does not work, and the failure is silent on write: bytes go in, nothing comes out, and the store's WAL can never be merged again.

Nothing else uses blob-v2 — RolloutStore, DatagenStore and GenericStore all store binary payloads inline — so this only affects a ContextStore explicitly opened with blob_columns set. It is off by default, which is why it went unnoticed.

Evidence

Writing a 4-byte payload and reading it back through every path, with and without the option:

blob_columns=false   get_blob=Some(4)  list=Some(4)  direct_scan=Some(4)
blob_columns=true    get_blob=None     list=None     direct_scan=None
                     MERGE FAILED: Schema error: WAL generation column
                     'binary_payload' has type Struct("position": UInt64,
                     "size": UInt64), expected LargeBinary

Two distinct failures:

1. Every read returns None. get_blob, list, and even a direct dataset.scan() all come back empty.

2. WAL merge fails permanently. This is the worse one. Blob-v2 stores the column physically as Struct{position, size} — a pointer into a separate blob file — while the base table's schema declares LargeBinary. align_batch_to_schema rejects the mismatch, so every merge attempt for that store fails. Flushed generations accumulate under _mem_wal/ without bound, read amplification grows without bound, and the error repeats on every sweeper tick.

Why blob-v2 cannot work here

Reads go exclusively through the MemWAL LSM scanner (LsmScanner), which has no blob-materialization step. A blob-v2 column reads back as None through it by construction. This was already known and documented in rollout_store.rs, which is why rollout stores artifact bytes inline:

binary_payload holds artifact bytes as a plain inline LargeBinary column, not a blob-v2 offloaded column. Rollout reads go exclusively through the MemWAL LSM scanner, which has no blob-materialization step: a blob-v2 (lance-encoding:blob) column reads back as None through it [...] Inline storage is therefore the only encoding that round-trips.

ContextStore is the only store still using blob-v2, and its read path is entirely lsm_scanner()-based. The two facts were never checked against each other.

Why the tests did not catch it

The three existing tests (test_blob_binary_payload, test_blob_text_payload, test_blob_both_columns) write a record and then assert on a freshly constructed schema object:

store.add(std::slice::from_ref(&record)).await.unwrap();

// Verify schema has blob metadata on binary_payload
let schema = ContextStore::schema(&store.blob_columns);
let field = schema.field_with_name("binary_payload").unwrap();
assert_eq!(field.metadata().get("lance-encoding:blob"), Some(&"true".to_string()));

That asserts ContextStore::schema() sets the metadata it was asked to set. It never touches the store the record was written to, and never reads a byte back. The tests pass whether or not the data is retrievable.

Fix

Change blob_columns from offload to projection exclusion, matching what RolloutStore and GenericStore already do:

• store the column inline as LargeBinary (no lance-encoding:blob metadata)
• exclude it from default scan projections, so list/search never materialize it
• fetch it per row on demand (get_blob, or get with explicit columns)

The observable contract for callers is unchanged — bulk reads still do not pay for large payloads — but the bytes are actually retrievable. Column-oriented storage means an unprojected column is never read off object storage in the first place, so the cost model is the same.

Measured on this path, inline payloads are not a problem at the sizes this is meant for: 5 MB writes in 80 ms / reads in 42 ms, 40 MB in 483/262 ms, 120 MB in 1.39 s/760 ms, all surviving a WAL merge.

Data already written with blob_columns enabled is not recoverable through this change — but it is not readable today either, so nothing that currently works is lost.

Blob-v2 can be reconsidered if LsmScanner gains a blob-materialization step upstream.

Scope

• Introduced in Add configurable V1 blob encoding for large payload columns #49
• Affects only ContextStore opened with blob_columns non-empty (default is empty)
• RolloutStore, DatagenStore, GenericStore unaffected — all inline already