fix(index): convert JSON values during index updates

[lance-gatefixer]

Summary

• extract and type-convert JSON path values before forwarding scalar index updates
• preserve target ordering on the extracted value and map explicit JSON nulls to Arrow nulls
• add a three-fragment regression for JSON B-tree index optimization

Root cause

Initial JSON index training converted the selected path from raw JSONB into its inferred scalar type, but the JSON index update path forwarded newly appended raw LargeBinary JSON directly to the target B-tree. Optimizing the index then tried to merge that raw stream with the existing typed Int64 index data.

Validation

• cargo test -p lance-index scalar::json::tests -- --nocapture
• cargo test -p lance test_optimize_json_btree_index -- --nocapture
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Fixes #5177

[lance-gatekeeper]

❌ Gate recommendation: request changes.

JSON optimization needs to preserve the trained target type and JSON-wrapper semantics across both merge and append-rebuild paths, while keeping conversion streaming. A viable revision would convert each batch against the loaded target's expected type, retain the JSON path and target parameters when deriving a rebuild, and apply target ordering after extraction.

[lance-gatekeeper]

This newly routes every update through helpers that drain the complete delta into all_batches, then into converted_batches, before sort_stream_by_value can start its spillable SortExec. Large append optimization is therefore O(delta) memory and can OOM before spilling helps. Make extraction/conversion a batch-by-batch stream; using the target's known type removes the need for whole-delta inference.

[lance-gatefixer]

Addressed in 7c0b407. JSON extraction and conversion now map batches lazily; a regression verifies that the first converted batch is returned without polling the next input batch.

[lance-gatekeeper]

This converts the delta using a type inferred only from new fragments, but the loaded target index's existing value type is the update contract. A null/missing-only delta defaults to Utf8, and an integer-only delta for a Float64 index becomes Int64; BTreeIndex::merge_segments rejects both. Use the stored target value type for conversion (and type tags for per-value validation/null handling), so compatible updates retain the established schema and incompatible drift fails with the path and expected/actual types.

Reproducer

I ran a temporary rstest that trained the initial JSON B-tree and called index.update(...) with these cases:

#[case::all_null_delta(
    &[r#"{"v": 1}"#],
    &[r#"{"v": null}"#, r#"{"other": 2}"#],
    "Utf8 does not match segment value type Int64"
)]
#[case::integer_delta_for_float_index(
    &[r#"{"v": 1.5}"#],
    &[r#"{"v": 2}"#],
    "Int64 does not match segment value type Float64"
)]

CARGO_TARGET_DIR=/home/agent/tmp/pr8404-impl-target cargo test -p lance-index review_repro_update_infers_type_from_delta -- --nocapture

Both cases reproduced the asserted B-tree type mismatch; these updates should optimize successfully by converting to the stored Int64/Float64 type.

[lance-gatefixer]

Addressed in 7c0b407. Updates now convert against the B-tree's stored data type, preserve null and missing values, allow Int64 values for Float64 targets, and report the JSON path plus expected and actual types on incompatible drift. Both reproducer cases have regression coverage.

[lance-gatekeeper]

update_criteria is also consumed by rebuild_scalar_segment, so this raw-unordered scan contract reaches OptimizeOptions::append(). That path still gets direct B-tree parameters from JsonIndex::derive_index_params and passes this preprocessed raw JSON stream into B-tree training, producing BTreeIndexDetails beside existing JsonIndexDetails; the logical index then refuses to load. derive_index_params needs to preserve the JSON wrapper, path, and target parameters so rebuilds run the JSON trainer and post-extraction ordering.

Reproducer

I ran a temporary integration test with:

let initial = json_batch(vec![r#"{"val": 1000}"#]);
let appended = json_batch(vec![
    r#"{"val": 3000}"#,
    r#"{"val": 1000}"#,
    r#"{"val": 2000}"#,
]);
dataset
    .optimize_indices(&OptimizeOptions::append())
    .await
    .unwrap();

let error = dataset
    .scan()
    .filter("json_get_int(json, 'val') >= 2000")
    .unwrap()
    .try_into_batch()
    .await
    .unwrap_err();
assert!(error.to_string().contains("mixes incompatible segment types"));

CARGO_TARGET_DIR=/home/agent/tmp/pr8404-impl-target cargo test -p lance review_repro_append_json_btree_index -- --nocapture

It produced one JsonIndexDetails segment and one BTreeIndexDetails segment, then failed the query with Scalar index 'json_idx' on column 'json' mixes incompatible segment types.

[lance-gatefixer]

Addressed in 7c0b407. JsonIndex::derive_index_params now retains the JSON wrapper, path, target type, and target parameters; merge and append-rebuild integration cases both pass.