fix: avoid deadlock when opening scan fragments

[lance-gatefixer]

Summary

• run legacy fragment-open futures as independent Tokio tasks before buffering them
• propagate task join failures through the scan stream
• add a regression test for progress while the outer future is not polled

Root cause

Legacy scans passed fragment-open futures directly to try_buffered. On GCS over HTTP/1, connection-pool dependencies can require another request to progress before an open completes, while the buffered stream poll order prevents that progress.

Spawning each open decouples its polling from the ordered and unordered legacy scan streams. Current-format v2 scans already use independently spawned fragment tasks.

Validation

• cargo test -p lance --lib io::exec::scan::tests
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Fixes #1835

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Independent fragment-open progress must remain bounded, cancellation-safe, and cover every reachable legacy scan path. Use datafusion::common::runtime::SpawnedTask as the shared mechanism across regular and pushdown legacy scans, with regression coverage through the real buffered-open pipeline.

[lance-gatekeeper]

Using a bare JoinHandle here violates DataFusion’s ExecutionPlan cancellation contract: dropping the scan drops this mapped handle, but Tokio detaches the child, so up to fragment_readahead network opens per cancelled scan keep running and can accumulate. datafusion::common::runtime::SpawnedTask is already used in filtered_read.rs; it preserves independent progress while aborting on drop, and the explicit JoinError mapping can remain.

Reproducer

At 99353d49a2194dde2a4956e61a2768f123e99b8d, I ran an ownership-equivalent case with:

CARGO_TARGET_DIR=/home/agent/tmp/gate8397-cancel-target cargo run --offline --quiet

The test waited for the child to start, dropped the returned mapped handle, released the child, and asserted that its completion channel had been cancelled. It exited 101 with:

spawned task completed after its outer future was dropped

The regression assertion was:

started_rx.await.unwrap();
drop(outer);
release_tx.send(()).unwrap();
let completion = tokio::time::timeout(
    std::time::Duration::from_secs(1),
    completed_rx,
)
.await
.unwrap();
assert!(
    completion.is_err(),
    "spawned task completed after its outer future was dropped"
);

[lance-gatefixer]

Addressed in 7c683f4. Replaced the detached Tokio handle with shared SpawnedTask ownership, so dropping a scan aborts in-flight opens while preserving join-error propagation.

[lance-gatekeeper]

This applies independent polling only to LanceStream::try_new_v1. Under the default scanner settings, an ordinary refined filter on a legacy dataset is routed to LancePushdownScanExec, where FragmentScanner::open still goes directly through .buffered(fragment_readahead). That is the same fragment-open scheduling boundary this change identifies as deadlock-prone, so the fix does not cover filtered legacy scans. Apply the shared cancellation-safe task wrapper to that path too while preserving its ordering and readahead bound.

[lance-gatefixer]

Addressed in 7c683f4. Routed both regular and pushdown legacy fragment opens through the shared cancellation-safe, ordered, fragment_readahead-bounded helper.

[lance-gatekeeper]

This proves only that Tokio starts a spawned task without polling its handle. It neither fails on the base revision nor drives either buffered fragment-open pipeline, so it still passes if both new call sites are removed and cannot guard #1835. Add a deterministic dependency regression through the regular and filtered legacy scan paths, including cancellation and bounded-readahead assertions.

[lance-gatefixer]

Addressed in 7c683f4. Added a parameterized real-pipeline regression for regular and pushdown legacy scans covering independent progress, the readahead bound, and cancellation.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The follow-up resolves the prior concerns at the shared scheduling boundary: regular and pushdown legacy fragment opens now use DataFusion’s abort-on-drop task wrapper while retaining ordered buffering and the configured readahead bound. The real-pipeline regression repeatedly verified independent progress, bounded concurrency, and cancellation for both paths.