fix(master): unwedge /experiments and drain compaction backlogs faster

[beinan]

Four production failures in the master, all reinforcing each other during a backlog. Observed live: 2/3 master replicas stuck on the stats lock, /experiments timing out, ~100k fragments accumulated (15k on a single experiment), and one experiment at 98+ pending WAL generations.

1. /experiments hung behind the stats scan

state.stats is an exclusive Mutex — every StatsStore read method takes &mut self, so reads cannot even run concurrently with each other. list_experiments used it as its only data source via a bare lock().await, with no timeout, no fallback, and no TimeoutLayer on the router. Any slow scan round wedged the endpoint outright.

And the round was slow by construction. scan_once_inner took the lock and then called retire_cold_experiments, a serial loop that for each cold experiment:

1. opens the store,
2. runs cleanup_own_shard — RETIRE_TIMEOUT, 600s,
3. waits on the process-wide compaction semaphore,
4. compacts — another 600s,
5. observes.

That is up to N_cold × ~20min of object-store IO inside the mutex that the UI depends on — and a semaphore wait nested inside a mutex hold, so an unrelated slow compaction elsewhere in the process extended it further. With stats_cold_retire_secs defaulting to 7 days, this path is live in any deployment more than a week old.

Two independent fixes:

• Retirement moved out of the critical section. It needs nothing from the lock — it reads a local snapshot and returns a set of names. It now runs before the lock is taken, leaving the critical section as the single commit it was meant to be.
• The handler no longer blocks. It try_locks, and on contention serves an in-memory snapshot published after each round, flagging the response stale. A list at most one scan interval old beats an unbounded hang, and the degradation is visible rather than silent. The search fallback — which takes the registry write lock and opens datasets on demand — is skipped while stale, since doing the most expensive work precisely when the process is busy is what turned contention into a second stall.

2. The sweeps ignored backlog size

list_above pushed limit into the scan before sorting, then sorted by name. Which rows came back was therefore scan order, effectively arbitrary — a 15k-fragment experiment had no better chance of being swept than a 17-fragment one, and the per-sweep cap could be spent entirely on nearly-clean experiments while the worst offenders sat untouched for many sweeps.

Now ordered by the threshold column descending (fragment_count for compaction, pending_wal_generations for merge), so each sweep spends its budget worst-first. The predicate is still pushed into the scan — that was the point of the original change — only the arbitrary pre-sort truncation is gone.

3. WAL merge starved compaction

MergeWal is an HTTP fan-out to every worker endpoint: slow in wall-clock terms, nearly free in local resources. It shared the single task_concurrency pool with compaction, so a backlog of slow fan-outs could occupy every slot — exactly when both backlogs are growing and both need to drain.

It now draws from its own budget: MERGE_WAL_CONCURRENCY, default 4. 0 restores the previous shared-pool behavior.

4. task_concurrency was documented as if it were global

It is per-process, so N replicas run up to N * task_concurrency tasks between them. Correctness never depended on the value — etcd claims and per-target locks already prevent two replicas touching one experiment — but the old wording invited operators to tune it as a cluster-wide bound. Doc-only.

Note for operators

This PR does not change any default. The fragment backlog is primarily paced by COMPACTION_MAX_SOURCE_FRAGMENTS (32) against COMPACTION_INTERVAL_SECS (600): with Compact deduped per target, one experiment drains at ~32 fragments per 10 minutes, i.e. ~3 days for 15k. Raising those is the operational lever; this PR makes sure the sweep budget is spent on the right experiments and that the UI survives while it drains.

Verification

• cargo build --workspace, cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo clippy --manifest-path crates/lance-context/Cargo.toml --all-targets -- -D warnings all clean.
• cargo test --workspace --all-targets — all green (217 core + 41 master + rest).
• cargo test -p lance-context-master --lib -- --ignored against a local etcd 3.7.0 — 20/20 pass.
• New tests: the stale-fallback path, and three covering worst-first sweep ordering, threshold preservation, and per-column ranking.
• The /experiments test was confirmed non-vacuous: it holds the stats lock for the duration of the call, reproducing the production wedge, and fails by timeout when reverted to the previous blocking lock().await.
• One existing assertion in threshold_query_returns_only_matching_rows encoded the old ordering and was updated; its stated intent ("exactly the rows over the threshold") is membership, which is unchanged. list — the UI pagination path — keeps its name ordering and is untouched.

🤖 Generated with Claude Code