ci: run the whole etcd suite, measure coverage, unskip remote tests

[beinan]

CI had infrastructure it was not using and blind spots it did not report. Four changes, one of which uncovered a real bug.

1. Run all 19 etcd-backed tests (was: 1)

rust-test.yml downloaded and started etcd 3.7.0, then ran exactly one test by name with --ignored --exact. The other 18 — the scheduler.rs, routes.rs, and state.rs etcd paths — never ran. Switching to cargo test -p lance-context-master --lib -- --ignored is nearly free since etcd is already up.

Turning them on surfaced 4 deterministic failures in tests that had never once executed:

scan_reconciles_removed_experiments — real production bug. StatsStore::write_snapshot early-returns on empty rows to stop a round of total observe failures from blanking the table and silently starving the compaction and WAL-merge sweeps (they read this table to decide what to enqueue). But the guard tested rows.is_empty() rather than why it was empty, conflating that hazard with "every experiment was legitimately deregistered." Impact beyond the test: a deployment draining to zero registered experiments serves stale rows indefinitely, and those rows keep feeding sweeps that open datasets whose registry entries are gone. Masked at N>1, which is why only a single-experiment test surfaces it. Fixed by splitting out replace_snapshot (honors empty); the scanner disambiguates at the call site using the failure count it already has. write_snapshot still refuses empty, so the original hazard stays defended.

startup_requeues_interrupted_local_tasks — stale test, production correct. Dropping a TaskClaim stops lease renewal but does not revoke the lease, so etcd holds the claim key for the rest of the TTL and recover_orphaned correctly declines to requeue — the task genuinely is still running. Added abandon_claim_for_test to revoke the lease, reaching the same key state a real crash reaches once the TTL elapses, without a 5s sleep. Assertions kept verbatim; mutation-tested (stubbing recover_orphaned to Ok(0) fails the test) to confirm it is not vacuous.

The two routes.rs endpoint tests — stale, predate the LSM write path. add() only appends to that handles in-memory memtable; rows are invisible to the separate handle the endpoint opens until flush()` seals them into a committed generation. Added the flush, all assertions intact. Not a polling situation — the seal is caller-triggered, so a timeout loop would just spin.

2. Measure coverage

Nothing measured it. New coverage job collects the normal run and the etcd-backed run into one lcov report via cargo llvm-cov --no-report + report, uploads to Codecov. codecov.yml is informational only — no gate can fail a build until we have a baseline.

3. Stop skipping the remote tests

test_rollout_remote.py is the only coverage of the remote/HTTP client path — lance-context-client has zero tests of its own — and all 4 tests silently skipped because CI installs a prebuilt wheel and never built the server binary. Now builds it, and a missing binary is a hard failure under CI (still a graceful skip locally) so this cannot regress to silent skipping.

4. Schedule the harness

test/harness/ asserts the ?source=fragments|wal|all selector semantics that nothing else exercises, and no workflow referenced it. Added a nightly workflow_dispatch-able job with log dumping on failure and if: always() teardown.

Verification

• 19/19 etcd tests pass, parallel and --test-threads=1, repeated runs
• Full cargo test --workspace --all-targets green (214 core, 62 server, 33 master, 18 api, ...)
• cargo fmt --check and cargo clippy --workspace --all-targets clean
• All 4 remote tests confirmed passing (not skipping) against a locally built binary; CI guard verified in both directions
• All workflow YAML validated

Note: python-test.yml timeout raised 30 → 45 min and the root workspace added to its rust-cache list, since that job now builds the server binary.

🤖 Generated with Claude Code