From d2a1f6ca7fa14a12578b10c2563a11fdb99fb199 Mon Sep 17 00:00:00 2001 From: Hakandede Date: Fri, 23 Oct 2026 01:14:42 +0300 Subject: [PATCH 1/2] Show index-build progress while the run waits on the {cd,_id} index The run builds a missing {cd:1,_id:1} index itself during mapping, but the dashboard showed nothing during that wait (hours to days on large sources): the run progress bar stays hidden until chunks exist, and index progress was only polled after clicking "Build missing indexes". - Track the collection mapping is blocked on (with its position in the map pass) around ensureIndex itself; an in-progress build lists as present but still blocks, so the hasRequiredIndex check is the wrong trigger. - Parse $currentOp: prefer the IndexBuildsCoordinator op (it carries progress) over the waiting client op, keep only {cd:1,_id:1} builds on the source db, and report the step, because the percentage restarts per step: scanning documents, sorting index keys (the builder reports "scanning collection" without counters for 30-45% of the build after the scan ends), writing index keys, applying writes made during the build. - Sharded clusters: through mongos every shard reports its own builder with per-shard counters; the slowest shard is reported with the number of shards still building. Secondaries omit command.$db; matched through ns. - Run $currentOp with readConcern local. The source client defaults to majority, which $currentOp refuses; the error was swallowed, so server-side progress never showed under the default config. Failures are now returned as serverOpsError and shown in the UI. - The Build missing indexes button joins builds already in flight instead of reporting "all indexed" while one is running. - Overview tab: a dedicated index bar (collection N of M, step N of 4, percentage, wait time), polled with the 5s slow tick so it covers builds the run starts itself and builds started by a dry run. Tests: - Unit (run in the CI unit job): the parser against real $currentOp recordings from MongoDB 8.0 (replica-set primary and secondary, 2-shard cluster via mongos), including whole-build replays; orchestrator index logic against a fake MongoDB. - Integration: the mapping-time state under the default readConcern majority, the listed-but-building case, and the button path. --- .github/workflows/ci.yml | 4 + src/http/ledger-viz-route.ts | 63 ++- src/runtime/chunk-orchestrator.ts | 90 +++-- src/runtime/ledger-engine.ts | 9 +- src/source/index-build-ops.ts | 127 +++++++ .../integration/index-build-progress.test.ts | 226 +++++++++++ .../unit/fixtures/currentop-index-build.json | 358 ++++++++++++++++++ .../streams/currentop-m8-primary.json | 1 + .../streams/currentop-m8-secondary.json | 1 + .../streams/currentop-m8-sharded-mongos.json | 1 + tests/unit/index-build-ops.test.ts | 232 ++++++++++++ .../unit/orchestrator-index-progress.test.ts | 213 +++++++++++ 12 files changed, 1291 insertions(+), 34 deletions(-) create mode 100644 src/source/index-build-ops.ts create mode 100644 tests/integration/index-build-progress.test.ts create mode 100644 tests/unit/fixtures/currentop-index-build.json create mode 100644 tests/unit/fixtures/streams/currentop-m8-primary.json create mode 100644 tests/unit/fixtures/streams/currentop-m8-secondary.json create mode 100644 tests/unit/fixtures/streams/currentop-m8-sharded-mongos.json create mode 100644 tests/unit/index-build-ops.test.ts create mode 100644 tests/unit/orchestrator-index-progress.test.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c783b15..6189b93 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -35,6 +35,10 @@ jobs: - name: Ingestion-matching differential harness run: npx vitest run tests/differential + # Pure unit tests (e.g. $currentOp index-build parsing from captured ops). + - name: Unit tests + run: npx vitest run tests/unit + integration: name: Integration (MongoDB 7 + ClickHouse 26.8) runs-on: ubuntu-latest diff --git a/src/http/ledger-viz-route.ts b/src/http/ledger-viz-route.ts index ee9d693..f6c7e5b 100644 --- a/src/http/ledger-viz-route.ts +++ b/src/http/ledger-viz-route.ts @@ -281,6 +281,13 @@ const PAGE = `
Failed chunks–
ETA–
+