test(dst): add deterministic simulation harness

[azimafroozeh]

What & why

This PR lands DST v1, the deterministic simulation harness designed in #507: a dev-only crate that runs engine operations in a fully seeded simulated world, plus the smallest engine seams that make that world controllable. A universe (one sampled operation stream) replays from its (scenario, seed) pair at sequential-universe scope; concurrent universes run under a measured envelope with no replay claim.

• New dev-only crate omnigraph-dst: scenario sampling, fault injection (storage faults, ack loss, bounded staleness, corruption, entropy), crash-state enumeration, five observation channels judged by 21 oracles behind typed detectors, concurrent universes, and the cost counting pass with its checked-in golden (recorded in RFC-031 §11).
• Engine footprint, additive and public: three seams (dst_ids, dst_clock, dst_gate) plus injection constructors (init_with_storage, open_read_only_with_storage); canonically ordered hash walks; fail replaced by fail-parallel with scenario serialization restored on the shim.
• Two workflows: a PR job running the deterministic suite, and a nightly whose planner gives each night a disjoint seed interval, so nights never re-search the same seeds and a re-run of one night replays exactly.
• The crate ships its own .cargo/config.toml for --cfg tokio_unstable (tokio's seeded runtime RNG); root-invoked builds stay flag-free, the upstream default shape.
• Rebasing onto current main confirmed the harness closes its loop twice: the birth-contract pin flipped from the cleanup-brick class to clean opens because the bug: after a failed init, cleanup deletes a completed store's schema files but not its data, leaving it unable to open or re-init #495 fix landed, and the cost golden re-baselined over upstream's retunes as a named per-operation diff.
• Evidence: DST v1 produced 12 findings; four are public (bug: branch merge with a net-zero delta fails permanently with "table version already exists" #473, bug: apply_schema breaks traversal queries on open handles until the store is reopened #494, bug: after a failed init, cleanup deletes a completed store's schema files but not its data, leaving it unable to open or re-init #495, bug: operations racing branch_delete fail with an unclassified "Not found" branch-ref error #515). Crash-window coverage is 50 of 66 crossed, with each of the 16 dark windows named with its unlock condition.

Local verification

• cargo test from crates/omnigraph-dst on the rebased tree: 25 lib + 42 integration + 1 torn-init green (~26 s); instruments stay #[ignore]d
• cargo clippy --workspace --all-targets in both CI shapes (default and failpoints superset, --locked) plus cargo fmt --all --check: clean
• Full workspace gate not re-run locally after the rebase: CI runs it with RUST_MIN_STACK=16777216; it was green on the previous base

Greptile Summary

The PR adds a deterministic simulation-testing crate and the engine seams, failpoint registry changes, storage injection points, fixtures, documentation, and CI workflows needed to exercise it.

• Adds seeded sequential and concurrent simulation universes with fault injection, observation channels, detectors, and checked-in cost/census evidence.
• Introduces engine clock, identity, scheduling, storage, and failpoint seams while migrating to fail-parallel.
• Adds a blocking DST workflow and sharded nightly exploration workflow; the nightly seed planner currently derives reruns from the new execution date rather than the original run date.

Confidence Score: 4/5

The nightly workflow should be fixed before merging because rerunning an older run searches a new seed interval instead of replaying the original night.

Both nightly fleets calculate their seed ownership from the current date inside the executing step, so reruns and sufficiently delayed jobs lose the stable run-to-seed mapping promised by the harness.

Files Needing Attention: .github/workflows/dst-nightly.yml

Important Files Changed

Filename	Overview
.github/workflows/dst-nightly.yml	Adds sharded deterministic and concurrent nightly fleets, but recomputes seed ownership from the rerun's execution date.
crates/omnigraph-dst/src/harness.rs	Implements the seeded sequential simulation harness, operation execution, fault reconciliation, observations, and final reports.
crates/omnigraph-dst/src/concurrent.rs	Adds concurrent writer, reader, maintenance, branch, fault, and scheduling universes with history-based judgment.
crates/omnigraph/src/db/omnigraph.rs	Exposes injected-storage initialization and read-only opening while threading DST seams through graph construction.
crates/omnigraph/src/failpoints.rs	Replaces the process-global fail API with a shared fail-parallel registry and scoped configuration wrappers.
crates/omnigraph/src/lance_access.rs	Exposes the process-wide Lance object-store registry needed by the simulation storage shim.
Cargo.toml	Adds the DST workspace crate, supporting dependencies and Tokio test utilities, and migrates failpoint dependencies.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Seed[Scenario and seed] --> Universe[DST universe]
    Universe --> Seams[Engine seams]
    Seams --> IDs[Seeded IDs]
    Seams --> Clock[Virtual clock]
    Seams --> Gate[Storage-call gate]
    Seams --> Storage[Injected storage and faults]
    Universe --> Ops[Engine operations]
    Ops --> Observe[Query, physical, history, claim, and time observations]
    Observe --> Detectors[Typed detectors and oracles]
    Detectors --> Report[Universe report and replay bundle]
    Nightly[Nightly planner] --> Seed

Loading

_{Reviews (1): Last reviewed commit: "imp" | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used (5)

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — OmniGraph write path
• Knowledge Base — Query execution engine
• Knowledge Base — Dev conventions and invariants
• Knowledge Base — Branching and merge

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[greptile-apps]

Execution date breaks nightly replay

When an older workflow run is rerun on a later UTC date, this step recalculates DST_FLEET_SEED_BASE from the new wall clock, causing the deterministic fleet to search a different interval instead of replaying the original night. The concurrent fleet repeats the same calculation at lines 97–106, and a scheduled job delayed across midnight is likewise assigned to the wrong nightly interval.

Knowledge Base Used: Dev conventions and invariants