Skip to content

Ordered loader retains earlier requests after a newer authoritative read succeeds #2296

Open

Description

@KyleAMathews

The Release workflow for the first RC failed the ordered-loader request-authority oracle on merged release commit 71bf598a3d13d37db28ac112431fd8be790957fe (#2217).

Executed CI evidence

  • Failed Test (db-2) job
  • Owner: packages/db/tests/query/ordered-source-loader-state-oracle.test.ts
  • Test: holds, retires, and recovers requests under generated settlement orders (random campaign)
  • Property: ordered-loader.request-authority
  • Seed: -845274422
  • Path: 2109:1:3:6:4:8:10:13:15
  • Failed after 2,110 generated tests, shrunk eight times.
  • Assertion checkpoint: checkAuthorityCase, line 1632, after the history drain and any explicit retry.
no leak: requests [3,4] are still held after authoritative request 5 succeeded
{"indexed":false,"limit":1,"settledFirst":true,"releaseFault":false,"syncRepair":true,"requiresFullSource":false,"joined":true,"snapshotRows":1,"needsMore":true,"retryTimer":false,"releaseTruncates":true,"offset":0,"releaseReenters":false,"releaseDisposes":false,"history":[{"type":"settle","pick":0,"ok":true},{"type":"settle","pick":0,"ok":true},{"type":"report","react":false},{"type":"move","delta":-1},{"type":"joinedBusy"},{"type":"settleNewest"}]}

Accepted law and limits of this report

The owner's “No leak” law states that once loading is quiet and no failure blocks it, the loader holds nothing issued before its last successful authoritative read. Retained demands can be read again on later refetches. The oracle uses the real loader with a controlled subscription; this mismatch establishes a failing request-ledger comparison, not incorrect public query rows or a measured production resource leak. Diagnose production retirement versus the oracle's model, replay ownership and observation checkpoint before changing either one.

This exact trace has not yet been replayed independently. The observed assertion is from the executed CI run; it is not a timeout or a reach failure.

Replay

Run from packages/db on the recorded commit:

TANSTACK_DB_ORACLE_PROPERTY=ordered-loader.request-authority \
TANSTACK_DB_ORACLE_SEED=-845274422 \
TANSTACK_DB_ORACLE_PATH=2109:1:3:6:4:8:10:13:15 \
pnpm exec vitest run --coverage.enabled=false --typecheck.enabled=false \
  tests/query/ordered-source-loader-state-oracle.test.ts -t 'random campaign'

Keep this history in the shared comparison and exercise nearby synchronous repair, release-triggered truncate and joined-work histories before a production repair. Show RED on the original code, GREEN after repair and a hostile mutant that fails. A real Collection/provider receiving witness is needed before claiming the public resource consequence is closed.

Related loader retirement investigations: #2223 (failed request across cursor reset) and #2191 (request authority/retirement). This trace has only successful explicit settlements; identity with either earlier report is not established. It is separate from the publication-retention report #2294 and multi-tab Electric report #2281.

The maintainer explicitly approved recording this counterexample as a follow-up and retrying the first RC release while leaving the defect open. No test, comparison or run budget has been weakened. A passing retry does not close this issue.

Activity

  1. KyleAMathews commented on Nov 3, 2026

    @KyleAMathews
    CollaboratorAuthor

    A second request-authority counterexample surfaced in CI for #2346. It reaches the same no-leak comparison through a different history from the original report. Whether both traces have the same root cause remains unresolved.

    Executed evidence

    • Failed database shard on 5911d5e00af0d947b36ac739f73016a6129d38cf.
    • Owner: packages/db/tests/query/ordered-source-loader-state-oracle.test.ts.
    • Property: ordered-loader.request-authority.
    • Random campaign failed after 2,541 histories and shrank eight times.
    • Seed: -416164704.
    • Shrink path: 2540:0:3:3:8:8:12:12:13.
    • Direct replay reproduces the same assertion on the PR and on an untouched Git archive of main at 1443dc92e5417394acbdc98b782caa9802848ce6, using the same installed dependencies. Both reach the comparison in checkAuthorityCase at line 2135.
    no leak: requests [2,3] are still held after authoritative request 4 succeeded
    
    {"indexed":false,"limit":1,"settledFirst":true,"releaseFault":false,"syncRepair":true,"requiresFullSource":false,"joined":true,"snapshotRows":1,"needsMore":true,"retryTimer":false,"releaseTruncates":false,"offset":0,"releaseReenters":true,"releaseDisposes":false,"releaseInvalidates":false,"history":[{"type":"truncate"},{"type":"settle","pick":0,"ok":true},{"type":"invalidateThenFailOldest"},{"type":"move","delta":-1},{"type":"joinedBusy"},{"type":"settle","pick":0,"ok":true}]}

    Run from packages/db on the recorded main commit:

    TANSTACK_DB_ORACLE_SEED=-416164704 \
    TANSTACK_DB_ORACLE_PATH=2540:0:3:3:8:8:12:12:13 \
    TANSTACK_DB_ORACLE_PROPERTY=ordered-loader.request-authority \
    pnpm exec vitest run tests/query/ordered-source-loader-state-oracle.test.ts \
      -t 'holds, retires, and recovers requests under generated settlement orders' \
      --coverage.enabled=false --typecheck.enabled=false --maxWorkers=1

    Scope and next experiment

    This confirms a baseline oracle failure, not incorrect public rows or a measured production resource leak. Determine whether the last authoritative read covers the held requests while joined work defers a reentrant pass, and whether production retirement or the oracle's coverage premise/checkpoint needs repair. Then exercise the corresponding real-subscription receiving path before claiming a public resource consequence is fixed. Preserve this history in the shared comparison and extend nearby generated histories; require RED/GREEN and a hostile mutant for a repair.

    The full trace and owner are also preserved in the coverage map and the review record.

    The maintainer directed recording this failure separately and merging #2346. Its corrected head acdfb6f14a8aebd83824ea88c717c821da3ce685 has 25 successful CI checks and one skipped CodeRabbit check. No loader test, comparison, seed, or run budget was changed to bypass this failure. A later green random campaign does not close either trace in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions