Skip to content

Stop the compression policy from reaching the collection health plan test's seed (#6944) - #6948

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/6944-collection-health-plan-compressed-chunk
Oct 30, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/6944-collection-health-plan-compressed-chunk

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Closes #6944.

Why

The dev push at b1dbfca failed FleetCollectionHealthRecoveryLiveTests.TheFollowUpStatement_ReadsThreeRunsPerPair_ThroughAnIndex_AgainstDevPostgres (run 40987085936): one plan index was compressed_server_id__ts_meta_v2_first_col_idx, not an idx_collection_log_ index. The product statement is fine. The test let the scratch store's compression policy touch its seed.

Cause, from the code:

  • The scratch database has TimescaleDB on, and migration V23 turns collection_log into a hypertable with a compression policy (compress_after 1 day, one-day chunks).
  • The test seeds 2,000 minutes ending at 12:00 UTC yesterday, so every seeded chunk is older than compress_after and eligible.
  • Nothing in the test stopped the scheduler. The policy job runs on its own clock, so a run landing between the seed and the plan compressed one chunk. A compressed chunk is read through its own compressed_... index. That is the "1 out of 2 items" in the CI failure: the other chunks still used the watermark index.
  • The result assertions (3 runs per pair) passed in CI, so the statement returns the right rows on a compressed chunk. Only the plan shape differs, and the plan shape is not what that read is about on compressed data.
  • Settable blocking bands, so the colours agree with the alerts (#6927) #6938 does not touch any of this. It is a timing race that happened to land on that push.

What changes

Only Darling/Darling.Tests/FleetCollectionHealthRecoveryLiveTests.cs, in the one test:

  • When the scratch store has TimescaleDB, run _timescaledb_functions.stop_background_workers() right after the migrations, before seeding. This is how the other live tests handle it (about 130 use it).
  • After the seed and ANALYZE, assert zero compressed collection_log chunks, so a future compression shows as a clear failure.
  • Every existing guard is kept, including the idx_collection_log_ prefix, idx_collection_log_watermark present, idx_collection_log_time absent, no Seq Scan, and the Incremental Sort checks. Nothing was loosened.

Test plan

  • Reproduced the cause: with the fix reverted and compress_chunk forced on the seeded chunks, the test fails (Assert.Contains() Failure: Sub-string not found, the plan no longer reads the watermark index). Forcing all chunks gives a different first failing assert than CI's one-chunk case, but it is the same cause. I could not reproduce the random timing itself: the policy job did not land mid-test in any of the runs.
  • Build: 0 Warning(s), 0 Error(s).
  • FleetCollectionHealthRecoveryLiveTests with the fix, 5 runs and 4 earlier runs against a throwaway PostgreSQL cluster with shared_preload_libraries = 'timescaledb,pg_stat_statements': Total 3, Failed 0, Skipped 0 each time.
  • CollectorTrailingRuns*: no class has that name (Total 0). The only test naming CollectorTrailingRuns is in the file above.
  • Full suite not run on this laptop, CI decides.

Considered, not added

…lth plan test seeds (#6944)

The follow-up statement test seeds collection_log chunks that are all older than the one-day compress_after of the table's policy. TimescaleDB's scheduler runs that policy on its own clock, so a run landing between the seed and the plan compressed a chunk, and a compressed chunk is read through its own compressed_ index instead of idx_collection_log_watermark. The test now stops the scratch database's background workers right after the migrations, as the other live tests do, and asserts no chunk is compressed before it reads the plan. The index-name asserts are unchanged.
@erikdarlingdata
erikdarlingdata merged commit c0f61d9 into dev Oct 30, 2026
8 of 9 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/6944-collection-health-plan-compressed-chunk branch October 30, 2026 04:04
@erikdarlingdata erikdarlingdata mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant