Skip to content

[ML] Make CMonotonicTimeTest robust to sleep overshoot on CI - #3201

Merged
edsavage merged 4 commits into
elastic:mainfrom
edsavage:fix/monotonic-time-flaky-test
Sep 21, 2026
Merged

edsavage merged 4 commits into
elastic:mainfrom
edsavage:fix/monotonic-time-flaky-test

Conversation

@edsavage

@edsavage edsavage commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

CMonotonicTimeTest::testMilliseconds / testNanoseconds slept for one second and asserted the monotonic timer advanced by a value inside a fixed [900, 1200]ms window. That effectively tested std::this_thread::sleep_for accuracy rather than the timer itself. On oversubscribed CI machines (notably the macOS aarch64 Orka VMs) the sleep can overshoot well past the 1200ms upper bound, causing intermittent failures.

An example failure: ml-cpp-snapshot-builds #7092 (test_macos-aarch64-RelWithDebInfo, branch 9.5):

During 1 second the monotonic millisecond timer advanced by 1293 milliseconds
*** 1 failure is detected in the test module "Master Test Suite"

The timer was behaving correctly; it faithfully reported the ~1293ms that actually elapsed while the thread was descheduled.

Fix

The tests still sleep for a nominal one second, but replace the fragile fixed [900, 1200]ms window with two complementary checks:

  1. Cross-domain lower bound (diff > 900ms / 900000000ns): sleep_for guarantees it sleeps for at least the requested duration, so a reading well below one second means the monotonic clock is running too slowly or has stalled. This is the only genuinely independent check that the clock ticks at roughly real-time rate, because it compares against a different clock domain (the scheduler's sleep timer). A 10% margin absorbs coarse-timer granularity and cross-clock skew.

  2. Cross-clock consistency check (agreement with std::chrono::steady_clock over the same interval, within 5%): validates CMonotonicTime's own unit-scaling arithmetic. This bounds diff from above, but only against the actual elapsed time measured over the same window.

There is deliberately no upper bound relative to the nominal sleep duration. sleep_for only promises a minimum sleep and can overshoot arbitrarily on loaded/oversubscribed hosts - that's a scheduler property, not a timer defect - which is exactly what made the old diff < 1200 assertion flaky. The rationale is documented in a block comment in the test.

Test plan

  • Built ml_test_core locally (macOS aarch64, RelWithDebInfo).
  • Ran CMonotonicTimeTest: both cases pass; the two clocks agree to <1ms (ms case) and <1us (ns case).
  • Compiled the test with -Wall -Wextra: zero warnings (fixed a latent -Wsign-compare).

The millisecond and nanosecond timer tests slept for one second and then
asserted the monotonic timer had advanced by a value within a fixed
[900, 1200]ms window. This really tested the accuracy of sleep_for rather
than the timer: on oversubscribed CI machines (e.g. the macOS Orka VMs)
the sleep can overshoot substantially, which intermittently pushed the
measured interval past the 1200ms upper bound and failed the build.

Measure the elapsed interval independently with std::chrono::steady_clock
around the same sleep and assert the monotonic timer agrees with that
reference to within 5%. This validates what we actually care about - that
the timer tracks real elapsed time - and is immune to how long the machine
actually slept for.

Co-authored-by: Cursor <cursoragent@cursor.com>
@elasticsearchmachine

Copy link
Copy Markdown

Pinging @elastic/ml-core (Team:ML)

@edsavage edsavage added auto-backport Automatically merge backport PRs when CI passes v8.19.23 v9.4.8 v9.5.5 labels Sep 20, 2026
edsavage and others added 2 commits September 21, 2026 12:02
Co-authored-by: Cursor <cursoragent@cursor.com>
Combine two complementary checks in CMonotonicTimeTest: a cross-domain
lower bound (the timer must advance by at least ~the requested sleep, our
only independent check that the clock ticks at real-time rate) and the
steady_clock agreement check (which validates CMonotonicTime's unit-scaling
arithmetic). Document why there is deliberately no upper bound relative to
the nominal sleep duration - sleep_for can overshoot arbitrarily on loaded
CI hosts, which is a scheduler property, not a timer defect.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Address the remaining timing assertions, measurement skew, documentation, and formatting issues.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Updates CMonotonicTimeTest to tolerate sleep overshoot by comparing elapsed time with steady_clock.

Changes:

  • Adds millisecond and nanosecond reference-clock measurements.
  • Uses a 5% agreement tolerance.
  • Updates test rationale and diagnostics.
File Summary
lib/​core/​unittest/​CMonotonicTimeTest.cc Updates monotonic timer tests, assertions, and logging.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lib/core/unittest/CMonotonicTimeTest.cc
Comment thread lib/core/unittest/CMonotonicTimeTest.cc Outdated
@valeriy42
valeriy42 self-requested a review September 21, 2026 06:46
…here)

Reword the check-2 comment: steady_clock and CMonotonicTime agree because
they observe the same elapsed real time, but they do not share the same
hardware counter on every platform - the coarse millisecond paths read a
different source (Linux CLOCK_MONOTONIC_COARSE, Windows GetTickCount64).
Describe it as a cross-clock consistency check that validates our scaling
arithmetic rather than claiming identical-counter coverage.

Co-authored-by: Cursor <cursoragent@cursor.com>
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

💚 All backports created successfully

Status Branch Result
9.4
9.5
8.19

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

elastic-vault-github-plugin-prod Bot added a commit that referenced this pull request Sep 21, 2026
…3205)

The millisecond and nanosecond timer tests slept for one second and then
asserted the monotonic timer had advanced by a value within a fixed
[900, 1200]ms window. This really tested the accuracy of sleep_for rather
than the timer: on oversubscribed CI machines (e.g. the macOS Orka VMs)
the sleep can overshoot substantially, which intermittently pushed the
measured interval past the 1200ms upper bound and failed the build.

Measure the elapsed interval independently with std::chrono::steady_clock
around the same sleep and assert the monotonic timer agrees with that
reference to within 5%. This validates what we actually care about - that
the timer tracks real elapsed time - and is immune to how long the machine
actually slept for.

(cherry picked from commit d6a62eb)

Co-authored-by: Ed Savage <ed.savage@elastic.co>
elastic-vault-github-plugin-prod Bot added a commit that referenced this pull request Sep 21, 2026
…3206)

The millisecond and nanosecond timer tests slept for one second and then
asserted the monotonic timer had advanced by a value within a fixed
[900, 1200]ms window. This really tested the accuracy of sleep_for rather
than the timer: on oversubscribed CI machines (e.g. the macOS Orka VMs)
the sleep can overshoot substantially, which intermittently pushed the
measured interval past the 1200ms upper bound and failed the build.

Measure the elapsed interval independently with std::chrono::steady_clock
around the same sleep and assert the monotonic timer agrees with that
reference to within 5%. This validates what we actually care about - that
the timer tracks real elapsed time - and is immune to how long the machine
actually slept for.

(cherry picked from commit d6a62eb)

Co-authored-by: Ed Savage <ed.savage@elastic.co>
elastic-vault-github-plugin-prod Bot added a commit that referenced this pull request Sep 22, 2026
…3207)

The millisecond and nanosecond timer tests slept for one second and then
asserted the monotonic timer had advanced by a value within a fixed
[900, 1200]ms window. This really tested the accuracy of sleep_for rather
than the timer: on oversubscribed CI machines (e.g. the macOS Orka VMs)
the sleep can overshoot substantially, which intermittently pushed the
measured interval past the 1200ms upper bound and failed the build.

Measure the elapsed interval independently with std::chrono::steady_clock
around the same sleep and assert the monotonic timer agrees with that
reference to within 5%. This validates what we actually care about - that
the timer tracks real elapsed time - and is immune to how long the machine
actually slept for.

(cherry picked from commit d6a62eb)

Co-authored-by: Ed Savage <ed.savage@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically merge backport PRs when CI passes :ml >test v8.19.23 v9.4.8 v9.5.5 v9.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants