Skip to content

test(assignor): fix flaky PyPy CI leg (disable Hypothesis deadline)#732

Closed
wbarnha wants to merge 1 commit into
masterfrom
claude/fix-pypy-hypothesis-deadline
Closed

test(assignor): fix flaky PyPy CI leg (disable Hypothesis deadline)#732
wbarnha wants to merge 1 commit into
masterfrom
claude/fix-pypy-hypothesis-deadline

Conversation

@wbarnha

@wbarnha wbarnha commented Jul 21, 2026

Copy link
Copy Markdown
Member

Problem

Master CI intermittently goes red on the Python pypy3.11 leg. The failure is:

hypothesis.errors.FlakyFailure: … produces unreliable results:
  Falsified on the first call but did not on a subsequent one
hypothesis.errors.DeadlineExceeded: Test took 5103.32ms, which exceeds the
  deadline of 4000.00ms …

in tests/meticulous/assignor/test_copartitioned_assignor.py::test_remove_clients (and its siblings).

Root cause

Those copartitioned-assignor property tests now run on PyPy (the PyPy skipif guards were removed when full PyPy support landed). They carried a fixed 4000 ms Hypothesis deadline. PyPy's JIT warmup makes a single example's timing vary by more than a second, so an example intermittently exceeds 4000 ms on its first run but not on retry — which Hypothesis reports as DeadlineExceededFlakyFailure. The assignment produced is perfectly valid; only the timing is nondeterministic. This flakes the (non-required) PyPy leg and turns the master run red.

Fix

Set TEST_DEADLINE = None, disabling the Hypothesis deadline for these four property tests — exactly what the DeadlineExceeded message recommends. These tests assert assignment correctness (validity, stickiness, termination), not performance, so the deadline adds only flakiness. The property assertions are unchanged, and the CI job's own timeout still guards against a genuine hang.

Verification

pytest tests/meticulous/assignor/test_copartitioned_assignor.py4 passed (run takes ~5.7 s, confirming examples legitimately approach/exceed the old 4 s deadline). flake8/black clean.

🤖 Generated with Claude Code


Generated by Claude Code

The copartitioned-assignor property tests run on PyPy since the PyPy
skips were removed. They carried a fixed 4000ms Hypothesis deadline, but
PyPy's JIT warmup makes a single example's timing vary by over a second,
so runs intermittently exceed it and Hypothesis reports DeadlineExceeded
/ FlakyFailure -- failing the (non-required) PyPy leg and turning master
CI red even though the assignment produced is valid.

These tests assert assignment correctness, not performance, so set
deadline=None (as the DeadlineExceeded message itself recommends). The
property assertions are unchanged and the job timeout still guards
against a real hang.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.11%. Comparing base (5bf3405) to head (c506022).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #732   +/-   ##
=======================================
  Coverage   95.11%   95.11%           
=======================================
  Files         104      104           
  Lines       11169    11169           
  Branches     1206     1206           
=======================================
  Hits        10623    10623           
  Misses        452      452           
  Partials       94       94           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wbarnha
wbarnha enabled auto-merge July 21, 2026 17:12
@wbarnha
wbarnha disabled auto-merge July 21, 2026 17:12
@wbarnha wbarnha closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant