Skip to content

fix(background_tasks): warm the invalidation publish connection from the listener - #882

Merged
antosubash merged 3 commits into
mainfrom
fix/issue-880-invalidation-cold-publish
Nov 7, 2026
Merged

antosubash merged 3 commits into
mainfrom
fix/issue-880-invalidation-cold-publish

Conversation

@antosubash

Copy link
Copy Markdown
Owner

Summary

Closes #880.

RedisInvalidationTransport.publish runs under a 1 s asyncio.timeout, while the request that made the write waits for its response. The client opened its command connection only on first use. So the first publish after boot, or after a Redis outage, also paid the TCP connect inside that second. Under load the connect was cancelled (seen in CI, #865) and the invalidation was lost; other workers kept the stale value until its TTL.

  • After every successful SUBSCRIBE, at boot and after each reconnect, the listener now sends one PING on the command client (_warm_publish_connection). That opens the pooled connection the next publish reuses.
  • The PING is bounded by SUBSCRIBE_TIMEOUT_SECONDS. A failure is logged at DEBUG and never restarts the listener. start() still never waits on Redis.
  • Unchanged: the 1 s budget, the bus's log-and-continue handling of a failed publish, and settings.

Design: docs/superpowers/specs/2026-11-07-invalidation-warm-publish-design.md. Rejected alternatives:

  • a separate connect budget (it doubles how long a request can be held);
  • a background sender queue with retry (it changes the transport's failure contract and loses queued messages at shutdown).

Verification

  • New test_background_tasks_invalidation_warmup.py:
    • fake client: one ping after subscribe, another after a reconnect, none before a subscribe succeeds;
    • fake client: a ping that fails or hangs leaves the listener delivering;
    • real Redis: CLIENT LIST shows the warmed connection, and a publish runs on that same connection without opening a new one. Connections are found by a per-test client_name, because CI workers share one Redis.
    • Without the production change, the three warm-up tests fail.
  • make lint green.
  • make test-py: 8,204 passed, 28 skipped, with the Redis tests running.
  • make test-js: 900 passed.
  • Reviews:

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

…the listener

The client's command connection opened lazily, so the first publish after boot
or after an outage paid the connect inside its 1 s request-path budget and lost
the message under load. The listener now PINGs after every successful
SUBSCRIBE; a failed warm-up is logged at DEBUG and never restarts it.

Closes #880

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Deploying simple-module-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: d9e745f
Status: ✅  Deploy successful!
Preview URL: https://d01f862f.simple-module-python.pages.dev
Branch Preview URL: https://fix-issue-880-invalidation-c.simple-module-python.pages.dev

View logs

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-11-07T14:08:31.828689Z d9e745f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@antosubash
antosubash merged commit 8f3b9c9 into main Nov 7, 2026
17 checks passed
@antosubash
antosubash deleted the fix/issue-880-invalidation-cold-publish branch November 7, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

background_tasks: an invalidation published on a cold Redis client can be lost to the 1 s publish timeout

1 participant