[Toldry]

Before: since about 00:20 UTC on 2026-10-17 the bot scanned 0 comments. The heartbeat showed reads: 0 requests, a lag of about -6 × 10^18 IDs, and about 10^12 "IDs skipped" every 10 minutes. The r/all listing had returned one comment ID of about 6 × 10^18 (real IDs are about 8 × 10^10). The reader skipped ahead to it, never saw a higher head again, and stopped reading.

After: such an ID is ignored. If a bogus position ever gets in anyway, the reader resets itself within a minute. The heartbeat now warns when heads were found but nothing was read. Deploying this also recovers the stuck production reader, because a saved cursor ahead of the newest comment is ignored at startup.

How:

• fetch_head drops listing IDs more than MAX_BACKLOG_IDS from the median (newest_consistent_id).
• poll_head rejects a head further past the last accepted head (or the saved cursor) than MAX_BACKLOG_IDS + 2,000 IDs/s × elapsed. It logs this once per streak and counts it as rejected in the heartbeat.
• After 6 polls in a row where the head is far below the reader's position, the reader restarts at the newest comment.
• If heads were found but no reads were made, the heartbeat logs a WARNING saying the reader is stalled.

Tests:

• Unit tests:
  • an outlier ID or a malformed ID in the listing
  • a rejected bogus jump
  • a real jump after time has passed
  • a replay of the incident with bogus heads mixed in
  • a head going slightly backwards
  • the self-reset
  • a restart with a bogus first head
  • a restart with a cursor ahead of the head
  • reader threads that keep reading while heads are bogus
  • the heartbeat stall warning and the rejected count
• Live tests:
  • the head is a real comment written in the last hour
  • the real CommentReader runs for 45 s and must read at least 200 comments with a sane lag. This is the check that would have caught this outage before the deploy.

Locally: uv run pytest gives 1804 passed. The live tests run only on master in CI.

---

Generated by Claude Code