Repository navigation
feat: read every new comment by ID range, resume after restarts - #98
Merged
Merged
Conversation
The r/all comment listing returns at most 100 comments per request, one request at a time, so the bot saw about 850 comments a minute by day and missed most of r/all. Comment IDs are numbered in order site-wide, so the bot now fetches blocks of 100 consecutive IDs with /api/info on STREAM_READERS threads spread over the 4 accounts, keeping only what r/all would show. STREAM_SOURCE=listing keeps the old stream as a fallback. The reader saves its position (stream_cursors) every 30 s and at shutdown, so restarts and redeploys resume instead of skipping comments. The listing stream now resumes after the newest comment seen after an error. The heartbeat also reports requests, comments per request, seconds per request, how far behind the newest comment the bot is, and CPU use (process share of a core and VM steal time). Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYGhPFNh3ZvVEr2YXt8Qmr
Toldry
marked this pull request as ready for review
October 16, 2026 20:03
…CPU figures GCP's Always Free tier limits instance hours, disk and egress, not CPU, so the heartbeat's CPU and steal figures are removed. scripts/verify_comment_ids.py checks against live Reddit that comment IDs follow creation time, that ID blocks come back mostly full with the fields the r/all filter reads, how fast IDs are handed out, and what each request costs. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYGhPFNh3ZvVEr2YXt8Qmr
# Conflicts: # AGENTS.md
The live check showed /api/info comments carry no over_18 or quarantine field, so the reader's NSFW filter did nothing. Phase 1 now drops matches on NSFW or quarantined posts, reading the submission it already fetches. The r/all listing sometimes returns nothing for limit=1, which would have stopped the head poll; it now asks for 25 and takes the newest. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYGhPFNh3ZvVEr2YXt8Qmr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Querent · project thread
Before: the bot read r/all through PRAW's comment listing stream, 100 comments per request and one request at a time, about 7 s each by day. On 2026-10-15/16 it saw about 850 comments a minute by day and up to 2,500 at night, when Reddit answered faster, so it missed most of r/all. Every restart (one per push to master) and every stream error skipped whatever was posted in between.
After: the bot reads every new comment by ID. Reddit numbers comments in base 36, in order across the whole site. The main thread polls the newest r/all comment ID every 10 s and queues the blocks of 100 IDs up to it.
STREAM_READERSthreads (default 4, taking turns between the 4 accounts) each fetch a block with one/api/inforequest and pass on the comments r/all would show (not NSFW, quarantined, private or on a user profile). Requests per minute follow Reddit's comment rate, about one per 100 new comments.The reader saves its position (
stream_cursors) every 30 s and at shutdown, so a restart resumes where it stopped, up to 200,000 IDs back.STREAM_SOURCE=listingbrings back the old stream, which now resumes after the newest comment seen when it hits an error.The heartbeat also says how comments were read, for example
reads: 412 requests, 87.3 comments/request, 0.95 s/request | head: 50 polls, 1.70 s/poll | behind: 9 s, 340 IDs.How:
comment_reader.py(CommentReader), andmain.run_streampicks the source.StreamMetricsgains the reading figures.stream_cursorstable withsave_stream_cursor/get_stream_cursor. I ran the schema script twice against PostgreSQL 16 to check it is idempotent.Phase1Workers.submitis now locked, because several readers call it.scripts/verify_comment_ids.py, a read-only live check of the reader's assumptions to run on the VM before merging.Not verified here, because this environment can't reach Reddit:
/api/infocarryover_18,quarantineandsubreddit_type.scripts/verify_comment_ids.pychecks both. So does the new live testtest_live_comment_reader_reads_recent_comments_by_id, which runs in the post-mergeintegration-testsjob and blocks the deploy if the assumption is wrong.Known trade-offs, added to AGENTS.md section 9:
🤖 Generated with Claude Code
https://claude.ai/code/session_01FYGhPFNh3ZvVEr2YXt8Qmr