Fix duplicate ingestion jobs starving newly enabled sources - #105
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The production ingestion queue accumulated over 74,000 waiting jobs because each scheduler tick created a new minute-based job ID for sources that were already queued. Those sources also occupied the first page of the due-source query, preventing 38 newly enabled California police feeds from reaching a worker.
Use BullMQ deduplication per source for scheduled and manual runs, lasting through waiting and execution. Page past sources already represented in Redis so later overdue sources can be admitted, with at most 50 new requests per tick. Unique job IDs retain completed history without blocking future runs.
Adds an explicit queue repair CLI: audit by default;
--applybacks up pending metadata and creates or locates a deduplicated replacement before removing legacy requests. It preserves force flags and leaves active jobs, unrelated job types, source configuration and ingested records untouched. Apply after deployment to consolidate the existing backlog.Validation: 1,641 local tests passed, including four real-Redis integration tests for concurrent deduplication, rescheduling after success/failure, pagination past queued sources and safe queue repair; one local PostgreSQL migration test skipped. Eight discovery checks passed, lint passed with existing warnings. CI now provides Redis for these integration tests. Separately, initial production reads succeeded for all 39 California police sources, loading 230 announcements.