Skip to content

Fix duplicate ingestion jobs starving newly enabled sources - #105

Merged
ralyodio merged 1 commit into
mainfrom
fix/ingest-queue-dedup
Sep 14, 2026
Merged

ralyodio merged 1 commit into
mainfrom
fix/ingest-queue-dedup

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

The production ingestion queue accumulated over 74,000 waiting jobs because each scheduler tick created a new minute-based job ID for sources that were already queued. Those sources also occupied the first page of the due-source query, preventing 38 newly enabled California police feeds from reaching a worker.

Use BullMQ deduplication per source for scheduled and manual runs, lasting through waiting and execution. Page past sources already represented in Redis so later overdue sources can be admitted, with at most 50 new requests per tick. Unique job IDs retain completed history without blocking future runs.

Adds an explicit queue repair CLI: audit by default; --apply backs up pending metadata and creates or locates a deduplicated replacement before removing legacy requests. It preserves force flags and leaves active jobs, unrelated job types, source configuration and ingested records untouched. Apply after deployment to consolidate the existing backlog.

Validation: 1,641 local tests passed, including four real-Redis integration tests for concurrent deduplication, rescheduling after success/failure, pagination past queued sources and safe queue repair; one local PostgreSQL migration test skipped. Eight discovery checks passed, lint passed with existing warnings. CI now provides Redis for these integration tests. Separately, initial production reads succeeded for all 39 California police sources, loading 230 announcements.

@ralyodio
ralyodio merged commit a2f50c8 into main Sep 14, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant