Skip to content

fix: stop opening a SQLite connection per raw job in discovery - #16

Open
gurmehakk20 wants to merge 1 commit into
Aditya-00a:mainfrom
gurmehakk20:fix/discovery-n-plus-one-sqlite-connections
Open

gurmehakk20 wants to merge 1 commit into
Aditya-00a:mainfrom
gurmehakk20:fix/discovery-n-plus-one-sqlite-connections

Conversation

@gurmehakk20

Copy link
Copy Markdown

Summary

  • run_job_scout() called count_company_active_jobs() and has_applied_to_title() once per raw job scanned instead of once per unique company/title, and each call opened its own fresh SQLite connection — ~30,000 throwaway connections per discovery cycle at real scale (~14,800 raw jobs/cycle).
  • Replaced both with bulk_company_active_counts() and bulk_applied_titles(), each a single batched query; the loop now does in-memory lookups. Same semantics, same log output.
  • Confirmed present on main (not a local-only issue) before fixing.

Root cause

See agent/backend/db/jobs_repository.py (old count_company_active_jobs / has_applied_to_title) and the loop in agent/backend/services/job_scout.py that called them per raw job. A clean local benchmark of just one of the two functions at real scale took 14.55s; the batched version takes 0.001s. Under a live process with Windows/filesystem overhead this compounded into multi-minute stalls per discovery cycle, observed directly while running the agent.

Test plan

  • Added tests/test_jobs_repository_bulk_lookups.py: correctness vs. old per-item semantics, a regression guard asserting exactly 2 SQLite connections regardless of catalog size, and a scale smoke test
  • Full suite: pytest -q → 55 passed
  • ruff check clean on changed/new files

@vercel

vercel Bot commented Oct 22, 2026

Copy link
Copy Markdown

@gurmehakk20 is attempting to deploy a commit to the Aditya Sakhale's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant