Skip to content

Add IDSA scraper - #8

Open
KouateMuhamed wants to merge 3 commits into
MedARC-AI:mainfrom
KouateMuhamed:feat/idsa-scraper
Open

Add IDSA scraper#8
KouateMuhamed wants to merge 3 commits into
MedARC-AI:mainfrom
KouateMuhamed:feat/idsa-scraper

Conversation

@KouateMuhamed

@KouateMuhamed KouateMuhamed commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an AMFV-native scraper for IDSA practice guidelines through the shared scraper registry.
  • Include current and archived guidelines by default, exclude in-development records, and mark archived documents with quality_flags=["outdated"].
  • Add quality metadata for short guideline pages and retrievable external/PDF links.
  • Remove Table of Contents navigation and empty headings from scraped content.

Tests

  • uv run ruff check datasets
  • uv run ruff format --check datasets
  • uv run pytest datasets/test

QA

  • Scraped 79 current and archived IDSA documents from the live site with zero errors.
  • Confirmed 25 archived documents carry the outdated quality flag.
  • Confirmed metadata contains only HTTP/HTTPS links, every IDSA document has section_count=1, and generated Markdown has no empty headings.
  • Retained short documents with quality_flags=["short_content"].

@CLAassistant

CLAassistant commented Jul 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@warner-benjamin

Copy link
Copy Markdown
Collaborator

Thanks for the PR. This looks good, but it also grabs out of date guidelines. If we process those we'll want to add a metadata flag that they are out of date.

Comment thread datasets/amfv_datasets/scraping/idsa.py Outdated
Comment thread datasets/amfv_datasets/scraping/idsa.py
Comment thread datasets/amfv_datasets/scraping/idsa.py
Comment thread datasets/amfv_datasets/scraping/idsa.py Outdated
@KouateMuhamed

Copy link
Copy Markdown
Contributor Author

@zndr27 I addressed the review feedback in b60e4d6: archived guidelines are included by default and marked outdated, in-development records are excluded, link metadata is HTTP/HTTPS-only, empty headings are removed, and IDSA section_count is consistently 1. The datasets suite and Ruff checks pass; a live crawl completed 79/79 documents with 25 outdated and no metadata/link/heading issues. A re-review would be appreciated.

@KouateMuhamed
KouateMuhamed requested a review from zndr27 August 26, 2026 15:13
@KouateMuhamed

KouateMuhamed commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

@warner-benjamin CI run 32984321093 ended with startup_failure before any job step ran, so this is not a test failure. I cannot rerun it without repository admin permission. Local verification is green: 48 datasets tests, Ruff check, Ruff format check, and the 79-document live crawl all passed. Could a maintainer rerun the workflow?

@zndr27

zndr27 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Thank you. @warner-benjamin pinging you for final approval

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants