Add IDSA scraper - #8
Conversation
|
Thanks for the PR. This looks good, but it also grabs out of date guidelines. If we process those we'll want to add a metadata flag that they are out of date. |
5a157e4 to
4353f05
Compare
4353f05 to
4808499
Compare
|
@zndr27 I addressed the review feedback in b60e4d6: archived guidelines are included by default and marked outdated, in-development records are excluded, link metadata is HTTP/HTTPS-only, empty headings are removed, and IDSA section_count is consistently 1. The datasets suite and Ruff checks pass; a live crawl completed 79/79 documents with 25 outdated and no metadata/link/heading issues. A re-review would be appreciated. |
|
@warner-benjamin CI run 32984321093 ended with startup_failure before any job step ran, so this is not a test failure. I cannot rerun it without repository admin permission. Local verification is green: 48 datasets tests, Ruff check, Ruff format check, and the 79-document live crawl all passed. Could a maintainer rerun the workflow? |
|
Thank you. @warner-benjamin pinging you for final approval |
Summary
quality_flags=["outdated"].Tests
uv run ruff check datasetsuv run ruff format --check datasetsuv run pytest datasets/testQA
outdatedquality flag.section_count=1, and generated Markdown has no empty headings.quality_flags=["short_content"].