Skip to content

fix(crawlers): Report robots.txt skips and fetch each robots.txt once in AdaptivePlaywrightCrawler - #2452

Open
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt
Open

Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt

Commits

  1. Commits on Nov 15, 2026