Skip to content

fix(crawlers): Report robots.txt skips and fetch each robots.txt once in AdaptivePlaywrightCrawler - #2452

Open
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt
Open

Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt

report robots.txt skips and fetch each robots.txt once in `AdaptivePl…

cb8888d
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 notice
Add labels
succeeded Nov 15, 2026 in 6s