Skip to content

fix(crawlers): Report robots.txt skips and fetch each robots.txt once in AdaptivePlaywrightCrawler - #2452

Open
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt
Open

Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:adaptive-robots-txt

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • Links that AdaptivePlaywrightCrawler skips in enqueue_links because of robots.txt now reach its on_skipped_request callback.
  • The crawler fetches the robots.txt file of a domain once. Each of its sub crawlers used to fetch the file again, up to three times per domain during a detection run.

Testing

  • Integration tests for skipped links with static and browser requests, and for a single robots.txt fetch per domain.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 15, 2026
@Mantisus
Mantisus requested a review from Pijukatel November 15, 2026 18:29
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant