Skip to content

fix(crawlers): Keep the crawler usable after a failed run - #2255

Open
vdusek wants to merge 3 commits into
masterfrom
fix/crawler-failed-run
Open

vdusek wants to merge 3 commits into
masterfrom
fix/crawler-failed-run

Conversation

@vdusek

@vdusek vdusek commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #2229, split out of #2239.

Crawler state after a failed run

  • An exception during setup (e.g. add_requests) left _running set, so every later run() raised RuntimeError. The whole run body is now inside the try/finally.
  • After an earlier successful run, a failed run's implicit purge dropped the requests it left pending. A _last_run_failed flag now skips that purge, even with purge_request_queue=True. A failure in the post-crawl steps (statistics, _save_crawler_state()) counts as a failed run.

AutoscaledPool

  • run.result.exception() is guarded against a cancelled future. On SIGINT it raised CancelledError in the orchestrator's finally, and the error log was lost.
  • The orchestrator's fallback error message now reads Unpropagated exception in worker task orchestrator, so it can be told apart from the same message in run().

✍️ Drafted by Claude Code

@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Sep 24, 2026
@vdusek vdusek self-assigned this Sep 24, 2026
@codecov

codecov Bot commented Sep 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.73%. Comparing base (45bb0e2) to head (f3c83e5).

Files with missing lines Patch % Lines
src/crawlee/crawlers/_basic/_basic_crawler.py 87.50% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2255      +/-   ##
==========================================
- Coverage   93.74%   93.73%   -0.01%     
==========================================
  Files         181      181              
  Lines       12884    12890       +6     
==========================================
+ Hits        12078    12083       +5     
- Misses        806      807       +1     
Flag Coverage Δ
unit 93.73% <88.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek changed the title fix(crawlers): Keep the crawler usable after a failed run fix(crawlers)!: Keep the crawler usable after a failed run Sep 24, 2026
@vdusek vdusek changed the title fix(crawlers)!: Keep the crawler usable after a failed run fix(crawlers): Keep the crawler usable after a failed run Sep 24, 2026
@vdusek
vdusek requested a review from Pijukatel September 24, 2026 13:06
@vdusek
vdusek marked this pull request as ready for review September 24, 2026 13:06
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Sep 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants