Skip to content

fix(browsers): Release a page whose close never finishes - #2441

Open
PrashanthLande wants to merge 2 commits into
apify:masterfrom
PrashanthLande:fix/page-close-timeout
Open

PrashanthLande wants to merge 2 commits into
apify:masterfrom
PrashanthLande:fix/page-close-timeout

Conversation

@PrashanthLande

@PrashanthLande PrashanthLande commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • PlaywrightCrawler hangs forever when Chromium never finishes a page.close(). This happens when a page navigates itself with JavaScript (e.g. location.reload()) right as the crawler closes it. crawler.run() never returns, even when wrapped in asyncio.wait_for.
  • Ports fix(browser-pool): survive a page.close() that never settles crawlee#4116 (v3: fix(browser-pool): survive a page.close() that never settles crawlee#4108), with one difference: only the page close itself is time limited, not the close hooks. BrowserPool now gives page.close() 5 seconds. If it does not finish, it logs a warning, releases the page from its browser and retires the browser, so it gets no new pages and is closed once its remaining pages are closed.
  • Pre and post page close hooks run without a time limit, as before, so slow user hooks are not cancelled.
  • The close wrapper is now always installed, not only when close hooks are registered.
  • Adds BrowserController.register_page_closed(), so the page is released even when its close event never arrives. It has a no-op default, so existing custom controllers keep working.
  • Uses a fixed 5 s limit like JS (PAGE_CLOSE_TIMEOUT_MILLIS) rather than operation_timeout, since a healthy close takes milliseconds. Happy to make it configurable if you prefer.
  • Note: when the close times out, the post page close hooks still run, since the page has been released from the pool. Before, a hanging close meant they never ran.
  • Known limitation: if closing a retired browser itself hangs, the inactive-browser cleanup task blocks too. Same as JS; left out of scope.

Issues

  • None. Found in production on a WordPress site using LiteSpeed Cache guest mode, which reloads the page for new visitors. After all requests were handled, the log kept repeating Crawled 3/3 pages, 0 failed requests for over 20 minutes, and a coroutine dump of the stuck process showed the worker task waiting on Page.close inside the crawler's deferred cleanup.

Testing

  • Added tests in tests/integration/browsers/test_browser_pool.py:
    • a close that never finishes releases the page and retires the browser (fails without the fix);
    • pre and post close hooks slower than the time limit run to completion and do not retire the browser.
  • The repro below simulates the hang by patching Playwright's internal Page.close to never finish (the real trigger, a page reloading itself during close, is timing-dependent and hard to reproduce on demand). Before the fix crawler.run() hung forever; now it logs the warning and returns after about 6 s.
  • Lint and type checks pass.
Repro (simulated hang)
import asyncio

from playwright._impl._page import Page as ImplPage

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def never_finishing_close(self, *args, **kwargs):
    # Simulates Chromium never finishing the close, as it does when the page is navigating at that moment.
    await asyncio.Event().wait()


ImplPage.close = never_finishing_close


async def main() -> None:
    crawler = PlaywrightCrawler()

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Handled {context.request.url}')

    await crawler.run(['https://crawlee.dev'])
    print('crawler.run() returned')


asyncio.run(main())

Checklist

  • CI passed

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 13, 2026
@vdusek
vdusek requested a review from Mantisus November 13, 2026 12:42

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, but in its current form it introduces a breaking change. Could you update it so the timeout applies only to the page close?

Comment thread src/crawlee/browsers/_browser_pool.py Outdated
page: Any = crawlee_page.page
page.close = close_with_hooks
try:
await asyncio.wait_for(close_sequence(), timeout)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also cancels user hooks that take longer than 5 seconds, which is a breaking change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Mantisus , updated: the timeout now applies only to page.close() itself. Pre and post page close hooks run without a time limit, as before. Added test_slow_page_close_hooks_are_not_cancelled to cover it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants