Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/03_concepts/02_requests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The crawler's `max_crawl_depth` option limits how deep the crawl goes. Requests

## Binding to a session

Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. For details on sessions, see [Session management](./session-management).
Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. In the failed request handler, `context.session` is then a retired session with the bound id, not taken from the pool. For details on sessions, see [Session management](./session-management).

## Request lifecycle

Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/03_crawling_context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ For details on the crawler-specific attributes, see [HTTP crawlers](./http-crawl

The `request` attribute is the <ApiLink to="class/Request">`Request`</ApiLink> being processed. It carries the URL, the label used for routing, user data and the retry count. For details, see [Requests](./requests).

The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request, or `None` when the crawler runs without a session pool. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).
The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).

The `proxy_info` attribute is a <ApiLink to="class/ProxyInfo">`ProxyInfo`</ApiLink> describing the proxy used for the request, or `None` when no proxy is configured. For details, see [Proxy management](./proxy-management).

Expand Down
6 changes: 0 additions & 6 deletions docs/03_concepts/15_cookie_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,6 @@ Access the jar through <ApiLink to="class/Session#cookies">`context.session.cook
{ReadWriteCookies}
</RunnableCodeBlock>

:::note

`context.session` is `None` only when the crawler runs without a session pool (<ApiLink to="class/BasicCrawler#__init__">`use_session_pool`</ApiLink> set to `False`). With the default pool it's always set. The examples guard with `if context.session is None: return` to cover that case.

:::

A cookie has a `name`, a `value`, and optional parameters such as `domain`. If a cookie has no `domain`, it applies to any domain, so a login cookie set that way is sent to every host you crawl. Set `domain` on anything sensitive. For the full set of parameters, see <ApiLink to="class/CookieParam">`CookieParam`</ApiLink>. To dump the whole jar, use <ApiLink to="class/SessionCookies#get_cookies_as_dicts">`get_cookies_as_dicts()`</ApiLink>, and to load several cookies at once, use <ApiLink to="class/SessionCookies#set_cookies">`set_cookies()`</ApiLink>.

## Seeding cookies on new sessions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: PlaywrightCrawlingContext) -> None:
if context.session is None:
return

# The browser receives the `visited` cookie during the first navigation.
# It's synced back onto the session after the handler returns, so on the
# second request it's already on `context.session`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# Set a cookie on the session. It's sent with every following
# request that runs on this session.
context.session.cookies.set(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

if context.request.user_data.get('prepared'):
# Cookies from the setup step are on this pinned session.
context.log.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

state = await context.use_state(default_value={})
if not state.get('cookies'):
# First pass: establish cookies once and store them for every session,
Expand All @@ -30,7 +27,7 @@ async def apply_cookies(context: BasicCrawlingContext) -> None:
# Runs before navigation. Apply the shared cookies to whichever session
# handles the request, so every session ends up with them.
state = await context.use_state(default_value={})
if context.session and (cookies := state.get('cookies')):
if cookies := state.get('cookies'):
context.session.cookies.set_cookies(cast('list[CookieParam]', cookies))

await crawler.run(['https://httpbingo.org/cookies'])
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# First attempt: establish cookies, then raise to trigger a retry.
if not context.session.cookies:
await context.send_request('https://httpbingo.org/cookies/set?logged_in=1')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ async def main() -> None:
)
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
use_session_pool=True,
save_response_cookies=True,
)

# ...
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ async def main() -> None:
)
crawler = PlaywrightCrawler(
proxy_configuration=proxy_configuration,
use_session_pool=True,
save_response_cookies=True,
)

# ...
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,17 @@ async def basic_handler(context: HttpCrawlingContext) -> None:
# Initialize the session and bind the next request to this session if needed
@crawler.router.handler(label='session_init')
async def session_init(context: HttpCrawlingContext) -> None:
next_requests = []
if context.session:
context.log.info('Init session %s', context.session.id)
next_request = Request.from_url(
'https://a.placeholder.com', session_id=context.session.id
)
next_requests.append(next_request)

await context.add_requests(next_requests)
context.log.info('Init session %s', context.session.id)
next_request = Request.from_url(
'https://a.placeholder.com', session_id=context.session.id
)
await context.add_requests([next_request])

# Handle errors when a session is blocked and no longer available in the pool
# when attempting to execute requests bound to it
@crawler.failed_request_handler
async def error_processing(context: BasicCrawlingContext, error: Exception) -> None:
if isinstance(error, RequestCollisionError) and context.session:
if isinstance(error, RequestCollisionError):
context.log.error(
'Request %s failed, because the bound session is unavailable',
context.request.url,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,12 @@ async def basic_handler(context: HttpCrawlingContext) -> None:
# Handler for session initialization (authentication, initial cookies, etc.)
@crawler.router.handler(label='session_init')
async def session_init(context: HttpCrawlingContext) -> None:
if context.session:
context.log.info('Init session %s', context.session.id)
context.log.info('Init session %s', context.session.id)

# Monitor if our session gets blocked and explicitly stop the crawler
@crawler.error_handler
async def error_processing(context: BasicCrawlingContext, error: Exception) -> None:
if isinstance(error, SessionError) and context.session:
if isinstance(error, SessionError):
context.log.info('Session %s blocked', context.session.id)
crawler.stop()

Expand Down
17 changes: 8 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = BasicCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -33,13 +31,14 @@ async def default_handler(context: BasicCrawlingContext) -> None:
page_content = (await response.read()).decode()
title_match = re.search(r'<title(?:.*?)>(.*?)</title>', page_content)

if context.session and (title := title_match.group(1) if title_match else None):
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
title = title_match.group(1) if title_match else None

if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: BeautifulSoupCrawlingContext) -> None:
title = context.soup.title.get_text() if context.soup.title else None

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
17 changes: 8 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_http.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = HttpCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -29,13 +27,14 @@ async def default_handler(context: HttpCrawlingContext) -> None:
page_content = (await context.http_response.read()).decode()
title_match = re.search(r'<title(?:.*?)>(.*?)</title>', page_content)

if context.session and (title := title_match.group(1) if title_match else None):
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
title = title_match.group(1) if title_match else None

if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
15 changes: 6 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_parsel.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = ParselCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: ParselCrawlingContext) -> None:
title = context.selector.css('title::text').get()

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = PlaywrightCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: PlaywrightCrawlingContext) -> None:
title = await context.page.title()

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
4 changes: 0 additions & 4 deletions docs/04_guides/code_examples/login_crawler/http_login.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,6 @@ async def request_handler(context: HttpCrawlingContext) -> None:
async def login_handler(context: HttpCrawlingContext) -> None:
context.log.info('Processing login at %s ...', context.request.url)

# Verify that a session is available before proceeding
if not context.session:
raise RuntimeError('Session not found')

# Parse the API response containing authentication tokens and user data
data = json.loads(await context.http_response.read())

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,6 @@ async def request_handler(context: PlaywrightCrawlingContext) -> None:
async def login_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info('Processing login %s ...', context.request.url)

# Check if the session is available
if not context.session:
raise RuntimeError('Session not found')

# Entering data into the form, `delay` to simulate human typing
# Without this, the data will be entered instantly
await context.page.type('#userName', 'crawlee_test', delay=100)
Expand Down
4 changes: 0 additions & 4 deletions docs/04_guides/code_examples/scrapy_migration/crawlee_post.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@ async def main() -> None:

@crawler.router.default_handler
async def login_page(context: ParselCrawlingContext) -> None:
# The CSRF token is tied to the session cookie issued for this GET.
if not context.session:
raise RuntimeError('Session not found')

# highlight-start
# Like Scrapy's `FormRequest.from_response`, the helper keeps the hidden
# `csrf_token` field, encodes the data and sets the `Content-Type` header.
Expand Down
Loading
Loading