Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/03_concepts/02_requests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The crawler's `max_crawl_depth` option limits how deep the crawl goes. Requests

## Binding to a session

Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. For details on sessions, see [Session management](./session-management).
Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. In the failed request handler, `context.session` is then a retired session with the bound id, not taken from the pool. For details on sessions, see [Session management](./session-management).

## Request lifecycle

Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/03_crawling_context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ For details on the crawler-specific attributes, see [HTTP crawlers](./http-crawl

The `request` attribute is the <ApiLink to="class/Request">`Request`</ApiLink> being processed. It carries the URL, the label used for routing, user data and the retry count. For details, see [Requests](./requests).

The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request, or `None` when the crawler runs without a session pool. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).
The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).

The `proxy_info` attribute is a <ApiLink to="class/ProxyInfo">`ProxyInfo`</ApiLink> describing the proxy used for the request, or `None` when no proxy is configured. For details, see [Proxy management](./proxy-management).

Expand Down
6 changes: 0 additions & 6 deletions docs/03_concepts/15_cookie_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,6 @@ Access the jar through <ApiLink to="class/Session#cookies">`context.session.cook
{ReadWriteCookies}
</RunnableCodeBlock>

:::note

`context.session` is `None` only when the crawler runs without a session pool (<ApiLink to="class/BasicCrawler#__init__">`use_session_pool`</ApiLink> set to `False`). With the default pool it's always set. The examples guard with `if context.session is None: return` to cover that case.

:::

A cookie has a `name`, a `value`, and optional parameters such as `domain`. If a cookie has no `domain`, it applies to any domain, so a login cookie set that way is sent to every host you crawl. Set `domain` on anything sensitive. For the full set of parameters, see <ApiLink to="class/CookieParam">`CookieParam`</ApiLink>. To dump the whole jar, use <ApiLink to="class/SessionCookies#get_cookies_as_dicts">`get_cookies_as_dicts()`</ApiLink>, and to load several cookies at once, use <ApiLink to="class/SessionCookies#set_cookies">`set_cookies()`</ApiLink>.

## Seeding cookies on new sessions
Expand Down
10 changes: 5 additions & 5 deletions docs/03_concepts/22_statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ The raw data lives in a <ApiLink to="class/StatisticsState">`StatisticsState`</A
- `requests_retries` counts requests that needed at least one retry, and `request_retry_histogram` shows how many requests finished or failed after 0, 1, 2, and more retries.
- `request_total_finished_duration`, `request_total_failed_duration`, `request_min_duration`, and `request_max_duration` measure request processing time.
- `requests_with_status_code` maps HTTP status codes to the number of responses that returned them.
- `crawler_started_at`, `crawler_finished_at`, and the computed `crawler_runtime` describe the run itself.
- `crawler_started_at`, `crawler_finished_at`, and the computed `crawler_runtime` describe the crawl itself.

From the state, <ApiLink to="class/Statistics#calculate">`Statistics.calculate`</ApiLink> derives a <ApiLink to="class/FinalStatistics">`FinalStatistics`</ApiLink> object. It's a frozen summary with the request counts, the retry histogram as a list, average finished and failed durations, total duration, per-minute rates for finished and failed requests, and the crawler runtime. The same summary appears in the periodic and final statistics logs.

## Reading statistics

There are two ways to get the statistics of a crawl:

- <ApiLink to="class/BasicCrawler#run">`crawler.run()`</ApiLink> returns the `FinalStatistics` of the run once it ends.
- <ApiLink to="class/BasicCrawler#run">`crawler.run()`</ApiLink> returns the `FinalStatistics` once the run ends.
- <ApiLink to="class/BasicCrawler#statistics">`crawler.statistics`</ApiLink> gives you the `Statistics` instance, so you can read the full `state` or call `calculate()` at any time, including from a request handler while the crawl is running.

The following example reads the summary returned by `run()`, then looks at state fields that `FinalStatistics` doesn't include, and the errors collected during the run.
Expand All @@ -37,7 +37,7 @@ The following example reads the summary returned by `run()`, then looks at state
{ReadFinalStatisticsExample}
</RunnableCodeBlock>

A crawler that you run again with `run()` after a completed run resets its statistics at the start of the new run. To keep the numbers from a previous run, store the returned `FinalStatistics` before calling `run()` again.
A crawler that you run again with `run()` continues the crawl, so its statistics add up the runs. The runtime counts only the time the runs took, not the time between them. The `FinalStatistics` that a later `run()` returns include the earlier runs too. To count from zero again, call `crawler.statistics.reset()` before the next `run()`.

## Periodic logging

Expand All @@ -50,7 +50,7 @@ The crawler's constructor also accepts `statistics_log_format`. When the crawler

## Persistence

`Statistics` can save its state to a key-value store, so a crawl that gets migrated, aborted, or restarted continues counting from where it left off. When persistence is on, the state is loaded from the store when the crawl starts, saved on every `PERSIST_STATE` event, and saved once more when the crawl ends. For details about the event, see [Events](./events).
`Statistics` can save its state to a key-value store, so a crawl that gets migrated, aborted, or restarted continues counting from where it left off. When persistence is on, the state is loaded from the store when the crawl starts, saved on every `PERSIST_STATE` event, and saved once more when a run ends. For details about the event, see [Events](./events).

The following options control persistence:

Expand All @@ -74,6 +74,6 @@ A custom `Statistics` logs its periodic messages through the `crawlee.statistics

## Error tracking

Each `Statistics` instance holds an `error_tracker`, which groups errors from failed request attempts by their type, the place in the code where they were raised, and a similar message. Its `total` and `unique_error_count` properties count all errors and the distinct groups, and `get_most_common_errors()` returns the most frequent groups with their counts. At the end of a run that had errors, the crawler logs both counts. A second tracker, `error_tracker_retry`, collects session errors that caused a retry with a new session.
Each `Statistics` instance holds an `error_tracker`, which groups errors from failed request attempts by their type, the place in the code where they were raised, and a similar message. Its `total` and `unique_error_count` properties count all errors and the distinct groups, and `get_most_common_errors()` returns the most frequent groups with their counts. At the end of a run, the crawler logs both counts if there are any errors. A repeated `run()` keeps the errors of the earlier runs in the tracker, so the counts include them too. A second tracker, `error_tracker_retry`, collects session errors that caused a retry with a new session.

With `save_error_snapshots=True`, the error tracker also saves a snapshot of the page on the first occurrence of each unique error. For details and examples, see [Capturing page snapshots on errors](./error-handling#capturing-page-snapshots-on-errors).
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: PlaywrightCrawlingContext) -> None:
if context.session is None:
return

# The browser receives the `visited` cookie during the first navigation.
# It's synced back onto the session after the handler returns, so on the
# second request it's already on `context.session`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# Set a cookie on the session. It's sent with every following
# request that runs on this session.
context.session.cookies.set(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

if context.request.user_data.get('prepared'):
# Cookies from the setup step are on this pinned session.
context.log.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

state = await context.use_state(default_value={})
if not state.get('cookies'):
# First pass: establish cookies once and store them for every session,
Expand All @@ -30,7 +27,7 @@ async def apply_cookies(context: BasicCrawlingContext) -> None:
# Runs before navigation. Apply the shared cookies to whichever session
# handles the request, so every session ends up with them.
state = await context.use_state(default_value={})
if context.session and (cookies := state.get('cookies')):
if cookies := state.get('cookies'):
context.session.cookies.set_cookies(cast('list[CookieParam]', cookies))

await crawler.run(['https://httpbingo.org/cookies'])
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# First attempt: establish cookies, then raise to trigger a retry.
if not context.session.cookies:
await context.send_request('https://httpbingo.org/cookies/set?logged_in=1')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ async def main() -> None:
)
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
use_session_pool=True,
save_response_cookies=True,
)

# ...
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ async def main() -> None:
)
crawler = PlaywrightCrawler(
proxy_configuration=proxy_configuration,
use_session_pool=True,
save_response_cookies=True,
)

# ...
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,17 @@ async def basic_handler(context: HttpCrawlingContext) -> None:
# Initialize the session and bind the next request to this session if needed
@crawler.router.handler(label='session_init')
async def session_init(context: HttpCrawlingContext) -> None:
next_requests = []
if context.session:
context.log.info('Init session %s', context.session.id)
next_request = Request.from_url(
'https://a.placeholder.com', session_id=context.session.id
)
next_requests.append(next_request)

await context.add_requests(next_requests)
context.log.info('Init session %s', context.session.id)
next_request = Request.from_url(
'https://a.placeholder.com', session_id=context.session.id
)
await context.add_requests([next_request])

# Handle errors when a session is blocked and no longer available in the pool
# when attempting to execute requests bound to it
@crawler.failed_request_handler
async def error_processing(context: BasicCrawlingContext, error: Exception) -> None:
if isinstance(error, RequestCollisionError) and context.session:
if isinstance(error, RequestCollisionError):
context.log.error(
'Request %s failed, because the bound session is unavailable',
context.request.url,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,12 @@ async def basic_handler(context: HttpCrawlingContext) -> None:
# Handler for session initialization (authentication, initial cookies, etc.)
@crawler.router.handler(label='session_init')
async def session_init(context: HttpCrawlingContext) -> None:
if context.session:
context.log.info('Init session %s', context.session.id)
context.log.info('Init session %s', context.session.id)

# Monitor if our session gets blocked and explicitly stop the crawler
@crawler.error_handler
async def error_processing(context: BasicCrawlingContext, error: Exception) -> None:
if isinstance(error, SessionError) and context.session:
if isinstance(error, SessionError):
context.log.info('Session %s blocked', context.session.id)
crawler.stop()

Expand Down
17 changes: 8 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = BasicCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -33,13 +31,14 @@ async def default_handler(context: BasicCrawlingContext) -> None:
page_content = (await response.read()).decode()
title_match = re.search(r'<title(?:.*?)>(.*?)</title>', page_content)

if context.session and (title := title_match.group(1) if title_match else None):
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
title = title_match.group(1) if title_match else None

if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: BeautifulSoupCrawlingContext) -> None:
title = context.soup.title.get_text() if context.soup.title else None

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
17 changes: 8 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_http.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = HttpCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -29,13 +27,14 @@ async def default_handler(context: HttpCrawlingContext) -> None:
page_content = (await context.http_response.read()).decode()
title_match = re.search(r'<title(?:.*?)>(.*?)</title>', page_content)

if context.session and (title := title_match.group(1) if title_match else None):
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
title = title_match.group(1) if title_match else None

if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
15 changes: 6 additions & 9 deletions docs/03_concepts/code_examples/session_management/sm_parsel.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = ParselCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: ParselCrawlingContext) -> None:
title = context.selector.css('title::text').get()

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ async def main() -> None:
# to manage concurrent sessions and proxy rotation
crawler = PlaywrightCrawler(
proxy_configuration=proxy_configuration,
# Activates the Session pool (default is true).
use_session_pool=True,
# Overrides default Session pool configuration.
session_pool=SessionPool(max_pool_size=100),
)
Expand All @@ -27,13 +25,12 @@ async def main() -> None:
async def default_handler(context: PlaywrightCrawlingContext) -> None:
title = await context.page.title()

if context.session:
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.
if title == 'Blocked':
context.session.retire()
elif title == 'Not sure if blocked, might also be a connection error':
context.session.mark_bad()
else:
context.session.mark_good() # BasicCrawler handles this automatically.

await crawler.run(['https://crawlee.dev/'])

Expand Down
Loading
Loading