Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/03_concepts/02_requests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The crawler's `max_crawl_depth` option limits how deep the crawl goes. Requests

## Binding to a session

Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. For details on sessions, see [Session management](./session-management).
Pass `session_id` to `from_url` to bind a request to one session from the crawler's <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. The crawler then processes the request with that session only. If the session isn't available anymore, the request fails with a <ApiLink to="class/RequestCollisionError">`RequestCollisionError`</ApiLink> and isn't retried. In the failed request handler, `context.session` is then a retired session with the bound id, not taken from the pool. For details on sessions, see [Session management](./session-management).

## Request lifecycle

Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/03_crawling_context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ For details on the crawler-specific attributes, see [HTTP crawlers](./http-crawl

The `request` attribute is the <ApiLink to="class/Request">`Request`</ApiLink> being processed. It carries the URL, the label used for routing, user data and the retry count. For details, see [Requests](./requests).

The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request, or `None` when the crawler runs without a session pool. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).
The `session` attribute is the <ApiLink to="class/Session">`Session`</ApiLink> assigned to the request. You can call `mark_bad()` or `retire()` on it from the handler when a page shows signs of blocking. For details, see [Session management](./session-management).

The `proxy_info` attribute is a <ApiLink to="class/ProxyInfo">`ProxyInfo`</ApiLink> describing the proxy used for the request, or `None` when no proxy is configured. For details, see [Proxy management](./proxy-management).

Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/06_playwright_crawler.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ The following example crawls the Hacker News website using Chromium. The crawler

## Advanced configuration

The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> uses other Crawlee components under the hood, notably <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink> and <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>. These components let you to configure the browser and context settings, launch multiple browsers, and apply pre-navigation hooks. You can create your own instances of these components and pass them to the <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> constructor.
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> uses other Crawlee components under the hood, notably <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink> and <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>. These components let you configure the browser and context settings, launch multiple browsers, and apply pre-navigation hooks. You can create your own instances of these components and pass them to the <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> constructor. If you pass in a <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink>, enter it with `async with` and keep it open until the crawler finishes.

- The <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink> manages how browsers are launched and how browser contexts are created. It accepts [browser launch](https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch) and [new context](https://playwright.dev/python/docs/api/class-browser#browser-new-context) options.
- The <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink> manages the lifecycle of browser instances (launching, recycling, etc.). You can customize its behavior to suit your needs.
Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/07_playwright_crawler_adaptive.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ In the following example you can see how to create and configure <ApiLink to="cl

To control which pages are crawled by which method you can use following arguments:

<ApiLink to="class/RenderingTypePredictor">`RenderingTypePredictor`</ApiLink> - Class that can give recommendations about which sub crawler should be used for specific url. Predictor will also recommend to use both sub crawlers for some page from time to time, to check that the given recommendation was correct. Predictor should be able to learn from previous results and gradually give more reliable recommendations.
<ApiLink to="class/RenderingTypePredictor">`RenderingTypePredictor`</ApiLink> - Class that can give recommendations about which sub crawler should be used for specific url. Predictor will also recommend to use both sub crawlers for some page from time to time, to check that the given recommendation was correct. Predictor should be able to learn from previous results and gradually give more reliable recommendations. If you pass in a predictor, enter it with `async with` before you run the crawler.

`result_checker` - Is a function that checks result created from crawling a page. By default, it always returns `True`.

Expand Down
4 changes: 2 additions & 2 deletions docs/03_concepts/08_pydantic_ai_crawler.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ crawler = PydanticAiCrawler(model=model)

## Extractors

An extractor turns a page into your schema. Extractors implement different strategies for working with the LLM, and each one uses a <ApiLink to="class/PydanticAiHtmlDistiller">`PydanticAiHtmlDistiller`</ApiLink> to shape the model's input. Crawlee ships two.
An extractor turns a page into your schema. Extractors implement different strategies for working with the LLM, and each one uses a <ApiLink to="class/PydanticAiHtmlDistiller">`PydanticAiHtmlDistiller`</ApiLink> to shape the model's input. Crawlee ships two. An extractor you pass to the crawler is yours to enter, so run the crawler inside `async with extractor`. The crawler enters only the default extractor it builds from `model`.

### PydanticAiDirectExtractor

Expand All @@ -96,7 +96,7 @@ To focus the model on the data you want, use `additional_instructions`:

### PydanticAiSelectorExtractor

<ApiLink to="class/PydanticAiSelectorExtractor">`PydanticAiSelectorExtractor`</ApiLink> asks the model for reusable CSS selectors on the first page of a route, caches them, and reuses them with no model call on later pages of the same layout, so it scales to large sites. When a page matches none of the cached selectors (a different markup variant), it generates and caches a new set, so one bucket can hold several variants. If selector generation fails, or the schema shape is unsupported, it degrades to the `fallback` extractor when one is set, and raises otherwise. Selectors are bucketed by `cache_tag`, that defaults to the request label, so each route keeps its own set. The cache is persisted to a <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink>, so a later run reuses selectors learned earlier.
<ApiLink to="class/PydanticAiSelectorExtractor">`PydanticAiSelectorExtractor`</ApiLink> asks the model for reusable CSS selectors on the first page of a route, caches them, and reuses them with no model call on later pages of the same layout, so it scales to large sites. When a page matches none of the cached selectors (a different markup variant), it generates and caches a new set, so one bucket can hold several variants. If selector generation fails, or the schema shape is unsupported, it degrades to the `fallback` extractor when one is set, and raises otherwise. Selectors are bucketed by `cache_tag`, that defaults to the request label, so each route keeps its own set. The cache is persisted to a <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink>, so a later run reuses selectors learned earlier. Entering the extractor also enters its `fallback`. An extractor can be entered only once at a time, so give each selector extractor a fallback of its own.

<CodeBlock className="language-python">
{SelectorExtractorExample}
Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/13_http_clients.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ HttpClient --|> CurlImpersonateHttpClient

Crawlee currently provides three main HTTP clients: <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>, which uses the `impit` library, <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink>, which uses the `httpx2` library with `browserforge` for custom HTTP headers and fingerprints, and <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>, which uses the `curl-cffi` library. You can switch between them by setting the `http_client` parameter when initializing a crawler class. The default HTTP client is <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>. For more details on anti-blocking features, see our [avoid getting blocked guide](../guides/avoid-blocking).

Below are examples of how to configure the HTTP client for the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>:
Below are examples of how to configure the HTTP client for the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>. A client you pass in is yours to enter, so each example runs the crawler inside `async with http_client`:

<Tabs>
<TabItem value="ParselHttpxExample" label="ParselCrawler with HTTPX">
Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/14_session_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Now, let's explore examples of how to use the <ApiLink to="class/SessionPool">`S
</TabItem>
</Tabs>

These examples demonstrate the basics of configuring and using the <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>.
These examples demonstrate the basics of configuring and using the <ApiLink to="class/SessionPool">`SessionPool`</ApiLink>. A crawler doesn't enter a pool you pass to it, so enter the pool with `async with` yourself and keep it open until the crawler finishes.

Please, bear in mind that <ApiLink to="class/SessionPool">`SessionPool`</ApiLink> requires some time to establish a stable pool of working IPs. During the initial setup, you may encounter errors as the pool identifies and filters out blocked or non-functional IPs. This stabilization period is expected and will improve over time.

Expand Down
6 changes: 0 additions & 6 deletions docs/03_concepts/15_cookie_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,6 @@ Access the jar through <ApiLink to="class/Session#cookies">`context.session.cook
{ReadWriteCookies}
</RunnableCodeBlock>

:::note

`context.session` is `None` only when the crawler runs without a session pool (<ApiLink to="class/BasicCrawler#__init__">`use_session_pool`</ApiLink> set to `False`). With the default pool it's always set. The examples guard with `if context.session is None: return` to cover that case.

:::

A cookie has a `name`, a `value`, and optional parameters such as `domain`. If a cookie has no `domain`, it applies to any domain, so a login cookie set that way is sent to every host you crawl. Set `domain` on anything sensitive. For the full set of parameters, see <ApiLink to="class/CookieParam">`CookieParam`</ApiLink>. To dump the whole jar, use <ApiLink to="class/SessionCookies#get_cookies_as_dicts">`get_cookies_as_dicts()`</ApiLink>, and to load several cookies at once, use <ApiLink to="class/SessionCookies#set_cookies">`set_cookies()`</ApiLink>.

## Seeding cookies on new sessions
Expand Down
6 changes: 5 additions & 1 deletion docs/03_concepts/16_http_headers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,11 @@ from crawlee.crawlers import HttpCrawler
from crawlee.http_clients import ImpitHttpClient

# Send plain requests with no browser-like headers.
crawler = HttpCrawler(http_client=ImpitHttpClient(browser=None))
http_client = ImpitHttpClient(browser=None)
crawler = HttpCrawler(http_client=http_client)

async with http_client:
await crawler.run(['https://httpbin.org/headers'])
```

The opt-out is named differently on each client:
Expand Down
2 changes: 1 addition & 1 deletion docs/03_concepts/22_statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ The statistics a crawler creates for itself have persistence enabled and use the

## Custom statistics

To change any of these options, create a `Statistics` instance and pass it to the crawler through the `statistics` argument. Use <ApiLink to="class/Statistics#with_default_state">`Statistics.with_default_state`</ApiLink> to get an instance with the standard `StatisticsState` model.
To change any of these options, create a `Statistics` instance and pass it to the crawler through the `statistics` argument. Use <ApiLink to="class/Statistics#with_default_state">`Statistics.with_default_state`</ApiLink> to get an instance with the standard `StatisticsState` model. The crawler enters the instance on every run, so don't enter it yourself. Crawlers that run one after another can share one instance, which then adds up their runs, including the requests that `max_requests_per_crawl` counts. Two crawlers can't run with the same instance at the same time.

The following example logs the statistics every second in the `inline` format, persists them under a custom key, and reads the persisted record back after the run.

Expand Down
23 changes: 13 additions & 10 deletions docs/03_concepts/code_examples/cookie_management/initial_cookies.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,18 @@


async def main() -> None:
session_pool = SessionPool(
# Seed every new session in the pool with these cookies.
create_session_settings={
'cookies': [
{'name': 'consent', 'value': 'accepted', 'domain': 'httpbingo.org'},
{'name': 'locale', 'value': 'en_US', 'domain': 'httpbingo.org'},
],
},
)

crawler = HttpCrawler(
session_pool=SessionPool(
# Seed every new session in the pool with these cookies.
create_session_settings={
'cookies': [
{'name': 'consent', 'value': 'accepted', 'domain': 'httpbingo.org'},
{'name': 'locale', 'value': 'en_US', 'domain': 'httpbingo.org'},
],
},
),
session_pool=session_pool,
)

@crawler.router.default_handler
Expand All @@ -23,7 +25,8 @@ async def handler(context: HttpCrawlingContext) -> None:
response = await context.http_response.read()
context.log.info(response.decode())

await crawler.run(['https://httpbingo.org/cookies'])
async with session_pool:
await crawler.run(['https://httpbingo.org/cookies'])


if __name__ == '__main__':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,15 @@


async def main() -> None:
# A single session, so both requests run on the same one.
session_pool = SessionPool(max_pool_size=1)

crawler = PlaywrightCrawler(
# A single session, so both requests run on the same one.
session_pool=SessionPool(max_pool_size=1),
session_pool=session_pool,
)

@crawler.router.default_handler
async def handler(context: PlaywrightCrawlingContext) -> None:
if context.session is None:
return

# The browser receives the `visited` cookie during the first navigation.
# It's synced back onto the session after the handler returns, so on the
# second request it's already on `context.session`.
Expand All @@ -33,7 +32,8 @@ async def handler(context: PlaywrightCrawlingContext) -> None:
]
)

await crawler.run(['https://httpbingo.org/cookies/set?visited=1'])
async with session_pool:
await crawler.run(['https://httpbingo.org/cookies/set?visited=1'])


if __name__ == '__main__':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# Set a cookie on the session. It's sent with every following
# request that runs on this session.
context.session.cookies.set(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

if context.request.user_data.get('prepared'):
# Cookies from the setup step are on this pinned session.
context.log.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ async def main() -> None:

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

state = await context.use_state(default_value={})
if not state.get('cookies'):
# First pass: establish cookies once and store them for every session,
Expand All @@ -30,7 +27,7 @@ async def apply_cookies(context: BasicCrawlingContext) -> None:
# Runs before navigation. Apply the shared cookies to whichever session
# handles the request, so every session ends up with them.
state = await context.use_state(default_value={})
if context.session and (cookies := state.get('cookies')):
if cookies := state.get('cookies'):
context.session.cookies.set_cookies(cast('list[CookieParam]', cookies))

await crawler.run(['https://httpbingo.org/cookies'])
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,25 @@


async def main() -> None:
session_pool = SessionPool(
# A single session, so every attempt runs on the same one.
max_pool_size=1,
create_session_settings={
# Keep the single session alive for the whole crawl.
'max_usage_count': 999_999,
'max_age': timedelta(hours=999_999),
'max_error_score': 100,
},
)

crawler = HttpCrawler(
# With a single session, don't burn retries trying to rotate to another one.
max_session_rotations=0,
session_pool=SessionPool(
# A single session, so every attempt runs on the same one.
max_pool_size=1,
create_session_settings={
# Keep the single session alive for the whole crawl.
'max_usage_count': 999_999,
'max_age': timedelta(hours=999_999),
'max_error_score': 100,
},
),
session_pool=session_pool,
)

@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
if context.session is None:
return

# First attempt: establish cookies, then raise to trigger a retry.
if not context.session.cookies:
await context.send_request('https://httpbingo.org/cookies/set?logged_in=1')
Expand All @@ -34,7 +33,8 @@ async def handler(context: HttpCrawlingContext) -> None:
# The retry runs on the same session, so the cookies are still here.
context.log.info('Cookies: %s', context.session.cookies.get_cookies_as_dicts())

await crawler.run(['https://httpbingo.org/cookies'])
async with session_pool:
await crawler.run(['https://httpbingo.org/cookies'])


if __name__ == '__main__':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@


async def main() -> None:
# Only treat 403 as a blocking status code, not 401
session_pool = SessionPool(create_session_settings={'blocked_status_codes': [403]})

crawler = HttpCrawler(
max_request_retries=2,
# Only treat 403 as a blocking status code, not 401
session_pool=SessionPool(create_session_settings={'blocked_status_codes': [403]}),
session_pool=session_pool,
# Don't treat 401 responses as errors
ignore_http_error_status_codes=[UNAUTHORIZED_CODE],
)
Expand All @@ -38,7 +40,8 @@ async def default_handler(context: HttpCrawlingContext) -> None:
# Trigger a retry with our updated headers
raise HttpStatusCodeError('Unauthorized', status_code=UNAUTHORIZED_CODE)

await crawler.run(['http://httpbingo.org/status/401'])
async with session_pool:
await crawler.run(['http://httpbingo.org/status/401'])


if __name__ == '__main__':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ async def request_handler(context: ParselCrawlingContext) -> None:
await context.push_data(data)

# Run the crawler with the initial list of URLs.
await crawler.run(['https://crawlee.dev'])
async with http_client:
await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
Expand Down
Loading
Loading