feat(crawl): add ZenRowsCrawlClient for the Crawl API

[StefanoGuerrini]

What this adds

A zenrows.crawl package with ZenRowsCrawlClient, also exported from the top level (from zenrows import ZenRowsCrawlClient). Crawl is in Beta: the README, the module docstring and the client docstring say so. There is no runtime warning.

• ZenRowsCrawlClient(api_key, *, base_url=None, timeout=30.0, retries=3, verify_ssl=True, user_agent=..., httpx_args=None). The default base URL is https://api.zenrows.com/v1. Override it with base_url= only; the client reads no environment variable.
• create(url, *, depth, max_items=None, max_pages=None, include_patterns=None, exclude_patterns=None, output_format: Literal["html"] | None = None, idempotency_key=None) -> Crawl sends only the fields you set. Without output_format, a crawl returns URLs only. A crawl stays on the start URL's registrable domain; subdomains count.
• get(crawl_id, *, cursor=None, limit=None) -> CrawlWithResults
• list_crawls(*, cursor=None, limit=None) -> CrawlList and iter_crawls(*, page_size=None)
• iter_results(crawl_id, *, page_size=None) follows next_cursor and returns at the first empty page or a null cursor. It does not poll: on a running crawl it yields only what the crawl has kept so far. Call wait first to read every result.
• stop(crawl_id) -> CrawlStop is idempotent.
• get_content(crawl_id, result_or_content_id) -> str returns one kept URL's HTML.
• iter_download(crawl_id) -> CrawlDownload opens the NDJSON file. Iterate it for parsed DownloadLines, and read .status (from X-Crawl-Status). It is also a context manager. download(crawl_id, path) -> CrawlDownloadFile streams the file to disk and returns .path and .status. On a running crawl, status is running and the file holds only what the crawl has kept so far.
• DownloadLine.content is str | dict | None, so a line with object content parses.
• wait(crawl_id, *, timeout=600.0, poll_interval=2.0, max_poll_interval=15.0) -> Crawl polls with backoff. A failed crawl is returned, not raised; read crawl.error.code and crawl.error.detail. A timeout raises WaiterTimeout and leaves the crawl running.
• Responses parse into pydantic v2 models. Unknown fields are ignored, and every response enum is open: a value added later parses as UNKNOWN and keeps its raw value.
• Errors raise CrawlAPIError, with status_code, code, detail, problem and retry_after. code is None when the body has no code. The README lists every code:
  • 400 invalid_request / unknown_parameter / invalid_cursor.
  • 403 REQS008 reads "Crawl is not enabled for this account", and exc.not_enabled is true for it.
  • 404 crawl_not_found / content_not_found.
  • 409 idempotency_request_in_flight: retry after the first request finishes.
  • 422 invalid_parameter / invalid_start_url / domain_not_allowed, and idempotency_key_reused (use a new key or no key).
  • 429 too_many_crawls, with retry_after: the account has reached its limit of active jobs (3 by default), shared with its Batch jobs. create never retries it, with or without an idempotency key.
• The package now ships a py.typed marker (in the wheel and the sdist), so type checkers use its types.

The SDK's shared HTTP transport now takes the error mapping and logger as constructor arguments, gains request() and stream() helpers, and takes the retryable statuses per request. ProblemDetail.code is None when the body has no code; BatchAPIError keeps its "internal" fallback. Existing clients behave as before.

Tests

• make check passes (ty, ruff check, ruff format --check).
• make test: 206 passed, 1 deselected (the e2e test).
• New e2e test, tests/e2e/test_crawl_e2e.py:
  • It is marked e2e and deselected from make test.
  • It skips unless ZENROWS_API_KEY, ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL are set. It passes ZENROWS_CRAWL_BASE_URL as base_url=. ZENROWS_E2E_CRAWL_INCLUDE is optional.
  • Run it with make test-e2e. DEVELOPMENT.md has a "Running the Crawl e2e test" section.
• I ran make test-e2e against a live deployment before the latest changes: 1 passed. I did not run it again after them.

Docs: a README section ("Quickstart — Crawl API (beta)"), examples/11_crawl.py, and DEVELOPMENT.md notes.

Release: this PR does not change the version. Do a minor bump (1.6.0) in the release commit.

[StefanoGuerrini]

Crawl Beta public-surface review

This review checks the public API against what production serves to a crawl-beta customer today: Crawl 475016a (docs/openapi/api.yaml) and gateway e382348. crawl-beta is at 100%. crawl-beta-extract (output_format: json, discovery: pagination) is staff-only. Reviewed at 9b3a2c6: make check (ty, ruff, ruff format) is clean, and make test passes 200 tests (1 e2e test deselected; I did not run it).

Verdict: changes needed. Items 1 to 7 become breaking changes if they ship as they are.

Fix before merge: breaking if shipped

1. No Beta marking. The README marks Batch "Private beta" and Extract "(beta)", but not Crawl. Add "(beta)" to the Crawl README section and say Beta in the zenrows.crawl module docstring and the ZenRowsCrawlClient docstring (proposed decision 1). Do not add a runtime warning.
2. iter_results follows a running crawl until it ends (src/zenrows/crawl/client.py:217). It sleeps and polls with no timeout, and the README documents this (README.md:526). Proposed decision 2: return at the first empty page. Callers who depend on the live follow break when this changes.
3. download and iter_download drop X-Crawl-Status (src/zenrows/crawl/client.py:258, :271). The status only goes to a log warning (:326), so a caller cannot tell a partial file from a full one. Return the status with the lines or the path (proposed decision 3). Adding it later changes the return types.
4. Crawl has no discovery and no extract_calls (src/zenrows/crawl/models.py:93). Production always returns discovery. Add discovery: list[Discovery] (open enum: links, pagination) and extract_calls: int | None = None (proposed decision 6). Adding a required field later breaks callers that build a Crawl, such as test fixtures.
5. DownloadLine.content is str | None (src/zenrows/crawl/models.py:175). A line with object content raises ValidationError (I checked this), so iter_download fails on a staff json crawl in the same org. Type it str | dict[str, Any] | None. Widening it later breaks callers that treat it as str.
6. code falls back to "internal" (src/zenrows/crawl/errors.py:41). A 502 with a non-JSON body gives code == "internal", a value the API never sends (production uses internal_error). Use None when the body has no code (proposed decision 5), and type code as str | None.
7. The client reads ZENROWS_CRAWL_BASE_URL (src/zenrows/crawl/client.py:81, comment at :43). Proposed decision 9: the SDKs take the base URL as a constructor option only. Remove the env read now; removing it after release breaks callers who set it. The e2e test, Makefile test-e2e and DEVELOPMENT.md can pass base_url= from their own variable.

Fix before merge: behaviour and docs

8. A keyed create retries 429 too_many_crawls. The shared transport retries 429 on a POST with an Idempotency-Key up to retries times and sleeps each Retry-After (src/zenrows/batch/_transport.py:33, :132, :152). The create docstring and the README say the caller gets the 429 with retry_after. too_many_crawls is a capacity limit, so do not retry it on create (proposed decision 7). Keep the 5xx and network retries.
9. wait() defaults to 300 s (src/zenrows/crawl/client.py:299). MCP and CLI use 600 s, and the README and the example pass 600. All surfaces will use one value; we have not picked the value yet (proposed decision 8).
10. The README error section is incomplete (README.md:558). The 429 limit is the account's active jobs: 3 by default, shared with its Batch jobs. The section leaves out 400 invalid_request/unknown_parameter/invalid_cursor, 409 idempotency_request_in_flight (retry later), and 422 domain_not_allowed/idempotency_key_reused (never retry).
11. The example always keeps /product/ (examples/11_crawl.py:29). action="append" with default=["/product/"] appends each --include to the default, so a user cannot remove it. Use default=None and fall back to ["/product/"] when it is empty.

Matches production

• create() takes url and depth as required, output_format="html" only, and no discovery. It sends only the arguments the caller set. Documented ranges and defaults match the spec.
• Response enums are open: a staff output_format: "json" parses as OutputFormat.UNKNOWN with the raw value. The models ignore unknown fields. CrawlErrorCode has all five run error codes.
• iter_crawls() ends on an absent next_cursor. list_crawls/iter_crawls(page_size=) follow the Batch client's names.
• wait() returns a failed crawl and does not raise. On timeout it raises WaiterTimeout and leaves the crawl running.
• 403 REQS008 maps to CrawlAPIError.not_enabled, and retry_after reads Retry-After.
• The _transport.py change is private: the new request()/stream() and the two new constructor arguments have defaults, and Batch behaviour is the same (its tests pass).

Can wait

These are additive and need no change now:

• OutputFormat has no JSON member.
• get_content() has no download=True option, and its docstring says "HTML text" only.
• CrawlAPIError does not expose the RateLimit headers.

Release: the PR leaves the version at 1.5.0, and the repo already has the v1.5.0 tag. Do a minor bump (1.6.0) in the release commit.

[StefanoGuerrini]

Replaced by #31: the same code, with a history that drops the AI co-author trailer. The review on this PR still applies.