feat(crawl): add ZenRowsCrawlClient for the Crawl API

[StefanoGuerrini]

Summary

This PR adds ZenRowsCrawlClient, a typed client for the Crawl API, in a new zenrows.crawl package. The shared HTTP transport now serves both the Batch and the Crawl clients. It caps a Retry-After wait at 30 s for both clients. depth is optional on create: without it the crawl has no depth limit. max_pages and max_items default to 100 on the server. stop retries a 429, 502, 503 or 504 answer, waiting the Retry-After value (at most 30 s, or 1 s when it is missing or unusable). The PR also adds a first-install acceptance test for the Crawl client under acceptance/crawl/.

This PR also carries #32 and #35, merged into this branch.

This PR also carries #36 and #37, folded into this branch.

Squash commit message

feat(crawl): add ZenRowsCrawlClient for the Crawl API

Add the zenrows.crawl package with ZenRowsCrawlClient: create, get,
list, results, stop, content, download and wait. Parse responses into
pydantic v2 models with open enums. Raise CrawlAPIError for every
non-2xx response, and export it and the client from zenrows.

Make depth optional on create and on the Crawl model: without it the
crawl has no depth limit and the API echoes no depth. Document the
server defaults of 100 for max_pages and max_items.

Let stop retry a 429, 502, 503 (crawl_busy included) or 504 up to
`retries` more times, since repeating a stop is safe. It waits the
Retry-After value, at most 30 seconds, or 1 second when the value is
missing or unusable. Any other status and network errors still raise
at once.

Let the shared HTTP transport take an error mapping, a logger and
per-request retry statuses. Cap a Retry-After wait at 30 seconds and
ignore a NaN or infinite value; this also applies to
ZenRowsBatchClient. CrawlAPIError.code is None when the body has no
code. Batch still reports a missing code as "internal", in both
ProblemDetail.code and BatchAPIError.code.

Ship a py.typed marker. Add an opt-in e2e test (make test-e2e), a
README quickstart, examples/11_crawl.py and DEVELOPMENT.md notes.

Add acceptance/crawl/, a first-install acceptance test. It builds the
wheel, installs it in a clean container, runs the README Crawl
quickstart and checks every Crawl feature against the live API.
Exclude /acceptance from the sdist. make check also lints acceptance/.

Squash-merge with this message: the branch commits carry trailer lines that should not land.

Changes since #30

This PR replaces #30. The review of the first version is here. Since that review, these things changed:

• content() names its argument content_id.
• The top-level zenrows package exports CrawlAPIError.
• The transport waits at most 30 s for a Retry-After.
• stop retries a 503 crawl_busy, and now also a 429, 502, any other 503 and 504.
• depth is optional, and the documented defaults for max_pages and max_items are 100.
• The branch adds the acceptance test.

What this adds

Import the client from the package or from the top level: from zenrows import ZenRowsCrawlClient, CrawlAPIError. README.md and the docstrings call Crawl "(new)". README.md and the package docstring add one line that Crawl is still evolving. The client gives no runtime warning.

Method	Behavior
ZenRowsCrawlClient(api_key, *, base_url=None, timeout=30.0, retries=3, verify_ssl=True, user_agent=DEFAULT_USER_AGENT, httpx_args=None)	Default User-Agent zenrows-crawl-python/<version>. Default base URL https://api.zenrows.com/v1; override it with base_url= only. The client reads no environment variable.
create(url, *, depth=None, max_items=None, max_pages=None, include_patterns=None, exclude_patterns=None, output_format: Literal["html"] | None = None, idempotency_key=None) -> Crawl	Sends only the fields you set. Without depth the crawl has no depth limit: it follows links until max_items or max_pages stops it, or no link is left, and Crawl.depth reads None. max_pages and max_items default to 100 on the server. Only the type hint checks output_format; the API decides. Without output_format, a crawl returns URLs only. A crawl stays on the start URL's registrable domain; subdomains count.
get(crawl_id, *, cursor=None, limit=None) -> CrawlWithResults	One crawl with one page of results.
list(*, cursor=None, limit=None) -> CrawlList	One page of crawls, newest first. next_cursor is None on the last page.
results(crawl_id, *, limit=None)	Follows next_cursor, limit URLs per request, and stops at the first empty page or a null cursor. It does not poll. On a running crawl it yields only what the crawl kept so far; call wait first.
stop(crawl_id) -> CrawlStop	Idempotent. Retries a 429, 502, 503 or 504; see the next section.
content(crawl_id, content_id) -> str	One kept URL's HTML. content_id is a CrawlResult, its content_url, or a content id.
download(crawl_id) -> CrawlDownload	Opens the NDJSON file. Iterate lines for parsed DownloadLines; read .status from X-Crawl-Status. It is also a context manager. On a running crawl, status is running and the file holds what the crawl kept so far. DownloadLine.content is str | dict | None.
wait(crawl_id, *, timeout=600.0) -> Crawl	Polls with backoff (2 s, x1.5, up to 15 s) until the crawl ends or timeout passes. It never raises for the crawl's state. On timeout it returns the crawl, still running. It returns a failed crawl; read crawl.error.code and crawl.error.detail.

Retries: the client retries up to retries times on 429, 502, 503, 504 and network errors. It retries GETs, and a create that carries an idempotency_key. create never retries a 429. The client uses jittered exponential backoff and honors a Retry-After up to 30 s.

Models: responses parse into pydantic v2 models. The models ignore unknown fields. Every response enum is open: a value added later parses as UNKNOWN and keeps its raw value.

Errors: the client raises CrawlAPIError, with status_code, code, detail, problem and retry_after. code is None when the body has no code. retry_after is None when Retry-After is missing, negative, NaN or infinite. README.md lists every code:

Status	Code	Meaning
400	invalid_request, unknown_parameter, invalid_cursor	Fix the request.
403	REQS008	Crawl is not enabled for this account. Compare exc.code with the exported CRAWL_NOT_ENABLED_CODE.
404	crawl_not_found, content_not_found	Not found.
409	idempotency_request_in_flight	Retry after the first request finishes.
422	invalid_parameter, invalid_start_url, domain_not_allowed	Fix the request.
422	idempotency_key_reused	Use a new key, or no key.
429	too_many_crawls	The account reached its limit of active jobs (3 by default), shared with its Batch jobs. create never retries it.
503	crawl_busy	The API did not save the stop, and the crawl is still running. stop retries it, then raises it. A second stop is safe.

stop retries every 429, 502, 503 and 504, whatever the code, then raises the last error.

Packaging: the package now ships a py.typed marker, so type checkers use its types.

Shared transport: it now takes the error mapping and the logger as constructor arguments. It gains request() and stream() helpers and takes the retryable statuses per request. These changes apply to ZenRowsBatchClient too. Batch still reports a missing code as "internal", in both ProblemDetail.code and BatchAPIError.code. Otherwise the existing clients behave as before.

Stop retries

A stop can collide with the crawl while the crawl saves its own progress. The API retries that collision for under 2 s. If the collision stays, the API answers 503 with code: crawl_busy and Retry-After: 1. Stop is a POST without an Idempotency-Key, so the shared transport never retries it. Without this change, the caller saw an error while the crawl kept running. A stop is safe to repeat (stopping an ended crawl answers 200 with its status), so stop retries the transient statuses itself; the shared transport's rule for POST is unchanged.

Case	stop behavior
429, 502, 503 (any code, crawl_busy included) or 504	Retries up to retries more times (default 3), then raises the last CrawlAPIError. retries=0 turns the retry off.
Wait between tries	The Retry-After value, at most 30 s. A missing, negative, NaN or infinite value gives 1 s.
Any other status (for example 500 or a 4xx), a network error	Raises at once.

Acceptance test

acceptance/crawl/run.sh tests the release artifact the way a new user meets it. It builds the wheel with python -m build in its own container. Then it installs the wheel in a clean, digest-pinned container with no source, caches or tokens. It runs the README.md Crawl quickstart as written, then checks 1 to 21 and a mypy --strict check. It stops every crawl it started, also on a failure or Ctrl-C. It writes acceptance/crawl/out/crawl-first-install-report.md and makes sure that no file under out/ holds the key. acceptance/crawl/README.md lists each check with its expected result. The harness runs on macOS and Linux, and make check lints it.

Input	Default
CRAWL_URL	https://www.scrapingcourse.com/ecommerce/
CRAWL_INCLUDE	/product/
CRAWL_API_BASE	https://api.zenrows.com/v1
REF	unset: this checkout
OUT_DIR	moves the output

Check 1's offline half creates without depth against a stand-in server and checks that no depth is sent or read back. Check 21 expects the documented default of 100 for max_pages and max_items; it passes once the live API makes depth optional and applies the new defaults.

Cost: a run starts five crawls, each with depth 1. Every crawl but check 19's sets max_pages 10 or less; check 19's leaves it out so check 21 can read the default, and it fetches only the start page. On the default CRAWL_URL, a run fetches about 10 pages, and at most 140. Check 9 can start a sixth crawl, which adds at most 10 pages. At most 2 crawls run at the same time, under the account limit of 3 active jobs. Do not run two copies against one account at the same time.

Packaging: pyproject.toml excludes /acceptance from the sdist, so the harness never ships in the package.

Tests

Test	Result or coverage
make check (ty, ruff check, ruff format --check)	Passes.
make test	Passes. It deselects the e2e test.
wait	Returns a running crawl on timeout, and returns a failed crawl.
create	Without depth it sends {"url": ...} only and reads depth is None; test_create_sends_depth_when_given sends depth=1.
list, results	list pages with a cursor, and results sends limit. list parses a page that mixes crawls with and without depth.
Transport Retry-After	3600 waits 30 s; an unusable value gives retry_after=None.
content	A caller can pass content_id= by keyword.
stop on 503 crawl_busy	crawl_busy then 200 returns after one 1.0 s wait. With retries=2 and crawl_busy on every call, it raises after 3 calls, waits [1.0, 1.0].
test_stop_retries_transient_status	A 429, 502, 503 (not crawl_busy) or 504 once, then 200, returns after one 2.0 s wait (Retry-After: 2).
test_stop_surfaces_502_after_retries	With retries=3 and 502 on every call, it raises the 502 after 4 calls, waits [1.0, 1.0, 1.0].
test_stop_does_not_retry_other_status	A 500 raises at once.
test_stop_waits_at_most_30_seconds	For 503 crawl_busy and for 429: 3600 waits 30.0 s, -1 and nan wait 1.0 s, 2 waits 2.0 s.
tests/e2e/test_crawl_e2e.py, marked e2e	Skips unless ZENROWS_API_KEY, ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL are set; ZENROWS_E2E_CRAWL_INCLUDE is optional. Passes ZENROWS_CRAWL_BASE_URL as base_url=. DEVELOPMENT.md has a "Running the Crawl e2e test" section.

The e2e test passed against a live deployment before the method renames. It has not run since then, nor since the content_id rename.

How to try it

make check
make test
make test-e2e   # needs the three environment variables above

examples/11_crawl.py runs a crawl end to end. README.md has a Crawl API (new) quickstart, with the client options base_url, retries and timeout.

The acceptance test needs bash, git and Docker. Run it from the repository root:

export ZENROWS_API_KEY="$(cat path/to/key-file)"
bash acceptance/crawl/run.sh              # every check
bash acceptance/crawl/run.sh --no-key     # offline checks only; the live checks are SKIP
bash acceptance/crawl/run.sh --report     # rebuild the report from out/
REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub

Without ZENROWS_API_KEY, run.sh stops and asks for it.

Release

This PR does not change the version. Do a minor bump (1.6.0) in the release commit.