Version: 0.9.2 (Docker, unclecode/crawl4ai:0.9.2, tag digest verified current)
What happens
POST /crawl with crawler_config.wait_for set to a CSS selector that never matches the page returns an opaque HTTP 500 instead of a per-URL error result:
{"error": "Internal server error", "correlation_id": "..."}Repro
curl -s -X POST http://localhost:8090/crawl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <token>' \
-d '{"urls":["https://www.reddit.com/r/selfhosted/"],"crawler_config":{"wait_until":"networkidle","wait_for":"css:.quote"}}'(.quote exists on quotes.toscrape.com, not on Reddit.)
Results (all against the same 0.9.2 container):
| URL |
wait_for |
Result |
| quotes.toscrape.com/js/ |
css:.quote (matches) |
HTTP 200, content OK |
| www.reddit.com/r/selfhosted/ |
css:.quote (no match) |
HTTP 500, opaque error |
| api-docs.deepseek.com |
css:article (no match at check time) |
HTTP 500, opaque error |
| www.reddit.com/r/selfhosted/ |
css:shreddit-post (matches) |
HTTP 200, content OK |
So this is a crash on an unmatched selector, not a general wait_for failure.
Related observations on the same endpoint (0.9.2)
- Bad
wait_until enum value (e.g. "not-a-real-value") → HTTP 500, not a 422 validation error.
- Unknown fields inside
crawler_config (e.g. bogus_param_xyz) → silently ignored (HTTP 200); the schema accepts arbitrary keys without complaint.
Root-cause hypothesis
Same symptom family as #2116 / #2117 — the Docker API promotes unsuccessful crawls to HTTP 500 and _http_exception_handler swallows the detail. #2117 addresses that for the single-URL /md and /llm/{url} handlers; /crawl appears to have the same failure mode but is not covered. Note: the 500s return within a few seconds, which looks like an immediate exception when the selector never appears rather than a page_timeout wait — worth confirming whether it is the same if not result.success: raise HTTPException(500) path or a separate crash in the wait_for handling.
Expected behavior
An unmatched wait_for should surface as a per-URL error result (or a 4xx with the reason preserved), not a generic 500 — the same treatment PR #2117 proposes for the single-URL handlers.
Investigation and repro produced with AI assistance (automated agent); reviewed and filed by a human.
Version: 0.9.2 (Docker,
unclecode/crawl4ai:0.9.2, tag digest verified current)What happens
POST /crawlwithcrawler_config.wait_forset to a CSS selector that never matches the page returns an opaque HTTP 500 instead of a per-URL error result:{"error": "Internal server error", "correlation_id": "..."}Repro
(
.quoteexists on quotes.toscrape.com, not on Reddit.)Results (all against the same 0.9.2 container):
css:.quote(matches)css:.quote(no match)css:article(no match at check time)css:shreddit-post(matches)So this is a crash on an unmatched selector, not a general wait_for failure.
Related observations on the same endpoint (0.9.2)
wait_untilenum value (e.g."not-a-real-value") → HTTP 500, not a 422 validation error.crawler_config(e.g.bogus_param_xyz) → silently ignored (HTTP 200); the schema accepts arbitrary keys without complaint.Root-cause hypothesis
Same symptom family as #2116 / #2117 — the Docker API promotes unsuccessful crawls to HTTP 500 and
_http_exception_handlerswallows the detail. #2117 addresses that for the single-URL/mdand/llm/{url}handlers;/crawlappears to have the same failure mode but is not covered. Note: the 500s return within a few seconds, which looks like an immediate exception when the selector never appears rather than apage_timeoutwait — worth confirming whether it is the sameif not result.success: raise HTTPException(500)path or a separate crash in the wait_for handling.Expected behavior
An unmatched
wait_forshould surface as a per-URL error result (or a 4xx with the reason preserved), not a generic 500 — the same treatment PR #2117 proposes for the single-URL handlers.Investigation and repro produced with AI assistance (automated agent); reviewed and filed by a human.