Playwright protocol applies neither the IP filter nor the URL filters to the requests it makes

[rzo1]

What happens

IPFilterRules is only used by the okhttp protocol, where it runs as a network interceptor. The playwright protocol never constructs it, so http.filter.ipaddress.include and http.filter.ipaddress.exclude have no effect on a topology that fetches with playwright. The module already installs a route handler that sees every request the browser makes, but that handler only aborts resource types listed in playwright.skip.resource.types and otherwise resumes. This covers the navigation itself, its redirect hops, and every subresource, XHR, iframe and script request the rendered page issues.

Where

external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:243-256.

page.route( lambdaUrl -> true, route -> { // abort if we know the main page is a redirection if (status.get() != -1) { ... } else if (resourceTypesToSkip.contains(route.request().resourceType())) { route.abort(); } else { route.resume(); } });

The IP filter itself lives in core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java and is wired up only in core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java.

Why it matters

This is documented: crawler-default.yaml:161-162 and docs/src/main/asciidoc/configuration.adoc:259-260 both say the IP filter is okhttp only, and both keys ship commented out, so nobody was told something untrue. The gap is still worth closing. A crawler that renders pages in a browser makes far more requests than one that fetches bytes, and every one of them leaves the worker with no address check, including requests the fetched page decides to make. An operator who moves a topology from okhttp to playwright loses the control silently, because the keys stay in the configuration and stop doing anything.

Reproduction

No automated test. configure() launches or connects to a real Chromium, so exercising the route handler needs a browser plus an HTTP server bound to a filtered address.

Manual steps:

1. Configure a topology with the playwright protocol and set http.filter.ipaddress.exclude: "localhost,sitelocal".
2. Run a small HTTP server on the loopback interface and fetch its URL through the protocol. It is fetched, whereas the okhttp protocol fails the fetch with an IOException.
3. Serve a page whose body requests a subresource from a loopback URL. The request is issued by the browser and the route handler resumes it.

Suggested fix

Build an IPFilterRules instance in HttpProtocol.configure() from the same configuration keys, and apply it inside the existing page.route handler: resolve the request host and call route.abort() when the address is rejected. Apply URLFilters in the same place so redirect hops and subresource requests are checked against the crawl scope. Update docs/src/main/asciidoc/configuration.adoc so the "OkHttp only" note reflects the new state. Resolution in the handler adds a lookup per request, so cache results per host. A separate DNS lookup is also not the address the browser ends up connecting to, which is worth stating in the documentation.