Skip to content

Honour ignoreOutsideDomain when set on its own in HostURLFilter - #2074

Open
rzo1 wants to merge 1 commit into
mainfrom
fix/hosturlfilter-domain-config
Open

Honour ignoreOutsideDomain when set on its own in HostURLFilter#2074
rzo1 wants to merge 1 commit into
mainfrom
fix/hosturlfilter-domain-config

Conversation

@rzo1

@rzo1 rzo1 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

HostURLFilter.configure() read the ignoreOutsideDomain parameter but tested the ignoreOutsideHost node for null. A configuration naming only ignoreOutsideDomain was silently discarded and the filter let every URL through; one naming only ignoreOutsideHost threw a NullPointerException at topology startup. Each parameter is now tested on its own.

Topologies setting only ignoreOutsideDomain will start restricting outlinks to the source domain, as configured. Configurations setting both keys, including all three archetypes, are unaffected.

For all changes

  • Is there a issue associated with this PR? Is it referenced in the commit message? - no issue

  • Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve? - no issue

  • Has your PR been rebased against the latest commit within the target branch (typically main)?

  • Is your initial contribution a single, squashed commit?

  • Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

  • Have you ensured that the full suite of tests is executed via mvn clean verify? - core module tests only (348 run, all green)
  • Have you written or updated unit tests to verify your changes? - two new cases in HostURLFilterTest, both fail without the fix
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? - no new dependencies
  • If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a
  • If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a

@rzo1 rzo1 added this to the 4.0.0 milestone Aug 27, 2026
HostURLFilter.configure() read the ignoreOutsideDomain parameter but tested
the ignoreOutsideHost node for null. A configuration naming only
ignoreOutsideDomain was therefore discarded and the filter let every URL
through, while one naming only ignoreOutsideHost failed with a
NullPointerException while the topology was starting. Now each parameter is
tested independently. Topologies which set only ignoreOutsideDomain start
restricting the crawl to the source domain, as configured.
@rzo1
rzo1 force-pushed the fix/hosturlfilter-domain-config branch from 47f3163 to 30135b0 Compare August 27, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants