IPFilterRules has no keyword for link-local addresses and the documented example leaves them allowed

[rzo1]

What happens

IPFilterRules.parseIPRules() understands three keywords: localhost and loopback map to InetAddress::isLoopbackAddress, and sitelocal maps to InetAddress::isSiteLocalAddress. For IPv4, isSiteLocalAddress covers 10/8, 172.16/12 and 192.168/16 only. Addresses in 169.254.0.0/16, 100.64.0.0/10 and 0.0.0.0/8, and IPv6 addresses in fd00::/8 and fe80::/10, match none of the keywords, so the example shipped in crawler-default.yaml, "localhost,sitelocal", still allows them. Anything else is parsed as a CIDR block or a single address, and a value that parses as neither is dropped with a LOG.error, so a typo in an exclude list removes a rule the operator thought was in force.

Where

core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java:111-129, keys http.filter.ipaddress.include and http.filter.ipaddress.exclude. The example is at core/src/main/resources/crawler-default.yaml:162 and the same keyword list is in the class javadoc at lines 45-52.

case "sitelocal":
    rules.add(InetAddress::isSiteLocalAddress);
    break;

Why it matters

An operator who turns the filter on and copies the shipped example gets loopback and RFC1918 blocked and nothing else. On a cloud worker the link-local range is the one that carries an unauthenticated instance metadata service, and it is reachable from the worker whatever DNS name the crawled link used, because the check runs on the resolved socket address. The ranges can be written today as plain CIDR rules, so this is about the vocabulary and the example being incomplete rather than a missing capability. The interceptor is also skipped when a proxy is configured, as its javadoc notes.

Reproduction

Save as core/src/test/java/org/apache/stormcrawler/protocol/IPFilterRulesKeywordsTest.java.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.stormcrawler.protocol;

import static org.junit.jupiter.api.Assertions.assertFalse;

import com.google.common.net.InetAddresses;
import java.net.InetAddress;
import java.util.HashMap;
import java.util.Map;
import org.junit.jupiter.api.Test;

/** The rule keywords and the example shipped in crawler-default.yaml. */
class IPFilterRulesKeywordsTest {

    private static InetAddress ip(String address) {
        return InetAddresses.forString(address);
    }

    private static IPFilterRules exclude(Object exclude) {
        Map<String, Object> conf = new HashMap<>();
        conf.put(IPFilterRules.EXCLUDE_RULES_KEY, exclude);
        return new IPFilterRules(conf);
    }

    /** The example from crawler-default.yaml. */
    @Test
    void documentedExampleBlocksTheNonRoutableRanges() {
        IPFilterRules r = exclude("localhost,sitelocal");
        assertFalse(r.accept(ip("169.254.169.254")), "IPv4 link-local");
        assertFalse(r.accept(ip("100.64.0.1")), "CGNAT");
        assertFalse(r.accept(ip("fd00::1")), "IPv6 unique local");
        assertFalse(r.accept(ip("fe80::1")), "IPv6 link-local");
    }

    /** There is no keyword for link-local addresses. */
    @Test
    void linkLocalKeywordIsSupported() {
        IPFilterRules r = exclude("linklocal");
        assertFalse(r.accept(ip("169.254.169.254")));
        assertFalse(r.accept(ip("fe80::1")));
    }
}

Run it:

mvn -pl core test -Dtest=IPFilterRulesKeywordsTest

Both tests fail on main. It sits next to the existing IPFilterRulesTest.

[main] ERROR org.apache.stormcrawler.protocol.IPFilterRules - Failed to parse linklocal as CIDR,
    ignoring it while configuring IP rules (http.filter.ipaddress.exclude)
[ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
[ERROR]   IPFilterRulesKeywordsTest.documentedExampleBlocksTheNonRoutableRanges:45
    IPv4 link-local ==> expected: <false> but was: <true>
[ERROR]   IPFilterRulesKeywordsTest.linkLocalKeywordIsSupported:55 expected: <false> but was: <true>

Suggested fix

Add a linklocal keyword mapping to InetAddress::isLinkLocalAddress in IPFilterRules.parseIPRules, and consider anylocal and multicast alongside it. Update the class javadoc and the commented example in crawler-default.yaml to "loopback,sitelocal,linklocal,100.64.0.0/10,fd00::/8". Change the unparseable-rule branch to throw instead of logging, so a mistyped exclude rule stops the topology rather than silently widening what it will connect to; that is a behaviour change for anyone who has a bad rule in their config today, and the existing invalidRuleIsIgnored test in IPFilterRulesTest needs to be updated with it.

[abhinav-phi]

Hi! I'd like to work on this.
Plan: I'll add a keyword for link-local addresses to IPFilterRules, fix the documented example so it doesn't leave those addresses allowed, and add tests for link-local ranges.
/assign take