Politeness queues and the robots.txt cache key on the raw host string, so percent-escaped host aliases get their own queue and their own robots.txt fetch

[rzo1]

What happens

FetchItem.create uses u.getHost() as the politeness queue id and HttpRobotRulesParser.getCacheKey builds protocol:host:port from the same string. Both only lowercase it. okhttp percent-decodes the host when it parses the URL, so http://exampl%65.org/ and http://example.org/ are one origin at connect time but two queue ids and two robots.txt cache entries in the bolt. BasicURLNormalizer does not close the gap either: it lowercases the host but leaves percent-escapes in place.

Where

core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java:189 and :197 (the same lowercasing at :170, and SimpleFetcherBolt.java:628 and :631), core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:69-81, core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLNormalizer.java:136. Config keys involved: fetcher.queue.mode (default byHost), fetcher.server.delay, fetcher.threads.number.

    key = u.getHost();
...
    queueId = key.toLowerCase(Locale.ROOT);

    String host = url.getHost().toLowerCase(Locale.ROOT);
    return protocol + ":" + host + ":" + port;

Why it matters

One server can be addressed under many host spellings that all resolve to the same origin, and each spelling gets its own delay clock and its own robots.txt download. The per-host delay is then applied several times in parallel to the same server, bounded by fetcher.threads.number (10 by default), not by the number of aliases. Robots rules are still obeyed, since every alias re-fetches and re-parses the same file, so this is wasted requests and a politeness overrun on the remote side rather than a rules bypass. A crawl that follows outlinks can pick these up from any page that links to the aliased forms.

Reproduction

Save as core/src/test/java/org/apache/stormcrawler/protocol/HostAliasCacheKeyTest.java.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.stormcrawler.protocol;

import java.net.URL;
import okhttp3.HttpUrl;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

/**
 * Two URLs whose host strings differ only by percent-escaping or by a trailing dot are sent to the
 * same server by okhttp, so they must share one robots.txt cache entry.
 */
class HostAliasCacheKeyTest {

    @Test
    void okhttpCollapsesHostAliases() {
        // what the client actually connects to
        Assertions.assertEquals(
                "example.org", HttpUrl.parse("http://%65xample.org/a").host(), "percent-escaped");
        Assertions.assertEquals(
                "example.org", HttpUrl.parse("http://exampl%65.org/a").host(), "percent-escaped");
        Assertions.assertEquals(
                "example.org", HttpUrl.parse("http://EXAMPLE.org/a").host(), "upper case");
    }

    @Test
    void robotsCacheKeyIsTheSameForHostAliases() throws Exception {
        String canonical = HttpRobotRulesParser.getCacheKey(new URL("http://example.org/a"));
        String escaped = HttpRobotRulesParser.getCacheKey(new URL("http://exampl%65.org/a"));
        Assertions.assertEquals(canonical, escaped);
    }
}

Run it:

mvn -pl core test -Dtest=HostAliasCacheKeyTest

okhttpCollapsesHostAliases passes and shows what the client connects to; robotsCacheKeyIsTheSameForHostAliases fails on main.

[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.151 s <<< FAILURE! -- in org.apache.stormcrawler.protocol.HostAliasCacheKeyTest
[ERROR] org.apache.stormcrawler.protocol.HostAliasCacheKeyTest.robotsCacheKeyIsTheSameForHostAliases -- Time elapsed: 0.018 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <http:example.org:80> but was: <http:exampl%65.org:80>

Suggested fix

Add one host canonicalisation helper in URLUtil that percent-decodes the host, lowercases it and strips a trailing dot, and call it from FetchItem.create, SimpleFetcherBolt.getPolitenessKey and HttpRobotRulesParser.getCacheKey so all three agree with what okhttp will connect to. Alternatively reject URLs whose host contains a percent escape in BasicURLNormalizer.filter, which is stricter and drops such URLs from the crawl entirely; that changes what an existing crawl accepts, so it needs a release note. The first option is the smaller change and does not alter which URLs are crawled, only how they are grouped.

[abhinav-phi]

Hi! I'd like to take this up.
Plan: I'll normalize percent-escaped hosts before keying the politeness queues and the robots.txt cache, so the same host doesn't get treated as different hosts — with regression tests.
/assign take