Skip to content

pool: firefly flow-id for HTTP-TPC does not match the data connection #8305

Closed

Description

@ShawnMcKee

Summary

For HTTP third-party copies, the flow-id in the fireflies a pool emits does not describe the TCP connection that carries the data:

  1. The remote address and port are those of the host named in the transfer URL, resolved at the door. They are not taken from the connection the pool opens. If the remote storage redirects (for example another dCache) or its name has several addresses, the firefly names a host that the data does not go to.
  2. For a push (PUT), the start marker carries the local port of the pool's first connection (to the remote door), while the end marker carries the local port of the data connection. The two markers of one transfer therefore have different flow-ids.
  3. The copy of each firefly that is sent to the "remote peer" on UDP 10514 goes to that same wrong address.

A network device or collector cannot match such a firefly to the packets of the flow, which is the purpose of the flow-id.

Observed

Pool umfs19_1 at AGLT2, dCache 11.2.10. Fireflies and the pool's outgoing TCP connections were captured together with tcpdump.

Push, AGLT2 pool to MWT2 (gfal-copy --copy-mode push, 536 MB)

Destination URL: davs://webdav.mwt2.org:2881/.... webdav.mwt2.org has nine AAAA records.

Local port Remote host Remote port
Real: first PUT, answered with a redirect 55138 uct2-webdav02.mwt2.org 2881
Real: data connection 44034 uct2-s96.mwt2.org (a pool) 21468
start firefly 55138 uct2-webdav05.mwt2.org 2881
end firefly 44034 uct2-webdav05.mwt2.org 2881

Both fireflies were also sent to uct2-webdav05.mwt2.org:10514, a host that took no part in the transfer.

Pulls in production traffic

In a 200-second capture the pool started two TPC pulls. In both, the firefly's remote endpoint differed from the connection that used the firefly's local port:

start firefly src-ip:src-port Real connection from the same local port
2605:9a00:10:200a:f816:3eff:fe90:55f8:2881 2605:9a00:10:200a:8616:cff:fe5c:6bb0:22324
2a06:93c1:129:2e0::6c:2880 2a06:93c1:129:2e0::74:23618

For pulls the local port is right, because the HTTP client follows the redirect before the marker is sent. Only the remote endpoint is wrong.

Cause

Line numbers are for master at 36f093eed2. The 11.2 branch is structured differently but has the same logic.

  • The door resolves the URL host once and puts the result into the protocol info:
    RemoteTransferHandler.java:1157, new InetSocketAddress(_destination.getHost(), port).
  • The pool uses that address as the remote endpoint of the start marker:
    RemoteHttpDataTransferProtocol.startFlowMarker(), ipInfo.getSocketAddress() (line 242).
  • The end marker does the same: DefaultPostTransferService.java:241, ipProtocolInfo.getSocketAddress().
  • The local endpoint is taken from the real connection (_transfer.localAddress() in onConnected()), but the start marker is sent only once (_startMarkerSent). In the PUT redirect loop (RemoteHttpTransferClient, around line 599) onConnected runs for every attempt, so the marker goes out for the first connection, which is the one that gets redirected. _localEndpoint is then updated, and the end marker reports the later value.

RemoteHttpTransferClient.remoteAddress() already returns the real remote address of the most recent request. It is used for remoteConnections() but not for the flow markers.

Suggested fix

  • Take the remote endpoint of the flow markers from the connection (RemoteHttpTransferClient.remoteAddress()), as is already done for the local endpoint.
  • For PUT, send the start marker for the connection that is not redirected, so that it describes the data connection.
  • Record that pair of endpoints on the mover and use the same pair for the end marker. The post-transfer verification request opens a further connection to the door, so the endpoints must be captured when the data connection is made, not read at the end.

With that, the firefly sent to the remote peer would also reach the host that takes part in the flow.

I can prepare a pull request if this approach is acceptable.

Related

Activity

  1. DmitryLitvintsev commented on Nov 13, 2026

    @DmitryLitvintsev
    Member

    Hi Shawn ,

    check the patch
    575397c
    that was pushed to upstream yesterday. It mostly deals with Proxied WebDAV HTTPs
    Another, related patch is in the review:
    01d10cb

    I believe it does not solve the problem you report as this has to do with redirected RemoteTrasnferManager managed TPC transfers but mat affect analysis if yours is not bases on the HEAD of the trunk.

  2. ShawnMcKee commented on Nov 13, 2026

    @ShawnMcKee
    ContributorAuthor

    A larger sample from the same pool: 15 minutes of production traffic, 13 TPC pulls started. For each start firefly I looked up the TCP connection that used the firefly's local port.

    Firefly remote endpoint vs real connection Pulls
    address and port both differ 10
    address differs, port is the same 2
    both match 1

    Examples:

    start firefly src-ip:src-port Real connection
    2001:760:4228::9:443 2001:760:4228::90:22656
    2401:de00:2:33::13:9000 2401:de00:2:33::98:9001
    2620:0:210:1::c6:443 2620:0:210:1::c4:443

    The last row is the "address differs, port is the same" case: no redirect to another port, but the host name has several addresses and the pool connected to a different one than the door resolved.

  3. ShawnMcKee commented on Nov 13, 2026

    @ShawnMcKee
    ContributorAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions