Repository navigation
Description
Summary
For HTTP third-party copies, the flow-id in the fireflies a pool emits does not describe the TCP connection that carries the data:
- The remote address and port are those of the host named in the transfer URL, resolved at the door. They are not taken from the connection the pool opens. If the remote storage redirects (for example another dCache) or its name has several addresses, the firefly names a host that the data does not go to.
- For a push (
PUT), thestartmarker carries the local port of the pool's first connection (to the remote door), while theendmarker carries the local port of the data connection. The two markers of one transfer therefore have different flow-ids. - The copy of each firefly that is sent to the "remote peer" on UDP 10514 goes to that same wrong address.
A network device or collector cannot match such a firefly to the packets of the flow, which is the purpose of the flow-id.
Observed
Pool umfs19_1 at AGLT2, dCache 11.2.10. Fireflies and the pool's outgoing TCP connections were captured together with tcpdump.
Push, AGLT2 pool to MWT2 (gfal-copy --copy-mode push, 536 MB)
Destination URL: davs://webdav.mwt2.org:2881/.... webdav.mwt2.org has nine AAAA records.
| Local port | Remote host | Remote port | |
|---|---|---|---|
Real: first PUT, answered with a redirect |
55138 | uct2-webdav02.mwt2.org |
2881 |
| Real: data connection | 44034 | uct2-s96.mwt2.org (a pool) |
21468 |
start firefly |
55138 | uct2-webdav05.mwt2.org |
2881 |
end firefly |
44034 | uct2-webdav05.mwt2.org |
2881 |
Both fireflies were also sent to uct2-webdav05.mwt2.org:10514, a host that took no part in the transfer.
Pulls in production traffic
In a 200-second capture the pool started two TPC pulls. In both, the firefly's remote endpoint differed from the connection that used the firefly's local port:
start firefly src-ip:src-port |
Real connection from the same local port |
|---|---|
2605:9a00:10:200a:f816:3eff:fe90:55f8:2881 |
2605:9a00:10:200a:8616:cff:fe5c:6bb0:22324 |
2a06:93c1:129:2e0::6c:2880 |
2a06:93c1:129:2e0::74:23618 |
For pulls the local port is right, because the HTTP client follows the redirect before the marker is sent. Only the remote endpoint is wrong.
Cause
Line numbers are for master at 36f093eed2. The 11.2 branch is structured differently but has the same logic.
- The door resolves the URL host once and puts the result into the protocol info:
RemoteTransferHandler.java:1157,new InetSocketAddress(_destination.getHost(), port). - The pool uses that address as the remote endpoint of the
startmarker:
RemoteHttpDataTransferProtocol.startFlowMarker(),ipInfo.getSocketAddress()(line 242). - The
endmarker does the same:DefaultPostTransferService.java:241,ipProtocolInfo.getSocketAddress(). - The local endpoint is taken from the real connection (
_transfer.localAddress()inonConnected()), but thestartmarker is sent only once (_startMarkerSent). In thePUTredirect loop (RemoteHttpTransferClient, around line 599)onConnectedruns for every attempt, so the marker goes out for the first connection, which is the one that gets redirected._localEndpointis then updated, and theendmarker reports the later value.
RemoteHttpTransferClient.remoteAddress() already returns the real remote address of the most recent request. It is used for remoteConnections() but not for the flow markers.
Suggested fix
- Take the remote endpoint of the flow markers from the connection (
RemoteHttpTransferClient.remoteAddress()), as is already done for the local endpoint. - For
PUT, send thestartmarker for the connection that is not redirected, so that it describes the data connection. - Record that pair of endpoints on the mover and use the same pair for the
endmarker. The post-transfer verification request opens a further connection to the door, so the endpoints must be captured when the data connection is made, not read at the end.
With that, the firefly sent to the remote peer would also reach the host that takes part in the flow.
I can prepare a pull request if this approach is acceptable.
Related
- pool: use pool host name as firefly syslog HOSTNAME #8303 (syslog HOSTNAME of fireflies) and pool: fix start-time and end-time of firefly end markers #8304 (times in
endmarkers) concern other fields of the same messages. - Independently of this issue, dCache sets
srcto the remote party anddstto the pool regardless of the direction of the data. That is not what this report is about.
Activity
DmitryLitvintsev commentedon Nov 13, 2026
Hi Shawn ,
check the patch
575397c
that was pushed to upstream yesterday. It mostly deals with Proxied WebDAV HTTPs
Another, related patch is in the review:
01d10cb
I believe it does not solve the problem you report as this has to do with redirected RemoteTrasnferManager managed TPC transfers but mat affect analysis if yours is not bases on the HEAD of the trunk.
ShawnMcKee commentedon Nov 13, 2026
A larger sample from the same pool: 15 minutes of production traffic, 13 TPC pulls started. For each
startfirefly I looked up the TCP connection that used the firefly's local port.Examples:
startfireflysrc-ip:src-port2001:760:4228::9:4432001:760:4228::90:226562401:de00:2:33::13:90002401:de00:2:33::98:90012620:0:210:1::c6:4432620:0:210:1::c4:443The last row is the "address differs, port is the same" case: no redirect to another port, but the host name has several addresses and the pool connected to a different one than the door resolved.
ShawnMcKee commentedon Nov 13, 2026
pool: report the data connection in firefly markers of HTTP-TPC trans…