Repository navigation
Description
What is wrong
A firefly ties a WAN flow to its owner (the experiment or virtual organization) and to an activity. It has two halves:
- The
contextblock says who and what:experiment-idandactivity-id. - The
flow-idsays which flow: the real source address and port, destination address and port, and protocol of the TCP connection that carries the file data.
Routers, collectors and marking agents use the five flow-id values to find the packets that the owner and activity apply to. If one of them belongs to a different connection, the owner and activity are attached to no packets, or to the wrong ones.
This report is about the flow-id half. The experiment and activity values are not what it is about.
A client transfer through a door involves two connections. Only the second carries the file:
| Connection | Client end | dCache end | What it carries |
|---|---|---|---|
| 1. Control | client address, port B | door, port 1094 (xrootd) or 2880 (WebDAV) | Open request and the redirect to the pool. No file data. |
| 2. Data | client address, port D | pool, port C (the mover port) | The file. This is the flow the firefly is about. |
The pool's firefly takes one end from each:
| Client end | Pool end | |
|---|---|---|
| A correct firefly names connection 2 | client address, port D | pool address, port C |
| dCache sends | the address the door saw, port B | pool address, port C |
A real example from our pool (https, 14 November 2026): the firefly said client port 54964. The data arrived from client port 47742, on pool port 47548. Port 54964 was the client's connection to the door.
So the pair of ports in the firefly never existed as a connection. In our measurement below, the client port was wrong in 54 of 55 transfers and right in none.
Details that follow from the same cause:
- The client address is also the one the door saw. Nothing guarantees that the client reaches the pool from the same address (dual-stack client, NAT, multi-homed host). This follows from the code; I have not observed a case.
- The
startmarker is sent before the client has connected to the pool, so the pool cannot know port D at that moment. It is sent whether or not the client ever connects. - Port C is correct, but it is the mover's listening port, shared by all transfers of that protocol on the pool. Only port D distinguishes one transfer from another between the same two hosts, and port D is the value that is missing.
The flowd-go developers ran into this on the FNAL testbed while preparing the SC26 demo: for an xrdcp transfer, the ports in the firefly did not line up with the ones xrdcp -d 3 reported, and the firefly carried one port from each of the client's two connections. flowd-go on the pool host could not match the firefly to the flow it had to mark.
This is the client-transfer counterpart of #8305, which covers HTTP-TPC. There, too, the firefly names a host and port from the pre-transfer exchange with a door instead of the data connection.
Observed
Pools umfs19_1 and umfs19_2 at AGLT2, dCache 11.2.10 plus the change of #8303 (which does not touch the flow-id), 14 November 2026.
For 10 minutes, the TCP connections arriving at the pools' mover ports were captured with tcpdump (SYN packets only). They were compared with the endpoints the pools logged for the start marker of each transfer (org.dcache.scitags at debug level). For on-site transfers the marker is suppressed by pool.firefly.excludes, but the same endpoints are logged, so those transfers were used as well.
| xrootd | https | Total | |
|---|---|---|---|
| Client transfers started | 53 | 2 | 55 |
| Logged client port equals the port of the real connection | 0 | 0 | 0 |
| Logged client port differs from the real connection | 52 | 2 | 54 |
| No connection from the logged address within 30 s | 1 | 0 | 1 |
Of the 55 transfers, 54 were on-site (marker suppressed) and one was external (marker sent).
Examples:
| Protocol | Client | Client port in the marker | Client port of the real connection | Pool port |
|---|---|---|---|---|
| https (marker sent) | 2001:48a8:68f7:8001:192:41:237:92 |
54964 | 47742 | 47548 |
| https | 2001:48a8:68f7:1:192:41:230:169 |
56226 | 51084 | 47548 |
| xrootd | 2001:48a8:68f7:1:192:41:231:176 |
53236 | 36076 | 33625 |
In all 54 matched cases the client's connection arrived within the same second as the marker was logged.
All 53 xrootd transfers used one port per pool (44402 on umfs19_1, 33625 on umfs19_2).
Cause
Line numbers are for master at 36f093eed2. 11.2.10 has the same code.
- The door puts the client's address, as the door sees it, into the protocol info:
XrootdTransfer.createXrootdProtocolInfo()passesgetClientAddress()(line 145), and
DcacheResourceFactory.createProtocolInfo(address)does the same for WebDAV (line 1966). - The pool sends the
startmarker inNettyTransferService.executeMover()(lines 380 to 388). The remote endpoint is((IpProtocolInfo) mover.getProtocolInfo()).getSocketAddress(), and the local endpoint is built fromgetServerAddress().getPort(). Only after that does it callsendAddressToDoor(), which is what allows the door to redirect the client to the pool. So the marker is always sent before the data connection exists. - The
endmarker uses the same remote endpoint:DefaultPostTransferService.java:241,ipProtocolInfo.getSocketAddress(). - The pool sees the real connection when the client presents the mover's UUID:
XrootdPoolRequestHandler(line 402) andHttpPoolRequestHandler(line 748) call_server.openFile(uuid, ...). At that point the Netty channel has the real remote and local addresses. They are not used for the flow markers.
Suggested fix
The rule to implement: the flow-id of both markers is read from the socket that carries the data, never from what a door reported.
- Send the
startmarker when a client attaches to the mover (theopenFilecall), with both endpoints taken from the Netty channel. - Record that pair of endpoints on the mover and use the same pair for the
endmarker, so that both markers of a transfer carry the same flow-id. - The copy of each firefly that is sent to the remote peer on UDP 10514 would then go to the address the client really uses.
Points to decide:
- Timing. Today the
startfirefly is sent before the first packet of the flow, but with ports that match no connection. With this change it is sent when the client has connected, with the right ports, so the first packets of the connection precede it. A correct flow-id matters more than an early one: the client's port cannot be known before the client connects. - Several connections to one mover. If more than one connection can attach to the same mover (reconnects, parallel requests), it has to be decided whether each one gets its own pair of markers.
- Movers that no client connects to. With this change they would emit no markers. Today they emit a
startmarker for a flow that never comes into existence.
I can prepare a pull request if this approach is acceptable.
Related
- pool: firefly flow-id for HTTP-TPC does not match the data connection #8305 and pool: report the data connection in firefly markers of HTTP-TPC transfers #8306: the same problem for HTTP-TPC, where the pool opens the connection.
- pool: use pool host name as firefly syslog HOSTNAME #8303 (syslog HOSTNAME of fireflies) and pool: fix start-time and end-time of firefly end markers #8304 (times in
endmarkers) concern other fields of the same messages. - The WebDAV door change in review (Review Board 14842) makes the door send the fireflies for transfers it proxies. There the door owns the connection to the remote site and takes the endpoints from it, so this problem does not arise.
- Independently of this issue, dCache sets
srcto the remote party anddstto the pool regardless of the direction of the data. That is not what this report is about.
Activity
xrootd: take firefly flow-id of third-party copies from the source co…