Skip to content

pool: report the data connection in firefly markers of HTTP-TPC transfers - #8306

Merged
chubnastia merged 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-tpc-flow-id
Nov 19, 2026
Merged

chubnastia merged 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-tpc-flow-id

Conversation

@ShawnMcKee

Copy link
Copy Markdown
Contributor

Fixes #8305.

Problem

For HTTP third-party copies, the flow-id of the firefly markers a pool emits does not describe the TCP connection that carries the data:

  • The remote endpoint is the address the door resolved from the host name in the transfer URL (RemoteTransferHandler, new InetSocketAddress(_destination.getHost(), port)). If the remote server redirects the request, or its name has several addresses, the marker names a host the data does not go to. The copy of the marker for the remote peer is sent to that host as well.
  • For uploads, RemoteHttpTransferClient reported a connection after each request had completed. A redirected request completes at once, so the start marker described the connection to the redirecting server while the end marker described the data connection. Without a redirect, the start marker was only sent after all data had been sent.

Observed on an AGLT2 pool (11.2.10), with fireflies and TCP connections captured together:

Push to a remote dCache Local port Remote Remote port
Real: first PUT, redirected 55138 door A 2881
Real: data connection 44034 a pool 21468
start marker 55138 door B 2881
end marker 44034 door B 2881

In 15 minutes of production traffic, 12 of 13 pulls had a marker whose remote endpoint differed from the connection. Details are in #8305.

Fix

  • RemoteHttpTransferClient: for an upload, onConnected is called just before the first byte of the entity is written, instead of after each request. With expect-continue the entity is not written to a server that redirects the request, so only the data connection is reported. Downloads are unchanged: the callback already runs after the redirects have been followed.
  • RemoteHttpDataTransferProtocol: the start marker takes its remote endpoint from the connection (RemoteHttpTransferClient.remoteAddress()), as it already does for the local endpoint. It falls back to the address in the protocol info.
  • Mover#getRemoteEndpoint and MoverProtocol#getRemoteEndpoint are added, and DefaultPostTransferService uses them for the end marker, so both markers carry the same flow-id.

This follows the approach of the in-review door-side change (commit 01d10cb, "webdav: send SciTags fireflies for transfers proxied by the door"), which takes the endpoints from the connection and marks a push at the first write of the entity. The two changes conflict only in RemoteHttpTransferClientTest, where both add a test at the same place.

Behaviour change to be aware of

An upload that is rejected before any data is sent no longer triggers onConnected. It gets no start marker, the mover reports no local endpoint for it, and its end marker falls back to the derived local endpoint and the protocol-info address, as for other movers without endpoint information.

Testing

  • Two new tests in RemoteHttpTransferClientTest: the connection is reported before the upload completes, and for an upload redirected by a server that does not send "100 Continue" (as a dCache door behaves) only the data connection is reported. Both fail without the change to RemoteHttpTransferClient.
  • Ran the PR pipeline's unit-test job locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5174 tests, 0 failures, 0 errors.
  • Not tested on a pool. The change is against master; the pool I can test on runs 11.2, where this code is structured differently. The mover and DefaultPostTransferService changes are covered only by compilation and the existing tests.

Signed-off-by: Shawn McKee smckee@umich.edu

🤖 Generated with Claude Code

chubnastia pushed a commit that referenced this pull request Nov 19, 2026
Motivation:
A firefly ties a network flow to an experiment and an activity. The
flow-id must therefore name the TCP connection that carries the data.

For xrootd and http(s) transfers that a door redirects to a pool, it
does not (#8307). NettyTransferService sends the start marker when the
mover is created, before the client has connected to the pool, and takes
the remote endpoint from the protocol info. That is the client's
connection to the door. The client then connects to the pool from a
different port, so the marker carries one port from each connection and
matches no flow. The end marker repeats the same endpoint. On a
production pool the client port was wrong in 54 of 55 transfers and
right in none.

Modification:
The start marker is no longer sent from executeMover. NettyMoverChannel
gains connected(remote, local), which the protocol handlers call with
the addresses of the Netty channel once a client has attached to the
mover: HttpPoolRequestHandler for GET and PUT, XrootdPoolRequestHandler
when a file is opened for reading or writing. The first such connection
is recorded on the mover and announced with the start marker.

Mover gains getRemoteEndpoint() (as in #8306) and isPassive().
DefaultPostTransferService uses the recorded remote endpoint for the end
marker, so both markers of a transfer carry the same flow-id. For a
passive mover that no client connected to, no end marker is sent, as
there was no flow and no start marker.

Not covered: when a pool is the destination of an xrootd third-party
copy, the file arrives over a connection that the pool's third-party
client (TpcWriteDescriptor) opens to the source. The requesting client's
connection carries no file data and is not reported, so no marker is
sent for such transfers until the third-party client reports its own
connection.

Result:
Fireflies for xrootd and http(s) client transfers name the connection
that carries the data. The start marker is sent when the client has
connected to the pool instead of before. Movers that no client connects
to emit no markers.

Testing:
- New NettyTransferServiceFlowMarkerTest.
- PR pipeline unit-test job run locally in
  maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify
  -DskipTests=false -Drun.slow.tests): 5178 tests, 0 failures.
- Backported to 11.2.10 and run on a production pool. TCP connections
  to the mover ports were captured for 10 minutes and compared with the
  endpoints the pool logged. Before: client port right in 0 of 55
  transfers. After: right in 65 of 65 (64 xrootd, 1 https). Start and
  end markers carried the same flow-id in 67 of 67 transfers.

Fixes: #8307
Acked-by:
Target: master
Require-book: no
Require-notes: yes

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Shawn McKee <smckee@umich.edu>
@chubnastia

Copy link
Copy Markdown
Contributor

Hi, Shawn! Thanks, this looks good! A few small things before we merge:

  • doGet: after a 202 retry the data arrives on a new connection, but onConnected has already reported the first one. Could you pass the context to the retry and call onConnected after the loop?
  • On PUT, onConnected can run twice (if the body is resent after a redirect) and overwrite the endpoints after the start marker was sent, so start and end end up with different flow-ids. Maybe record them only once?
  • It would be good to make _localEndpoint/_remoteEndpoint volatile.
  • A pull that fails before connecting currently sends an end marker with port 0 and no start. Could we skip that, similar to the isPassive() check in pool: take firefly flow-id of client transfers from the data connection #8308?
  • There's a small import conflict in RemoteHttpTransferClientTest after rebasing.

…fers

Motivation:
For HTTP third-party copies the flow-id of the firefly markers does not
describe the connection that carries the data (dCache#8305):

- The remote endpoint is the address the door resolved from the host
  name in the transfer URL. If the remote server redirects the request,
  or its name has several addresses, the marker names a host the data
  does not go to. The marker is also sent to that host.

- For uploads, the transfer client reported a connection after each
  request had completed. A request that is redirected completes at
  once, so the start marker described the connection to the redirecting
  server, while the end marker described the data connection. If the
  request was not redirected, the start marker was only sent once all
  data had been sent.

Observed on a pool: a push to a remote dCache produced markers naming
<door address>:2881, with local port 55138 in the start marker and
44034 in the end marker, while the data went from local port 44034 to
<pool address>:21468. Of 13 pulls in production traffic, 12 had a
marker whose remote endpoint differed from the connection.

Modification:
RemoteHttpTransferClient reports the connection of an upload just
before the first byte of the entity is written, instead of after each
request. With expect-continue the entity is not written to a server
that redirects the request, so only the data connection is reported.

For a download it reports the connection after the retry loop, and the
retried request uses the same context: while a server answers 202 the
connection may be closed, and the data then arrives on a new one.

The connection is reported once. If a server asks for a redirection
only after it has received the file, the file is sent again; the second
connection is not reported, so that start and end markers agree. It
is logged at debug level by the org.dcache.scitags logger
(event=connection-unreported), so that a site can tell whether this
happens.

RemoteHttpDataTransferProtocol takes the remote endpoint of the start
marker from that connection, as it already does for the local endpoint,
and falls back to the address in the protocol info. Both endpoint
fields are volatile.

Add Mover#getRemoteEndpoint and MoverProtocol#getRemoteEndpoint.
DefaultPostTransferService uses it for the end marker, so that start
and end markers carry the same flow-id.

Add Mover#isActive and MoverProtocol#isActive for a mover that connects
to the remote party itself and reports that connection.
DefaultPostTransferService sends no end marker for such a mover if it
never connected, as it already does for a passive mover nobody
connected to. Before, a pull that failed on the DNS lookup or the TLS
handshake sent an end marker with local port 0 and no start marker.

Result:
The firefly markers of an HTTP-TPC transfer describe the TCP connection
that carries the data, for pulls and pushes, also when the remote
server redirects the request, makes the pool retry the request, or its
name has several addresses. The start marker of a push is sent when the
upload starts. A transfer that fails before the pool has a connection
to report, for example on the DNS lookup or the TLS handshake, sends
no markers.

Fixes: dCache#8305
Acked-by:
Target: master
Require-book: no
Require-notes: yes

Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ShawnMcKee

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Rebased onto current master and updated (5c37b8329d). Point by point:

  1. GET after a 202 retry. The retried request now uses the same context, and the connection is reported after the retry loop, so the endpoints are those of the connection the data arrives on. New test shouldReportConnectionOfRetriedDownload: the server answers 202 with Connection: close, then 200 on a new connection, and only the second is reported. It takes 5 s because DELAY_BETWEEN_REQUESTS is fixed.
  2. PUT body sent again after a redirect. The client reports the connection once (reportConnection() with an AtomicBoolean), so the endpoints cannot change after the start marker. Start and end then name the first connection, the one to the server that asked for the redirect only after receiving data. The second connection is logged at debug level by the org.dcache.scitags logger (event=connection-unreported reason=data-sent-again), so a site can see whether this happens; with the 5-minute EXPECT_100_TIMEOUT it should be rare. New test shouldReportConnectionOnlyOnceWhenUploadIsSentAgain.
  3. volatile. _localEndpoint and _remoteEndpoint are volatile.
  4. Pull that fails before connecting. Added Mover#isActive / MoverProtocol#isActive, the counterpart of isPassive(): a mover that connects to the remote party itself and reports that connection. DefaultPostTransferService sends no end marker for an active mover without a local endpoint, next to the existing check for a passive mover nobody connected to. New DefaultPostTransferServiceTest covers both. We saw this case eight times in 20 hours on two AGLT2 pools (four DNS failures, four rejected certificates).
  5. Import conflict in RemoteHttpTransferClientTest: resolved in the rebase.

The two new client tests fail on the previous version of this PR and pass now. The PR pipeline's unit-test job, run locally in maven:3.9.12-eclipse-temurin-21 on this commit: 5206 tests, 0 failures, 0 errors.

@ShawnMcKee
ShawnMcKee force-pushed the scitags/firefly-tpc-flow-id branch from d13302f to 5c37b83 Compare November 19, 2026 13:18
@chubnastia
chubnastia merged commit 46bdae0 into dCache:master Nov 19, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pool: firefly flow-id for HTTP-TPC does not match the data connection

2 participants