Repository navigation
pool: report the data connection in firefly markers of HTTP-TPC transfers - #8306
Merged
Merged
Conversation
This was referenced Nov 14, 2026
chubnastia
pushed a commit
that referenced
this pull request
Nov 19, 2026
Motivation: A firefly ties a network flow to an experiment and an activity. The flow-id must therefore name the TCP connection that carries the data. For xrootd and http(s) transfers that a door redirects to a pool, it does not (#8307). NettyTransferService sends the start marker when the mover is created, before the client has connected to the pool, and takes the remote endpoint from the protocol info. That is the client's connection to the door. The client then connects to the pool from a different port, so the marker carries one port from each connection and matches no flow. The end marker repeats the same endpoint. On a production pool the client port was wrong in 54 of 55 transfers and right in none. Modification: The start marker is no longer sent from executeMover. NettyMoverChannel gains connected(remote, local), which the protocol handlers call with the addresses of the Netty channel once a client has attached to the mover: HttpPoolRequestHandler for GET and PUT, XrootdPoolRequestHandler when a file is opened for reading or writing. The first such connection is recorded on the mover and announced with the start marker. Mover gains getRemoteEndpoint() (as in #8306) and isPassive(). DefaultPostTransferService uses the recorded remote endpoint for the end marker, so both markers of a transfer carry the same flow-id. For a passive mover that no client connected to, no end marker is sent, as there was no flow and no start marker. Not covered: when a pool is the destination of an xrootd third-party copy, the file arrives over a connection that the pool's third-party client (TpcWriteDescriptor) opens to the source. The requesting client's connection carries no file data and is not reported, so no marker is sent for such transfers until the third-party client reports its own connection. Result: Fireflies for xrootd and http(s) client transfers name the connection that carries the data. The start marker is sent when the client has connected to the pool instead of before. Movers that no client connects to emit no markers. Testing: - New NettyTransferServiceFlowMarkerTest. - PR pipeline unit-test job run locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5178 tests, 0 failures. - Backported to 11.2.10 and run on a production pool. TCP connections to the mover ports were captured for 10 minutes and compared with the endpoints the pool logged. Before: client port right in 0 of 55 transfers. After: right in 65 of 65 (64 xrootd, 1 https). Start and end markers carried the same flow-id in 67 of 67 transfers. Fixes: #8307 Acked-by: Target: master Require-book: no Require-notes: yes Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Shawn McKee <smckee@umich.edu>
Contributor
|
Hi, Shawn! Thanks, this looks good! A few small things before we merge:
|
…fers Motivation: For HTTP third-party copies the flow-id of the firefly markers does not describe the connection that carries the data (dCache#8305): - The remote endpoint is the address the door resolved from the host name in the transfer URL. If the remote server redirects the request, or its name has several addresses, the marker names a host the data does not go to. The marker is also sent to that host. - For uploads, the transfer client reported a connection after each request had completed. A request that is redirected completes at once, so the start marker described the connection to the redirecting server, while the end marker described the data connection. If the request was not redirected, the start marker was only sent once all data had been sent. Observed on a pool: a push to a remote dCache produced markers naming <door address>:2881, with local port 55138 in the start marker and 44034 in the end marker, while the data went from local port 44034 to <pool address>:21468. Of 13 pulls in production traffic, 12 had a marker whose remote endpoint differed from the connection. Modification: RemoteHttpTransferClient reports the connection of an upload just before the first byte of the entity is written, instead of after each request. With expect-continue the entity is not written to a server that redirects the request, so only the data connection is reported. For a download it reports the connection after the retry loop, and the retried request uses the same context: while a server answers 202 the connection may be closed, and the data then arrives on a new one. The connection is reported once. If a server asks for a redirection only after it has received the file, the file is sent again; the second connection is not reported, so that start and end markers agree. It is logged at debug level by the org.dcache.scitags logger (event=connection-unreported), so that a site can tell whether this happens. RemoteHttpDataTransferProtocol takes the remote endpoint of the start marker from that connection, as it already does for the local endpoint, and falls back to the address in the protocol info. Both endpoint fields are volatile. Add Mover#getRemoteEndpoint and MoverProtocol#getRemoteEndpoint. DefaultPostTransferService uses it for the end marker, so that start and end markers carry the same flow-id. Add Mover#isActive and MoverProtocol#isActive for a mover that connects to the remote party itself and reports that connection. DefaultPostTransferService sends no end marker for such a mover if it never connected, as it already does for a passive mover nobody connected to. Before, a pull that failed on the DNS lookup or the TLS handshake sent an end marker with local port 0 and no start marker. Result: The firefly markers of an HTTP-TPC transfer describe the TCP connection that carries the data, for pulls and pushes, also when the remote server redirects the request, makes the pool retry the request, or its name has several addresses. The start marker of a push is sent when the upload starts. A transfer that fails before the pool has a connection to report, for example on the DNS lookup or the TLS handshake, sends no markers. Fixes: dCache#8305 Acked-by: Target: master Require-book: no Require-notes: yes Signed-off-by: Shawn McKee <smckee@umich.edu> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
Author
|
Thanks for the review. Rebased onto current master and updated (
The two new client tests fail on the previous version of this PR and pass now. The PR pipeline's unit-test job, run locally in |
ShawnMcKee
force-pushed
the
scitags/firefly-tpc-flow-id
branch
from
November 19, 2026 13:18
d13302f to
5c37b83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8305.
Problem
For HTTP third-party copies, the
flow-idof the firefly markers a pool emits does not describe the TCP connection that carries the data:RemoteTransferHandler,new InetSocketAddress(_destination.getHost(), port)). If the remote server redirects the request, or its name has several addresses, the marker names a host the data does not go to. The copy of the marker for the remote peer is sent to that host as well.RemoteHttpTransferClientreported a connection after each request had completed. A redirected request completes at once, so thestartmarker described the connection to the redirecting server while theendmarker described the data connection. Without a redirect, thestartmarker was only sent after all data had been sent.Observed on an AGLT2 pool (11.2.10), with fireflies and TCP connections captured together:
PUT, redirectedstartmarkerendmarkerIn 15 minutes of production traffic, 12 of 13 pulls had a marker whose remote endpoint differed from the connection. Details are in #8305.
Fix
RemoteHttpTransferClient: for an upload,onConnectedis called just before the first byte of the entity is written, instead of after each request. With expect-continue the entity is not written to a server that redirects the request, so only the data connection is reported. Downloads are unchanged: the callback already runs after the redirects have been followed.RemoteHttpDataTransferProtocol: thestartmarker takes its remote endpoint from the connection (RemoteHttpTransferClient.remoteAddress()), as it already does for the local endpoint. It falls back to the address in the protocol info.Mover#getRemoteEndpointandMoverProtocol#getRemoteEndpointare added, andDefaultPostTransferServiceuses them for theendmarker, so both markers carry the same flow-id.This follows the approach of the in-review door-side change (commit 01d10cb, "webdav: send SciTags fireflies for transfers proxied by the door"), which takes the endpoints from the connection and marks a push at the first write of the entity. The two changes conflict only in
RemoteHttpTransferClientTest, where both add a test at the same place.Behaviour change to be aware of
An upload that is rejected before any data is sent no longer triggers
onConnected. It gets nostartmarker, the mover reports no local endpoint for it, and itsendmarker falls back to the derived local endpoint and the protocol-info address, as for other movers without endpoint information.Testing
RemoteHttpTransferClientTest: the connection is reported before the upload completes, and for an upload redirected by a server that does not send "100 Continue" (as a dCache door behaves) only the data connection is reported. Both fail without the change toRemoteHttpTransferClient.maven:3.9.12-eclipse-temurin-21(mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5174 tests, 0 failures, 0 errors.DefaultPostTransferServicechanges are covered only by compilation and the existing tests.Signed-off-by: Shawn McKee smckee@umich.edu
🤖 Generated with Claude Code