Skip to content

pool, webdav: name the supplier of the data as source in fireflies - #8314

Draft
ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-src-supplier
Draft

ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-src-supplier

Conversation

@ShawnMcKee

Copy link
Copy Markdown
Contributor

Draft, for discussion on #8312. It implements option 2 there: src is the supplier of the data.

What changes in a firefly

Transfer src dst
Upload, third-party pull remote party dCache node (pool or door)
Download, third-party push dCache node (pool or door) remote party
  • usage is counted from the side of the source: sent is the data of the transfer. Before, it was counted from the side of the dCache node.
  • Every firefly carries a top-level "dcache-version", so a consumer can tell this convention from the earlier one, whose fireflies do not have the key.

Unchanged, and tied to remote/local rather than src/dst: the firefly is sent to the remote party, the syslog header names the dCache node, and the excludes apply when the remote and the local address are both inside the configured subnets.

Why

  • A reader, and a collector that stores the record as sent, takes src for the sender. On the ESnet collector, for 19 November 00:00 to 14:00 UTC, all 4,168 end records in which an AGLT2 pool had sent the data name that pool as dst.
  • Two dCache instances report the same transfer with mirrored tuples today. With this change they report the same flow-id.
  • XRootD names the supplier of the data as source where it knows it (XrdNetPMarkFF.cc).

How

TransferLifeCycle#onStart and the onEnd for transfers without a mover take the remote endpoint, the local endpoint and a DataDirection (INBOUND: the remote party supplies the data; OUTBOUND: this node does). The onEnd for movers derives the direction from MoverInfoMessage#isFileCreated. Each caller passes what it already knows:

Sender Direction from
Pool, client transfers (NettyTransferService) the mover's I/O mode
Pool, HTTP-TPC (RemoteHttpDataTransferProtocol) pull or push
Door, relayed transfers (DcacheResourceFactory) upload or download
Door, proxied TPC (RemoteTransferHandler) Direction.PULL / PUSH

The debug lines of the org.dcache.scitags logger report source and destination as in the firefly and gain a direction field.

Open questions

Testing

  • TransferLifeCycleTest: start and end markers for both directions, usage for both directions with and without a mover, excludes for both directions. NettyTransferServiceFlowMarkerTest: a reading mover reports outbound, a writing mover inbound. FlowMarkerTest: the version key, validated against the schema.
  • The PR pipeline's unit-test job, run locally in maven:3.9.12-eclipse-temurin-21 on this commit: 5212 tests, 0 failures, 0 errors.
  • Not deployed on a pool.

🤖 Generated with Claude Code

Motivation:
dCache always puts the remote party in src-ip/src-port of a firefly and
itself in dst-ip/dst-port, whatever the direction of the data (dCache#8312).
For a download or a third-party push the data flows from dst to src.

- A reader, and a collector that stores the record as sent, takes src
  for the sender. On one collector, all 4,168 'end' records of 14 hours
  in which an AGLT2 pool had sent the data name that pool as dst.
- Two dCache instances report the same transfer with mirrored tuples,
  so a consumer that identifies a flow by its tuple sees two flows.
- XRootD names the supplier of the data as source where it knows it.

A consumer can also not tell which convention a firefly follows,
because nothing in it identifies the dCache release.

Modification:
TransferLifeCycle#onStart and the onEnd for transfers without a mover
take the remote endpoint, the local endpoint and a DataDirection
(INBOUND: the remote party supplies the data; OUTBOUND: this node
does). The onEnd for movers derives the direction from
MoverInfoMessage#isFileCreated. The source of the flow is the remote
endpoint for inbound and the local endpoint for outbound transfers.

Unchanged, and tied to remote/local rather than src/dst: the firefly is
sent to the remote party, the excludes apply when the remote and the
local address are both inside the configured subnets, and the address
family is that of the local endpoint.

The callers pass the direction they already know: NettyTransferService
from the mover's I/O mode, RemoteHttpDataTransferProtocol from pull or
push, the WebDAV door from upload or download for relayed transfers and
from the transfer's direction for proxied third-party copies.

"usage" is counted from the side of the source: "sent" is what the
source sent to the destination, the data of the transfer. Before, it
was counted from the side of the dCache node. With the source as src
this keeps "sent" the larger value, so a collector that swaps src and
dst when received exceeds sent leaves the record alone.

Every firefly carries a top-level "dcache-version". The firefly schema
closes flow-id, flow-lifecycle, usage and context to additional
properties, but not the top level, where storage-stats already is.

The debug lines of the org.dcache.scitags logger report source and
destination as in the firefly and gain a direction field.

The SciTags chapter of TheBook describes the convention.

Result:
In fireflies sent by pools and WebDAV doors, src is the end that
supplies the data and dst the end that receives it, and usage.sent is
the amount of data transferred. Both ends of a transfer between two
dCache instances report the same flow-id. Consumers that relied on src
being the remote party need to be adapted; dcache-version tells the
conventions apart, as fireflies of earlier releases do not carry it.

Fixes: dCache#8312
Acked-by:
Target: master
Require-book: yes
Require-notes: yes

Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ShawnMcKee

Copy link
Copy Markdown
Contributor Author

This change now runs in production at AGLT2 on one pool host and one door host (backported to 11.2.10, together with #8313), since 19 November. What we checked:

Pool, 20 minutes of production traffic against a packet capture. 8 transfers were marked: 6 HTTP-TPC pulls and 2 downloads by remote clients.

  • In all 8, the start marker names a connection that is in the capture, with the supplier of the data as src: the remote server for the pulls, the pool for the downloads.
  • All 8 end markers carry the same flow as their start.
  • usage.sent equals the bytes transferred and usage.received is 0 in all 8, for the pulls as well.
  • All 16 fireflies carry "dcache-version":"11.2.10".

Door, relayed uploads from a client at our other site. For 2 of 2 uploads the door sent start and end with src = the client's connection (the port matches curl's local port) and dst = the door on 2880, to the client and to the collector; the end has sent=209715200, received=0.

ESnet collector. It stores the new fireflies as sent: downloads from that pool now have the pool as src. It drops dcache-version, since its pipeline keeps a fixed list of keys.

Two things this showed:

  • The value of dcache-version is the jar's Implementation-Version, which is 11.2.10 for our backport as for the stock release. A consumer can rely on the key being present, not on comparing the number, unless the convention changes with a release.
  • Not covered: a third-party push or an xrootd flow that crosses the site boundary, and transfers proxied by a door (that code is not in 11.2).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant