Repository navigation
pool, webdav: name the supplier of the data as source in fireflies - #8314
ShawnMcKee wants to merge 1 commit into
Conversation
Motivation: dCache always puts the remote party in src-ip/src-port of a firefly and itself in dst-ip/dst-port, whatever the direction of the data (dCache#8312). For a download or a third-party push the data flows from dst to src. - A reader, and a collector that stores the record as sent, takes src for the sender. On one collector, all 4,168 'end' records of 14 hours in which an AGLT2 pool had sent the data name that pool as dst. - Two dCache instances report the same transfer with mirrored tuples, so a consumer that identifies a flow by its tuple sees two flows. - XRootD names the supplier of the data as source where it knows it. A consumer can also not tell which convention a firefly follows, because nothing in it identifies the dCache release. Modification: TransferLifeCycle#onStart and the onEnd for transfers without a mover take the remote endpoint, the local endpoint and a DataDirection (INBOUND: the remote party supplies the data; OUTBOUND: this node does). The onEnd for movers derives the direction from MoverInfoMessage#isFileCreated. The source of the flow is the remote endpoint for inbound and the local endpoint for outbound transfers. Unchanged, and tied to remote/local rather than src/dst: the firefly is sent to the remote party, the excludes apply when the remote and the local address are both inside the configured subnets, and the address family is that of the local endpoint. The callers pass the direction they already know: NettyTransferService from the mover's I/O mode, RemoteHttpDataTransferProtocol from pull or push, the WebDAV door from upload or download for relayed transfers and from the transfer's direction for proxied third-party copies. "usage" is counted from the side of the source: "sent" is what the source sent to the destination, the data of the transfer. Before, it was counted from the side of the dCache node. With the source as src this keeps "sent" the larger value, so a collector that swaps src and dst when received exceeds sent leaves the record alone. Every firefly carries a top-level "dcache-version". The firefly schema closes flow-id, flow-lifecycle, usage and context to additional properties, but not the top level, where storage-stats already is. The debug lines of the org.dcache.scitags logger report source and destination as in the firefly and gain a direction field. The SciTags chapter of TheBook describes the convention. Result: In fireflies sent by pools and WebDAV doors, src is the end that supplies the data and dst the end that receives it, and usage.sent is the amount of data transferred. Both ends of a transfer between two dCache instances report the same flow-id. Consumers that relied on src being the remote party need to be adapted; dcache-version tells the conventions apart, as fireflies of earlier releases do not carry it. Fixes: dCache#8312 Acked-by: Target: master Require-book: yes Require-notes: yes Signed-off-by: Shawn McKee <smckee@umich.edu> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
This change now runs in production at AGLT2 on one pool host and one door host (backported to 11.2.10, together with #8313), since 19 November. What we checked: Pool, 20 minutes of production traffic against a packet capture. 8 transfers were marked: 6 HTTP-TPC pulls and 2 downloads by remote clients.
Door, relayed uploads from a client at our other site. For 2 of 2 uploads the door sent start and end with ESnet collector. It stores the new fireflies as sent: downloads from that pool now have the pool as Two things this showed:
|
Draft, for discussion on #8312. It implements option 2 there:
srcis the supplier of the data.What changes in a firefly
srcdstusageis counted from the side of the source:sentis the data of the transfer. Before, it was counted from the side of the dCache node."dcache-version", so a consumer can tell this convention from the earlier one, whose fireflies do not have the key.Unchanged, and tied to remote/local rather than
src/dst: the firefly is sent to the remote party, the syslog header names the dCache node, and the excludes apply when the remote and the local address are both inside the configured subnets.Why
srcfor the sender. On the ESnet collector, for 19 November 00:00 to 14:00 UTC, all 4,168endrecords in which an AGLT2 pool had sent the data name that pool asdst.flow-id.XrdNetPMarkFF.cc).How
TransferLifeCycle#onStartand theonEndfor transfers without a mover take the remote endpoint, the local endpoint and aDataDirection(INBOUND: the remote party supplies the data;OUTBOUND: this node does). TheonEndfor movers derives the direction fromMoverInfoMessage#isFileCreated. Each caller passes what it already knows:NettyTransferService)RemoteHttpDataTransferProtocol)DcacheResourceFactory)RemoteTransferHandler)Direction.PULL/PUSHThe debug lines of the
org.dcache.scitagslogger report source and destination as in the firefly and gain adirectionfield.Open questions
srcanddstwhenusage.received > usage.sent. Countingusagefrom the source's side keepssentthe larger value, so that rule leaves these records alone. Is that the reading ofusageSciTags wants?src= remote party,dst= dCache node #8312). With this change that matches when dCache sends the data and not when it receives.dcache-version: is a top-level key acceptable, and is the name right? The schema closesflow-id,flow-lifecycle,usageandcontextto additional properties but not the top level, wherestorage-statsalready is.FlowMarker; I will rebase whichever is merged second.Testing
TransferLifeCycleTest: start and end markers for both directions, usage for both directions with and without a mover, excludes for both directions.NettyTransferServiceFlowMarkerTest: a reading mover reports outbound, a writing mover inbound.FlowMarkerTest: the version key, validated against the schema.maven:3.9.12-eclipse-temurin-21on this commit: 5212 tests, 0 failures, 0 errors.🤖 Generated with Claude Code