Repository navigation
pool: add result code, file size, I/O mode, queue time and pool name to firefly storage statistics - #8313
pool: add result code, file size, I/O mode, queue time and pool name to firefly storage statistics#8313ShawnMcKee wants to merge 1 commit into
Conversation
…to firefly storage statistics Motivation: With pool.firefly.storage-statistics enabled, the 'end' firefly carries a "storage-stats" section with the bytes transferred, the duration and the disk read/write rates and active/idle times. Several things a consumer needs to interpret these numbers are missing: - The outcome. A transfer that failed before any data moved sends an 'end' firefly in which every value is zero; nothing distinguishes it from an empty file that was transferred successfully. - The file size. Without it a partial transfer cannot be told from a complete one. - The direction. Whether the pool read or wrote the file has to be inferred from which of usage.sent and usage.received is non-zero. - The time spent waiting for a mover. connection-time starts when the mover runs, so a transfer delayed by a full mover queue looks the same as one that started at once. - The pool. The syslog header names the host, but a host commonly runs several pools. All but the queue time are already in the MoverInfoMessage the statistics are taken from. The queue time is not: InfoMessage has a timeQueued field, but the pool never sets it for a mover. Modification: FlowMarkerBuilder#withStats adds five keys to "storage-stats": result-code return code of the transfer, 0 is success file-size size of the file in bytes io-mode "write" if the pool received the file, "read" otherwise queue-time time the request waited in the mover queue, milliseconds pool-name cell name of the pool Mover gains getQueueTime(). AbstractMover records when it is created, which is when the request is added to the mover queue, and when execute() is called, which is when the queue grants it a slot; the difference is the queue time. DefaultPostTransferService copies it into the MoverInfoMessage. The SciTags chapter of TheBook and the description of pool.firefly.storage-statistics list the content of the section. Unit tests cover the new keys for a successful upload and a failed download, and the queue time before, at and after execution. Result: When storage statistics are enabled, 'end' fireflies tell whether the transfer succeeded, how large the file is, whether the pool read or wrote it, how long the request was queued and which pool served it. Nothing changes if pool.firefly.storage-statistics is false, which is the default. As a side effect the mover's billing record now carries its queue time: the "queuingTime" field of the Kafka record of a mover, which was always 0, has the real value. Acked-by: Target: master Require-book: yes Require-notes: yes Signed-off-by: Shawn McKee <smckee@umich.edu> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
An update for the "not yet deployed on a pool" line: this change now runs on one production pool host at AGLT2 (backported to 11.2.10, together with #8314), since 19 November 15:45 UTC. From a packet capture of the fireflies that host sent in the first 67 minutes, 40
One block as sent, for a pull: "storage-stats":{"read-active":"0","connection-time":2196,"queue-time":0,"pool-name":"umfs19_1","read-bw":0,"result-code":0,"io-mode":"write","write-idle":"PT1.262989575S","write-bw":456505082,"read-idle":"0","file-size":1048576,"bytes-transferred":1048576,"write-active":"PT0.003129317S"}A note for consumers: the ESnet collector stores the block through a fixed list of keys ( |
|
A production result for All four are HTTP third-party pulls ( The remote server answered the GET with a 500, so a connection existed and the pool sent start and end fireflies. Without |
Problem
With
pool.firefly.storage-statistics=true, theendfirefly carries astorage-statsblock (bytes transferred, duration, disk read/write rates, active/idle times). A consumer cannot interpret those numbers without a few more facts, which the block does not carry:endfirefly in which every value is zero. Nothing distinguishes it from an empty file transferred successfully. We saw eight such fireflies on two AGLT2 pools in 20 hours (pulls that failed on a DNS lookup or a rejected certificate).usage.sentandusage.receivedis non-zero.connection-timestarts when the mover runs, so a transfer delayed by a full mover queue looks the same as one that started at once.Change
FlowMarkerBuilder#withStatsadds five keys tostorage-stats:result-code0is successfile-sizeio-modewriteif the pool received the file,readif it sent itqueue-timepool-nameFour of these are already in the
MoverInfoMessagethe statistics are taken from. The queue time is not:InfoMessagehas atimeQueuedfield, but the pool never sets it for a mover. So:MovergainsgetQueueTime().AbstractMoverrecords when it is created (when the request is added to the mover queue) and whenexecute()is called (when the queue grants it a slot). The difference is the queue time.DefaultPostTransferServicecopies it into theMoverInfoMessage.The SciTags chapter of TheBook and the description of
pool.firefly.storage-statisticsnow list the content of the block.Nothing changes when
pool.firefly.storage-statisticsisfalse, which is the default.Side effect
Because the mover's billing record now carries its queue time, the
queuingTimefield of the Kafka record of a mover, which was always0, has the real value. The default billing text format for movers does not print it.Things for review
io-modefollows the mover'sIoMode;directionwas avoided because of thesrc/dstdiscussion in Firefly flow-id: dCache always setssrc= remote party,dst= dCache node #8312.storage-statsto a fixed schema (the ESnet collector stores it asvalues.storage.*) will see five new keys.Testing
FlowMarkerStorageStatsTest(the new keys for a successful upload and a failed download; no block unless requested) andMoverQueueTimeTest(queue time before, at and after execution).maven:3.9.12-eclipse-temurin-21(mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests) on this commit: 5204 tests, 0 failures, 0 errors.🤖 Generated with Claude Code