Skip to content

pool: add result code, file size, I/O mode, queue time and pool name to firefly storage statistics - #8313

Open
ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-end-stats
Open

ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-end-stats

Conversation

@ShawnMcKee

Copy link
Copy Markdown
Contributor

Problem

With pool.firefly.storage-statistics=true, the end firefly carries a storage-stats block (bytes transferred, duration, disk read/write rates, active/idle times). A consumer cannot interpret those numbers without a few more facts, which the block does not carry:

  • The outcome. A transfer that fails before any data moves sends an end firefly in which every value is zero. Nothing distinguishes it from an empty file transferred successfully. We saw eight such fireflies on two AGLT2 pools in 20 hours (pulls that failed on a DNS lookup or a rejected certificate).
  • The file size. Without it a partial transfer cannot be told from a complete one.
  • The direction. Whether the pool read or wrote the file has to be inferred from which of usage.sent and usage.received is non-zero.
  • The time spent waiting for a mover. connection-time starts when the mover runs, so a transfer delayed by a full mover queue looks the same as one that started at once.
  • The pool. The syslog header names the host, but a host commonly runs several pools.

Change

FlowMarkerBuilder#withStats adds five keys to storage-stats:

Key Value
result-code return code of the transfer; 0 is success
file-size size of the file in bytes
io-mode write if the pool received the file, read if it sent it
queue-time time the request waited in the pool's mover queue, in milliseconds
pool-name cell name of the pool

Four of these are already in the MoverInfoMessage the statistics are taken from. The queue time is not: InfoMessage has a timeQueued field, but the pool never sets it for a mover. So:

  • Mover gains getQueueTime().
  • AbstractMover records when it is created (when the request is added to the mover queue) and when execute() is called (when the queue grants it a slot). The difference is the queue time.
  • DefaultPostTransferService copies it into the MoverInfoMessage.

The SciTags chapter of TheBook and the description of pool.firefly.storage-statistics now list the content of the block.

Nothing changes when pool.firefly.storage-statistics is false, which is the default.

Side effect

Because the mover's billing record now carries its queue time, the queuingTime field of the Kafka record of a mover, which was always 0, has the real value. The default billing text format for movers does not print it.

Things for review

  • The key names are a proposal. io-mode follows the mover's IoMode; direction was avoided because of the src/dst discussion in Firefly flow-id: dCache always sets src = remote party, dst = dCache node #8312.
  • Consumers that map storage-stats to a fixed schema (the ESnet collector stores it as values.storage.*) will see five new keys.
  • The error message is deliberately not sent; it can contain host names and paths.

Testing

  • New unit tests: FlowMarkerStorageStatsTest (the new keys for a successful upload and a failed download; no block unless requested) and MoverQueueTimeTest (queue time before, at and after execution).
  • Ran the PR pipeline's unit-test job locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests) on this commit: 5204 tests, 0 failures, 0 errors.
  • Not yet deployed on a pool.

🤖 Generated with Claude Code

…to firefly storage statistics

Motivation:
With pool.firefly.storage-statistics enabled, the 'end' firefly carries
a "storage-stats" section with the bytes transferred, the duration and
the disk read/write rates and active/idle times. Several things a
consumer needs to interpret these numbers are missing:

- The outcome. A transfer that failed before any data moved sends an
  'end' firefly in which every value is zero; nothing distinguishes it
  from an empty file that was transferred successfully.
- The file size. Without it a partial transfer cannot be told from a
  complete one.
- The direction. Whether the pool read or wrote the file has to be
  inferred from which of usage.sent and usage.received is non-zero.
- The time spent waiting for a mover. connection-time starts when the
  mover runs, so a transfer delayed by a full mover queue looks the same
  as one that started at once.
- The pool. The syslog header names the host, but a host commonly runs
  several pools.

All but the queue time are already in the MoverInfoMessage the
statistics are taken from. The queue time is not: InfoMessage has a
timeQueued field, but the pool never sets it for a mover.

Modification:
FlowMarkerBuilder#withStats adds five keys to "storage-stats":

  result-code  return code of the transfer, 0 is success
  file-size    size of the file in bytes
  io-mode      "write" if the pool received the file, "read" otherwise
  queue-time   time the request waited in the mover queue, milliseconds
  pool-name    cell name of the pool

Mover gains getQueueTime(). AbstractMover records when it is created,
which is when the request is added to the mover queue, and when
execute() is called, which is when the queue grants it a slot; the
difference is the queue time. DefaultPostTransferService copies it
into the MoverInfoMessage.

The SciTags chapter of TheBook and the description of
pool.firefly.storage-statistics list the content of the section.

Unit tests cover the new keys for a successful upload and a failed
download, and the queue time before, at and after execution.

Result:
When storage statistics are enabled, 'end' fireflies tell whether the
transfer succeeded, how large the file is, whether the pool read or
wrote it, how long the request was queued and which pool served it.
Nothing changes if pool.firefly.storage-statistics is false, which is
the default.

As a side effect the mover's billing record now carries its queue time:
the "queuingTime" field of the Kafka record of a mover, which was always
0, has the real value.

Acked-by:
Target: master
Require-book: yes
Require-notes: yes

Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ShawnMcKee

Copy link
Copy Markdown
Contributor Author

An update for the "not yet deployed on a pool" line: this change now runs on one production pool host at AGLT2 (backported to 11.2.10, together with #8314), since 19 November 15:45 UTC.

From a packet capture of the fireflies that host sent in the first 67 minutes, 40 end fireflies:

  • All 40 carry the five new keys.
  • io-mode is write for the 31 HTTP-TPC pulls and the 3 client uploads, and read for the 6 client downloads.
  • file-size equals bytes-transferred in all 40.
  • pool-name is umfs19_1 or umfs19_2, the two pools on the host.
  • result-code is 0 and queue-time is 0 in all 40. So the non-trivial values of those two have not been seen in production yet: no transfer failed after connecting in that period, and no mover had to wait.

One block as sent, for a pull:

"storage-stats":{"read-active":"0","connection-time":2196,"queue-time":0,"pool-name":"umfs19_1","read-bw":0,"result-code":0,"io-mode":"write","write-idle":"PT1.262989575S","write-bw":456505082,"read-idle":"0","file-size":1048576,"bytes-transferred":1048576,"write-active":"PT0.003129317S"}

A note for consumers: the ESnet collector stores the block through a fixed list of keys (values.storage.*), so it currently drops the new ones.

@ShawnMcKee

Copy link
Copy Markdown
Contributor Author

A production result for result-code. Since 19 November AGLT2 runs this change (together with #8314) on all pools. In the first three hours after the last pool switched (20:00–22:45 UTC), 4 of 7,514 end fireflies from our pools in the ESnet collector index carry "result-code":10027 (CacheException.THIRD_PARTY_TRANSFER_FAILED); all the others carry 0.

All four are HTTP third-party pulls (remotehttpsdatatransfer, io-mode write, file-size 0, bytes-transferred 0) on four different pools, from three remote sites. The pool log has the matching failure at the same second, for example:

19 Nov 2026 17:01:16 (umfs33_2) [door:WebDAV-dcdum03@webdav-dcdum03Domain:AAZgqDmGg6g RemoteTransferManager PoolAcceptFile 00005EC0A50F02CE4ED0B0D2EEC01A8B9128] Transfer failed: rejected GET: 500 Server Error

The remote server answered the GET with a 500, so a connection existed and the pool sent start and end fireflies. Without result-code the end firefly reads as a successful transfer of 0 bytes; with it, a collector can tell the failure apart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant