Skip to content

pool: fix start-time and end-time of firefly end markers - #8304

Merged
chubnastia merged 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-end-marker-times
Nov 19, 2026
Merged

chubnastia merged 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-end-marker-times

Conversation

@ShawnMcKee

Copy link
Copy Markdown
Contributor

Problem

The end firefly reports the transfer interval one transfer duration too late. TransferLifeCycle.onEnd() builds it as:

Instant startedAt = Instant.ofEpochMilli(mover.getTimestamp());
Instant finishedAt = startedAt.plusMillis(transferDurationMillis);

MoverInfoMessage is created when the transfer has finished, so its timestamp is the end of the transfer, not its start. The result:

  • start-time is the moment the transfer ended,
  • end-time lies in the future when the marker is sent,
  • the start and end markers of the same transfer disagree about when it started.

The duration (end-time − start-time) is correct.

Observed on an AGLT2 pool (11.2.10) for one HTTP-TPC pull of 50.4 s, same flow in both markers:

marker sent (current-time) start-time end-time
start 16:23:15.710Z 16:23:15.710Z
end 16:24:06.215Z 16:24:06.214Z 16:24:56.639Z

The transfer ran from 16:23:15.7 to 16:24:06.2.

Cause

This came in with my change for a review comment on #8044, which asked for the start time to be derived from moverInfo.getTimestamp(). I used the timestamp as the start itself. Before that the code used Instant.now() as end and subtracted the duration. master, 12.1, 12.0 and 11.2 are affected.

Fix

Keep deriving the times from the mover info timestamp, but use it as end-time and subtract the connection time to get start-time. For the example above this gives a start-time of 16:23:15.789Z, within 80 ms of the start marker.

Testing

  • The unit test that pinned the old behaviour is updated (shouldDeriveEndMarkerTimesFromTransferEnd) and now also checks that end-time is not after the marker's current-time. It fails on the unfixed code and passes with the fix.
  • Ran the PR pipeline's unit-test job locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5172 tests, 0 failures, 0 errors.
  • Not yet deployed on a pool.

Note: #8303 touches neighbouring lines in TransferLifeCycleTest; whichever is merged second will need a rebase, which I will do.

Signed-off-by: Shawn McKee smckee@umich.edu

🤖 Generated with Claude Code

@chubnastia

Copy link
Copy Markdown
Contributor

Thanks, Shawn! Unfortunately it conflicts with master, because onEnd was refactored. Could you rebase?

Motivation:
The 'end' firefly reports the time interval of the transfer as
start-time = MoverInfoMessage#getTimestamp and end-time = start-time +
connection time. The mover info message is created when the transfer
has finished, so its timestamp is the end of the transfer, not its
start. As a result both times are too late by the duration of the
transfer: start-time is the moment the transfer ended and end-time
lies in the future when the marker is sent. The start and end markers
of the same transfer also disagree about when it started.

Observed on a pool for one HTTP-TPC pull of 50.4 s:

  start marker, sent 16:23:15.710Z: start-time 16:23:15.710Z
  end marker,   sent 16:24:06.215Z: start-time 16:24:06.214Z
                                    end-time   16:24:56.639Z

The transfer ran from 16:23:15.7 to 16:24:06.2.

The start-time was derived from the mover info timestamp following a
review comment on dCache#8044; the timestamp was then used as the start
itself.

Modification:
Use the mover info timestamp as end-time and derive start-time by
subtracting the connection time. The duration is unchanged.

The unit test that pinned the previous behaviour is updated, and now
also checks that end-time does not lie after the marker's
current-time. The SciTags chapter of TheBook described the times of
end markers the old way and is corrected.

Rebased onto the onEnd refactoring of 985824c: the fix is now in
onEnd(src, dst, MoverInfoMessage), which hands the derived start-time
and the duration to the common onEnd. End markers of doors, which pass
their own start time, are not affected.

Result:
The start-time and end-time of 'end' fireflies describe the interval
in which the transfer took place, and agree with the start-time of
the corresponding 'start' firefly. For the example above the derived
start-time is 16:23:15.789Z.

Acked-by:
Target: master, 12.1, 12.0, 11.2
Require-book: no
Require-notes: yes

Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ShawnMcKee
ShawnMcKee force-pushed the scitags/firefly-end-marker-times branch from d4c19eb to 9709fb7 Compare November 19, 2026 13:14
@ShawnMcKee

Copy link
Copy Markdown
Contributor Author

Rebased onto current master (9709fb7f5e).

After the onEnd refactoring the fix sits in onEnd(src, dst, MoverInfoMessage): it hands the derived start-time and the duration to the common onEnd. End markers sent by doors pass their own start time and are not affected.

I also corrected one sentence in the SciTags chapter of TheBook, which described the times of end markers the old way.

The PR pipeline's unit-test job, run locally in maven:3.9.12-eclipse-temurin-21 on this commit: 5198 tests, 0 failures, 0 errors.

@chubnastia
chubnastia merged commit f8f6cf2 into dCache:master Nov 19, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants