SOLR-18344: Report in-progress backup status on /replication?command=details

[idantepper]

https://issues.apache.org/jira/browse/SOLR-18344

Description

The backup key in the /replication?command=details response is only populated once a backup has finished. While one runs there is no sign of it, and the key still holds the previous backup's payload — including "status": "success".

So a client that issues command=backup and then polls command=details reads the stale success of an earlier backup as the result of the one it just started, and concludes the new backup is already done. With no prior backup the key is simply absent, so a poller cannot distinguish "running" from "never started" either.

Solution

The plumbing already existed and was only used once. SnapShooter.createSnapAsync takes a Consumer<NamedList<?>> that ReplicationHandler wires to its volatile snapShootDetails field, which getReplicationDetails already publishes under the backup key — it was just invoked a single time, at the very end of the snapshot.

That consumer is now kept in a volatile progressListener field and emitted through as the copy loop advances, adding two in-progress statuses ahead of the existing terminal ones:

status	When	Extra keys
waiting for commit	Published synchronously, before the worker thread starts	none
running	After each backupRepo.copyFileFrom	fileCount, finishedFileCount
success / exception	Unchanged	unchanged

The synchronous first publish is the part that fixes the stale-success read. It carries no file counts because the index commit — and with it the list of files to copy — has not been resolved yet.

All in-progress statuses carry startTime, directoryName, and snapshotName; a null snapshotName is omitted rather than reported, matching how CoreSnapshotResponse renders a completed snapshot via putIfNotNull.

The change is purely additive to the response, involves no API change, and ReplicationHandler is untouched — which keeps this to a single main-source file.

Tests

TestSnapshotCoreBackup#testBackupReportsProgressWhileRunning collects every status the handler would publish and asserts on the whole sequence, rather than racing a live backup by polling — so it is deterministic. It asserts that:

• the first report is waiting for commit, carrying neither fileCount nor finishedFileCount;
• running reports follow, with a non-decreasing finishedFileCount;
• every in-progress report identifies its own snapshot (startTime, snapshotName, directoryName);
• the last running report accounts for every file in the completed backup;
• the resulting backup still passes simpleBackupCheck.

I verified the test fails without the fix — with the three progress emissions removed it fails on Backup was never reported as running.

./gradlew tidy updateLicenses check -x test was run. Two tasks fail in my local environment for reasons unrelated to this change, both of which reproduce on unmodified main:

• :rat — my checkout is a git worktree, where .git is a file rather than a directory. rat-sources.gradle treats that as "not a git repository", so the git-index lookup returns null and the exclude "dev-docs" / exclude "**/AGENTS.md" / exclude "**/.*" rules in that same branch never apply; rat then scans the whole tree and flags upstream files such as dev-docs/*.adoc, .github/* and AGENTS.md.
• :solr-ref-guide:buildLocalAntoraSite — my repository path contains a space, which the npx invocation does not quote (Error: Cannot find module '/Users/idantepper/Desktop/Developing').

All other checks pass, including spotlessCheck, ecjLint, forbiddenApis, validateLogCalls, validateSourcePatterns and javadoc.

AI assistance disclosure

This change was developed with the assistance of Claude (Anthropic), following the guidance in dev-docs/how-to-contribute.adoc. The commit carries a Co-Authored-By trailer. I have reviewed the diff and the test in full, confirmed the test fails without the fix, and I take responsibility for the contribution.

Checklist

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch.
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide — backup-restore.adoc §"Backup Status" now documents the two in-progress statuses, and states that the reported status is retained until the next backup starts. (An earlier revision of this description claimed the ref guide did not document this response; that was wrong.)
• I have added a changelog entry for my change.

[epugh]

so is the idea that a success is just going to hangaround for forever, or until the next snapshot is begun? I guess that is how it works...

[idantepper]

Yes, exactly that. snapShootDetails on ReplicationHandler is a plain volatile field that is never cleared -- getReplicationDetails publishes it under backup whenever it is non-null. It gets overwritten by the next backup, or by a snapshot deletion, and it is only back to null after a core reload or node restart, at which point the backup key is simply absent again.

That lifetime is pre-existing and this PR does not change it. What it changes is when the first overwrite lands: it used to be at the end of the next backup, so a stale success survived the entire run of the backup that was meant to replace it. Now it is replaced the moment the next backup is requested, which is the part that fixes the stale read.

You prompted me to go look at the docs, and my checklist claim was wrong: backup-restore.adoc -- "Backup Status" does document the shape of this response, it just only showed the completed success payload. I have pushed a ref-guide commit that adds the two in-progress statuses and states this retention behaviour explicitly, so a reader knows a success may describe an earlier backup rather than one just requested.

Drive-by in the same section, shout if you would rather it went separately: it said a failure reports snapShootException, which appears nowhere in the codebase -- the key is exception.

[epugh]

is this a pattern that we follow elsewhere in Solr for these types of things?

[idantepper]

Yes -- Solr has a family of "record what the code published, then assert on the collected sequence" helpers rather than racing the thing under test. The three closest to this one:

• TrackingBackupRepository (solr/test-framework/src/java/org/apache/solr/core/TrackingBackupRepository.java) is the nearest, and it is in this same feature area: it wraps the repository and records every copyIndexFileFrom / createOutput / createDirectory into a synchronized list so a test can assert on what a backup actually did. AbstractIncrementalBackupTest asserts on copiedFiles() / outputsCreated() / directoriesCreated(), as do LocalFSCloudIncrementalBackupTest and the S3/GCS/HDFS backup tests. Same seam as here -- the per-file copy inside a running backup -- and the same shape.
• SoftAutoCommitTest.MockEventListener registers a SolrEventListener that offers each async commit / newSearcher event into a LinkedBlockingQueue, and the test asserts on the ordering of what arrived. That is the precedent for asserting on an asynchronous sequence rather than polling for a single state.
• SnapshotBackupAPITest.TrackingSnapshotBackupAPI overrides doSnapShoot and records what the handler passed instead of running a live backup -- the same seam this test uses. The Consumer<NamedList<?>> overload of ReplicationHandler.doSnapShoot is what ReplicationHandler itself calls, so this is not a test-only backdoor.

The alternative would have been BackupStatusChecker, which polls command=details over HTTP (TestRestoreCore, TestReplicationHandlerBackup, TestStressThreadBackup). I did not use it because it is deliberately terminal-state-only -- it returns null for anything that is not success -- and its own javadoc says it is "NOT suitable/safe ... because the replication handler API provides no reliable way to check the results of a specific backup before the results of another backup may overwrite them internally". Polling it for an intermediate status would flake: on a test-sized index the copy loop can finish between two 50ms polls.

Worth recording that the new statuses do not disturb that helper either -- non-success statuses still return null, and neither the exception check nor the startsWith("Unable to delete") check can match waiting for commit / running.

Happy to add an HTTP-level assertion on top if you would prefer one, though it could only assert "not success yet" rather than a specific in-progress status.

Unrelated: the red check is TestGracefulJettyShutdown.testSingleShardInFlightRequestsDuringShutDown failing on a jetty HTTP/2 ClosedChannelException during shutdown -- nothing to do with this change; gradle check is green.

[epugh]

"repots backup details" maybe?

[epugh]

pending tests completing, and me giving it a manual test, and maybe a tweak to chagnelyg, this looks great!

[epugh]

realizing I should just mark it request changes till the changelog gets sorted ;-)