Skip to content

Log and report a failed data update check by polling or the file watcher, and go on checking - #146

Draft
jwrosewell wants to merge 9 commits into
rcfrom
fix/data-update-check-failures
Draft

jwrosewell wants to merge 9 commits into
rcfrom
fix/data-update-check-failures

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A check for a data update that runs on its own, from update polling or from the file watcher, runs inside a task of the data update service's scheduler. On rc b33daa0 an error thrown by that check, or by an update callback the application registered with onUpdateComplete, leaves the task and is held by its ScheduledFuture, which nothing reads. So the error is never logged and the check is never reported. Three things follow.

  1. The file watcher stops for good. The watcher's task is scheduled to repeat (DataUpdateServiceDefault line 615 at rc b33daa0f calls dataFileUpdated from it), and a repeating task that throws is never run again. One callback that throws once, for example while logging a success, means no later data file is ever applied by the watcher.
  2. Polling can stop for good. The polling timer is a task that runs once and schedules the next (lines 295 and 544). An error thrown before the next one is scheduled leaves the data file holding a finished timer, and registering only sets a timer when there is none, so no later check runs.
  3. A failed check is reported to nobody. The callbacks are called only at the end of a check that did not throw (lines 317 and 454). That is true of a programmatic check too, which passes the error to its caller without reporting the check.

The same goes for a Java Error thrown by the engine's refresh, for example from its native code. The refresh is tried up to 10 times, but the retries catch only Exception (line 363), so an Error leaves the check at once, before the data file is registered again for the next one.

What the specification says

data-updates.md at 706de1f0, "Errors", lines 474 to 479:

A failure in a check started by update polling or by the file watcher MUST be caught and logged, rather than being lost or ending the process, and is then reported to onUpdateCallbacks. A failure MUST NOT stop later checks for updates, so a later good data file is still applied. A programmatic check, the update on start-up and an update from memory pass the error to their caller instead of logging it.

The same section, lines 487 to 495:

An error of any other kind is logged at Error with the message in the last row, also with the error attached. An error raised by a callback in onUpdateCallbacks is logged in the same way on update polling and the file watcher, as Error during check for data update when it is of the same kind as the Error rows, and with the last row's message otherwise. On a programmatic check and the update on start-up it passes to the caller. Either way the callbacks after the one that raised the error are not called, and on update polling any reports of the check not yet made are not made.

The table's last two rows (lines 514 and 515) give a callback's error "None, already reported" as its status, and any other error the status AUTO_UPDATE_UNKNOWN_ERROR and the message "An unhandled error occurred while checking for automatic updates for engine '<Engine type name>'".

"Programmatic update", line 349:

on fail, report the failure to onUpdateCallbacks, then pass the error to the caller without logging it

What changed

All in DataUpdateServiceDefault, plus one new status.

  1. AUTO_UPDATE_UNKNOWN_ERROR is added to DataUpdateService.AutoUpdateStatus (line 134), the status the specification's table gives a failed check that no other row covers. Nothing is removed or renamed.
  2. The check is split from its report. The body of the check is now runCheck (line 268) and the call to each callback is report (line 1036), so each kind of check can handle a failure in either one as the specification says.
  3. Update polling. Both polling timers now run checkForUpdateOnTimer (line 248). An error from the check is logged at Error with the table's message and the error attached, the check is reported as AUTO_UPDATE_UNKNOWN_ERROR, and scheduleNextCheck (line 1066) clears the finished timer, without interrupting it, and registers the data file again, which sets the next one. An error from a callback is logged the same way by reportOnTimer (line 1050), and the callbacks after it are not called.
  4. The file watcher. dataFileUpdated (line 430) now runs the unchanged steps, moved into applyWatchedFile (line 457), and handles an error from them or from a callback as polling does. Nothing leaves the watcher's task, so it keeps running.
  5. A Java Error is handled as any other failure. Each of the places above catches RuntimeException and Error, so an Error from a refresh during update polling or a file watcher check is logged, reported and followed by the next check, rather than ending the timer's or the watcher's task. A refresh still retries only on an Exception.
  6. A programmatic check, and the update on start-up, report a check that threw as AUTO_UPDATE_UNKNOWN_ERROR and then pass the error to the caller unlogged, as before (line 223). An error from a callback still passes to the caller.

Java's data update service throws no error that carries one of the table's Error rows, as it logs those rows itself, so every error caught here takes the last row's message.

Tests

Written first (commit 55df4fd), in the new DataUpdateServiceCheckFailureTests. Each uses a mock logger and checks the Error entry by its message and the error object.

  • WatcherCheck_CallbackFails_LoggedAndWatchingCarriesOn. A callback that throws on every report. A replaced data file refreshes the engine, the error is logged, and a second, later data file still refreshes it. Not run on macOS, whose file watcher is too slow, as in the other watcher tests.
  • PollingCheck_CallbackFails_LoggedAndPollingCarriesOn. A 304 answer, reported once as AUTO_UPDATE_NOT_NEEDED to a callback that throws. The error is logged, nothing leaves the timer's task, and the next timer is set.
  • PollingCheck_CheckFails_ReportedAndNextCheckScheduled. The file system throws during the check. The error is logged, the check is reported as AUTO_UPDATE_UNKNOWN_ERROR, and the next timer is set.
  • PollingCheck_RefreshThrowsError_ReportedAndNextCheckScheduled (commit 6c13956). The data file is newer than the copy the engine uses and the engine's refresh throws an Error. The Error is logged, the check is reported as AUTO_UPDATE_UNKNOWN_ERROR, and the next timer is set.
  • ProgrammaticCheck_CheckFails_ReportedAndPassedToCaller. The same failure on checkForUpdate(engine). The caller gets the same error, the check is reported as AUTO_UPDATE_UNKNOWN_ERROR, and nothing is logged at Error.

The runs build pipeline/pipeline.engines with the modules it needs, on Java 17, and run every DataUpdateService*Tests class.

  • Red, the tests on rc b33daa0 (55df4fd with the workflow, 5b4d70e): run 43617219873. The new class failed 4 of 4. Three tests stopped with the error the callback or the file system threw, which left the timer's or the watcher's task, and the programmatic test failed with "expected:<[AUTO_UPDATE_UNKNOWN_ERROR]> but was:<[]>". The other 173 tests in the five classes passed.
  • Green, a9589a6 with the workflow (c1c20b9): run 43617240107. All 177 passed.
  • Red 2, the Error test on a9589a6 (6c13956 with the workflow, 4431751): run 43620147107. DataUpdateServiceCheckFailureTests failed 1 of 5, the new test, which stopped with the Error the refresh threw, and the other 173 tests in the five classes passed.
  • Green 2, the head (9051b19 with the workflow, 06bdb7c): run 43620156448. All 178 passed.

The first full build, 43620184966, failed in its four Windows jobs, all in WatcherCheck_CallbackFails_LoggedAndWatchingCarriesOn, which could not write the data file a second time, "The requested operation cannot be performed on a file with a user-mapped section open". The watcher checks that a changed file can be opened before it applies it, through FileWrapperDefault, whose reader maps the file, and Java never unmaps a file until the mapping is collected, so Windows refused the test's next write. 665aaa00 changes the test only, giving its file wrapper a reader that holds a copy of the bytes. Red 3, the head before it (9051b19) with a throwaway workflow on Ubuntu and Windows, and green 3, the new head with the same workflow, are red 3 and green 3. Red 3 failed on Windows in that test alone, with the same error at line 149, and passed on Ubuntu. Green 3 passed on both, DataUpdateServiceCheckFailureTests 5 of 5, DataUpdateServiceWatchedFileTests 9 of 9 and DataUpdateServiceTests 132 of 132.

Build

Run 43916434291 built 49e8960 merged with rc 81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Each of its ten Setting 'PullRequestSha' lines names 49e8960, and its Configure job printed "Left out 6 macOS entries".

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Before you upgrade

An update callback registered with onUpdateComplete can now be called with AUTO_UPDATE_UNKNOWN_ERROR, for a check that failed with an error no other status covers. A switch over AutoUpdateStatus with no default will not handle it. An error a callback throws during update polling or a file watcher check is now logged at Error, and the next check still runs.

Base branch

rc 81e3d4a is merged into the head, 49e8960, with a merge commit. Links to code in What changed are at 665aaa0, the commit before that merge. DataUpdateServiceCheckFailureTests passed 5 of 5 in all nine jobs.

What remains

Nothing for this change. #129, which changes the HTTP failure messages of the same class, is on rc, and a trial merge of the head with rc at c2d982a is clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant