Give FOS somewhere to report an imaging failure by mastacontrola · Pull Request #1207 · FOGProject/fogproject

mastacontrola · 2026-08-19T11:05:34Z

The server half of #1206. The FOS half is FOGProject/fos#152 — this one is useful on its own (an older FOS just never calls it), the FOS one is not useful without this.

The gap

handleError() in FOS prints a banner to the console and exit 1s. It makes no request to the server, so a real imaging failure — bad image, mount failure, partition error, any of the ~40 handleError call sites across fog.download, fog.upload and fog.mount — has always been invisible to FOG. The only failure the server ever heard about was a storage node problem, through fog.checkmount → blame.php, and that re-queues rather than fails.

#1205 wired HOST_IMAGE_FAIL to TaskQueue::checkout()'s catch, which covers "imaging ran and FOG then failed to record it". This covers the failure people actually mean.

service/taskerror.php takes the host identity FOS already sends everywhere else, plus the error text and the script that raised it, and fires HOST_IMAGE_FAIL with the payload #1205 defined — the same keys from either path, so a listener behaves identically whichever failure reached it.

Deliberately narrow, in three ways

It does not change the task's state. taskStates has no Failed — the five are Queued, Checked In, In-Progress, Complete, Cancelled. Reusing Cancelled loses the difference between "an admin stopped this" and "this broke"; adding a sixth means every place that enumerates states has to learn about it or a failed task becomes invisible there. That is a decision with UI and API consequences, it stays on #1206, and making the event fire does not depend on taking it.

It refuses a task that is not imaging. HOST_IMAGE_FAIL is an imaging event and this endpoint is reachable from a wipe or an inventory task too. Firing it there would be the defect #1202 just removed, pointing the other way. There is no event for a non-imaging task failing; noted on the issue.

It answers ## with 200 on every path, including every rejection. FOS calls this on its way out and cannot act on the reply. Answering identically also means the endpoint cannot be used to ask whether a given MAC has an active imaging task.

The text

Bounded to 500 characters and flattened to one line. This is unauthenticated in the same way every other service/*.php is — identified by MAC, reachable by anything that can reach the web tier — and the text lands in an administrator's Slack, ntfy or pushbullet message. An embedded newline there forges what looks like a second, separate notification, which is a better lie than anything markup could manage:

error=Could not mount images folder (fog.mount)
   Args Passed: --target /images
Host imaging completed successfully

comes out as one line, with the forged "success" visibly part of the failure text.

\p{C} rather than a newline-only strip, because a console-facing error string can carry terminal escapes too; preg_replace returns null rather than throwing on invalid UTF-8, so there is a byte-wise fallback — a machine with the wrong locale must not silently report nothing.

Not a REST route

service/*.php is served directly and is not in Route::defineRoutes(), so OpenAPI::document() is unaffected — same as blame.php and the Post_Stage endpoints. No schema change, so no FOG_SCHEMA bump. No change to Route::$validClasses, so no FogApi sync.

Verification

Live, against this install's real database, using a throwaway host on a locally-administered MAC (02:00:00:00:0E:11) that matches no hardware, so nothing could ever PXE boot into the task:

case	result
deploy task	`FOG: imaging failed on host … (task 62): Could not mount images folder (fog.mount) Args Passed: --target /images Host imaging completed successfully` — newlines flattened
capture task	notified and logged
wipe task	refused: `Task is not an imaging task`
unknown MAC	refused: `Invalid Host`
4000-char report	cut to exactly 500
every one of them	HTTP 200, body `##`

HOST_IMAGE_FAIL now appears in notifyEvents, which is proof notify() was genuinely reached and got past its guards. Fixture and shadow tree removed afterwards; host count back to 86.

tests/task-error-report.test.php, mutation-verified — dropping the control-character strip, the length bound, the imaging gate, the uniform ack, or adding a state change each fails the suite. Full suite: 68 passed, 0 failed.

🤖 Generated with Claude Code

https://claude.ai/code/session_013mJVe4CpK3rRbi9H5GubXd

The server half of #1206. FOS's handleError() prints a banner to the console and exits, making no request at all, so a real imaging failure -- bad image, mount failure, partition error -- has always been invisible to FOG. The only failure the server ever heard about was a storage node problem, through Blame, and that re-queues rather than fails. #1205 wired HOST_IMAGE_FAIL to checkout()'s catch, which covers "imaging ran and FOG then failed to record it"; this covers the failure people actually mean. service/taskerror.php takes the host identity FOS already sends everywhere else, plus the error text and the script that raised it, and fires HOST_IMAGE_FAIL with the payload #1205 defined -- same keys from either path, so a listener behaves the same whichever failure reached it. Deliberately narrow in three ways. It does not change the task's state. taskStates has no Failed; reusing Cancelled would lose the difference between "an admin stopped this" and "this broke", and adding a sixth state means every place that enumerates them has to learn about it or a failed task becomes invisible there. That is a decision with UI and API consequences, it is tracked on #1206, and making the event fire does not depend on taking it. It refuses a task that is not imaging. HOST_IMAGE_FAIL is an imaging event and this endpoint is reachable from a wipe or an inventory task too; firing it there would be the defect #1202 just removed, pointing the other way. It answers '##' with 200 on every path, including every rejection. FOS calls this on its way out and cannot act on the reply, and answering identically means the endpoint cannot be used to ask whether a MAC has an active imaging task. The reported text is bounded to 500 characters and flattened to one line. This is unauthenticated in the same way every other service/*.php is, and the text lands in an administrator's Slack, ntfy or pushbullet message -- an embedded newline there forges what looks like a second, separate notification, which is a better lie than anything markup could manage. Not a REST route -- service/*.php is served directly and is not in Route::defineRoutes() -- so OpenAPI::document() is unaffected, same as blame.php and the Post_Stage endpoints. Verified against the live 1.6 install with a throwaway host on a locally-administered MAC that matches no hardware: a deploy and a capture both notify and log with the host and task id, a wipe is refused as "Task is not an imaging task", an unknown MAC is refused as "Invalid Host", a 4000 character report is cut to exactly 500, embedded newlines come out as spaces, and every one of them answers 200. HOST_IMAGE_FAIL now appears in notifyEvents, which is proof notify() was genuinely reached. Gated by tests/task-error-report.test.php and mutation-verified. Co-Authored-By: Claude <noreply@anthropic.com>

…hema 338) (#1208) #1207 gave FOS an endpoint to report an imaging failure to. It fired HOST_IMAGE_FAIL and called error_log(), and that was the whole of it -- so a report either reached a notification plugin or it reached the web server's error log, mixed in with everything else the web tier writes, correlated with nothing. Neither is a place an admin goes to ask "what did this task actually say before it stopped". A report now lands in three places: a `taskLog` row, typed and with the text in it. This is the one correlated with the task: it carries taskID and the state the task was in; /var/log/fog/fos/fosreports.log, listed by the Log Viewer like any other log because 'fos' is now in FOGLogPaths; HOST_IMAGE_FAIL, as before -- errors only, imaging tasks only. SCHEMA 338 adds `logType` (default 'state') and `logText` (NULL) to taskLog. Every row in that table so far is a state transition, which is exactly what the default backfills them as, so TaskingElement::taskLog() is untouched: a state row costs no extra column. logText is NULL rather than '' so "no body" and "an empty body" stay distinguishable. The step is a closure, like 336, because ADD COLUMN has no IF NOT EXISTS below MariaDB 10.0.2 and a re-run has to converge rather than error. TYPES. FOS now sends `type=error` or `type=warning`, because handleWarning() reports too (FOGProject/fos#152). A warning means the machine carried on, so it is recorded and it does not fire the failure event -- announcing a failed deploy for a task that went on to succeed would be worse than saying nothing. A report with no type at all is an error: an older FOS reporting a real failure must not be downgraded into silence. The row is written BEFORE the imaging gate, so a failed Memtest or inventory is recorded against its task even though no imaging event can fire for it. That gate is still there for the event itself -- firing HOST_IMAGE_FAIL for a non-imaging task is the defect #1202 removed, in the other direction. THE LOG FILE gets its own subdirectory, created by the installer as the web user with httpd_sys_rw_content_t. Both halves matter. It is not the top level because $servicelogs is root's and holds the eight daemons' logs, and rotation renames and unlinks -- the same split ADR 0010 made for the plugin runner. And it is not created by PHP, because /opt/fog inherits usr_t and httpd_t may READ usr_t but not write it (GH-964), so a directory made without the relabel looks right and silently swallows every write on an enforcing host. error_log() is kept as the fallback, not the destination: a server whose web tree is updated but which has not been re-installed yet has nowhere to write, and a report that reaches no log at all is the exact failure this path exists to end. Rotation keeps one old copy at SERVICE_LOG_SIZE -- the daemons' setting, so an admin who has already decided how big a FOG log may get does not decide again. One generation rather than five: this file gains a line per failed task, not a line per poll. VERIFIED against a throwaway copy of a real FOG database in a container, with the web tier served from a shadow tree, so nothing live was written: step 338 applied through its own closure, backfilled all 52 existing rows to 'state', and ran clean a second time; an error on a Deploy task wrote its row, its log line and fired HOST_IMAGE_FAIL to a registered listener; a warning on the same task wrote its row and its log line and fired nothing; an error on a Memtest task wrote its row and its log line and fired nothing; a report with no type was recorded as an error; an embedded newline arrived flattened to spaces, as #1207 intended; the file rotated to .1 once it passed SERVICE_LOG_SIZE. The `ip` column is empty in those rows only because filter_input(INPUT_SERVER) returns NULL under the cli-server SAPI the harness used; the same TaskLog constructor fills it under php-fpm, which the 52 pre-existing rows show. tests/task-error-report.test.php grows the type routing, the row's position relative to both gates, the model mapping, the FOGLogPaths entry and the installer's two lines. Every one of them was mutation-verified. Downstream: none. `tasklog` is already a Route::$validClasses entry, so OpenAPI picks the two columns up from the regenerated manifest. Co-authored-by: JJ Fullmer <7743340+darksidemilk@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…ema 280) Ported from working-1.6 (#1207, #1208, #1209). This is a feature on a maintenance branch, and the reason it belongs here anyway is that FOS is not branched. FOGProject/fos#152 makes handleError() AND handleWarning() post a report to whatever server the machine booted from, and a 1.5 server is just as likely to be that server. Without this the POST reaches a 404 and 1.5 keeps the behaviour it has had since the beginning: a machine stops mid-image and says nothing to anyone. HOST_IMAGE_FAIL has two listeners in this tree (slack, pushbullet) and no core caller, so neither has ever fired on any 1.5 server. WHAT ARRIVES. service/taskerror.php takes mac, sysuuid, a type of error or warning, the text, and the script that raised it. A report lands in three places, none of which is the task's state: a `taskLog` row, typed, with the text in it -- the one correlated with the task, carrying taskID and the state the task was in; /var/log/fog/fos/fosreports.log, which the Log Viewer lists like any other; HOST_IMAGE_FAIL -- errors only, imaging tasks only. SCHEMA 280 adds `logType` (default 'state') and `logText` (NULL) to taskLog. Every row in that table so far is a state transition, which is what the default backfills them as, so TaskingElement::taskLog() is untouched. A closure, not a bare ALTER, because ADD COLUMN has no IF NOT EXISTS below MariaDB 10.0.2 and a re-run has to converge. A warning is recorded and fires nothing, because the machine carried on; announcing a failed deploy for a task that went on to succeed would be worse than silence. A report with NO type is an error -- on this branch that is the normal case rather than the exotic one, because a FOS newer than the server is what 1.5 will usually be talking to. THE INSTALLER PROBE HAD TO COME WITH IT, and this is the part that is not optional. installFOGDB() probes fogstorage's INSERT privilege with a throwaway row and reads any failure as "the grants need redoing", which is what makes it demand a database root password. That probe was positional: INSERT INTO taskLog VALUES ( 0, '999test', 3, '127.0.0.1', NOW(), 'fog'); Six values into what schema 280 makes an eight column table is error 1136, "Column count doesn't match value count" -- so this schema step alone would have made every 1.5 upgrade stop and ask for a root password nobody needs to type, on servers whose grants are perfectly correct. 1.6 shipped exactly that regression and had already been bitten once before by the same statement; see same commit as the cause rather than after it. THE LOG DIRECTORY is created by the installer as the web user with httpd_sys_rw_content_t, in its own subdirectory: $servicelogs is root's and holds the daemons' logs, and rotation renames and unlinks. error_log() stays as the fallback so a server updated but not yet re-installed still records something. 'fos' is added to all THREE lists 1.5 keeps -- StorageNode:: _getData(), status/getfiles.php and status/logtoview.php -- because they fail differently: miss the first two and the selector has no entry, miss the last and it answers "Invalid Folder". VERIFIED against a throwaway copy of a real 1.5 database (2079 hosts, schema 278) in a container, with the web tier served from a shadow tree, so the live 1.5 install was never written to: step 280 applied through its own closure, backfilled all 7 existing rows to 'state', and ran clean a second time; the OLD probe against the migrated table -> ERROR 1136, as predicted; the new one -> ROW_COUNT() = 1; error on a Deploy task -> row, log line, HOST_IMAGE_FAIL to a listener; warning on the same task -> row and log line, no event; no type at all -> recorded as an error, event fired; error on a non-imaging task -> row and log line, no event; an embedded newline arrived flattened to spaces; the file rotated to .1 once it passed SERVICE_LOG_SIZE. WHAT IS DELIBERATELY NOT PORTED. The slack and pushbullet listeners are left exactly as they are. 1.6's #1202 rewrote them to name the image and the reason; here they keep reading only $data['HostName'], and the extra payload keys are simply ignored. Making them fire at all is the change this branch needed; changing what they say is a separate one. Two tests, both mutation-verified: task-error-report.test.php (the sanitizer run for real against stubbed base classes, plus the type routing, the row's position relative to both gates, and all three log-path lists) and installer-db-probes-name-columns.test.php (any positional INSERT anywhere in the installer). Co-Authored-By: Claude <noreply@anthropic.com>

…ema 280) (#1210) Ported from working-1.6 (#1207, #1208, #1209). This is a feature on a maintenance branch, and the reason it belongs here anyway is that FOS is not branched. FOGProject/fos#152 makes handleError() AND handleWarning() post a report to whatever server the machine booted from, and a 1.5 server is just as likely to be that server. Without this the POST reaches a 404 and 1.5 keeps the behaviour it has had since the beginning: a machine stops mid-image and says nothing to anyone. HOST_IMAGE_FAIL has two listeners in this tree (slack, pushbullet) and no core caller, so neither has ever fired on any 1.5 server. WHAT ARRIVES. service/taskerror.php takes mac, sysuuid, a type of error or warning, the text, and the script that raised it. A report lands in three places, none of which is the task's state: a `taskLog` row, typed, with the text in it -- the one correlated with the task, carrying taskID and the state the task was in; /var/log/fog/fos/fosreports.log, which the Log Viewer lists like any other; HOST_IMAGE_FAIL -- errors only, imaging tasks only. SCHEMA 280 adds `logType` (default 'state') and `logText` (NULL) to taskLog. Every row in that table so far is a state transition, which is what the default backfills them as, so TaskingElement::taskLog() is untouched. A closure, not a bare ALTER, because ADD COLUMN has no IF NOT EXISTS below MariaDB 10.0.2 and a re-run has to converge. A warning is recorded and fires nothing, because the machine carried on; announcing a failed deploy for a task that went on to succeed would be worse than silence. A report with NO type is an error -- on this branch that is the normal case rather than the exotic one, because a FOS newer than the server is what 1.5 will usually be talking to. THE INSTALLER PROBE HAD TO COME WITH IT, and this is the part that is not optional. installFOGDB() probes fogstorage's INSERT privilege with a throwaway row and reads any failure as "the grants need redoing", which is what makes it demand a database root password. That probe was positional: INSERT INTO taskLog VALUES ( 0, '999test', 3, '127.0.0.1', NOW(), 'fog'); Six values into what schema 280 makes an eight column table is error 1136, "Column count doesn't match value count" -- so this schema step alone would have made every 1.5 upgrade stop and ask for a root password nobody needs to type, on servers whose grants are perfectly correct. 1.6 shipped exactly that regression and had already been bitten once before by the same statement; see same commit as the cause rather than after it. THE LOG DIRECTORY is created by the installer as the web user with httpd_sys_rw_content_t, in its own subdirectory: $servicelogs is root's and holds the daemons' logs, and rotation renames and unlinks. error_log() stays as the fallback so a server updated but not yet re-installed still records something. 'fos' is added to all THREE lists 1.5 keeps -- StorageNode:: _getData(), status/getfiles.php and status/logtoview.php -- because they fail differently: miss the first two and the selector has no entry, miss the last and it answers "Invalid Folder". VERIFIED against a throwaway copy of a real 1.5 database (2079 hosts, schema 278) in a container, with the web tier served from a shadow tree, so the live 1.5 install was never written to: step 280 applied through its own closure, backfilled all 7 existing rows to 'state', and ran clean a second time; the OLD probe against the migrated table -> ERROR 1136, as predicted; the new one -> ROW_COUNT() = 1; error on a Deploy task -> row, log line, HOST_IMAGE_FAIL to a listener; warning on the same task -> row and log line, no event; no type at all -> recorded as an error, event fired; error on a non-imaging task -> row and log line, no event; an embedded newline arrived flattened to spaces; the file rotated to .1 once it passed SERVICE_LOG_SIZE. WHAT IS DELIBERATELY NOT PORTED. The slack and pushbullet listeners are left exactly as they are. 1.6's #1202 rewrote them to name the image and the reason; here they keep reading only $data['HostName'], and the extra payload keys are simply ignored. Making them fire at all is the change this branch needed; changing what they say is a separate one. Two tests, both mutation-verified: task-error-report.test.php (the sanitizer run for real against stubbed base classes, plus the type routing, the row's position relative to both gates, and all three log-path lists) and installer-db-probes-name-columns.test.php (any positional INSERT anywhere in the installer). Co-authored-by: JJ Fullmer <7743340+darksidemilk@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

… the task log (#1211) * Give a task the host died on a state of its own (schema 339) A FOS error report is recorded and announced (#1207, #1208), but the task itself was left exactly where it was: Queued or In-Progress, forever. The task list still said the machine was working on it, and because Host:: loadTask() and getActiveTaskCount() count those states as live, the host could not be re-tasked until somebody noticed and cancelled it by hand. Adds taskStates row 6, Failed, and moves the task to it when FOS reports an error. A warning never does -- a warning means FOS carried on. Not Cancelled, which was the alternative. Cancelled means an administrator stopped it; losing the difference between "somebody stopped this" and "this broke" costs the operator the one fact they open the task list to find. Adding a sixth state is additive because every "is this task live" test in the tree is an allowlist -- getQueuedStates() plus getProgressState() -- so a state nobody listed is inactive by construction. No call site had to learn about it, and Complete and Cancelled rows already persist in `tasks` (20 and 33 of them on the reference install), so a Failed row lingering is the existing behavior rather than new litter. The state is written for every task type, not just imaging ones: a Memtest the host died on is as finished as a deploy. Only HOST_IMAGE_FAIL stays imaging-specific. Guarded on the taskStates row existing, so a web tree updated ahead of its database leaves the task alone rather than pointing it at a state that is not there. Verified end to end against an isolated copy of this server's database (podman MariaDB, shadow web tree, throwaway host on a locally-administered MAC): an error on a deploy task moved it to 6 and dropped the host's active task count to 0; an error on a wipe task did the same; a warning left the task Queued; every path still answers an identical `##`. Step 339 re-runs clean. All seven new assertions mutation-tested. Co-Authored-By: Claude <noreply@anthropic.com> * Show failed tasks in Task Management's Recent pane A Failed task is in no pane otherwise. It is not an active state, so the active pane excludes it by construction, and the Recent pane -- the only view of finished tasks there is -- asked for Complete and Cancelled only. The state would have existed with nothing in the UI able to show it. Recent's state filter becomes All / Complete / Cancelled / Failed, and the default includes Failed. 'both' becomes 'all' because there are three finished states now and a two-way label was about to start lying; getRecentTasks() still reads any unrecognised value as all-of-them, so a page rendered before this keeps working. FOGBase gains the getFailedState() passthrough its five siblings already have, so pages reach it the same way they reach the others. Verified against the isolated lab database: an error report moved the fixture task to 6 with taskStateChangedTime stamped (Task::set() already does that for any state change), and the pane's own WHERE returns 54 rows where the old one returned 53. Three new assertions mutation-tested. Co-Authored-By: Claude <noreply@anthropic.com> * Show the task log, which nothing has ever displayed taskLog has been written since 1.2 -- a row per task state transition -- and there has never been a way to read it short of SQL. That stopped being merely untidy at schema 338, when FOS reports started landing in the same table: the text a machine sends when a deploy dies was being stored and shown to nobody. Adds a Logs tab to Task Management: time, host, task type, state at the time, entry type, message, who recorded it. Errors and warnings are badged (danger/warning) because the type is what tells an operator whether the machine stopped or carried on. Defaults to reports rather than everything. State rows outnumber reports several-fold -- 52 against 2 on the reference install after two test reports -- and would bury the thing the tab exists to surface; All is one click away, as are errors, warnings and state changes on their own. Read through a derived table: taskLog and tasks both have `taskID` and `taskStateID` columns, and complex() builds its select list as bare backticked names, so an unqualified name across that join is ambiguous and the query dies. Aliasing inside the subquery gives every column a name of its own; MariaDB merges a derived table with no aggregate, so this is not a materialisation. tests/task-log-view.test.php pins the three things that fail silently: a grid column asking for a key the endpoint does not emit, a filter radio posting a value the switch does not handle (which shows the default and looks like it worked), and the table id drifting between the pane and its builder. Six mutations, all caught. Verified against the isolated lab database: the endpoint returns the mapped row shape through complex() with the joins resolving host name, task type and state icons, and each filter's WHERE returns the expected split (2 reports, 1 error, 1 warning, 52 state, 54 all). FOG_BCACHE_VER bumped -- fog.task.list.js changed. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: JJ Fullmer <7743340+darksidemilk@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

mastacontrola mentioned this pull request Aug 19, 2026

Report failures and warnings to the server, tagged with which they are FOGProject/fos#152

Merged

mastacontrola merged commit 16fbe26 into working-1.6 Aug 19, 2026
3 checks passed

mastacontrola deleted the fix-1206-task-error-report branch August 19, 2026 11:07

mastacontrola mentioned this pull request Aug 19, 2026

Give a FOS report somewhere to live: taskLog and a log of its own (schema 338) #1208

Merged

mastacontrola mentioned this pull request Aug 19, 2026

Let 1.5 record what FOS tells it: taskLog gets a type and a body (schema 280) #1210

Merged

mastacontrola mentioned this pull request Aug 19, 2026

Make a FOS failure visible: a Failed task state, and a tab that shows the task log #1211

Merged

Navigation Menu

Uh oh!

Give FOS somewhere to report an imaging failure - #1207

Give FOS somewhere to report an imaging failure#1207
mastacontrola merged 1 commit into
working-1.6from
fix-1206-task-error-report

mastacontrola commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mastacontrola commented Aug 19, 2026

The gap

Deliberately narrow, in three ways

The text

Not a REST route

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants