Skip to content

Make a FOS failure visible: a Failed task state, and a tab that shows the task log - #1211

Merged
mastacontrola merged 3 commits into
working-1.6from
fix-1206-failed-task-state
Aug 19, 2026
Merged

Make a FOS failure visible: a Failed task state, and a tab that shows the task log#1211
mastacontrola merged 3 commits into
working-1.6from
fix-1206-failed-task-state

Conversation

@mastacontrola

@mastacontrola mastacontrola commented Aug 19, 2026

Copy link
Copy Markdown
Member

Follow-up to #1207 / #1208. FOS reports an error, FOG records it — and then nothing about the system changed: the task stayed where it was, and the row it wrote landed in a table nothing has ever displayed.

Two commits, one job: make a FOS failure something you can see.

1. A Failed task state (schema 339)

A task the host died on stayed Queued or In-Progress forever. Host::loadTask() and Host::getActiveTaskCount() count those states as live, so the host could not be re-tasked until somebody noticed and cancelled the task by hand.

  • taskStates row 6, Failed — "Host reported that the task could not be completed."
  • TaskState::getFailedState(), with a FAILED_STATE hook like every other state getter, plus the FOGBase passthrough its five siblings already have
  • TaskError moves the task to it on an error. A warning never does — a warning means FOS carried on.

Not Cancelled, which was the alternative: Cancelled means an administrator stopped it, and losing the difference between "somebody stopped this" and "this broke" costs the operator the one fact they open the task list to find.

Adding a sixth state is additive because every "is this task live" test in the tree is an allowlistgetQueuedStates() plus getProgressState() — so a state nobody listed is inactive by construction. Complete and Cancelled rows already persist in tasks (20 and 33 of them on the reference install), so a Failed row lingering is existing behavior, not new litter.

The state is written for every task type, not just imaging ones — a Memtest the host died on is as finished as a deploy. Only HOST_IMAGE_FAIL stays imaging-specific. _markFailed() is guarded on the taskStates row existing, so a web tree updated ahead of its database leaves the task alone rather than pointing it at a state that is not there. Apart from stateID the endpoint writes nothing about the task — it is unauthenticated, matched by MAC like every FOS endpoint, so what it may change stays a list of one.

Recent pane: the state would otherwise be invisible — not active, and Recent asked for Complete and Cancelled only. Its filter becomes All / Complete / Cancelled / Failed with Failed in the default. (bothall, since a two-way label was about to start lying; the endpoint still reads any unrecognised value as all-of-them.)

2. A tab that shows the task log

taskLog has been written since 1.2 and displayed never. Schema 338 started putting FOS report text in the same table, so the message a machine sends when a deploy dies was being stored and shown to nobody.

New Logs tab on Task Management: time, host, task type, state at the time, entry type, message, recorded by. Errors and warnings are badged, because the type is what tells you whether the machine stopped or carried on. Defaults to reports — state rows outnumber them several-fold and would bury the thing the tab exists to surface — with All / Errors / Warnings / State changes one click away.

Read through a derived table: taskLog and tasks both carry taskID and taskStateID, and complex() builds its select list as bare backticked names, so an unqualified name across that join is ambiguous and the query dies. Aliasing inside the subquery gives every column a name of its own; MariaDB merges a derived table with no aggregate in it.

FOG_BCACHE_VER bumped — fog.task.list.js changed.

Verification

Against an isolated copy of this server's database (podman MariaDB on :13306, shadow web tree, throwaway host on a locally-administered MAC — nothing live was touched):

case result
error on a deploy task state → 6, getActiveTaskCount() → 0, loadTask() invalid
error on a wipe task (non-imaging) state → 6
warning state stays Queued, taskLog row typed warning
every endpoint path, unknown MAC included identical ## body
step 339 run twice converges, taskStates still 6 rows
Recent pane's own WHERE 54 rows where the old one returned 53
getTaskLogs() through complex() correct row shape, joins resolving host name and both icons
each log filter's WHERE 2 reports / 1 error / 1 warning / 52 state / 54 all

Ten new assertions across tests/task-error-report.test.php and the new tests/task-log-view.test.php, every one mutation-tested. Full suite green (56 files).

Downstream

None — no route classes added or removed.

darksidemilk and others added 3 commits August 19, 2026 08:07
A FOS error report is recorded and announced (#1207, #1208), but the task
itself was left exactly where it was: Queued or In-Progress, forever. The
task list still said the machine was working on it, and because Host::
loadTask() and getActiveTaskCount() count those states as live, the host
could not be re-tasked until somebody noticed and cancelled it by hand.

Adds taskStates row 6, Failed, and moves the task to it when FOS reports an
error. A warning never does -- a warning means FOS carried on.

Not Cancelled, which was the alternative. Cancelled means an administrator
stopped it; losing the difference between "somebody stopped this" and "this
broke" costs the operator the one fact they open the task list to find.

Adding a sixth state is additive because every "is this task live" test in
the tree is an allowlist -- getQueuedStates() plus getProgressState() -- so
a state nobody listed is inactive by construction. No call site had to
learn about it, and Complete and Cancelled rows already persist in `tasks`
(20 and 33 of them on the reference install), so a Failed row lingering is
the existing behavior rather than new litter.

The state is written for every task type, not just imaging ones: a Memtest
the host died on is as finished as a deploy. Only HOST_IMAGE_FAIL stays
imaging-specific. Guarded on the taskStates row existing, so a web tree
updated ahead of its database leaves the task alone rather than pointing it
at a state that is not there.

Verified end to end against an isolated copy of this server's database
(podman MariaDB, shadow web tree, throwaway host on a locally-administered
MAC): an error on a deploy task moved it to 6 and dropped the host's active
task count to 0; an error on a wipe task did the same; a warning left the
task Queued; every path still answers an identical `##`. Step 339 re-runs
clean. All seven new assertions mutation-tested.

Co-Authored-By: Claude <noreply@anthropic.com>
A Failed task is in no pane otherwise. It is not an active state, so the
active pane excludes it by construction, and the Recent pane -- the only
view of finished tasks there is -- asked for Complete and Cancelled only.
The state would have existed with nothing in the UI able to show it.

Recent's state filter becomes All / Complete / Cancelled / Failed, and the
default includes Failed. 'both' becomes 'all' because there are three
finished states now and a two-way label was about to start lying;
getRecentTasks() still reads any unrecognised value as all-of-them, so a
page rendered before this keeps working.

FOGBase gains the getFailedState() passthrough its five siblings already
have, so pages reach it the same way they reach the others.

Verified against the isolated lab database: an error report moved the
fixture task to 6 with taskStateChangedTime stamped (Task::set() already
does that for any state change), and the pane's own WHERE returns 54 rows
where the old one returned 53. Three new assertions mutation-tested.

Co-Authored-By: Claude <noreply@anthropic.com>
taskLog has been written since 1.2 -- a row per task state transition -- and
there has never been a way to read it short of SQL. That stopped being
merely untidy at schema 338, when FOS reports started landing in the same
table: the text a machine sends when a deploy dies was being stored and
shown to nobody.

Adds a Logs tab to Task Management: time, host, task type, state at the
time, entry type, message, who recorded it. Errors and warnings are badged
(danger/warning) because the type is what tells an operator whether the
machine stopped or carried on.

Defaults to reports rather than everything. State rows outnumber reports
several-fold -- 52 against 2 on the reference install after two test
reports -- and would bury the thing the tab exists to surface; All is one
click away, as are errors, warnings and state changes on their own.

Read through a derived table: taskLog and tasks both have `taskID` and
`taskStateID` columns, and complex() builds its select list as bare
backticked names, so an unqualified name across that join is ambiguous and
the query dies. Aliasing inside the subquery gives every column a name of
its own; MariaDB merges a derived table with no aggregate, so this is not a
materialisation.

tests/task-log-view.test.php pins the three things that fail silently: a
grid column asking for a key the endpoint does not emit, a filter radio
posting a value the switch does not handle (which shows the default and
looks like it worked), and the table id drifting between the pane and its
builder. Six mutations, all caught.

Verified against the isolated lab database: the endpoint returns the
mapped row shape through complex() with the joins resolving host name,
task type and state icons, and each filter's WHERE returns the expected
split (2 reports, 1 error, 1 warning, 52 state, 54 all).

FOG_BCACHE_VER bumped -- fog.task.list.js changed.

Co-Authored-By: Claude <noreply@anthropic.com>
@mastacontrola mastacontrola changed the title Give a task the host died on a state of its own (schema 339) Make a FOS failure visible: a Failed task state, and a tab that shows the task log Aug 19, 2026
@mastacontrola
mastacontrola merged commit 15eeed5 into working-1.6 Aug 19, 2026
3 checks passed
@mastacontrola
mastacontrola deleted the fix-1206-failed-task-state branch August 19, 2026 13:22
mastacontrola added a commit that referenced this pull request Aug 19, 2026
Ported from 1.6 #1211, completing the port of #1210. A FOS error report is
recorded there, and the task itself was left exactly where it was: Queued or
In-Progress, forever. The task page still said the machine was working on
it, and because Host::loadTask() and getActiveTaskCount() count those states
as live, the host could not be re-tasked until somebody noticed and
cancelled it by hand.

Adds taskStates row 6, Failed, and moves the task to it when FOS reports an
error. A warning never does -- a warning means FOS carried on.

Not Cancelled, which was the alternative. Cancelled means an administrator
stopped it; losing the difference between "somebody stopped this" and "this
broke" costs the operator the one fact they open the task list to find.

Additive on a maintenance branch, which is why it is portable at all: every
"is this task live" test in the tree is an allowlist -- getQueuedStates()
plus getProgressState() -- so a state nobody listed is inactive by
construction, and no call site had to learn about it. 1.5 has no history
pane, so a Failed task is as invisible in the UI as a Complete or Cancelled
one already is; what changes is that the host is free and the task has
stopped claiming otherwise.

The state is written for every task type, not just imaging ones: a Memtest
the host died on is as finished as a deploy. Only HOST_IMAGE_FAIL stays
imaging-specific. Guarded on the taskStates row existing, so a web tree
updated ahead of its database leaves the task alone rather than pointing it
at a state that is not there.

Verified against an isolated copy of the 1.5 database (podman MariaDB,
shadow web tree, throwaway host on a locally-administered MAC, 2079 real
hosts): a warning left the task Queued and wrote a row typed warning; an
error moved it to 6 and dropped the host's active task count to zero; error,
warning and unknown-MAC all still answer an identical `##`. Step 281 applies
clean on a database at 278. Seven assertions mutation-tested.

Co-authored-by: JJ Fullmer <7743340+darksidemilk@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
mastacontrola added a commit that referenced this pull request Aug 19, 2026
v1.6.11 carries FOGProject/fog-plugins#21, which was merged the morning after
v1.6.10 was cut and so has never reached a server.

That change matters more than its own PR made it sound, because the FOS
reporting work landed in between. #1206/#1211/#1217/#1223 give a failed task a
stored, multi-line report of what FOS actually said, and TaskError sends the
flattened opening of it as HOST_IMAGE_FAIL's Reason -- but every bundled
listener on v1.6.10 ignores that key and pushes the fixed string "This host
has failed to image".

Confirmed live before cutting the release. A report whose stored row read

    fog.download: failed to restore partition 2
      partclone.ntfs: /dev/sda2 is busy
      ERROR: win11-split part 2 checksum mismatch
      exit code 1 (fog.download)

pushed, in its entirety: "fos-deploy-test Failed" / "This host has failed to
image". So the whole point of storing the trace stopped at the server.

One line, because the sha256 is fetched from the release alongside the
tarball rather than pinned here; bin/fetch-plugins.sh reads this constant and
verifies what it downloads against it. Verified end to end on this pin:
"Fetching plugins v1.6.11" -> "Plugins at v1.6.11", and all three imagefail
listeners in the fetched tree read Reason.

The 1.5 line keeps its plugins in-tree and has no pin, so its equivalent is a
code change: #1226.

Co-authored-by: JJ Fullmer <7743340+darksidemilk@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants