Let a failed remote delivery leave a trace on the host - #396
Conversation
Delivery ends in `|| true` on purpose: a session without its briefing is still a session, and a launch must never be blocked by one. But it also ended in `>/dev/null 2>&1`, and that is a different decision wearing the same clothes. When #395's SyntaxError made every delivery a no-op, the reason was written to /dev/null on five days' worth of dials, so neither machine held a word about why every remote host was empty. Failure stays non-fatal. Its stderr now reaches the host's own dial log as `delivery install failed <reason>`, beside the launch decisions it belongs next to. The reason is flattened to one line, bounded, and taken from the *tail* — a python traceback ends with the line that names the fault, and keeping the head threw exactly that away: measured, `NotADirectoryError` fell outside the first 400 bytes of the very failure this was written to explain. A succeeding delivery still writes nothing. The sweep dials every host every minute, and a line per healthy tick would bury the one that isn't. `aFailedDeliveryLeavesNoStampBehind` asserted `!script.contains("printf")` as a proxy for "the shell must not write the stamp". It now asserts the rule itself: every printf in the fragment targets dials.log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: scgopi <scgopireddy@gmail.com>
Five findings from an adversarial review of the previous commit, four of them reproduced end to end. `tail -c` cuts bytes, so the detail landed mid-character on any host with a non-ASCII path, and one orphan continuation byte makes grep and sed fail on the *whole* file under a UTF-8 locale — a change written to make one failure legible would have hidden every dial on the host. The cut now passes through `iconv -c`, and a test writes a failure whose path runs through the boundary and greps the log afterwards. The 400-byte detail also broke DialLog's own bound. The trim keeps the last `keptLines` lines, which stands in for a byte budget only while a line stays under `maxBytes / keptLines` = 209 bytes; at 445 the trim can never get back under `maxBytes`, so every later append by every loop on that host re-reads and rewrites a 2.2 MB file forever. The size is now derived from that budget rather than chosen, and `DialLogBoundTests` locks the arithmetic so raising `keptLines` breaks there first. The trim's scratch file was a fixed `dials.log.tmp` — a race every loop on a host shares, measured at 40 concurrent fragments turning a 5000-line log into 34. Rare enough to survive before; this commit made it fire on every append, so the name is now per-process. A non-zero exit with empty stderr logged no reason at all. An OOM-killed installer is the realistic case, and the exit code is the one datum always available, so it falls back to `rc=<code>`. Last, the test I changed in the previous commit was a loose proxy that accepted what the old line forbade: the text after a rogue write picks up `dials.log` from the next fragment. It now asserts the rule directly — no redirect names the stamp, and every append targets the dial log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: scgopi <scgopireddy@gmail.com>
Review findings addressed —
|
| # | Finding | Fix |
|---|---|---|
| F1 HIGH | tail -c 400 splits UTF-8 mid-character; one orphan byte makes grep/sed fail on the whole log |
cut piped through iconv -c -f UTF-8 -t UTF-8 |
| F2 HIGH | 445-byte lines vs a 209-byte per-line budget — keptLines × 445 = 2.2 MB against a 1 MB cap, so the trim can never get under it again |
detail size derived from maxBytes / keptLines, locked by DialLogBoundTests |
| F3 MED | fixed dials.log.tmp name — a race this PR made fire on every append; 40 concurrent fragments left 34 of 5000 lines |
per-process $$ scratch name, trim factored into one place |
| F4 MED | non-zero exit with empty stderr logged no reason and dropped the exit code | falls back to rc=<code> |
| F5 TEST | my replacement assertion was a loose proxy that accepted what the old line forbade | asserts the rule directly: no redirect names the stamp, every append targets the dial log |
F1 deserves emphasis: it would have made this change net-negative. A feature written to make one failure legible would have made every other dial entry on that host unfindable. Confirmed here — grep -c "ensure fresh" returned nothing on a log that visibly contained the line.
F5 is mine to own. The previous commit's body claimed that assertion "asserts the rule itself". It did not, and the counterexample is a rogue printf 'stamp' > …/.shim-stamp; passing because the text after it picks up dials.log from the next fragment's trim.
The review also cleared five properties by running them: no variable collision with the ensure's own gc_rc/gc_boot/gc_last/GRAPHCODE_RESUME_ID; the exit-status contract holds through the full exec zsh -l -i -c round trip and inside { install || { log; false; }; } && launch; printf format escape is impossible; no file contents reach the log; and a succeeding delivery writes nothing.
Also hardened without a finding: \r is now stripped, since tr handled only \n and \t.
Verification
| Step | Result |
|---|---|
xcodebuild -scheme graphcode test |
exit=0, ** TEST SUCCEEDED **, Test run with 1935 tests in 204 suites passed |
graphcode-cli / graphcoded builds |
exit=0 each |
swiftlint / swift format lint --strict |
0 errors / exit=0 |
This PR is deliberately not going into the 0.1.74 stable. Nothing here is user-visible; the fix users are waiting on is #395, already in 0.1.74-beta1. It rides the release after.
Re-review at 80a088a: the derived budget was still over by one byte. `wc -c` is what the trim measures the file with and it counts the terminator, so a line computed as exactly 209 bytes was 210 on disk and 5000 of them came to 1,050,000 against a 1,048,576 cap — the permanent re-trim was still reachable, just 2.4x further away. The overhead literal now carries its own `\n`, so the budget and the trim measure the same thing. The test written to lock that arithmetic left the newline out the same way and passed on the bug, which is the real lesson: it recomputed the line instead of looking at one. There is now a test that runs a failing delivery and measures the bytes the fragment actually wrote. Also from the re-review: - `iconv` is glibc's; musl and busybox images have none, and there the detail degraded to the exit code alone. Falls back to stripping high bytes, which costs a non-ASCII path its accents and keeps the log valid, which is the property worth keeping. - The stamp assertion matched only the tilde spelling, so a write using `$HOME` — this codebase's own spelling, the one `DialLog` uses — walked past it, as did two spaces or an fd. It matches any redirect at a target ending in the stamp's name now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: scgopi <scgopireddy@gmail.com>
Follow-up to #395. That fixed the broken installer; this fixes the reason nobody noticed for five days.
The decision worth separating
The delivery fragment ended
>/dev/null 2>&1 || true. Those are two different decisions wearing the same clothes:|| true— keep. A session without its briefing is still a session, and delivery must never block the launch it precedes.>/dev/null 2>&1— wrong. When the embedded python raisedSyntaxErroron every dial, the reason went to/dev/null. Neither machine held a word about why every remote host was empty.The bug was never that the install could fail. It was that failure left no trace anywhere.
Change
Failure stays non-fatal, and now reports to the host's own dial log beside the launch decisions it belongs next to:
dials.logsplits on newlines.NotADirectoryErrorfell outside the first 400 bytes of the very failure this was written to explain.printfargument, not inside the format — unlikesession/dial/event, which are literals this codebase controls. A%sin a remote python's error must not be able to reformat the line it is written to.One existing test changed
aFailedDeliveryLeavesNoStampBehindasserted!script.contains("printf")as a proxy for "the shell must not write the stamp". Aprintfnow legitimately appears, so it asserts the rule itself: everyprintfin the fragment targetsdials.log. The stamp assertions around it are untouched.TDD evidence
RED: xcodebuild -only-testing:graphcodeTests/RemoteInstallerExecutionTests with main's RemoteGraphAccess.swift and DialLog.swift in place -> exit=65, aFailedDeliveryLeavesItsReasonInTheHostsDialLog fails because no dials.log exists at all
GREEN: xcodebuild -scheme graphcode -destination platform=macOS test -> exit=0, ** TEST SUCCEEDED **, Test run with 1929 tests in 203 suites passed
REGRESSION: full suite plus graphcode-cli and graphcoded builds, swiftlint and swift format lint -> exit=0 on every step, 0 lint errors, the 1927 tests from the 0.1.74-beta1 tree all still pass