Skip to content

Let a failed remote delivery leave a trace on the host - #396

Merged
scgopi merged 3 commits into
mainfrom
fix/delivery-leaves-a-trace
Sep 20, 2026
Merged

scgopi merged 3 commits into
mainfrom
fix/delivery-leaves-a-trace

Conversation

@scgopi

@scgopi scgopi commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Follow-up to #395. That fixed the broken installer; this fixes the reason nobody noticed for five days.

The decision worth separating

The delivery fragment ended >/dev/null 2>&1 || true. Those are two different decisions wearing the same clothes:

  • || truekeep. A session without its briefing is still a session, and delivery must never block the launch it precedes.
  • >/dev/null 2>&1wrong. When the embedded python raised SyntaxError on every dial, the reason went to /dev/null. Neither machine held a word about why every remote host was empty.

The bug was never that the install could fail. It was that failure left no trace anywhere.

Change

Failure stays non-fatal, and now reports to the host's own dial log beside the launch decisions it belongs next to:

2026-09-20T21:38:23Z delivery install failed NotADirectoryError: [Errno 20] Not a directory: ...
  • Flattened to one line — every reader of dials.log splits on newlines.
  • Bounded at 400 bytes, and taken from the tail. A python traceback opens with frames and interpreter paths and ends with the line that names the fault. Keeping the head threw the answer away: measured, NotADirectoryError fell outside the first 400 bytes of the very failure this was written to explain.
  • The detail rides as a printf argument, not inside the format — unlike session/dial/event, which are literals this codebase controls. A %s in a remote python's error must not be able to reformat the line it is written to.
  • A succeeding delivery writes nothing. The sweep dials every host every minute; a line per healthy tick would bury the one that isn't.

One existing test changed

aFailedDeliveryLeavesNoStampBehind asserted !script.contains("printf") as a proxy for "the shell must not write the stamp". A printf now legitimately appears, so it asserts the rule itself: every printf in the fragment targets dials.log. The stamp assertions around it are untouched.

TDD evidence

RED: xcodebuild -only-testing:graphcodeTests/RemoteInstallerExecutionTests with main's RemoteGraphAccess.swift and DialLog.swift in place -> exit=65, aFailedDeliveryLeavesItsReasonInTheHostsDialLog fails because no dials.log exists at all

GREEN: xcodebuild -scheme graphcode -destination platform=macOS test -> exit=0, ** TEST SUCCEEDED **, Test run with 1929 tests in 203 suites passed

REGRESSION: full suite plus graphcode-cli and graphcoded builds, swiftlint and swift format lint -> exit=0 on every step, 0 lint errors, the 1927 tests from the 0.1.74-beta1 tree all still pass

scgopi and others added 2 commits September 20, 2026 14:45
Delivery ends in `|| true` on purpose: a session without its briefing is
still a session, and a launch must never be blocked by one. But it also
ended in `>/dev/null 2>&1`, and that is a different decision wearing the
same clothes. When #395's SyntaxError made every delivery a no-op, the
reason was written to /dev/null on five days' worth of dials, so neither
machine held a word about why every remote host was empty.

Failure stays non-fatal. Its stderr now reaches the host's own dial log
as `delivery install failed <reason>`, beside the launch decisions it
belongs next to. The reason is flattened to one line, bounded, and taken
from the *tail* — a python traceback ends with the line that names the
fault, and keeping the head threw exactly that away: measured,
`NotADirectoryError` fell outside the first 400 bytes of the very
failure this was written to explain.

A succeeding delivery still writes nothing. The sweep dials every host
every minute, and a line per healthy tick would bury the one that isn't.

`aFailedDeliveryLeavesNoStampBehind` asserted `!script.contains("printf")`
as a proxy for "the shell must not write the stamp". It now asserts the
rule itself: every printf in the fragment targets dials.log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: scgopi <scgopireddy@gmail.com>
Five findings from an adversarial review of the previous commit, four of
them reproduced end to end.

`tail -c` cuts bytes, so the detail landed mid-character on any host with
a non-ASCII path, and one orphan continuation byte makes grep and sed
fail on the *whole* file under a UTF-8 locale — a change written to make
one failure legible would have hidden every dial on the host. The cut now
passes through `iconv -c`, and a test writes a failure whose path runs
through the boundary and greps the log afterwards.

The 400-byte detail also broke DialLog's own bound. The trim keeps the
last `keptLines` lines, which stands in for a byte budget only while a
line stays under `maxBytes / keptLines` = 209 bytes; at 445 the trim can
never get back under `maxBytes`, so every later append by every loop on
that host re-reads and rewrites a 2.2 MB file forever. The size is now
derived from that budget rather than chosen, and `DialLogBoundTests`
locks the arithmetic so raising `keptLines` breaks there first.

The trim's scratch file was a fixed `dials.log.tmp` — a race every loop
on a host shares, measured at 40 concurrent fragments turning a
5000-line log into 34. Rare enough to survive before; this commit made it
fire on every append, so the name is now per-process.

A non-zero exit with empty stderr logged no reason at all. An OOM-killed
installer is the realistic case, and the exit code is the one datum
always available, so it falls back to `rc=<code>`.

Last, the test I changed in the previous commit was a loose proxy that
accepted what the old line forbade: the text after a rogue write picks up
`dials.log` from the next fragment. It now asserts the rule directly —
no redirect names the stamp, and every append targets the dial log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: scgopi <scgopireddy@gmail.com>
@scgopi

scgopi commented Sep 20, 2026

Copy link
Copy Markdown
Owner Author

Review findings addressed — 80a088a9

ReviewPR396 reviewed this adversarially and found five defects, four reproduced end to end. I independently reproduced the two HIGH ones before acting. All five are fixed.

# Finding Fix
F1 HIGH tail -c 400 splits UTF-8 mid-character; one orphan byte makes grep/sed fail on the whole log cut piped through iconv -c -f UTF-8 -t UTF-8
F2 HIGH 445-byte lines vs a 209-byte per-line budget — keptLines × 445 = 2.2 MB against a 1 MB cap, so the trim can never get under it again detail size derived from maxBytes / keptLines, locked by DialLogBoundTests
F3 MED fixed dials.log.tmp name — a race this PR made fire on every append; 40 concurrent fragments left 34 of 5000 lines per-process $$ scratch name, trim factored into one place
F4 MED non-zero exit with empty stderr logged no reason and dropped the exit code falls back to rc=<code>
F5 TEST my replacement assertion was a loose proxy that accepted what the old line forbade asserts the rule directly: no redirect names the stamp, every append targets the dial log

F1 deserves emphasis: it would have made this change net-negative. A feature written to make one failure legible would have made every other dial entry on that host unfindable. Confirmed here — grep -c "ensure fresh" returned nothing on a log that visibly contained the line.

F5 is mine to own. The previous commit's body claimed that assertion "asserts the rule itself". It did not, and the counterexample is a rogue printf 'stamp' > …/.shim-stamp; passing because the text after it picks up dials.log from the next fragment's trim.

The review also cleared five properties by running them: no variable collision with the ensure's own gc_rc/gc_boot/gc_last/GRAPHCODE_RESUME_ID; the exit-status contract holds through the full exec zsh -l -i -c round trip and inside { install || { log; false; }; } && launch; printf format escape is impossible; no file contents reach the log; and a succeeding delivery writes nothing.

Also hardened without a finding: \r is now stripped, since tr handled only \n and \t.

Verification

Step Result
xcodebuild -scheme graphcode test exit=0, ** TEST SUCCEEDED **, Test run with 1935 tests in 204 suites passed
graphcode-cli / graphcoded builds exit=0 each
swiftlint / swift format lint --strict 0 errors / exit=0

This PR is deliberately not going into the 0.1.74 stable. Nothing here is user-visible; the fix users are waiting on is #395, already in 0.1.74-beta1. It rides the release after.

Re-review at 80a088a: the derived budget was still over by one byte.
`wc -c` is what the trim measures the file with and it counts the
terminator, so a line computed as exactly 209 bytes was 210 on disk and
5000 of them came to 1,050,000 against a 1,048,576 cap — the permanent
re-trim was still reachable, just 2.4x further away.

The overhead literal now carries its own `\n`, so the budget and the
trim measure the same thing.

The test written to lock that arithmetic left the newline out the same
way and passed on the bug, which is the real lesson: it recomputed the
line instead of looking at one. There is now a test that runs a failing
delivery and measures the bytes the fragment actually wrote.

Also from the re-review:

- `iconv` is glibc's; musl and busybox images have none, and there the
  detail degraded to the exit code alone. Falls back to stripping high
  bytes, which costs a non-ASCII path its accents and keeps the log
  valid, which is the property worth keeping.
- The stamp assertion matched only the tilde spelling, so a write using
  `$HOME` — this codebase's own spelling, the one `DialLog` uses — walked
  past it, as did two spaces or an fd. It matches any redirect at a
  target ending in the stamp's name now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: scgopi <scgopireddy@gmail.com>
@scgopi
scgopi merged commit bef14ad into main Sep 20, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant