fix(ci): the icarus gate was blind to every test failure (missing pipefail) - #374
Merged
Conversation
…efail)
The required 'GF-T spec KATs under Icarus Verilog' check could not fail
on a failing spec. Its loop is
if ! $T27C icarus-simulate specs/${spec}.t27 | tee /tmp/${spec}.simlog; then
and a bash pipeline's exit status is the LAST command's -- tee, which
always succeeds. No 'shell:' or 'defaults:' is declared, so GitHub's
default 'bash -e {0}' applies WITHOUT pipefail. t27c correctly exits 1
and prints 'Error: Icarus simulation reported test/bench failures', but
the pipe swallowed it and the gate reported green.
Demonstrated: injecting a failing assertion into crc16 and running the
exact CI construct yields STATUS=0 -- a false green. With
'set -o pipefail' the same input yields STATUS=1.
So the gate effectively only ran the bench budgets and the wave-55
coverage guard. Every 'N specs pass under icarus' claim rested on local
verification, not on this check. This is the broken-ruler failure mode:
the instrument that was supposed to measure correctness could not
report incorrectness.
Fixes both pipe-swallowing sites:
- gft-verilog-kat.yml: the icarus simulate loop (the critical one)
- spec-drift-guard.yml: the typecheck '| grep -q' loop, same class
The typecheck change was verified safe first: across all 107 specs the
t27c exit code and the grep verdict never diverge, so pipefail cannot
introduce a false positive there.
Verified both directions: an injected failing assertion is now DETECTED
(STATUS=1, previously 0), and the healthy 107-spec corpus still
simulates green under pipefail (STATUS=0) -- no false positives.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The required icarus check could not fail on a failing spec
Its loop is:
A bash pipeline's exit status is the last command's —
tee, which always succeeds. Noshell:ordefaults:is declared, so GitHub's defaultbash -e {0}applies without pipefail.t27ccorrectly exits 1 and printsError: Icarus simulation reported test/bench failures, but the pipe swallowed it and the gate reported green.Demonstrated: injecting a failing assertion into
crc16and running the exact CI construct yieldsSTATUS=0— a false green. Withset -o pipefailthe same input yieldsSTATUS=1.So the gate effectively only ran the bench budgets and the wave-55 coverage guard. Every "N specs pass under icarus" claim rested on local verification, not on this check. This is the broken-ruler failure mode: the instrument meant to measure correctness could not report incorrectness.
Fixes both pipe-swallowing sites
gft-verilog-kat.yml— the icarus simulate loop (the critical one)spec-drift-guard.yml— the typecheck| grep -qloop, same classThe typecheck change was verified safe first: across all 107 specs the t27c exit code and the grep verdict never diverge, so pipefail cannot introduce a false positive there.
Verified both directions
🤖 Generated with Claude Code