Skip to content

Add style checks and fix what they exposed - #333

Merged
jserv merged 6 commits into
masterfrom
hook
Sep 7, 2026
Merged

Add style checks and fix what they exposed#333
jserv merged 6 commits into
masterfrom
hook

Conversation

@jserv

@jserv jserv commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

This branch gives shecc one way to check its own style, runnable identically by a contributor and by CI, and then follows through on what became visible once those checks could be trusted. make check-style covers trailing newlines, clang-format 20, commentflow comment reflow, and shfmt with ShellCheck; make indent rewrites exactly the set the checks cover; make install-hooks puts the same checks in front of a commit without making the tools mandatory to have; and make check-hooks tests that machinery.

The recurring defect this turned up is a check that reports success without having checked. check-newline.sh called file(1) without verifying it was installed, so a machine lacking it blocked commits on a bogus violation instead of standing aside, and it never noticed a short reply that would silently reclassify the rest of the tree. All four checkers sourced common.sh without checking that it loaded. Separately, make sanitizer shared its object file with the ordinary build, so after any plain build -- which make check performs -- it relinked an uninstrumented object against the sanitizer runtime and make check-sanitizer ran the whole suite having intercepted nothing. Counted in symbols: 32 __asan references after a clean sanitizer build, 1 after a plain one.

Removing -Wno-uninitialized needs a __noreturn macro on the three diagnostic exits, and that macro expands to nothing under __SHECC__ -- which is how a latent preprocessor bug surfaced. shecc hangs or crashes on any macro with an empty replacement list, ordinary C that the tree had simply never contained. pp_preprocess_internal() leaves its list head uninitialized and hands the caller a pointer into a frame that is about to die, so an empty expansion builds a cycle in the token list. It presented as a self-compile spinning forever in pp_strip_layout(), nowhere near the preprocessor.

With that fixed, all ten -Wno- flags come out of CFLAGS_TO_CHECK. Four hid nothing on either compiler; the rest were hiding work, including an uninitialized read in build_r_idom() that its forward twin has always guarded against. Twenty-four sites also printed a diagnostic and then abort()ed, which discards the buffer, so the MAX_TYPES guard wrote zero bytes into a build log, and error_at() now exits rather than aborting, finishing for source errors what c77b8be started for the command line.

The last commit splits the four functions that had grown past reading: emit_ph2_ir() 1543 lines to 398, read_body_statement() 896 to 63, reg_alloc() 830 to 155, lex_token() 737 to 28. It is code motion, and was checked as such rather than asserted: a frozen copy of the compiler's own source, compiled by the compiler built before and after the split, is byte-identical on all three targets. That is a stronger claim than the suite makes, since stage1 == stage2 only proves the compiler is deterministic.

Verified on arm, riscv and x64: each bootstraps with stage1 byte-identical to stage2 and passes all four suites. gcc and clang both report zero warnings on all three targets with nothing suppressed. Sixteen adversarial inputs -- deep nesting, long identifiers, unterminated literals, exhausted limits -- draw no AddressSanitizer or UndefinedBehaviorSanitizer report from a build that is now genuinely instrumented. Every commit in the series builds and passes make check-style on its own.

Deliberately left out: the file-scope objects stay as they are. There are 120, not the larger number a naive grep suggests; none is dead and none is used by only one function, so the count reflects genuinely shared state, and the context-struct alternative would spend one of only eight parameter slots on every signature.


Summary by cubic

Adds a style-checking pipeline (make check-style, make indent, make install-hooks, make check-hooks) that CI and contributors run identically, then fixes the real defects the checks exposed: the sanitizer build ran uninstrumented no matter how it was reached, empty macro expansions crashed the preprocessor, and ten -Wno- flags were hiding warnings including an uninitialized read. The last commit splits four oversized functions as pure code motion.

Style checks

  • make check-style covers newlines, clang-format 20, commentflow reflow, and shfmt/ShellCheck; make indent rewrites exactly what the checks cover and now also adds final newlines.
  • Checks exit 0 clean, 1 violation, 2 could-not-run, and only 1 blocks a commit; missing tools, an unreadable index, and sparse-checkout paths all stand aside instead of reporting clean.
  • File enumeration now happens in the current shell so a git failure actually exits 2, collects only regular files, and cleans up its temp listing on interrupt; the pre-commit hook treats only exit 2 as advisory, and file lists are passed with -- to avoid dash-prefixed paths.
  • make install-hooks refuses to link a hook a linked worktree would silently run as nothing and never replaces an existing hook; make check-hooks tests both refusals and the staged-snapshot judging.

Bugs the checks exposed

  • make sanitizer and make check-sanitizer both now carry the sanitizer flags and build into out/sanitize, so instrumentation can no longer drop out of either entry point.
  • Macros with empty replacement lists hung or crashed shecc by building a cycle in the token list; the empty splice is now skipped.
  • Removed all ten -Wno- flags; the hidden warnings included an uninitialized read in build_r_idom(), 134 opcode bytes implicitly narrowed in emit_byte() (now int), and thirteen () declarations.
  • error_at() and its fallbacks now exit with status 1 instead of aborting, and the print-then-abort sites flush so diagnostics reach build logs.
  • Split emit_ph2_ir(), read_body_statement(), reg_alloc(), and lex_token(); output is byte-identical on all three targets.

Written for commit 8672866. Summary will update on new commits.

Review in cubic

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added a commit that referenced this pull request Sep 7, 2026
The previous commit gave the sanitizer build its own objects, which fixed
"make sanitizer" and left "make check-sanitizer" as broken as it found
it. A target-specific variable reaches its own target and that target's
prerequisites, and the flags were declared only on "sanitizer", so
reaching $(OUT)/$(STAGE0)-sanitizer through check-sanitizer built
$(SAN_OBJS) with the ordinary CFLAGS.

Run on its own, "make check-sanitizer" therefore compiled the whole
suite against a binary with no instrumentation in it at all: zero __asan
references, against 32 for a real sanitizer build. It passed, as it was
always going to.

The flag lists move into SAN_CFLAGS and SAN_LDFLAGS and both goals carry
them, so either entry point gives 32.

Reported by cubic on #333.
jserv added a commit that referenced this pull request Sep 7, 2026
Six more of them, in the same family as the ones already fixed here.

list_files() let git ls-files fail into a process substitution, where
the status is discarded: an index git cannot read produced an empty file
set, which every checker reads as nothing to complain about. It exits 2
now.

check-shell.sh threw away a verdict it already had. A missing ShellCheck
exits 2 so the hook stands aside, but shfmt runs first, and a real
formatting violation was being relabelled "could not run" and waved
through. A violation now outranks the linter that could not run.

check-format.sh handed clang-format paths with no file behind them --
a sparse checkout, or a deletion staged but not committed -- and failed
the style check on the resulting error.

check-newline.sh read tail's output without its status, so an unreadable
file looked like one ending in a newline. It also grows --write, and
"make indent" calls it, because indent is meant to apply what
check-style verifies and the final-newline rule was the one part it left
for the developer. A carriage return stays reported rather than
rewritten: that is an edit to content, not to layout.

git-pre-commit.sh filtered staged paths with ACMR, so a file that
changed type was staged and never checked.

test-git-hooks.sh probed the default tool names rather than the ones the
checkers resolve, rejecting a valid COMMENTFLOW or SHFMT override, and
did not probe file(1) at all -- which check-newline.sh now requires, and
whose absence would have surfaced as an unrelated assertion failure.
CONTRIBUTING.md says the same thing to the reader.

Reported by cubic on #333.
jserv added a commit that referenced this pull request Sep 7, 2026
error_at() had two fallbacks that disagreed after the exit contract
changed. A NULL location exits, but a source file no longer in
SRC_FILE_MAP still reached fatal() and its core dump. Neither says
anything about what went wrong: the caller reached error_at() because it
found a mistake in the input, and losing the text to quote changes what
can be shown, not what happened. Both exit now.

elf_write_all() has two definitions under HOST_BUFFERED_STDIO, and only
the one that gets compiled here picked up the const on buf. Neither
writes through it, so both carry it.

The four statement handlers extracted from read_body_statement() each
ended in a return the extraction appended, which no path reaches. That
they are unreachable is also the proof the extraction was sound: gcc
does not report control reaching the end of any of them once the
appended returns are gone, so no caller ever fell through the original
if-block into the statement kinds below it.

Reported by cubic on #333.
jserv added a commit that referenced this pull request Sep 7, 2026
Three section headings were adjacent comment blocks, and commentflow
merged each into the sentence beneath it: "div/mod emulation Preserve
the values of the dividend and divisor", "Default capacities for common
data structures Arena sizes optimized based on typical usage patterns",
and "variable definition Depth of the SSA renaming stack".

Two read as one thought once the heading becomes the start of the
sentence, so they are reworded rather than split, which also survives
the next reflow. The third heading labels the whole variable-definition
region of defs.h rather than the constant beneath it, so it keeps its
own block and gains the blank line that stops commentflow reaching past
it.

Reported by cubic on #333.
jserv added a commit that referenced this pull request Sep 7, 2026
.git/hooks is shared across linked worktrees, so the link has to name
the main worktree, and the main worktree may be on a branch that does
not carry the script. git runs a dangling hook by running nothing and
saying nothing, so "make install-hooks" from a linked worktree could
report success and leave every worktree unchecked.

Installing now checks the target exists and exits 2 with what to do
about it. Making the shared hook a dispatcher that resolves the
committing worktree would remove the constraint rather than report it,
but that is a design change and this is not the branch for it.

Reported by cubic on #333.
cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added a commit that referenced this pull request Sep 7, 2026
The enumeration guard added in 8394174 did not work. list_files() ran
inside "< <(...)", so its exit 2 ended the subshell, the loop saw end of
input, and an enumeration that never happened arrived as an empty array
-- which every checker reads as a clean tree. The bug being fixed was
still there, one level further in.

collect_files() replaces it and fills FILES in the current shell, taking
the listing through a temporary file so git's status can be read where
it can still stop anything. All four checkers now exit 2 when git cannot
enumerate the worktree, verified with a git that fails on ls-files. It
also drops index entries with no file behind them, which check-format.sh
was working around on its own.

Three more of the same shape. git-pre-commit.sh treated every status but
0 and 1 as "tool unavailable", so a checker killed by a signal was a
note and the commit went through; only 2 is advisory now. shfmt,
ShellCheck and commentflow were handed file lists with no "--" in front,
so a path beginning with a dash would have been read as an option.
install-git-hooks.sh checked that its source exists but not that it can
be run, and git ignores a hook it cannot execute without saying so.

Two comments in parser.c that the reflow welded onto the function
beneath them are deleted rather than separated: both were already
orphaned on master, labelling functions that are not there --
"Helper function to calculate element size for pointer operations" sits
above accept_compound_assign_op(), and "if first token is type" above
read_global_init(). Restoring the boundary would preserve a comment that
lies about the code under it.

Reported by cubic on #333.
cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added a commit that referenced this pull request Sep 7, 2026
The two paths added in 0e73408 and 2a9c02c refuse to install a hook that
git would ignore, and neither had a test. On a branch about checks that
report success without checking, that is the wrong thing to leave
uncovered. test-git-hooks.sh now makes the source non-executable, then
moves it away, and asserts the installer exits 2 and leaves no hook
behind in either case; it also passes a stray argument to --uninstall.
Each assertion fails if its guard is removed.

collect_files() filtered with -e, which accepts a directory: a submodule
gitlink or a symlink to one is an index entry no file-oriented checker
can read. It takes -f now.

Its temporary listing is also removed through an EXIT trap rather than
only on the paths that return normally, matching check-format.sh and
check-newline.sh, so an interrupted run cannot leave it behind. The trap
is cleared before returning, which is safe because every caller installs
its own afterwards.

Reported by cubic on #333.
Mechanical result of running "make indent" over the tree, so that the
checks added in the next commit start from a clean base.

No C code changes. Verified rather than asserted: with comments stripped
and whitespace normalized, all 21 touched files under src, lib, tools
and tests are byte-identical to their previous revisions.

Reflow merges a comment block into the one beneath it, which is right
when they are one thought and wrong when the first is a section heading.
Five headings are handled here rather than left to read as run-on
sentences. Three are reworded into the sentence that follows, so that a
later reflow leaves them alone. One keeps its own block and gains a
blank line, because it labels a region rather than the constant under
it. Two are deleted: they were already orphaned before this branch,
naming functions that are not there.

The shell reformatting is shfmt reading .editorconfig: function braces
move to their own line, case arms get their own ";;", and redirections
gain a space. Three test scripts also had a "Check result" heading
absorbed into the sentence beneath it; those are split apart again.
The coding convention job installed clang-format-18 inline and ran two
scripts that between them could not fail usefully: check-format.sh
counted XML replacement tags to build an exit code, and check-newline.sh
covered only src, tools and tests. Neither had a local equivalent, so
the first time a contributor learned their patch was misformatted was
after pushing it.

Four checks now live behind "make check-style", each a script that CI
and a developer run the same way: trailing newlines and LF endings,
clang-format 20, commentflow comment reflow, and shfmt plus ShellCheck
for the shell. They share one file-discovery helper, so the set "make
indent" rewrites is exactly the set the checks cover, and a new
untracked file is held to the same rule that formats it.

Every check honors one exit-code contract: 0 clean, 1 violation, 2 could
not run. That third value is what lets "make install-hooks" put a
pre-commit hook in front of the same checks without making the style
tools mandatory to commit. The hook judges a snapshot of the index
rather than the working tree, so a partially staged file is checked as
committed, and a checker whose tool is missing prints a note and stands
aside. CI remains the authority. An existing local hook is never
replaced, and a hook git would ignore, because its source is missing or
not executable, is refused rather than installed.

The contract only means something if nothing can report a clean tree
without having looked at one, so the ways that could happen are closed
here rather than discovered later. Enumeration failures exit 2 instead
of yielding an empty list, and the listing is read in the shell that
acts on it, because a failure inside a process substitution ends only
the subshell. A formatting violation already found outranks a linter
that could not run. Tool absence is probed rather than assumed, and only
status 2 is advisory, so a checker killed by a signal blocks. File lists
are passed after "--", cover regular files only, and skip index entries
with no file behind them.

The tools come from their own releases with the published sha256
verified, rather than from apt: a formatter the distribution moves out
from under the tree turns "make check-style" into a disagreement between
two machines. clang-format is the exception, pinned to major version 20
through the LLVM apt repository via a keyring holding only the
fingerprint we name.
"make sanitizer" adds the instrumentation through a target-specific
CFLAGS, but both it and the ordinary link depended on the same
out/src/main.o, so whichever ran last decided what the other got.

One direction is loud: "make" after "make sanitizer" fails at the link
on undefined __asan_ and __ubsan_ symbols.

The other is silent, and is the one that matters. "make sanitizer" after
any ordinary build, which "make check" performs, finds the object newer
than its source, relinks it against the sanitizer runtime without
rebuilding it, and produces a binary that intercepts nothing. Counted in
symbols: 32 __asan references after a clean sanitizer build, 1 after a
plain build followed by "make sanitizer". So "make check-sanitizer" ran
the whole suite and reported success having checked nothing.

A target-specific variable also reaches only its own target and that
target's prerequisites, so reaching the sanitizer binary through
check-sanitizer built its objects with the ordinary flags: run on its
own, that goal instrumented nothing at all, zero __asan references.

The sanitizer objects now live under out/sanitize, and both goals carry
the flags, so every entry point gives 32 and either build order works.
"#define EMPTY" is ordinary C, and shecc hangs or crashes on it. So does
a function-like macro with an empty body. The tree has never contained
one, which is why it went unnoticed until a header wanted an attribute
macro that expands to nothing under __SHECC__.

pp_preprocess_internal() assembles its result behind a local list head
and hands the caller two things: the first token, and end_of_token for
splicing the next one on. An expansion that produces no tokens gets both
wrong. The head is never initialized, so the return value is whatever
the stack happened to hold, and end_of_token is left pointing at that
head, a local of a frame that is about to go away, which all six callers
write through.

What the caller then builds is a token list with a cycle in it. The
crash is wherever the cycle is first walked, so it surfaced as a
self-compile spinning forever in pp_strip_layout(), nowhere near the
preprocessor.

The head is initialized, end_of_token is NULL rather than a dead pointer
when nothing was produced, and every caller skips the splice instead of
performing it with a cursor that means nothing. Two tests cover both
macro shapes; without the fix the first one aborts.
CFLAGS_TO_CHECK carried ten -Wno- flags. Measured one at a time, four of
them hid nothing at all on either compiler, and the rest were hiding
work. All ten are gone; gcc and clang are clean on arm, riscv and x64
with nothing switched off.

-Wno-uninitialized was the expensive one. It was there because fatal(),
usage_error() and error_at() never return but could not say so: shecc
parses no attributes, so annotating them would not compile itself. A
__noreturn macro that is empty under __SHECC__ solves that, and with it
every remaining report in the class was real or explicable. The real one
is build_r_idom(), whose pred is read without an initializer when no
successor has a postdominator yet. Its forward twin build_idom() has
carried the guard, and a comment explaining it, all along; the
suppression is why nobody saw the reverse copy lacked it.
Reintroducing the defect now fails the build.

-Wno-overflow hid 134 x86-64 opcode bytes above 0x7F handed to a char
parameter. The bytes were right, but the narrowing was implicit 134
times over, which would have hidden a real one. emit_byte() takes an int
and truncates once, on purpose, and no call site changes.

The rest were smaller: nine "%p" arguments that were typed pointers,
four parameters kept for signature symmetry, and thirteen "()"
declarations, which in C means unspecified arguments rather than none.

Separately, a diagnostic that reached a pipe did not survive it.
Twenty-four sites printed and then abort()ed, which discards the buffer,
so the MAX_TYPES guard, reachable from ordinary source, wrote zero bytes
into a build log. All of them flush now.

The three exits are also split by cause rather than by habit. error_at()
reports a mistake in the input and exits: a syntax error should not
raise SIGABRT, wake the system crash handler or leave a core behind,
which cost 1.1 s per diagnostic here and made 24 of the suite's tests
pay it. Losing the text to quote changes what can be shown and not what
happened, so both of its fallbacks exit too. fatal() keeps abort(),
because a broken invariant is worth a core dump, and usage_error()
already exited for the same reason.

The rest is what the analysis turned up alongside: 89 const qualifiers
on pointers that are only read, an unreachable statement after a
now-noreturn call, an unreachable ternary branch, three dead stores, a
dead counter, a redundant condition, a duplicated if, the bare 10 that
bounded three operator stacks, and two spelling mistakes.
emit_ph2_ir() was 1543 lines, read_body_statement() 896, reg_alloc()
830, lex_token() 737. Each is a dispatcher with its work inlined, so
each splits along a seam that is already there rather than one invented
for the occasion.

lex_token() dispatched on the first character of a token through fifty
if-blocks. It now offers the character to six readers, layout, numbers,
literals, punctuation, operators and words, each of which claims what it
knows and returns NULL for the rest, and is 28 lines.

emit_ph2_ir() is a 218-line prologue of fusion attempts and a switch
over 43 opcode groups. The groups become eight emitters named for the
instruction family they cover, taking only the prologue values they use,
and the function is 398 lines.

read_body_statement() already delegated return, if and while to
handle_*_statement() helpers; switch, for, do, record declarations and
the declaration tail now go the same way, leaving a 63-line dispatcher
that reads as the grammar it implements.

reg_alloc() is two loops. Their bodies become reg_alloc_global() and
reg_alloc_bb(), and the function is 155 lines. Three continues that
ended an iteration of the block loop end the call instead.

This is code motion and nothing else, checked as such rather than
asserted: a frozen copy of the compiler's own source, compiled by the
compiler built before and after, is byte-identical on all three targets.
That is a stronger statement than the suite makes, since stage1 equal to
stage2 only proves the compiler is deterministic. That every extracted
handler returns on all paths is part of the same proof: gcc reports no
control reaching the end of any of them, so no caller ever fell out of
the original if-block into the statement kinds below it.
@jserv
jserv merged commit a8fca6b into master Sep 7, 2026
30 checks passed
@jserv
jserv deleted the hook branch September 7, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant