Skip to content

perf(cli)!: write compact JSON through a streaming encoder - #337

Merged
OmarAlJarrah merged 2 commits into
mainfrom
perf/cli-compact-streaming-output
Aug 9, 2026
Merged

perf(cli)!: write compact JSON through a streaming encoder#337
OmarAlJarrah merged 2 commits into
mainfrom
perf/cli-compact-streaming-output

Conversation

@OmarAlJarrah

Copy link
Copy Markdown
Member

Summary

-o wrote 2-space-indented JSON, and it wrote it by marshalling the whole document into a byte
slice before the destination was even opened. Both cost more than they are worth for a file nobody
reads by eye: the indented artifact is about twice the bytes, json.MarshalIndent compacts and
then re-indents into a second buffer, and the finished slice was then handed to the writer as one
more live copy of the output.

-o now writes compact JSON through a json.Encoder aimed straight at the temp file that
replaceFile publishes. Stdout keeps its indented form — a person is reading it — and --pretty
restores the indented form for a file.

On a synthetic 800-path, 800-schema, 6400-property OpenAPI 3.1 spec (664 KB of YAML, no
diagnostics), measured on this branch:

before (indented) after (compact)
artifact 7,455,607 bytes 3,789,578 bytes (1.97× smaller)
encode time ~40 ms ~18.6 ms (2.15× faster)
encode allocations ~22.8 MB ~10.5 MB (2.2× less)
whole compile … -o run 0.495 s 0.484 s

The last row is the honest one: encoding is a small share of a full compile, so the end-to-end
saving on this spec is a couple of percent. The win is the artifact and the transient memory, not
the wall clock.

Indented output deliberately keeps using MarshalIndent rather than the encoder. A
json.Encoder with SetIndent indents into a second buffer of its own and allocates roughly twice
as much for byte-identical output, so routing both forms through the encoder would have made the
default stdout path worse.

Two properties the old order provided are kept rather than assumed:

  • A failed encode never disturbs the destination. Marshalling first used to guarantee that by
    never opening a file it could not fill. Encoding into the temp file guarantees it because the
    destination is only ever reached by the rename — and the temp file is removed on the way out.
    scripts/verify-atomic-output.sh, which is not in CI, was run by hand and passes.
  • Ordering is untouched. Maps still emit in sorted-key order and slices in source order.

ir/irtest is untouched, so every IR golden keeps its indented bytes.

Breaking

The bytes -o writes change. Anything that diffs, hashes or byte-compares a morphic compile -o
artifact across this change will see a difference in formatting only; --pretty restores the
previous bytes exactly. Stdout is unaffected.

Test plan

  • TestRun_FileOutputIsCompact — the artifact holds one newline, its trailing one, and unmarshals
    to the compiled document. Red before the change (91 newlines).
  • TestRun_PrettyRestoresIndentedFile-o --pretty writes byte-for-byte what stdout writes,
    which is what -o used to write. Checked outside the suite too: cmp reports the --pretty
    artifact identical to the artifact the previous build produced for the same spec.
  • TestRun_CompactOutputKeepsDocumentOrder — the determinism check. json.Compact of the indented
    artifact must reproduce the compact one byte for byte; removing whitespace cannot reorder
    anything, so the two agreeing means the encoder emitted the same keys in the same sorted order
    and the same slices in the same source order as the marshaller it replaced. It also compiles
    twice and compares, and asserts the fixture really does force a reordering (its schemas are
    declared Zeta, Alpha, Mid) so the check is not vacuous.
  • TestWriteParsed_MarshalErrorLeavesFileUntouched now pins the structural half as well: a
    document that will not marshal creates exactly one temp file, removes it, and leaves the
    existing destination byte-for-byte intact.
  • TestEncodeDocument_ErrorPaths covers both forms against both failures. The compact form needs
    it: json.Encoder reports a refusing destination and an unmarshallable document as one error,
    so without the pass-through writer that records what the destination said, every disk failure
    would be reported as a marshal failure.
  • Reverting the production change with the tests kept: forcing files back to indented reddens the
    two format tests; marshalling into a buffer before replaceFile reddens the temp-file test.
  • cmd/morphic/testdata/compile-help.txt regenerated with -update; it changes by the new flag
    and the description sentence and nothing else. It was red before regenerating.
  • Full gate green in a clean worktree: gofmt, go vet, golangci-lint, go build, and
    scripts/check-coverage.sh at 100%.

Notes

#308 (-o /dev/null fails and its exit 2 masks the diagnostics' exit 1) is in this output path and
is neither fixed nor worsened here: it fails at temp-file creation, before anything is encoded, so
the message and the exit code are identical before and after.

Ordering

Branched from main and intended to land after #312, which touches the same file for different
reasons. Three conflicts, all mechanical: compileOptions/newCompileFlags in compile.go (keep
#312's specOptions split, add the pretty field and its BoolVar), compileFlagNames in
command_test.go (add "pretty" to compile's list), and the README's CLI section (add the
--pretty row and the -o wording to #312's rewritten table).

Closes #80

The three conflicts this branch predicted, resolved as it said: compile.go keeps
#312's specOptions split and gains the pretty field, command_test.go keeps
#312's two-list block and gains "pretty", and the README keeps #312's
two-command flag table with the -o wording changed and a --pretty row added.
#335 also landed on the README since, so its diagnostics and exit-code prose is
kept whole rather than reverted to this branch's older paragraph.

One interaction the branch could not predict: #330's
TestRun_TerminatorAsFlagValue asserts the artifact contains `"name": "Tiny"` —
the indented spelling. That test is about which argument "--" became, not about
formatting, and -o is compact now, so it matched text it never meant to pin. It
decodes the artifact instead, which is what it was always asking.
@OmarAlJarrah
OmarAlJarrah merged commit ac04260 into main Aug 9, 2026
1 check passed
@OmarAlJarrah
OmarAlJarrah deleted the perf/cli-compact-streaming-output branch August 9, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cli: default -o output to compact JSON written through a streaming encoder

1 participant