perf(cli)!: write compact JSON through a streaming encoder - #337
Merged
Conversation
The three conflicts this branch predicted, resolved as it said: compile.go keeps #312's specOptions split and gains the pretty field, command_test.go keeps #312's two-list block and gains "pretty", and the README keeps #312's two-command flag table with the -o wording changed and a --pretty row added. #335 also landed on the README since, so its diagnostics and exit-code prose is kept whole rather than reverted to this branch's older paragraph. One interaction the branch could not predict: #330's TestRun_TerminatorAsFlagValue asserts the artifact contains `"name": "Tiny"` — the indented spelling. That test is about which argument "--" became, not about formatting, and -o is compact now, so it matched text it never meant to pin. It decodes the artifact instead, which is what it was always asking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-owrote 2-space-indented JSON, and it wrote it by marshalling the whole document into a byteslice before the destination was even opened. Both cost more than they are worth for a file nobody
reads by eye: the indented artifact is about twice the bytes,
json.MarshalIndentcompacts andthen re-indents into a second buffer, and the finished slice was then handed to the writer as one
more live copy of the output.
-onow writes compact JSON through ajson.Encoderaimed straight at the temp file thatreplaceFilepublishes. Stdout keeps its indented form — a person is reading it — and--prettyrestores the indented form for a file.
On a synthetic 800-path, 800-schema, 6400-property OpenAPI 3.1 spec (664 KB of YAML, no
diagnostics), measured on this branch:
compile … -orunThe last row is the honest one: encoding is a small share of a full compile, so the end-to-end
saving on this spec is a couple of percent. The win is the artifact and the transient memory, not
the wall clock.
Indented output deliberately keeps using
MarshalIndentrather than the encoder. Ajson.EncoderwithSetIndentindents into a second buffer of its own and allocates roughly twiceas much for byte-identical output, so routing both forms through the encoder would have made the
default stdout path worse.
Two properties the old order provided are kept rather than assumed:
never opening a file it could not fill. Encoding into the temp file guarantees it because the
destination is only ever reached by the rename — and the temp file is removed on the way out.
scripts/verify-atomic-output.sh, which is not in CI, was run by hand and passes.ir/irtestis untouched, so every IR golden keeps its indented bytes.Breaking
The bytes
-owrites change. Anything that diffs, hashes or byte-compares amorphic compile -oartifact across this change will see a difference in formatting only;
--prettyrestores theprevious bytes exactly. Stdout is unaffected.
Test plan
TestRun_FileOutputIsCompact— the artifact holds one newline, its trailing one, and unmarshalsto the compiled document. Red before the change (91 newlines).
TestRun_PrettyRestoresIndentedFile—-o --prettywrites byte-for-byte what stdout writes,which is what
-oused to write. Checked outside the suite too:cmpreports the--prettyartifact identical to the artifact the previous build produced for the same spec.
TestRun_CompactOutputKeepsDocumentOrder— the determinism check.json.Compactof the indentedartifact must reproduce the compact one byte for byte; removing whitespace cannot reorder
anything, so the two agreeing means the encoder emitted the same keys in the same sorted order
and the same slices in the same source order as the marshaller it replaced. It also compiles
twice and compares, and asserts the fixture really does force a reordering (its schemas are
declared
Zeta,Alpha,Mid) so the check is not vacuous.TestWriteParsed_MarshalErrorLeavesFileUntouchednow pins the structural half as well: adocument that will not marshal creates exactly one temp file, removes it, and leaves the
existing destination byte-for-byte intact.
TestEncodeDocument_ErrorPathscovers both forms against both failures. The compact form needsit:
json.Encoderreports a refusing destination and an unmarshallable document as one error,so without the pass-through writer that records what the destination said, every disk failure
would be reported as a marshal failure.
two format tests; marshalling into a buffer before
replaceFilereddens the temp-file test.cmd/morphic/testdata/compile-help.txtregenerated with-update; it changes by the new flagand the description sentence and nothing else. It was red before regenerating.
gofmt,go vet,golangci-lint,go build, andscripts/check-coverage.shat 100%.Notes
#308 (
-o /dev/nullfails and its exit 2 masks the diagnostics' exit 1) is in this output path andis neither fixed nor worsened here: it fails at temp-file creation, before anything is encoded, so
the message and the exit code are identical before and after.
Ordering
Branched from
mainand intended to land after #312, which touches the same file for differentreasons. Three conflicts, all mechanical:
compileOptions/newCompileFlagsincompile.go(keep#312's
specOptionssplit, add theprettyfield and itsBoolVar),compileFlagNamesincommand_test.go(add"pretty"to compile's list), and the README's CLI section (add the--prettyrow and the-owording to #312's rewritten table).Closes #80