Skip to content

Add a lossless source model - #5

Merged
konojunya merged 1 commit into
mainfrom
jj/add-lossless-source-model
Sep 3, 2026
Merged

Add a lossless source model#5
konojunya merged 1 commit into
mainfrom
jj/add-lossless-source-model

Conversation

@konojunya

Copy link
Copy Markdown
Contributor

Background

Task 6 needs compiler-owned comment and trivia handling before the formatter is implemented. The existing AST keeps decoded syntax values and the normalized IR intentionally excludes formatting data, while the lexer previously discarded whitespace and comments.

Implementation

  • add public parse_lossless and parse_lossless_bytes entry points for syntactically valid Stack source
  • expose a separate flat lossless document containing exact token/trivia text, decoded string values, comments, CRLF, and end-exclusive spans
  • reconstruct UTF-8 source byte-for-byte by concatenating the lossless sequence
  • reuse the existing lexer and recursive-descent parser, leaving established AST and normalized IR types unchanged
  • document the boundary in ADR-0005 and the compiler frontend specification
  • pin the merged canonical formatter specification revision e40ad5dc230ab58f8a211130af96ae09c0e523bc

Testing

  • cargo +stable fmt --check
  • cargo +stable test (35 unit tests, 5 integration tests)
  • STACK_SPECIFICATION_DIR=../specification cargo +stable test --features conformance --test conformance
  • cargo +stable clippy --all-targets --all-features -- -D warnings
  • RUSTDOCFLAGS="-D warnings" cargo +stable doc --no-deps
  • cargo +stable llvm-cov --lib --all-features --workspace --fail-under-lines 95 --fail-under-functions 95 --fail-under-regions 95 (97.92% regions, 98.98% functions, 98.71% lines)
  • cargo +1.85.0 test
  • cargo +1.85.0 clippy --all-targets --all-features -- -D warnings
  • git diff --check

Review focus / concerns

  • lossless parsing intentionally requires lexical and syntax validity but remains independent of semantic validation
  • token text is owned so the public model has no source lifetime; decoded strings add a modest allocation cost
  • comments exclude their following line ending, which remains a separate whitespace segment and keeps reconstruction exact
  • the parser validates the same token vector before a lossless document is returned; errors never expose a partial model

@konojunya
konojunya merged commit 8e9dc1c into main Sep 3, 2026
2 checks passed
@konojunya
konojunya deleted the jj/add-lossless-source-model branch September 3, 2026 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant