Add a lossless source model

README.md

@@ -14,7 +14,7 @@ Development and primary CI follow the latest stable Rust and Cargo releases thro
 
 ```text
 Stack source
-    -> lexical analysis
+    -> lexical analysis -----> lossless tokens and trivia
     -> syntax AST
     -> semantic validation
     -> normalized Diagram IR
@@ -46,12 +46,19 @@ STACK_SPECIFICATION_DIR=../specification cargo test --features conformance --tes
 
 JSON support is development-only and does not add a runtime dependency to the compiler library. CI checks out the recorded specification revision before running the suite.
 
+## Lossless Source Model
+
+`parse_lossless` and `parse_lossless_bytes` expose every authored token, whitespace segment, line comment, original string spelling, CRLF sequence, and end-exclusive source span. Concatenating token text through `lossless::Document::reconstruct` reproduces the input byte-for-byte. This source-oriented API is separate from normalized IR and performs no filesystem access.
+
+Lossless parsing succeeds for syntactically valid source even when semantic validation would later reject it. Lexical and syntax errors return diagnostics without a partial document.
+
 ## Architecture
 
 - [`docs/decisions/0001-build-a-portable-rust-compiler-core.md`](./docs/decisions/0001-build-a-portable-rust-compiler-core.md)
 - [`docs/decisions/0002-separate-syntax-ast-from-normalized-ir.md`](./docs/decisions/0002-separate-syntax-ast-from-normalized-ir.md)
 - [`docs/decisions/0003-use-a-handwritten-parser.md`](./docs/decisions/0003-use-a-handwritten-parser.md)
 - [`docs/decisions/0004-consume-a-pinned-conformance-suite.md`](./docs/decisions/0004-consume-a-pinned-conformance-suite.md)
+- [`docs/decisions/0005-add-a-lossless-source-model.md`](./docs/decisions/0005-add-a-lossless-source-model.md)
 - [`docs/specs/compiler-frontend.md`](./docs/specs/compiler-frontend.md)
 
 ## License

docs/decisions/0005-add-a-lossless-source-model.md

@@ -0,0 +1,56 @@
+# ADR-0005: Add a Lossless Source Model
+
+## Status
+
+Accepted
+
+## Date
+
+2026-09-03
+
+## Context
+
+The canonical formatter must preserve comments and their token gaps while normalizing whitespace and authored string escapes. The existing syntax AST intentionally stores decoded semantic values and declaration spans, while the lexer discards whitespace and comments. Normalized IR excludes all source trivia by contract.
+
+Implementing another parser in the formatter would let syntax handling drift from the compiler. Adding trivia to normalized IR would make a portable semantic representation depend on source spelling. Mutating every existing AST node to carry trivia would also expand an API whose current responsibility is syntax and validation.
+
+## Decision
+
+Add a separate public lossless source model. `parse_lossless` and `parse_lossless_bytes` run the same lexer and recursive-descent parser used by the established parse path, then expose a flat source-order sequence containing:
+
+- language tokens with their exact authored text and end-exclusive span;
+- decoded values for string tokens while retaining the original escape spelling;
+- whitespace segments, including tabs, LF, and CRLF;
+- complete line-comment lexemes without absorbing their line endings;
+- a zero-width end token.
+
+The source text of all tokens and trivia segments concatenates to the original UTF-8 input byte-for-byte. Lossless parsing requires lexical and syntactic validity but is independent of semantic validation, so a syntactically valid document with name or layout errors remains inspectable. Lexical or syntax errors return the existing portable diagnostics without a partial lossless document.
+
+The AST and normalized IR types remain unchanged. The source model does not read files, consult the network or clock, or add a runtime dependency.
+
+## Alternatives Considered
+
+### Put trivia in the normalized IR
+
+- Pros: Downstream tools would consume one representation.
+- Cons: Source spelling and comments are not portable diagram meaning and would destabilize semantic interchange.
+- Rejected: Normalized IR must remain renderer-independent and formatting-free.
+
+### Add trivia fields throughout the existing AST
+
+- Pros: Formatting structure and source text would live in one tree.
+- Cons: Broadly changes established syntax types and makes semantic consumers carry formatter-only data.
+- Rejected: A separate token model is additive and keeps existing responsibilities intact.
+
+### Tokenize again in each formatter
+
+- Pros: No compiler API addition.
+- Cons: Duplicates string, comment, operator, and position behavior and can drift from compiler diagnostics.
+- Rejected: The compiler must own one lexical interpretation of Stack source.
+
+## Consequences
+
+- Formatters and editors can preserve authored bytes without reimplementing lexical rules.
+- Callers use the lossless tokens alongside the existing AST when structural context is required.
+- The model owns each token's exact text and decoded strings, trading modest allocation for a simple lifetime-free public API.
+- Invalid UTF-8, byte order marks, and syntax errors remain diagnostic-only results.

docs/specs/compiler-frontend.md

@@ -29,6 +29,7 @@ The primary users are Stack CLI, browser, editor, layout, and hosted-service imp
 src/ast.rs          Syntax-oriented AST and spans
 src/diagnostic.rs   Portable diagnostic types and codes
 src/lexer.rs        UTF-8 text tokenization
+src/lossless.rs     Exact tokens, trivia, authored text, and spans
 src/parser.rs       Recursive-descent grammar implementation
 src/validation.rs   Semantic validation and normalization
 src/validation/     Focused validation unit tests
@@ -45,12 +46,16 @@ docs/decisions/     Architectural decisions
 ```rust
 pub fn parse(source: &str) -> ParseOutput;
 pub fn parse_bytes(source: &[u8]) -> ParseOutput;
+pub fn parse_lossless(source: &str) -> LosslessParseOutput;
+pub fn parse_lossless_bytes(source: &[u8]) -> LosslessParseOutput;
 pub fn compile(source: &str) -> CompileOutput;
 pub fn compile_bytes(source: &[u8]) -> CompileOutput;
 ```
 
 `ParseOutput` contains an AST only when decoding, lexing, and parsing succeed. `CompileOutput` contains normalized IR only when all compiler-stage errors are absent. Both outputs contain diagnostics. Semantic warnings may accompany successful IR.
 
+`LosslessParseOutput` contains an exact token-and-trivia document only when decoding, lexing, and syntax parsing succeed. Its tokens retain original text and spans, while string tokens additionally expose decoded values. Reconstructing the document concatenates every token and trivia segment without normalization. Semantic validation remains a separate stage.
+
 ## Code Style
 
 Prefer explicit domain types and exhaustive matches:
@@ -115,7 +120,7 @@ match operator {
 
 - WebAssembly bindings
 - Native CLI commands
-- Formatter and comment-preserving concrete syntax tree
+- Canonical formatter implementation
 - Theme and icon resolution
 - Layout and renderer integration
 - Multi-error syntax recovery

src/lib.rs

@@ -9,6 +9,7 @@
 pub mod ast;
 pub mod diagnostic;
 pub mod ir;
+pub mod lossless;
 
 mod lexer;
 mod parser;
@@ -34,6 +35,15 @@ pub struct CompileOutput {
     pub diagnostics: Vec<Diagnostic>,
 }
 
+/// Output of syntactic parsing into the lossless source model.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct LosslessParseOutput {
+    /// Lossless document, present only when lexical and syntax parsing succeeds.
+    pub document: Option<lossless::Document>,
+    /// Lexical and syntax diagnostics.
+    pub diagnostics: Vec<Diagnostic>,
+}
+
 /// Parses UTF-8 Stack source into a source-oriented AST.
 pub fn parse(source: &str) -> ParseOutput {
     match parser::parse(source) {
@@ -52,17 +62,45 @@ pub fn parse(source: &str) -> ParseOutput {
 pub fn parse_bytes(source: &[u8]) -> ParseOutput {
     match std::str::from_utf8(source) {
         Ok(source) => parse(source),
-        Err(error) => {
-            let position = position_after_valid_prefix(&source[..error.valid_up_to()]);
-            ParseOutput {
+        Err(error) => ParseOutput {
+            document: None,
+            diagnostics: vec![invalid_utf8_diagnostic(source, error)],
+        },
+    }
+}
+
+/// Parses UTF-8 Stack source into exact authored tokens and trivia.
+pub fn parse_lossless(source: &str) -> LosslessParseOutput {
+    let tokens = match lexer::tokenize(source) {
+        Ok(tokens) => tokens,
+        Err(diagnostic) => {
+            return LosslessParseOutput {
                 document: None,
-                diagnostics: vec![Diagnostic::error(
-                    "STK1001",
-                    "Input is not valid UTF-8.",
-                    diagnostic::Span::point(position),
-                )],
-            }
+                diagnostics: vec![*diagnostic],
+            };
         }
+    };
+
+    match parser::parse_tokens(tokens.clone()) {
+        Ok(_) => LosslessParseOutput {
+            document: Some(lossless::Document::from_lexer_tokens(source, tokens)),
+            diagnostics: Vec::new(),
+        },
+        Err(diagnostic) => LosslessParseOutput {
+            document: None,
+            diagnostics: vec![*diagnostic],
+        },
+    }
+}
+
+/// Decodes and parses Stack source bytes into exact authored tokens and trivia.
+pub fn parse_lossless_bytes(source: &[u8]) -> LosslessParseOutput {
+    match std::str::from_utf8(source) {
+        Ok(source) => parse_lossless(source),
+        Err(error) => LosslessParseOutput {
+            document: None,
+            diagnostics: vec![invalid_utf8_diagnostic(source, error)],
+        },
     }
 }
 
@@ -122,10 +160,22 @@ fn position_after_valid_prefix(prefix: &[u8]) -> diagnostic::SourcePosition {
     position
 }
 
+fn invalid_utf8_diagnostic(source: &[u8], error: std::str::Utf8Error) -> Diagnostic {
+    let position = position_after_valid_prefix(&source[..error.valid_up_to()]);
+    Diagnostic::error(
+        "STK1001",
+        "Input is not valid UTF-8.",
+        diagnostic::Span::point(position),
+    )
+}
+
 #[cfg(test)]
 mod tests {
+    use crate::lossless::TokenKind;
+
     use super::{
-        compile, compile_bytes, parse, parse_bytes, position_after_valid_prefix, validate,
+        compile, compile_bytes, parse, parse_bytes, parse_lossless, parse_lossless_bytes,
+        position_after_valid_prefix, validate,
     };
 
     #[test]
@@ -165,4 +215,69 @@ mod tests {
             crate::diagnostic::SourcePosition::start()
         );
     }
+
+    #[test]
+    fn lossless_entry_points_preserve_trivia_escapes_and_crlf() {
+        let source = concat!(
+            "// leading\r\n",
+            "stack 1.0\r\n",
+            "diagram \"\\u56F3\" {\r\n",
+            "\tnode api \"API\" // trailing\r\n",
+            "}\r\n",
+        );
+        let output = parse_lossless(source);
+        assert!(output.diagnostics.is_empty());
+        let Some(document) = output.document else {
+            return;
+        };
+
+        assert_eq!(document.reconstruct(), source);
+        assert!(document.tokens().iter().any(|token| {
+            matches!(&token.kind, TokenKind::String(value) if value == "図")
+                && token.text == "\"\\u56F3\""
+        }));
+        assert!(
+            document.tokens().iter().any(|token| {
+                token.kind == TokenKind::Whitespace && token.text.contains("\r\n")
+            })
+        );
+        assert!(
+            document.tokens().iter().any(|token| {
+                token.kind == TokenKind::LineComment && token.text == "// trailing"
+            })
+        );
+
+        let bytes_output = parse_lossless_bytes(source.as_bytes());
+        assert_eq!(bytes_output.document, Some(document));
+    }
+
+    #[test]
+    fn lossless_entry_points_report_lexical_syntax_and_encoding_errors() {
+        let bom = parse_lossless("\u{feff}stack 1.0");
+        assert!(bom.document.is_none());
+        assert_eq!(bom.diagnostics[0].code, "STK1002");
+
+        let syntax = parse_lossless("stack 1.0 diagram \"x\" {");
+        assert!(syntax.document.is_none());
+        assert_eq!(syntax.diagnostics[0].code, "STK2003");
+
+        let encoding = parse_lossless_bytes(b"stack 1.0\r\n\xff");
+        assert!(encoding.document.is_none());
+        assert_eq!(encoding.diagnostics[0].code, "STK1001");
+        assert_eq!(encoding.diagnostics[0].span.start.line, 2);
+    }
+
+    #[test]
+    fn lossless_syntax_model_keeps_semantically_invalid_source() {
+        let source = concat!(
+            "stack 1.0\n",
+            "diagram \"Duplicate\" {\n",
+            "  node api \"First\"\n",
+            "  node api \"Second\"\n",
+            "}\n",
+        );
+
+        assert!(parse_lossless(source).document.is_some());
+        assert!(compile(source).diagram.is_none());
+    }
 }