add box prefix support for auto-boxing stack types in DataEnum

[ehwan]

Description

This PR introduces the box keyword prefix syntax sugar for %tokentype and NonTerminal RuleType definitions.
Previously, to prevent parser data stack enum inflation from large AST nodes, users had to manually wrap types in Box<...> and write boilerplate code to package/unbox values inside reduce actions. With this change, prefixing a type definition with box automatically compiles it into a Box wrapper in the internal Data stack enum, while presenting clean, unboxed values directly to user-defined reduce actions and the public parser APIs.
To ensure zero overhead on the parser generator compilation phase, type prefix checking is optimized to run in $O(1)$ time by verifying token structure directly rather than doing full syntax parsing.

Key Changes

• Type Checking Utilities (utils.rs): Added highly efficient $O(1)$ has_box_prefix and strip_box_prefix checks to identify and strip the box keyword from token streams when followed by a type definition.
• Placeholder Resolution (grammar.rs): Supported placeholder type inference for boxed structures (e.g., box _), resolving types while maintaining the boxed keyword structure.
• Auto-boxing and Auto-dereferencing Codegen (emit.rs):
  • Automatically wraps data enum variant fields in Box<T> if defined with the box prefix.
  • Automatically dereferences (*val) when popping variables in reduce actions, making the popped binding unboxed.
  • Automatically wraps pushed variables in Box::new(...) during rule reduction returns and token feed actions.
  • Kept public traits and API definitions (such as type Term and type StartType) using the unboxed types.
• Documentation: Updated SYNTAX.md under the Memory Optimization with Box section to document the usage and mechanics of the new syntax sugar.

[gemini-code-assist]

Code Review

This pull request introduces support for a box keyword prefix in the parser generator's syntax to automatically box internal stack values, optimizing memory footprint while allowing unboxed types in reduce actions. The feedback suggests optimizing the box prefix detection in utils.rs by avoiding expensive syn::parse2 calls, and improving consistency in emit.rs by using the newly exposed crate::utils::remove_whitespaces helper instead of a local duplicate.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Performance & Complexity Improvement

Using syn::parse2::<syn::Type> to validate the remaining token stream on every check is highly inefficient and adds unnecessary overhead to the parser generator, especially for large grammars with many rules.

Since box is a reserved keyword in Rust, any type definition starting with box followed by at least one other token is unambiguously a boxed type. If the user provides invalid syntax after box, the Rust compiler will catch it anyway when compiling the generated code.

We can simplify both has_box_prefix and strip_box_prefix to simply check the first token and the presence of a second token. This avoids the expensive parsing step entirely, making the check $O(1)$ and significantly faster.

pub(crate) fn has_box_prefix(ts: &proc_macro2::TokenStream) -> bool {
    let mut iter = ts.clone().into_iter();
    if let Some(proc_macro2::TokenTree::Ident(ident)) = iter.next() {
        ident.to_string() == "box" && iter.next().is_some()
    } else {
        false
    }
}

pub(crate) fn strip_box_prefix(ts: proc_macro2::TokenStream) -> proc_macro2::TokenStream {
    let mut iter = ts.clone().into_iter();
    if let Some(proc_macro2::TokenTree::Ident(ident)) = iter.next() {
        if ident.to_string() == "box" && iter.next().is_some() {
            let mut iter = ts.into_iter();
            let _ = iter.next(); // Consume 'box'
            return iter.collect();
        }
    }
    ts
}

[gemini-code-assist]

Consistency & Code Duplication

Use crate::utils::remove_whitespaces instead of the local remove_whitespaces helper function to maintain consistency with line 937 and avoid redundant helpers.

Suggested change

	.entry(remove_whitespaces(storage_type.to_string()))
	.entry(crate::utils::remove_whitespaces(storage_type.to_string()))