fix: external language scanner parsing - #52
Merged
hkimura-intersys merged 1 commit intoJul 27, 2026
Conversation
commit 7b2e8d8 Author: Hannah <hannah.kimura@intersystems.com> Date: Mon Jul 27 11:06:31 2026 -0400 fix: add external delimiter fix to SQL/JavaScript cases The bug described in the commit before this explains how if a delimiter is included in a string, it was wrongfully counted in the scanner's parsing of external languages. This was fixed by Tom Dyar for python, and that fix is extended to other languages with this commit. commit 9bbc870 Author: Thomas Dyar <tdyar@intersystems.com> Date: Sat Jul 25 05:18:11 2026 -0400 fix(udl scanner): lex Language = python bodies with string awareness `lex_fenced_text` balances braces without knowing anything about string literals or comments, so a `{` inside a Python string ends the method body early. A single line like s = "dict literal { unbalanced" leaves the counter unbalanced, and the body token swallows the rest of the class. Every following member is lost from the tree, and the only signal is an ERROR range at the closing brace of the class. `external_method_body_content` is shared with XData (XML and JSON), triggers, SQL queries and storage, and the scanner gets no language context (`in_body` is never read). Teaching `lex_fenced_text` about quotes would break XML prose containing apostrophes and SQL `--` comments, so this splits the token instead of patching it in place: - `_python_method_keywords` and `_python_trigger_keywords` match `Language = python` only, so the grammar can select a different body token. Both alias `method_keyword_python_language` back to `method_keyword_external_language`. - `python_method_body_content` is lexed by `lex_python_fenced_text`, which tracks brace depth while skipping `#` comments, single and triple quotes, prefixed strings, and backslash escapes. An unterminated single-quoted string stops at the newline so brace counting resumes. - `method_keyword_external_language` is narrowed to tsql and ispl. Those two, plus XData, triggers, queries and storage, keep `external_method_body_content` and the existing brace counter. - The two Python rules in `injections.scm` point at the new token so Python highlighting keeps working. The only change to parser output is `external_method_body_content` -> `python_method_body_content` on `Language = python` bodies. The playground grammar keeps its own near-copy of the UDL scanner, so the same changes are ported there; without that its external token indices skew and unrelated Owner-keyword tests fail. New corpus cases in `udl/test/corpus/python-method.txt` cover the brace in a string, the quoting edge cases (triple quotes, f-strings, raw strings, escaped quotes, comments, nested dict literals), a Python trigger, and a tsql method as a control. All five grammars pass: 2745 parses, 0 failures. Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the bug fix (done by @isc-tdyar) described in #50 and extends that to also include the bug fix for
tsql,ispl, andjavascriptcases. Essentially, if a delimiter is found within a string, it is not counted as an actual delimiter. There are more edge cases described in #50 that are specific to Python as well.Testing
This was tested on all the generated files in %SYS, as well as all the test cases and workflows in this repo. Everything is passing.