Fix/string aware fenced text - #51
Closed
hkimura-intersys wants to merge 2 commits into
Closed
Conversation
`lex_fenced_text` balances braces without knowing anything about string
literals or comments, so a `{` inside a Python string ends the method body
early. A single line like
s = "dict literal { unbalanced"
leaves the counter unbalanced, and the body token swallows the rest of the
class. Every following member is lost from the tree, and the only signal is
an ERROR range at the closing brace of the class.
`external_method_body_content` is shared with XData (XML and JSON),
triggers, SQL queries and storage, and the scanner gets no language context
(`in_body` is never read). Teaching `lex_fenced_text` about quotes would
break XML prose containing apostrophes and SQL `--` comments, so this
splits the token instead of patching it in place:
- `_python_method_keywords` and `_python_trigger_keywords` match
`Language = python` only, so the grammar can select a different body
token. Both alias `method_keyword_python_language` back to
`method_keyword_external_language`.
- `python_method_body_content` is lexed by `lex_python_fenced_text`, which
tracks brace depth while skipping `#` comments, single and triple quotes,
prefixed strings, and backslash escapes. An unterminated single-quoted
string stops at the newline so brace counting resumes.
- `method_keyword_external_language` is narrowed to tsql and ispl. Those
two, plus XData, triggers, queries and storage, keep
`external_method_body_content` and the existing brace counter.
- The two Python rules in `injections.scm` point at the new token so Python
highlighting keeps working.
The only change to parser output is `external_method_body_content` ->
`python_method_body_content` on `Language = python` bodies.
The playground grammar keeps its own near-copy of the UDL scanner, so the
same changes are ported there; without that its external token indices skew
and unrelated Owner-keyword tests fail.
New corpus cases in `udl/test/corpus/python-method.txt` cover the brace in a
string, the quoting edge cases (triple quotes, f-strings, raw strings,
escaped quotes, comments, nested dict literals), a Python trigger, and a
tsql method as a control. All five grammars pass: 2745 parses, 0 failures.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
The bug described in the commit before this explains how if a delimiter is included in a string, it was wrongfully counted in the scanner's parsing of external languages. This was fixed by Tom Dyar for python, and that fix is extended to other languages with this commit.
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR extends @isc-tdyar's PR here to use that bug fix (only the string cases) for
tsql,ispl, andjavascriptas well. Details are described in the PR linked. The version is bumped tov1.9.14as well.