fix(udl scanner): lex Language = python bodies with string awareness

[isc-tdyar]

A { inside a Python string literal ends a Language = python method body
early, and everything after it in the class is lost from the tree.

Reproduction

Class My.Brace
{

ClassMethod PyBad() [ Language = python ]
{
    s = "dict literal { unbalanced"
    return s
}

ClassMethod Real() As %String
{
    Quit ##class(My.Tgt).Hit()
}

}

Before this change, tree-sitter parse gives one
external_method_body_content spanning [4, 1] - [15, 0]: the body token
runs past its own closing brace and consumes Real along with it. Real
never appears in the tree. The only signal is an ERROR range on the closing
brace of the class, so a consumer that tolerates partial parses sees a class
with one method instead of two.

The cause is lex_fenced_text in udl/src/scanner.c. It counts { and }
with no knowledge of string literals or comments, so the { inside the
string increments the depth and the real closing brace only brings it back to
one.

Why this splits the token instead of patching the brace counter

external_method_body_content is shared by XData (XML and JSON), triggers,
SQL queries and storage. The scanner has no language context when it runs
(in_body is declared and assigned but never read), so it cannot tell which
of those it is lexing.

Teaching lex_fenced_text to skip quotes would therefore apply Python rules
to XML prose, where an apostrophe in don't would open a string that never
closes, and to SQL, where -- is a comment and # is not. So the grammar now
selects a different body token when the language is Python:

• _python_method_keywords and _python_trigger_keywords match
  Language = python only. Both alias method_keyword_python_language back
  to method_keyword_external_language, so the keyword node is unchanged.
• python_method_body_content is lexed by a new lex_python_fenced_text,
  which tracks brace depth while skipping # line comments, single and triple
  quotes, prefixed strings, and backslash escapes. An unterminated
  single-quoted string stops at the newline so brace counting resumes on the
  next line.
• method_keyword_external_language is narrowed to tsql and ispl. Those
  two, plus XData, triggers, queries and storage, keep
  external_method_body_content and the existing brace counter.
• The two Python rules in injections.scm point at the new token, so Python
  highlighting still works. Verified with tree-sitter query: the injection
  fires on the Python body and the range now stops at the correct brace.

Effect on parser output

One node name changes: external_method_body_content becomes
python_method_body_content on Language = python bodies. The new keyword
rules are hidden and aliased specifically to keep it to that. This is a breaking
change for a consumer matching that node on Python bodies, which is why it is
called out here rather than buried in the diff.

Playground grammar

objectscript/src/scanner.c is a near-copy of the UDL scanner, differing only
in the function prefix and two core-scanner flags, so the same changes are
ported there. Without that, its external token indices skew against the
regenerated grammar.json and unrelated Owner keyword tests fail.

Tests

New corpus cases in udl/test/corpus/python-method.txt:

• the brace inside a string, with a following ObjectScript method that must
  still parse, including its nested ##class(...) call
• quoting edge cases in one body: "close } brace", '{ open only',
  f"fstring {a} interp", """triple with } and { braces""",
  '''other triple }''', # comment with } and {, "escaped \" quote }",
  r"raw \ backslash }", and a real nested dict literal {"k": "v"}
• a Python trigger with a brace in a string
• a Language = tsql method as a control, asserting it still produces
  external_method_body_content

The existing Valid ClassMethod case in udl/test/corpus/class-method.txt
is updated for the new body token name, and the mirrored playground corpus is
regenerated through the pre-commit sync.

All five grammars pass, 2745 parses and 0 failures:

grammar	parses	failures
objectscript	918	0
udl	482	0
core	431	0
expr	476	0
objectscript_routine	438	0

Also run locally: npm run lint (clean), make checkquery, make lintquery,
make formatquery (no diff), npm test (3/3, which compiles the modified
scanner with no warnings), cargo test --lib (4/4, including the injections
query load), make test, and the CI examples loop over examples/*.cls.

Generated artifacts (parser.c, grammar.json, node-types.json) are
committed in sync with the grammar change per CONTRIBUTING, for the two
grammars that change: udl and objectscript. core, expr and
objectscript_routine are left untouched. udl/src/parser.c is large, so the
diff is mostly that file.

Context

Found while indexing IRIS source with a tree-sitter based code graph tool,
where a single Python method with a brace in a string silently dropped the
rest of the class from the index.

[hkimura-intersys]

thank you for catching this! I'm gonna run a few more tests with this change, and then should be able to merge this in. One thing before merging, commits must have verified signatures (if you set up a GPG key and set signing as default true, this will be resolved): https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification

[hkimura-intersys]

this is a great catch and the fix works without increasing the parser.c size by much. I know that some edge cases are python specific, but in general this bug could also happen in tsql, javascript and maybe? ispl, for example with something like SELECT '}' FROM Foo:

I like that Python is parsed separately because it has edge cases that doesn't apply to the other languages. But, this made me realize that for the other languages we should probably also add this fix (something like if the delimiter is within a string it shouldn't be counted). I am going to work off of this PR to add this fix to tsql, ispl and javascript cases (this will be done in scanner.h and referenced from the udl and objectscript scanners.

[hkimura-intersys]

closing this, as it can't be merged with an unsigned commit, so this commit is going to merged with my commit in #52

[isc-tdyar]

oh, I amended the commit, after adding ssh signing key - hopefully that all actually worked! I will make sure future commits are signed :)

[hkimura-intersys]

oh, I amended the commit, after adding ssh signing key - hopefully that all actually worked! I will make sure future commits are signed :)

Thank you! I merged your fix in with my commit as well with PR #52