Language = python method body scanner drops all following methods in the class

[isc-tdyar]

Summary

The ObjectScript UDL grammar lexes Language = python (and other
non-ObjectScript) method bodies as one opaque external_method_body_content
token using a plain brace counter. The counter is not string- or comment-aware,
so a { or } inside a Python string literal or # comment desynchronizes
it. The body then runs past its own closing brace and swallows the remainder of
the class.

Every method after the offending one disappears from the graph along with all
their edges. The only signal is a soft parse_partial warning naming a single
line range.

Reproduction

Two files. Tgt.cls:

Class My.Tgt Extends %RegisteredObject
{

ClassMethod Hit() As %String
{
    Quit "hit"
}

}

Brace.cls, where the { inside the Python string is ordinary Python:

Class My.Brace Extends %RegisteredObject
{

ClassMethod PyBad() As %String [ Language = python ]
{
    s = "dict literal { unbalanced"
    return s
}

ClassMethod RealOne() As %String
{
    Do ##class(My.Tgt).Hit()
    Quit "one"
}

ClassMethod RealTwo() As %String
{
    Do ##class(My.Tgt).Hit()
    Quit "two"
}

}

Index the directory, then:

MATCH (n:Method) RETURN n.name, n.qualified_name, n.lines

Observed:

rows: 2
  Hit     My.Tgt.Hit        lines=4
  PyBad   My.Brace.PyBad    lines=19   <-- swallowed the rest of the class

RealOne and RealTwo are absent. And:

MATCH (a)-[r:CALLS]->(b) RETURN a.name, b.qualified_name

returns zero rows. Both CALLS -> My.Tgt.Hit edges are gone. Hit itself is
indexed, so the call sites were never parsed rather than failing to resolve.

The index response reports only:

"parse_partial": {"files": [{"path": "Brace.cls", "error_ranges": "23-23"}], "count": 1}

A stray balanced } inside a string (s = "close } brace") fails
symmetrically: the body truncates early and ERROR nodes appear.

Root cause

internal/cbm/vendored/grammars/objectscript_udl/scanner.c:17-34.
lex_fenced_text tracks only nesting depth on { / }. Dispatched at
:60-66.

The grammar's language-based production switch works correctly.
parser.c:1585 defines sym_method_keyword_language; Language = python and
tsql yield external_method_keywords plus an opaque
external_method_body_content, while Language = cache and objectscript
yield parsed statement nodes. Only the fenced-text lexer is at fault.

Suggested fix

Make lex_fenced_text skip string literals and line comments while counting.
For Python bodies that means ', ", ''', """, string prefixes (f, r,
b, u and combinations), backslash escapes, and # to end-of-line.

Question before I write code

scanner.c is vendored from
intersystems/tree-sitter-objectscript,
which you audited and vendored yourself in #590. Patching it in-tree diverges
from the upstream grammar and will conflict on the next re-vendor.

How would you prefer this handled?

1. PR to intersystems/tree-sitter-objectscript, then you re-vendor. Clean,
   but gated on that repo's review latency.
2. In-tree patch to the vendored scanner.c with a comment marking the local
   delta, and an upstream PR filed in parallel.
3. Something else, such as you taking the vendored-file change yourself as with
   the scanner restore in feat(objectscript): InterSystems IRIS ObjectScript language support #467.

Happy to do the work either way. I'd rather not touch a vendored file without
your direction.

Related

The embedded-Python graph-representation issue (filed separately) depends on this being
fixed. Re-parsing the Python body is only meaningful once the body's byte range
is correct.

[isc-tdyar]

Update on the vendored-file question above: I went with option 1 and filed
the scanner fix upstream, so there's no in-tree patch to the vendored grammar
to review here.

intersystems/tree-sitter-objectscript#50

Summary of what it does, since it decides one thing on your behalf. Rather
than teaching lex_fenced_text about quotes, it splits the token.
external_method_body_content is shared with XData (XML and JSON), triggers,
SQL queries and storage, and the scanner has no language context, so a
blanket quote-skipping change would break XML prose containing apostrophes
and SQL -- comments. Instead Language = python bodies now get a separate
python_method_body_content token lexed by a Python-aware brace counter that
skips # comments, single and triple quotes, prefixed strings and backslash
escapes. tsql, ispl, XData, triggers, queries and storage keep the existing
token and the existing counter.

The only change to parser output is that node name:
external_method_body_content becomes python_method_body_content on
Language = python bodies. That means the re-vendor is not a drop-in. CBM's
extractor matches the old name, so whenever you next re-vendor, the
external_method_body_content handling needs to accept both names or this
issue reappears with the body simply missing instead of over-long. I'm happy
to send that CBM-side change as a small PR whenever the grammar version you
want to vendor is settled, or to leave it to you.

No action needed from you on this issue in the meantime. Upstream review
latency is out of my hands, and #1262 stays blocked until the byte ranges are
correct.

[DeusData]

Thanks for taking the scanner fix upstream and for documenting the non-drop-in node-name change.

Current main still matches the vendored external_method_body_content shape; there is no python_method_body_content handling in the extractor yet. Your warning is therefore binding for the eventual re-vendor: accepting the upstream grammar without teaching CBM both names would turn the over-long body into a missing body.

I have routed the parser-loss bug to 0.9.1-rc at priority/high. No awaiting-reporter label is applied because the next move is upstream review and then our re-vendor, not another report from you. Thank you for avoiding an untracked vendored delta.