[SPARK-59255][SQL] Extend parse_sql to parse SQL batches

[srielau]

What changes were proposed in this pull request?

Extend the experimental parse_sql function so it can parse a batch of SQL statements ('select 1; select 2') instead of a single statement.

parse_sql now:

• Splits the input with SqlStatementSplitter (the same splitter used by SparkSqlParser.splitStatements).
• Parses each statement independently and returns a JSON array of statement objects.
• Adds start (1-based offset in the original batch) and length (trimmed statement text, excluding surrounding whitespace and the terminating semicolon) on every statement object.
• Continues after a parse failure so later statements are still described.

Well-formed BEGIN ... END scripts remain a single array element. Nested error locations stay statement-relative; start is relative to the original batch. Empty or comment-only input returns []. NULL still returns SQL NULL.

The splitter now records source positions internally so spans are taken from token offsets rather than reconstructed with indexOf (which would mis-bind when a dropped comment repeats later statement text).

Why are the changes needed?

Users of the experimental parse_sql function asked to parse batches such as 'select 1; select 2'. Source spans are needed so consumers can highlight each sub-statement in the original text.

JIRA: https://issues.apache.org/jira/browse/SPARK-59255

Does this PR introduce any user-facing change?

Yes, behind spark.sql.function.parseSql.enabled (still off by default; the JSON contract is documented as evolving).

Previously a successful parse returned one JSON object:

{"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}

Now the same input is wrapped in an array and includes source spans:

[{"start":1,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}]

A two-statement batch:

SELECT parse_sql('select 1; select 2')

[
  {"start":1,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]},
  {"start":11,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}
]

JSON paths such as $.statement_identifier become $[0].statement_identifier. Empty SQL that previously produced a parse-failure object now returns [].

How was this patch tested?

• SqlStatementSplitterSuite (including comment / empty-; span recovery)
• ParseSqlResultSuite and ParseSqlSuite
• SQLQueryTestSuite -- -z parse-sql.sql (goldens regenerated)
• ExpressionsSchemaSuite and ExpressionInfoSuite example-output check
• catalyst/scalastyle and sql/scalastyle

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Grok 4.6

[cloud-fan]

Review summary

The new batch result shape, per-statement error handling, valid compound-script handling, documentation, and broad ASCII test coverage are consistent. The remaining issue is the source-position boundary: supplementary Unicode characters can corrupt both the advertised spans and the splitter's candidate validation, so the positional conversion and a focused regression should be fixed before merge. The pinned Build check was still in progress; no test failures were available to assess.

Findings

1 total: 0 P0, 1 P1, 0 P2, 0 P3.

Blocking (P1)

• Convert ANTLR offsets before using them as String spans — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:66 — see inline.

[cloud-fan]

Review summary

The batch parsing design and the prior supplementary-Unicode correctness fix are sound, but three source-span issues remain. Repeated from-zero offset conversions make ordinary multi-statement batches quadratic; String.trim leaves lexer-recognized Unicode whitespace in the returned spans; and clients are not told that the public coordinates count UTF-16 code units. These should be addressed before relying on the new batch contract.

Findings

3 total: 0 P0, 0 P1, 3 P2, 0 P3.

Non-blocking (P2)

• Document the UTF-16 unit used by source spans — sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala:42 — see inline.
• Avoid rescanning the SQL prefix for every statement position — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:197 — see inline.
• Trim spans with Spark SQL whitespace semantics — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:210 — see inline.

Re-review status

Prior AI findings: 1 addressed, 0 still present; additional unresolved findings in this review: 3.

New attribution: 1 newly introduced, 2 late catch, 0 previously raised, 0 unattributed.

Remaining prior AI findings

No prior AI findings remain.

Verification

• The previous supplementary-Unicode correctness failure is resolved at the pinned head; the retained performance issue is a distinct consequence of the new conversion strategy.

PR metadata suggestions

• Define the coordinate unit for start and length in the PR description; the current implementation and tests use UTF-16 code units, while the examples only cover ASCII.

[cloud-fan]

Non-blocking (P2): The returned start and length values are UTF-16 code-unit coordinates, but this public description does not name that unit. For SELECT '😀😀'; SELECT 2;, the API reports the first length as 13 and the second start as 16, while code-point-indexed clients reasonably expect 11 and 14 and will slice the wrong text. Please state explicitly that start is a 1-based UTF-16 code-unit offset and length is a UTF-16 code-unit count, and include a non-BMP example so clients can interpret the fields reliably.

[cloud-fan]

Non-blocking (P2): This conversion starts at index 0 for every statement, and tryParseRegion performs the same from-zero conversion for candidate boundaries. A batch of many short statements therefore rescans progressively larger prefixes, making ordinary splitting quadratic despite the documented O(n) contract. Please build the code-point-to-UTF-16 boundary mapping once, or maintain equivalent incremental offsets, so token-boundary lookup is constant time.

[cloud-fan]

Non-blocking (P2): String.trim only removes characters up to U+0020, but the Spark SQL lexer also treats Unicode spaces such as U+00A0 as whitespace. Consequently, parse_sql("\u00a0SELECT 1\u00a0;") retains both NBSPs and reports start 1/length 10 instead of the promised trimmed span at start 2/length 8. Please trim with the lexer's whitespace semantics and derive the leading UTF-16 offset from the same boundary.