Skip to content

perf(analyser): make identifierTooLong cheap and complete - #897

Merged
psteinroe merged 1 commit into
mainfrom
perf/identifier-too-long
Oct 26, 2026
Merged

psteinroe merged 1 commit into
mainfrom
perf/identifier-too-long

Conversation

@psteinroe

Copy link
Copy Markdown
Collaborator

Three follow-ups to identifierTooLong from #885 (recommended, so it runs on every statement).

Cost

The rule ran pgls_query::scan on every statement, because the parser has already truncated the names in the AST. Scanning costs about as much as parsing, and the rule was about 84% of the cost of all rules ported in #885.

It now skips the scan unless the statement could contain a long identifier: a run of more than 63 bytes that can be part of an unquoted identifier ([A-Za-z0-9_$] or a non-ASCII byte), or more than 63 bytes between two quotes. A name is never longer than its text (downcasing keeps the length, "" and Unicode escapes are longer than what they stand for), so the check can't miss one. Quotes in strings or comments can't hide a quoted identifier either: inside one, quotes come in pairs, so only an odd run of quotes counts as a boundary.

Measured on a debug build as on-CPU thread time (minimum of 5 runs; the machine was busy, so wall-clock is meaningless):

before after
scan step over the PG 15–18 regression corpus (172,586 statements) 2.36 s 0.28 s (1,850 statements, 1.1%, still get scanned)
rule over the PG 18 corpus, minus the no-rule baseline (46,158 statements) +0.93 s +0.19 s or less, within noise
rule on a synthetic 10k-statement migration, minus the baseline +279 ms ~0 ms

For comparison, 6.3% of the corpus statements contain a ", so "skip unless there's a quote" would have scanned six times as many.

U&"..." identifiers

Only IDENT tokens were checked, so this was silent although Postgres prints a truncation notice:

create table U&"aaaa…(64 bytes)…" (id int);
create table U&"x!0061!0061…(80 escapes)" uescape '!' (id int);

UIDENT tokens are now decoded like str_udeescape does (4- and 6-digit escapes, surrogate pairs, doubled escape characters, and the UESCAPE character as check_uescapechar allows it). Identifiers whose escapes Postgres rejects are skipped. Each case in the spec was checked against Postgres for the NOTICE.

One diagnostic per name

A long table name used three times in a statement gave three identical diagnostics. Each distinct name is now reported once per statement, at its first occurrence. Unquoted names are compared, and shown, downcased like Postgres does (ASCII only), so Long_Name… and long_name… are the same name.

Tests

Spec files edge_cases.sql (multi-byte names, doubled quotes, strings/comments/dollar-quoted bodies, a quote in a string next to a long quoted identifier with spaces, the repeated name) and unicode_escapes.sql; unit tests for the pre-filter (including the cases that must pass it) and the U& decoding.

Not covered

libpg_query (PG 17) fails to parse a comment between U&"..." and UESCAPE, which Postgres accepts, so such a statement is reported as a syntax error and never reaches the rule.

Skip the scan unless the statement has a run of identifier bytes or a quoted
section longer than 63 bytes, which is about 1% of the statements of the
Postgres regression suite. Decode U&"..." identifiers, including UESCAPE,
and report each name once per statement.
@psteinroe
psteinroe merged commit fb814bd into main Oct 26, 2026
11 checks passed
@psteinroe
psteinroe deleted the perf/identifier-too-long branch October 26, 2026 07:54
psteinroe added a commit that referenced this pull request Oct 26, 2026
…ction and banDuplicateColumnAssignments (#909)

Stacked on #905, and merges #897 (both change `identifierTooLong`), so until #897 lands the diff here includes it; the change of this PR is the last commit.

Three of the rules ported from Squawk in #885 get fixes, using the fix API from #859. Squawk has fixes for these; `requireEnumValueOrdering`'s Squawk fix inserts a placeholder that doesn't run, so it isn't ported.

**identifierTooLong: safe**

The fix spells out the name Postgres already uses: the identifier truncated to 63 bytes at a character boundary, the way Postgres truncates it. Since #897 reports each name once per statement, the fix rewrites every identifier of the statement that stands for the name:

```sql
select aaaa…(64).id from aaaa…(64) where AAAA…(64).id = 1;
-- fix: select aaaa…(63).id from aaaa…(63) where AAAA…(63).id = 1;
```

Unquoted identifiers keep their case as written, quoted ones are re-quoted from the truncated name (so a `""` cut in half can't break the quoting). `U&"..."` identifiers are left alone, because rewriting them would leave a `UESCAPE` clause behind.

The report proposed making it unsafe for quoted names. I made it safe for all: the statement does exactly the same afterwards, quoted or not, which is what `FixKind::Safe` means. Other statements that use the long name get their own diagnostic and fix, and leaving them as they are is still equivalent. The fix doesn't pick a good short name, that's still up to the user (the note says so).

**banUncommittedTransaction: unsafe**

Adds `commit;` on a new line after the last statement of the file, or of the migration section the `BEGIN` is in:

```sql
-- section: up transaction:false
begin;
create table users (id bigint);
commit;            -- added

-- section: down transaction:false
```

Unsafe because it changes what the migration does (its changes are now applied), and whether `COMMIT` belongs at the end, or a `ROLLBACK` was meant, is a guess. To know where the file or section ends, `AnalysedFileContext` now knows where its statements start and where the last one ends, including statements that don't parse, via `end_offset()`, relative to the current statement like other fix edits.

**banDuplicateColumnAssignments: unsafe**

Keeps the last assignment of a SET clause (UPDATE, ON CONFLICT DO UPDATE, MERGE) and removes the earlier ones with their commas. Only if every assignment to the column is a plain `col = value`: no fix for INSERT column lists (the values would have to move too), `(a, b) = ...` assignments, or subscripts and fields. Unsafe because we can't tell which value was meant.

**Tests**

Every spec snapshot of the three rules now shows the fixed SQL, and the test harness checks it still parses. New specs: `banDuplicateColumnAssignments/fixes.sql` (comments between assignments, ON CONFLICT, and the cases without a fix), `banUncommittedTransaction/fix_in_section.sql`.

**Risks**

The fixes are plain token edits, independent of the fix infrastructure changes in flight. One interaction to watch: #906 checks each fix by applying it to the statement text and re-parsing. The COMMIT fix's edit lies past the end of the `BEGIN` statement (at the end of the file or section), so `RuleFix::apply` on the statement text would slice out of range. Whichever of the two lands second has to handle edits outside the statement, e.g. skip the re-parse check for them or apply the fix to the file text.

**Follow-ups:** #906 keeps fixes that edit past the end of their statement without re-parsing them, so the `commit;` fix works with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant