Repository navigation
perf(analyser): make identifierTooLong cheap and complete - #897
Merged
Merged
Conversation
Skip the scan unless the statement has a run of identifier bytes or a quoted section longer than 63 bytes, which is about 1% of the statements of the Postgres regression suite. Decode U&"..." identifiers, including UESCAPE, and report each name once per statement.
psteinroe
added a commit
that referenced
this pull request
Oct 26, 2026
…ction and banDuplicateColumnAssignments (#909) Stacked on #905, and merges #897 (both change `identifierTooLong`), so until #897 lands the diff here includes it; the change of this PR is the last commit. Three of the rules ported from Squawk in #885 get fixes, using the fix API from #859. Squawk has fixes for these; `requireEnumValueOrdering`'s Squawk fix inserts a placeholder that doesn't run, so it isn't ported. **identifierTooLong: safe** The fix spells out the name Postgres already uses: the identifier truncated to 63 bytes at a character boundary, the way Postgres truncates it. Since #897 reports each name once per statement, the fix rewrites every identifier of the statement that stands for the name: ```sql select aaaa…(64).id from aaaa…(64) where AAAA…(64).id = 1; -- fix: select aaaa…(63).id from aaaa…(63) where AAAA…(63).id = 1; ``` Unquoted identifiers keep their case as written, quoted ones are re-quoted from the truncated name (so a `""` cut in half can't break the quoting). `U&"..."` identifiers are left alone, because rewriting them would leave a `UESCAPE` clause behind. The report proposed making it unsafe for quoted names. I made it safe for all: the statement does exactly the same afterwards, quoted or not, which is what `FixKind::Safe` means. Other statements that use the long name get their own diagnostic and fix, and leaving them as they are is still equivalent. The fix doesn't pick a good short name, that's still up to the user (the note says so). **banUncommittedTransaction: unsafe** Adds `commit;` on a new line after the last statement of the file, or of the migration section the `BEGIN` is in: ```sql -- section: up transaction:false begin; create table users (id bigint); commit; -- added -- section: down transaction:false ``` Unsafe because it changes what the migration does (its changes are now applied), and whether `COMMIT` belongs at the end, or a `ROLLBACK` was meant, is a guess. To know where the file or section ends, `AnalysedFileContext` now knows where its statements start and where the last one ends, including statements that don't parse, via `end_offset()`, relative to the current statement like other fix edits. **banDuplicateColumnAssignments: unsafe** Keeps the last assignment of a SET clause (UPDATE, ON CONFLICT DO UPDATE, MERGE) and removes the earlier ones with their commas. Only if every assignment to the column is a plain `col = value`: no fix for INSERT column lists (the values would have to move too), `(a, b) = ...` assignments, or subscripts and fields. Unsafe because we can't tell which value was meant. **Tests** Every spec snapshot of the three rules now shows the fixed SQL, and the test harness checks it still parses. New specs: `banDuplicateColumnAssignments/fixes.sql` (comments between assignments, ON CONFLICT, and the cases without a fix), `banUncommittedTransaction/fix_in_section.sql`. **Risks** The fixes are plain token edits, independent of the fix infrastructure changes in flight. One interaction to watch: #906 checks each fix by applying it to the statement text and re-parsing. The COMMIT fix's edit lies past the end of the `BEGIN` statement (at the end of the file or section), so `RuleFix::apply` on the statement text would slice out of range. Whichever of the two lands second has to handle edits outside the statement, e.g. skip the re-parse check for them or apply the fix to the file text. **Follow-ups:** #906 keeps fixes that edit past the end of their statement without re-parsing them, so the `commit;` fix works with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three follow-ups to
identifierTooLongfrom #885 (recommended, so it runs on every statement).Cost
The rule ran
pgls_query::scanon every statement, because the parser has already truncated the names in the AST. Scanning costs about as much as parsing, and the rule was about 84% of the cost of all rules ported in #885.It now skips the scan unless the statement could contain a long identifier: a run of more than 63 bytes that can be part of an unquoted identifier (
[A-Za-z0-9_$]or a non-ASCII byte), or more than 63 bytes between two quotes. A name is never longer than its text (downcasing keeps the length,""and Unicode escapes are longer than what they stand for), so the check can't miss one. Quotes in strings or comments can't hide a quoted identifier either: inside one, quotes come in pairs, so only an odd run of quotes counts as a boundary.Measured on a debug build as on-CPU thread time (minimum of 5 runs; the machine was busy, so wall-clock is meaningless):
For comparison, 6.3% of the corpus statements contain a
", so "skip unless there's a quote" would have scanned six times as many.U&"..."identifiersOnly
IDENTtokens were checked, so this was silent although Postgres prints a truncation notice:UIDENTtokens are now decoded likestr_udeescapedoes (4- and 6-digit escapes, surrogate pairs, doubled escape characters, and theUESCAPEcharacter ascheck_uescapecharallows it). Identifiers whose escapes Postgres rejects are skipped. Each case in the spec was checked against Postgres for the NOTICE.One diagnostic per name
A long table name used three times in a statement gave three identical diagnostics. Each distinct name is now reported once per statement, at its first occurrence. Unquoted names are compared, and shown, downcased like Postgres does (ASCII only), so
Long_Name…andlong_name…are the same name.Tests
Spec files
edge_cases.sql(multi-byte names, doubled quotes, strings/comments/dollar-quoted bodies, a quote in a string next to a long quoted identifier with spaces, the repeated name) andunicode_escapes.sql; unit tests for the pre-filter (including the cases that must pass it) and theU&decoding.Not covered
libpg_query (PG 17) fails to parse a comment between
U&"..."andUESCAPE, which Postgres accepts, so such a statement is reported as a syntax error and never reaches the rule.