Skip to content

fix(catalog): synthesize Postgres system columns (xmin, ctid, ...) (#3742)#4459

Open
luongs3 wants to merge 1 commit into
sqlc-dev:mainfrom
luongs3:fix/3742-postgres-system-columns
Open

fix(catalog): synthesize Postgres system columns (xmin, ctid, ...) (#3742)#4459
luongs3 wants to merge 1 commit into
sqlc-dev:mainfrom
luongs3:fix/3742-postgres-system-columns

Conversation

@luongs3
Copy link
Copy Markdown

@luongs3 luongs3 commented May 28, 2026

Problem

Fixes #3742.

sqlc errors with column "..." does not exist when a PostgreSQL query references any of the six system columns (tableoid, xmin, cmin, xmax, cmax, ctid) unless database.managed: true is configured.

The earlier fix (#2871 / #1745) only added these columns inside the PostgreSQL analyzer, which is initialized only when a live database is attached. Code-only pipelines that build the catalog purely from schema files still hit the same failure 17 months later. The issue thread on #3742 confirms this — swallowstalker already diagnosed it, and werjo-gridfuse noted that requiring a live DB doesn't work for code-gen pipelines.

Repro on main (no live DB):

CREATE TABLE authors (id BIGSERIAL PRIMARY KEY, name text NOT NULL, bio text);
-- query.sql
SELECT xmin, ctid FROM authors;
sql/query.sql:2:8: column "xmin" does not exist

Fix

Synthesize the six PostgreSQL system columns inside compiler.QueryCatalog.GetTable whenever the engine is postgresql. The catalog itself is left untouched (so MySQL/SQLite tables are not polluted), and the synthesized columns are marked with a new Column.IsSystem flag so SELECT * / RETURNING * expansion skips them — matching PostgreSQL's own behavior. Explicit references like SELECT xmin, ctid FROM foo and aliased forms like a.xmin resolve to the correct types (oid/xid/cid -> pgtype.Uint32, tid -> pgtype.TID).

Changed files:

  • internal/compiler/query.go — add Column.IsSystem.
  • internal/compiler/query_catalog.go — track engine on QueryCatalog; append pgSystemColumns in GetTable when engine is postgresql.
  • internal/compiler/expand.go — skip IsSystem columns in * expansion.
  • internal/compiler/output_columns.go — skip IsSystem columns in the output-columns * branch (mirrors expand.go).
  • internal/endtoend/testdata/system_columns_3742/postgresql/pgx/v5/ — new fixture exercising all six system columns, a table alias, and SELECT * (which must omit them). Golden files committed.

Tests

go test ./internal/compiler/... ./internal/sql/... ./internal/engine/... ./internal/codegen/...   # all pass
go test -run TestExamples ./internal/endtoend/                                                    # pass
go vet ./...                                                                                       # clean
gofmt -l <touched files>                                                                           # clean

Local end-to-end verification against the exact schema + query from the issue:

  • Before fix: EXIT=1, sql/query.sql:2:8: column "xmin" does not exist
  • After fix: EXIT=0, generates pgtype.Uint32 for xmin, pgtype.TID for ctid. SELECT * still expands to id, name, bio only.

What does NOT change

  • MySQL and SQLite catalogs are untouched.
  • The managed-database analyzer path (feat(postgresql): Support system columns on tables #2871) is unchanged and still works for users on database.managed: true.
  • SELECT * continues to omit system columns (matches PostgreSQL behavior).
  • The existing select_system end-to-end fixture (managed-db) is left alone; the new fixture covers the non-managed code path that was broken.

…qlc-dev#3742)

Problem
-------
sqlc errored with "column does not exist" when a PostgreSQL query
referenced any of the six system columns (tableoid, xmin, cmin, xmax,
cmax, ctid) unless `database.managed: true` was configured. The 2023 fix
(sqlc-dev#2871 / sqlc-dev#1745) only added the columns inside the PostgreSQL analyzer,
which is initialized only when a live database is attached. Code-only
pipelines that build the catalog purely from schema files still hit the
same "column \"xmin\" does not exist" failure 17 months later (sqlc-dev#3742).

Fix
---
Synthesize the six PostgreSQL system columns inside
`compiler.QueryCatalog.GetTable` whenever the engine is `postgresql`.
The catalog itself is left untouched (so MySQL/SQLite tables are not
polluted), and the synthesized columns are marked with a new
`Column.IsSystem` flag so `SELECT *` / `RETURNING *` expansion skips
them — matching PostgreSQL's own behavior. Explicit references like
`SELECT xmin, ctid FROM foo` and aliased forms like `a.xmin` resolve to
the correct types (oid/xid/cid -> pgtype.Uint32, tid -> pgtype.TID).

Changed files
-------------
- internal/compiler/query.go: add `Column.IsSystem`.
- internal/compiler/query_catalog.go: track engine on QueryCatalog;
  append `pgSystemColumns` in GetTable when engine == postgresql.
- internal/compiler/expand.go: skip IsSystem columns in `*` expansion.
- internal/compiler/output_columns.go: skip IsSystem columns in the
  output-columns `*` branch (mirror of expand.go).
- internal/endtoend/testdata/system_columns_3742/postgresql/pgx/v5/:
  new fixture exercising all six system columns, a table alias, and
  SELECT * (which must omit system columns). Golden files committed.

Tests
-----
- `go test ./internal/compiler/... ./internal/sql/... ./internal/engine/... ./internal/codegen/...` — all pass.
- `go test -run TestExamples ./internal/endtoend/` — passes.
- `go vet` + `gofmt -l` clean on touched files.

Local verification
------------------
Built `/tmp/sqlc-dev-3742` from this branch and ran it against the exact
schema + query in the issue. Before fix: `EXIT=1, column "xmin" does
not exist`. After fix: `EXIT=0`, generates `pgtype.Uint32` for xmin,
`pgtype.TID` for ctid. Confirmed `SELECT *` still expands to only the
user columns (id, name, bio) — system columns are not included.

What does NOT change
--------------------
- MySQL / SQLite catalogs are untouched.
- The managed-database analyzer path (sqlc-dev#2871) is untouched and still
  works for users on `database.managed: true`.
- `SELECT *` continues to omit system columns (PG-compatible).
- The existing `select_system` endtoend fixture (managed-db) is left
  alone; the new fixture covers the non-managed code path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Postgres system columns not working

1 participant