Skip to content

nbtree: opt-in leaf key prefix compression - #53

Open
gburd wants to merge 1 commit into
masterfrom
nbtkpc
Open

gburd wants to merge 1 commit into
masterfrom
nbtkpc

Conversation

@gburd

@gburd gburd commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

nbtree: opt-in leaf key prefix compression

Opened against this fork's master so CI runs before cfbot does. This branch
carries the version posted to pgsql-hackers (CF 7523, v9 and later): one
commit on top of postgres/postgres master.

v9 replaces the single-column, varlena-only design of v1-v8. The design
write-up is the thread message that carries v9; the commit message below
describes what the code does.

Commit message

nbtree: opt-in leaf key prefix compression

Add a prefix_compression storage parameter for btree indexes (default
off).  With it on, a leaf page may store the leading bytes that a run of
its tuples share just once, in a per-page directory, and keep only each
tuple's remainder.  Indexes on keys with long shared prefixes (URLs,
jsonb paths, zero-padded codes, multicolumn keys with low-cardinality
leading columns) get much smaller.

Sharing is decided by equality of stored bytes, never by opclass
equality.  That keeps it sound for every type, including types whose
equal values can differ in representation (numeric 1.0 and 1.00): only
bytes that are already identical are shared, so an index-only scan gets
back exactly what was stored, and no equalimage support is required.  A
tuple shares the longest common prefix of its data area with its
group's anchor, so whole leading columns and leading bytes of the first
differing column are both covered.

A compressed leaf carries the new BTP_PREFIX flag and one directory
item at the offset where the first data item would otherwise be.  Its
data items follow in key order, so P_FIRSTDATAKEY still depends only on
the page flags, binary search is unchanged, and neither
BTPageOpaqueData nor BTREE_VERSION changes.  The directory holds one or
more groups of consecutive items, each with an anchor (a copy of the
key part of its first member).  A coded item keeps its real heap TID in
t_tid and any posting list in the standard layout, so code that reads
only TIDs (VACUUM, LP_DEAD hinting, deletion, posting list splits) works
on coded items unchanged.  Pivot tuples are never coded.  Readers get a
decoded tuple through _bt_getitem(); _bt_readpage() skips decoding when
a scan needs only heap TIDs and every scan key is already satisfied for
the page.

Leaves are compressed when they are written whole.  CREATE INDEX fills
each leaf according to a running estimate of its coded size, honoring
fillfactor in coded bytes; a page split writes both halves plain and
codes each.  An insert codes the new tuple against its neighbor's group.
Deduplication and bottom-up deletion work on decoded copies of a coded
page's items, and deduplication re-codes the result against the page's
existing directory, so prefix compression and deduplication compose on
one page.  An emptied coded leaf drops its directory and becomes a plain
page again, as page deletion expects.

A coded page can hold more than one page of decoded data but must still
split into two plain pages.  The directory caches the decoded size of
the page's items; inserts split before that size would exceed what a
balanced split can always place on two pages.  For the rare case where
a large incoming tuple leaves no legal split point, the existing items
are split alone and the insert is retried.

Coding and decoding use only the page and the tuple, never the catalog,
so WAL replay rebuilds coded pages without full-page images.  An insert
logs the coded tuple.  xl_btree_split gains a flags field saying whether
the original page and each half were coded; redo decodes the original
page, collects the left half's items as the primary did and lays them
out with the same routine, while the right half arrives as its coded
items.  Deduplication and posting list splits on coded pages are
replayed by repeating the primary's steps.  The xl_btree_split change
needs an XLOG_PAGE_MAGIC bump.

Indexes without the parameter, and pages without the flag, are read
exactly as before, so pg_upgrade and existing indexes need nothing.
Turning the parameter on or off with ALTER INDEX rewrites nothing: each
leaf converts the next time it is written whole, and coded leaves stay
readable with the parameter off.

amcheck decodes coded items and checks the directory's cached sizes.
pageinspect's bt_page_items shows coded items decoded, with itemlen
giving the stored size.  A new recovery test replays every coded-page
WAL record on a standby with wal_consistency_checking enabled.

On 2 million URL-like keys (one shared 37-byte prefix plus a random
number), compared with the same index without the parameter: the index
is 37% smaller when grown by inserts and 35% smaller when built (52%
for zero-padded sequential keys); inserts take about 10% more time and
write 17% less WAL; CREATE INDEX takes about 10% longer; warm reads are
unchanged and cold-cache index scans are 25-35% faster.

Discussion: https://postgr.es/m/at-01F1wsXLRH0qo@floki

Changes since v9 as posted (v10/v11/v12)

  • v12: Split-fit accounting fix (the directory's cached decoded size counted
    unaligned IndexTupleSize; the split and build estimate use MAXALIGN, so the
    gate could undercount by up to 7B/item and over-admit). Count MAXALIGN'd
    size. Added a StaticAssert tying BTREE_PREFIX_CODE_OVERHEAD to the header +
    posting-pointer sizes, and a redo assert on the split left-item count.
    Threading: the two decode scratch buffers in _bt_compare and
    _bt_prefix_try_plain_page were function-local statics; made them a stack
    local and a caller-owned buffer respectively (reentrancy for the
    threaded-server work). No format change.
  • v11: README grouping section rewritten (doc only).
  • v10: Multiple prefix groups per page trigger correctly (reported by Yao
    Feng); regression test groups_compress.
  • Removed unused _bt_prefix_ngroups (clang -Wunused-function).
  • t/061_nbtree_prefix.pl uses a physical replication slot.
  • Rebased on current master.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

📜 Change history & discussion (Agora / pg.ddx.io)

I have the full picture now. Let me quickly confirm the prior-art threads (Matthias's dynamic prefix truncation and the old 2005/2014 prefix btree threads are confirmed from search results). I have enough to write the report.

🧵 Related discussion

  • nbtree: opt-in leaf key prefix compression — Greg Burd, 2026-10-26. This is the thread for this PR (not yet committed; 002ac2adde7 is not in the upstream index). v1 through v9 posted here. Key points:
    • Per-index prefix_compression storage parameter, off by default.
    • New per-page BTP_PREFIX leaf format: one PREFIX item holds shared bytes, each data item stored "coded" with its prefix replaced by a one-byte length. No BTREE_VERSION bump — pages "fault forward" only when rewritten (build/split/dedup), so old and new pages coexist, safe across pg_upgrade/replication.
    • Composes with deduplication; bottom-up deletion runs unchanged on coded pages.
    • WAL redo reconstructs coded bytes via a shared _bt_split_leaf_layout (no FPI per change), with a bounded FPI fallback. Author flags this and posting-list-split-on-coded-page as the fiddliest areas.
  • Peter Geoghegan's review (2026-11-03) is the substantive design critique. Note it carefully:
    • Peter attached his own April 2026 vibe-coded prefix-compression prototype for comparison. Tradeoff: Peter's composes poorly with dedup but does column-level compression; Burd's composes with dedup but only compresses a varlena first key column (no multi-column/column-level compression).
    • Core objections: patch leads with WAL mechanics instead of the architectural tradeoffs; the compressed prefix is not used as a low key (doesn't bound the page key space), so late-inserted in-range-but-uncompressible tuples are simply left uncoded — Peter wants justification. He cites Modern B-Tree Techniques on key-space vs ad-hoc prefix truncation and asks where this design sits in that taxonomy. He also wants a hard guarantee that a legal page split is always available even in pathological cases.
    • Blunt meta-point: AI-generated prototypes converging on "complete" mean little; wants a motivated design-space discussion first.
    • Motivation (from Burd's reply): tied to his "HOT for Expression Indexes" work — making JSONB heap updates HOT-able.
  • Reviewer Yao Feng also engaged (reply).

🔗 Related commits / prior art

  • Prefix compression of B-Tree keys — Claudio Freire, 2014. Earliest modern discussion of exactly this idea; never committed.
  • prefix btree implementation — Qingqing Zhou, 2005, added to TODO by Bruce Momjian. The original feature request.
  • btree: implement dynamic prefix truncation — Matthias van de Meent. Distinct and complementary: that is search-time dynamic prefix skipping of _bt_compare, not on-disk compression. Burd's patch changes the on-disk representation; Matthias's does not. Don't conflate them.
  • Suffix truncation (committed) is the adjacent shipped feature: _bt_truncate() and "Making all nbtree entries unique by having heap TIDs participate in comparisons" (Peter Geoghegan, PG12). This patch is prefix compression on leaf pages, orthogonal to suffix truncation on high keys.
  • v2 was rebased over Peter's d42c5457e5 (multi-level page-deletion WAL fix), per the author — not a dependency, just a collision in nbtpage.c.

📋 Commitfest

  • CF #62, entry 7523 — "nbtree: opt-in leaf key prefix compression", status Needs review, author Greg Burd.

🧭 Context for reviewers

  • This is an unmerged, actively-contested patch. The PR commit 002ac2adde7 does not exist upstream; treat the PR as a snapshot of an in-flight patch (upstream was at v9+).
  • The central unresolved design questions are Peter's, not cosmetic: (1) why first-column-varlena-only instead of column-level compression, which likely can't be retrofitted post-commit; (2) prefix-as-base-key vs. not using it as a low key; (3) a provable guarantee that a legal split always exists on a coded page. Any review should start there, not at the WAL code.
  • On-disk compatibility claim ("no version bump, faults forward, downgradeable by turning the GUC off") is the load-bearing safety argument — verify the coexistence and byte-for-byte redo claims (wal_consistency_checking=all) hold, since there is no version gate to catch a standby divergence.
  • Competing implementation exists (Peter's April prototype, attached to his review) with opposite tradeoffs on dedup composition vs. column-level scope — a committed design may need to borrow from both.

Generated by pg-history via the Agora MCP server (pg.ddx.io).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 11 issue(s).

  • 11 inline, 0 in summary

Comment thread src/include/access/nbtxlog.h Outdated
Comment on lines +165 to +169
* xl_btree_split flag bits. The left (original) page is restored from a
* full-page image when it is prefix-compressed, so it needs no flag; the
* right page is rebuilt from logged items, so redo must know to set
* BTP_PREFIX on it (its PREFIX item is included in the logged tuples).
*/

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This newly-added comment is factually wrong and contradicts the actual redo code. The only reader of XLH_SPLIT_RPREFIX is btree_xlog_split() (nbtxlog.c:291-296), which restores the prefix-compressed right page from a full-page image (it requires BLK_RESTORED and elog(ERROR)s otherwise). It does NOT rebuild the page from logged items, and it never sets BTP_PREFIX explicitly. Correspondingly the primary registers block 1 with REGBUF_STANDARD | REGBUF_FORCE_IMAGE (no REGBUF_WILL_INIT, no logged tuple data) when P_HASPREFIX(ropaque) (nbtinsert.c:2157-2159). So the claims "the right page is rebuilt from logged items" and "its PREFIX item is included in the logged tuples" are both false. The flag's real purpose is to tell redo that block 1 must be restored from its forced FPI (and thus to skip the item-replay path). Please correct the comment to match the code. (high confidence)

Suggested change
* xl_btree_split flag bits. The left (original) page is restored from a
* full-page image when it is prefix-compressed, so it needs no flag; the
* right page is rebuilt from logged items, so redo must know to set
* BTP_PREFIX on it (its PREFIX item is included in the logged tuples).
*/
* xl_btree_split flag bits. Both halves of a prefix-compressed leaf split
* are restored from full-page images forced on the primary (their coded
* items are laid out in a physical order that _bt_restore_page would not
* reproduce). The left (original) page is block 0, whose FPI redo already
* detects via BLK_RESTORED, so it needs no flag. This flag tells redo that
* block 1 (the right page) must likewise be restored from its forced FPI
* rather than rebuilt from logged items.
*/

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +578 to +581
SELECT
(SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix) =
(SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix WHERE u IS NOT NULL OR u IS NULL)
AS digest_consistent;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "full differential digest" is tautological and cannot catch any regression. Both subqueries read the same table btree_prefix via the same access path, and the predicate u IS NOT NULL OR u IS NULL is always true, so the two digests are byte-for-byte identical by construction. digest_consistent returns t even if prefix compression is reverted or corrupts data. The comment promises comparing "the prefix index, the uncompressed index, and a sequential scan" but the SQL compares none of those. To actually distinguish the implementations, force each access path explicitly and compare: e.g. digest of a scan forced through btree_prefix_on (prefix) vs one forced through btree_prefix_off (uncompressed) vs a seqscan (enable_indexscan=off), and assert all three agree. As written this test is worthless.

Comment thread src/test/regress/sql/btree_index.sql Outdated
-- by keys that share a long prefix (URL-like), keys that share almost nothing
-- (md5), fixed-width zero-padded codes, and oversized (>126-byte) values.
--
CREATE TABLE btree_prefix (u text, v text);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Locale/collation portability defect. The text column u is created with the database default collation, so =, BETWEEN, u > ..., ORDER BY u, and string_agg(u, '|' ORDER BY u) all depend on the server's collation. Under a non-C collation the recorded row order in expected/btree_index.out (e.g. the backward-scan output and every digest) will differ, breaking the buildfarm. This same test file already follows the C-collation convention (create temp table btree_bpchar (f1 text collate "C")). Declare the key column as u text COLLATE "C" (and the digest/ORDER BY accordingly) so contents and expected output are deterministic across platforms.

Comment thread src/test/regress/sql/btree_index.sql Outdated
SELECT 'https://example.com/other/' || lpad(g::text, 8, '0'), md5((-g)::text)
FROM generate_series(1, 10000) g;
-- Oversized first-column values (stored with shared = 0) and a NULL.
INSERT INTO btree_prefix VALUES (repeat('x', 300), 'big'), (NULL, 'nullkey');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boundary coverage is missing at the one place the encoding actually switches. BTREE_PREFIX_MAX is 126 and BTREE_PREFIX_RAW is 127 (see nbtree.h); the shared-count field is a single uint8 masked to 7 bits, so the coded/raw transition happens around 126/127 shared bytes, not at "300". repeat('x', 300) merely has zero shared prefix with the URL keys (shared = 0), which is a different path from the >126 case the comment claims to exercise. Add keys that share exactly 125/126/127 bytes with the page prefix to cover the boundary where the shared encoding changes behavior.

ALTER INDEX btree_prefix_off SET (prefix_compression = on);
SELECT reloptions FROM pg_class WHERE relname = 'btree_prefix_off';
-- A non-text (non-varlena) first column silently stores no prefix, but is legal.
CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The non-varlena index is created immediately before DROP TABLE and never queried, so this only asserts that CREATE INDEX doesn't error. The comment claims the "silently stores no prefix, but is legal" path is covered, but the correctness of scans over that index (the actual behavior being tested) is never exercised. Add at least one query using this index (forced via enable_seqscan=off) and verify it returns the expected rows, or drop the index assertion if it adds no value.

Comment thread src/backend/access/nbtree/README Outdated
Comment on lines +1131 to +1132
Prefix split halves, and a deduplication rewrite of a prefix page, are logged
by full-page image rather than by replaying individual tuples, because the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor documentation imprecision. The code (nbtinsert.c) forces the left page's full-page image based on whether the pre-split original page is BTP_PREFIX (P_HASPREFIX(origpage)), not based on whether the resulting left half ends up prefix-compressed -- a left half can end up plain yet still be FPI-logged because its source items are coded against the original prefix. The phrase "Prefix split halves ... are logged by full-page image" reads as if only the halves that remain prefix-compressed are FPI'd. Consider clarifying that the left half is FPI-logged whenever the original page was prefix-compressed. Low-confidence/low-severity nit; the overall mechanism is correct.

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +554 to +556
SELECT md5(string_agg(u, '|' ORDER BY u)) =
(SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix) AS ios_ok
FROM (SELECT u FROM btree_prefix ORDER BY u) s;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "index-only scan" check is tautological, same defect as the full differential digest below. Both operands derive from the identical source btree_prefix and are wrapped in string_agg(u, '|' ORDER BY u), which sorts its input, so the two md5 digests are byte-for-byte equal by construction regardless of which access path (prefix index, uncompressed index, or seqscan) each side uses. If key decoding were broken, both sides would return the same wrong digest and ios_ok would still be t. The test cannot catch a regression in the new prefix-decode path. To actually test decoding, compare the prefix index against an independent oracle, e.g. force the IOS over btree_prefix_on and compare its ordered output to the original literal values inserted (or to a seqscan result that provably did not use the index).

Comment thread src/test/regress/sql/btree_index.sql Outdated
-- by keys that share a long prefix (URL-like), keys that share almost nothing
-- (md5), fixed-width zero-padded codes, and oversized (>126-byte) values.
--
CREATE TABLE btree_prefix (u text, v text);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second column v is populated by every INSERT (md5 values) but never referenced by any query in this test block. It is dead test data that only adds noise and insert cost. Either drop the column (the index is on u only) or use it, e.g. to exercise a two-column prefix-compressed index.

Comment thread src/include/access/nbtree.h Outdated
Comment on lines +1243 to +1247
#define BTTupleStorageSize(relation) \
(BTGetPrefixCompression(relation) ? \
(BLCKSZ + (BLCKSZ / (int) MAXALIGN(sizeof(IndexTupleData) + 2)) * \
(int) MAXALIGN(BTREE_PREFIX_MAX)) : \
BLCKSZ)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory-safety bug (high confidence): BTTupleStorageSize falls back to plain BLCKSZ whenever BTGetPrefixCompression(relation) is false, but BTP_PREFIX pages are not tied to the current reloption value. Per the README and the code (flags are only cleared on split/empty/dedup-rewrite, never by an ALTER INDEX ... SET (prefix_compression = off)), BTP_PREFIX leaves persist in an index after the option is turned off. _bt_getitem() decodes based on the page flag P_HASPREFIX(opaque), not the reloption, so an index-only scan over a surviving BTP_PREFIX leaf will decode tuples whose combined size can exceed BLCKSZ and overflow so->currTuples/so->markTuples (allocated as half * 2 in btrescan). The sizing must not depend on the live option; size the buffer for the worst case whenever prefix compression is a btree index possibility (e.g. drop the BTGetPrefixCompression gate, or key it on whether the index could ever contain BTP_PREFIX pages).

Comment thread src/backend/access/nbtree/README Outdated
Comment on lines +1127 to +1129
parameter on with ALTER INDEX does not rewrite anything; each leaf is converted
the next time it is written whole, and compressed and uncompressed leaves
coexist in the same index indefinitely.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation gap (moderate confidence): this paragraph only describes turning the parameter on and asserts compressed and uncompressed leaves "coexist in the same index indefinitely", but it never states what happens when the parameter is turned off. The code keeps BTP_PREFIX on existing pages (flags are cleared only on split/empty/dedup-rewrite), so BTP_PREFIX leaves survive an ALTER INDEX ... SET (prefix_compression = off) and remain readable. This is not merely cosmetic: it is exactly the invariant the index-only-scan buffer sizing (BTTupleStorageSize in nbtree.h) must honor. Document that turning the option off also does not rewrite anything, so prefix pages can be read back with the option off.

@gburd

gburd commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Addressed the review in 6edd15c:

  • Memory-safety bug (nbtree.h BTTupleStorageSize) — fixed. It no longer
    gates on the live reloption; a BTP_PREFIX leaf survives ALTER ... SET (prefix_compression = off) and _bt_getitem decodes on the page flag, so
    the IOS buffers are now sized for a decoded page on every btree index.
    Verified: build-on, ALTER SET off, then an index-only scan over the
    surviving compressed leaves returns all rows, digest matches an independent
    oracle, server stays up (previously this overran currTuples).
  • Tautological regress tests — reworked. Comparisons are now against
    independent oracles (literal values built with generate_series, and an
    index-only-scan digest compared to a forced seqscan), the access path is
    pinned with EXPLAIN, the key column is COLLATE "C", the 125/126/127
    shared-byte encoding boundary is covered, the non-varlena-first-column index
    is queried, reading prefix pages after the option is turned off is exercised,
    and the unused second column is dropped.
  • Wrong XLH_SPLIT_RPREFIX comment (nbtxlog.h) and the matching _bt_split
    comment
    — corrected: both split halves are restored from forced full-page
    images; the flag tells redo to restore the right page from its image, not to
    rebuild it from logged tuples.
  • README — documents that turning the option off rewrites nothing (prefix
    pages stay readable, which is what the IOS buffer sizing must honor) and
    clarifies that the left half is FPI-logged whenever the pre-split page was
    prefix-compressed.

Re-ran on Fedora: regress + isolation + amcheck + pageinspect + pgstattuple all
green, and 027_stream_regress with wal_consistency_checking.

The four larger follow-ups (compose with deduplication, re-enable bottom-up
deletion on prefix pages, drop the split/dedup FPI, and the consequent extra
split WAL) remain as described in the PR body; they are deliberately separate
steps and the WAL ones are gated behind a throughput/WAL A/B benchmark that is
not yet done.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 19 issue(s).

  • 17 inline, 2 in summary

📄 src/backend/access/nbtree/README

This paragraph describes the opposite of what the code does. A prefix split does NOT full-page-image its halves as the primary mechanism. nbtinsert.c registers block 0 with REGBUF_STANDARD and block 1 with REGBUF_WILL_INIT (not REGBUF_FORCE_IMAGE), and btree_xlog_split_prefix()/the prefix-dedup redo branch explicitly rebuild each coded half from the logged plain items of the pre-split page -- see the code comments "No full-page image is needed" (nbtxlog.c) and "avoiding a full-page image" (nbtxlog.h). A forced FPI is used only as an overflow fallback when the plain payload would exceed BLCKSZ (XLH_SPLIT_RFPI / XLH_DEDUP_RFPI). The README should state that redo reconstructs the coded layout from logged plain items (because _bt_restore_page's item-by-item replay cannot reproduce it), with FPI only as the overflow fallback -- not that FPI is the normal logging mechanism.


📄 src/backend/access/nbtree/README

This claim is inaccurate. A deduplication rewrite of a prefix page is NOT FPI-logged by default. In nbtdedup.c (_bt_dedup_pass), block 0 is registered with REGBUF_STANDARD and the chosen prefix plus the deduped plain items are logged via XLogRegisterBufData so redo re-lays-out the coded page. A forced full-page image (REGBUF_FORCE_IMAGE / XLH_DEDUP_RFPI) is used only as a fallback when the packed plain payload would exceed BLCKSZ (dedup_rfpi, set when off > BLCKSZ). The normal path logs individual (plain) tuples, the opposite of what this sentence states. (high confidence)

Comment thread src/backend/access/nbtree/nbtpage.c Outdated
Comment on lines +1221 to +1222
* high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum
* record so redo does the same.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is inaccurate: nothing about the PREFIX-item drop is recorded in the vacuum record. xl_btree_vacuum only carries ndeleted/nupdated and the offset/tuple payloads (unchanged by this patch), and btree_xlog_vacuum() re-derives the drop by independently re-evaluating P_HASPREFIX(opaque) && PageGetMaxOffsetNumber(page) == BTPageGetPrefixOff(opaque) after replaying the deletes. Redo does the same thing, but not because it was recorded. Reword to reflect that redo recomputes the same condition from the post-delete page state. (high confidence)

Suggested change
* high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum
* record so redo does the same.
* high key (_bt_mark_page_halfdead). Redo (btree_xlog_vacuum) re-derives
* this from the post-delete page state, so nothing extra is logged.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +279 to +280
IndexTuple rightitems[MaxIndexTuplesPerPage];
int nright = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stack buffer overflow risk during redo. rightitems[] and dummyright[] are fixed-size stack arrays sized to MaxIndexTuplesPerPage, but a prefix-compressed leaf can physically hold MORE line pointers than that limit: MaxIndexTuplesPerPage is derived assuming each tuple is at least MAXALIGN(sizeof(IndexTupleData) + 1) bytes, whereas a coded tuple can be smaller (1-byte varlena header + 1-byte shared code). The primary correctly sizes these arrays dynamically as palloc_array(IndexTuple, maxoff + 2) (see _bt_split in nbtinsert.c). The while (datalen > 0) loop increments nright with no bound check and trusts the WAL record's self-describing IndexTupleSize, so a legitimately overfull (or corrupt) record overruns the stack. Allocate these with palloc sized from the record, matching the primary.

Comment thread src/include/access/nbtxlog.h Outdated
Comment on lines +181 to +182
#define XLH_SPLIT_LPREFIX_ORIG 0x02 /* left half prefix is the orig page's */
#define XLH_SPLIT_RPREFIX_ORIG 0x04 /* right half prefix is the orig page's */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead scaffolding and misleading documentation. XLH_SPLIT_LPREFIX_ORIG and XLH_SPLIT_RPREFIX_ORIG are defined but never referenced anywhere (neither the primary in nbtinsert.c nor redo in nbtxlog.c reads them). The comment claims prefix bytes are "not logged ... when the *_PREFIX_ORIG flag is set", but the actual code always logs the prefix bytes (XLogRegisterBufData of leftprefix/rightprefix) and redo always reads them from the logged block data. Either the described optimization was never implemented (so remove the two flags and fix the comment), or the implementation is incomplete. As written, the comment describes behavior that does not exist.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +402 to +403
IndexTuple dummyright[MaxIndexTuplesPerPage];
int dummynr;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unbounded fixed-size stack array as the right/left half of the split path: dummyright is MaxIndexTuplesPerPage, but _bt_split_leaf_collect writes *nright entries for a prefix page that can exceed MaxIndexTuplesPerPage. Even though this redo path only needs the left items, _bt_split_leaf_collect unconditionally fills the right array too, so dummyright can overflow. Size it to match the primary (maxoff + 2) via palloc.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +700 to +701
IndexTuple items[MaxIndexTuplesPerPage];
int nitems = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unbounded stack array in the dedup redo path: items[MaxIndexTuplesPerPage] is filled by a while (datalen > 0) loop over trusted WAL data, with nitems++ and no bound check. A prefix-compressed dedup result page can hold more coded items than MaxIndexTuplesPerPage. Allocate from the record size instead of a fixed stack array.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +313 to +318
if (spagenumber != P_NONE)
{
right_hikey = (IndexTuple) datapos;
datapos += MAXALIGN(IndexTupleSize(right_hikey));
datalen -= MAXALIGN(IndexTupleSize(right_hikey));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unaligned IndexTuple access during redo. The primary logs the right half's prefix as rightprefixlen raw bytes with no MAXALIGN padding (see nbtinsert.c: roff += rightprefixlen before copying the high key/items). rightprefixlen is the number of shared first-column bytes from _bt_prefix_choose() and is an arbitrary value, not a multiple of MAXALIGN. Here datapos += xlrec->rightprefixlen then casting to IndexTuple and calling IndexTupleSize(right_hikey) (and likewise for each rightitems[] entry below) dereferences t_info at an unaligned address. On strict-alignment platforms (ARM, s390x, etc.) this is undefined behavior / a crash. Either MAXALIGN-pad the prefix on the primary before the high key/items, or memcpy the tuples into aligned storage in redo before reading them.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +705 to +710
if (xlrec->flags & XLH_DEDUP_NOT_RIGHTMOST)
{
hikey = (IndexTuple) datapos;
datapos += MAXALIGN(IndexTupleSize(hikey));
datalen -= MAXALIGN(IndexTupleSize(hikey));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unaligned IndexTuple access in the prefix-dedup redo path. The primary packs prefixlen raw (unpadded) prefix bytes before the high key and deduped items (see nbtdedup.c: off = dedup_prefixlen with no MAXALIGN). After datapos += xlrec->prefixlen, casting to IndexTuple and calling IndexTupleSize(hikey) / IndexTupleSize(it) reads t_info at a non-MAXALIGN'd address whenever prefixlen % MAXALIGN != 0, which crashes on strict-alignment architectures. Pad the prefix to MAXALIGN on the primary, or copy tuples into aligned buffers in redo.

Comment thread src/backend/access/nbtree/nbtsearch.c Outdated
Comment on lines +698 to +699
IndexTuple itup;
BTItemBuf itupbuf;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTItemBuf is PGAlignedBlock, i.e. a full BLCKSZ-sized buffer (8KB by default, up to 32KB). Declaring it as a local in _bt_compare() -- the single hottest routine in nbtree, called O(log n) times per descent and in every page binary search -- inflates the stack frame by BLCKSZ on every call, including internal-page and non-prefix-index comparisons where _bt_getitem() never decodes (BTP_PREFIX is a leaf-only flag). Since &itupbuf is address-taken, the compiler cannot elide it. Consider having the caller supply the scratch buffer (so it is reserved once per descent rather than per comparison), or otherwise avoid paying this cost on the non-prefix fast path. (performance, moderate confidence)

Comment thread src/backend/access/nbtree/nbtree.c Outdated
Comment on lines +438 to +441
Size half = BTTupleStorageSize(scan->indexRelation);

so->currTuples = (char *) palloc(half * 2);
so->markTuples = so->currTuples + half;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This grows the per-scan index-only tuple workspace from BLCKSZ * 2 (16KB) to BTTupleStorageSize * 2, which is roughly 9x larger (~144KB at default BLCKSZ: 8192 + (8192/16)*128 = 73728 per half). This larger allocation is now paid by every index-only scan on every btree index at btrescan time, even indexes that have never used prefix compression. The nbtree.h comment justifies the worst-case sizing (a surviving BTP_PREFIX page can exist on any index), but the allocation is unconditional rather than gated on the index actually being capable of carrying prefix pages. Worth confirming this per-scan memory regression is acceptable, or sizing it based on whether the index can contain BTP_PREFIX pages. (performance, moderate confidence)

Comment thread src/include/access/nbtxlog.h Outdated

/* xl_btree_dedup flag bits */
#define XLH_DEDUP_NOT_RIGHTMOST 0x01 /* prefix payload includes a high key */
#define XLH_DEDUP_RFPI 0x02 /* prefix page carried as a forced image */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XLH_DEDUP_RFPI is write-only dead scaffolding. The primary sets it (nbtdedup.c:387) but no code ever reads it: the dedup redo path relies on XLogReadBufferForRedo() returning BLK_RESTORED for the forced full-page image and short-circuiting before it would inspect this flag. Either drop the flag (and stop setting it), or if it is kept purely as documentation, say so explicitly. As written it is a knob with no caller, which the community rejects (YAGNI). Note this is a separate flag from the already-flagged XLH_SPLIT_LPREFIX_ORIG/RPREFIX_ORIG.

Comment thread src/backend/access/nbtree/nbtdedup.c Outdated
Comment on lines +72 to +76
bool pageprefix = P_HASPREFIX(opaque);
BTItemBuf curbuf; /* decode scratch for the loop's current item */
BTItemBuf basebuf; /* decode scratch kept for the pending base */
XLogRecPtr recptr;
IndexTuple *dedup_plain = NULL; /* deduped plain items, for prefix WAL */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire prefix-handling block in _bt_dedup_pass is unreachable. The only caller, _bt_delete_or_dedup_one_page() in nbtinsert.c, now guards the call with !P_HASPREFIX(...), so _bt_dedup_pass() is never entered with a BTP_PREFIX page and pageprefix is always false here. That makes all the if (pageprefix) decode branches, the _bt_prefix_compress_page() re-compression block, the dedup_plain/dedup_hikey/dedup_walbuf packing, and the dedup_islogprefix ? 0 : state->nintervals WAL branch dead code (together with the matching XLOG_BTREE_DEDUP prefix-redo path in nbtxlog.c). This is ~150 lines of complex, untested WAL-logging scaffolding for a path that is deliberately not wired up (the nbtinsert.c comment states "Deduplication is not yet applied to prefix-compressed pages"). Drop it until dedup on prefix pages is actually enabled, per YAGNI. (high confidence)

Comment thread src/backend/access/nbtree/nbtsort.c Outdated
Comment on lines +1049 to +1051
if (state->btps_prefix == NULL)
state->btps_prefix = _bt_prefix_candidate(itup,
&state->btps_prefixlen);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

state->btps_prefix is set to a pointer into itup (via _bt_prefix_candidate, which returns VARDATA_ANY(val) of the tuple's first column). But itup here comes from tuplesort_getindextuple(), whose returned tuple is only valid until the next fetch on that sortstate. On the next _bt_buildadd() the previous itup's memory has been reused by the tuplesort, so btps_prefix now dangles and the later _bt_prefix_shared(state->btps_prefix, ...) reads stale/garbage bytes, yielding a meaningless shared estimate (and reading memory this code does not own). It does not corrupt the page (the hard fit check and the recompute in _bt_prefix_compress_page protect correctness), but the fillfactor estimate is effectively random. Copy the candidate prefix bytes into stable storage in BTPageState instead of keeping a pointer into the transient tuple. (medium confidence)

Comment thread src/backend/access/nbtree/README Outdated
Comment on lines +1150 to +1154
Deduplication and bottom-up index deletion are not currently applied to
BTP_PREFIX pages (such a page splits instead): their in-place rewrite of a page
cannot always be re-compressed within one page, and prefix compression already
reclaims much of the space they target. Composing them with prefix
compression is future work.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The claim that bottom-up index deletion is not applied to BTP_PREFIX pages contradicts the implementation. In nbtinsert.c (_bt_delete_or_dedup_one_page), only the deduplication pass is gated by !P_HASPREFIX(...); the bottom-up deletion pass (_bt_bottomupdel_pass) runs unconditionally, and its own comment states "Bottom-up deletion above is safe on prefix pages -- it only removes items, never rewrites survivors." So bottom-up deletion IS applied to prefix pages; only deduplication is skipped. Reword to scope the "not currently applied / splits instead" statement to deduplication alone.

Comment thread src/backend/access/nbtree/README Outdated
Comment on lines +1088 to +1090
With the prefix_compression storage parameter (default off) a leaf page may
front-code the first key column of each of its data tuples against a per-page
prefix, which shrinks indexes on keys that share long leading bytes (URLs,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opening description omits that prefix compression applies only when the first key column is of a varlena (variable-length) type. _bt_prefix_enabled() gates on TupleDescAttr(...,0)->attlen == -1, and nbtree.h documents the same requirement ("whose first key column is of a varlena type"). As written, the README implies any first key column is front-coded, which is misleading for an index whose leading column is fixed-length (it is silently never compressed). (moderate confidence)

Comment on lines +608 to +610
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_seqscan = on;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_indexonlyscan is set off here but never reset, so the session ends with a non-default GUC. Every other GUC block in this file restores what it changes (see the reset enable_seqscan/indexscan/bitmapscan at the earlier tests), and this block itself resets enable_seqscan/enable_bitmapscan at lines 615-616 and resets enable_indexscan to its default via SET ... = on at line 614. Leaving enable_indexonlyscan permanently off is inconsistent and sloppy. Add RESET enable_indexonlyscan; alongside the other resets (and prefer RESET enable_indexscan; over SET enable_indexscan = on; for consistency with the file's idiom).

Suggested change
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_seqscan = on;
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_seqscan = on;

Comment thread src/backend/access/nbtree/nbtinsert.c Outdated
Comment on lines +2034 to +2044
* Leaf halves are built as whole pages now, so they can be prefix-
* compressed from the start (Strategy 1: a page is faulted into the
* BTP_PREFIX format only when it is written whole). Both halves of a
* prefix split are restored from full-page images in redo (their coded
* physical layout would not be reproduced by item replay): the left half
* (block 0) is FPI-logged whenever the original page was prefix-compressed
* and redo detects the image via BLK_RESTORED, so it needs no record flag;
* the right half (block 1) records XLH_SPLIT_RPREFIX so redo restores it
* from its forced image rather than rebuilding it from logged tuples.
* dataitem copies and the lefthighkey copy are freed via the short-lived
* insert context.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment describes an abandoned FPI-based design and flatly contradicts the code and the redo path. In reality neither half is restored from a full-page image in the normal case: block 0 is registered with plain REGBUF_STANDARD (no forced image) and btree_xlog_split_prefix() rebuilds the left half from the pre-split page on BLK_NEEDS_REDO; the right half is carried as its logged plain items (rightwalbuf) and laid out coded by _bt_split_leaf_layout(), only falling back to an FPI under XLH_SPLIT_RFPI. The flag named here, XLH_SPLIT_RPREFIX, does not exist (the real flags are XLH_SPLIT_LEAFPREFIX and XLH_SPLIT_RFPI; XLH_SPLIT_RPREFIX_ORIG is defined but unused). This stale comment will badly mislead anyone auditing WAL/redo correctness and should be rewritten to match the tuple-replay design actually implemented (and correctly described near the XLogInsert call below).

@github-actions

Copy link
Copy Markdown

📊 OCR posted 16/17 inline comment(s).

1 could not be posted

src/backend/access/nbtree/nbtinsert.c L1959: Validation Failed: {"resource":"PullRequestReviewComment","code":"custom","field":"pull_request_review_thread.line","message":"could not be resolved"} - https://docs.github.com/rest/pulls/comments#create-a-review-comment-for-a-pull-request

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 23 issue(s).

  • 22 inline, 1 in summary

📄 src/backend/access/nbtree/README

This WAL paragraph contradicts the shipped implementation; it describes a design that was not the one committed. The code does NOT use a full-page image as the primary redo mechanism for prefix splits/dedup:

  • Left half (block 0) is registered with plain REGBUF_STANDARD (nbtinsert.c _bt_split, XLogRegisterBuffer(0, buf, REGBUF_STANDARD)) -- there is no forced FPI. Redo rebuilds the left half from the original page's items via _bt_split_leaf_collect + _bt_split_leaf_layout (btree_xlog_split, nbtxlog.c).
  • Right half (block 1) is registered REGBUF_WILL_INIT and rebuilt from logged PLAIN items; REGBUF_FORCE_IMAGE is used only in the XLH_SPLIT_RFPI overflow fallback, not 'whenever it ends up prefix-compressed'.
  • A dedup rewrite of a prefix page is likewise carried as the prefix + plain items (xl_btree_dedup prefixlen payload), with an FPI only via the XLH_DEDUP_RFPI fallback.

The nbtxlog.h header comment states exactly the opposite of this paragraph ('avoiding a full-page image', 'logs no full-page image: redo rebuilds each half the way the primary did'). Rewrite this paragraph to describe item-replay as the primary path with FPI only as a bounded (>BLCKSZ payload) fallback, consistent with XLH_SPLIT_RFPI / XLH_DEDUP_RFPI.

Comment thread src/backend/access/nbtree/nbtpage.c Outdated
Comment on lines +1221 to +1222
* high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum
* record so redo does the same.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is inaccurate. Nothing about the PREFIX drop is "recorded in the vacuum record": xl_btree_vacuum only carries ndeleted/nupdated plus the offset arrays (see nbtxlog.h), and no field is added here. Redo (btree_xlog_vacuum, nbtxlog.c) independently re-derives the same condition from the page state after replaying the deletions (P_HASPREFIX && PageGetMaxOffsetNumber == BTPageGetPrefixOff). Stating it is recorded is misleading and would steer a future maintainer wrong if the record format is changed. Reword to match the redo path, e.g. "Redo re-derives this from the replayed page state in btree_xlog_vacuum."

Suggested change
* high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum
* record so redo does the same.
* high key (nbtpage.c's _bt_mark_page_halfdead). btree_xlog_vacuum
* re-derives this from the replayed page state, so no field is recorded.

Comment thread src/backend/access/nbtree/nbtsearch.c Outdated
Comment on lines +698 to +699
IndexTuple itup;
BTItemBuf itupbuf;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTItemBuf is PGAlignedBlock, i.e. a full BLCKSZ (8KB default) stack buffer. _bt_compare is the hottest comparison routine in nbtree, invoked repeatedly inside the binary-search loops (_bt_binsrch, _bt_binsrch_insert) at every level of a descent. This adds ~8KB to its stack frame on every call, paid unconditionally even for the vast majority of indexes that are not prefix-compressed. Worth confirming the added per-call frame size is acceptable on the descent hot path (stack depth, not just CPU); the other new BTItemBuf locals in nbtreadpage.c are per-page rather than per-comparison, so this call site is the most sensitive one.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +307 to +318
if (xlrec->rightprefixlen > 0)
{
rightprefix = datapos;
datapos += xlrec->rightprefixlen;
datalen -= xlrec->rightprefixlen;
}
if (spagenumber != P_NONE)
{
right_hikey = (IndexTuple) datapos;
datapos += MAXALIGN(IndexTupleSize(right_hikey));
datalen -= MAXALIGN(IndexTupleSize(right_hikey));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unaligned memory access on strict-alignment platforms (ARM/SPARC/PPC, etc.).

The right-half payload is laid out as [rightprefix bytes][right_hikey][items...], and the prefix bytes are logged WITHOUT MAXALIGN padding (see nbtinsert.c: roff = rightprefixlen; ... roff += MAXALIGN(...)). XLogRecGetBlockData only MAXALIGNs the start of the block data. After datapos += xlrec->rightprefixlen (rightprefixlen can be 1..126, including odd values), datapos is no longer aligned, so (IndexTuple) datapos and IndexTupleSize(right_hikey) / the subsequent item loop dereference t_info (a uint16) at an unaligned address. _bt_prefix_encode/PageAddItem then read the whole tuple header from an unaligned source.

The pre-existing _bt_restore_page deliberately avoids this by using memcpy into an aligned IndexTupleData and a void *items[] array (see its comment about not assuming alignment). This new path regresses. Either MAXALIGN the prefix length in the logged payload on both the write and read sides, or memcpy the header out before calling IndexTupleSize. The left-half path (block 0) happens to be safe only because the prefix is logged last there.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +703 to +719
datapos += xlrec->prefixlen;
datalen -= xlrec->prefixlen;
if (xlrec->flags & XLH_DEDUP_NOT_RIGHTMOST)
{
hikey = (IndexTuple) datapos;
datapos += MAXALIGN(IndexTupleSize(hikey));
datalen -= MAXALIGN(IndexTupleSize(hikey));
}
while (datalen > 0)
{
IndexTuple it = (IndexTuple) datapos;
Size sz = MAXALIGN(IndexTupleSize(it));

items[nitems++] = it;
datapos += sz;
datalen -= sz;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unaligned-access hazard as the split right-half path. The dedup payload is [prefix bytes][hikey][items...] with the prefix logged unpadded (nbtdedup.c packs off += dedup_prefixlen with no MAXALIGN). After datapos += xlrec->prefixlen with an odd prefixlen, (IndexTuple) datapos on the hikey and every item in the loop is an unaligned IndexTuple dereference (IndexTupleSize reads t_info). This will SIGBUS/UB on strict-alignment architectures. Pad the prefix to MAXALIGN on the write side (and mirror it here), or memcpy each tuple header out before use, consistent with _bt_restore_page.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +400 to +409
rightitems[0] = NULL; /* not reused below */
{
IndexTuple dummyright[MaxIndexTuplesPerPage];
int dummynr;

_bt_split_leaf_collect(origpage, xlrec->firstrightoff,
xlrec->newitemoff, newitem, newitemonleft,
origpagepostingoff, nposting,
leftitems, &nleft, dummyright, &dummynr);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rightitems[0] = NULL; followed by a nested block declaring a throwaway dummyright[MaxIndexTuplesPerPage] only to discard the collected right half is an awkward workaround. It burns an extra MaxIndexTuplesPerPage-entry stack array (~ several KB) purely to satisfy the _bt_split_leaf_collect signature. Consider letting _bt_split_leaf_collect accept NULL right arrays (skip the right side) so redo can collect only the half it needs, rather than reusing rightitems and a dummy array.

Comment thread src/backend/access/nbtree/nbtxlog.c Outdated
Comment on lines +319 to +327
while (datalen > 0)
{
IndexTuple it = (IndexTuple) datapos;
Size sz = MAXALIGN(IndexTupleSize(it));

rightitems[nright++] = it;
datapos += sz;
datalen -= sz;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The while (datalen > 0) loop appends to rightitems[] (sized MaxIndexTuplesPerPage) with no bound check on nright. A corrupt record with an oversized datalen would overrun the on-stack array. The pre-existing _bt_restore_page has the same unchecked pattern, so this matches established nbtree-redo convention, but given this runs on standbys where a corrupt record is possible, consider an Assert(nright < MaxIndexTuplesPerPage) (and likewise in the dedup path) to fail loudly rather than smash the stack.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 14 issue(s).

  • 14 inline, 0 in summary

FROM generate_series(1, 20) g;
CREATE INDEX test_prefix_idx ON test_prefix (u) WITH (prefix_compression = on);
CREATE INDEX test_plain_idx ON test_prefix (u);
SELECT btpo_flags & 512 <> 0 AS coded FROM bt_page_stats('test_prefix_idx', 1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The magic literal 512 here is the BTP_PREFIX flag bit (1 << 9). An unadorned numeric flag mask is opaque to readers and brittle if the flag ever moves. Since pageinspect doesn't expose the symbolic name in SQL, at minimum add a short comment noting this tests the BTP_PREFIX bit, so the intent is clear in the test output/source.

Comment on lines +76 to +77
FULL JOIN bt_page_items('test_plain_idx', 1) p ON p.itemoffset = c.itemoffset - 1
WHERE c.itemoffset > 1 AND (c.data IS DISTINCT FROM p.data OR c.htid IS DISTINCT FROM p.htid);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FULL JOIN is partly defeated by WHERE c.itemoffset > 1: rows where c is unmatched-NULL (i.e., a plain item with no corresponding coded item) are dropped because NULL > 1 is not true, so a prefix page with fewer data items than the plain page would not be flagged as a mismatch. It happens to be harmless here (both indexes hold the same 20 unique rows with no dead tuples, so counts are equal), but the FULL JOIN is misleading about what it verifies. Consider also asserting equal row counts, or drop the FULL to a plain JOIN to match the actual intent.

Comment on lines 24 to 25
static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state,
TM_IndexDeleteOp *delstate);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This blank line is cosmetic churn on an untouched region unrelated to prefix compression. PostgreSQL rejects whitespace-only changes to lines a patch does not otherwise need to touch; please drop it to keep the diff minimal.

Suggested change
static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state,
TM_IndexDeleteOp *delstate);
static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state,
TM_IndexDeleteOp *delstate);

Comment on lines 296 to 298
XLogBeginInsert();

XLogRegisterBuffer(0, buf, REGBUF_STANDARD);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cosmetic blank line inserted into the XLOG block; it is unrelated to prefix compression and is pure whitespace churn on an otherwise untouched region. Remove to keep the diff minimal.

Suggested change
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);

Comment thread src/backend/access/nbtree/nbtsearch.c Outdated
Comment on lines +722 to +724
static BTItemBuf prefixbuf;

itup = _bt_getitem(page, offnum, &prefixbuf);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a static BTItemBuf (an 8KB PGAlignedBlock) here is a surprising deviation from the established pattern: every other _bt_getitem caller (nbtdedup.c, nbtreadpage.c, nbtxlog.c) uses a stack-local BTItemBuf. The static costs 8KB of permanent per-process BSS even for indexes that never use prefix compression, and it silently retains a decoded tuple across calls. The only reason to avoid a stack-local here is the 8KB frame on this hot, recursive comparison routine -- but that rationale is undocumented, so a future reader will read this as a footgun (the returned pointer is clobbered by the next _bt_compare). Correctness is fine today because itup never escapes the function, but please either document why a stack-local is not used or reuse the standard pattern. At minimum, add a comment explaining the static lifetime.

Comment thread src/include/access/nbtree.h Outdated
/*
* prototypes for functions in nbtprefix.c
*/
extern const char *_bt_page_get_prefix(Page page, Size *len);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_bt_page_get_prefix is a dead prototype: it has no definition anywhere (not in nbtprefix.c) and no caller in the backend or contrib. Every other prototype in this block is defined and used; this one is dangling (leftover from an earlier design or an unimplemented helper). Remove it per YAGNI/minimal-diff discipline — an unused declaration will also eventually trip -Wmissing-prototypes-style checks and confuses readers about the module's API.

Comment on lines +726 to +727
SELECT pg_relation_size('btree_prefix_grp_on') <
pg_relation_size('btree_prefix_grp_off') AS groups_compress;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion is not portable across BLCKSZ. The _grp indexes hold only 151 small (int4, ~35-byte text) tuples, so the result of _on < _off depends entirely on the uncompressed index spilling into a second leaf page while the compressed one fits in one. That is borderline even at the default 8kB BLCKSZ, and at 16kB/32kB BLCKSZ both indexes fit on a single leaf page, making pg_relation_size equal and groups_compress evaluate to f -- breaking the recorded expected output t. Other size/page-dependent tests in this very file explicitly document their BLCKSZ assumption (see lines 311, 422, 434). Either make this comparison robust across block sizes (e.g. insert enough rows that compression changes the leaf-page count by a wide margin regardless of BLCKSZ) or gate/document the BLCKSZ assumption as the surrounding tests do. The large URL-keyed build_compresses/insert_compresses checks (33% byte threshold over hundreds of pages) are far safer; this 151-row strict-inequality is the fragile one.

Comment on lines +609 to +611
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_seqscan = on;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_indexonlyscan is set off here but never reset. Its session default is true (costsize.c), and every other GUC this new section touches is RESET at the end of its block. Although btree_index.sql runs in its own session so this does not corrupt other regress files, leaving the session in a non-default state at end-of-file is inconsistent with the test's own cleanup discipline and is a maintenance footgun if blocks are reordered or added later. Add a matching RESET enable_indexonlyscan; once the seqscan-oracle comparison is done.

Comment on lines +660 to +661
SELECT (SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix_bu) =
md5(string_agg(x, '|' ORDER BY x)) AS bottomup_ok

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block's only assertion (bottomup_ok digest + a count) is a correctness-after-churn check: it passes whenever the index returns correct rows, regardless of whether the bottom-up deletion pass (_bt_bottomupdel_pass) actually fires on a coded page. _bt_bottomupdel_pass only runs when a leaf is about to split AND the UPDATE was flagged logically-unchanged, and removal is snapshot/timing-dependent (acknowledged in the comment). So this test does not reliably exercise the coded-page deletion code it claims to cover and would still pass if that path regressed or was skipped. Consider verifying the path is reached (e.g. via pageinspect on BTP_PREFIX leaves, or an index-size assertion that the pass actually reclaimed space), or soften the comment to state it is a correctness-under-churn check rather than a guaranteed exercise of bottom-up deletion.

Comment on lines +551 to +556
-- Confirm the index really is used (and is prefix-compressed), so the queries
-- below exercise the coded page format rather than silently falling back.
SET enable_seqscan = off;
SET enable_bitmapscan = off;
EXPLAIN (costs off)
SELECT u FROM btree_prefix WHERE u = 'https://example.com/path/to/resource/00012345';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment claims this EXPLAIN confirms the index "is prefix-compressed", but EXPLAIN (costs off) output (Index Only Scan using btree_prefix_on) reveals nothing about the leaf page format (BTP_PREFIX). If prefix_compression=on silently fell back to plain pages, this EXPLAIN would be byte-for-byte identical and the test would still pass. In fact, every correctness check in this section (equality/range/backward/IOS/dedup digests, counts) exercises index correctness and would pass equally on an uncompressed index -- i.e. they survive the feature being reverted. The only assertions that can actually detect "compression did not happen" are the two BLCKSZ-dependent pg_relation_size ratio checks (_sz, _grp), and _grp is already flagged as borderline-nonportable. Either drop the "(and is prefix-compressed)" overclaim from the comment, or add a deterministic assertion that the page is really coded (the pageinspect/amcheck additions in this same PR expose the flag).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 17 issue(s).

  • 17 inline, 0 in summary

Comment on lines +76 to +77
FULL JOIN bt_page_items('test_plain_idx', 1) p ON p.itemoffset = c.itemoffset - 1
WHERE c.itemoffset > 1 AND (c.data IS DISTINCT FROM p.data OR c.htid IS DISTINCT FROM p.htid);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low confidence / low severity, test robustness: this is written as a FULL JOIN to compare both indexes, but the predicate c.itemoffset > 1 silently discards every plain-index row that has no matching coded row (for an unmatched plain row c.itemoffset is NULL, so the WHERE drops it). As a result the query only detects coded items that differ from or are missing in the plain index; it will NOT detect the case where the prefix index is missing data items that the plain index has. Consider also asserting the data-item counts match, e.g. SELECT count(*) FROM bt_page_items('test_prefix_idx',1) WHERE itemoffset>1 equals SELECT count(*) FROM bt_page_items('test_plain_idx',1), so a dropped/extra item on the prefix page cannot pass unnoticed.

values[j++] = Int16GetDatum(offset);
values[j++] = ItemPointerGetDatum(&itup->t_tid);
values[j++] = Int16GetDatum(IndexTupleSize(itup));
values[j++] = Int16GetDatum(ItemIdGetLength(id));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unconditionally changes itemlen from IndexTupleSize(itup) (the unaligned size stored in t_info) to ItemIdGetLength(id) (the MAXALIGN'd line-pointer length). For prefix pages that correctly reports the coded size, but it also changes the reported itemlen for every item on plain (non-BTP_PREFIX) pages: index_form_tuple() stores the unaligned size in t_info, so whenever a tuple's real size is not a multiple of MAXALIGN (e.g. a text key of odd byte length) the two values differ (e.g. 15 vs 16). This is a backward-incompatible change to long-standing bt_page_items output for ordinary indexes, and is unrelated to prefix compression. The existing regression value (itemlen | 16) does not catch it because that int tuple is already MAXALIGN'd. Restrict the on-page length to the decoded case and keep IndexTupleSize(itup) otherwise (confidence: high).

Comment on lines +1710 to +1711
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz);
origprefix_array = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory leak on the prefix-compressed split path. _bt_prefix_decode_items() allocates ditems plus one CopyIndexTuple() per data item (potentially up to ~2 pages' worth of tuples), but _bt_split() never frees either the array or the decoded copies. Compare _bt_dedup_pass/_bt_bottomupdel_pass, which both free their ditems arrays and members. A statement doing many splits on a coded index (e.g. a large COPY or INSERT..SELECT) accumulates this leak until the surrounding memory context is reset. Free ditems[i] (skipping the nposting slot, which is freed by the caller) and ditems before returning from _bt_split.

Comment thread src/backend/access/nbtree/nbtreadpage.c Outdated
Comment on lines +665 to +666
ItemId iid = NULL;
BTItemBuf firstbuf,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iid is now dead: it is initialized to NULL and immediately discarded with (void) iid;, with no remaining use after the switch to _bt_getitem(). Remove the declaration and the (void) iid; line rather than voiding an unused variable (pgsql-hackers routinely rejects this kind of leftover).

Comment thread src/backend/access/nbtree/nbtreadpage.c Outdated
Comment on lines +687 to +688
(void) iid;
firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead statement: (void) iid; only exists to suppress an unused-variable warning after iid lost its users. Drop it together with the iid declaration above.

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +615 to +617
SET enable_indexscan = on;
RESET enable_seqscan;
RESET enable_bitmapscan;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

State hygiene: these three lines mix SET enable_indexscan = on (restore to default via SET) with RESET for the siblings, and leave enable_indexonlyscan at the non-default off set on line 610. Use RESET enable_indexscan; and add RESET enable_indexonlyscan; so no enable_* GUC leaks past this block into the rest of btree_index.sql.

Suggested change
SET enable_indexscan = on;
RESET enable_seqscan;
RESET enable_bitmapscan;
RESET enable_indexscan;
RESET enable_indexonlyscan;
RESET enable_seqscan;
RESET enable_bitmapscan;

Comment on lines +701 to +702
SELECT pg_relation_size('btree_prefix_sz_on') * 3 <
pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These relation-size comparisons are a portability/stability risk. The build/insert worthwhile threshold in the implementation (_bt_prefix_worthwhile: codedsz16 <= plainsz15) only guarantees a ~6% saving, while these assertions demand ~33% (on3 < off2). For this specific highly-compressible URL data the actual margin is large, so it will hold today, but the ratio depends on page geometry and is not normalized to block_size; under a non-default BLCKSZ (directory overhead per page and tuples-per-leaf change) the boolean could flip. The groups_compress check below is a bare strict < with no margin at all, making it the most fragile. Consider documenting the dependence on default BLCKSZ, or expressing the assertion so it is robust to block size.

Suggested change
SELECT pg_relation_size('btree_prefix_sz_on') * 3 <
pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses;
SELECT pg_relation_size('btree_prefix_sz_on') * 3 <
pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses;

Comment on lines +551 to +552
-- Confirm the index really is used (and is prefix-compressed), so the queries
-- below exercise the coded page format rather than silently falling back.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment claims the EXPLAIN confirms the index "is prefix-compressed". EXPLAIN (costs off) only shows the chosen plan node and index name; it cannot show that any leaf is in BTP_PREFIX coded form. Nothing here would catch a silent fallback where the index is used but no leaf actually compressed. Tone the comment down to what it verifies (the index is used, not bypassed), or add a pageinspect/amcheck assertion on the coded leaf format if coded-ness must be proven.

Suggested change
-- Confirm the index really is used (and is prefix-compressed), so the queries
-- below exercise the coded page format rather than silently falling back.
-- Confirm the index is used (not bypassed) so the queries below exercise the
-- index access path rather than silently falling back to a seq scan.

Comment on lines +726 to +727
SELECT pg_relation_size('btree_prefix_grp_on') <
pg_relation_size('btree_prefix_grp_off') AS groups_compress;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This groups_compress assertion is fragile across non-default BLCKSZ. The dataset is tiny (151 small tuples: one (1001,'a') plus 150 short URLs). pg_relation_size is quantized to whole pages, so the strict on < off comparison requires the plain index to occupy at least one more leaf page than the compressed one. With a larger BLCKSZ (16K/32K) both indexes may fit in a single leaf page, giving equal pg_relation_size and turning this into false -> test failure. Elsewhere this suite explicitly guards page-count assumptions (the dedup test notes "assumes BLCKSZ is 8192 or less"); this check has no such guard. Either add many more rows so the compressed index saves a full page under any supported BLCKSZ, or make the assertion BLCKSZ-robust (e.g. compare via pageinspect/amcheck that a leaf actually formed multiple groups) rather than relying on whole-page size granularity.

Comment on lines +631 to +633
CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on);
SET enable_seqscan = off;
SELECT length(u) FROM btree_prefix WHERE length(u) = 300;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block does not verify what its comment claims. The comment asserts "A non-varlena key column is compressed too", but the only check is a scan returning one row for length(u) = 300, with no EXPLAIN to confirm the index is even chosen and nothing that confirms any leaf is in BTP_PREFIX form. The int4 expression values are overwhelmingly the constant 45 (all URL rows have length 45), so this index path is dominated by deduplication, not prefix compression. This query returns the same single row whether or not prefix compression is enabled -- i.e. it still passes with the feature reverted -- so it does not actually exercise or guard the non-varlena prefix-compression code. Either drop the overclaiming comment or strengthen the test to assert compression actually occurred for a non-varlena key.

@gburd
gburd force-pushed the master branch 5 times, most recently from 48fd38b to 4b8e031 Compare November 12, 2026 18:15

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 16 issue(s).

  • 16 inline, 0 in summary

values[j++] = Int16GetDatum(offset);
values[j++] = ItemPointerGetDatum(&itup->t_tid);
values[j++] = Int16GetDatum(IndexTupleSize(itup));
values[j++] = Int16GetDatum(ItemIdGetLength(id));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes itemlen for all pages, not just prefix leaves. The old value, IndexTupleSize(itup), is the unaligned tuple size (t_info & INDEX_SIZE_MASK); ItemIdGetLength(id) is the MAXALIGN'd length stored in the line pointer. For any index whose stored tuples are not already a multiple of MAXALIGN (very common for text/varlena keys, e.g. the data | 01 00 ... int8 case happens to be 16 only by luck), itemlen now reports a larger, MAXALIGN'd value for existing non-prefix indexes. That is an unjustified, user-visible change to an existing output column of bt_page_items.

The decode-only requirement (coded size instead of the decoded tuple's IndexTupleSize) is only needed on the BTP_PREFIX data-item path. Keep the historical behavior for everything else, e.g. report ItemIdGetLength(id) only when the item was decoded, and IndexTupleSize(itup) otherwise. Confidence: high.

Comment on lines +1092 to +1095
* - soft: finish the leaf once it is fillfactor-full as it stands; -
* hard: finish it if admitting itup could leave it unable to hold its
* tuples and high key, or (coded) beyond the split-fit budget (see
* _bt_prefix_split_needed), or out of line pointers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inline bullet list is garbled: "soft" and "hard" items are run together across the line wrap (... as it stands; - then * hard: ...), so the two cases read as one sentence. Break them onto separate lines so each bullet stands on its own, e.g.

    • soft: finish the leaf once it is fillfactor-full as it stands;
    • hard: finish it if admitting itup could leave it unable to hold its
  • tuples and high key, or (coded) beyond the split-fit budget (see
  • _bt_prefix_split_needed), or out of line pointers.

Comment on lines +1341 to +1343
/* lay out the final leaf of a prefix_compression index */
if (state != NULL && state->btps_items != NULL)
_bt_flush_coded(wstate, state, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pre-existing comment "Each iteration of this loop completes one more level of the tree." describes the for loop below, but the new prefix-flush block was inserted between the comment and the loop it documents, so the comment now reads as describing the flush. Move the flush block above this comment (or move the comment down to sit directly on the for) to keep the comment attached to the code it explains.

Comment on lines +1711 to +1712
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz);
origprefix_array = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory leak on the heavily-compressed split path: _bt_prefix_decode_items() allocates the ditems array and, for every data item, a CopyIndexTuple() of its decoded form. In the origprefix_array branch none of this is ever freed before _bt_split returns (the end of the function only frees lefthighkey). Each leaf split on an oversized coded page leaks ~2 pages of decoded tuples plus the array. _bt_doinsert/_bt_split run in the caller's context, which for VACUUM/COPY/CREATE INDEX inserts can be long-lived, so this accumulates. Free the decoded items and the array before returning (taking care that ditems[origpagepostingoff] may have been overwritten with nposting, which is freed separately on the posting-split path).

(high confidence)

Comment thread src/backend/access/nbtree/nbtreadpage.c Outdated
Comment on lines +665 to +666
ItemId iid = NULL;
BTItemBuf firstbuf,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead code / unnecessary churn. After switching to _bt_getitem(), the iid local is no longer used anywhere in _bt_set_startikey, and the (void) iid; cast only exists to silence the unused-variable warning that this leftover declaration itself causes. Drop both ItemId iid = NULL; and (void) iid; for a clean minimal diff.

(high confidence)

Suggested change
ItemId iid = NULL;
BTItemBuf firstbuf,
BTItemBuf firstbuf,

Comment on lines +726 to +727
SELECT pg_relation_size('btree_prefix_grp_on') <
pg_relation_size('btree_prefix_grp_off') AS groups_compress;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These size-ratio assertions (on < off*2/3) and groups_compress (on < off) encode fixed compression-effectiveness expectations. The per-tuple coding ratio is BLCKSZ-independent, and the URL keys share ~40 bytes so the margin looks comfortable for build_compresses/insert_compresses. The btree_prefix_grp case is the thin one: a single page with only 151 tuples where the on<off comparison has no explicit margin and the directory item plus per-group overhead eats into the gain -- if fillfactor/alignment shifts this on a non-default BLCKSZ the boolean could flip. Consider asserting a margin here too (e.g. on * 10 < off * 9) rather than a bare strict inequality. (low confidence)

Comment on lines +677 to +679
INSERT INTO btree_prefix_dd
SELECT 'https://example.com/path/to/resource/' || lpad((g % 200)::text, 8, '0')
FROM generate_series(1, 40000) g;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btree_index runs in a large parallel group (select_into ... btree_index hash_index update delete namespace prepared_xacts). This block adds ~75k inserted rows (20k+10k+5k+40k + bottom-up 4k with 3 full-table UPDATEs + size/group builds), a VACUUM, and several full index builds -- a noticeable bump to that group's wall time and temp space under the parallel schedule. Confirm each generate_series size is the minimum needed to force the targeted page path (split / dedup / bottom-up). In particular the 40000-row dedup set and the 20000-row size sets look larger than required to fill a single leaf and trigger the pass. (low confidence)

DELETE FROM pfx_dup WHERE u < 'https://example.com/path/to/resource/00000150';
VACUUM pfx_dup;

-- Non-key updates: bottom-up deletion on coded pages of a unique index.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says "bottom-up deletion on coded pages of a unique index," but pfx_upd_u is a non-unique index built on u, which has only 500 distinct values across 20000 rows (g % 500) -- i.e. heavily duplicated, the opposite of unique. The only unique index here is the PRIMARY KEY on id, which is not prefix-compressed and is not what this scenario exercises. Fix the comment to describe the actual index (a non-unique, low-cardinality prefix-compressed index).

Suggested change
-- Non-key updates: bottom-up deletion on coded pages of a unique index.
-- Non-key updates: bottom-up deletion on coded pages of a low-cardinality index.

@@ -0,0 +1,123 @@

# Copyright (c) 2026, PostgreSQL Global Development Group

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spurious leading blank line before the copyright header. Every other recovery TAP test (e.g. 060_multixact_crashed_updater.pl) begins with the # Copyright ... line on line 1. Remove the leading blank line to match project convention and keep the diff clean.

Suggested change
# Copyright (c) 2026, PostgreSQL Global Development Group
# Copyright (c) 2026, PostgreSQL Global Development Group

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +615 to +617
SET enable_indexscan = on;
RESET enable_seqscan;
RESET enable_bitmapscan;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_indexonlyscan is turned off at line 610 but never restored (only enable_indexscan is re-enabled here, via SET ... = on rather than the file's usual RESET). Since enable_indexonlyscan defaults to true, the leak persists for the rest of the session and silently downgrades every later block that the author intends to run as an index-only scan to a plain index scan that fetches from the heap. In particular the bottomup_ok query (line 660), whose comment explicitly claims "an index-only scan of the key column", no longer exercises IOS decoding of coded leaves -- the feature path the test is meant to cover. The test still passes, which hides the gap. Reset the GUC (and prefer RESET for enable_indexscan to match the surrounding style).

Suggested change
SET enable_indexscan = on;
RESET enable_seqscan;
RESET enable_bitmapscan;
RESET enable_indexscan;
RESET enable_indexonlyscan;
RESET enable_seqscan;
RESET enable_bitmapscan;

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 18 issue(s).

  • 18 inline, 0 in summary

Comment on lines +1348 to +1352
if (P_HASPREFIX(topaque))
{
itup = _bt_getitem(state->target, offset, &state->itembuf);
tupsize = IndexTupleSize(itup);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use-after-overwrite on prefix-compressed unique indexes. On a BTP_PREFIX leaf, _bt_getitem(..., &state->itembuf) returns a tuple that lives in state->itembuf and is only valid "until the workspace is reused" (see the BTItemBuf contract in nbtree.h). But bt_entry_unique_check() saves a pointer into this tuple in lVis.tid (lVis->tid = BTreeTupleGetHeapTID(itup) / BTreeTupleGetPostingN(itup, i)), and that pointer is retained across loop iterations to be dereferenced later by bt_report_duplicate() and the ItemPointerIsValid(lVis.tid) checks. On the next iteration this line re-decodes into the same itembuf, clobbering the bytes lVis.tid points at, so a cross-row uniqueness violation reports garbage heap TIDs (or reads freed/overwritten memory). Prefix compression is only gated by the prefix_compression reloption and is permitted on unique indexes, so this path is reachable. The added SQL test only exercises a non-unique prefix index, so it does not catch this. Fix by copying the lVis TID into stable storage (e.g. a dedicated ItemPointerData in BtreeLastVisibleEntry) rather than aliasing the decode workspace, and add a unique prefix-index case to the regression test.

Comment on lines +2090 to +2094
if (P_ISLEAF(opaque))
firstitup = CopyIndexTuple(_bt_getitem(rightpage, *rightfirstoffset,
&state->nextbuf));
else
firstitup = (IndexTuple) PageGetItem(rightpage, rightitem);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment "this relies on right page memory remaining allocated" is now only accurate for the internal (non-leaf) branch. The leaf branch copies the decoded tuple with CopyIndexTuple(), precisely because the decode workspace (nextbuf) and the right page are both transient; the resulting scankey no longer depends on right-page memory. Consider updating the comment to explain why the leaf path must copy (decoded tuple lives in a reusable BTItemBuf) while the internal path can point at page memory.

Comment on lines +199 to +201
CREATE INDEX prefix_bug_idx ON prefix_bug (v) WITH (prefix_compression = on);
SELECT bt_index_check('prefix_bug_idx', heapallindexed => true);
SELECT bt_index_parent_check('prefix_bug_idx', heapallindexed => true, rootdescend => true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test-coverage gap: this exercises only a non-unique prefix-compressed index. The uniqueness-checking path in verify_nbtree.c (gated on state->indexinfo->ii_Unique) is never reached here, yet that path is where prefix decoding interacts most dangerously with amcheck's cross-row state (lVis retains a pointer into the per-item BTItemBuf decode workspace). Add a UNIQUE prefix-compressed index case -- including adjacent equal keys / posting lists on the same leaf -- so bt_entry_unique_check()/bt_report_duplicate() run against decoded items. A healthy-index check will not surface a wrong-TID report, so also consider a corruption-injection or at least a duplicate-key scenario that drives the duplicate-reporting branch.

Comment on lines +380 to +382
Assert(P_HASPREFIX(oopaque));
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey,
&dhikeysz);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ditems array and the per-item CopyIndexTuple copies allocated by _bt_prefix_decode_items are never freed in this split-redo branch (only litems is pfreed at line 510). WAL replay (PerformWalRecovery/ApplyWalRecord) does NOT reset a memory context per record, so these allocations accumulate in the startup process's long-lived context for the entire duration of recovery. Each coded split replays one array plus one palloc per decoded data item (up to ~MaxIndexTuplesPerPage), which over millions of split records is an unbounded, data-proportional leak during crash/standby recovery. The comment below claiming "decoded items ... go with the redo memory context" is misleading since there is no such per-record reset. Free ditems (and ideally its element tuples) before returning, as is done for litems.

Comment on lines +608 to +609
(void) _bt_prefix_decode_items(page, &ditems, &dummyhikey,
&dummyhikeysz);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same leak as the split path: ditems (array) and its per-item CopyIndexTuple copies from _bt_prefix_decode_items are never freed in the dedup-redo branch. Recovery does not reset a memory context per WAL record, so each replayed dedup of a coded page leaks one array plus one tuple copy per data item into the startup process's long-lived context. Free ditems and its elements before PageRestoreTempPage returns.

Comment on lines +701 to +702
SELECT pg_relation_size('btree_prefix_sz_on') * 3 <
pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate confidence] BLCKSZ assumption undocumented. The 3:2 size-ratio assertions (build_compresses/insert_compresses) depend on page packing, which varies with BLCKSZ, alignment, and 32/64-bit layout. The 40-byte shared prefix over 20000 keys gives a wide margin so this is likely robust, but the file's convention (lines 311, 422, 434) is to spell out BLCKSZ sensitivity explicitly. Add a one-line note that these size checks assume the default 8K BLCKSZ, consistent with the surrounding tests, so a buildfarm failure on a large-block animal is diagnosable.

Comment on lines +603 to +605
-- After the churn, an index-only scan of the live keys must equal the same
-- keys fetched from the heap by a sequential scan (an independent path): if
-- decoding the coded leaves were wrong, the two digests would differ.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate confidence] Comment overstates what is exercised. The VACUUM btree_prefix at line 599 runs before the 5000-row re-insert at lines 600-602, so the freshly inserted rows are not all-visible when the digest is captured. The plan for those rows is therefore a plain index scan with heap fetches, not a pure index-only scan. The digest equality still holds (correctness is fine), but the comment "an index-only scan of the live keys" overstates the IOS coverage. Either VACUUM again after the re-insert to actually set the VM, or soften the comment.

ItemId iid = PageGetItemId(page, minoff);

pstate.finaltup = (IndexTuple) PageGetItem(page, iid);
pstate.finaltup = _bt_getitem(page, minoff, &finalbuf);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks the pstate->finaltup == tuple pointer-identity check in _bt_advance_array_keys() on prefix-compressed pages. For a backward SK_SEARCHARRAY scan, finaltup is decoded here into finalbuf. When the descending loop below reaches offnum == minoff, it computes itup = _bt_getitem(page, offnum, &itupbuf) (a different BTItemBuf), and that itup flows into _bt_checkkeys -> _bt_advance_array_keys, where line ~2709 does if (!all_required_satisfied && pstate->finaltup == tuple). On a plain page both _bt_getitem() calls return the same raw PageGetItem pointer, so the identity holds; on a BTP_PREFIX page they decode into distinct buffers, so finaltup == tuple is never true even when they are the same logical tuple. The primitive-scan boundary logic that relies on recognizing "caller's tuple is also the page's finaltup" then silently misbehaves (wrong primitive-scan continuation decisions for array-key backward scans on coded leaves). The identity check needs to compare by offset (e.g. pstate->offnum == minoff/a finaltup offset) rather than by pointer, or finaltup and the minoff item must share one decode buffer.

-- After the churn, an index-only scan of the live keys must equal the same
-- keys fetched from the heap by a sequential scan (an independent path): if
-- decoding the coded leaves were wrong, the two digests would differ.
SET enable_indexonlyscan = on;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_indexonlyscan is turned on here (line 606) and off at line 610 but never restored to its default; enable_indexscan/enable_seqscan are likewise hand-set to assumed defaults (SET ... = on) at lines 611/615 instead of RESET. The rest of this file consistently pairs the set enable_* to false toggles with reset enable_* (lines 300-301, 390-392, 476-477). Harmless at end-of-session, but inconsistent and brittle if queries are later appended; prefer RESET to put the GUCs back to their real defaults.

-- removable duplicates for the pass to find. (Bloat is quantified in the
-- performance benchmark, not here, since removability is timing-dependent.)
--
CREATE TABLE btree_prefix_bu (id int PRIMARY KEY, u text COLLATE "C", filler text);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block's only assertions are a digest round-trip (bottomup_ok) and a count, both of which pass regardless of whether a bottom-up deletion pass ever ran, whether the leaf was BTP_PREFIX-coded, or even whether prefix_compression had any effect (plain full-key leaves round-trip identically). So the block does not actually verify the stated "bottom-up index deletion on prefix-compressed pages" path was exercised; a regression that silently stopped coding these leaves would not be caught here. Consider adding a positive check that the coded path is active (e.g. a size/page-level assertion, or use amcheck/pageinspect as the other blocks and files do) so this test can fail when the feature is reverted. The same limitation applies to the runtime-dedup block below.

@gburd
gburd force-pushed the master branch 5 times, most recently from eeb0ab5 to 2dc53a9 Compare November 14, 2026 22:26

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 14 issue(s).

  • 14 inline, 0 in summary

Comment on lines +508 to +511
if (leafpage &&
P_HASPREFIX(BTPageGetOpaque(page)) &&
offset >= P_FIRSTDATAKEY(BTPageGetOpaque(page)))
itup = _bt_getitem(page, offset, &uargs->decodebuf);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low confidence / non-blocking: this extends the raw-bytea path (bt_page_items(get_raw_page(...))) to run _bt_prefix_decode() on an attacker-controllable page image. _bt_prefix_decode() does pure byte-splicing with no bounds checks: a corrupt code->group indexes the anchor array out of bounds, and for a posting-list member a bogus NPosting makes plainsz = MAXALIGN(keysz + nhtids*sizeof(ItemPointerData)) overrun the fixed BLCKSZ decodebuf (the plainsz <= BLCKSZ guard is only an Assert, compiled out in production). This is consistent with pageinspect's existing superuser-only "trust the image" model (the surrounding code already reads posting metadata from raw input), so it is not a new risk class, but it is worth the author's awareness as a decode-specific overrun vector on the bytea entry point.

Comment on lines +192 to +194
INSERT INTO prefix_bug
SELECT 'https://example.com/path/to/resource/' || lpad(g::text, 8, '0')
FROM generate_series(1, 20000) g;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prefix check inserts 40,000 rows and then runs both a heapallindexed bt_index_check and a bt_index_parent_check with rootdescend => true. rootdescend performs a full root-to-leaf descent per leaf tuple, making this by far the most expensive check in this file (the existing 100k-row checks on bttest_a/b do not use rootdescend). Confirm 20,000+20,000 is the minimum needed to force splits + dedup + prefix coding; if a smaller count still exercises multi-level splits and posting-list dedup, prefer it to keep make check/buildfarm time down. (low confidence on the exact threshold)

Comment on lines +1348 to +1352
if (P_HASPREFIX(topaque))
{
itup = _bt_getitem(state->target, offset, &state->itembuf);
tupsize = IndexTupleSize(itup);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dangling pointer into a reused decode workspace. On a prefix page itup now points into state->itembuf (line 1350), and bt_entry_unique_check() stores lVis->tid = BTreeTupleGetHeapTID(itup) / posting TID, i.e. a pointer into itembuf. lVis->tid outlives this iteration and is dereferenced in a later iteration (e.g. bt_report_duplicate() at line 875-876, reached via lines 1676-1679). But the next iteration re-decodes the current item into the same itembuf at line 1350, clobbering the bytes lVis->tid points at. The reported "last visible entry" heap TID in the duplicate-corruption message is then wrong/garbage for unique prefix-compressed indexes. The read-ahead path at lines 2087-2091 already uses CopyIndexTuple() for exactly this lifetime reason; the unique path must likewise copy the decoded tuple into storage that outlives the loop (e.g. copy into a dedicated buffer and point lVis->tid at that copy) before stashing a pointer into lVis.

Comment on lines +198 to +199
INSERT INTO prefix_bug VALUES ('zzz-no-shared-prefix'), (NULL);
CREATE INDEX prefix_bug_idx ON prefix_bug (v) WITH (prefix_compression = on);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new prefix-compression test index is non-unique, so it never exercises bt_entry_unique_check() on a BTP_PREFIX leaf -- the code path where a decoded data item is passed into the unique check (and where lVis->tid ends up pointing into the reused decode workspace). Add a unique prefix-compressed index with duplicate keys (or at least WITH (prefix_compression = on) on a unique index) so the decode-plus-unique-check interaction is covered, not just the happy non-unique path.

Comment thread src/include/access/nbtree.h Outdated
/*
* prototypes for functions in nbtprefix.c
*/
extern const char *_bt_page_get_prefix(Page page, Size *len);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead prototype: _bt_page_get_prefix is declared here but has no definition in nbtprefix.c (or anywhere else) and no caller in the tree. A declaration with no implementation is a bisectability/linker hazard if anything ever references it, and an exported prototype with no definition is dead "flexibility" that the community rejects on sight (YAGNI). Remove it, or add the definition and a caller if it is genuinely needed.

Comment on lines +609 to +611
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_seqscan = on;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_indexonlyscan is set off here and never reset. Every subsequent query in this file (the btree_prefix_bu and btree_prefix_dd blocks) runs with IOS disabled, so despite their comments explicitly claiming "an index-only scan of the key column" (line 657) and the dedup block's IOS claim, those scans are plain Index Scans, not index-only scans. That matters here: the IOS-only workspace-growth path in nbtreadpage.c (so->currTuples && !so->bigTuples, which allocates a bigger buffer because a prefix page can decode to more than one BLCKSZ) is reachable only from an index-only scan, so it is exercised only by the single digest at lines 607-608 and nowhere else. Either keep IOS enabled through the bu/dd blocks (and RESET it at the end alongside the other GUCs), or fix the comments to stop claiming IOS coverage they do not have. As written this also violates the "every SET has a matching RESET" discipline.

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +615 to +617
SET enable_indexscan = on;
RESET enable_seqscan;
RESET enable_bitmapscan;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent GUC restore: enable_indexscan is restored with SET ... = on while enable_seqscan/enable_bitmapscan use RESET. Use RESET enable_indexscan for symmetry and to restore the session default rather than pinning an explicit value. (enable_indexonlyscan, set off at line 610, is also missing from this cleanup.)

Comment thread src/test/regress/sql/btree_index.sql Outdated
Comment on lines +629 to +631
-- A non-varlena key column is compressed too (shared bytes need not be text);
-- verify a scan over such an index returns correct rows.
CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment claims a non-varlena key "is compressed too", but prefix sharing in nbtprefix.c (_bt_prefix_shared_len) compares raw stored bytes from the start of the data area. For an int4 key the datum is stored in native endianness, so on little-endian platforms (x86_64/ARM64 — the bulk of the buildfarm) consecutive small int values share no leading bytes (the varying low byte comes first), and essentially no compression occurs. Moreover, unlike the _sz/_grp blocks, this test asserts nothing about size — it only checks one row round-trips (which a plain btree does too). So the comment overstates what the test proves and is platform-dependent. Either assert the size reduction (and only where it actually holds), or reword the comment to drop the "is compressed" claim and describe what is really being verified (correct round-trip of a fixed-width key through the coded format).

Comment on lines +726 to +727
SELECT pg_relation_size('btree_prefix_grp_on') <
pg_relation_size('btree_prefix_grp_off') AS groups_compress;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This size assertion is block-size fragile. With only 151 small rows, both the compressed and uncompressed indexes collapse to a single leaf page under a larger BLCKSZ (e.g. 16KB/32KB buildfarm members), making pg_relation_size equal and this strict < return f while the expected output records t. The regression suite is expected to pass under non-default BLCKSZ (see stats.sql/temp.sql normalizing by current_setting('block_size')). The _sz block above avoids this by using 20000 rows and a ratio, which spans many pages at any block size. Either scale the row count so the uncompressed index exceeds one leaf page at the maximum supported BLCKSZ, or normalize/compare in a block-size-independent way.

Comment on lines 686 to +688
/* minoff is an offset to the lowest non-pivot tuple on the page */
iid = PageGetItemId(pstate->page, pstate->minoff);
firsttup = (IndexTuple) PageGetItem(pstate->page, iid);
(void) iid;
firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (void) iid; exists only to silence the unused-variable warning for the now-removed iid local; remove it together with the iid declaration above.

Suggested change
/* minoff is an offset to the lowest non-pivot tuple on the page */
iid = PageGetItemId(pstate->page, pstate->minoff);
firsttup = (IndexTuple) PageGetItem(pstate->page, iid);
(void) iid;
firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf);
/* minoff is an offset to the lowest non-pivot tuple on the page */
firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf);

@gburd
gburd force-pushed the master branch 9 times, most recently from 63423da to 361c211 Compare November 19, 2026 20:45
Add a prefix_compression storage parameter, default off.  A leaf page
can store common leading bytes once in a directory and retain each
member's remaining bytes.  This can reduce index size when neighboring
keys have long identical prefixes.  Sharing compares stored bytes, not
opclass equality, and decoding restores the original representation.
Variable-length headers can prevent sharing even when text values have
long common prefixes.

Keep pivot tuples plain, retain ordinary heap TIDs and posting lists,
and decode leaf keys through a shared accessor.  The page flag selects
the representation independently of the current reloption.  Changing
the option does not rewrite existing pages.  Multiple groups and
posting-list deduplication can coexist on a coded leaf.

Account for directory and member representation limits as well as
physical page space.  Bound decoded occupancy so a coded leaf can split
into two plain pages.  Use the ordinary split picker when decoding fits
on one page, and honor rightmost fillfactor among legal array split
candidates otherwise.  Split existing items and retry when a large
incoming tuple leaves no legal boundary.

Reconstruct coded split, deduplication, and posting-list insertion WAL
records without consulting catalogs.  The xl_btree_split flags field
changes the WAL format and requires an XLOG_PAGE_MAGIC bump at
integration.  This patch leaves that bump to the committer.

Validate coded metadata before copying decoded bytes.  Extend amcheck
and pageinspect to understand coded leaves, and copy the last visible
heap TID when amcheck checks uniqueness across decoded tuples.  Cover
representation limits, build admission, corruption diagnostics,
fillfactor selection, and standby replay in regression and TAP tests.

Neither BTREE_VERSION nor BTPageOpaqueData changes.  Older binaries
cannot read coded pages; same-build pg_upgrade testing does not establish
cross-major compatibility.  Coding adds work to builds, insertion, and
key-reading scans.  No claim of zero default-off overhead or a general
query speedup is made.

Reviewed-by: Peter Geoghegan
Reviewed-by: Yao Feng
Discussion: https://postgr.es/m/at-01F1wsXLRH0qo@floki

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 21 issue(s).

  • 21 inline, 0 in summary

Comment on lines 1277 to 1281
}


/*
* Loop over page items, starting from first non-highkey item, not high

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gratuitous double blank line (lines 1278-1279). This is diff churn unrelated to the prefix-decode change; pgindent will not collapse it. Drop the extra blank line to keep the diff minimal.

Suggested change
}
/*
* Loop over page items, starting from first non-highkey item, not high
}
/*
* Loop over page items, starting from first non-highkey item, not high

Comment on lines 296 to 298
XLogBeginInsert();

XLogRegisterBuffer(0, buf, REGBUF_STANDARD);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gratuitous whitespace churn: a blank line added immediately after XLogBeginInsert(); inside an otherwise-unmodified WAL block. Remove it to keep the diff minimal.

Suggested change
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);

{
ereport(ERROR,
(errcode(ERRCODE_INDEX_CORRUPTED),
errmsg("invalid prefix %s at offset %u", field, offnum)));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These ERRCODE_INDEX_CORRUPTED messages substitute an internal field name (field, e.g. "anchor offset", "residual extent") via %s and are not wrapped in _(). Per PostgreSQL conventions, never-translated diagnostics that interpolate internal identifiers should use errmsg_internal() so translators are not asked to translate a message whose content varies with an untranslatable field name. Consider errmsg_internal() here.

Suggested change
errmsg("invalid prefix %s at offset %u", field, offnum)));
errmsg_internal("invalid prefix %s at offset %u", field, offnum)));

Comment on lines +134 to +136
/*
* _bt_scan_skip_key -- can _bt_readpage skip decoding the current item?
*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new helper and its doc comment are inserted between the _bt_readpage doc comment (which ends with "Returns true if any matching items found on the page, false if none.") and the _bt_readpage function definition. This splits _bt_readpage's doc comment from the function it documents, making it read as if it describes _bt_scan_skip_key. Move the _bt_scan_skip_key definition and its comment above the _bt_readpage doc-comment block (or after the function), so each doc comment sits directly on top of the function it describes.

Comment on lines +487 to +489
if (!_bt_prefix_form_page(leftpage, left_hikey, litems, nlitems,
true))
elog(PANIC, "failed to lay out prefix-compressed left page after split");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor inconsistency in error-handling level: the new coded left-half path uses elog(PANIC, ...) while the adjacent non-coded path (and the pre-existing split redo code) uses elog(ERROR, ...) for the equivalent "can't happen in redo" PageAddItem/form-page failures. Both are promoted to a recovery failure, but mixing PANIC and ERROR for equivalent redo-invariant violations within the same function is gratuitously inconsistent. Consider using ERROR here to match the surrounding non-coded path (lines 494/503), or document why this specific failure warrants PANIC. (low confidence on which is "right"; the file already mixes both, but this new hunk introduces the divergence.)

Comment on lines +721 to +722
SELECT pg_relation_size('btree_prefix_sz_on') * 3 <
pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These size-ratio assertions use a fixed 20000-row dataset that does not scale with block_size, while the compressed/uncompressed outputs are compared at a hardcoded 3:2 ratio. On non-default block sizes (e.g. 32KB pages) per-leaf packing, tuple-per-page counts and the directory/anchor overhead change, so the compressed:uncompressed ratio is not guaranteed to stay below 2/3 with margin the way it does at 8KB. The grouped case below correctly scales its row count by current_setting('block_size'), but these two do not. Either scale the dataset by block_size as well, or document/verify the 3:2 margin holds across all supported block sizes (1KB-32KB) to avoid buildfarm flakiness.

Comment on lines +1712 to +1713
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz);
origprefix_array = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory leak: in the origprefix_array path ditems is allocated by _bt_prefix_decode_items, which CopyIndexTuples each of the (up to two pages') decoded items, and dhikey is a separately palloc'd copy. Neither dhikey, nor the per-element copies in ditems, nor the ditems array itself is ever freed: _bt_split has a single exit (return rbuf) with no pfree(ditems). _bt_split runs in the caller's (non-reset) context, so every split of a heavily-compressed page leaks a full decoded page's worth of tuples; under a bulk load this accumulates unbounded. dhikey should be pfree'd right here, and ditems/its elements should be freed before return rbuf.

Suggested change
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz);
origprefix_array = true;
(void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz);
if (dhikey)
pfree(dhikey);
origprefix_array = true;

Comment on lines 2375 to 2377
/* split's done */
*newitemplaced = placenewitem;
return rbuf;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Companion to the leak above: before this single return, free the decoded-array workspace on the origprefix_array path, e.g. if (ditems) { for (int k = 0; k < maxoff - P_FIRSTDATAKEY(oopaque) + 1; k++) pfree(ditems[k]); pfree(ditems); }. Note ditems[origpagepostingoff - ...] may have been overwritten with nposting (freed by the caller's posting-split cleanup), so skip that element or free it before the overwrite to avoid a double free.

EXTRA_INSTALL = src/test/modules/injection_points contrib/amcheck
EXTRA_INSTALL = src/test/modules/injection_points contrib/amcheck contrib/pageinspect

TAP_TESTS = 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This TAP_TESTS = 1 adds the new prefix tests (001_prefix.pl, 002_prefix_split.pl) to a module that is deliberately gated behind injection points: the parent src/test/modules/Makefile puts nbtree in SUBDIRS only when enable_injection_points=yes (otherwise ALWAYS_SUBDIRS), and this module's own Makefile makes check a no-op (@echo "injection points are disabled") in that case. The matching meson.build also does if not get_option('injection_points') / subdir_done().

The two new TAP tests do NOT use injection points (they only need amcheck + pageinspect). As placed, they will silently NOT run on the default build (injection points disabled), which is the common buildfarm/cfbot/developer configuration -- so the prefix-compression TAP coverage is gated behind an unrelated, non-default build option. Either host these TAP tests in a module that is not injection-point-gated (e.g. alongside the amcheck/recovery prefix tests already added), or restructure so the TAP tests run unconditionally while the injection-point-dependent regress/isolation tests remain gated.

Comment on lines +576 to +578
SELECT (SELECT md5(string_agg(u, '|' ORDER BY u))
FROM btree_prefix WHERE u IS NOT NULL) =
md5(string_agg(x, '|' ORDER BY x)) AS ios_ok

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression-guard weakness: this "oracle" check is meant to prove the coded-page decode path returns correct keys, but it does not force an index-only scan and does not EXPLAIN-verify the plan. At this point enable_indexscan is still on (only seqscan/bitmapscan are off). A plain Index Scan is therefore a legal plan, and in a plain Index Scan the returned u is fetched from the heap, not decoded from the coded index tuple. If the planner picks an Index Scan, this comparison passes trivially even with a broken index decoder -- i.e. the test would still pass with the decode path reverted. Add SET enable_indexscan = off; (or SET enable_indexonlyscan = on plus disabling index scan) before the digest and assert the plan with EXPLAIN (costs off) so an index-only scan is genuinely exercised. The same gap applies to the ios_digest capture below (line 607-608), which is also only guarded by seqscan/bitmap off with indexscan still enabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant