Repository navigation
Conversation
📜 Change history & discussion (Agora / pg.ddx.io)I have the full picture now. Let me quickly confirm the prior-art threads (Matthias's dynamic prefix truncation and the old 2005/2014 prefix btree threads are confirmed from search results). I have enough to write the report. 🧵 Related discussion
🔗 Related commits / prior art
📋 Commitfest
🧭 Context for reviewers
Generated by pg-history via the Agora MCP server (pg.ddx.io). |
| * xl_btree_split flag bits. The left (original) page is restored from a | ||
| * full-page image when it is prefix-compressed, so it needs no flag; the | ||
| * right page is rebuilt from logged items, so redo must know to set | ||
| * BTP_PREFIX on it (its PREFIX item is included in the logged tuples). | ||
| */ |
There was a problem hiding this comment.
This newly-added comment is factually wrong and contradicts the actual redo code. The only reader of XLH_SPLIT_RPREFIX is btree_xlog_split() (nbtxlog.c:291-296), which restores the prefix-compressed right page from a full-page image (it requires BLK_RESTORED and elog(ERROR)s otherwise). It does NOT rebuild the page from logged items, and it never sets BTP_PREFIX explicitly. Correspondingly the primary registers block 1 with REGBUF_STANDARD | REGBUF_FORCE_IMAGE (no REGBUF_WILL_INIT, no logged tuple data) when P_HASPREFIX(ropaque) (nbtinsert.c:2157-2159). So the claims "the right page is rebuilt from logged items" and "its PREFIX item is included in the logged tuples" are both false. The flag's real purpose is to tell redo that block 1 must be restored from its forced FPI (and thus to skip the item-replay path). Please correct the comment to match the code. (high confidence)
| * xl_btree_split flag bits. The left (original) page is restored from a | |
| * full-page image when it is prefix-compressed, so it needs no flag; the | |
| * right page is rebuilt from logged items, so redo must know to set | |
| * BTP_PREFIX on it (its PREFIX item is included in the logged tuples). | |
| */ | |
| * xl_btree_split flag bits. Both halves of a prefix-compressed leaf split | |
| * are restored from full-page images forced on the primary (their coded | |
| * items are laid out in a physical order that _bt_restore_page would not | |
| * reproduce). The left (original) page is block 0, whose FPI redo already | |
| * detects via BLK_RESTORED, so it needs no flag. This flag tells redo that | |
| * block 1 (the right page) must likewise be restored from its forced FPI | |
| * rather than rebuilt from logged items. | |
| */ |
| SELECT | ||
| (SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix) = | ||
| (SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix WHERE u IS NOT NULL OR u IS NULL) | ||
| AS digest_consistent; |
There was a problem hiding this comment.
This "full differential digest" is tautological and cannot catch any regression. Both subqueries read the same table btree_prefix via the same access path, and the predicate u IS NOT NULL OR u IS NULL is always true, so the two digests are byte-for-byte identical by construction. digest_consistent returns t even if prefix compression is reverted or corrupts data. The comment promises comparing "the prefix index, the uncompressed index, and a sequential scan" but the SQL compares none of those. To actually distinguish the implementations, force each access path explicitly and compare: e.g. digest of a scan forced through btree_prefix_on (prefix) vs one forced through btree_prefix_off (uncompressed) vs a seqscan (enable_indexscan=off), and assert all three agree. As written this test is worthless.
| -- by keys that share a long prefix (URL-like), keys that share almost nothing | ||
| -- (md5), fixed-width zero-padded codes, and oversized (>126-byte) values. | ||
| -- | ||
| CREATE TABLE btree_prefix (u text, v text); |
There was a problem hiding this comment.
Locale/collation portability defect. The text column u is created with the database default collation, so =, BETWEEN, u > ..., ORDER BY u, and string_agg(u, '|' ORDER BY u) all depend on the server's collation. Under a non-C collation the recorded row order in expected/btree_index.out (e.g. the backward-scan output and every digest) will differ, breaking the buildfarm. This same test file already follows the C-collation convention (create temp table btree_bpchar (f1 text collate "C")). Declare the key column as u text COLLATE "C" (and the digest/ORDER BY accordingly) so contents and expected output are deterministic across platforms.
| SELECT 'https://example.com/other/' || lpad(g::text, 8, '0'), md5((-g)::text) | ||
| FROM generate_series(1, 10000) g; | ||
| -- Oversized first-column values (stored with shared = 0) and a NULL. | ||
| INSERT INTO btree_prefix VALUES (repeat('x', 300), 'big'), (NULL, 'nullkey'); |
There was a problem hiding this comment.
Boundary coverage is missing at the one place the encoding actually switches. BTREE_PREFIX_MAX is 126 and BTREE_PREFIX_RAW is 127 (see nbtree.h); the shared-count field is a single uint8 masked to 7 bits, so the coded/raw transition happens around 126/127 shared bytes, not at "300". repeat('x', 300) merely has zero shared prefix with the URL keys (shared = 0), which is a different path from the >126 case the comment claims to exercise. Add keys that share exactly 125/126/127 bytes with the page prefix to cover the boundary where the shared encoding changes behavior.
| ALTER INDEX btree_prefix_off SET (prefix_compression = on); | ||
| SELECT reloptions FROM pg_class WHERE relname = 'btree_prefix_off'; | ||
| -- A non-text (non-varlena) first column silently stores no prefix, but is legal. | ||
| CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on); |
There was a problem hiding this comment.
The non-varlena index is created immediately before DROP TABLE and never queried, so this only asserts that CREATE INDEX doesn't error. The comment claims the "silently stores no prefix, but is legal" path is covered, but the correctness of scans over that index (the actual behavior being tested) is never exercised. Add at least one query using this index (forced via enable_seqscan=off) and verify it returns the expected rows, or drop the index assertion if it adds no value.
| Prefix split halves, and a deduplication rewrite of a prefix page, are logged | ||
| by full-page image rather than by replaying individual tuples, because the |
There was a problem hiding this comment.
Minor documentation imprecision. The code (nbtinsert.c) forces the left page's full-page image based on whether the pre-split original page is BTP_PREFIX (P_HASPREFIX(origpage)), not based on whether the resulting left half ends up prefix-compressed -- a left half can end up plain yet still be FPI-logged because its source items are coded against the original prefix. The phrase "Prefix split halves ... are logged by full-page image" reads as if only the halves that remain prefix-compressed are FPI'd. Consider clarifying that the left half is FPI-logged whenever the original page was prefix-compressed. Low-confidence/low-severity nit; the overall mechanism is correct.
| SELECT md5(string_agg(u, '|' ORDER BY u)) = | ||
| (SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix) AS ios_ok | ||
| FROM (SELECT u FROM btree_prefix ORDER BY u) s; |
There was a problem hiding this comment.
This "index-only scan" check is tautological, same defect as the full differential digest below. Both operands derive from the identical source btree_prefix and are wrapped in string_agg(u, '|' ORDER BY u), which sorts its input, so the two md5 digests are byte-for-byte equal by construction regardless of which access path (prefix index, uncompressed index, or seqscan) each side uses. If key decoding were broken, both sides would return the same wrong digest and ios_ok would still be t. The test cannot catch a regression in the new prefix-decode path. To actually test decoding, compare the prefix index against an independent oracle, e.g. force the IOS over btree_prefix_on and compare its ordered output to the original literal values inserted (or to a seqscan result that provably did not use the index).
| -- by keys that share a long prefix (URL-like), keys that share almost nothing | ||
| -- (md5), fixed-width zero-padded codes, and oversized (>126-byte) values. | ||
| -- | ||
| CREATE TABLE btree_prefix (u text, v text); |
There was a problem hiding this comment.
The second column v is populated by every INSERT (md5 values) but never referenced by any query in this test block. It is dead test data that only adds noise and insert cost. Either drop the column (the index is on u only) or use it, e.g. to exercise a two-column prefix-compressed index.
| #define BTTupleStorageSize(relation) \ | ||
| (BTGetPrefixCompression(relation) ? \ | ||
| (BLCKSZ + (BLCKSZ / (int) MAXALIGN(sizeof(IndexTupleData) + 2)) * \ | ||
| (int) MAXALIGN(BTREE_PREFIX_MAX)) : \ | ||
| BLCKSZ) |
There was a problem hiding this comment.
Memory-safety bug (high confidence): BTTupleStorageSize falls back to plain BLCKSZ whenever BTGetPrefixCompression(relation) is false, but BTP_PREFIX pages are not tied to the current reloption value. Per the README and the code (flags are only cleared on split/empty/dedup-rewrite, never by an ALTER INDEX ... SET (prefix_compression = off)), BTP_PREFIX leaves persist in an index after the option is turned off. _bt_getitem() decodes based on the page flag P_HASPREFIX(opaque), not the reloption, so an index-only scan over a surviving BTP_PREFIX leaf will decode tuples whose combined size can exceed BLCKSZ and overflow so->currTuples/so->markTuples (allocated as half * 2 in btrescan). The sizing must not depend on the live option; size the buffer for the worst case whenever prefix compression is a btree index possibility (e.g. drop the BTGetPrefixCompression gate, or key it on whether the index could ever contain BTP_PREFIX pages).
| parameter on with ALTER INDEX does not rewrite anything; each leaf is converted | ||
| the next time it is written whole, and compressed and uncompressed leaves | ||
| coexist in the same index indefinitely. |
There was a problem hiding this comment.
Documentation gap (moderate confidence): this paragraph only describes turning the parameter on and asserts compressed and uncompressed leaves "coexist in the same index indefinitely", but it never states what happens when the parameter is turned off. The code keeps BTP_PREFIX on existing pages (flags are cleared only on split/empty/dedup-rewrite), so BTP_PREFIX leaves survive an ALTER INDEX ... SET (prefix_compression = off) and remain readable. This is not merely cosmetic: it is exactly the invariant the index-only-scan buffer sizing (BTTupleStorageSize in nbtree.h) must honor. Document that turning the option off also does not rewrite anything, so prefix pages can be read back with the option off.
|
Addressed the review in 6edd15c:
Re-ran on Fedora: regress + isolation + amcheck + pageinspect + pgstattuple all The four larger follow-ups (compose with deduplication, re-enable bottom-up |
There was a problem hiding this comment.
🔍 OCR found 19 issue(s).
- 17 inline, 2 in summary
📄 src/backend/access/nbtree/README
This paragraph describes the opposite of what the code does. A prefix split does NOT full-page-image its halves as the primary mechanism. nbtinsert.c registers block 0 with REGBUF_STANDARD and block 1 with REGBUF_WILL_INIT (not REGBUF_FORCE_IMAGE), and btree_xlog_split_prefix()/the prefix-dedup redo branch explicitly rebuild each coded half from the logged plain items of the pre-split page -- see the code comments "No full-page image is needed" (nbtxlog.c) and "avoiding a full-page image" (nbtxlog.h). A forced FPI is used only as an overflow fallback when the plain payload would exceed BLCKSZ (XLH_SPLIT_RFPI / XLH_DEDUP_RFPI). The README should state that redo reconstructs the coded layout from logged plain items (because _bt_restore_page's item-by-item replay cannot reproduce it), with FPI only as the overflow fallback -- not that FPI is the normal logging mechanism.
📄 src/backend/access/nbtree/README
This claim is inaccurate. A deduplication rewrite of a prefix page is NOT FPI-logged by default. In nbtdedup.c (_bt_dedup_pass), block 0 is registered with REGBUF_STANDARD and the chosen prefix plus the deduped plain items are logged via XLogRegisterBufData so redo re-lays-out the coded page. A forced full-page image (REGBUF_FORCE_IMAGE / XLH_DEDUP_RFPI) is used only as a fallback when the packed plain payload would exceed BLCKSZ (dedup_rfpi, set when off > BLCKSZ). The normal path logs individual (plain) tuples, the opposite of what this sentence states. (high confidence)
| * high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum | ||
| * record so redo does the same. |
There was a problem hiding this comment.
The comment is inaccurate: nothing about the PREFIX-item drop is recorded in the vacuum record. xl_btree_vacuum only carries ndeleted/nupdated and the offset/tuple payloads (unchanged by this patch), and btree_xlog_vacuum() re-derives the drop by independently re-evaluating P_HASPREFIX(opaque) && PageGetMaxOffsetNumber(page) == BTPageGetPrefixOff(opaque) after replaying the deletes. Redo does the same thing, but not because it was recorded. Reword to reflect that redo recomputes the same condition from the post-delete page state. (high confidence)
| * high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum | |
| * record so redo does the same. | |
| * high key (_bt_mark_page_halfdead). Redo (btree_xlog_vacuum) re-derives | |
| * this from the post-delete page state, so nothing extra is logged. |
| IndexTuple rightitems[MaxIndexTuplesPerPage]; | ||
| int nright = 0; |
There was a problem hiding this comment.
Stack buffer overflow risk during redo. rightitems[] and dummyright[] are fixed-size stack arrays sized to MaxIndexTuplesPerPage, but a prefix-compressed leaf can physically hold MORE line pointers than that limit: MaxIndexTuplesPerPage is derived assuming each tuple is at least MAXALIGN(sizeof(IndexTupleData) + 1) bytes, whereas a coded tuple can be smaller (1-byte varlena header + 1-byte shared code). The primary correctly sizes these arrays dynamically as palloc_array(IndexTuple, maxoff + 2) (see _bt_split in nbtinsert.c). The while (datalen > 0) loop increments nright with no bound check and trusts the WAL record's self-describing IndexTupleSize, so a legitimately overfull (or corrupt) record overruns the stack. Allocate these with palloc sized from the record, matching the primary.
| #define XLH_SPLIT_LPREFIX_ORIG 0x02 /* left half prefix is the orig page's */ | ||
| #define XLH_SPLIT_RPREFIX_ORIG 0x04 /* right half prefix is the orig page's */ |
There was a problem hiding this comment.
Dead scaffolding and misleading documentation. XLH_SPLIT_LPREFIX_ORIG and XLH_SPLIT_RPREFIX_ORIG are defined but never referenced anywhere (neither the primary in nbtinsert.c nor redo in nbtxlog.c reads them). The comment claims prefix bytes are "not logged ... when the *_PREFIX_ORIG flag is set", but the actual code always logs the prefix bytes (XLogRegisterBufData of leftprefix/rightprefix) and redo always reads them from the logged block data. Either the described optimization was never implemented (so remove the two flags and fix the comment), or the implementation is incomplete. As written, the comment describes behavior that does not exist.
| IndexTuple dummyright[MaxIndexTuplesPerPage]; | ||
| int dummynr; |
There was a problem hiding this comment.
Same unbounded fixed-size stack array as the right/left half of the split path: dummyright is MaxIndexTuplesPerPage, but _bt_split_leaf_collect writes *nright entries for a prefix page that can exceed MaxIndexTuplesPerPage. Even though this redo path only needs the left items, _bt_split_leaf_collect unconditionally fills the right array too, so dummyright can overflow. Size it to match the primary (maxoff + 2) via palloc.
| IndexTuple items[MaxIndexTuplesPerPage]; | ||
| int nitems = 0; |
There was a problem hiding this comment.
Same unbounded stack array in the dedup redo path: items[MaxIndexTuplesPerPage] is filled by a while (datalen > 0) loop over trusted WAL data, with nitems++ and no bound check. A prefix-compressed dedup result page can hold more coded items than MaxIndexTuplesPerPage. Allocate from the record size instead of a fixed stack array.
| if (spagenumber != P_NONE) | ||
| { | ||
| right_hikey = (IndexTuple) datapos; | ||
| datapos += MAXALIGN(IndexTupleSize(right_hikey)); | ||
| datalen -= MAXALIGN(IndexTupleSize(right_hikey)); | ||
| } |
There was a problem hiding this comment.
Unaligned IndexTuple access during redo. The primary logs the right half's prefix as rightprefixlen raw bytes with no MAXALIGN padding (see nbtinsert.c: roff += rightprefixlen before copying the high key/items). rightprefixlen is the number of shared first-column bytes from _bt_prefix_choose() and is an arbitrary value, not a multiple of MAXALIGN. Here datapos += xlrec->rightprefixlen then casting to IndexTuple and calling IndexTupleSize(right_hikey) (and likewise for each rightitems[] entry below) dereferences t_info at an unaligned address. On strict-alignment platforms (ARM, s390x, etc.) this is undefined behavior / a crash. Either MAXALIGN-pad the prefix on the primary before the high key/items, or memcpy the tuples into aligned storage in redo before reading them.
| if (xlrec->flags & XLH_DEDUP_NOT_RIGHTMOST) | ||
| { | ||
| hikey = (IndexTuple) datapos; | ||
| datapos += MAXALIGN(IndexTupleSize(hikey)); | ||
| datalen -= MAXALIGN(IndexTupleSize(hikey)); | ||
| } |
There was a problem hiding this comment.
Same unaligned IndexTuple access in the prefix-dedup redo path. The primary packs prefixlen raw (unpadded) prefix bytes before the high key and deduped items (see nbtdedup.c: off = dedup_prefixlen with no MAXALIGN). After datapos += xlrec->prefixlen, casting to IndexTuple and calling IndexTupleSize(hikey) / IndexTupleSize(it) reads t_info at a non-MAXALIGN'd address whenever prefixlen % MAXALIGN != 0, which crashes on strict-alignment architectures. Pad the prefix to MAXALIGN on the primary, or copy tuples into aligned buffers in redo.
| IndexTuple itup; | ||
| BTItemBuf itupbuf; |
There was a problem hiding this comment.
BTItemBuf is PGAlignedBlock, i.e. a full BLCKSZ-sized buffer (8KB by default, up to 32KB). Declaring it as a local in _bt_compare() -- the single hottest routine in nbtree, called O(log n) times per descent and in every page binary search -- inflates the stack frame by BLCKSZ on every call, including internal-page and non-prefix-index comparisons where _bt_getitem() never decodes (BTP_PREFIX is a leaf-only flag). Since &itupbuf is address-taken, the compiler cannot elide it. Consider having the caller supply the scratch buffer (so it is reserved once per descent rather than per comparison), or otherwise avoid paying this cost on the non-prefix fast path. (performance, moderate confidence)
| Size half = BTTupleStorageSize(scan->indexRelation); | ||
|
|
||
| so->currTuples = (char *) palloc(half * 2); | ||
| so->markTuples = so->currTuples + half; |
There was a problem hiding this comment.
This grows the per-scan index-only tuple workspace from BLCKSZ * 2 (16KB) to BTTupleStorageSize * 2, which is roughly 9x larger (~144KB at default BLCKSZ: 8192 + (8192/16)*128 = 73728 per half). This larger allocation is now paid by every index-only scan on every btree index at btrescan time, even indexes that have never used prefix compression. The nbtree.h comment justifies the worst-case sizing (a surviving BTP_PREFIX page can exist on any index), but the allocation is unconditional rather than gated on the index actually being capable of carrying prefix pages. Worth confirming this per-scan memory regression is acceptable, or sizing it based on whether the index can contain BTP_PREFIX pages. (performance, moderate confidence)
|
|
||
| /* xl_btree_dedup flag bits */ | ||
| #define XLH_DEDUP_NOT_RIGHTMOST 0x01 /* prefix payload includes a high key */ | ||
| #define XLH_DEDUP_RFPI 0x02 /* prefix page carried as a forced image */ |
There was a problem hiding this comment.
XLH_DEDUP_RFPI is write-only dead scaffolding. The primary sets it (nbtdedup.c:387) but no code ever reads it: the dedup redo path relies on XLogReadBufferForRedo() returning BLK_RESTORED for the forced full-page image and short-circuiting before it would inspect this flag. Either drop the flag (and stop setting it), or if it is kept purely as documentation, say so explicitly. As written it is a knob with no caller, which the community rejects (YAGNI). Note this is a separate flag from the already-flagged XLH_SPLIT_LPREFIX_ORIG/RPREFIX_ORIG.
| bool pageprefix = P_HASPREFIX(opaque); | ||
| BTItemBuf curbuf; /* decode scratch for the loop's current item */ | ||
| BTItemBuf basebuf; /* decode scratch kept for the pending base */ | ||
| XLogRecPtr recptr; | ||
| IndexTuple *dedup_plain = NULL; /* deduped plain items, for prefix WAL */ |
There was a problem hiding this comment.
This entire prefix-handling block in _bt_dedup_pass is unreachable. The only caller, _bt_delete_or_dedup_one_page() in nbtinsert.c, now guards the call with !P_HASPREFIX(...), so _bt_dedup_pass() is never entered with a BTP_PREFIX page and pageprefix is always false here. That makes all the if (pageprefix) decode branches, the _bt_prefix_compress_page() re-compression block, the dedup_plain/dedup_hikey/dedup_walbuf packing, and the dedup_islogprefix ? 0 : state->nintervals WAL branch dead code (together with the matching XLOG_BTREE_DEDUP prefix-redo path in nbtxlog.c). This is ~150 lines of complex, untested WAL-logging scaffolding for a path that is deliberately not wired up (the nbtinsert.c comment states "Deduplication is not yet applied to prefix-compressed pages"). Drop it until dedup on prefix pages is actually enabled, per YAGNI. (high confidence)
| if (state->btps_prefix == NULL) | ||
| state->btps_prefix = _bt_prefix_candidate(itup, | ||
| &state->btps_prefixlen); |
There was a problem hiding this comment.
state->btps_prefix is set to a pointer into itup (via _bt_prefix_candidate, which returns VARDATA_ANY(val) of the tuple's first column). But itup here comes from tuplesort_getindextuple(), whose returned tuple is only valid until the next fetch on that sortstate. On the next _bt_buildadd() the previous itup's memory has been reused by the tuplesort, so btps_prefix now dangles and the later _bt_prefix_shared(state->btps_prefix, ...) reads stale/garbage bytes, yielding a meaningless shared estimate (and reading memory this code does not own). It does not corrupt the page (the hard fit check and the recompute in _bt_prefix_compress_page protect correctness), but the fillfactor estimate is effectively random. Copy the candidate prefix bytes into stable storage in BTPageState instead of keeping a pointer into the transient tuple. (medium confidence)
| Deduplication and bottom-up index deletion are not currently applied to | ||
| BTP_PREFIX pages (such a page splits instead): their in-place rewrite of a page | ||
| cannot always be re-compressed within one page, and prefix compression already | ||
| reclaims much of the space they target. Composing them with prefix | ||
| compression is future work. |
There was a problem hiding this comment.
The claim that bottom-up index deletion is not applied to BTP_PREFIX pages contradicts the implementation. In nbtinsert.c (_bt_delete_or_dedup_one_page), only the deduplication pass is gated by !P_HASPREFIX(...); the bottom-up deletion pass (_bt_bottomupdel_pass) runs unconditionally, and its own comment states "Bottom-up deletion above is safe on prefix pages -- it only removes items, never rewrites survivors." So bottom-up deletion IS applied to prefix pages; only deduplication is skipped. Reword to scope the "not currently applied / splits instead" statement to deduplication alone.
| With the prefix_compression storage parameter (default off) a leaf page may | ||
| front-code the first key column of each of its data tuples against a per-page | ||
| prefix, which shrinks indexes on keys that share long leading bytes (URLs, |
There was a problem hiding this comment.
This opening description omits that prefix compression applies only when the first key column is of a varlena (variable-length) type. _bt_prefix_enabled() gates on TupleDescAttr(...,0)->attlen == -1, and nbtree.h documents the same requirement ("whose first key column is of a varlena type"). As written, the README implies any first key column is front-coded, which is misleading for an index whose leading column is fixed-length (it is silently never compressed). (moderate confidence)
| SET enable_indexscan = off; | ||
| SET enable_indexonlyscan = off; | ||
| SET enable_seqscan = on; |
There was a problem hiding this comment.
enable_indexonlyscan is set off here but never reset, so the session ends with a non-default GUC. Every other GUC block in this file restores what it changes (see the reset enable_seqscan/indexscan/bitmapscan at the earlier tests), and this block itself resets enable_seqscan/enable_bitmapscan at lines 615-616 and resets enable_indexscan to its default via SET ... = on at line 614. Leaving enable_indexonlyscan permanently off is inconsistent and sloppy. Add RESET enable_indexonlyscan; alongside the other resets (and prefer RESET enable_indexscan; over SET enable_indexscan = on; for consistency with the file's idiom).
| SET enable_indexscan = off; | |
| SET enable_indexonlyscan = off; | |
| SET enable_seqscan = on; | |
| SET enable_indexscan = off; | |
| SET enable_indexonlyscan = off; | |
| SET enable_seqscan = on; |
| * Leaf halves are built as whole pages now, so they can be prefix- | ||
| * compressed from the start (Strategy 1: a page is faulted into the | ||
| * BTP_PREFIX format only when it is written whole). Both halves of a | ||
| * prefix split are restored from full-page images in redo (their coded | ||
| * physical layout would not be reproduced by item replay): the left half | ||
| * (block 0) is FPI-logged whenever the original page was prefix-compressed | ||
| * and redo detects the image via BLK_RESTORED, so it needs no record flag; | ||
| * the right half (block 1) records XLH_SPLIT_RPREFIX so redo restores it | ||
| * from its forced image rather than rebuilding it from logged tuples. | ||
| * dataitem copies and the lefthighkey copy are freed via the short-lived | ||
| * insert context. |
There was a problem hiding this comment.
This comment describes an abandoned FPI-based design and flatly contradicts the code and the redo path. In reality neither half is restored from a full-page image in the normal case: block 0 is registered with plain REGBUF_STANDARD (no forced image) and btree_xlog_split_prefix() rebuilds the left half from the pre-split page on BLK_NEEDS_REDO; the right half is carried as its logged plain items (rightwalbuf) and laid out coded by _bt_split_leaf_layout(), only falling back to an FPI under XLH_SPLIT_RFPI. The flag named here, XLH_SPLIT_RPREFIX, does not exist (the real flags are XLH_SPLIT_LEAFPREFIX and XLH_SPLIT_RFPI; XLH_SPLIT_RPREFIX_ORIG is defined but unused). This stale comment will badly mislead anyone auditing WAL/redo correctness and should be rewritten to match the tuple-replay design actually implemented (and correctly described near the XLogInsert call below).
|
📊 OCR posted 16/17 inline comment(s). 1 could not be posted
|
There was a problem hiding this comment.
🔍 OCR found 23 issue(s).
- 22 inline, 1 in summary
📄 src/backend/access/nbtree/README
This WAL paragraph contradicts the shipped implementation; it describes a design that was not the one committed. The code does NOT use a full-page image as the primary redo mechanism for prefix splits/dedup:
- Left half (block 0) is registered with plain REGBUF_STANDARD (nbtinsert.c _bt_split, XLogRegisterBuffer(0, buf, REGBUF_STANDARD)) -- there is no forced FPI. Redo rebuilds the left half from the original page's items via _bt_split_leaf_collect + _bt_split_leaf_layout (btree_xlog_split, nbtxlog.c).
- Right half (block 1) is registered REGBUF_WILL_INIT and rebuilt from logged PLAIN items; REGBUF_FORCE_IMAGE is used only in the XLH_SPLIT_RFPI overflow fallback, not 'whenever it ends up prefix-compressed'.
- A dedup rewrite of a prefix page is likewise carried as the prefix + plain items (xl_btree_dedup prefixlen payload), with an FPI only via the XLH_DEDUP_RFPI fallback.
The nbtxlog.h header comment states exactly the opposite of this paragraph ('avoiding a full-page image', 'logs no full-page image: redo rebuilds each half the way the primary did'). Rewrite this paragraph to describe item-replay as the primary path with FPI only as a bounded (>BLCKSZ payload) fallback, consistent with XLH_SPLIT_RFPI / XLH_DEDUP_RFPI.
| * high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum | ||
| * record so redo does the same. |
There was a problem hiding this comment.
This comment is inaccurate. Nothing about the PREFIX drop is "recorded in the vacuum record": xl_btree_vacuum only carries ndeleted/nupdated plus the offset arrays (see nbtxlog.h), and no field is added here. Redo (btree_xlog_vacuum, nbtxlog.c) independently re-derives the same condition from the page state after replaying the deletions (P_HASPREFIX && PageGetMaxOffsetNumber == BTPageGetPrefixOff). Stating it is recorded is misleading and would steer a future maintainer wrong if the record format is changed. Reword to match the redo path, e.g. "Redo re-derives this from the replayed page state in btree_xlog_vacuum."
| * high key (nbtpage.c's _bt_mark_page_halfdead). Recorded in the vacuum | |
| * record so redo does the same. | |
| * high key (nbtpage.c's _bt_mark_page_halfdead). btree_xlog_vacuum | |
| * re-derives this from the replayed page state, so no field is recorded. |
| IndexTuple itup; | ||
| BTItemBuf itupbuf; |
There was a problem hiding this comment.
BTItemBuf is PGAlignedBlock, i.e. a full BLCKSZ (8KB default) stack buffer. _bt_compare is the hottest comparison routine in nbtree, invoked repeatedly inside the binary-search loops (_bt_binsrch, _bt_binsrch_insert) at every level of a descent. This adds ~8KB to its stack frame on every call, paid unconditionally even for the vast majority of indexes that are not prefix-compressed. Worth confirming the added per-call frame size is acceptable on the descent hot path (stack depth, not just CPU); the other new BTItemBuf locals in nbtreadpage.c are per-page rather than per-comparison, so this call site is the most sensitive one.
| if (xlrec->rightprefixlen > 0) | ||
| { | ||
| rightprefix = datapos; | ||
| datapos += xlrec->rightprefixlen; | ||
| datalen -= xlrec->rightprefixlen; | ||
| } | ||
| if (spagenumber != P_NONE) | ||
| { | ||
| right_hikey = (IndexTuple) datapos; | ||
| datapos += MAXALIGN(IndexTupleSize(right_hikey)); | ||
| datalen -= MAXALIGN(IndexTupleSize(right_hikey)); | ||
| } |
There was a problem hiding this comment.
Unaligned memory access on strict-alignment platforms (ARM/SPARC/PPC, etc.).
The right-half payload is laid out as [rightprefix bytes][right_hikey][items...], and the prefix bytes are logged WITHOUT MAXALIGN padding (see nbtinsert.c: roff = rightprefixlen; ... roff += MAXALIGN(...)). XLogRecGetBlockData only MAXALIGNs the start of the block data. After datapos += xlrec->rightprefixlen (rightprefixlen can be 1..126, including odd values), datapos is no longer aligned, so (IndexTuple) datapos and IndexTupleSize(right_hikey) / the subsequent item loop dereference t_info (a uint16) at an unaligned address. _bt_prefix_encode/PageAddItem then read the whole tuple header from an unaligned source.
The pre-existing _bt_restore_page deliberately avoids this by using memcpy into an aligned IndexTupleData and a void *items[] array (see its comment about not assuming alignment). This new path regresses. Either MAXALIGN the prefix length in the logged payload on both the write and read sides, or memcpy the header out before calling IndexTupleSize. The left-half path (block 0) happens to be safe only because the prefix is logged last there.
| datapos += xlrec->prefixlen; | ||
| datalen -= xlrec->prefixlen; | ||
| if (xlrec->flags & XLH_DEDUP_NOT_RIGHTMOST) | ||
| { | ||
| hikey = (IndexTuple) datapos; | ||
| datapos += MAXALIGN(IndexTupleSize(hikey)); | ||
| datalen -= MAXALIGN(IndexTupleSize(hikey)); | ||
| } | ||
| while (datalen > 0) | ||
| { | ||
| IndexTuple it = (IndexTuple) datapos; | ||
| Size sz = MAXALIGN(IndexTupleSize(it)); | ||
|
|
||
| items[nitems++] = it; | ||
| datapos += sz; | ||
| datalen -= sz; | ||
| } |
There was a problem hiding this comment.
Same unaligned-access hazard as the split right-half path. The dedup payload is [prefix bytes][hikey][items...] with the prefix logged unpadded (nbtdedup.c packs off += dedup_prefixlen with no MAXALIGN). After datapos += xlrec->prefixlen with an odd prefixlen, (IndexTuple) datapos on the hikey and every item in the loop is an unaligned IndexTuple dereference (IndexTupleSize reads t_info). This will SIGBUS/UB on strict-alignment architectures. Pad the prefix to MAXALIGN on the write side (and mirror it here), or memcpy each tuple header out before use, consistent with _bt_restore_page.
| rightitems[0] = NULL; /* not reused below */ | ||
| { | ||
| IndexTuple dummyright[MaxIndexTuplesPerPage]; | ||
| int dummynr; | ||
|
|
||
| _bt_split_leaf_collect(origpage, xlrec->firstrightoff, | ||
| xlrec->newitemoff, newitem, newitemonleft, | ||
| origpagepostingoff, nposting, | ||
| leftitems, &nleft, dummyright, &dummynr); | ||
| } |
There was a problem hiding this comment.
This rightitems[0] = NULL; followed by a nested block declaring a throwaway dummyright[MaxIndexTuplesPerPage] only to discard the collected right half is an awkward workaround. It burns an extra MaxIndexTuplesPerPage-entry stack array (~ several KB) purely to satisfy the _bt_split_leaf_collect signature. Consider letting _bt_split_leaf_collect accept NULL right arrays (skip the right side) so redo can collect only the half it needs, rather than reusing rightitems and a dummy array.
| while (datalen > 0) | ||
| { | ||
| IndexTuple it = (IndexTuple) datapos; | ||
| Size sz = MAXALIGN(IndexTupleSize(it)); | ||
|
|
||
| rightitems[nright++] = it; | ||
| datapos += sz; | ||
| datalen -= sz; | ||
| } |
There was a problem hiding this comment.
The while (datalen > 0) loop appends to rightitems[] (sized MaxIndexTuplesPerPage) with no bound check on nright. A corrupt record with an oversized datalen would overrun the on-stack array. The pre-existing _bt_restore_page has the same unchecked pattern, so this matches established nbtree-redo convention, but given this runs on standbys where a corrupt record is possible, consider an Assert(nright < MaxIndexTuplesPerPage) (and likewise in the dedup path) to fail loudly rather than smash the stack.
| FROM generate_series(1, 20) g; | ||
| CREATE INDEX test_prefix_idx ON test_prefix (u) WITH (prefix_compression = on); | ||
| CREATE INDEX test_plain_idx ON test_prefix (u); | ||
| SELECT btpo_flags & 512 <> 0 AS coded FROM bt_page_stats('test_prefix_idx', 1); |
There was a problem hiding this comment.
The magic literal 512 here is the BTP_PREFIX flag bit (1 << 9). An unadorned numeric flag mask is opaque to readers and brittle if the flag ever moves. Since pageinspect doesn't expose the symbolic name in SQL, at minimum add a short comment noting this tests the BTP_PREFIX bit, so the intent is clear in the test output/source.
| FULL JOIN bt_page_items('test_plain_idx', 1) p ON p.itemoffset = c.itemoffset - 1 | ||
| WHERE c.itemoffset > 1 AND (c.data IS DISTINCT FROM p.data OR c.htid IS DISTINCT FROM p.htid); |
There was a problem hiding this comment.
The FULL JOIN is partly defeated by WHERE c.itemoffset > 1: rows where c is unmatched-NULL (i.e., a plain item with no corresponding coded item) are dropped because NULL > 1 is not true, so a prefix page with fewer data items than the plain page would not be flagged as a mismatch. It happens to be harmless here (both indexes hold the same 20 unique rows with no dead tuples, so counts are equal), but the FULL JOIN is misleading about what it verifies. Consider also asserting equal row counts, or drop the FULL to a plain JOIN to match the actual intent.
| static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state, | ||
| TM_IndexDeleteOp *delstate); |
There was a problem hiding this comment.
This blank line is cosmetic churn on an untouched region unrelated to prefix compression. PostgreSQL rejects whitespace-only changes to lines a patch does not otherwise need to touch; please drop it to keep the diff minimal.
| static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state, | |
| TM_IndexDeleteOp *delstate); | |
| static void _bt_bottomupdel_finish_pending(Page page, BTDedupState state, | |
| TM_IndexDeleteOp *delstate); |
| XLogBeginInsert(); | ||
|
|
||
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); |
There was a problem hiding this comment.
Cosmetic blank line inserted into the XLOG block; it is unrelated to prefix compression and is pure whitespace churn on an otherwise untouched region. Remove to keep the diff minimal.
| XLogBeginInsert(); | |
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); | |
| XLogBeginInsert(); | |
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); |
| static BTItemBuf prefixbuf; | ||
|
|
||
| itup = _bt_getitem(page, offnum, &prefixbuf); |
There was a problem hiding this comment.
Using a static BTItemBuf (an 8KB PGAlignedBlock) here is a surprising deviation from the established pattern: every other _bt_getitem caller (nbtdedup.c, nbtreadpage.c, nbtxlog.c) uses a stack-local BTItemBuf. The static costs 8KB of permanent per-process BSS even for indexes that never use prefix compression, and it silently retains a decoded tuple across calls. The only reason to avoid a stack-local here is the 8KB frame on this hot, recursive comparison routine -- but that rationale is undocumented, so a future reader will read this as a footgun (the returned pointer is clobbered by the next _bt_compare). Correctness is fine today because itup never escapes the function, but please either document why a stack-local is not used or reuse the standard pattern. At minimum, add a comment explaining the static lifetime.
| /* | ||
| * prototypes for functions in nbtprefix.c | ||
| */ | ||
| extern const char *_bt_page_get_prefix(Page page, Size *len); |
There was a problem hiding this comment.
_bt_page_get_prefix is a dead prototype: it has no definition anywhere (not in nbtprefix.c) and no caller in the backend or contrib. Every other prototype in this block is defined and used; this one is dangling (leftover from an earlier design or an unimplemented helper). Remove it per YAGNI/minimal-diff discipline — an unused declaration will also eventually trip -Wmissing-prototypes-style checks and confuses readers about the module's API.
| SELECT pg_relation_size('btree_prefix_grp_on') < | ||
| pg_relation_size('btree_prefix_grp_off') AS groups_compress; |
There was a problem hiding this comment.
This assertion is not portable across BLCKSZ. The _grp indexes hold only 151 small (int4, ~35-byte text) tuples, so the result of _on < _off depends entirely on the uncompressed index spilling into a second leaf page while the compressed one fits in one. That is borderline even at the default 8kB BLCKSZ, and at 16kB/32kB BLCKSZ both indexes fit on a single leaf page, making pg_relation_size equal and groups_compress evaluate to f -- breaking the recorded expected output t. Other size/page-dependent tests in this very file explicitly document their BLCKSZ assumption (see lines 311, 422, 434). Either make this comparison robust across block sizes (e.g. insert enough rows that compression changes the leaf-page count by a wide margin regardless of BLCKSZ) or gate/document the BLCKSZ assumption as the surrounding tests do. The large URL-keyed build_compresses/insert_compresses checks (33% byte threshold over hundreds of pages) are far safer; this 151-row strict-inequality is the fragile one.
| SET enable_indexscan = off; | ||
| SET enable_indexonlyscan = off; | ||
| SET enable_seqscan = on; |
There was a problem hiding this comment.
enable_indexonlyscan is set off here but never reset. Its session default is true (costsize.c), and every other GUC this new section touches is RESET at the end of its block. Although btree_index.sql runs in its own session so this does not corrupt other regress files, leaving the session in a non-default state at end-of-file is inconsistent with the test's own cleanup discipline and is a maintenance footgun if blocks are reordered or added later. Add a matching RESET enable_indexonlyscan; once the seqscan-oracle comparison is done.
| SELECT (SELECT md5(string_agg(u, '|' ORDER BY u)) FROM btree_prefix_bu) = | ||
| md5(string_agg(x, '|' ORDER BY x)) AS bottomup_ok |
There was a problem hiding this comment.
This block's only assertion (bottomup_ok digest + a count) is a correctness-after-churn check: it passes whenever the index returns correct rows, regardless of whether the bottom-up deletion pass (_bt_bottomupdel_pass) actually fires on a coded page. _bt_bottomupdel_pass only runs when a leaf is about to split AND the UPDATE was flagged logically-unchanged, and removal is snapshot/timing-dependent (acknowledged in the comment). So this test does not reliably exercise the coded-page deletion code it claims to cover and would still pass if that path regressed or was skipped. Consider verifying the path is reached (e.g. via pageinspect on BTP_PREFIX leaves, or an index-size assertion that the pass actually reclaimed space), or soften the comment to state it is a correctness-under-churn check rather than a guaranteed exercise of bottom-up deletion.
| -- Confirm the index really is used (and is prefix-compressed), so the queries | ||
| -- below exercise the coded page format rather than silently falling back. | ||
| SET enable_seqscan = off; | ||
| SET enable_bitmapscan = off; | ||
| EXPLAIN (costs off) | ||
| SELECT u FROM btree_prefix WHERE u = 'https://example.com/path/to/resource/00012345'; |
There was a problem hiding this comment.
The comment claims this EXPLAIN confirms the index "is prefix-compressed", but EXPLAIN (costs off) output (Index Only Scan using btree_prefix_on) reveals nothing about the leaf page format (BTP_PREFIX). If prefix_compression=on silently fell back to plain pages, this EXPLAIN would be byte-for-byte identical and the test would still pass. In fact, every correctness check in this section (equality/range/backward/IOS/dedup digests, counts) exercises index correctness and would pass equally on an uncompressed index -- i.e. they survive the feature being reverted. The only assertions that can actually detect "compression did not happen" are the two BLCKSZ-dependent pg_relation_size ratio checks (_sz, _grp), and _grp is already flagged as borderline-nonportable. Either drop the "(and is prefix-compressed)" overclaim from the comment, or add a deterministic assertion that the page is really coded (the pageinspect/amcheck additions in this same PR expose the flag).
| FULL JOIN bt_page_items('test_plain_idx', 1) p ON p.itemoffset = c.itemoffset - 1 | ||
| WHERE c.itemoffset > 1 AND (c.data IS DISTINCT FROM p.data OR c.htid IS DISTINCT FROM p.htid); |
There was a problem hiding this comment.
Low confidence / low severity, test robustness: this is written as a FULL JOIN to compare both indexes, but the predicate c.itemoffset > 1 silently discards every plain-index row that has no matching coded row (for an unmatched plain row c.itemoffset is NULL, so the WHERE drops it). As a result the query only detects coded items that differ from or are missing in the plain index; it will NOT detect the case where the prefix index is missing data items that the plain index has. Consider also asserting the data-item counts match, e.g. SELECT count(*) FROM bt_page_items('test_prefix_idx',1) WHERE itemoffset>1 equals SELECT count(*) FROM bt_page_items('test_plain_idx',1), so a dropped/extra item on the prefix page cannot pass unnoticed.
| values[j++] = Int16GetDatum(offset); | ||
| values[j++] = ItemPointerGetDatum(&itup->t_tid); | ||
| values[j++] = Int16GetDatum(IndexTupleSize(itup)); | ||
| values[j++] = Int16GetDatum(ItemIdGetLength(id)); |
There was a problem hiding this comment.
This unconditionally changes itemlen from IndexTupleSize(itup) (the unaligned size stored in t_info) to ItemIdGetLength(id) (the MAXALIGN'd line-pointer length). For prefix pages that correctly reports the coded size, but it also changes the reported itemlen for every item on plain (non-BTP_PREFIX) pages: index_form_tuple() stores the unaligned size in t_info, so whenever a tuple's real size is not a multiple of MAXALIGN (e.g. a text key of odd byte length) the two values differ (e.g. 15 vs 16). This is a backward-incompatible change to long-standing bt_page_items output for ordinary indexes, and is unrelated to prefix compression. The existing regression value (itemlen | 16) does not catch it because that int tuple is already MAXALIGN'd. Restrict the on-page length to the decoded case and keep IndexTupleSize(itup) otherwise (confidence: high).
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz); | ||
| origprefix_array = true; |
There was a problem hiding this comment.
Memory leak on the prefix-compressed split path. _bt_prefix_decode_items() allocates ditems plus one CopyIndexTuple() per data item (potentially up to ~2 pages' worth of tuples), but _bt_split() never frees either the array or the decoded copies. Compare _bt_dedup_pass/_bt_bottomupdel_pass, which both free their ditems arrays and members. A statement doing many splits on a coded index (e.g. a large COPY or INSERT..SELECT) accumulates this leak until the surrounding memory context is reset. Free ditems[i] (skipping the nposting slot, which is freed by the caller) and ditems before returning from _bt_split.
| ItemId iid = NULL; | ||
| BTItemBuf firstbuf, |
There was a problem hiding this comment.
iid is now dead: it is initialized to NULL and immediately discarded with (void) iid;, with no remaining use after the switch to _bt_getitem(). Remove the declaration and the (void) iid; line rather than voiding an unused variable (pgsql-hackers routinely rejects this kind of leftover).
| (void) iid; | ||
| firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf); |
There was a problem hiding this comment.
Dead statement: (void) iid; only exists to suppress an unused-variable warning after iid lost its users. Drop it together with the iid declaration above.
| SET enable_indexscan = on; | ||
| RESET enable_seqscan; | ||
| RESET enable_bitmapscan; |
There was a problem hiding this comment.
State hygiene: these three lines mix SET enable_indexscan = on (restore to default via SET) with RESET for the siblings, and leave enable_indexonlyscan at the non-default off set on line 610. Use RESET enable_indexscan; and add RESET enable_indexonlyscan; so no enable_* GUC leaks past this block into the rest of btree_index.sql.
| SET enable_indexscan = on; | |
| RESET enable_seqscan; | |
| RESET enable_bitmapscan; | |
| RESET enable_indexscan; | |
| RESET enable_indexonlyscan; | |
| RESET enable_seqscan; | |
| RESET enable_bitmapscan; |
| SELECT pg_relation_size('btree_prefix_sz_on') * 3 < | ||
| pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses; |
There was a problem hiding this comment.
These relation-size comparisons are a portability/stability risk. The build/insert worthwhile threshold in the implementation (_bt_prefix_worthwhile: codedsz16 <= plainsz15) only guarantees a ~6% saving, while these assertions demand ~33% (on3 < off2). For this specific highly-compressible URL data the actual margin is large, so it will hold today, but the ratio depends on page geometry and is not normalized to block_size; under a non-default BLCKSZ (directory overhead per page and tuples-per-leaf change) the boolean could flip. The groups_compress check below is a bare strict < with no margin at all, making it the most fragile. Consider documenting the dependence on default BLCKSZ, or expressing the assertion so it is robust to block size.
| SELECT pg_relation_size('btree_prefix_sz_on') * 3 < | |
| pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses; | |
| SELECT pg_relation_size('btree_prefix_sz_on') * 3 < | |
| pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses; |
| -- Confirm the index really is used (and is prefix-compressed), so the queries | ||
| -- below exercise the coded page format rather than silently falling back. |
There was a problem hiding this comment.
The comment claims the EXPLAIN confirms the index "is prefix-compressed". EXPLAIN (costs off) only shows the chosen plan node and index name; it cannot show that any leaf is in BTP_PREFIX coded form. Nothing here would catch a silent fallback where the index is used but no leaf actually compressed. Tone the comment down to what it verifies (the index is used, not bypassed), or add a pageinspect/amcheck assertion on the coded leaf format if coded-ness must be proven.
| -- Confirm the index really is used (and is prefix-compressed), so the queries | |
| -- below exercise the coded page format rather than silently falling back. | |
| -- Confirm the index is used (not bypassed) so the queries below exercise the | |
| -- index access path rather than silently falling back to a seq scan. |
| SELECT pg_relation_size('btree_prefix_grp_on') < | ||
| pg_relation_size('btree_prefix_grp_off') AS groups_compress; |
There was a problem hiding this comment.
This groups_compress assertion is fragile across non-default BLCKSZ. The dataset is tiny (151 small tuples: one (1001,'a') plus 150 short URLs). pg_relation_size is quantized to whole pages, so the strict on < off comparison requires the plain index to occupy at least one more leaf page than the compressed one. With a larger BLCKSZ (16K/32K) both indexes may fit in a single leaf page, giving equal pg_relation_size and turning this into false -> test failure. Elsewhere this suite explicitly guards page-count assumptions (the dedup test notes "assumes BLCKSZ is 8192 or less"); this check has no such guard. Either add many more rows so the compressed index saves a full page under any supported BLCKSZ, or make the assertion BLCKSZ-robust (e.g. compare via pageinspect/amcheck that a leaf actually formed multiple groups) rather than relying on whole-page size granularity.
| CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on); | ||
| SET enable_seqscan = off; | ||
| SELECT length(u) FROM btree_prefix WHERE length(u) = 300; |
There was a problem hiding this comment.
This block does not verify what its comment claims. The comment asserts "A non-varlena key column is compressed too", but the only check is a scan returning one row for length(u) = 300, with no EXPLAIN to confirm the index is even chosen and nothing that confirms any leaf is in BTP_PREFIX form. The int4 expression values are overwhelmingly the constant 45 (all URL rows have length 45), so this index path is dominated by deduplication, not prefix compression. This query returns the same single row whether or not prefix compression is enabled -- i.e. it still passes with the feature reverted -- so it does not actually exercise or guard the non-varlena prefix-compression code. Either drop the overclaiming comment or strengthen the test to assert compression actually occurred for a non-varlena key.
48fd38b to
4b8e031
Compare
| values[j++] = Int16GetDatum(offset); | ||
| values[j++] = ItemPointerGetDatum(&itup->t_tid); | ||
| values[j++] = Int16GetDatum(IndexTupleSize(itup)); | ||
| values[j++] = Int16GetDatum(ItemIdGetLength(id)); |
There was a problem hiding this comment.
This changes itemlen for all pages, not just prefix leaves. The old value, IndexTupleSize(itup), is the unaligned tuple size (t_info & INDEX_SIZE_MASK); ItemIdGetLength(id) is the MAXALIGN'd length stored in the line pointer. For any index whose stored tuples are not already a multiple of MAXALIGN (very common for text/varlena keys, e.g. the data | 01 00 ... int8 case happens to be 16 only by luck), itemlen now reports a larger, MAXALIGN'd value for existing non-prefix indexes. That is an unjustified, user-visible change to an existing output column of bt_page_items.
The decode-only requirement (coded size instead of the decoded tuple's IndexTupleSize) is only needed on the BTP_PREFIX data-item path. Keep the historical behavior for everything else, e.g. report ItemIdGetLength(id) only when the item was decoded, and IndexTupleSize(itup) otherwise. Confidence: high.
| * - soft: finish the leaf once it is fillfactor-full as it stands; - | ||
| * hard: finish it if admitting itup could leave it unable to hold its | ||
| * tuples and high key, or (coded) beyond the split-fit budget (see | ||
| * _bt_prefix_split_needed), or out of line pointers. |
There was a problem hiding this comment.
The inline bullet list is garbled: "soft" and "hard" items are run together across the line wrap (... as it stands; - then * hard: ...), so the two cases read as one sentence. Break them onto separate lines so each bullet stands on its own, e.g.
-
- soft: finish the leaf once it is fillfactor-full as it stands;
-
- hard: finish it if admitting itup could leave it unable to hold its
- tuples and high key, or (coded) beyond the split-fit budget (see
- _bt_prefix_split_needed), or out of line pointers.
| /* lay out the final leaf of a prefix_compression index */ | ||
| if (state != NULL && state->btps_items != NULL) | ||
| _bt_flush_coded(wstate, state, true); |
There was a problem hiding this comment.
The pre-existing comment "Each iteration of this loop completes one more level of the tree." describes the for loop below, but the new prefix-flush block was inserted between the comment and the loop it documents, so the comment now reads as describing the flush. Move the flush block above this comment (or move the comment down to sit directly on the for) to keep the comment attached to the code it explains.
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz); | ||
| origprefix_array = true; |
There was a problem hiding this comment.
Memory leak on the heavily-compressed split path: _bt_prefix_decode_items() allocates the ditems array and, for every data item, a CopyIndexTuple() of its decoded form. In the origprefix_array branch none of this is ever freed before _bt_split returns (the end of the function only frees lefthighkey). Each leaf split on an oversized coded page leaks ~2 pages of decoded tuples plus the array. _bt_doinsert/_bt_split run in the caller's context, which for VACUUM/COPY/CREATE INDEX inserts can be long-lived, so this accumulates. Free the decoded items and the array before returning (taking care that ditems[origpagepostingoff] may have been overwritten with nposting, which is freed separately on the posting-split path).
(high confidence)
| ItemId iid = NULL; | ||
| BTItemBuf firstbuf, |
There was a problem hiding this comment.
Dead code / unnecessary churn. After switching to _bt_getitem(), the iid local is no longer used anywhere in _bt_set_startikey, and the (void) iid; cast only exists to silence the unused-variable warning that this leftover declaration itself causes. Drop both ItemId iid = NULL; and (void) iid; for a clean minimal diff.
(high confidence)
| ItemId iid = NULL; | |
| BTItemBuf firstbuf, | |
| BTItemBuf firstbuf, |
| SELECT pg_relation_size('btree_prefix_grp_on') < | ||
| pg_relation_size('btree_prefix_grp_off') AS groups_compress; |
There was a problem hiding this comment.
These size-ratio assertions (on < off*2/3) and groups_compress (on < off) encode fixed compression-effectiveness expectations. The per-tuple coding ratio is BLCKSZ-independent, and the URL keys share ~40 bytes so the margin looks comfortable for build_compresses/insert_compresses. The btree_prefix_grp case is the thin one: a single page with only 151 tuples where the on<off comparison has no explicit margin and the directory item plus per-group overhead eats into the gain -- if fillfactor/alignment shifts this on a non-default BLCKSZ the boolean could flip. Consider asserting a margin here too (e.g. on * 10 < off * 9) rather than a bare strict inequality. (low confidence)
| INSERT INTO btree_prefix_dd | ||
| SELECT 'https://example.com/path/to/resource/' || lpad((g % 200)::text, 8, '0') | ||
| FROM generate_series(1, 40000) g; |
There was a problem hiding this comment.
btree_index runs in a large parallel group (select_into ... btree_index hash_index update delete namespace prepared_xacts). This block adds ~75k inserted rows (20k+10k+5k+40k + bottom-up 4k with 3 full-table UPDATEs + size/group builds), a VACUUM, and several full index builds -- a noticeable bump to that group's wall time and temp space under the parallel schedule. Confirm each generate_series size is the minimum needed to force the targeted page path (split / dedup / bottom-up). In particular the 40000-row dedup set and the 20000-row size sets look larger than required to fill a single leaf and trigger the pass. (low confidence)
| DELETE FROM pfx_dup WHERE u < 'https://example.com/path/to/resource/00000150'; | ||
| VACUUM pfx_dup; | ||
|
|
||
| -- Non-key updates: bottom-up deletion on coded pages of a unique index. |
There was a problem hiding this comment.
The comment says "bottom-up deletion on coded pages of a unique index," but pfx_upd_u is a non-unique index built on u, which has only 500 distinct values across 20000 rows (g % 500) -- i.e. heavily duplicated, the opposite of unique. The only unique index here is the PRIMARY KEY on id, which is not prefix-compressed and is not what this scenario exercises. Fix the comment to describe the actual index (a non-unique, low-cardinality prefix-compressed index).
| -- Non-key updates: bottom-up deletion on coded pages of a unique index. | |
| -- Non-key updates: bottom-up deletion on coded pages of a low-cardinality index. |
| @@ -0,0 +1,123 @@ | |||
|
|
|||
| # Copyright (c) 2026, PostgreSQL Global Development Group | |||
There was a problem hiding this comment.
Spurious leading blank line before the copyright header. Every other recovery TAP test (e.g. 060_multixact_crashed_updater.pl) begins with the # Copyright ... line on line 1. Remove the leading blank line to match project convention and keep the diff clean.
| # Copyright (c) 2026, PostgreSQL Global Development Group | |
| # Copyright (c) 2026, PostgreSQL Global Development Group |
| SET enable_indexscan = on; | ||
| RESET enable_seqscan; | ||
| RESET enable_bitmapscan; |
There was a problem hiding this comment.
enable_indexonlyscan is turned off at line 610 but never restored (only enable_indexscan is re-enabled here, via SET ... = on rather than the file's usual RESET). Since enable_indexonlyscan defaults to true, the leak persists for the rest of the session and silently downgrades every later block that the author intends to run as an index-only scan to a plain index scan that fetches from the heap. In particular the bottomup_ok query (line 660), whose comment explicitly claims "an index-only scan of the key column", no longer exercises IOS decoding of coded leaves -- the feature path the test is meant to cover. The test still passes, which hides the gap. Reset the GUC (and prefer RESET for enable_indexscan to match the surrounding style).
| SET enable_indexscan = on; | |
| RESET enable_seqscan; | |
| RESET enable_bitmapscan; | |
| RESET enable_indexscan; | |
| RESET enable_indexonlyscan; | |
| RESET enable_seqscan; | |
| RESET enable_bitmapscan; |
| if (P_HASPREFIX(topaque)) | ||
| { | ||
| itup = _bt_getitem(state->target, offset, &state->itembuf); | ||
| tupsize = IndexTupleSize(itup); | ||
| } |
There was a problem hiding this comment.
Use-after-overwrite on prefix-compressed unique indexes. On a BTP_PREFIX leaf, _bt_getitem(..., &state->itembuf) returns a tuple that lives in state->itembuf and is only valid "until the workspace is reused" (see the BTItemBuf contract in nbtree.h). But bt_entry_unique_check() saves a pointer into this tuple in lVis.tid (lVis->tid = BTreeTupleGetHeapTID(itup) / BTreeTupleGetPostingN(itup, i)), and that pointer is retained across loop iterations to be dereferenced later by bt_report_duplicate() and the ItemPointerIsValid(lVis.tid) checks. On the next iteration this line re-decodes into the same itembuf, clobbering the bytes lVis.tid points at, so a cross-row uniqueness violation reports garbage heap TIDs (or reads freed/overwritten memory). Prefix compression is only gated by the prefix_compression reloption and is permitted on unique indexes, so this path is reachable. The added SQL test only exercises a non-unique prefix index, so it does not catch this. Fix by copying the lVis TID into stable storage (e.g. a dedicated ItemPointerData in BtreeLastVisibleEntry) rather than aliasing the decode workspace, and add a unique prefix-index case to the regression test.
| if (P_ISLEAF(opaque)) | ||
| firstitup = CopyIndexTuple(_bt_getitem(rightpage, *rightfirstoffset, | ||
| &state->nextbuf)); | ||
| else | ||
| firstitup = (IndexTuple) PageGetItem(rightpage, rightitem); |
There was a problem hiding this comment.
The comment "this relies on right page memory remaining allocated" is now only accurate for the internal (non-leaf) branch. The leaf branch copies the decoded tuple with CopyIndexTuple(), precisely because the decode workspace (nextbuf) and the right page are both transient; the resulting scankey no longer depends on right-page memory. Consider updating the comment to explain why the leaf path must copy (decoded tuple lives in a reusable BTItemBuf) while the internal path can point at page memory.
| CREATE INDEX prefix_bug_idx ON prefix_bug (v) WITH (prefix_compression = on); | ||
| SELECT bt_index_check('prefix_bug_idx', heapallindexed => true); | ||
| SELECT bt_index_parent_check('prefix_bug_idx', heapallindexed => true, rootdescend => true); |
There was a problem hiding this comment.
Test-coverage gap: this exercises only a non-unique prefix-compressed index. The uniqueness-checking path in verify_nbtree.c (gated on state->indexinfo->ii_Unique) is never reached here, yet that path is where prefix decoding interacts most dangerously with amcheck's cross-row state (lVis retains a pointer into the per-item BTItemBuf decode workspace). Add a UNIQUE prefix-compressed index case -- including adjacent equal keys / posting lists on the same leaf -- so bt_entry_unique_check()/bt_report_duplicate() run against decoded items. A healthy-index check will not surface a wrong-TID report, so also consider a corruption-injection or at least a duplicate-key scenario that drives the duplicate-reporting branch.
| Assert(P_HASPREFIX(oopaque)); | ||
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, | ||
| &dhikeysz); |
There was a problem hiding this comment.
The ditems array and the per-item CopyIndexTuple copies allocated by _bt_prefix_decode_items are never freed in this split-redo branch (only litems is pfreed at line 510). WAL replay (PerformWalRecovery/ApplyWalRecord) does NOT reset a memory context per record, so these allocations accumulate in the startup process's long-lived context for the entire duration of recovery. Each coded split replays one array plus one palloc per decoded data item (up to ~MaxIndexTuplesPerPage), which over millions of split records is an unbounded, data-proportional leak during crash/standby recovery. The comment below claiming "decoded items ... go with the redo memory context" is misleading since there is no such per-record reset. Free ditems (and ideally its element tuples) before returning, as is done for litems.
| (void) _bt_prefix_decode_items(page, &ditems, &dummyhikey, | ||
| &dummyhikeysz); |
There was a problem hiding this comment.
Same leak as the split path: ditems (array) and its per-item CopyIndexTuple copies from _bt_prefix_decode_items are never freed in the dedup-redo branch. Recovery does not reset a memory context per WAL record, so each replayed dedup of a coded page leaks one array plus one tuple copy per data item into the startup process's long-lived context. Free ditems and its elements before PageRestoreTempPage returns.
| SELECT pg_relation_size('btree_prefix_sz_on') * 3 < | ||
| pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses; |
There was a problem hiding this comment.
[moderate confidence] BLCKSZ assumption undocumented. The 3:2 size-ratio assertions (build_compresses/insert_compresses) depend on page packing, which varies with BLCKSZ, alignment, and 32/64-bit layout. The 40-byte shared prefix over 20000 keys gives a wide margin so this is likely robust, but the file's convention (lines 311, 422, 434) is to spell out BLCKSZ sensitivity explicitly. Add a one-line note that these size checks assume the default 8K BLCKSZ, consistent with the surrounding tests, so a buildfarm failure on a large-block animal is diagnosable.
| -- After the churn, an index-only scan of the live keys must equal the same | ||
| -- keys fetched from the heap by a sequential scan (an independent path): if | ||
| -- decoding the coded leaves were wrong, the two digests would differ. |
There was a problem hiding this comment.
[moderate confidence] Comment overstates what is exercised. The VACUUM btree_prefix at line 599 runs before the 5000-row re-insert at lines 600-602, so the freshly inserted rows are not all-visible when the digest is captured. The plan for those rows is therefore a plain index scan with heap fetches, not a pure index-only scan. The digest equality still holds (correctness is fine), but the comment "an index-only scan of the live keys" overstates the IOS coverage. Either VACUUM again after the re-insert to actually set the VM, or soften the comment.
| ItemId iid = PageGetItemId(page, minoff); | ||
|
|
||
| pstate.finaltup = (IndexTuple) PageGetItem(page, iid); | ||
| pstate.finaltup = _bt_getitem(page, minoff, &finalbuf); |
There was a problem hiding this comment.
This breaks the pstate->finaltup == tuple pointer-identity check in _bt_advance_array_keys() on prefix-compressed pages. For a backward SK_SEARCHARRAY scan, finaltup is decoded here into finalbuf. When the descending loop below reaches offnum == minoff, it computes itup = _bt_getitem(page, offnum, &itupbuf) (a different BTItemBuf), and that itup flows into _bt_checkkeys -> _bt_advance_array_keys, where line ~2709 does if (!all_required_satisfied && pstate->finaltup == tuple). On a plain page both _bt_getitem() calls return the same raw PageGetItem pointer, so the identity holds; on a BTP_PREFIX page they decode into distinct buffers, so finaltup == tuple is never true even when they are the same logical tuple. The primitive-scan boundary logic that relies on recognizing "caller's tuple is also the page's finaltup" then silently misbehaves (wrong primitive-scan continuation decisions for array-key backward scans on coded leaves). The identity check needs to compare by offset (e.g. pstate->offnum == minoff/a finaltup offset) rather than by pointer, or finaltup and the minoff item must share one decode buffer.
| -- After the churn, an index-only scan of the live keys must equal the same | ||
| -- keys fetched from the heap by a sequential scan (an independent path): if | ||
| -- decoding the coded leaves were wrong, the two digests would differ. | ||
| SET enable_indexonlyscan = on; |
There was a problem hiding this comment.
enable_indexonlyscan is turned on here (line 606) and off at line 610 but never restored to its default; enable_indexscan/enable_seqscan are likewise hand-set to assumed defaults (SET ... = on) at lines 611/615 instead of RESET. The rest of this file consistently pairs the set enable_* to false toggles with reset enable_* (lines 300-301, 390-392, 476-477). Harmless at end-of-session, but inconsistent and brittle if queries are later appended; prefer RESET to put the GUCs back to their real defaults.
| -- removable duplicates for the pass to find. (Bloat is quantified in the | ||
| -- performance benchmark, not here, since removability is timing-dependent.) | ||
| -- | ||
| CREATE TABLE btree_prefix_bu (id int PRIMARY KEY, u text COLLATE "C", filler text); |
There was a problem hiding this comment.
This block's only assertions are a digest round-trip (bottomup_ok) and a count, both of which pass regardless of whether a bottom-up deletion pass ever ran, whether the leaf was BTP_PREFIX-coded, or even whether prefix_compression had any effect (plain full-key leaves round-trip identically). So the block does not actually verify the stated "bottom-up index deletion on prefix-compressed pages" path was exercised; a regression that silently stopped coding these leaves would not be caught here. Consider adding a positive check that the coded path is active (e.g. a size/page-level assertion, or use amcheck/pageinspect as the other blocks and files do) so this test can fail when the feature is reverted. The same limitation applies to the runtime-dedup block below.
eeb0ab5 to
2dc53a9
Compare
| if (leafpage && | ||
| P_HASPREFIX(BTPageGetOpaque(page)) && | ||
| offset >= P_FIRSTDATAKEY(BTPageGetOpaque(page))) | ||
| itup = _bt_getitem(page, offset, &uargs->decodebuf); |
There was a problem hiding this comment.
Low confidence / non-blocking: this extends the raw-bytea path (bt_page_items(get_raw_page(...))) to run _bt_prefix_decode() on an attacker-controllable page image. _bt_prefix_decode() does pure byte-splicing with no bounds checks: a corrupt code->group indexes the anchor array out of bounds, and for a posting-list member a bogus NPosting makes plainsz = MAXALIGN(keysz + nhtids*sizeof(ItemPointerData)) overrun the fixed BLCKSZ decodebuf (the plainsz <= BLCKSZ guard is only an Assert, compiled out in production). This is consistent with pageinspect's existing superuser-only "trust the image" model (the surrounding code already reads posting metadata from raw input), so it is not a new risk class, but it is worth the author's awareness as a decode-specific overrun vector on the bytea entry point.
| INSERT INTO prefix_bug | ||
| SELECT 'https://example.com/path/to/resource/' || lpad(g::text, 8, '0') | ||
| FROM generate_series(1, 20000) g; |
There was a problem hiding this comment.
The prefix check inserts 40,000 rows and then runs both a heapallindexed bt_index_check and a bt_index_parent_check with rootdescend => true. rootdescend performs a full root-to-leaf descent per leaf tuple, making this by far the most expensive check in this file (the existing 100k-row checks on bttest_a/b do not use rootdescend). Confirm 20,000+20,000 is the minimum needed to force splits + dedup + prefix coding; if a smaller count still exercises multi-level splits and posting-list dedup, prefer it to keep make check/buildfarm time down. (low confidence on the exact threshold)
| if (P_HASPREFIX(topaque)) | ||
| { | ||
| itup = _bt_getitem(state->target, offset, &state->itembuf); | ||
| tupsize = IndexTupleSize(itup); | ||
| } |
There was a problem hiding this comment.
Dangling pointer into a reused decode workspace. On a prefix page itup now points into state->itembuf (line 1350), and bt_entry_unique_check() stores lVis->tid = BTreeTupleGetHeapTID(itup) / posting TID, i.e. a pointer into itembuf. lVis->tid outlives this iteration and is dereferenced in a later iteration (e.g. bt_report_duplicate() at line 875-876, reached via lines 1676-1679). But the next iteration re-decodes the current item into the same itembuf at line 1350, clobbering the bytes lVis->tid points at. The reported "last visible entry" heap TID in the duplicate-corruption message is then wrong/garbage for unique prefix-compressed indexes. The read-ahead path at lines 2087-2091 already uses CopyIndexTuple() for exactly this lifetime reason; the unique path must likewise copy the decoded tuple into storage that outlives the loop (e.g. copy into a dedicated buffer and point lVis->tid at that copy) before stashing a pointer into lVis.
| INSERT INTO prefix_bug VALUES ('zzz-no-shared-prefix'), (NULL); | ||
| CREATE INDEX prefix_bug_idx ON prefix_bug (v) WITH (prefix_compression = on); |
There was a problem hiding this comment.
The new prefix-compression test index is non-unique, so it never exercises bt_entry_unique_check() on a BTP_PREFIX leaf -- the code path where a decoded data item is passed into the unique check (and where lVis->tid ends up pointing into the reused decode workspace). Add a unique prefix-compressed index with duplicate keys (or at least WITH (prefix_compression = on) on a unique index) so the decode-plus-unique-check interaction is covered, not just the happy non-unique path.
| /* | ||
| * prototypes for functions in nbtprefix.c | ||
| */ | ||
| extern const char *_bt_page_get_prefix(Page page, Size *len); |
There was a problem hiding this comment.
Dead prototype: _bt_page_get_prefix is declared here but has no definition in nbtprefix.c (or anywhere else) and no caller in the tree. A declaration with no implementation is a bisectability/linker hazard if anything ever references it, and an exported prototype with no definition is dead "flexibility" that the community rejects on sight (YAGNI). Remove it, or add the definition and a caller if it is genuinely needed.
| SET enable_indexscan = off; | ||
| SET enable_indexonlyscan = off; | ||
| SET enable_seqscan = on; |
There was a problem hiding this comment.
enable_indexonlyscan is set off here and never reset. Every subsequent query in this file (the btree_prefix_bu and btree_prefix_dd blocks) runs with IOS disabled, so despite their comments explicitly claiming "an index-only scan of the key column" (line 657) and the dedup block's IOS claim, those scans are plain Index Scans, not index-only scans. That matters here: the IOS-only workspace-growth path in nbtreadpage.c (so->currTuples && !so->bigTuples, which allocates a bigger buffer because a prefix page can decode to more than one BLCKSZ) is reachable only from an index-only scan, so it is exercised only by the single digest at lines 607-608 and nowhere else. Either keep IOS enabled through the bu/dd blocks (and RESET it at the end alongside the other GUCs), or fix the comments to stop claiming IOS coverage they do not have. As written this also violates the "every SET has a matching RESET" discipline.
| SET enable_indexscan = on; | ||
| RESET enable_seqscan; | ||
| RESET enable_bitmapscan; |
There was a problem hiding this comment.
Inconsistent GUC restore: enable_indexscan is restored with SET ... = on while enable_seqscan/enable_bitmapscan use RESET. Use RESET enable_indexscan for symmetry and to restore the session default rather than pinning an explicit value. (enable_indexonlyscan, set off at line 610, is also missing from this cleanup.)
| -- A non-varlena key column is compressed too (shared bytes need not be text); | ||
| -- verify a scan over such an index returns correct rows. | ||
| CREATE INDEX btree_prefix_int ON btree_prefix ((length(u))) WITH (prefix_compression = on); |
There was a problem hiding this comment.
The comment claims a non-varlena key "is compressed too", but prefix sharing in nbtprefix.c (_bt_prefix_shared_len) compares raw stored bytes from the start of the data area. For an int4 key the datum is stored in native endianness, so on little-endian platforms (x86_64/ARM64 — the bulk of the buildfarm) consecutive small int values share no leading bytes (the varying low byte comes first), and essentially no compression occurs. Moreover, unlike the _sz/_grp blocks, this test asserts nothing about size — it only checks one row round-trips (which a plain btree does too). So the comment overstates what the test proves and is platform-dependent. Either assert the size reduction (and only where it actually holds), or reword the comment to drop the "is compressed" claim and describe what is really being verified (correct round-trip of a fixed-width key through the coded format).
| SELECT pg_relation_size('btree_prefix_grp_on') < | ||
| pg_relation_size('btree_prefix_grp_off') AS groups_compress; |
There was a problem hiding this comment.
This size assertion is block-size fragile. With only 151 small rows, both the compressed and uncompressed indexes collapse to a single leaf page under a larger BLCKSZ (e.g. 16KB/32KB buildfarm members), making pg_relation_size equal and this strict < return f while the expected output records t. The regression suite is expected to pass under non-default BLCKSZ (see stats.sql/temp.sql normalizing by current_setting('block_size')). The _sz block above avoids this by using 20000 rows and a ratio, which spans many pages at any block size. Either scale the row count so the uncompressed index exceeds one leaf page at the maximum supported BLCKSZ, or normalize/compare in a block-size-independent way.
| /* minoff is an offset to the lowest non-pivot tuple on the page */ | ||
| iid = PageGetItemId(pstate->page, pstate->minoff); | ||
| firsttup = (IndexTuple) PageGetItem(pstate->page, iid); | ||
| (void) iid; | ||
| firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf); |
There was a problem hiding this comment.
This (void) iid; exists only to silence the unused-variable warning for the now-removed iid local; remove it together with the iid declaration above.
| /* minoff is an offset to the lowest non-pivot tuple on the page */ | |
| iid = PageGetItemId(pstate->page, pstate->minoff); | |
| firsttup = (IndexTuple) PageGetItem(pstate->page, iid); | |
| (void) iid; | |
| firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf); | |
| /* minoff is an offset to the lowest non-pivot tuple on the page */ | |
| firsttup = _bt_getitem(pstate->page, pstate->minoff, &firstbuf); |
63423da to
361c211
Compare
Add a prefix_compression storage parameter, default off. A leaf page can store common leading bytes once in a directory and retain each member's remaining bytes. This can reduce index size when neighboring keys have long identical prefixes. Sharing compares stored bytes, not opclass equality, and decoding restores the original representation. Variable-length headers can prevent sharing even when text values have long common prefixes. Keep pivot tuples plain, retain ordinary heap TIDs and posting lists, and decode leaf keys through a shared accessor. The page flag selects the representation independently of the current reloption. Changing the option does not rewrite existing pages. Multiple groups and posting-list deduplication can coexist on a coded leaf. Account for directory and member representation limits as well as physical page space. Bound decoded occupancy so a coded leaf can split into two plain pages. Use the ordinary split picker when decoding fits on one page, and honor rightmost fillfactor among legal array split candidates otherwise. Split existing items and retry when a large incoming tuple leaves no legal boundary. Reconstruct coded split, deduplication, and posting-list insertion WAL records without consulting catalogs. The xl_btree_split flags field changes the WAL format and requires an XLOG_PAGE_MAGIC bump at integration. This patch leaves that bump to the committer. Validate coded metadata before copying decoded bytes. Extend amcheck and pageinspect to understand coded leaves, and copy the last visible heap TID when amcheck checks uniqueness across decoded tuples. Cover representation limits, build admission, corruption diagnostics, fillfactor selection, and standby replay in regression and TAP tests. Neither BTREE_VERSION nor BTPageOpaqueData changes. Older binaries cannot read coded pages; same-build pg_upgrade testing does not establish cross-major compatibility. Coding adds work to builds, insertion, and key-reading scans. No claim of zero default-off overhead or a general query speedup is made. Reviewed-by: Peter Geoghegan Reviewed-by: Yao Feng Discussion: https://postgr.es/m/at-01F1wsXLRH0qo@floki
| } | ||
|
|
||
|
|
||
| /* | ||
| * Loop over page items, starting from first non-highkey item, not high |
There was a problem hiding this comment.
Gratuitous double blank line (lines 1278-1279). This is diff churn unrelated to the prefix-decode change; pgindent will not collapse it. Drop the extra blank line to keep the diff minimal.
| } | |
| /* | |
| * Loop over page items, starting from first non-highkey item, not high | |
| } | |
| /* | |
| * Loop over page items, starting from first non-highkey item, not high |
| XLogBeginInsert(); | ||
|
|
||
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); |
There was a problem hiding this comment.
Gratuitous whitespace churn: a blank line added immediately after XLogBeginInsert(); inside an otherwise-unmodified WAL block. Remove it to keep the diff minimal.
| XLogBeginInsert(); | |
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); | |
| XLogBeginInsert(); | |
| XLogRegisterBuffer(0, buf, REGBUF_STANDARD); |
| { | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_INDEX_CORRUPTED), | ||
| errmsg("invalid prefix %s at offset %u", field, offnum))); |
There was a problem hiding this comment.
These ERRCODE_INDEX_CORRUPTED messages substitute an internal field name (field, e.g. "anchor offset", "residual extent") via %s and are not wrapped in _(). Per PostgreSQL conventions, never-translated diagnostics that interpolate internal identifiers should use errmsg_internal() so translators are not asked to translate a message whose content varies with an untranslatable field name. Consider errmsg_internal() here.
| errmsg("invalid prefix %s at offset %u", field, offnum))); | |
| errmsg_internal("invalid prefix %s at offset %u", field, offnum))); |
| /* | ||
| * _bt_scan_skip_key -- can _bt_readpage skip decoding the current item? | ||
| * |
There was a problem hiding this comment.
The new helper and its doc comment are inserted between the _bt_readpage doc comment (which ends with "Returns true if any matching items found on the page, false if none.") and the _bt_readpage function definition. This splits _bt_readpage's doc comment from the function it documents, making it read as if it describes _bt_scan_skip_key. Move the _bt_scan_skip_key definition and its comment above the _bt_readpage doc-comment block (or after the function), so each doc comment sits directly on top of the function it describes.
| if (!_bt_prefix_form_page(leftpage, left_hikey, litems, nlitems, | ||
| true)) | ||
| elog(PANIC, "failed to lay out prefix-compressed left page after split"); |
There was a problem hiding this comment.
Minor inconsistency in error-handling level: the new coded left-half path uses elog(PANIC, ...) while the adjacent non-coded path (and the pre-existing split redo code) uses elog(ERROR, ...) for the equivalent "can't happen in redo" PageAddItem/form-page failures. Both are promoted to a recovery failure, but mixing PANIC and ERROR for equivalent redo-invariant violations within the same function is gratuitously inconsistent. Consider using ERROR here to match the surrounding non-coded path (lines 494/503), or document why this specific failure warrants PANIC. (low confidence on which is "right"; the file already mixes both, but this new hunk introduces the divergence.)
| SELECT pg_relation_size('btree_prefix_sz_on') * 3 < | ||
| pg_relation_size('btree_prefix_sz_off') * 2 AS build_compresses; |
There was a problem hiding this comment.
These size-ratio assertions use a fixed 20000-row dataset that does not scale with block_size, while the compressed/uncompressed outputs are compared at a hardcoded 3:2 ratio. On non-default block sizes (e.g. 32KB pages) per-leaf packing, tuple-per-page counts and the directory/anchor overhead change, so the compressed:uncompressed ratio is not guaranteed to stay below 2/3 with margin the way it does at 8KB. The grouped case below correctly scales its row count by current_setting('block_size'), but these two do not. Either scale the dataset by block_size as well, or document/verify the 3:2 margin holds across all supported block sizes (1KB-32KB) to avoid buildfarm flakiness.
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz); | ||
| origprefix_array = true; |
There was a problem hiding this comment.
Memory leak: in the origprefix_array path ditems is allocated by _bt_prefix_decode_items, which CopyIndexTuples each of the (up to two pages') decoded items, and dhikey is a separately palloc'd copy. Neither dhikey, nor the per-element copies in ditems, nor the ditems array itself is ever freed: _bt_split has a single exit (return rbuf) with no pfree(ditems). _bt_split runs in the caller's (non-reset) context, so every split of a heavily-compressed page leaks a full decoded page's worth of tuples; under a bulk load this accumulates unbounded. dhikey should be pfree'd right here, and ditems/its elements should be freed before return rbuf.
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz); | |
| origprefix_array = true; | |
| (void) _bt_prefix_decode_items(origpage, &ditems, &dhikey, &dhikeysz); | |
| if (dhikey) | |
| pfree(dhikey); | |
| origprefix_array = true; |
| /* split's done */ | ||
| *newitemplaced = placenewitem; | ||
| return rbuf; |
There was a problem hiding this comment.
Companion to the leak above: before this single return, free the decoded-array workspace on the origprefix_array path, e.g. if (ditems) { for (int k = 0; k < maxoff - P_FIRSTDATAKEY(oopaque) + 1; k++) pfree(ditems[k]); pfree(ditems); }. Note ditems[origpagepostingoff - ...] may have been overwritten with nposting (freed by the caller's posting-split cleanup), so skip that element or free it before the overwrite to avoid a double free.
| EXTRA_INSTALL = src/test/modules/injection_points contrib/amcheck | ||
| EXTRA_INSTALL = src/test/modules/injection_points contrib/amcheck contrib/pageinspect | ||
|
|
||
| TAP_TESTS = 1 |
There was a problem hiding this comment.
This TAP_TESTS = 1 adds the new prefix tests (001_prefix.pl, 002_prefix_split.pl) to a module that is deliberately gated behind injection points: the parent src/test/modules/Makefile puts nbtree in SUBDIRS only when enable_injection_points=yes (otherwise ALWAYS_SUBDIRS), and this module's own Makefile makes check a no-op (@echo "injection points are disabled") in that case. The matching meson.build also does if not get_option('injection_points') / subdir_done().
The two new TAP tests do NOT use injection points (they only need amcheck + pageinspect). As placed, they will silently NOT run on the default build (injection points disabled), which is the common buildfarm/cfbot/developer configuration -- so the prefix-compression TAP coverage is gated behind an unrelated, non-default build option. Either host these TAP tests in a module that is not injection-point-gated (e.g. alongside the amcheck/recovery prefix tests already added), or restructure so the TAP tests run unconditionally while the injection-point-dependent regress/isolation tests remain gated.
| SELECT (SELECT md5(string_agg(u, '|' ORDER BY u)) | ||
| FROM btree_prefix WHERE u IS NOT NULL) = | ||
| md5(string_agg(x, '|' ORDER BY x)) AS ios_ok |
There was a problem hiding this comment.
Regression-guard weakness: this "oracle" check is meant to prove the coded-page decode path returns correct keys, but it does not force an index-only scan and does not EXPLAIN-verify the plan. At this point enable_indexscan is still on (only seqscan/bitmapscan are off). A plain Index Scan is therefore a legal plan, and in a plain Index Scan the returned u is fetched from the heap, not decoded from the coded index tuple. If the planner picks an Index Scan, this comparison passes trivially even with a broken index decoder -- i.e. the test would still pass with the decode path reverted. Add SET enable_indexscan = off; (or SET enable_indexonlyscan = on plus disabling index scan) before the digest and assert the plan with EXPLAIN (costs off) so an index-only scan is genuinely exercised. The same gap applies to the ios_digest capture below (line 607-608), which is also only guarded by seqscan/bitmap off with indexscan still enabled.
nbtree: opt-in leaf key prefix compression
Opened against this fork's
masterso CI runs before cfbot does. This branchcarries the version posted to pgsql-hackers (CF 7523, v9 and later): one
commit on top of postgres/postgres master.
v9 replaces the single-column, varlena-only design of v1-v8. The design
write-up is the thread message that carries v9; the commit message below
describes what the code does.
Commit message
Changes since v9 as posted (v10/v11/v12)
unaligned IndexTupleSize; the split and build estimate use MAXALIGN, so the
gate could undercount by up to 7B/item and over-admit). Count MAXALIGN'd
size. Added a StaticAssert tying BTREE_PREFIX_CODE_OVERHEAD to the header +
posting-pointer sizes, and a redo assert on the split left-item count.
Threading: the two decode scratch buffers in _bt_compare and
_bt_prefix_try_plain_page were function-local statics; made them a stack
local and a caller-owned buffer respectively (reentrancy for the
threaded-server work). No format change.
Feng); regression test groups_compress.