Repository navigation
Conversation
01237c9 to
3b56d29
Compare
c2a5026 to
d8fed9e
Compare
c999d4e to
d3a10f3
Compare
14db4ac to
bad9884
Compare
There was a problem hiding this comment.
🔍 OCR found 31 issue(s).
- 25 inline, 6 in summary (inline capped at 25)
📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L68)
sed -i.bak '...' is not portable: GNU sed treats -i.bak as in-place with a .bak backup suffix, but BSD/macOS sed requires a space (-i .bak) and otherwise interprets .bak as the script. The README advertises these scripts as Linux/FreeBSD-portable, but this invocation (and the matching mv "$hh.bak" "$hh") only works on GNU sed. On BSD sed this either errors or writes the backup to the wrong name, leaving the modified source in place -- compounding the in-place-edit footgun above.
📄 src/test/modules/test_locator_tableam/test_locator_tableam.c (L145-L150)
This chain-follow loop is modeled on heap_get_latest_tid(), but it drops that function's lower-bound offset check. The canonical loop guards with offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(page) precisely because it is chasing possibly-bogus t_ctid links. Here, if a ctid ever carried offset 0 (InvalidOffsetNumber), PageGetItemId(page, 0) indexes pd_linp[-1], an out-of-bounds read. In a well-formed heap chain this won't occur, but since this code exists to harden the no-retain contract against exactly such edge cases, mirror the canonical bound check.
💡 Suggested change
Before:
if (ItemPointerGetOffsetNumber(&cur) <= PageGetMaxOffsetNumber(page))
{
lp = PageGetItemId(page, ItemPointerGetOffsetNumber(&cur));
if (ItemIdIsNormal(lp))
htup = (HeapTupleHeader) PageGetItem(page, lp);
}
After:
if (ItemPointerGetOffsetNumber(&cur) >= FirstOffsetNumber &&
ItemPointerGetOffsetNumber(&cur) <= PageGetMaxOffsetNumber(page))
{
lp = PageGetItemId(page, ItemPointerGetOffsetNumber(&cur));
if (ItemIdIsNormal(lp))
htup = (HeapTupleHeader) PageGetItem(page, lp);
}
📄 src/test/regress/pg_regress.c (L1246-L1246)
This execl -> execlp change is unrelated to the feature this patch implements and is effectively a no-op. shellprog is always SHELLPROG, which every build system defines as an absolute path ($(SHELL) i.e. /bin/sh for the makefiles, and a hardcoded /bin/sh in meson). When the first argument contains a /, execlp does no PATH search and behaves identically to execl, so this buys nothing while adding an unrelated PATH-lookup semantic. Unrelated churn like this is a top rejection reason on pgsql-hackers; drop it from this patch (or split it into its own justified commit).
💡 Suggested change
Before:
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
After:
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
📄 src/test/regress/sql/tsearch.sql (L763-L764)
The comment names the wrong function: tsvector_update_trigger() sets the column via heap_modify_tuple_by_cols() (see tsvector_op.c), not heap_modify_tuple(). Per the "comments must describe what the code does now" rule, correct the identifier (or drop the implementation-specific reference) so it does not drift from the actual code.
💡 Suggested change
Before:
-- tsvector_update_trigger() uses heap_modify_tuple() to set column 'a'
-- without going through the executor's SET-clause tracking.
After:
-- tsvector_update_trigger() uses heap_modify_tuple_by_cols() to set column 'a'
-- without going through the executor's SET-clause tracking.
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L486-L486)
This test calls CREATE EXTENSION amcheck, but src/test/subscription/Makefile does not list contrib/amcheck in EXTRA_INSTALL (it only installs contrib/hstore and injection_points). Under the autoconf make check path (prove_check), the temporary install tree therefore won't contain amcheck, and this statement fails with "could not open extension control file" — a hard failure, not a clean skip. The companion src/test/recovery/Makefile was correctly updated to add contrib/amcheck; the subscription Makefile needs the same addition for 039/040.
📄 src/test/subscription/t/040_hot_indexed_replica_identity.pl (L115-L115)
CREATE EXTENSION amcheck will fail under the autoconf make check path: src/test/subscription/Makefile's EXTRA_INSTALL does not include contrib/amcheck, so the module is absent from the test install tree and this is a hard error rather than a clean skip. Add contrib/amcheck to the subscription Makefile's EXTRA_INSTALL (as src/test/recovery/Makefile already does for 061/062). Also note the minor inconsistency with 039, which uses CREATE EXTENSION IF NOT EXISTS amcheck; align the two for consistency.
| SET enable_indexscan = off; | ||
| SET enable_bitmapscan = on; | ||
| -- Must return the row. | ||
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; |
There was a problem hiding this comment.
This test claims to guard the BitmapAnd-over-a-split-block path (bloom bitmap intersected with the btree bitmap, where VISIBILITYMAP_LOCATOR_SPLIT forces a union+recheck instead of an intersection), but it only asserts the result count. Nothing forces the planner to actually choose a BitmapAnd of both indexes. On a one-row table, the planner may pick a single Bitmap Index Scan on one index and apply the other qual as a heap filter/recheck, in which case the BitmapAnd-with-split-block code is never exercised and the test passes as a no-op — it would not catch a regression in the very code it claims to protect. The parallel cases in contrib/pageinspect/sql/hot_indexed_updates.sql pin the plan shape with EXPLAIN (COSTS OFF) for exactly this reason; this test should do the same to verify a BitmapAnd node is produced. (moderate confidence)
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; | |
| EXPLAIN (COSTS OFF) SELECT count(*) FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; | |
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This shebang change is unrelated to the HOT-indexed-updates work and is the only file in the entire tree using #!/usr/bin/env perl. Every other perl script under src/, contrib/, and doc/ uses #!/usr/bin/perl (or #! /usr/bin/perl), so this deviates from a consistent, deliberate project convention. It also changes which perl is resolved at runtime (PATH-dependent instead of fixed), and is exactly the kind of unrelated churn that gets a patch rejected on -hackers and creates a needless merge conflict with upstream. Revert this to keep the diff minimal.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
This entire file is personal developer tooling (clangd editor config with a contributor-specific CompilationDatabase path and hardcoded -I../../../../src/include / -I. include paths) that does not belong in a patch destined for pgsql-hackers/commitfest. It pollutes the shared source tree, is environment-specific, and violates minimal-diff discipline. Keep it in your local environment or add it to a personal ignore file; do not commit it as part of the feature patch.
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
This file is personal developer tooling and should not be committed to the repository. Worse, it hard-codes source line numbers as breakpoints (e.g. heapam.c:3210, heapam.c:4019, heapam.c:4033, heapam.c:4101, heapam.c:4147). Line-number breakpoints are extremely brittle: the companion diffs in this change set move heap_update and the surrounding HOT code by hundreds of lines (per heapam.c, heap_update now begins near line 3298, not 3210), so these breakpoints will silently land on unrelated code. Any future edit to these files shifts them again. If GDB helpers are wanted at all, break on function names, not line numbers -- but this file should not be in the tree.
| # Redirect-with-data: initialize bitmap accumulator from collapsed redirect | ||
| break heapam_indexscan.c:182 |
There was a problem hiding this comment.
More hard-coded line-number breakpoints (heapam_indexscan.c:182/250/297, indexam.c:299, execIndexing.c:370, pruneheap.c:1802/1836/1863/1287/2936). The diffs to these very files in this change set relocate all of this code, so each of these breakpoints will resolve to the wrong line. Line-number breakpoints committed alongside the code they target are a guaranteed maintenance hazard.
| if (scandesc->xs_entry_needs_recheck) | ||
| continue; |
There was a problem hiding this comment.
The continue path that skips stale entries has no CHECK_FOR_INTERRUPTS(). The two sibling loops that implement this exact HOT-indexed skip idiom both add one in the skip branch: nodeIndexscan.c (ExecIndexScan) and repack.c's identity probe. A long run of stale index entries for the searched key (a row whose key was cycled through many HOT-indexed updates) can be skipped here, and btree only checks for interrupts at page boundaries, so many matches on one leaf page would spin without a cancellation point. For consistency with the other probe sites and robustness, add a CHECK_FOR_INTERRUPTS() to the skip branch. (low confidence)
| if (scandesc->xs_entry_needs_recheck) | |
| continue; | |
| if (scandesc->xs_entry_needs_recheck) | |
| { | |
| CHECK_FOR_INTERRUPTS(); | |
| continue; | |
| } |
| * context; the caller owns it and must bms_free when done. We never hand | ||
| * out a borrowed pointer to the cached copy because relcache invalidation | ||
| * can rebuild rd_indexcxt in place even while a refcount is held. |
There was a problem hiding this comment.
The stated justification is inaccurate. The in-place index reload path (RelationReloadIndexInfo, used by RelationRebuildRelation when rd_indexcxt != NULL) explicitly preserves rd_indexcxt and never rebuilds it; the array fields (indkey/indexprs/indpred) that rd_indattr derives from cannot change there. The full-rebuild path destroys and swaps the whole entry rather than rebuilding rd_indexcxt "in place". So the real reason to always hand back a copy is simply that the cached bitmap is relcache-owned (freed with rd_indexcxt on full rebuild), not that the context is rebuilt in place under a held refcount. Please reword to avoid asserting behavior the code does not have (comment-accuracy, low confidence on wording, high confidence the "in place" claim is wrong).
| BENCH=${BENCH:-/scratch/siu-bench} | ||
| JOBS=${JOBS:-$( (command -v nproc >/dev/null && nproc) || sysctl -n hw.ncpu 2>/dev/null || echo 8 )} | ||
| if [ -z "${REPO:-}" ]; then | ||
| for candidate in "$HOME/ws/postgres/tepid" "$BENCH/repo" /scratch/pg; do |
There was a problem hiding this comment.
This whole directory (src/test/benchmarks/siu/) is developer benchmarking tooling, not a regression/TAP/isolation suite. It is not referenced by any Makefile or meson.build (confirmed: no benchmarks reference exists in the build system), will never run in the buildfarm, and hardcodes host-specific absolute paths (/scratch/siu-bench) and a private developer branch name (tepid, $HOME/ws/postgres/tepid). For a pgsql-hackers patch this is out-of-scope churn bundled into the feature: a patch should contain only what the change requires, and the tree has no precedent src/test/benchmarks framework to slot into. Recommend splitting these scripts out of the feature patch entirely (keep them in the -hackers thread as a reproducible-benchmark attachment instead). This applies equally to run.sh, soak.sh, split_ab.sh, bloat.sh, and the .sql workloads in this directory.
| if [ "$off" = 1 ]; then | ||
| # Force heap to advertise no split signal: BitmapAnd then always | ||
| # intersects exactly, as it did before the feature. | ||
| sed -i.bak 's/\.bitmap_and_inexact = heap_bitmap_and_inexact,/.bitmap_and_inexact = NULL,\/* split_ab: split-off *\//' \ | ||
| "$hh" |
There was a problem hiding this comment.
This does an in-place sed -i.bak edit of a tracked source file (heapam_handler.c) and relies solely on an EXIT trap to restore it. The sed target .bitmap_and_inexact = heap_bitmap_and_inexact, does match the real registration (heapam_handler.c:2955, confirmed), so the edit works -- but the restore is fragile: a SIGKILL/power loss, or any failure after git checkout --detach but before the trap's mv -f $hh.bak $hh runs, leaves the user's working tree with a silently-modified committed source file (and the build divergence is invisible unless they git status). Mutating tracked source in-place to produce a benchmark variant is a footgun against the user's own checkout. Prefer building split_off from a throwaway git worktree/git stash-applied patch or a compile-time -D guard rather than editing the file in place. (Part of the broader scoping concern: these scripts should not be in the feature patch at all.)
| -- bitmap_and_mixed: quantifies the cost of the per-block | ||
| -- VISIBILITYMAP_LOCATOR_SPLIT BitmapAnd signal specifically, not the SIU | ||
| -- write-path win the other workloads measure. |
There was a problem hiding this comment.
This leading comment is a dense design narrative (feature-internals justification) rather than a description of what the query does, and it carries developer-branch framing ("the split fix", "reset_state's freshly-clustered layout"). In a mailing-list patch this reads as noise. The technical terms it asserts (VISIBILITYMAP_LOCATOR_SPLIT, crossed-attribute bitmap) do match the real feature code, so it is accurate -- but it should be trimmed to a short statement of the workload's shape. Lower priority than the scoping/footgun issues; if the scripts are split out of the patch this is moot.
There was a problem hiding this comment.
🔍 OCR found 29 issue(s).
- 25 inline, 4 in summary (inline capped at 25)
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L299-L302)
The ALTER-to-off reread test has no explicit synchronization that the apply worker has processed the catalog invalidation before the next batch is applied. The worker learns of the new mode via subscription_change_cb -> maybe_reread_subscription(), driven by invalidation processing between transactions; wait_for_catchup inside apply_updates_and_read only guarantees LSN catch-up, not that the worker observed the ALTER's syscache inval before applying the 81..100 batch. If the first post-ALTER transaction is applied before the invalidation is picked up, $post_alter_hotidx could be > 0 and this assertion would flake. Consider polling pg_stat_subscription/forcing a worker restart, or otherwise confirming the mode was re-read before measuring.
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L492-L494)
This test ends with done_testing() but, unlike 040, never calls $subscriber->stop/$publisher->stop. Teardown works via PostgreSQL::Test::Cluster's END-block auto-stop, so this is not a correctness problem, but the inconsistency with 040 is a minor hygiene nit; consider stopping both nodes explicitly for consistency.
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L193-L198)
Minor performance/maintainability nit: this fires one UPDATE per id through a separate safe_psql (hence a separate psql process/connection) for every row in the range. For a 40-wide range (case 3) this spawns 40 psql invocations per batch, and the helper is called many times across the three modes. The per-tuple pgstat counters (n_tup_upd, n_tup_hot_upd, n_tup_hot_indexed_upd) are aggregate, so a single ranged statement UPDATE $table SET payload = payload + 1 WHERE id BETWEEN $id_lo AND $id_hi produces identical counter deltas while being far cheaper.
💡 Suggested change
Before:
for my $i ($id_lo .. $id_hi)
{
$publisher->safe_psql('postgres',
"UPDATE $table SET payload = payload + 1 WHERE id = $i");
}
$publisher->wait_for_catchup($sub_name);
After:
$publisher->safe_psql('postgres',
"UPDATE $table SET payload = payload + 1 WHERE id BETWEEN $id_lo AND $id_hi");
$publisher->wait_for_catchup($sub_name);
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L2-L2)
Style nit: the file begins with a blank line before the copyright header. The PostgreSQL convention (as followed by sibling file 040) is for the # Copyright ... comment to be the first line of the TAP script. Drop the leading blank line for consistency.
💡 Suggested change
Before:
# Copyright (c) 2026, PostgreSQL Global Development Group
After:
# Copyright (c) 2026, PostgreSQL Global Development Group
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
Personal developer-environment scaffolding does not belong in a PostgreSQL patch destined for pgsql-hackers. .clangd, .gdbinit, flake.nix, shell.nix, pg-aliases.sh, and glibc-no-fortify-warning.patch are being committed into the repository root (they are not listed in any .gitignore). This violates the minimal-diff / one-logical-change discipline and will draw immediate rejection. These artifacts belong in a developer's local tree (e.g. a global gitignore or a personal untracked config), not in the committed patch. Drop them from the change set. [high confidence]
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
Besides being out-of-scope developer tooling, this file hardcodes absolute source line numbers (e.g. heapam.c:3210, heapam.c:4019, heapam.c:4033, pruneheap.c:1802, indexam.c:299). These break the moment any line above them shifts and are meaningless outside the author's exact tree, so they cannot be maintained upstream. Remove this file from the patch. [high confidence]
| @@ -1243,7 +1243,7 @@ spawn_process(const char *cmdline) | |||
| char *cmdline2; | |||
|
|
|||
| cmdline2 = psprintf("exec %s", cmdline); | |||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |||
There was a problem hiding this comment.
Changing execl to execlp alters lookup semantics: execlp performs a PATH search when shellprog is not an absolute path. shellprog = SHELLPROG is compiled from $(SHELL), which is normally an absolute path, so this is a no-op today; but if SHELLPROG is ever a bare name, the regression harness would resolve the shell from PATH, which is a behavior and security posture change (picking up an unintended shell binary). This change is unrelated to the feature this PR delivers and lacks any justification. Split it out or revert it. [moderate confidence]
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
The shebang change from #!/usr/bin/perl to #!/usr/bin/env perl is an unrelated, behavior-changing tweak to a committer-facing tool: it picks the first perl on PATH instead of the system interpreter. It is also inconsistent with the rest of the tree, which uses #!/usr/bin/perl (e.g. src/tools/copyright.pl, src/tools/git_changelog, src/tools/mark_pgdllimport.pl). Drop this hunk or submit it as a separate, justified patch. [high confidence]
| alias pg-perf='perf top -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1)' | ||
| alias pg-htop='htop -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | tr "\n" "," | sed "s/,$//")' | ||
|
|
||
| pg-stats() { |
There was a problem hiding this comment.
Beyond being out-of-scope tooling: pg-enable-cores/pg-disable-cores mutate the system-wide sysctl kernel.core_pattern via sudo tee, affecting every process on the host, not just this dev tree. This is a footgun with machine-global side effects. Likewise pg-test-one hardcodes --port=40099, which can collide with concurrent runs. These confirm the file is tied to the author's machine and unsuitable for the committed tree. [low confidence]
| * with 2-bit _vm forks that are not rewritten. The committer re-picks both | ||
| * values at commit time, along with the copy in t/010_vm_conversion.pl. | ||
| */ | ||
| #define VISIBILITYMAP_WIDTH_CHANGE_CAT_VER 202610201 |
There was a problem hiding this comment.
The value 202610201 is duplicated verbatim in t/010_vm_conversion.pl ($width_change_cat_ver) and the committer must also keep it equal to the final CATALOG_VERSION_NO installed by the VM-widening commit. If any of these three drift, a cluster initdb'd in the window gets its 2-bit _vm forks silently left unconverted (data-correctness hazard, not a hard error). This is a genuine foot-gun, so it is worth adding a hard cross-check: in a backend build the constant could be StaticAssert'd against CATALOG_VERSION_NO, and 010_vm_conversion.pl could read the value from the installed header rather than re-hardcoding it, to prevent the two from diverging unnoticed. (Confidence: moderate; the risk is real but the comment already flags it as committer-owned.)
| TEPID_REV=${TEPID_REV:-tepid} | ||
| MASTER_REV=${MASTER_REV:-$(git merge-base "$TEPID_REV" origin/master 2>/dev/null || git merge-base "$TEPID_REV" master)} |
There was a problem hiding this comment.
These benchmark scripts are hardwired to a private development branch and a developer's personal paths: build.sh defaults REPO to $HOME/ws/postgres/tepid and /scratch/pg, TEPID_REV to tepid, and bakes -siubench/tepid into the build; split_ab.sh and the README do the same. Nothing in the tree (no Makefile/meson rule) references src/test/benchmarks, so this is a brand-new directory of unhooked, machine-specific tooling. For a patch destined for pgsql-hackers this fails the minimality/patch-hygiene bar: a committed benchmark should be self-contained and reproducible for anyone, not reference a committer's home directory or an unpublished branch name. Recommend dropping this harness from the patch (keep it on the thread as an attachment) or rewriting it to be branch/path agnostic.
| echo "=== building $name ($rev) into $prefix" | ||
| rm -rf "$prefix" | ||
| mkdir -p "$prefix" | ||
| git checkout --quiet --detach "$rev" |
There was a problem hiding this comment.
build_variant runs git checkout --quiet --detach "$rev" on the developer's live working tree and relies solely on an EXIT trap to restore the original HEAD. A hard kill (SIGKILL/OOM/power loss) during the long meson compile leaves the user's repo on a detached HEAD at MASTER/TEPID with no restore -- the same working-tree-mutation footgun already flagged for split_ab.sh's sed -i.bak. Prefer git worktree add into $BENCH (or git archive | tar -x) so the checkout under benchmark never touches the user's working copy.
| -o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null | ||
| sleep 2 | ||
| } |
There was a problem hiding this comment.
start_pg waits for the server with a bare sleep 2 rather than polling readiness. On a slow/loaded host initdb+start can take longer than 2s, so the immediately-following setup_schemas/psql_as can hit "the database system is starting up" and fail; on a fast host it needlessly wastes time. pg_ctl already supports -w (wait for ready) -- use pg_ctl ... -w start and drop the sleep. Same pattern in soak.sh and split_ab.sh.
| local t=0 | ||
| while [ "$t" -lt "$DURATION" ]; do | ||
| sleep "$SAMPLE" | ||
| t=$((t + SAMPLE)) |
There was a problem hiding this comment.
The sampler loop advances t in fixed $SAMPLE steps but pgbench runs for exactly $DURATION seconds. When $DURATION is not an integer multiple of $SAMPLE (e.g. the default 900/60 happens to divide, but any override need not), the final tick either overshoots past pgbench exit (sampling a quiesced server, so the last row reports a misleadingly low write rate) or truncates the last window. Either compute the number of ticks as DURATION/SAMPLE and guard the final partial interval, or break the loop as soon as pgbench ($pgb) has exited.
There was a problem hiding this comment.
🔍 OCR found 31 issue(s).
- 25 inline, 6 in summary (inline capped at 25)
📄 src/test/benchmarks/siu/scripts/split_ab.sh (L134-L135)
grep -oP (Perl regex, \K) is GNU-grep-specific and absent from BSD/macOS grep, where -P is unsupported. On those platforms both extractions return empty, tps/lat are blank, and every iteration takes the "FAILED (no tps/lat)" path, silently producing an empty summary. The sibling run.sh extracts the same pgbench fields with portable awk; use the same approach here for consistency and portability. (high confidence)
💡 Suggested change
Before:
tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1)
lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1)
After:
tps=$(awk '/tps = /{print $3}' "$log" | tail -1)
lat=$(awk '/latency average = /{print $4}' "$log" | tail -1)
📄 src/test/benchmarks/siu/scripts/run.sh (L36-L37)
Doc/code drift and a latent failure: the header documents the default as "0,1,4,8,WIDE_COLS", but the code hardcodes 16. If a user overrides WIDE_COLS (e.g. to 8) without also overriding WIDE_STEPS, the wide_16 step calls build_wide_set_clause 16, producing c9=:v,...,c16=:v for columns that seed_wide_table never created, and pgbench aborts the run. Either derive the default from WIDE_COLS (as the comment claims) or validate each step is <= WIDE_COLS. (moderate confidence)
💡 Suggested change
Before:
WIDE_COLS=${WIDE_COLS:-16}
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,16}
After:
WIDE_COLS=${WIDE_COLS:-16}
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,$WIDE_COLS}
📄 src/test/modules/test_locator_tableam/sql/noretain.sql (L186-L186)
These three pg_ls_tmpdir() checks assume the server's temp directory reflects only this session's spill files. pg_ls_tmpdir() lists every temp file in the current tablespace's pgsql_tmp, including those created concurrently by other backends (e.g. an autovacuum worker that spills), and autovacuum is on by default even in the pg_regress test instance. That makes two of these checks fragile:
- Line 186 sizes
temp_file_limitascurrent temp-dir size + 40KB. An unrelated temp file present at that instant inflates the limit, so the subsequentDOblock may not hitconfiguration_limit_exceeded, and the expectedNOTICE: spill failedwould not fire. - Line 210 compares two successive directory totals; if another backend's temp file appears or disappears between the two measurements,
releasedcan flip.
The count(*) > 0 check at line 154 is robust, but the size-based ones are not deterministic. Consider restricting to this backend's own temp files (e.g. filtering by the backend PID prefix in the temp file name) or asserting the property in a way that does not depend on a shared directory snapshot.
📄 src/test/regress/regress.c (L1761-L1764)
result_names[] is a hand-maintained parallel array to the TM_Result enum (tableam.h) with no compile-time link. The runtime check EXPECT_TRUE(result >= TM_Ok && result <= TM_WouldBlock) validates the enum range but silently assumes this array has exactly TM_WouldBlock + 1 entries and in matching order. If TM_Result ever gains a value, this array goes stale with no build failure: a value inserted before TM_WouldBlock yields a wrong name, and one appended after it indexes out of bounds past the range guard. Add a StaticAssertStmt(lengthof(result_names) == TM_WouldBlock + 1, ...) to tie the two together.
📄 src/test/regress/regress.c (L1767-L1767)
cid = GetCurrentCommandId(false) - PG_GETARG_INT32(2) can underflow: CommandId is unsigned, so a cid_delta larger than the current command counter (or a negative int32) wraps to a huge value. The in-tree callers only pass 0 and 1 so this is safe today, but since the function is a general exported test helper with no validation it is an easy footgun for future callers. Consider rejecting a delta that exceeds the current command id, or at least documenting the precondition in the header comment.
📄 src/test/recovery/t/061_hot_indexed_vm.pl (L98-L102)
This comment describes the existing table (the else branch), but it sits directly above the if ($fpw eq 'on' && $update eq $updates[0]) block. Since UPDATE fresh is logged before UPDATE existing, $updates[0] is the fresh table's record, which is what the if branch asserts carries the applied image. A reader naturally attaches this comment to the if case it precedes, making it read as if existing is $updates[0]. Move the explanation into the else branch (or reword to name fresh as the image-carrying case) so it matches the code it governs.
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
These hard-coded source line numbers do not match the current tree and will mislead anyone who sources this file. Verified against the actual sources:
heapam.c:3210is insideheap_delete's toast handling (relkind != RELKIND_MATVIEW), not theheap_updateentry.heapam.c:4019/4024/4033point at a comment, a blank line, andXLogRegisterBuffer()in the tuple-lock path -- not the "pure HOT / HOT-indexed / size check" decisions.heapam_indexscan.c:182isIndexScanHeapData *hscan = ...inheapam_index_scan_end;:250is*all_dead = first_call;;:297is abreakfor a bogus TID -- none are the described bitmap-accumulation points.pruneheap.c:1287is ado_prune = ...assignment, not the "execute phase".
Line-number breakpoints are stale on the very first code edit and cannot be reviewed. This is personal debugging scaffolding that should not be committed to a tree destined for pgsql-hackers/commitfest; keep it out of the series (local gitignore).
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
Unrelated change that is inconsistent with the tree-wide convention. Every other Perl script in the repository (30+ scripts, incl. all of src/tools/*.pl, genbki.pl, gen_node_support.pl, etc.) uses #!/usr/bin/perl; after this change pgindent is the only script using #!/usr/bin/env perl. This is a committer-owned tooling script and the shebang change is not required by the HOT-indexed-updates feature -- it should not ride along with a feature patch and should be dropped to keep the diff minimal.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| pg-enable-cores() { | ||
| ulimit -c unlimited | ||
| if ! [ -w /proc/sys/kernel/core_pattern ]; then | ||
| echo "Setting kernel.core_pattern (requires sudo)..." | ||
| echo "core.%p" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || { |
There was a problem hiding this comment.
pg-enable-cores/pg-disable-cores mutate the system-wide kernel.core_pattern sysctl via sudo, a machine-global side effect that is surprising to ship in a repository helper script. Combined with the many unquoted/edge-case constructs elsewhere (e.g. basename $CC, pkill -f "postgres.*-D.*$PG_DATA_DIR", find ... -mtime +$days -delete), this is personal developer tooling rather than project infrastructure and should not be committed to a tree intended for pgsql-hackers submission.
| # into system glibc or release builds. | ||
| patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: { | ||
| patches = (oldAttrs.patches or []) ++ [ | ||
| ./glibc-no-fortify-warning.patch |
There was a problem hiding this comment.
This devShell patches glibc with ./glibc-no-fortify-warning.patch. The patch file does exist at the repo root, but it is not part of this change set / review group -- make sure it is committed together with these files, otherwise flake evaluation of every glibc devShell (default, gcc, clang) fails with a missing-path error. More broadly, this entire Nix dev environment is personal tooling unrelated to the HOT-indexed-updates feature and will draw an immediate "patch does more than one thing" rejection on pgsql-hackers; keep it out of the submitted series.
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath = true; is a NixOS module option, but here it is placed inside a flake-utils.lib.eachDefaultSystem per-system output set. At the flake-output level this attribute is never interpreted as NixOS configuration; it is a stray/dead attribute that does nothing. Either drop it or move it into an actual NixOS module. (moderate confidence)
| *) | ||
| echo "reset_state: unknown table $table" >&2 | ||
| return 1 | ||
| ;; |
There was a problem hiding this comment.
reset_state returns 1 on an unknown table, but every call site invokes it bare under set -e, so a genuine mismatch aborts the whole run rather than being handled - while the surrounding pgbench steps are wrapped in || true and set +e/set -e toggles, making the error-handling policy inconsistent (some failures abort, some are swallowed). A failed reseed between workloads would carry bloat/index-size state forward and silently skew the per-workload deltas the CSV reports. Consider making reset failures consistently fatal and verifying the reseed succeeded before measuring.
|
|
||
| start_pg() { | ||
| local v=$1 datadir=$BENCH/_data_$v | ||
| [ -d "$datadir" ] && find "$datadir" -mindepth 1 -delete && rmdir "$datadir" |
There was a problem hiding this comment.
Same destructive-deletion footgun as run.sh: find "$datadir" -mindepth 1 -delete && rmdir with datadir=$BENCH/_data_$v. An empty exported BENCH passes set -u and yields datadir=/_data_master. Guard that $datadir is nonempty and under $BENCH before deleting.
| @@ -0,0 +1,430 @@ | |||
| #!/usr/bin/env bash | |||
| # A/B pgbench harness for tepid: master (upstream) vs tepid (HOT-indexed). | |||
There was a problem hiding this comment.
These benchmark scripts are not referenced by any Makefile or meson.build (confirmed: no build/test wiring points at src/test/benchmarks/siu/). They are also not deterministic regression/TAP tests and cannot run in CI (they require a pre-built two-variant layout under $BENCH and specific git revisions master/tepid). For a pgsql-hackers submission, performance claims need a reproducible benchmark, but an ad hoc harness carrying developer-specific variant names ("tepid"), absolute paths, and a git merge-base ... origin/master default is unusual to commit to the tree and has no established src/test/benchmarks/ precedent. Consider keeping these out of the patch itself and instead posting them / their numbers to the design thread; if they must be in-tree, the README should cite the pgsql-hackers Message-Id for the SIU design discussion they support.
| @@ -1243,7 +1243,7 @@ spawn_process(const char *cmdline) | |||
| char *cmdline2; | |||
|
|
|||
| cmdline2 = psprintf("exec %s", cmdline); | |||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |||
There was a problem hiding this comment.
This execl -> execlp change is unrelated to the HOT-indexed (SIU) feature this patch implements and should be dropped to keep the diff minimal (a top rejection reason on -hackers). shellprog is only ever set from the compile-time SHELLPROG macro, which is always an absolute path ($(SHELL) from the Makefile, /bin/sh from meson). For an absolute path execlp behaves identically to execl, so this provides no benefit here. Worse, it introduces a footgun/behavior change: if shellprog ever lacked a slash, execlp would perform a $PATH search, making the test harness resolve the shell non-deterministically and from an attacker-influenceable environment. If there is a genuine need (e.g. a relative shell on some platform), it belongs in a separate, justified commit with that rationale; otherwise revert to execl. (high confidence)
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| local v=$1 datadir=$BENCH/_data_split_$v | ||
| rm -rf "$datadir"; mkdir -p "$datadir" |
There was a problem hiding this comment.
Same destructive-deletion footgun confirmed for run.sh/soak.sh exists here, and is worse: rm -rf "$datadir" with datadir=$BENCH/_data_split_$v. An explicitly-exported empty BENCH='' passes set -u, yielding datadir=/_data_split_split_off and an rm -rf against an absolute root-level path. Guard that $BENCH is non-empty (and ideally that $datadir lives under $BENCH) before deleting. (high confidence)
| local v=$1 datadir=$BENCH/_data_split_$v | |
| rm -rf "$datadir"; mkdir -p "$datadir" | |
| local v=$1 | |
| [ -n "${BENCH:-}" ] || { echo "split_ab: BENCH must be set and non-empty" >&2; exit 1; } | |
| local datadir=$BENCH/_data_split_$v | |
| rm -rf "$datadir"; mkdir -p "$datadir" |
There was a problem hiding this comment.
🔍 OCR found 35 issue(s).
- 25 inline, 10 in summary (inline capped at 25)
📄 src/include/access/heapam.h (L180-L180)
xs_prefix_all_dead is written but never read. heap_hot_search_buffer() stores into hscan->xs_prefix_all_dead (heapam_indexscan.c:772) and resets it (:776), but no code consumes the struct field. The only leaf-kill path, heapam_index_kill_item(), keys solely off all_dead/kill_prior_tuple and never inspects xs_prefix_all_dead; the stale-entry consumers (heapam_index_entry_needs_recheck callers) only continue, they do not kill the arriving leaf. So the documented contract here ("Combined with a stale verdict, this lets the index-access layer kill the arriving leaf") describes behavior that is not implemented. This is dead/speculative scaffolding on an exported struct, and the comment is aspirational rather than descriptive. Either wire up the kill-stale-leaf optimization that reads this field, or drop the field (and the corresponding prefix_all_dead out-param of heap_hot_search_buffer) until it has a consumer. Confidence: high.
📄 src/test/benchmarks/siu/scripts/run.sh (L31-L32)
These are developer-local scratch scripts, not part of the automated test suite. They hardcode absolute paths (/scratch/siu-bench), assume a tepid git branch exists, require meson, and are not wired into any Makefile/meson target under src/test/ (verified: src/test/benchmarks/ is introduced only by this change and nothing in src/test/Makefile/meson.build references it). pgsql-hackers routinely rejects committing out-of-tree scratch tooling of this kind. If benchmark reproducibility must be preserved, attach it to the -hackers thread rather than the tree, or at minimum document that it is non-CI, path-portable tooling and remove the hardcoded scratch defaults. (medium, high confidence)
📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L67)
This mutates tracked source in place: sed -i.bak rewrites src/backend/access/heap/heapam_handler.c (the target string .bitmap_and_inexact = heap_bitmap_and_inexact, matches heapam_handler.c:2947, so the edit is real and effective), and restoration depends solely on an EXIT trap. A SIGKILL, an OOM kill during meson compile, or any trap that fires before mv "$hh.bak" "$hh" leaves the checkout with a patched heapam_handler.c and/or a dangling .bak — silently corrupting the developer's working tree. Editing tracked backend source as a side effect of a benchmark is a footgun; prefer building the split_off variant from a committed branch/patch or a runtime toggle rather than an in-place sed on tracked code. (medium, high confidence)
📄 src/test/benchmarks/siu/scripts/build.sh (L38-L40)
Destructive, env-var-driven operations with no confirmation: rm -rf "$prefix" where prefix=$BENCH/$name, plus git checkout --quiet --detach "$rev" which silently moves the user off their branch. A mistyped/empty BENCH (e.g. if the :- default were ever removed, or name is empty) makes rm -rf target an unintended directory, and the branch restore relies on an EXIT trap that does not fire on SIGKILL/crash — leaving the repo detached. Data-loss hazard; at minimum validate BENCH/name are non-empty before rm -rf, and document that an interrupted run can leave a detached HEAD. (medium, high confidence)
📄 src/test/benchmarks/siu/scripts/run.sh (L102-L104)
Fixed sleep 2 after pg_ctl ... start (which is not even invoked with -w) is brittle synchronization: on a slow/loaded machine the server may not be accepting connections yet when setup_schemas/psql_as runs, causing spurious failures. Use pg_ctl -w start and/or poll with pg_isready before issuing psql. Same pattern exists in soak.sh and split_ab.sh. (low, high confidence)
📄 src/test/benchmarks/siu/scripts/run.sh (L37-L37)
The default WIDE_STEPS hardcodes the literal 16, contradicting the documented default "0,1,4,8,WIDE_COLS" at the top of the file (line 24). If a user overrides WIDE_COLS to anything smaller than 16 but leaves WIDE_STEPS at its default, the wide_16 step causes build_wide_set_clause 16 to emit c1=:v,...,c16=:v while seed_wide_table only created columns c1..cWIDE_COLS. pgbench then issues UPDATE wide_table SET ..., c16=:v ... referencing nonexistent columns, which errors at runtime. Derive the final step from $WIDE_COLS so the two stay in sync.
💡 Suggested change
Before:
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,16}
After:
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,$WIDE_COLS}
📄 src/test/modules/index/t/001_concurrent_hot.pl (L10-L11)
The skip guard only checks the enable_injection_points env var, but line 18 then hard-depends on CREATE EXTENSION injection_points succeeding. Every other injection-point TAP test in this tree (e.g. test_misc/t/010_index_concurrently_upsert.pl) adds a second guard after starting the node:
plan skip_all => 'Extension injection_points not installed'
unless $node->check_extension('injection_points');
Without it, an environment where the env var is set but the extension files are not actually installed fails with a hard error instead of skipping cleanly. Recommend adding the check_extension guard for consistency and clean skipping. (low confidence / low severity)
📄 src/test/recovery/t/061_hot_indexed_vm.pl (L100-L103)
Fragile identity check: $update eq $updates[0] uses string equality to mean "this is the first WAL record". This works only because the two HOT_UPDATE lines differ textually (different rel/block/LSN). If a future rmgrdesc change ever made the two lines textually identical, both loop iterations would take the if branch and the existing-table assertion (the else/unlike check that redo must replay the delta) would silently never run, turning a real regression into a passing test. Prefer an index-based loop so the intent is explicit and immune to duplicate-line matches.
💡 Suggested change
Before:
for my $update (@updates)
{
if ($fpw eq 'on' && $update eq $updates[0])
{
After:
for my $i (0 .. $#updates)
{
my $update = $updates[$i];
if ($fpw eq 'on' && $i == 0)
{
📄 src/tools/pgindent/pgindent (L1-L1)
This shebang change is unrelated to the PR's purpose (the HOT-indexed/TAM feature) and should be dropped from the patch. Unrelated changes are a top rejection reason on pgsql-hackers. More importantly, it diverges from the established convention: every other Perl script in the tree (30+ scripts, including the sibling src/tools/copyright.pl and src/tools/mark_pgdllimport.pl) uses #!/usr/bin/perl. If a portability improvement for pgindent is genuinely desired, it belongs in its own dedicated patch applied consistently across the tree, with a reference to the -hackers discussion. Confidence: high.
💡 Suggested change
Before:
+#!/usr/bin/env perl
After:
#!/usr/bin/perl
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L379-L380)
like($err, qr/^$/, ...) can intermittently fail: psql may emit non-error chatter on stderr (e.g. a NOTICE, or a libpq connection notice) that has nothing to do with the INSERT succeeding. The adjacent is($r, 0, ...) already asserts the command succeeded via the exit code, which is the reliable signal. The empty-stderr regex is both redundant with the return-code check and a portability/flakiness footgun. Consider dropping this assertion (or using on_error_stop + checking only $r).
| SET enable_seqscan = off; | ||
| SET enable_indexscan = off; | ||
| SET enable_bitmapscan = on; | ||
| -- Must return the row. | ||
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; |
There was a problem hiding this comment.
The test asserts it "guards" the BitmapAnd union-and-recheck path, but it never verifies the plan actually is a BitmapAnd. On a single-row, un-ANALYZEd table the planner is free to pick a single Bitmap Index Scan on one index with the other predicate applied as a recheck/filter, in which case the row is still returned (count = 1) while none of the bitmap_and_inexact code the comment describes is exercised. The sibling test (test_locator_tableam/sql/bitmap_hooks.sql) and the existing bloom tests above (lines 24-26, 68-70) all pin the plan with EXPLAIN (COSTS OFF). Add an EXPLAIN (COSTS OFF) for this query to confirm a BitmapAnd is chosen, and disable enable_indexonlyscan / set max_parallel_workers_per_gather = 0 so the plan is deterministic across machines. (high confidence)
| SET enable_seqscan = off; | |
| SET enable_indexscan = off; | |
| SET enable_bitmapscan = on; | |
| -- Must return the row. | |
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; | |
| SET enable_seqscan = off; | |
| SET enable_indexscan = off; | |
| SET enable_indexonlyscan = off; | |
| SET enable_bitmapscan = on; | |
| SET max_parallel_workers_per_gather = 0; | |
| -- Must use a BitmapAnd of the two indexes... | |
| EXPLAIN (COSTS OFF) SELECT count(*) FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; | |
| -- ...and must return the row. | |
| SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22; |
| SET enable_seqscan = off; | ||
| SET enable_indexscan = off; | ||
| SET enable_bitmapscan = on; |
There was a problem hiding this comment.
These GUCs are left changed when the block ends only via the matching RESETs below, but if the SELECT errors out the session-level GUC state leaks into subsequent tests in the same file. Minor, but the surrounding file convention (lines 12-14, 20-22, 149-151) is consistent SET/RESET pairing; this is fine as-is. Lower-priority note: consider whether fillfactor = 50 and the explicit id/changed btree indexes are all load-bearing for reproducing the HOT-indexed update, or trim to the minimal schema needed. (low confidence)
| ParameterNames: true | ||
| DeducedTypes: true | ||
| CompileFlags: | ||
| CompilationDatabase: build/ # Search build/ directory for compile_commands.json |
There was a problem hiding this comment.
This file is developer-local IDE scaffolding (clangd config pinned to build/, -I. and -I../../../../src/include) that assumes one specific working directory and build layout. It is not part of the PostgreSQL source/test/doc change and does not belong in the tree. For a patch destined for pgsql-hackers/commitfest, committing personal editor config to the repo root is out-of-scope churn that will be rejected. Keep it in a personal (untracked) gitignore instead.
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
These breakpoints are pinned to hardcoded line numbers in heavily-modified files and are already stale. For example break heapam.c:3210 is annotated "Main entry: heap_update", but heapam.c:3210 is actually a relkind check in the toast-deletion portion of a different function; and break pruneheap.c:1802 is annotated as chain-collapsing bitmap collection, but line 1802 is priorXmax = HeapTupleHeaderGetUpdateXid(htup); at the tail of a chain-advance loop. Line-number breakpoints drift the instant any of these files change, so they silently land on the wrong code. Beyond the correctness drift, this is personal debugging scaffolding that does not belong in the PostgreSQL tree.
| alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"' | ||
|
|
||
| # Database management | ||
| alias pg-init='trash "$PG_DATA_DIR" 2>/dev/null || rm -rf "$PG_DATA_DIR"; "$PG_INSTALL_DIR/bin/initdb" --debug --no-clean "$PG_DATA_DIR"' |
There was a problem hiding this comment.
pg-init does rm -rf "$PG_DATA_DIR" on a value derived entirely from an environment variable. If PG_DATA_DIR is unset or empty when this file is sourced outside the Nix shell, this expands to rm -rf "" (or an unexpected path), a silent data-loss footgun. The same unguarded pattern appears in pg-full-clean. Committing shell helpers that rm -rf env-derived paths with no non-empty guard is unsafe; and more broadly this personal shell library does not belong in the PostgreSQL tree.
| * Return a Bitmapset that contains the set of modified (changed) indexed | ||
| * attributes between oldtup and newtup. | ||
| * | ||
| * Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that operates |
There was a problem hiding this comment.
This comment line runs well past the ~78-column limit the tree uses for block comments (the surrounding lines wrap at ~78). pgindent will not fix comment wrapping, so this should be rewrapped by hand to stay consistent.
(low confidence / style)
| * Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that operates | |
| * Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that | |
| * operates |
| if (!isnull) | ||
| { | ||
| indexprs = (List *) stringToNode(TextDatumGetCString(datum)); | ||
| pull_varattnos((Node *) indexprs, 1, &attrs); | ||
| } |
There was a problem hiding this comment.
On the first call for a given index relcache entry (before the cached rd_indattr exists), the parsed node trees from stringToNode() and the TextDatumGetCString() strings are palloc'd in the caller's current memory context and never freed. Callers such as execIndexing.c and heapam_indexscan.c invoke this on per-tuple paths; the first invocation may land in a long-lived context, leaking the parse tree until that context resets. Existing helpers like RelationGetIndexExpressions build such trees in rd_indexcxt. Consider doing the parse/pull_varattnos work in a short-lived context (or pfreeing the intermediate trees and CStrings) before returning, so nothing leaks into the caller's context.
| RelationIdUpdatesInPlace(Oid relationId) | ||
| { | ||
| Relation rel; | ||
| bool result; | ||
|
|
||
| RelationIdCacheLookup(relationId, rel); |
There was a problem hiding this comment.
The header comment states "the caller must hold a lock on the relation," but neither path enforces or documents it via an assertion. The sibling RelationIdGetRelation() calls AssertCouldGetRelation() precisely to catch the no-lock misuse this comment warns about. Since this is a new exported API advertised as a planner hot path, add AssertCouldGetRelation() (matching RelationIdGetRelation) so a missing lock is caught in assert builds rather than risking a TOCTOU-style race.
| * with 2-bit _vm forks that are not rewritten. The committer re-picks both | ||
| * values at commit time, along with the copy in t/010_vm_conversion.pl. | ||
| */ | ||
| #define VISIBILITYMAP_WIDTH_CHANGE_CAT_VER 202610201 |
There was a problem hiding this comment.
This hardcoded value is stale relative to the tree it ships in and is a silent data-corruption footgun. The current CATALOG_VERSION_NO in this branch is 202611121, which is already greater than VISIBILITYMAP_WIDTH_CHANGE_CAT_VER (202610201). The rewrite guard in transfer_relfile()/check_new_cluster() uses cat_ver < VISIBILITYMAP_WIDTH_CHANGE_CAT_VER, so:
- A cluster that still has the 2-bit VM but whose catversion is >= 202610201 (e.g. anything initdb'd in this branch after 202610201 but before the widening actually landed) is NOT rewritten, and its 2-bit _vm forks are then read as 4-bit -> wrong all-visible/all-frozen bits -> index-only scans return wrong rows and VACUUM can advance relfrozenxid incorrectly.
- A cluster that already has the 4-bit VM but a catversion < 202610201 is rewritten (doubled) a second time -> corrupt _vm.
The value must exactly equal the CATALOG_VERSION_NO installed by the commit that actually widens the VM; as written it is a guess that is already contradicted by the branch's own catversion. Verify the widening commit's final catversion and set this (and the duplicate literal in t/010_vm_conversion.pl line 42) to that exact number. Confidence: high.
| /* Fast path: return a copy of the cached bitmap. */ | ||
| if (indexRel->rd_indattr != NULL) | ||
| return bms_copy(indexRel->rd_indattr); |
There was a problem hiding this comment.
Empty indexed-attr set permanently defeats the cache and repeats the per-call leak. For an index whose referenced-attribute set is empty (e.g. CREATE INDEX ON t ((1)) or any constant-expression index), attrs is NULL, so bms_copy(attrs) stores NULL into rd_indattr. The fast-path check if (indexRel->rd_indattr != NULL) then never fires, so every subsequent call re-runs stringToNode()/TextDatumGetCString() and leaks those allocations in the caller's context (the confirmed leak finding is about the first call; this makes it every call). Since this runs per-index per-row on the HOT-indexed write path (execIndexing.c) and per index scan (heapam_indexscan.c), the caching optimization silently never engages for such indexes. Use a separate bool to mark the cache as populated (or cache an empty-but-non-NULL sentinel) rather than relying on a non-NULL pointer.
There was a problem hiding this comment.
🔍 OCR found 31 issue(s).
- 25 inline, 6 in summary (inline capped at 25)
📄 src/test/benchmarks/siu/scripts/build.sh (L15-L15)
Portability/minimalism (high confidence): these benchmark drivers are committed under src/test/benchmarks but hardcode machine-specific absolute paths and a private branch name as defaults -- $HOME/ws/postgres/tepid, /scratch/pg, BENCH=/scratch/siu-bench, and TEPID_REV=tepid. A benchmark that only runs on the author's directory layout and branch cannot be reproduced by a reviewer and should not land in a patch. Make REPO/BENCH required (no silent default) or derive REPO from the script location, and drop the tepid branch default (it is meaningless outside the author's clone).
📄 src/test/benchmarks/siu/scripts/run.sh (L85-L86)
Data-loss hazard / robustness (low confidence): find "$datadir" -mindepth 1 -delete && rmdir "$datadir" unconditionally wipes $BENCH/_data_$v. If BENCH is misconfigured to point at a real directory (it is user-supplied and defaults to a hardcoded /scratch path), this deletes it with no confirmation. Separately, server readiness after pg_ctl ... start is handled with a fixed sleep 2 rather than polling (e.g. pg_ctl -w start or pg_isready), which is racy on a busy/slow host and can cause the first psql/pgbench to fail intermittently. The same pattern is duplicated in soak.sh.
📄 src/test/modules/index/t/002_unique_hot.pl (L131-L133)
The final NULLS DISTINCT block creates table t but, unlike every case in the loop above (which ends with DROP TABLE t), never drops it before $node->stop. Harmless today since the node is torn down immediately, but it breaks the "DROP objects the test creates" discipline and would silently collide if another case were appended after this block. Add a cleanup for consistency.
💡 Suggested change
Before:
is($node->safe_psql('postgres', 'SELECT count(*) FROM t'),
'2', 'NULLS DISTINCT permits a second NULL expression key');
$node->stop;
After:
is($node->safe_psql('postgres', 'SELECT count(*) FROM t'),
'2', 'NULLS DISTINCT permits a second NULL expression key');
$node->safe_psql('postgres', 'DROP TABLE t');
$node->stop;
📄 src/bin/pg_upgrade/file.c (L335-L338)
Per c.h, StaticAssertDecl() "doesn't need to be used in new code" and is a declaration that must sit where a variable declaration is valid. These are placed correctly (at the top of the function block, before statements), so this compiles, but the modern idiom is a plain static_assert(). Minor style nit, not a blocker.
📄 src/test/benchmarks/siu/scripts/soak.sh (L84-L88)
Robustness: this script runs under set -euo pipefail and backgrounds a long-running pgbench (pgb=$!) and a postmaster, but installs no EXIT/ERR trap to tear them down. Any transient failure of a psql_as call inside the sampling loop (lines 95-101) aborts the whole script and leaves the backgrounded pgbench and the started postmaster orphaned (still holding the port and datadir). The sibling drivers run.sh and bloat.sh guard this with a trap '... pg_ctl ... stop' EXIT; soak.sh should do the same (and kill $pgb).
📄 src/test/regress/pg_regress.c (L1246-L1246)
Unrelated, unexplained change: switching execl to execlp is not required by this patch (the HOT-indexed table-AM feature) and is a minimal-diff violation. shellprog is SHELLPROG, which is hardcoded to an absolute path (/bin/sh under meson, $(SHELL) under make, normally absolute). execlp only differs from execl when the program name contains no slash, in which case it does a PATH lookup. Since no accompanying Makefile/meson change makes SHELLPROG a bare name, this change has no effect for the normal build and only introduces a subtly different (PATH-dependent, less predictable) shell-resolution behavior with no test and no rationale. Drop this hunk, or if it is genuinely needed (e.g. for a Nix dev environment), split it into its own patch with the corresponding build-system change and a clear justification. [high confidence]
💡 Suggested change
Before:
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
After:
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
| break heapam.c:4019 | ||
| break heapam.c:4024 | ||
| break heapam.c:4033 |
There was a problem hiding this comment.
These hard-coded absolute source line breakpoints are brittle and already wrong against the committed tree. For example, this block claims break heapam.c:4019 is the "pure HOT (no indexed columns changed)" decision, but heapam.c:4019 is actually if (PageIsAllVisible(page)) inside the tuple-lock WAL path, unrelated to any HOT decision. Line-number breakpoints drift with every edit and are meaningless outside the author's working tree. Use function-name breakpoints (as the rest of this file already does), or better, do not commit a personal .gdbinit at all.
| break heap_hot_search_buffer | ||
|
|
||
| # Redirect-with-data: initialize bitmap accumulator from collapsed redirect | ||
| break heapam_indexscan.c:182 |
There was a problem hiding this comment.
Confirmed stale: break heapam_indexscan.c:182 is annotated as "Redirect-with-data: initialize bitmap accumulator from collapsed redirect", but line 182 is inside heapam_index_scan_end() (IndexScanHeapData *hscan = ...), which has nothing to do with redirect-with-data accumulation. The remaining line-number breakpoints (heapam_indexscan.c:250/297, indexam.c:299, execIndexing.c:370, pruneheap.c:1802/1836/1863/1287/2936) have the same problem. Replace with symbol breakpoints or drop this file from the patch.
| patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: { | ||
| patches = (oldAttrs.patches or []) ++ [ | ||
| ./glibc-no-fortify-warning.patch | ||
| ]; | ||
| }); |
There was a problem hiding this comment.
This dev-shell patches glibc's features.h via ./glibc-no-fortify-warning.patch, a workaround scoped to one contributor's Nix environment. Along with flake.nix/pg-aliases.sh/.clangd/.gdbinit, this is developer-local scaffolding unrelated to the HOT-indexed-updates feature and should not be committed to the PostgreSQL source tree. It needlessly expands the patch footprint (and drags glibc-no-fortify-warning.patch and flake.lock into the tree too). Keep the dev environment untracked.
| }); | ||
|
|
||
| # Use LLVM for modern PostgreSQL development | ||
| llvmPkgs = pkgs-unstable.llvmPackages_21; |
There was a problem hiding this comment.
llvmPackages_21 is pulled from nixpkgs-unstable, which moves independently of the pinned stable channel. Even with flake.lock committed, mixing an unstable input for the toolchain makes the dev environment harder to reproduce over time. Minor relative to the larger issue (this whole file is developer-local scaffolding that should not be in the patch), but flagging the reproducibility smell.
| @@ -0,0 +1,45 @@ | |||
| { | |||
| description = "PostgreSQL development environment"; | |||
There was a problem hiding this comment.
flake.nix (with its companion shell.nix and a committed flake.lock) wires up a complete personal Nix dev environment. This is unrelated to the feature and is exactly the kind of non-minimal, environment-specific churn that gets patches rejected on pgsql-hackers. PostgreSQL does not ship a Nix flake; keep this untracked rather than committing it.
| if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber && | ||
| PageGetMaxOffsetNumber(page) == FirstOffsetNumber) | ||
| PageGetMaxOffsetNumber(page) == FirstOffsetNumber && | ||
| ((PageHeader) page)->pd_upper == | ||
| ((PageHeader) page)->pd_special - MAXALIGN(heaptup->t_len)) |
There was a problem hiding this comment.
This heap_insert INIT_PAGE tightening (and its mirror in log_heap_update below) is a wal_consistency_checking fix for a scenario -- vacuum's lp_truncate_only pass (PRUNE_VACUUM_CLEANUP) leaving orphan tuple bytes behind an LP_UNUSED slot -- that is independent of the HOT-indexed feature and predates this patch. Bundling an orthogonal WAL-consistency fix into this large feature patch hurts bisectability and review (one commit, one logical change). Consider splitting it into its own patch with its own test, unless it is strictly required by the new VM/stub code paths (the comment does not establish that dependency).
| /* pg_stat_get_tuples_hot_indexed_upd_skipped */ | ||
| Datum | ||
| pg_stat_get_tuples_hot_indexed_upd_skipped(PG_FUNCTION_ARGS) | ||
| { |
There was a problem hiding this comment.
These two functions hand-copy the exact boilerplate of the existing PG_STAT_GET_IDXENTRY_INT64(stat) macro (defined a few lines below at the comment "Accessor macro for index stats entries"), which already expands to a function that fetches pgstat_fetch_stat_idxentry(relid), returns 0 on NULL, and otherwise casts idxentry->stat to int64. The only reason the macro can't be reused directly is that it hardcodes the pg_stat_get_idx_ name prefix while these prosrc names use pg_stat_get_tuples_hot_indexed_upd_. Consider either reusing the macro (keeping the pg_stat_get_idx_* naming for the SQL prosrc) to avoid the copy-paste, or at least generalizing the macro to take an explicit function-name token. As written this is redundant boilerplate that will drift from the macro.
| afterTriggers.maximages = Max(64, afterTriggers.maximages * 2); | ||
| afterTriggers.images = afterTriggers.images ? | ||
| repalloc_huge(afterTriggers.images, | ||
| afterTriggers.maximages * sizeof(AfterTriggerImage)) : | ||
| MemoryContextAllocHuge(afterTriggers.event_cxt, | ||
| afterTriggers.maximages * sizeof(AfterTriggerImage)); |
There was a problem hiding this comment.
The images[] descriptor array grows with the total image count, including spilled ones: the grow block (and nimages++) runs on both the in-memory and file paths. So spilling the tuple bodies to image_file does not bound this array's memory - a huge bulk UPDATE on an in-place AM still accumulates one AfterTriggerImage (~24 bytes) per row in event_cxt for the life of the transaction, partly defeating the stated purpose of the spill ("so that a large bulk update does not hold every row in memory").
Separately, the allocation size afterTriggers.maximages * sizeof(AfterTriggerImage) is a uint32 * size_t product. The only guard is maximages > PG_UINT32_MAX / 2; on 32-bit platforms where Size is 32-bit, maximages * sizeof(AfterTriggerImage) overflows Size well before that bound (e.g. maximages near 2^31 times 24), passing a wrapped, far-too-small size to MemoryContextAllocHuge/repalloc_huge and causing a heap buffer overflow on subsequent images[nimages] writes. Guard the byte product (e.g. via pg_mul_size_overflow or capping maximages against MaxAllocHugeSize / sizeof(AfterTriggerImage)).
|
|
||
| #include "storage/itemptr.h" | ||
| #include "utils/dsa.h" | ||
| #include "utils/relcache.h" |
There was a problem hiding this comment.
Pulling the full utils/relcache.h into tidbitmap.h is heavier than needed. The two new prototypes use Relation only as a by-value pointer type, so a forward declaration suffices and keeps this widely-included header (via access/relscan.h) minimal, per PG header hygiene. relcache.h drags in access/tupdesc.h and common/relpath.h transitively.
Confidence: low (style/maintainability nit, not a correctness issue).
| #include "utils/relcache.h" | |
| /* avoid pulling in relcache.h; Relation is only used in prototypes below */ | |
| typedef struct RelationData *Relation; |
| trap "test ! -e $hh.bak || mv -f $hh.bak $hh; git checkout --quiet $orig" EXIT | ||
| git checkout --quiet --detach "$TEPID_REV" |
There was a problem hiding this comment.
Footgun (high confidence): this mutates the user's working repository in place. It runs git checkout --quiet --detach "$TEPID_REV" and sed -i.bak on a tracked source file (src/backend/access/heap/heapam_handler.c, whose .bitmap_and_inexact = heap_bitmap_and_inexact, line is confirmed to exist), relying solely on a single EXIT trap to restore state. If the script is SIGKILLed, the machine loses power, or the trap otherwise fails, the user's checkout is left detached and/or heapam_handler.c is left edited (with the .bak possibly already consumed by the restore of the other arm). A patch that silently corrupts a developer's tree should not ship. Build the split_off variant from a disposable worktree (git worktree add) or a clean clone under $BENCH instead of editing the live checkout.
Squashes the four CI commits this fork carried on top of upstream into
one, and cuts the carry from 3776 lines to ~1000.
Removed:
- .github/{README,QUICKSTART}.md and .github/docs/* (2273 lines). Beyond
being bulk we rebase hourly, they had drifted into being wrong: they
described a daily (not hourly) sync, Claude 3.5 Sonnet via the direct
Anthropic API (we run Opus on Bedrock), a static IAM user (we use
OIDC), and a Windows dependency-builder workflow that the same commit
which documented it had already deleted. 16 referenced paths did not
exist. An AI reviewer reading this repo ingests that as truth.
- sync-upstream-manual.yml (252 lines). sync-upstream.yml already has
workflow_dispatch; the sole behavioral difference was a force_push
toggle whose "false" setting cannot work after a rebase anyway.
- .github/.gitignore, which only ignored scripts/ai-review/, a directory
that does not exist here.
Rewritten:
- sync-upstream.yml -> fork-sync-upstream.yml, 260 lines to 47. git
rebase already handles the fast-forward, no-op and diverged cases that
the ahead/behind arithmetic hand-rolled, so all that goes away, as do
the issue open/comment/close machinery (a scheduled-run failure is
already emailed to the one consumer, and the run log carries what the
issue body would) and the "dev setup|dev v[0-9]" commit allowlist,
which matched zero commits on master and had since it was written.
Sync drops to every three hours: hourly meant 24 full-history clones
of an 840MB repo per day, each force-push also re-triggering pg-ci.yml
on master, to collect a handful of upstream commits.
Renamed:
- ocr-review.yml, ocr-model-check.yml -> fork-*.yml. Upstream owns
.github/ too (it has touched pg-ci.yml three times this year), so
fork-owned workflows now live under a name prefix upstream will not
collide with, making the rebase structurally conflict-free and letting
the sync guard match fork-owned paths exactly rather than allowing all
of .github/.
Unchanged: the OCR review workflows' logic and .github/ocr/* (rule.json,
context.md, litellm.yaml, pg-history.py) - the part that carries value.
The OCR review workflows and their config were the last bulk this fork carried on top of upstream: ~970 lines rebased onto postgres/postgres every few hours, editable only by amending a commit that gets force- pushed. They are now reusable (workflow_call) workflows in the sidecar repo gburd/ci-workflows, and master carries two caller stubs instead. Carry after this: 3 files, ~110 lines, one commit. The sidecar is also where the value is: ocr/rule.json and ocr/context.md (the PostgreSQL committer-grade review corpus) become normal reviewable files with normal PRs and history, rather than a blob inside a commit that is rewritten on every sync. Mechanics worth knowing: - The review jobs check ci-workflows out at github.job_workflow_sha, so the rules always come from the same commit as the logic using them, and pinning @v1 (or a sha) in the stub pins both. This replaces the old "git show origin/<default_branch>:.github/ocr/..." dance, which existed only because the config had to be read off a branch other than the PR's. - AWS settings are passed as explicit inputs rather than read from vars inside the reusable workflow. vars does resolve across the boundary, but the contract stays visible at the call site. - A reusable workflow does not change the OIDC subject: it stays repo:gburd/postgres:*, so the ocr-bedrock-ci trust policy is unchanged. - ci-workflows must stay public; callers read it with GITHUB_TOKEN. The sync guard narrows to .github/workflows/fork-* accordingly, and sync-upstream keeps needing SYNC_TOKEN: the stubs are still workflow files, which GITHUB_TOKEN may not push.
This isn't a commit that will be submitting for review, it is purely for local developer tooling while developing this patch set. Ignore it.
An index stores a locator for every entry: the value it hands back to the table AM to identify the row. Every in-core table AM uses an ItemPointerData, a block number plus an in-block offset, and a good deal of core code leans on that shape. Bitmap scans split the locator into a block and an offset, GIN packs the offset into 11 bits, TID range scans expose its ordering to SQL, and nbtree breaks ties between equal keys with it. A table AM that identifies rows some other way has no way, today, to tell core any of this, and core has no way to ask. Give both sides a place to say what they mean. A table AM returns a per-relation locator descriptor from a new relation_locator callback, reached through RelationGetLocatorDesc() and cached on the relcache entry. The descriptor gives the locator's width (negative for a variable-width locator), a name for error messages, whether a row keeps its locator for its whole life (stable), whether the version an UPDATE replaced can still be fetched by its locator (old_version_retained, with RelationUpdatesInPlace() for the executor), and max_offset, the largest offset any of the relation's locators carries. max_offset has two readers today: the planner does not consider bitmap scans of a relation whose max_offset exceeds what a TID bitmap can hold, and ginbuild() refuses such a table, since GIN's posting lists and partial-match scans assume the same bound. This is the gap Jeff Davis ran into in "MaxOffsetNumber for Table AMs" (2021). The other fields describe properties that later commits in this series act on. amlocator.h and a new "Row Locators" section of tableam.sgml also write down what core relies on of every locator that no field states: that it travels as an ItemPointerData, that it is the row's ctid, which values are reserved, and the (block, offset) order that nbtree, ANALYZE, TID range scans and BRIN depend on. A table AM without the callback is treated as heap is, so out-of-tree AMs keep working untouched. Heap returns a single static descriptor that writes down what core already assumes of a TID, so for heap this is a no-op. Discussion: https://postgr.es/m/0bbeb784050503036344e1f08513f13b2083244b.camel@j-davis.com
The field is documented as "an update chain was followed", but that is not what its readers act on. heapam_tuple_lock sets it only after replacing *tid, and every reader (ExecLockRows, GetTupleForTrigger, ri_LockPKTuple, and the EvalPlanQual paths in ExecUpdate and ExecDelete) responds by re-evaluating the row it read earlier. What they need to know is that the version they locked is later than the one they named, because a concurrent transaction updated the row and committed. For heap that is the same thing as following the update chain, so the rename changes no behavior. It gives a table AM whose rows are not identified by a version-specific TID a definition it can implement, which "Support table AMs that update rows in place" completes.
Keep physical tuple links and transaction/command IDs inside the heap AM. Callers need to distinguish same-command and triggered modifications, current-transaction ownership, and partition moves, not interpret heap headers or special TID values. Translate only fields defined for the returned result, using the supplied command ID and preserving the creation-command check when a lock follows an update chain. Initialize failure facts on every call without losing retargeted across internal retries. Keep existing heap partition-move errors while allowing MERGE to recognize an AM-reported move. Exercise the booleans directly through the real lock callback, including poisoned output, child transactions, command visibility and retargeting. Check MERGE cardinality handling in a child transaction as well.
table_tuple_lock, table_tuple_update and table_tuple_delete take a locator, and for heap a TID names one version of a row. A table AM whose locator names only the row, as one that updates rows in place does, cannot tell from the call whether the caller read the row through its snapshot, in which case a newer committed version means the caller must re-evaluate, or has already locked the latest version itself, in which case it must not. Guessing either way is wrong: re-evaluating a row the command already locked runs EvalPlanQual where ON CONFLICT DO UPDATE and MERGE cannot take it, and assuming "already locked" whenever the transaction holds a lock loses concurrent updates in an UPDATE fed by a SELECT ... FOR UPDATE subquery, or in a FOR UPDATE OF join that reaches a locked row twice. Only the executor knows. Add TUPLE_LOCK_FLAG_LOCKED_VERSION, TABLE_UPDATE_LOCKED_VERSION and TABLE_DELETE_LOCKED_VERSION, meaning "this is the version this command has already locked", and pass them where that is so: after a BEFORE ROW trigger or EvalPlanQual has locked the row in ExecUpdate, ExecDelete and MERGE, for the row ON CONFLICT DO UPDATE and DO SELECT lock, and from logical replication apply, which locks the row first and has no snapshot. ExecLockRows and the RI triggers never pass them, since their rows come from snapshot scans. GetTupleForTrigger learns of it through a new row_locked argument to ExecBRUpdateTriggers and ExecBRDeleteTriggers. Heap ignores the new bits, since its TID already names a version, so nothing changes for heap.
Index AMs, the executor and every index-facing table AM callback pass a row's locator as an ItemPointerData, and validate_index() packs it into an int8 for CREATE INDEX CONCURRENTLY. A table AM whose locator descriptor declares a wider or variable-width locator can therefore not be indexed by any in-core index AM. Let the index side say whether it can do better. An index AM sets amcanvarlocator when it can store a locator that does not fit in an ItemPointerData, reported by pg_indexam_has_property() as "can_var_locator". Unless it does, building an index on a table whose locator is variable-width or wider than an ItemPointerData fails with an error naming both access methods and the locator. The check is in index_create(), before any catalog entry is made, and again in index_build(), because a table's locator can change under existing indexes: ALTER TABLE ... SET ACCESS METHOD rebuilds them through reindex_relation(), and REINDEX, TRUNCATE and VACUUM FULL rebuild them too. Every index build, including partition index cloning and constraint indexes, passes through one of the two. No in-core index AM sets the flag and no in-core table AM has a wide locator, so nothing changes for existing AMs. Carrying a wider locator through aminsert, index scans and vacuum is left for a later series; the flag lets an index AM and a table AM that agree on one find each other until then.
An AFTER ROW trigger event records the locators of the rows it fires for, and the rows are fetched by locator when the trigger runs, which for a deferred trigger is at COMMIT. That works for heap because a locator names a version and heap keeps old versions until VACUUM. For a table AM whose locator names only the row, the fetch returns whatever the row is when the trigger fires: OLD is already overwritten, and NEW may have changed again since. Let an event carry copies of its rows instead. A new AFTER_TRIGGER_IMAGES event kind records, alongside the locators, an index into a per-transaction store of copied rows: the OLD row of an UPDATE, and the NEW row of an INSERT or UPDATE for every row trigger other than a foreign key check or a unique recheck, which need the row's current version to decide whether it is still live. A copy is made once per row and shared by all the triggers that fire for it. Copies stay in memory up to work_mem and spill to a temporary file beyond that, owned by a resource owner of its own so an abort closes it without retrying a failed write. They are released at the end of a top-level statement once no deferred event remains, and a subtransaction abort rolls the store back to where it began. ExecARUpdateTriggers takes the old row the caller holds rather than fetching it again. Events carry copies only for a table whose locator descriptor says old_version_retained = false. The next commit makes the executor copy the old row before the write, so that the copy is the pre-update row. Heap says true, so heap events are unchanged.
Core assumes throughout that an UPDATE leaves the pre-update version
of a row readable through its locator: it re-fetches the old row by
TID when an AFTER trigger fires, for RETURNING OLD, and when
EvalPlanQual rechecks a joined row. Heap keeps the old version on
its page until VACUUM, so this is always true for heap. A table AM
that overwrites a row in place has replaced the old version by then,
so on such an AM AFTER ROW triggers and OLD transition tables see
the wrong values, and foreign-key enforcement misfires.
When the locator descriptor says old_version_retained = false:
- The executor copies the old row out of its slot before the
write, in ExecUpdate, MERGE and logical replication apply, so
RETURNING OLD and the AFTER ROW triggers read the copy.
- After-trigger events carry row copies (previous commit).
- EvalPlanQual carries an unlocked joined row whole (ROW_MARK_COPY)
instead of re-fetching it.
The AM must still keep a deleted row fetchable by its locator until
the deleting transaction ends, and a fetch by locator must return
the row's current version; amlocator.h states both. With the
version-naming bits from earlier in the series this is enough for
an in-place AM to give heap's answers.
Heap's descriptor says true, so no heap path changes.
A bitmap scan splits each locator into a group and a slot within it, the block and line pointer offset for heap, and BitmapAnd and BitmapOr combine their inputs group by group without touching the table. That is exact as long as every index entry for one row version names the same slot. A table AM that cannot promise that for some groups has no way to say so: an intersection would drop a row whose two entries name different slots, and a union over groups that do not match the AM's storage may need a recheck. Add two optional per-group callbacks to the locator descriptor. bitmap_and_inexact makes BitmapAnd union a flagged group's slots and mark it for recheck instead of intersecting it, and bitmap_or_inexact makes BitmapOr mark a flagged group for recheck. Each takes the group and a buffer the callback may keep pinned across calls on one scan. Every bitmap index scan records its table on the bitmap it fills (tbm_set_relation), so a combine at any depth of the bitmap tree consults the table's callbacks. Heap leaves both NULL here, so it takes the exact path it always did.
Add regression coverage for existing classic Heap-Only Tuple (HOT) update behavior, committed first so the behavioral changes in the later HOT-indexed commits are diffable against a known-good baseline. This commit adds tests so as to codify explicitly the HOT contract for the heap AM. The new hot_updates regression test exercises: - Basic HOT vs non-HOT update decisions - The all-or-none property across multiple indexes - Partial indexes and predicate handling - BRIN (summarizing) indexes allowing HOT updates - TOAST column handling with HOT - Unique constraint behavior - Multi-column indexes - Partitioned table HOT updates - HOT chain formation and the index-scan walk over a chain Authored-by: Greg Burd <greg@burd.me>
The executor needs to know which indexed attributes changed without assuming how a table AM represents their values. Add an optional modified_attrs callback and table_modified_attrs() wrapper that return the changed subset of a candidate bitmap. An AM without the callback conservatively reports every candidate as changed. This separates the comparison contract from the update interface. Heap implements the callback before the executor begins using it, so the update conversion preserves existing HOT and locking choices.
Provide heapam_modified_attrs(), the heap AM's implementation of the table AM modified_attrs callback introduced in the preceding commit. It answers "which of these attributes changed value between the old and new versions of a row?" using the heap's own notion of attribute equality (datum_image_eq on the logical slot values) and system-column semantics (the FirstLowInvalidHeapAttributeNumber offset convention and the tableoid special case). This is the heap-specific mechanism that previously lived in the executor as ExecCompareSlotAttrs(); moving it behind the table AM boundary keeps the executor agnostic of how any given AM stores its data or what its system columns mean, while the executor retains the policy decision of which indexes to maintain from the returned overlap. An AM that does not provide the callback is treated as though every candidate attribute changed (maintain all indexes).
Refactor executor update logic to determine which indexed columns have actually changed during an UPDATE operation rather than leaving this up to HeapDetermineColumnsInfo() in heap_update(). Finding this set of attributes is not heap-specific, but more general to all table AMs and having this information in the executor could inform other decisions about when index inserts are required and when they are not regardless of the table AM's MVCC implementation strategy. The heap-only tuple decision (HOT) in heap functions as it always has; what moves to the executor is only the determination of the "modified indexed attributes" (modified_idx_attrs). ExecUpdateModifiedIdxAttrs() replaces HeapDetermineColumnsInfo() and is called before table_tuple_update() crucially without the need for an exclusive buffer lock on the page that holds the tuple being updated. This reduces the time the buffer lock is held later within heapam_tuple_update() and heap_update(). Besides identifying the set of modified indexed attributes HeapDetermineColumnsInfo() was also partially responsible for the decision about what to WAL log for the replica identity key. That logic moves into heap_update() and into the replacement helper HeapUpdateModifiedIdxAttrs(), so simple_heap_update() and heapam_tuple_update() share the same logic since both call into heap_update(). Updates stemming from logical replication also use the new ExecUpdateModifiedIdxAttrs() in ExecSimpleRelationUpdate(). ExecUpdateModifiedIdxAttrs() uses ExecCompareSlotAttrs() to identify which attributes have changed and then intersects that with the set of indexed attributes to identify the modified indexed set, the modified_idx_attrs. This patch introduces a few helper functions to reduce code duplication and increase readability: HeapUpdateHotAllowable() and HeapUpdateDetermineLockmode(), used in both heap_update() and simple_heap_update(). heap_update() is now called with lockmode pre-determined and a boolean indicating whether the update may be HOT, both const. If during heap_update() the new tuple fits on the same page and that boolean is true, the update is HOT. So although the functions and timing of the HOT decision code have changed, none of the logic governing when HOT is allowed has changed. Development of this feature exposed nondeterministic behavior in three existing tests, which have been adjusted to avoid inconsistent results due to tuple ordering during heap page scans. Authored-by: Greg Burd <greg@burd.me> Discussion: https://www.postgresql.org/message-id/flat/78574B24-BE0A-42C5-8075-3FA9FA63B8FC%40amazon.com
An update that preserves existing row locators need not replace entries in indexes whose referenced attributes did not change. Separate this executor and index support from the heap storage implementation so other table access methods can use the contract. Track each index's referenced attributes independently of the narrow indexUnchanged hint. Let a table AM skip index entries that no longer name the row's current key, and add SO_INDEX_KEEP_STALE for the constraint checks that must see them, flagged, to compare keys themselves; provide an optional checked fetch for constraint enforcement. Access methods without that callback use the existing fetch operations. Distinguish index-only scans from requests to return index tuples when deciding whether to retain btree leaf pins. Prepare the selective-update counters and their flush paths here. The existing heap statistics call supplies false for the new argument until the heap implementation can produce such updates; this is the only heap change in this preparatory commit. Bump the statistics file format identifier for the added counters.
Define the on-disk representation a HOT-indexed update and its later prune/collapse produce, ahead of the code that reads or writes it: - HEAP_INDEXED_UPDATED (htup_details.h), the t_infomask2 bit marking a heap-only tuple whose producing UPDATE also changed an indexed column; and - access/hot_indexed.h, the inline fixed-size modified-attrs bitmap stored in the tail of such a tuple, plus the xid-free "collapse-survivor stub" format (HEAP_INDEXED_UPDATED with natts == 0, a forward link, and the segment's bitmap) and the accessors both share. The design itself is documented in README.HOT-INDEXED, which the next commit adds along with the code that writes this format. Co-authored-by: Greg Burd <greg@burd.me> Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>
A selective-indexed update, added by the next commit, plants a fresh index entry that points at the new heap-only tuple mid-chain while every unchanged index still points at the chain root. The two entries name different line pointers on the same heap block, and an exact BitmapAnd intersection of the two indexes would drop the row. The disagreement is always within one block, because a HOT chain never leaves its page, so what BitmapAnd needs is a per-block answer that it can read without touching the heap. Add VISIBILITYMAP_LOCATOR_SPLIT, read lock-free like ALL_VISIBLE, with functions to set, clear and test it. The visibility map widens from 2 to 4 bits per heap block to carry it, and pg_upgrade rewrites each _vm fork from the old layout, a block at a time, as it did for 9.6's frozen bit. The rewrite is gated on VISIBILITYMAP_WIDTH_CHANGE_CAT_VER, which equals the catalog version this commit installs. The bit is not part of visibility and is excluded from VISIBILITYMAP_VALID_BITS. Nothing sets the bit yet; the next commit sets it and points heap's bitmap_and_inexact callback at it. Bump catalog version.
An update that changes only some indexed attributes can preserve unchanged index entries by keeping the new tuple in the HOT chain. Record each hop's modified attributes in an inline bitmap. New entries address the new heap-only tuple; unchanged entries continue to address earlier members of the chain. Have index scans skip entries whose crossed attributes affect the index being scanned, inside the heap AM, so callers need no knowledge of the chains; a scan begun with SO_INDEX_KEEP_STALE gets them flagged instead. Give constraint fetches the arriving index too, so unrelated changes do not request an unsafe expression-key recheck under a B-tree leaf lock. Keep the key comparison for overlapping attributes: away-and-back changes can leave the key equal. Include pruning and vacuum support with the first writer. Preserve needed dead members as forwarding stubs, including their saved bitmap widths, and include existing stubs when pruning again. Resolve sibling redirects against the completed plan so no surviving anchor bypasses a changed-key hop or points at reclaimed storage. Keep pages with stubs or selective redirects out of the all-visible state. Set the visibility map locator-split bit when indexes can name different offsets for the same row. Log and replay the update and prune changes, initializing a new visibility map page before setting its split bit when redo has no applicable full-page image. When building an index, anchor each live row at the latest chain member that changed one of the index's attributes, or the root, which is where an UPDATE would have put the entry; pruning relies on that when it reclaims a member. Resolve existing anchors under the reference snapshot during concurrent index validation. Accept unchanged suffixes without inserting duplicate entries, but reject crossed changes even when the key cycles back. Adapt rewrite and bitmap scans to the new representation. Index deletion keeps all of its corruption checks except that an entry may point at a HOT-indexed heap-only tuple. Retain conservative fallbacks for catalog, replication and expression-index input changes. Cover repeated pruning, index scan order, direct anchors, concurrent index builds and unique checks here. Add recovery checks alongside the amcheck support they require in a later commit. Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>
Expose the HOT-indexed counters maintained by the write path in pg_stat_all_tables and pg_stat_all_indexes. Provide pg_relation_hot_indexed_stats() to inspect chain composition, and document the counters in monitoring.sgml. Teach pg_surgery to handle HOT-indexed tuples and collapse stubs. Cover that handling and concurrent update, vacuum and scan behavior here. Add the pageinspect and recovery suites with the following amcheck support, since those tests also verify heap integrity.
Recognize heap-only tuples with direct index anchors, xid-free collapse stubs and multiple redirects to one live tuple as valid heap representations, while continuing to check their chains. For heapallindexed, fingerprint the stored leaf keys with the TID of the snapshot-visible version reached through a valid anchor. Reject anchors that cross changes to the index, including away-and-back changes. Probe with the scanned heap version rather than a rebuilt anchor, so different valid anchors do not report missing entries and a broken chain cannot borrow coverage from another tuple. Use this path for heap handler aliases too. Validate raw TID bounds, physical links, tuple headers and bitmap bounds under the buffer share lock before walking visibility. In particular, never let an invalid block number reach a read that could extend the heap. Respect saved stub widths and ordinary off-page update links. Cover rebuilt anchors, concurrent updates and vacuum, aliases, old stub widths and corrupt keys, TIDs and chains. Check backend survival and unchanged heap storage after corruption errors. Carry selective chains and collapsed stubs through pg_upgrade as well. Add the pageinspect and recovery suites here, after both statistics and selective-aware amcheck are available. These include repeated prune replay and visibility map initialization without an applicable full-page image, with full-page writes and consistency checking both on and off, and visibility map tail truncation.
Core disqualifies HOT-indexed updates unconditionally on the logical-replication apply path: HeapUpdateHotAllowable returns HEAP_UPDATE_ALL_INDEXES whenever IsLogicalWorker() is true, because a HOT-indexed update of a replica-identity attribute leaves a stale index leaf that the apply worker's replica-identity lookups would otherwise be exposed to. This commit relaxes that unconditional bail-out with a per-subscription hot_indexed_on_apply option (subhotindexedonapply: off / subset_only (default) / always). HeapUpdateHotAllowable now consults it when running in an apply worker: "off" keeps the unconditional bail-out, "subset_only" allows the HOT-indexed path only when the relation's indexed attributes are a subset of its primary-key attributes, and "always" applies no apply-path gating. The apply worker's replica-identity lookups (see RelationFindReplTupleByIndex) cope with the stale leaf, but only when the indexed attributes are a subset of the replica identity, which is what the default subset_only enforces. Wire the option through CREATE/ALTER SUBSCRIPTION, pg_subscription, pg_dump, and psql's \dRs+, and document it (create_subscription, alter_subscription, catalogs). Cover apply under each mode (039), apply under REPLICA IDENTITY FULL and a non-PK USING INDEX whose key is cycled (040), and decoding of HOT-indexed update chains (test_decoding). Authored-by: Greg Burd <greg@burd.me>
src/test/modules/test_locator_tableam provides table AMs that store
rows exactly as heap does and differ only in the locator properties
they declare, so the parts of the locator contract that heap does
not exercise run in the regression tests:
- heap_noretain declares old_version_retained = false. A
SnapshotAny fetch by locator follows the update chain to the
row's current version, which is what an AM that updates in place
hands back, so a caller that wrongly re-fetches an old row after
an UPDATE gets the new one and the test fails. It covers AFTER
ROW UPDATE and DELETE triggers, OLD and NEW transition tables,
deferred constraint triggers, foreign key CASCADE and RESTRICT,
RETURNING OLD, MERGE, the trigger-copy spill under a small
work_mem, temp_file_limit failures, savepoint rollback, and a
cross-partition update.
- heap_wide declares a 16-byte locator, so every path that builds
an index on it must fail, and pg_indexam_has_property() reports
can_var_locator false for every in-core index AM.
- heap_bigoffset declares offsets up to MaxOffsetNumber, so the
planner must not consider bitmap scans of it and GIN must refuse
to index it.
- heap_orrecheck sets bitmap_or_inexact, so BitmapOr consults it.
The executor's version-naming bits are not exercised, since all
these AMs use heap's lock and update code. This is test scaffolding
for reviewers and is not meant to be committed.
A/B and single-variant benchmark scripts for HOT-indexed updates: build two postgres variants, run pgbench workloads exercising classic-HOT, non-HOT, and HOT-indexed paths, and a self-contained bloat probe that reports the skip count (index writes avoided on unchanged indexes) and changed-index bounding. Not for merge; kept for evaluating the feature.
There was a problem hiding this comment.
🔍 OCR found 37 issue(s).
- 25 inline, 12 in summary (inline capped at 25)
📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L67)
sed -i.bak is not portable: BSD/macOS sed treats -i.bak differently from GNU sed (BSD needs -i '' or -i .bak as a separate argument). On BSD this either errors or consumes the next arg as the suffix, which would corrupt the in-place edit of a tracked source file. Since this edits heapam_handler.c in the working tree, a portability failure here risks leaving the tree mutated. Given the README's FreeBSD-portability claim, use a temp-file + mv approach instead of relying on -i's nonstandard semantics.
Confidence: high.
📄 src/test/benchmarks/siu/scripts/run.sh (L85-L85)
find ... -mindepth is a non-POSIX GNU/BSD extension (not available in a strict POSIX find, e.g. Solaris, which PostgreSQL targets). More simply, this whole dance to empty-and-recreate a directory can be replaced with rm -rf "$datadir"; mkdir -p "$datadir" (as split_ab.sh's own start_pg already does on line 87), which is both portable and simpler. The inconsistency between the two drivers is itself a maintainability smell.
Confidence: medium.
💡 Suggested change
Before:
[ -d "$datadir" ] && find "$datadir" -mindepth 1 -delete && rmdir "$datadir"
After:
rm -rf "$datadir"; mkdir -p "$datadir"
📄 src/test/benchmarks/siu/scripts/build.sh (L12-L12)
These developer benchmark harnesses live under src/test/benchmarks/ but are not wired into any Makefile or meson.build (none exists in this directory), so they are not run by the buildfarm/meson test harness and do not behave like the rest of src/test. They also hardcode absolute paths (/scratch/siu-bench, $HOME/ws/postgres/tepid), fixed ports (57480/57481/57503), sleep 2 as startup synchronization, and GNU-only tooling. For a pgsql-hackers submission, supporting benchmark artifacts for a performance claim are normally posted to the thread, not committed under the test tree; if they are to be committed, they belong somewhere that makes clear they are not part of the regression/TAP suite, and the portability/hardcoded-path issues should be addressed. Please confirm intent.
Confidence: medium.
📄 src/test/benchmarks/siu/scripts/bloat.sh (L24-L24)
README (line 17) documents the default BENCH as /scratch/tepid-bench, but every script defaults to /scratch/siu-bench. The docs and code disagree on the single most important path a user must know. Reconcile them.
Confidence: high.
📄 src/test/benchmarks/siu/scripts/run.sh (L101-L104)
Fragile synchronization: pg_ctl ... start is launched without -w, so it returns before the postmaster accepts connections, and readiness is then assumed via a fixed sleep 2. On a loaded/slow host 2s may be insufficient and the following setup_schemas/psql_as call fails; on a fast host it wastes time. bloat.sh in this same series already does this correctly with -w start. Use pg_ctl ... -w start (and drop the sleep) so startup blocks until the server is ready. soak.sh and split_ab.sh have the identical start + sleep 2 pattern and should be fixed the same way.
💡 Suggested change
Before:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
sleep 2
}
After:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}
📄 src/test/benchmarks/siu/scripts/soak.sh (L47-L50)
Same fragile-startup pattern as run.sh: pg_ctl start without -w returns before the server accepts connections, and a fixed sleep 2 is used as the readiness barrier. Prefer pg_ctl ... -w start (as bloat.sh does) so startup blocks until ready and the sleep can be removed.
💡 Suggested change
Before:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
sleep 2
}
After:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}
📄 src/test/benchmarks/siu/scripts/split_ab.sh (L99-L102)
Same fragile-startup pattern as run.sh/soak.sh: pg_ctl start without -w plus a fixed sleep 2 as the readiness barrier. Use pg_ctl ... -w start so startup blocks until the server accepts connections and the sleep can be dropped.
💡 Suggested change
Before:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
sleep 2
}
After:
LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}
📄 src/test/benchmarks/siu/scripts/run.sh (L29-L32)
No cleanup trap is installed. With set -e, any failing step (a bad psql, a missing binary, or Ctrl-C) aborts the script mid-loop and leaves a running postmaster holding $PORT and an un-removed _data_* dir, which then breaks the next invocation. bloat.sh and split_ab.sh both install an EXIT trap that runs pg_ctl stop; run.sh (and soak.sh) should do the same so a failed/interrupted run cleans up its cluster.
📄 src/test/benchmarks/siu/scripts/soak.sh (L5-L8)
No cleanup trap: under set -e, a failing command or Ctrl-C aborts before stop_pg runs, leaving a postmaster on $PORT and an un-removed data dir that breaks the next run. Install an EXIT trap that stops the running cluster (as bloat.sh/split_ab.sh do).
📄 src/test/regress/pg_regress.c (L1246-L1246)
This execl -> execlp switch is unrelated to the table-AM lock-facts feature this patch implements, and it carries a behavioral change: execlp resolves shellprog via $PATH when it is not an absolute path, whereas execl does not. SHELLPROG is $(SHELL) (make) / /bin/sh (meson); $(SHELL) can be overridden to a bare name, in which case this now silently changes which shell is executed. If this is needed for a non-standard build environment (e.g. the added Nix setup), split it into its own commit with a rationale; otherwise drop it to keep the diff minimal.
📄 src/test/regress/sql/tsearch.sql (L776-L777)
This is a comment-only change in tsearch.sql, a file otherwise unrelated to the table-AM lock-facts feature in this patch. PostgreSQL patch hygiene penalizes rewording comments in files not functionally touched by the change, since it adds churn to the diff and forces the matching .out update. The explanation about SET-clause tracking belongs with the feature's own code/tests, not retrofitted onto the tsearch regression test. Confidence: low (style/hygiene, not correctness).
📄 src/test/subscription/t/039_hot_indexed_apply.pl (L371-L373)
The comment "The old key's leaf entry is stale after a HOT-indexed update" / "may now be stale" is only accurate for the always iteration. For off and subset_only, HeapUpdateHotAllowable() demotes the UPDATE to HEAP_UPDATE_ALL_INDEXES (tab_uk's indexed attrs {id,payload,tag} are not a subset of the PK {id}), so the old (0, 'mode_$mode') leaf is properly removed - no stale entry exists. Those two iterations therefore pass trivially and do NOT exercise the _bt_check_unique stale-leaf recheck the comment describes; only the always pass does. Consider rewording so the comment does not overstate what the off/subset_only iterations cover (they only confirm the INSERT still succeeds, not that a recheck occurred).
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
This entire file is local developer scaffolding unrelated to the HOT indexed updates feature. On pgsql-hackers a non-minimal diff that bundles unrelated tooling is a top rejection reason (patch does more than one thing). Additionally the committed config is non-portable: CompilationDatabase: build/ points at an uncommitted, local artifact (no compile_commands.json is tracked in the tree), and -I../../../../src/include only resolves from one specific subdirectory depth. This file should be dropped from the patch and kept local (e.g. via a personal gitignore).
| # ========================================================================= | ||
|
|
||
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
This file is local debugging scaffolding unrelated to the committable feature and should be dropped from the patch. Worse, the absolute line-number breakpoints are already stale against the very patch they document. For example, heap_update() in the modified heapam.c starts at the TM_Result heap_update(...) signature near line 3313, not 3210; the HOT decision block (if (hot_mode != HEAP_UPDATE_ALL_INDEXES)) is near line 4207, not 4019/4024; HEAP_INDEXED_UPDATED is set near line 4343, not 4101; and there is no 'restore on old tuple' code near 4147. Any edit to these heavily-modified files (heapam.c +712/-174, pruneheap.c +696/-47) invalidates every numeric breakpoint, making this committed documentation wrong on arrival.
| # ========================================================================= | ||
|
|
||
| # Compute indexed_attrs for HOT indexed update chain following | ||
| break indexam.c:299 |
There was a problem hiding this comment.
This breakpoint and its comment are fictional relative to the actual change. The only modification to indexam.c in this patch is a single line near 352 adding scan->xs_index_only = index_only_scan; inside index_beginscan_internal(); there is no 'indexed_attrs computation' at line 299. The comment describes code that does not exist here.
|
|
||
| # Clean commands | ||
| alias pg-clean='ninja -C "$PG_BUILD_DIR" clean' | ||
| alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"' |
There was a problem hiding this comment.
Footgun: this alias runs rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR" with no guard against unset/empty variables. These vars are only exported by the Nix devShell hook (shell.nix commonHookTail); if this script is sourced directly outside that shell, both expand to empty and the command collapses to rm -rf "" "", which resolves relative to the current directory. Guard with ${PG_BUILD_DIR:?}/${PG_INSTALL_DIR:?} (or an explicit emptiness check) before destructive deletion. The same applies to pg-init below (rm -rf "$PG_DATA_DIR") and the broad pkill -f "postgres.*-D.*$PG_DATA_DIR" in pg-stop, which with an empty var matches every running postgres.
| { | ||
| description = "PostgreSQL development environment"; |
There was a problem hiding this comment.
This flake is local developer-environment scaffolding unrelated to the HOT indexed updates feature. Bundling it into a feature patch is a non-minimal diff (the patch does more than one thing) and would draw a rejection on pgsql-hackers. It should be kept local, not committed as part of this series.
| if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber && | ||
| PageGetMaxOffsetNumber(page) == FirstOffsetNumber) | ||
| PageGetMaxOffsetNumber(page) == FirstOffsetNumber && | ||
| ((PageHeader) page)->pd_upper == | ||
| ((PageHeader) page)->pd_special - MAXALIGN(heaptup->t_len)) |
There was a problem hiding this comment.
This INIT_PAGE gating change (and the mirrored one in log_heap_update) fixes a pre-existing wal_consistency_checking hazard caused by vacuum's lp_truncate_only (PRUNE_VACUUM_CLEANUP) second pass leaving orphan tuple bytes behind an LP_UNUSED slot. Per the comment it is independent of the HOT-indexed/SIU feature this patch introduces. Bundling an unrelated crash/replay-consistency fix into a large feature patch hurts bisectability and reviewability; it should be split into its own commit (and ideally posted separately, since it is backpatchable on its own). (moderate confidence)
|
|
||
| #include "storage/itemptr.h" | ||
| #include "utils/dsa.h" | ||
| #include "utils/relcache.h" |
There was a problem hiding this comment.
The two new prototypes only need the Relation typedef, but this pulls the full relcache.h (and transitively access/tupdesc.h, common/relpath.h, nodes/bitmapset.h) into every includer of this fairly widely-included header. amlocator.h in this same patch deliberately avoids the analogous heavy include and forward-declares struct RelationData instead. Consider doing the same here (forward declare and use struct RelationData * in the prototypes) to keep the dependency minimal. Low confidence / minor.
| * This index was skipped because its key attributes did not | ||
| * change. When the overall update is a partial (selective-index) |
There was a problem hiding this comment.
The comment says the index "was skipped because its key attributes did not change", but the controlling flag ii_IndexNeedsUpdate is the wide rule documented just above (key, INCLUDE, expression, AND partial-predicate columns). An index whose INCLUDE/expression/predicate column changed would set ii_IndexNeedsUpdate=true and not reach this skip path, so "key attributes" is too narrow here. Suggest "its referenced attributes did not change" to match ii_IndexNeedsUpdate's semantics.
| * This index was skipped because its key attributes did not | |
| * change. When the overall update is a partial (selective-index) | |
| * This index was skipped because none of the attributes it | |
| * references changed. When the overall update is a partial | |
| * (selective-index) |
| indexedattrs = RelationGetIndexedAttrs(indexDesc); | ||
| indexInfo->ii_IndexNeedsUpdate = | ||
| row_moved || bms_overlap(indexedattrs, modified_idx_attrs); | ||
| bms_free(indexedattrs); |
There was a problem hiding this comment.
Performance: ExecSetIndexUnchanged() runs once per updated row (from ExecUpdateEpilogue / ExecSimpleRelationUpdate), and here calls RelationGetIndexedAttrs(indexDesc) for every index. That function always returns a fresh bms_copy() of the cached rd_indattr (by design, since the cached pointer must not be borrowed across relcache invalidation). So a bulk UPDATE of N rows on a table with M indexes performs N*M palloc+copy+bms_free cycles of an invariant bitmap on the update hot path.
The old index_unchanged_by_update() cached its per-index result via ii_CheckedUnchanged and did the work once per statement. The indexedattrs contents do not vary across rows within a statement; only modified_idx_attrs does. Consider computing/caching the indexed-attrs bitmap once per index per statement (or carrying the overlap result in IndexInfo) instead of recomputing it per row. Confidence: high.
| tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1) | ||
| lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1) |
There was a problem hiding this comment.
grep -oP uses PCRE, which is a GNU-grep extension and is not available in BSD/macOS grep. The README (lines 110-112) explicitly claims these scripts are "portable between Linux and FreeBSD", but this will fail on FreeBSD/macOS, silently producing empty tps/lat and taking the "FAILED (no tps/lat)" path on every iteration. Use a portable parser instead, e.g. awk '/tps = /{print $3}' "$log" | tail -1, consistent with how run.sh extracts TPS.
Confidence: high.
| tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1) | |
| lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1) | |
| tps=$(awk '/tps = /{print $3}' "$log" | tail -1) | |
| lat=$(awk '/latency average = /{print $4}' "$log" | tail -1) |
No description provided.