Skip to content

Tepid: only update indexes that have changed attributes - #32

Draft
gburd wants to merge 24 commits into
masterfrom
tepid
Draft

gburd wants to merge 24 commits into
masterfrom
tepid

Conversation

@gburd

@gburd gburd commented Jul 6, 2026

Copy link
Copy Markdown
Owner

No description provided.

@github-actions
github-actions Bot force-pushed the master branch 11 times, most recently from 01237c9 to 3b56d29 Compare July 7, 2026 20:47
@github-actions
github-actions Bot force-pushed the master branch 5 times, most recently from c2a5026 to d8fed9e Compare July 8, 2026 08:11
@github-actions
github-actions Bot force-pushed the master branch 9 times, most recently from c999d4e to d3a10f3 Compare July 9, 2026 08:20
@github-actions
github-actions Bot force-pushed the master branch 2 times, most recently from 14db4ac to bad9884 Compare July 9, 2026 17:14

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 31 issue(s).

  • 25 inline, 6 in summary (inline capped at 25)

📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L68)

sed -i.bak '...' is not portable: GNU sed treats -i.bak as in-place with a .bak backup suffix, but BSD/macOS sed requires a space (-i .bak) and otherwise interprets .bak as the script. The README advertises these scripts as Linux/FreeBSD-portable, but this invocation (and the matching mv "$hh.bak" "$hh") only works on GNU sed. On BSD sed this either errors or writes the backup to the wrong name, leaving the modified source in place -- compounding the in-place-edit footgun above.


📄 src/test/modules/test_locator_tableam/test_locator_tableam.c (L145-L150)

This chain-follow loop is modeled on heap_get_latest_tid(), but it drops that function's lower-bound offset check. The canonical loop guards with offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(page) precisely because it is chasing possibly-bogus t_ctid links. Here, if a ctid ever carried offset 0 (InvalidOffsetNumber), PageGetItemId(page, 0) indexes pd_linp[-1], an out-of-bounds read. In a well-formed heap chain this won't occur, but since this code exists to harden the no-retain contract against exactly such edge cases, mirror the canonical bound check.

💡 Suggested change

Before:

		if (ItemPointerGetOffsetNumber(&cur) <= PageGetMaxOffsetNumber(page))
		{
			lp = PageGetItemId(page, ItemPointerGetOffsetNumber(&cur));
			if (ItemIdIsNormal(lp))
				htup = (HeapTupleHeader) PageGetItem(page, lp);
		}

After:

		if (ItemPointerGetOffsetNumber(&cur) >= FirstOffsetNumber &&
			ItemPointerGetOffsetNumber(&cur) <= PageGetMaxOffsetNumber(page))
		{
			lp = PageGetItemId(page, ItemPointerGetOffsetNumber(&cur));
			if (ItemIdIsNormal(lp))
				htup = (HeapTupleHeader) PageGetItem(page, lp);
		}

📄 src/test/regress/pg_regress.c (L1246-L1246)

This execl -> execlp change is unrelated to the feature this patch implements and is effectively a no-op. shellprog is always SHELLPROG, which every build system defines as an absolute path ($(SHELL) i.e. /bin/sh for the makefiles, and a hardcoded /bin/sh in meson). When the first argument contains a /, execlp does no PATH search and behaves identically to execl, so this buys nothing while adding an unrelated PATH-lookup semantic. Unrelated churn like this is a top rejection reason on pgsql-hackers; drop it from this patch (or split it into its own justified commit).

💡 Suggested change

Before:

		execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

After:

		execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

📄 src/test/regress/sql/tsearch.sql (L763-L764)

The comment names the wrong function: tsvector_update_trigger() sets the column via heap_modify_tuple_by_cols() (see tsvector_op.c), not heap_modify_tuple(). Per the "comments must describe what the code does now" rule, correct the identifier (or drop the implementation-specific reference) so it does not drift from the actual code.

💡 Suggested change

Before:

-- tsvector_update_trigger() uses heap_modify_tuple() to set column 'a'
-- without going through the executor's SET-clause tracking.

After:

-- tsvector_update_trigger() uses heap_modify_tuple_by_cols() to set column 'a'
-- without going through the executor's SET-clause tracking.

📄 src/test/subscription/t/039_hot_indexed_apply.pl (L486-L486)

This test calls CREATE EXTENSION amcheck, but src/test/subscription/Makefile does not list contrib/amcheck in EXTRA_INSTALL (it only installs contrib/hstore and injection_points). Under the autoconf make check path (prove_check), the temporary install tree therefore won't contain amcheck, and this statement fails with "could not open extension control file" — a hard failure, not a clean skip. The companion src/test/recovery/Makefile was correctly updated to add contrib/amcheck; the subscription Makefile needs the same addition for 039/040.


📄 src/test/subscription/t/040_hot_indexed_replica_identity.pl (L115-L115)

CREATE EXTENSION amcheck will fail under the autoconf make check path: src/test/subscription/Makefile's EXTRA_INSTALL does not include contrib/amcheck, so the module is absent from the test install tree and this is a hard error rather than a clean skip. Add contrib/amcheck to the subscription Makefile's EXTRA_INSTALL (as src/test/recovery/Makefile already does for 061/062). Also note the minor inconsistency with 039, which uses CREATE EXTENSION IF NOT EXISTS amcheck; align the two for consistency.

SET enable_indexscan = off;
SET enable_bitmapscan = on;
-- Must return the row.
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test claims to guard the BitmapAnd-over-a-split-block path (bloom bitmap intersected with the btree bitmap, where VISIBILITYMAP_LOCATOR_SPLIT forces a union+recheck instead of an intersection), but it only asserts the result count. Nothing forces the planner to actually choose a BitmapAnd of both indexes. On a one-row table, the planner may pick a single Bitmap Index Scan on one index and apply the other qual as a heap filter/recheck, in which case the BitmapAnd-with-split-block code is never exercised and the test passes as a no-op — it would not catch a regression in the very code it claims to protect. The parallel cases in contrib/pageinspect/sql/hot_indexed_updates.sql pin the plan shape with EXPLAIN (COSTS OFF) for exactly this reason; this test should do the same to verify a BitmapAnd node is produced. (moderate confidence)

Suggested change
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;
EXPLAIN (COSTS OFF) SELECT count(*) FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;

@@ -1,4 +1,4 @@
#!/usr/bin/perl
#!/usr/bin/env perl

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shebang change is unrelated to the HOT-indexed-updates work and is the only file in the entire tree using #!/usr/bin/env perl. Every other perl script under src/, contrib/, and doc/ uses #!/usr/bin/perl (or #! /usr/bin/perl), so this deviates from a consistent, deliberate project convention. It also changes which perl is resolved at runtime (PATH-dependent instead of fixed), and is exactly the kind of unrelated churn that gets a patch rejected on -hackers and creates a needless merge conflict with upstream. Revert this to keep the diff minimal.

Suggested change
#!/usr/bin/env perl
#!/usr/bin/perl

Comment thread .clangd
Comment on lines +1 to +2
Diagnostics:
MissingIncludes: None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire file is personal developer tooling (clangd editor config with a contributor-specific CompilationDatabase path and hardcoded -I../../../../src/include / -I. include paths) that does not belong in a patch destined for pgsql-hackers/commitfest. It pollutes the shared source tree, is environment-specific, and violates minimal-diff discipline. Keep it in your local environment or add it to a personal ignore file; do not commit it as part of the feature patch.

Comment thread .gdbinit
Comment on lines +19 to +20
# Main entry: heap_update
break heapam.c:3210

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is personal developer tooling and should not be committed to the repository. Worse, it hard-codes source line numbers as breakpoints (e.g. heapam.c:3210, heapam.c:4019, heapam.c:4033, heapam.c:4101, heapam.c:4147). Line-number breakpoints are extremely brittle: the companion diffs in this change set move heap_update and the surrounding HOT code by hundreds of lines (per heapam.c, heap_update now begins near line 3298, not 3210), so these breakpoints will silently land on unrelated code. Any future edit to these files shifts them again. If GDB helpers are wanted at all, break on function names, not line numbers -- but this file should not be in the tree.

Comment thread .gdbinit
Comment on lines +88 to +89
# Redirect-with-data: initialize bitmap accumulator from collapsed redirect
break heapam_indexscan.c:182

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More hard-coded line-number breakpoints (heapam_indexscan.c:182/250/297, indexam.c:299, execIndexing.c:370, pruneheap.c:1802/1836/1863/1287/2936). The diffs to these very files in this change set relocate all of this code, so each of these breakpoints will resolve to the wrong line. Line-number breakpoints committed alongside the code they target are a guaranteed maintenance hazard.

Comment thread src/backend/utils/adt/ri_triggers.c Outdated
Comment on lines +2970 to +2971
if (scandesc->xs_entry_needs_recheck)
continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The continue path that skips stale entries has no CHECK_FOR_INTERRUPTS(). The two sibling loops that implement this exact HOT-indexed skip idiom both add one in the skip branch: nodeIndexscan.c (ExecIndexScan) and repack.c's identity probe. A long run of stale index entries for the searched key (a row whose key was cycled through many HOT-indexed updates) can be skipped here, and btree only checks for interrupts at page boundaries, so many matches on one leaf page would spin without a cancellation point. For consistency with the other probe sites and robustness, add a CHECK_FOR_INTERRUPTS() to the skip branch. (low confidence)

Suggested change
if (scandesc->xs_entry_needs_recheck)
continue;
if (scandesc->xs_entry_needs_recheck)
{
CHECK_FOR_INTERRUPTS();
continue;
}

Comment on lines +5394 to +5396
* context; the caller owns it and must bms_free when done. We never hand
* out a borrowed pointer to the cached copy because relcache invalidation
* can rebuild rd_indexcxt in place even while a refcount is held.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stated justification is inaccurate. The in-place index reload path (RelationReloadIndexInfo, used by RelationRebuildRelation when rd_indexcxt != NULL) explicitly preserves rd_indexcxt and never rebuilds it; the array fields (indkey/indexprs/indpred) that rd_indattr derives from cannot change there. The full-rebuild path destroys and swaps the whole entry rather than rebuilding rd_indexcxt "in place". So the real reason to always hand back a copy is simply that the cached bitmap is relcache-owned (freed with rd_indexcxt on full rebuild), not that the context is rebuilt in place under a held refcount. Please reword to avoid asserting behavior the code does not have (comment-accuracy, low confidence on wording, high confidence the "in place" claim is wrong).

Comment on lines +12 to +15
BENCH=${BENCH:-/scratch/siu-bench}
JOBS=${JOBS:-$( (command -v nproc >/dev/null && nproc) || sysctl -n hw.ncpu 2>/dev/null || echo 8 )}
if [ -z "${REPO:-}" ]; then
for candidate in "$HOME/ws/postgres/tepid" "$BENCH/repo" /scratch/pg; do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole directory (src/test/benchmarks/siu/) is developer benchmarking tooling, not a regression/TAP/isolation suite. It is not referenced by any Makefile or meson.build (confirmed: no benchmarks reference exists in the build system), will never run in the buildfarm, and hardcodes host-specific absolute paths (/scratch/siu-bench) and a private developer branch name (tepid, $HOME/ws/postgres/tepid). For a pgsql-hackers patch this is out-of-scope churn bundled into the feature: a patch should contain only what the change requires, and the tree has no precedent src/test/benchmarks framework to slot into. Recommend splitting these scripts out of the feature patch entirely (keep them in the -hackers thread as a reproducible-benchmark attachment instead). This applies equally to run.sh, soak.sh, split_ab.sh, bloat.sh, and the .sql workloads in this directory.

Comment on lines +64 to +68
if [ "$off" = 1 ]; then
# Force heap to advertise no split signal: BitmapAnd then always
# intersects exactly, as it did before the feature.
sed -i.bak 's/\.bitmap_and_inexact = heap_bitmap_and_inexact,/.bitmap_and_inexact = NULL,\/* split_ab: split-off *\//' \
"$hh"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does an in-place sed -i.bak edit of a tracked source file (heapam_handler.c) and relies solely on an EXIT trap to restore it. The sed target .bitmap_and_inexact = heap_bitmap_and_inexact, does match the real registration (heapam_handler.c:2955, confirmed), so the edit works -- but the restore is fragile: a SIGKILL/power loss, or any failure after git checkout --detach but before the trap's mv -f $hh.bak $hh runs, leaves the user's working tree with a silently-modified committed source file (and the build divergence is invisible unless they git status). Mutating tracked source in-place to produce a benchmark variant is a footgun against the user's own checkout. Prefer building split_off from a throwaway git worktree/git stash-applied patch or a compile-time -D guard rather than editing the file in place. (Part of the broader scoping concern: these scripts should not be in the feature patch at all.)

Comment on lines +1 to +3
-- bitmap_and_mixed: quantifies the cost of the per-block
-- VISIBILITYMAP_LOCATOR_SPLIT BitmapAnd signal specifically, not the SIU
-- write-path win the other workloads measure.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This leading comment is a dense design narrative (feature-internals justification) rather than a description of what the query does, and it carries developer-branch framing ("the split fix", "reset_state's freshly-clustered layout"). In a mailing-list patch this reads as noise. The technical terms it asserts (VISIBILITYMAP_LOCATOR_SPLIT, crossed-attribute bitmap) do match the real feature code, so it is accurate -- but it should be trimmed to a short statement of the workload's shape. Lower priority than the scoping/footgun issues; if the scripts are split out of the patch this is moot.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 29 issue(s).

  • 25 inline, 4 in summary (inline capped at 25)

📄 src/test/subscription/t/039_hot_indexed_apply.pl (L299-L302)

The ALTER-to-off reread test has no explicit synchronization that the apply worker has processed the catalog invalidation before the next batch is applied. The worker learns of the new mode via subscription_change_cb -> maybe_reread_subscription(), driven by invalidation processing between transactions; wait_for_catchup inside apply_updates_and_read only guarantees LSN catch-up, not that the worker observed the ALTER's syscache inval before applying the 81..100 batch. If the first post-ALTER transaction is applied before the invalidation is picked up, $post_alter_hotidx could be > 0 and this assertion would flake. Consider polling pg_stat_subscription/forcing a worker restart, or otherwise confirming the mode was re-read before measuring.


📄 src/test/subscription/t/039_hot_indexed_apply.pl (L492-L494)

This test ends with done_testing() but, unlike 040, never calls $subscriber->stop/$publisher->stop. Teardown works via PostgreSQL::Test::Cluster's END-block auto-stop, so this is not a correctness problem, but the inconsistency with 040 is a minor hygiene nit; consider stopping both nodes explicitly for consistency.


📄 src/test/subscription/t/039_hot_indexed_apply.pl (L193-L198)

Minor performance/maintainability nit: this fires one UPDATE per id through a separate safe_psql (hence a separate psql process/connection) for every row in the range. For a 40-wide range (case 3) this spawns 40 psql invocations per batch, and the helper is called many times across the three modes. The per-tuple pgstat counters (n_tup_upd, n_tup_hot_upd, n_tup_hot_indexed_upd) are aggregate, so a single ranged statement UPDATE $table SET payload = payload + 1 WHERE id BETWEEN $id_lo AND $id_hi produces identical counter deltas while being far cheaper.

💡 Suggested change

Before:

	for my $i ($id_lo .. $id_hi)
	{
		$publisher->safe_psql('postgres',
			"UPDATE $table SET payload = payload + 1 WHERE id = $i");
	}
	$publisher->wait_for_catchup($sub_name);

After:

	$publisher->safe_psql('postgres',
		"UPDATE $table SET payload = payload + 1 WHERE id BETWEEN $id_lo AND $id_hi");
	$publisher->wait_for_catchup($sub_name);

📄 src/test/subscription/t/039_hot_indexed_apply.pl (L2-L2)

Style nit: the file begins with a blank line before the copyright header. The PostgreSQL convention (as followed by sibling file 040) is for the # Copyright ... comment to be the first line of the TAP script. Drop the leading blank line for consistency.

💡 Suggested change

Before:


# Copyright (c) 2026, PostgreSQL Global Development Group

After:

# Copyright (c) 2026, PostgreSQL Global Development Group

Comment thread .clangd
Comment on lines +1 to +2
Diagnostics:
MissingIncludes: None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personal developer-environment scaffolding does not belong in a PostgreSQL patch destined for pgsql-hackers. .clangd, .gdbinit, flake.nix, shell.nix, pg-aliases.sh, and glibc-no-fortify-warning.patch are being committed into the repository root (they are not listed in any .gitignore). This violates the minimal-diff / one-logical-change discipline and will draw immediate rejection. These artifacts belong in a developer's local tree (e.g. a global gitignore or a personal untracked config), not in the committed patch. Drop them from the change set. [high confidence]

Comment thread .gdbinit
Comment on lines +19 to +20
# Main entry: heap_update
break heapam.c:3210

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides being out-of-scope developer tooling, this file hardcodes absolute source line numbers (e.g. heapam.c:3210, heapam.c:4019, heapam.c:4033, pruneheap.c:1802, indexam.c:299). These break the moment any line above them shifts and are meaningless outside the author's exact tree, so they cannot be maintained upstream. Remove this file from the patch. [high confidence]

@@ -1243,7 +1243,7 @@ spawn_process(const char *cmdline)
char *cmdline2;

cmdline2 = psprintf("exec %s", cmdline);
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing execl to execlp alters lookup semantics: execlp performs a PATH search when shellprog is not an absolute path. shellprog = SHELLPROG is compiled from $(SHELL), which is normally an absolute path, so this is a no-op today; but if SHELLPROG is ever a bare name, the regression harness would resolve the shell from PATH, which is a behavior and security posture change (picking up an unintended shell binary). This change is unrelated to the feature this PR delivers and lacks any justification. Split it out or revert it. [moderate confidence]

@@ -1,4 +1,4 @@
#!/usr/bin/perl
#!/usr/bin/env perl

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shebang change from #!/usr/bin/perl to #!/usr/bin/env perl is an unrelated, behavior-changing tweak to a committer-facing tool: it picks the first perl on PATH instead of the system interpreter. It is also inconsistent with the rest of the tree, which uses #!/usr/bin/perl (e.g. src/tools/copyright.pl, src/tools/git_changelog, src/tools/mark_pgdllimport.pl). Drop this hunk or submit it as a separate, justified patch. [high confidence]

Comment thread pg-aliases.sh
alias pg-perf='perf top -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1)'
alias pg-htop='htop -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | tr "\n" "," | sed "s/,$//")'

pg-stats() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond being out-of-scope tooling: pg-enable-cores/pg-disable-cores mutate the system-wide sysctl kernel.core_pattern via sudo tee, affecting every process on the host, not just this dev tree. This is a footgun with machine-global side effects. Likewise pg-test-one hardcodes --port=40099, which can collide with concurrent runs. These confirm the file is tied to the author's machine and unsuitable for the committed tree. [low confidence]

* with 2-bit _vm forks that are not rewritten. The committer re-picks both
* values at commit time, along with the copy in t/010_vm_conversion.pl.
*/
#define VISIBILITYMAP_WIDTH_CHANGE_CAT_VER 202610201

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value 202610201 is duplicated verbatim in t/010_vm_conversion.pl ($width_change_cat_ver) and the committer must also keep it equal to the final CATALOG_VERSION_NO installed by the VM-widening commit. If any of these three drift, a cluster initdb'd in the window gets its 2-bit _vm forks silently left unconverted (data-correctness hazard, not a hard error). This is a genuine foot-gun, so it is worth adding a hard cross-check: in a backend build the constant could be StaticAssert'd against CATALOG_VERSION_NO, and 010_vm_conversion.pl could read the value from the installed header rather than re-hardcoding it, to prevent the two from diverging unnoticed. (Confidence: moderate; the risk is real but the comment already flags it as committer-owned.)

Comment on lines +22 to +23
TEPID_REV=${TEPID_REV:-tepid}
MASTER_REV=${MASTER_REV:-$(git merge-base "$TEPID_REV" origin/master 2>/dev/null || git merge-base "$TEPID_REV" master)}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These benchmark scripts are hardwired to a private development branch and a developer's personal paths: build.sh defaults REPO to $HOME/ws/postgres/tepid and /scratch/pg, TEPID_REV to tepid, and bakes -siubench/tepid into the build; split_ab.sh and the README do the same. Nothing in the tree (no Makefile/meson rule) references src/test/benchmarks, so this is a brand-new directory of unhooked, machine-specific tooling. For a patch destined for pgsql-hackers this fails the minimality/patch-hygiene bar: a committed benchmark should be self-contained and reproducible for anyone, not reference a committer's home directory or an unpublished branch name. Recommend dropping this harness from the patch (keep it on the thread as an attachment) or rewriting it to be branch/path agnostic.

echo "=== building $name ($rev) into $prefix"
rm -rf "$prefix"
mkdir -p "$prefix"
git checkout --quiet --detach "$rev"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_variant runs git checkout --quiet --detach "$rev" on the developer's live working tree and relies solely on an EXIT trap to restore the original HEAD. A hard kill (SIGKILL/OOM/power loss) during the long meson compile leaves the user's repo on a detached HEAD at MASTER/TEPID with no restore -- the same working-tree-mutation footgun already flagged for split_ab.sh's sed -i.bak. Prefer git worktree add into $BENCH (or git archive | tar -x) so the checkout under benchmark never touches the user's working copy.

Comment on lines +102 to +104
-o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
sleep 2
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start_pg waits for the server with a bare sleep 2 rather than polling readiness. On a slow/loaded host initdb+start can take longer than 2s, so the immediately-following setup_schemas/psql_as can hit "the database system is starting up" and fail; on a fast host it needlessly wastes time. pg_ctl already supports -w (wait for ready) -- use pg_ctl ... -w start and drop the sleep. Same pattern in soak.sh and split_ab.sh.

Comment on lines +90 to +93
local t=0
while [ "$t" -lt "$DURATION" ]; do
sleep "$SAMPLE"
t=$((t + SAMPLE))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sampler loop advances t in fixed $SAMPLE steps but pgbench runs for exactly $DURATION seconds. When $DURATION is not an integer multiple of $SAMPLE (e.g. the default 900/60 happens to divide, but any override need not), the final tick either overshoots past pgbench exit (sampling a quiesced server, so the last row reports a misleadingly low write rate) or truncates the last window. Either compute the number of ticks as DURATION/SAMPLE and guard the final partial interval, or break the loop as soon as pgbench ($pgb) has exited.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 31 issue(s).

  • 25 inline, 6 in summary (inline capped at 25)

📄 src/test/benchmarks/siu/scripts/split_ab.sh (L134-L135)

grep -oP (Perl regex, \K) is GNU-grep-specific and absent from BSD/macOS grep, where -P is unsupported. On those platforms both extractions return empty, tps/lat are blank, and every iteration takes the "FAILED (no tps/lat)" path, silently producing an empty summary. The sibling run.sh extracts the same pgbench fields with portable awk; use the same approach here for consistency and portability. (high confidence)

💡 Suggested change

Before:

  tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1)
  lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1)

After:

  tps=$(awk '/tps = /{print $3}' "$log" | tail -1)
  lat=$(awk '/latency average = /{print $4}' "$log" | tail -1)

📄 src/test/benchmarks/siu/scripts/run.sh (L36-L37)

Doc/code drift and a latent failure: the header documents the default as "0,1,4,8,WIDE_COLS", but the code hardcodes 16. If a user overrides WIDE_COLS (e.g. to 8) without also overriding WIDE_STEPS, the wide_16 step calls build_wide_set_clause 16, producing c9=:v,...,c16=:v for columns that seed_wide_table never created, and pgbench aborts the run. Either derive the default from WIDE_COLS (as the comment claims) or validate each step is <= WIDE_COLS. (moderate confidence)

💡 Suggested change

Before:

WIDE_COLS=${WIDE_COLS:-16}
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,16}

After:

WIDE_COLS=${WIDE_COLS:-16}
WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,$WIDE_COLS}

📄 src/test/modules/test_locator_tableam/sql/noretain.sql (L186-L186)

These three pg_ls_tmpdir() checks assume the server's temp directory reflects only this session's spill files. pg_ls_tmpdir() lists every temp file in the current tablespace's pgsql_tmp, including those created concurrently by other backends (e.g. an autovacuum worker that spills), and autovacuum is on by default even in the pg_regress test instance. That makes two of these checks fragile:

  • Line 186 sizes temp_file_limit as current temp-dir size + 40 KB. An unrelated temp file present at that instant inflates the limit, so the subsequent DO block may not hit configuration_limit_exceeded, and the expected NOTICE: spill failed would not fire.
  • Line 210 compares two successive directory totals; if another backend's temp file appears or disappears between the two measurements, released can flip.

The count(*) > 0 check at line 154 is robust, but the size-based ones are not deterministic. Consider restricting to this backend's own temp files (e.g. filtering by the backend PID prefix in the temp file name) or asserting the property in a way that does not depend on a shared directory snapshot.


📄 src/test/regress/regress.c (L1761-L1764)

result_names[] is a hand-maintained parallel array to the TM_Result enum (tableam.h) with no compile-time link. The runtime check EXPECT_TRUE(result >= TM_Ok && result <= TM_WouldBlock) validates the enum range but silently assumes this array has exactly TM_WouldBlock + 1 entries and in matching order. If TM_Result ever gains a value, this array goes stale with no build failure: a value inserted before TM_WouldBlock yields a wrong name, and one appended after it indexes out of bounds past the range guard. Add a StaticAssertStmt(lengthof(result_names) == TM_WouldBlock + 1, ...) to tie the two together.


📄 src/test/regress/regress.c (L1767-L1767)

cid = GetCurrentCommandId(false) - PG_GETARG_INT32(2) can underflow: CommandId is unsigned, so a cid_delta larger than the current command counter (or a negative int32) wraps to a huge value. The in-tree callers only pass 0 and 1 so this is safe today, but since the function is a general exported test helper with no validation it is an easy footgun for future callers. Consider rejecting a delta that exceeds the current command id, or at least documenting the precondition in the header comment.


📄 src/test/recovery/t/061_hot_indexed_vm.pl (L98-L102)

This comment describes the existing table (the else branch), but it sits directly above the if ($fpw eq 'on' && $update eq $updates[0]) block. Since UPDATE fresh is logged before UPDATE existing, $updates[0] is the fresh table's record, which is what the if branch asserts carries the applied image. A reader naturally attaches this comment to the if case it precedes, making it read as if existing is $updates[0]. Move the explanation into the else branch (or reword to name fresh as the image-carrying case) so it matches the code it governs.

Comment thread .gdbinit
Comment on lines +19 to +20
# Main entry: heap_update
break heapam.c:3210

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These hard-coded source line numbers do not match the current tree and will mislead anyone who sources this file. Verified against the actual sources:

  • heapam.c:3210 is inside heap_delete's toast handling (relkind != RELKIND_MATVIEW), not the heap_update entry.
  • heapam.c:4019/4024/4033 point at a comment, a blank line, and XLogRegisterBuffer() in the tuple-lock path -- not the "pure HOT / HOT-indexed / size check" decisions.
  • heapam_indexscan.c:182 is IndexScanHeapData *hscan = ... in heapam_index_scan_end; :250 is *all_dead = first_call;; :297 is a break for a bogus TID -- none are the described bitmap-accumulation points.
  • pruneheap.c:1287 is a do_prune = ... assignment, not the "execute phase".
    Line-number breakpoints are stale on the very first code edit and cannot be reviewed. This is personal debugging scaffolding that should not be committed to a tree destined for pgsql-hackers/commitfest; keep it out of the series (local gitignore).

@@ -1,4 +1,4 @@
#!/usr/bin/perl
#!/usr/bin/env perl

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated change that is inconsistent with the tree-wide convention. Every other Perl script in the repository (30+ scripts, incl. all of src/tools/*.pl, genbki.pl, gen_node_support.pl, etc.) uses #!/usr/bin/perl; after this change pgindent is the only script using #!/usr/bin/env perl. This is a committer-owned tooling script and the shebang change is not required by the HOT-indexed-updates feature -- it should not ride along with a feature patch and should be dropped to keep the diff minimal.

Suggested change
#!/usr/bin/env perl
#!/usr/bin/perl

Comment thread pg-aliases.sh
Comment on lines +576 to +580
pg-enable-cores() {
ulimit -c unlimited
if ! [ -w /proc/sys/kernel/core_pattern ]; then
echo "Setting kernel.core_pattern (requires sudo)..."
echo "core.%p" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pg-enable-cores/pg-disable-cores mutate the system-wide kernel.core_pattern sysctl via sudo, a machine-global side effect that is surprising to ship in a repository helper script. Combined with the many unquoted/edge-case constructs elsewhere (e.g. basename $CC, pkill -f "postgres.*-D.*$PG_DATA_DIR", find ... -mtime +$days -delete), this is personal developer tooling rather than project infrastructure and should not be committed to a tree intended for pgsql-hackers submission.

Comment thread shell.nix
# into system glibc or release builds.
patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: {
patches = (oldAttrs.patches or []) ++ [
./glibc-no-fortify-warning.patch

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This devShell patches glibc with ./glibc-no-fortify-warning.patch. The patch file does exist at the repo root, but it is not part of this change set / review group -- make sure it is committed together with these files, otherwise flake evaluation of every glibc devShell (default, gcc, clang) fails with a missing-path error. More broadly, this entire Nix dev environment is personal tooling unrelated to the HOT-indexed-updates feature and will draw an immediate "patch does more than one thing" rejection on pgsql-hackers; keep it out of the submitted series.

Comment thread flake.nix
inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript;
};

environment.localBinInPath = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

environment.localBinInPath = true; is a NixOS module option, but here it is placed inside a flake-utils.lib.eachDefaultSystem per-system output set. At the flake-output level this attribute is never interpreted as NixOS configuration; it is a stray/dead attribute that does nothing. Either drop it or move it into an actual NixOS module. (moderate confidence)

Comment on lines +183 to +186
*)
echo "reset_state: unknown table $table" >&2
return 1
;;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reset_state returns 1 on an unknown table, but every call site invokes it bare under set -e, so a genuine mismatch aborts the whole run rather than being handled - while the surrounding pgbench steps are wrapped in || true and set +e/set -e toggles, making the error-handling policy inconsistent (some failures abort, some are swallowed). A failed reseed between workloads would carry bloat/index-size state forward and silently skew the per-workload deltas the CSV reports. Consider making reset failures consistently fatal and verifying the reseed succeeded before measuring.


start_pg() {
local v=$1 datadir=$BENCH/_data_$v
[ -d "$datadir" ] && find "$datadir" -mindepth 1 -delete && rmdir "$datadir"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same destructive-deletion footgun as run.sh: find "$datadir" -mindepth 1 -delete && rmdir with datadir=$BENCH/_data_$v. An empty exported BENCH passes set -u and yields datadir=/_data_master. Guard that $datadir is nonempty and under $BENCH before deleting.

@@ -0,0 +1,430 @@
#!/usr/bin/env bash
# A/B pgbench harness for tepid: master (upstream) vs tepid (HOT-indexed).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These benchmark scripts are not referenced by any Makefile or meson.build (confirmed: no build/test wiring points at src/test/benchmarks/siu/). They are also not deterministic regression/TAP tests and cannot run in CI (they require a pre-built two-variant layout under $BENCH and specific git revisions master/tepid). For a pgsql-hackers submission, performance claims need a reproducible benchmark, but an ad hoc harness carrying developer-specific variant names ("tepid"), absolute paths, and a git merge-base ... origin/master default is unusual to commit to the tree and has no established src/test/benchmarks/ precedent. Consider keeping these out of the patch itself and instead posting them / their numbers to the design thread; if they must be in-tree, the README should cite the pgsql-hackers Message-Id for the SIU design discussion they support.

@@ -1243,7 +1243,7 @@ spawn_process(const char *cmdline)
char *cmdline2;

cmdline2 = psprintf("exec %s", cmdline);
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This execl -> execlp change is unrelated to the HOT-indexed (SIU) feature this patch implements and should be dropped to keep the diff minimal (a top rejection reason on -hackers). shellprog is only ever set from the compile-time SHELLPROG macro, which is always an absolute path ($(SHELL) from the Makefile, /bin/sh from meson). For an absolute path execlp behaves identically to execl, so this provides no benefit here. Worse, it introduces a footgun/behavior change: if shellprog ever lacked a slash, execlp would perform a $PATH search, making the test harness resolve the shell non-deterministically and from an attacker-influenceable environment. If there is a genuine need (e.g. a relative shell on some platform), it belongs in a separate, justified commit with that rationale; otherwise revert to execl. (high confidence)

Suggested change
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

Comment on lines +86 to +87
local v=$1 datadir=$BENCH/_data_split_$v
rm -rf "$datadir"; mkdir -p "$datadir"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same destructive-deletion footgun confirmed for run.sh/soak.sh exists here, and is worse: rm -rf "$datadir" with datadir=$BENCH/_data_split_$v. An explicitly-exported empty BENCH='' passes set -u, yielding datadir=/_data_split_split_off and an rm -rf against an absolute root-level path. Guard that $BENCH is non-empty (and ideally that $datadir lives under $BENCH) before deleting. (high confidence)

Suggested change
local v=$1 datadir=$BENCH/_data_split_$v
rm -rf "$datadir"; mkdir -p "$datadir"
local v=$1
[ -n "${BENCH:-}" ] || { echo "split_ab: BENCH must be set and non-empty" >&2; exit 1; }
local datadir=$BENCH/_data_split_$v
rm -rf "$datadir"; mkdir -p "$datadir"

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 35 issue(s).

  • 25 inline, 10 in summary (inline capped at 25)

📄 src/include/access/heapam.h (L180-L180)

xs_prefix_all_dead is written but never read. heap_hot_search_buffer() stores into hscan->xs_prefix_all_dead (heapam_indexscan.c:772) and resets it (:776), but no code consumes the struct field. The only leaf-kill path, heapam_index_kill_item(), keys solely off all_dead/kill_prior_tuple and never inspects xs_prefix_all_dead; the stale-entry consumers (heapam_index_entry_needs_recheck callers) only continue, they do not kill the arriving leaf. So the documented contract here ("Combined with a stale verdict, this lets the index-access layer kill the arriving leaf") describes behavior that is not implemented. This is dead/speculative scaffolding on an exported struct, and the comment is aspirational rather than descriptive. Either wire up the kill-stale-leaf optimization that reads this field, or drop the field (and the corresponding prefix_all_dead out-param of heap_hot_search_buffer) until it has a consumer. Confidence: high.


📄 src/test/benchmarks/siu/scripts/run.sh (L31-L32)

These are developer-local scratch scripts, not part of the automated test suite. They hardcode absolute paths (/scratch/siu-bench), assume a tepid git branch exists, require meson, and are not wired into any Makefile/meson target under src/test/ (verified: src/test/benchmarks/ is introduced only by this change and nothing in src/test/Makefile/meson.build references it). pgsql-hackers routinely rejects committing out-of-tree scratch tooling of this kind. If benchmark reproducibility must be preserved, attach it to the -hackers thread rather than the tree, or at minimum document that it is non-CI, path-portable tooling and remove the hardcoded scratch defaults. (medium, high confidence)


📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L67)

This mutates tracked source in place: sed -i.bak rewrites src/backend/access/heap/heapam_handler.c (the target string .bitmap_and_inexact = heap_bitmap_and_inexact, matches heapam_handler.c:2947, so the edit is real and effective), and restoration depends solely on an EXIT trap. A SIGKILL, an OOM kill during meson compile, or any trap that fires before mv "$hh.bak" "$hh" leaves the checkout with a patched heapam_handler.c and/or a dangling .bak — silently corrupting the developer's working tree. Editing tracked backend source as a side effect of a benchmark is a footgun; prefer building the split_off variant from a committed branch/patch or a runtime toggle rather than an in-place sed on tracked code. (medium, high confidence)


📄 src/test/benchmarks/siu/scripts/build.sh (L38-L40)

Destructive, env-var-driven operations with no confirmation: rm -rf "$prefix" where prefix=$BENCH/$name, plus git checkout --quiet --detach "$rev" which silently moves the user off their branch. A mistyped/empty BENCH (e.g. if the :- default were ever removed, or name is empty) makes rm -rf target an unintended directory, and the branch restore relies on an EXIT trap that does not fire on SIGKILL/crash — leaving the repo detached. Data-loss hazard; at minimum validate BENCH/name are non-empty before rm -rf, and document that an interrupted run can leave a detached HEAD. (medium, high confidence)


📄 src/test/benchmarks/siu/scripts/run.sh (L102-L104)

Fixed sleep 2 after pg_ctl ... start (which is not even invoked with -w) is brittle synchronization: on a slow/loaded machine the server may not be accepting connections yet when setup_schemas/psql_as runs, causing spurious failures. Use pg_ctl -w start and/or poll with pg_isready before issuing psql. Same pattern exists in soak.sh and split_ab.sh. (low, high confidence)


📄 src/test/benchmarks/siu/scripts/run.sh (L37-L37)

The default WIDE_STEPS hardcodes the literal 16, contradicting the documented default "0,1,4,8,WIDE_COLS" at the top of the file (line 24). If a user overrides WIDE_COLS to anything smaller than 16 but leaves WIDE_STEPS at its default, the wide_16 step causes build_wide_set_clause 16 to emit c1=:v,...,c16=:v while seed_wide_table only created columns c1..cWIDE_COLS. pgbench then issues UPDATE wide_table SET ..., c16=:v ... referencing nonexistent columns, which errors at runtime. Derive the final step from $WIDE_COLS so the two stay in sync.

💡 Suggested change

Before:

WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,16}

After:

WIDE_STEPS=${WIDE_STEPS:-0,1,4,8,$WIDE_COLS}

📄 src/test/modules/index/t/001_concurrent_hot.pl (L10-L11)

The skip guard only checks the enable_injection_points env var, but line 18 then hard-depends on CREATE EXTENSION injection_points succeeding. Every other injection-point TAP test in this tree (e.g. test_misc/t/010_index_concurrently_upsert.pl) adds a second guard after starting the node:

plan skip_all => 'Extension injection_points not installed'
  unless $node->check_extension('injection_points');

Without it, an environment where the env var is set but the extension files are not actually installed fails with a hard error instead of skipping cleanly. Recommend adding the check_extension guard for consistency and clean skipping. (low confidence / low severity)


📄 src/test/recovery/t/061_hot_indexed_vm.pl (L100-L103)

Fragile identity check: $update eq $updates[0] uses string equality to mean "this is the first WAL record". This works only because the two HOT_UPDATE lines differ textually (different rel/block/LSN). If a future rmgrdesc change ever made the two lines textually identical, both loop iterations would take the if branch and the existing-table assertion (the else/unlike check that redo must replay the delta) would silently never run, turning a real regression into a passing test. Prefer an index-based loop so the intent is explicit and immune to duplicate-line matches.

💡 Suggested change

Before:

		for my $update (@updates)
		{
			if ($fpw eq 'on' && $update eq $updates[0])
			{

After:

		for my $i (0 .. $#updates)
		{
			my $update = $updates[$i];
			if ($fpw eq 'on' && $i == 0)
			{

📄 src/tools/pgindent/pgindent (L1-L1)

This shebang change is unrelated to the PR's purpose (the HOT-indexed/TAM feature) and should be dropped from the patch. Unrelated changes are a top rejection reason on pgsql-hackers. More importantly, it diverges from the established convention: every other Perl script in the tree (30+ scripts, including the sibling src/tools/copyright.pl and src/tools/mark_pgdllimport.pl) uses #!/usr/bin/perl. If a portability improvement for pgindent is genuinely desired, it belongs in its own dedicated patch applied consistently across the tree, with a reference to the -hackers discussion. Confidence: high.

💡 Suggested change

Before:

+#!/usr/bin/env perl

After:

#!/usr/bin/perl

📄 src/test/subscription/t/039_hot_indexed_apply.pl (L379-L380)

like($err, qr/^$/, ...) can intermittently fail: psql may emit non-error chatter on stderr (e.g. a NOTICE, or a libpq connection notice) that has nothing to do with the INSERT succeeding. The adjacent is($r, 0, ...) already asserts the command succeeded via the exit code, which is the reliable signal. The empty-stderr regex is both redundant with the return-code check and a portability/flakiness footgun. Consider dropping this assertion (or using on_error_stop + checking only $r).

Comment on lines +115 to +119
SET enable_seqscan = off;
SET enable_indexscan = off;
SET enable_bitmapscan = on;
-- Must return the row.
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test asserts it "guards" the BitmapAnd union-and-recheck path, but it never verifies the plan actually is a BitmapAnd. On a single-row, un-ANALYZEd table the planner is free to pick a single Bitmap Index Scan on one index with the other predicate applied as a recheck/filter, in which case the row is still returned (count = 1) while none of the bitmap_and_inexact code the comment describes is exercised. The sibling test (test_locator_tableam/sql/bitmap_hooks.sql) and the existing bloom tests above (lines 24-26, 68-70) all pin the plan with EXPLAIN (COSTS OFF). Add an EXPLAIN (COSTS OFF) for this query to confirm a BitmapAnd is chosen, and disable enable_indexonlyscan / set max_parallel_workers_per_gather = 0 so the plan is deterministic across machines. (high confidence)

Suggested change
SET enable_seqscan = off;
SET enable_indexscan = off;
SET enable_bitmapscan = on;
-- Must return the row.
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;
SET enable_seqscan = off;
SET enable_indexscan = off;
SET enable_indexonlyscan = off;
SET enable_bitmapscan = on;
SET max_parallel_workers_per_gather = 0;
-- Must use a BitmapAnd of the two indexes...
EXPLAIN (COSTS OFF) SELECT count(*) FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;
-- ...and must return the row.
SELECT count(*) AS bloom_bitmapand FROM bloom_hot_indexed WHERE bcol = 11 AND changed = 22;

Comment on lines +115 to +117
SET enable_seqscan = off;
SET enable_indexscan = off;
SET enable_bitmapscan = on;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These GUCs are left changed when the block ends only via the matching RESETs below, but if the SELECT errors out the session-level GUC state leaks into subsequent tests in the same file. Minor, but the surrounding file convention (lines 12-14, 20-22, 149-151) is consistent SET/RESET pairing; this is fine as-is. Lower-priority note: consider whether fillfactor = 50 and the explicit id/changed btree indexes are all load-bearing for reproducing the HOT-indexed update, or trim to the minimal schema needed. (low confidence)

Comment thread .clangd
ParameterNames: true
DeducedTypes: true
CompileFlags:
CompilationDatabase: build/ # Search build/ directory for compile_commands.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is developer-local IDE scaffolding (clangd config pinned to build/, -I. and -I../../../../src/include) that assumes one specific working directory and build layout. It is not part of the PostgreSQL source/test/doc change and does not belong in the tree. For a patch destined for pgsql-hackers/commitfest, committing personal editor config to the repo root is out-of-scope churn that will be rejected. Keep it in a personal (untracked) gitignore instead.

Comment thread .gdbinit
Comment on lines +19 to +20
# Main entry: heap_update
break heapam.c:3210

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These breakpoints are pinned to hardcoded line numbers in heavily-modified files and are already stale. For example break heapam.c:3210 is annotated "Main entry: heap_update", but heapam.c:3210 is actually a relkind check in the toast-deletion portion of a different function; and break pruneheap.c:1802 is annotated as chain-collapsing bitmap collection, but line 1802 is priorXmax = HeapTupleHeaderGetUpdateXid(htup); at the tail of a chain-advance loop. Line-number breakpoints drift the instant any of these files change, so they silently land on the wrong code. Beyond the correctness drift, this is personal debugging scaffolding that does not belong in the PostgreSQL tree.

Comment thread pg-aliases.sh
alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"'

# Database management
alias pg-init='trash "$PG_DATA_DIR" 2>/dev/null || rm -rf "$PG_DATA_DIR"; "$PG_INSTALL_DIR/bin/initdb" --debug --no-clean "$PG_DATA_DIR"'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pg-init does rm -rf "$PG_DATA_DIR" on a value derived entirely from an environment variable. If PG_DATA_DIR is unset or empty when this file is sourced outside the Nix shell, this expands to rm -rf "" (or an unexpected path), a silent data-loss footgun. The same unguarded pattern appears in pg-full-clean. Committing shell helpers that rm -rf env-derived paths with no non-empty guard is unsafe; and more broadly this personal shell library does not belong in the PostgreSQL tree.

Comment thread src/backend/executor/nodeModifyTable.c Outdated
* Return a Bitmapset that contains the set of modified (changed) indexed
* attributes between oldtup and newtup.
*
* Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that operates

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment line runs well past the ~78-column limit the tree uses for block comments (the surrounding lines wrap at ~78). pgindent will not fix comment wrapping, so this should be rewrapped by hand to stay consistent.

(low confidence / style)

Suggested change
* Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that operates
* Note: There is a similar function called HeapUpdateModifiedIdxAttrs() that
* operates

Comment on lines +5472 to +5476
if (!isnull)
{
indexprs = (List *) stringToNode(TextDatumGetCString(datum));
pull_varattnos((Node *) indexprs, 1, &attrs);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the first call for a given index relcache entry (before the cached rd_indattr exists), the parsed node trees from stringToNode() and the TextDatumGetCString() strings are palloc'd in the caller's current memory context and never freed. Callers such as execIndexing.c and heapam_indexscan.c invoke this on per-tuple paths; the first invocation may land in a long-lived context, leaking the parse tree until that context resets. Existing helpers like RelationGetIndexExpressions build such trees in rd_indexcxt. Consider doing the parse/pull_varattnos work in a short-lived context (or pfreeing the intermediate trees and CStrings) before returning, so nothing leaks into the caller's context.

Comment on lines +2239 to +2244
RelationIdUpdatesInPlace(Oid relationId)
{
Relation rel;
bool result;

RelationIdCacheLookup(relationId, rel);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header comment states "the caller must hold a lock on the relation," but neither path enforces or documents it via an assertion. The sibling RelationIdGetRelation() calls AssertCouldGetRelation() precisely to catch the no-lock misuse this comment warns about. Since this is a new exported API advertised as a planner hot path, add AssertCouldGetRelation() (matching RelationIdGetRelation) so a missing lock is caught in assert builds rather than risking a TOCTOU-style race.

* with 2-bit _vm forks that are not rewritten. The committer re-picks both
* values at commit time, along with the copy in t/010_vm_conversion.pl.
*/
#define VISIBILITYMAP_WIDTH_CHANGE_CAT_VER 202610201

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hardcoded value is stale relative to the tree it ships in and is a silent data-corruption footgun. The current CATALOG_VERSION_NO in this branch is 202611121, which is already greater than VISIBILITYMAP_WIDTH_CHANGE_CAT_VER (202610201). The rewrite guard in transfer_relfile()/check_new_cluster() uses cat_ver < VISIBILITYMAP_WIDTH_CHANGE_CAT_VER, so:

  • A cluster that still has the 2-bit VM but whose catversion is >= 202610201 (e.g. anything initdb'd in this branch after 202610201 but before the widening actually landed) is NOT rewritten, and its 2-bit _vm forks are then read as 4-bit -> wrong all-visible/all-frozen bits -> index-only scans return wrong rows and VACUUM can advance relfrozenxid incorrectly.
  • A cluster that already has the 4-bit VM but a catversion < 202610201 is rewritten (doubled) a second time -> corrupt _vm.

The value must exactly equal the CATALOG_VERSION_NO installed by the commit that actually widens the VM; as written it is a guess that is already contradicted by the branch's own catversion. Verify the widening commit's final catversion and set this (and the duplicate literal in t/010_vm_conversion.pl line 42) to that exact number. Confidence: high.

Comment on lines +5412 to +5414
/* Fast path: return a copy of the cached bitmap. */
if (indexRel->rd_indattr != NULL)
return bms_copy(indexRel->rd_indattr);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty indexed-attr set permanently defeats the cache and repeats the per-call leak. For an index whose referenced-attribute set is empty (e.g. CREATE INDEX ON t ((1)) or any constant-expression index), attrs is NULL, so bms_copy(attrs) stores NULL into rd_indattr. The fast-path check if (indexRel->rd_indattr != NULL) then never fires, so every subsequent call re-runs stringToNode()/TextDatumGetCString() and leaks those allocations in the caller's context (the confirmed leak finding is about the first call; this makes it every call). Since this runs per-index per-row on the HOT-indexed write path (execIndexing.c) and per index scan (heapam_indexscan.c), the caching optimization silently never engages for such indexes. Use a separate bool to mark the cache as populated (or cache an empty-but-non-NULL sentinel) rather than relying on a non-NULL pointer.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 31 issue(s).

  • 25 inline, 6 in summary (inline capped at 25)

📄 src/test/benchmarks/siu/scripts/build.sh (L15-L15)

Portability/minimalism (high confidence): these benchmark drivers are committed under src/test/benchmarks but hardcode machine-specific absolute paths and a private branch name as defaults -- $HOME/ws/postgres/tepid, /scratch/pg, BENCH=/scratch/siu-bench, and TEPID_REV=tepid. A benchmark that only runs on the author's directory layout and branch cannot be reproduced by a reviewer and should not land in a patch. Make REPO/BENCH required (no silent default) or derive REPO from the script location, and drop the tepid branch default (it is meaningless outside the author's clone).


📄 src/test/benchmarks/siu/scripts/run.sh (L85-L86)

Data-loss hazard / robustness (low confidence): find "$datadir" -mindepth 1 -delete && rmdir "$datadir" unconditionally wipes $BENCH/_data_$v. If BENCH is misconfigured to point at a real directory (it is user-supplied and defaults to a hardcoded /scratch path), this deletes it with no confirmation. Separately, server readiness after pg_ctl ... start is handled with a fixed sleep 2 rather than polling (e.g. pg_ctl -w start or pg_isready), which is racy on a busy/slow host and can cause the first psql/pgbench to fail intermittently. The same pattern is duplicated in soak.sh.


📄 src/test/modules/index/t/002_unique_hot.pl (L131-L133)

The final NULLS DISTINCT block creates table t but, unlike every case in the loop above (which ends with DROP TABLE t), never drops it before $node->stop. Harmless today since the node is torn down immediately, but it breaks the "DROP objects the test creates" discipline and would silently collide if another case were appended after this block. Add a cleanup for consistency.

💡 Suggested change

Before:

is($node->safe_psql('postgres', 'SELECT count(*) FROM t'),
	'2', 'NULLS DISTINCT permits a second NULL expression key');
$node->stop;

After:

is($node->safe_psql('postgres', 'SELECT count(*) FROM t'),
	'2', 'NULLS DISTINCT permits a second NULL expression key');
$node->safe_psql('postgres', 'DROP TABLE t');
$node->stop;

📄 src/bin/pg_upgrade/file.c (L335-L338)

Per c.h, StaticAssertDecl() "doesn't need to be used in new code" and is a declaration that must sit where a variable declaration is valid. These are placed correctly (at the top of the function block, before statements), so this compiles, but the modern idiom is a plain static_assert(). Minor style nit, not a blocker.


📄 src/test/benchmarks/siu/scripts/soak.sh (L84-L88)

Robustness: this script runs under set -euo pipefail and backgrounds a long-running pgbench (pgb=$!) and a postmaster, but installs no EXIT/ERR trap to tear them down. Any transient failure of a psql_as call inside the sampling loop (lines 95-101) aborts the whole script and leaves the backgrounded pgbench and the started postmaster orphaned (still holding the port and datadir). The sibling drivers run.sh and bloat.sh guard this with a trap '... pg_ctl ... stop' EXIT; soak.sh should do the same (and kill $pgb).


📄 src/test/regress/pg_regress.c (L1246-L1246)

Unrelated, unexplained change: switching execl to execlp is not required by this patch (the HOT-indexed table-AM feature) and is a minimal-diff violation. shellprog is SHELLPROG, which is hardcoded to an absolute path (/bin/sh under meson, $(SHELL) under make, normally absolute). execlp only differs from execl when the program name contains no slash, in which case it does a PATH lookup. Since no accompanying Makefile/meson change makes SHELLPROG a bare name, this change has no effect for the normal build and only introduces a subtly different (PATH-dependent, less predictable) shell-resolution behavior with no test and no rationale. Drop this hunk, or if it is genuinely needed (e.g. for a Nix dev environment), split it into its own patch with the corresponding build-system change and a clear justification. [high confidence]

💡 Suggested change

Before:

		execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

After:

		execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);

Comment thread .gdbinit
Comment on lines +27 to +29
break heapam.c:4019
break heapam.c:4024
break heapam.c:4033

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These hard-coded absolute source line breakpoints are brittle and already wrong against the committed tree. For example, this block claims break heapam.c:4019 is the "pure HOT (no indexed columns changed)" decision, but heapam.c:4019 is actually if (PageIsAllVisible(page)) inside the tuple-lock WAL path, unrelated to any HOT decision. Line-number breakpoints drift with every edit and are meaningless outside the author's working tree. Use function-name breakpoints (as the rest of this file already does), or better, do not commit a personal .gdbinit at all.

Comment thread .gdbinit
break heap_hot_search_buffer

# Redirect-with-data: initialize bitmap accumulator from collapsed redirect
break heapam_indexscan.c:182

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed stale: break heapam_indexscan.c:182 is annotated as "Redirect-with-data: initialize bitmap accumulator from collapsed redirect", but line 182 is inside heapam_index_scan_end() (IndexScanHeapData *hscan = ...), which has nothing to do with redirect-with-data accumulation. The remaining line-number breakpoints (heapam_indexscan.c:250/297, indexam.c:299, execIndexing.c:370, pruneheap.c:1802/1836/1863/1287/2936) have the same problem. Replace with symbol breakpoints or drop this file from the patch.

Comment thread shell.nix
Comment on lines +15 to +19
patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: {
patches = (oldAttrs.patches or []) ++ [
./glibc-no-fortify-warning.patch
];
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dev-shell patches glibc's features.h via ./glibc-no-fortify-warning.patch, a workaround scoped to one contributor's Nix environment. Along with flake.nix/pg-aliases.sh/.clangd/.gdbinit, this is developer-local scaffolding unrelated to the HOT-indexed-updates feature and should not be committed to the PostgreSQL source tree. It needlessly expands the patch footprint (and drags glibc-no-fortify-warning.patch and flake.lock into the tree too). Keep the dev environment untracked.

Comment thread shell.nix
});

# Use LLVM for modern PostgreSQL development
llvmPkgs = pkgs-unstable.llvmPackages_21;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llvmPackages_21 is pulled from nixpkgs-unstable, which moves independently of the pinned stable channel. Even with flake.lock committed, mixing an unstable input for the toolchain makes the dev environment harder to reproduce over time. Minor relative to the larger issue (this whole file is developer-local scaffolding that should not be in the patch), but flagging the reproducibility smell.

Comment thread flake.nix
@@ -0,0 +1,45 @@
{
description = "PostgreSQL development environment";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flake.nix (with its companion shell.nix and a committed flake.lock) wires up a complete personal Nix dev environment. This is unrelated to the feature and is exactly the kind of non-minimal, environment-specific churn that gets patches rejected on pgsql-hackers. PostgreSQL does not ship a Nix flake; keep this untracked rather than committing it.

Comment on lines 2151 to +2154
if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber &&
PageGetMaxOffsetNumber(page) == FirstOffsetNumber)
PageGetMaxOffsetNumber(page) == FirstOffsetNumber &&
((PageHeader) page)->pd_upper ==
((PageHeader) page)->pd_special - MAXALIGN(heaptup->t_len))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This heap_insert INIT_PAGE tightening (and its mirror in log_heap_update below) is a wal_consistency_checking fix for a scenario -- vacuum's lp_truncate_only pass (PRUNE_VACUUM_CLEANUP) leaving orphan tuple bytes behind an LP_UNUSED slot -- that is independent of the HOT-indexed feature and predates this patch. Bundling an orthogonal WAL-consistency fix into this large feature patch hurts bisectability and review (one commit, one logical change). Consider splitting it into its own patch with its own test, unless it is strictly required by the new VM/stub code paths (the comment does not establish that dependency).

Comment on lines +99 to +102
/* pg_stat_get_tuples_hot_indexed_upd_skipped */
Datum
pg_stat_get_tuples_hot_indexed_upd_skipped(PG_FUNCTION_ARGS)
{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two functions hand-copy the exact boilerplate of the existing PG_STAT_GET_IDXENTRY_INT64(stat) macro (defined a few lines below at the comment "Accessor macro for index stats entries"), which already expands to a function that fetches pgstat_fetch_stat_idxentry(relid), returns 0 on NULL, and otherwise casts idxentry->stat to int64. The only reason the macro can't be reused directly is that it hardcodes the pg_stat_get_idx_ name prefix while these prosrc names use pg_stat_get_tuples_hot_indexed_upd_. Consider either reusing the macro (keeping the pg_stat_get_idx_* naming for the SQL prosrc) to avoid the copy-paste, or at least generalizing the macro to take an explicit function-name token. As written this is redundant boilerplate that will drift from the macro.

Comment on lines +4455 to +4460
afterTriggers.maximages = Max(64, afterTriggers.maximages * 2);
afterTriggers.images = afterTriggers.images ?
repalloc_huge(afterTriggers.images,
afterTriggers.maximages * sizeof(AfterTriggerImage)) :
MemoryContextAllocHuge(afterTriggers.event_cxt,
afterTriggers.maximages * sizeof(AfterTriggerImage));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The images[] descriptor array grows with the total image count, including spilled ones: the grow block (and nimages++) runs on both the in-memory and file paths. So spilling the tuple bodies to image_file does not bound this array's memory - a huge bulk UPDATE on an in-place AM still accumulates one AfterTriggerImage (~24 bytes) per row in event_cxt for the life of the transaction, partly defeating the stated purpose of the spill ("so that a large bulk update does not hold every row in memory").

Separately, the allocation size afterTriggers.maximages * sizeof(AfterTriggerImage) is a uint32 * size_t product. The only guard is maximages > PG_UINT32_MAX / 2; on 32-bit platforms where Size is 32-bit, maximages * sizeof(AfterTriggerImage) overflows Size well before that bound (e.g. maximages near 2^31 times 24), passing a wrapped, far-too-small size to MemoryContextAllocHuge/repalloc_huge and causing a heap buffer overflow on subsequent images[nimages] writes. Guard the byte product (e.g. via pg_mul_size_overflow or capping maximages against MaxAllocHugeSize / sizeof(AfterTriggerImage)).


#include "storage/itemptr.h"
#include "utils/dsa.h"
#include "utils/relcache.h"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pulling the full utils/relcache.h into tidbitmap.h is heavier than needed. The two new prototypes use Relation only as a by-value pointer type, so a forward declaration suffices and keeps this widely-included header (via access/relscan.h) minimal, per PG header hygiene. relcache.h drags in access/tupdesc.h and common/relpath.h transitively.

Confidence: low (style/maintainability nit, not a correctness issue).

Suggested change
#include "utils/relcache.h"
/* avoid pulling in relcache.h; Relation is only used in prototypes below */
typedef struct RelationData *Relation;

Comment on lines +56 to +57
trap "test ! -e $hh.bak || mv -f $hh.bak $hh; git checkout --quiet $orig" EXIT
git checkout --quiet --detach "$TEPID_REV"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Footgun (high confidence): this mutates the user's working repository in place. It runs git checkout --quiet --detach "$TEPID_REV" and sed -i.bak on a tracked source file (src/backend/access/heap/heapam_handler.c, whose .bitmap_and_inexact = heap_bitmap_and_inexact, line is confirmed to exist), relying solely on a single EXIT trap to restore state. If the script is SIGKILLed, the machine loses power, or the trap otherwise fails, the user's checkout is left detached and/or heapam_handler.c is left edited (with the .bak possibly already consumed by the restore of the other arm). A patch that silently corrupts a developer's tree should not ship. Build the split_off variant from a disposable worktree (git worktree add) or a clean clone under $BENCH instead of editing the live checkout.

gburd and others added 24 commits November 20, 2026 11:42
Squashes the four CI commits this fork carried on top of upstream into
one, and cuts the carry from 3776 lines to ~1000.

Removed:

- .github/{README,QUICKSTART}.md and .github/docs/* (2273 lines). Beyond
  being bulk we rebase hourly, they had drifted into being wrong: they
  described a daily (not hourly) sync, Claude 3.5 Sonnet via the direct
  Anthropic API (we run Opus on Bedrock), a static IAM user (we use
  OIDC), and a Windows dependency-builder workflow that the same commit
  which documented it had already deleted. 16 referenced paths did not
  exist. An AI reviewer reading this repo ingests that as truth.

- sync-upstream-manual.yml (252 lines). sync-upstream.yml already has
  workflow_dispatch; the sole behavioral difference was a force_push
  toggle whose "false" setting cannot work after a rebase anyway.

- .github/.gitignore, which only ignored scripts/ai-review/, a directory
  that does not exist here.

Rewritten:

- sync-upstream.yml -> fork-sync-upstream.yml, 260 lines to 47. git
  rebase already handles the fast-forward, no-op and diverged cases that
  the ahead/behind arithmetic hand-rolled, so all that goes away, as do
  the issue open/comment/close machinery (a scheduled-run failure is
  already emailed to the one consumer, and the run log carries what the
  issue body would) and the "dev setup|dev v[0-9]" commit allowlist,
  which matched zero commits on master and had since it was written.
  Sync drops to every three hours: hourly meant 24 full-history clones
  of an 840MB repo per day, each force-push also re-triggering pg-ci.yml
  on master, to collect a handful of upstream commits.

Renamed:

- ocr-review.yml, ocr-model-check.yml -> fork-*.yml. Upstream owns
  .github/ too (it has touched pg-ci.yml three times this year), so
  fork-owned workflows now live under a name prefix upstream will not
  collide with, making the rebase structurally conflict-free and letting
  the sync guard match fork-owned paths exactly rather than allowing all
  of .github/.

Unchanged: the OCR review workflows' logic and .github/ocr/* (rule.json,
context.md, litellm.yaml, pg-history.py) - the part that carries value.
The OCR review workflows and their config were the last bulk this fork
carried on top of upstream: ~970 lines rebased onto postgres/postgres
every few hours, editable only by amending a commit that gets force-
pushed. They are now reusable (workflow_call) workflows in the sidecar
repo gburd/ci-workflows, and master carries two caller stubs instead.

Carry after this: 3 files, ~110 lines, one commit.

The sidecar is also where the value is: ocr/rule.json and ocr/context.md
(the PostgreSQL committer-grade review corpus) become normal reviewable
files with normal PRs and history, rather than a blob inside a commit
that is rewritten on every sync.

Mechanics worth knowing:

- The review jobs check ci-workflows out at github.job_workflow_sha, so
  the rules always come from the same commit as the logic using them, and
  pinning @v1 (or a sha) in the stub pins both. This replaces the old
  "git show origin/<default_branch>:.github/ocr/..." dance, which existed
  only because the config had to be read off a branch other than the PR's.

- AWS settings are passed as explicit inputs rather than read from vars
  inside the reusable workflow. vars does resolve across the boundary,
  but the contract stays visible at the call site.

- A reusable workflow does not change the OIDC subject: it stays
  repo:gburd/postgres:*, so the ocr-bedrock-ci trust policy is unchanged.

- ci-workflows must stay public; callers read it with GITHUB_TOKEN.

The sync guard narrows to .github/workflows/fork-* accordingly, and
sync-upstream keeps needing SYNC_TOKEN: the stubs are still workflow
files, which GITHUB_TOKEN may not push.
This isn't a commit that will be submitting for review, it is purely for
local developer tooling while developing this patch set.  Ignore it.
An index stores a locator for every entry: the value it hands back
to the table AM to identify the row.  Every in-core table AM uses
an ItemPointerData, a block number plus an in-block offset, and a
good deal of core code leans on that shape.  Bitmap scans split the
locator into a block and an offset, GIN packs the offset into 11
bits, TID range scans expose its ordering to SQL, and nbtree breaks
ties between equal keys with it.  A table AM that identifies rows
some other way has no way, today, to tell core any of this, and
core has no way to ask.

Give both sides a place to say what they mean.  A table AM returns
a per-relation locator descriptor from a new relation_locator
callback, reached through RelationGetLocatorDesc() and cached on
the relcache entry.  The descriptor gives the locator's width
(negative for a variable-width locator), a name for error
messages, whether a row keeps its locator for its whole life
(stable), whether the version an UPDATE replaced can still be
fetched by its locator (old_version_retained, with
RelationUpdatesInPlace() for the executor), and max_offset, the
largest offset any of the relation's locators carries.

max_offset has two readers today: the planner does not consider
bitmap scans of a relation whose max_offset exceeds what a TID
bitmap can hold, and ginbuild() refuses such a table, since GIN's
posting lists and partial-match scans assume the same bound.  This
is the gap Jeff Davis ran into in "MaxOffsetNumber for Table AMs"
(2021).  The other fields describe properties that later commits
in this series act on.

amlocator.h and a new "Row Locators" section of tableam.sgml also
write down what core relies on of every locator that no field
states: that it travels as an ItemPointerData, that it is the
row's ctid, which values are reserved, and the (block, offset)
order that nbtree, ANALYZE, TID range scans and BRIN depend on.

A table AM without the callback is treated as heap is, so
out-of-tree AMs keep working untouched.  Heap returns a single
static descriptor that writes down what core already assumes of a
TID, so for heap this is a no-op.

Discussion: https://postgr.es/m/0bbeb784050503036344e1f08513f13b2083244b.camel@j-davis.com
The field is documented as "an update chain was followed", but that
is not what its readers act on.  heapam_tuple_lock sets it only after
replacing *tid, and every reader (ExecLockRows, GetTupleForTrigger,
ri_LockPKTuple, and the EvalPlanQual paths in ExecUpdate and
ExecDelete) responds by re-evaluating the row it read earlier.  What
they need to know is that the version they locked is later than the
one they named, because a concurrent transaction updated the row and
committed.

For heap that is the same thing as following the update chain, so
the rename changes no behavior.  It gives a table AM whose rows are
not identified by a version-specific TID a definition it can
implement, which "Support table AMs that update rows in place"
completes.
Keep physical tuple links and transaction/command IDs inside the heap AM.
Callers need to distinguish same-command and triggered modifications,
current-transaction ownership, and partition moves, not interpret heap
headers or special TID values.

Translate only fields defined for the returned result, using the supplied
command ID and preserving the creation-command check when a lock follows
an update chain. Initialize failure facts on every call without losing
retargeted across internal retries. Keep existing heap partition-move
errors while allowing MERGE to recognize an AM-reported move.

Exercise the booleans directly through the real lock callback, including
poisoned output, child transactions, command visibility and retargeting.
Check MERGE cardinality handling in a child transaction as well.
table_tuple_lock, table_tuple_update and table_tuple_delete take a
locator, and for heap a TID names one version of a row.  A table AM
whose locator names only the row, as one that updates rows in place
does, cannot tell from the call whether the caller read the row
through its snapshot, in which case a newer committed version means
the caller must re-evaluate, or has already locked the latest
version itself, in which case it must not.  Guessing either way is
wrong: re-evaluating a row the command already locked runs
EvalPlanQual where ON CONFLICT DO UPDATE and MERGE cannot take it,
and assuming "already locked" whenever the transaction holds a lock
loses concurrent updates in an UPDATE fed by a SELECT ... FOR UPDATE
subquery, or in a FOR UPDATE OF join that reaches a locked row twice.

Only the executor knows.  Add TUPLE_LOCK_FLAG_LOCKED_VERSION,
TABLE_UPDATE_LOCKED_VERSION and TABLE_DELETE_LOCKED_VERSION, meaning
"this is the version this command has already locked", and pass
them where that is so: after a BEFORE ROW trigger or EvalPlanQual
has locked the row in ExecUpdate, ExecDelete and MERGE, for the row
ON CONFLICT DO UPDATE and DO SELECT lock, and from logical
replication apply, which locks the row first and has no snapshot.
ExecLockRows and the RI triggers never pass them, since their rows
come from snapshot scans.  GetTupleForTrigger learns of it through a
new row_locked argument to ExecBRUpdateTriggers and
ExecBRDeleteTriggers.

Heap ignores the new bits, since its TID already names a version,
so nothing changes for heap.
Index AMs, the executor and every index-facing table AM callback
pass a row's locator as an ItemPointerData, and validate_index()
packs it into an int8 for CREATE INDEX CONCURRENTLY.  A table AM
whose locator descriptor declares a wider or variable-width locator
can therefore not be indexed by any in-core index AM.

Let the index side say whether it can do better.  An index AM sets
amcanvarlocator when it can store a locator that does not fit in an
ItemPointerData, reported by pg_indexam_has_property() as
"can_var_locator".  Unless it does, building an index on a table
whose locator is variable-width or wider than an ItemPointerData
fails with an error naming both access methods and the locator.
The check is in index_create(), before any catalog entry is made,
and again in index_build(), because a table's locator can change
under existing indexes: ALTER TABLE ... SET ACCESS METHOD rebuilds
them through reindex_relation(), and REINDEX, TRUNCATE and VACUUM
FULL rebuild them too.  Every index build, including partition
index cloning and constraint indexes, passes through one of the two.

No in-core index AM sets the flag and no in-core table AM has a
wide locator, so nothing changes for existing AMs.  Carrying a
wider locator through aminsert, index scans and vacuum is left for
a later series; the flag lets an index AM and a table AM that agree
on one find each other until then.
An AFTER ROW trigger event records the locators of the rows it
fires for, and the rows are fetched by locator when the trigger
runs, which for a deferred trigger is at COMMIT.  That works for
heap because a locator names a version and heap keeps old versions
until VACUUM.  For a table AM whose locator names only the row, the
fetch returns whatever the row is when the trigger fires: OLD is
already overwritten, and NEW may have changed again since.

Let an event carry copies of its rows instead.  A new
AFTER_TRIGGER_IMAGES event kind records, alongside the locators,
an index into a per-transaction store of copied rows: the OLD row
of an UPDATE, and the NEW row of an INSERT or UPDATE for every row
trigger other than a foreign key check or a unique recheck, which
need the row's current version to decide whether it is still live.
A copy is made once per row and shared by all the triggers that
fire for it.  Copies stay in memory up to work_mem and spill to a
temporary file beyond that, owned by a resource owner of its own so
an abort closes it without retrying a failed write.  They are
released at the end of a top-level statement once no deferred event
remains, and a subtransaction abort rolls the store back to where
it began.  ExecARUpdateTriggers takes the old row the caller holds
rather than fetching it again.

Events carry copies only for a table whose locator descriptor says
old_version_retained = false.  The next commit makes the executor
copy the old row before the write, so that the copy is the
pre-update row.  Heap says true, so heap events are unchanged.
Core assumes throughout that an UPDATE leaves the pre-update version
of a row readable through its locator: it re-fetches the old row by
TID when an AFTER trigger fires, for RETURNING OLD, and when
EvalPlanQual rechecks a joined row.  Heap keeps the old version on
its page until VACUUM, so this is always true for heap.  A table AM
that overwrites a row in place has replaced the old version by then,
so on such an AM AFTER ROW triggers and OLD transition tables see
the wrong values, and foreign-key enforcement misfires.

When the locator descriptor says old_version_retained = false:

  - The executor copies the old row out of its slot before the
    write, in ExecUpdate, MERGE and logical replication apply, so
    RETURNING OLD and the AFTER ROW triggers read the copy.

  - After-trigger events carry row copies (previous commit).

  - EvalPlanQual carries an unlocked joined row whole (ROW_MARK_COPY)
    instead of re-fetching it.

The AM must still keep a deleted row fetchable by its locator until
the deleting transaction ends, and a fetch by locator must return
the row's current version; amlocator.h states both.  With the
version-naming bits from earlier in the series this is enough for
an in-place AM to give heap's answers.

Heap's descriptor says true, so no heap path changes.
A bitmap scan splits each locator into a group and a slot within
it, the block and line pointer offset for heap, and BitmapAnd and
BitmapOr combine their inputs group by group without touching the
table.  That is exact as long as every index entry for one row
version names the same slot.  A table AM that cannot promise that
for some groups has no way to say so: an intersection would drop a
row whose two entries name different slots, and a union over
groups that do not match the AM's storage may need a recheck.

Add two optional per-group callbacks to the locator descriptor.
bitmap_and_inexact makes BitmapAnd union a flagged group's slots and
mark it for recheck instead of intersecting it, and
bitmap_or_inexact makes BitmapOr mark a flagged group for recheck.
Each takes the group and a buffer the callback may keep pinned
across calls on one scan.  Every bitmap index scan records its
table on the bitmap it fills (tbm_set_relation), so a combine at any
depth of the bitmap tree consults the table's callbacks.

Heap leaves both NULL here, so it takes the exact path it always
did.
Add regression coverage for existing classic Heap-Only Tuple (HOT) update
behavior, committed first so the behavioral changes in the later HOT-indexed
commits are diffable against a known-good baseline.  This commit adds
tests so as to codify explicitly the HOT contract for the heap AM.

The new hot_updates regression test exercises:
- Basic HOT vs non-HOT update decisions
- The all-or-none property across multiple indexes
- Partial indexes and predicate handling
- BRIN (summarizing) indexes allowing HOT updates
- TOAST column handling with HOT
- Unique constraint behavior
- Multi-column indexes
- Partitioned table HOT updates
- HOT chain formation and the index-scan walk over a chain

Authored-by: Greg Burd <greg@burd.me>
The executor needs to know which indexed attributes changed without
assuming how a table AM represents their values.  Add an optional
modified_attrs callback and table_modified_attrs() wrapper that
return the changed subset of a candidate bitmap.  An AM without the
callback conservatively reports every candidate as changed.

This separates the comparison contract from the update interface.
Heap implements the callback before the executor begins using it,
so the update conversion preserves existing HOT and locking choices.
Provide heapam_modified_attrs(), the heap AM's implementation of the
table AM modified_attrs callback introduced in the preceding commit.  It
answers "which of these attributes changed value between the old and new
versions of a row?" using the heap's own notion of attribute equality
(datum_image_eq on the logical slot values) and system-column semantics
(the FirstLowInvalidHeapAttributeNumber offset convention and the
tableoid special case).

This is the heap-specific mechanism that previously lived in the
executor as ExecCompareSlotAttrs(); moving it behind the table AM
boundary keeps the executor agnostic of how any given AM stores its data
or what its system columns mean, while the executor retains the policy
decision of which indexes to maintain from the returned overlap.  An AM
that does not provide the callback is treated as though every candidate
attribute changed (maintain all indexes).
Refactor executor update logic to determine which indexed columns have
actually changed during an UPDATE operation rather than leaving this up
to HeapDetermineColumnsInfo() in heap_update(). Finding this set of
attributes is not heap-specific, but more general to all table AMs and
having this information in the executor could inform other decisions
about when index inserts are required and when they are not regardless
of the table AM's MVCC implementation strategy.

The heap-only tuple decision (HOT) in heap functions as it always has;
what moves to the executor is only the determination of the "modified
indexed attributes" (modified_idx_attrs).

ExecUpdateModifiedIdxAttrs() replaces HeapDetermineColumnsInfo() and is
called before table_tuple_update() crucially without the need for an
exclusive buffer lock on the page that holds the tuple being updated.
This reduces the time the buffer lock is held later within
heapam_tuple_update() and heap_update().

Besides identifying the set of modified indexed attributes
HeapDetermineColumnsInfo() was also partially responsible for the
decision about what to WAL log for the replica identity key. That logic
moves into heap_update() and into the replacement helper
HeapUpdateModifiedIdxAttrs(), so simple_heap_update() and
heapam_tuple_update() share the same logic since both call into
heap_update().

Updates stemming from logical replication also use the new
ExecUpdateModifiedIdxAttrs() in ExecSimpleRelationUpdate().

ExecUpdateModifiedIdxAttrs() uses ExecCompareSlotAttrs() to identify
which attributes have changed and then intersects that with the set of
indexed attributes to identify the modified indexed set, the
modified_idx_attrs.

This patch introduces a few helper functions to reduce code duplication
and increase readability: HeapUpdateHotAllowable() and
HeapUpdateDetermineLockmode(), used in both heap_update() and
simple_heap_update().

heap_update() is now called with lockmode pre-determined and a boolean
indicating whether the update may be HOT, both const. If during
heap_update() the new tuple fits on the same page and that boolean is
true, the update is HOT. So although the functions and timing of the
HOT decision code have changed, none of the logic governing when HOT is
allowed has changed.

Development of this feature exposed nondeterministic behavior in three
existing tests, which have been adjusted to avoid inconsistent results
due to tuple ordering during heap page scans.

Authored-by: Greg Burd <greg@burd.me>
Discussion: https://www.postgresql.org/message-id/flat/78574B24-BE0A-42C5-8075-3FA9FA63B8FC%40amazon.com
An update that preserves existing row locators need not replace
entries in indexes whose referenced attributes did not change.
Separate this executor and index support from the heap storage
implementation so other table access methods can use the contract.

Track each index's referenced attributes independently of the
narrow indexUnchanged hint.  Let a table AM skip index entries that
no longer name the row's current key, and add SO_INDEX_KEEP_STALE for
the constraint checks that must see them, flagged, to compare keys
themselves; provide an optional checked fetch for constraint
enforcement.  Access methods without that callback use
the existing fetch operations.  Distinguish index-only scans from
requests to return index tuples when deciding whether to retain
btree leaf pins.

Prepare the selective-update counters and their flush paths here.
The existing heap statistics call supplies false for the new
argument until the heap implementation can produce such updates;
this is the only heap change in this preparatory commit.

Bump the statistics file format identifier for the added counters.
Define the on-disk representation a HOT-indexed update and its later
prune/collapse produce, ahead of the code that reads or writes it:

- HEAP_INDEXED_UPDATED (htup_details.h), the t_infomask2 bit marking a
  heap-only tuple whose producing UPDATE also changed an indexed column; and
- access/hot_indexed.h, the inline fixed-size modified-attrs bitmap stored in
  the tail of such a tuple, plus the xid-free "collapse-survivor stub" format
  (HEAP_INDEXED_UPDATED with natts == 0, a forward link, and the segment's
  bitmap) and the accessors both share.

The design itself is documented in README.HOT-INDEXED, which the next commit
adds along with the code that writes this format.

Co-authored-by: Greg Burd <greg@burd.me>
Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>
A selective-indexed update, added by the next commit, plants a
fresh index entry that points at the new heap-only tuple mid-chain
while every unchanged index still points at the chain root.  The
two entries name different line pointers on the same heap block,
and an exact BitmapAnd intersection of the two indexes would drop
the row.  The disagreement is always within one block, because a
HOT chain never leaves its page, so what BitmapAnd needs is a
per-block answer that it can read without touching the heap.

Add VISIBILITYMAP_LOCATOR_SPLIT, read lock-free like ALL_VISIBLE,
with functions to set, clear and test it.  The visibility map
widens from 2 to 4 bits per heap block to carry it, and pg_upgrade
rewrites each _vm fork from the old layout, a block at a time, as
it did for 9.6's frozen bit.  The rewrite is gated on
VISIBILITYMAP_WIDTH_CHANGE_CAT_VER, which equals the catalog
version this commit installs.  The bit is not part of visibility
and is excluded from VISIBILITYMAP_VALID_BITS.

Nothing sets the bit yet; the next commit sets it and points heap's
bitmap_and_inexact callback at it.

Bump catalog version.
An update that changes only some indexed attributes can preserve
unchanged index entries by keeping the new tuple in the HOT chain.
Record each hop's modified attributes in an inline bitmap. New
entries address the new heap-only tuple; unchanged entries continue
to address earlier members of the chain.

Have index scans skip entries whose crossed attributes affect the
index being scanned, inside the heap AM, so callers need no
knowledge of the chains; a scan begun with SO_INDEX_KEEP_STALE
gets them flagged instead. Give constraint fetches the arriving index too, so
unrelated changes do not request an unsafe expression-key recheck
under a B-tree leaf lock. Keep the key comparison for overlapping
attributes: away-and-back changes can leave the key equal.

Include pruning and vacuum support with the first writer. Preserve
needed dead members as forwarding stubs, including their saved
bitmap widths, and include existing stubs when pruning again.
Resolve sibling redirects against the completed plan so no
surviving anchor bypasses a changed-key hop or points at reclaimed
storage. Keep pages with stubs or selective redirects out of the
all-visible state.

Set the visibility map locator-split bit when indexes can name
different offsets for the same row. Log and replay the update and
prune changes, initializing a new visibility map page before
setting its split bit when redo has no applicable full-page image.

When building an index, anchor each live row at the latest chain
member that changed one of the index's attributes, or the root,
which is where an UPDATE would have put the entry; pruning relies
on that when it reclaims a member. Resolve existing anchors under
the reference snapshot during concurrent index validation. Accept
unchanged suffixes without inserting duplicate entries, but reject
crossed changes even when the key cycles back. Adapt rewrite and
bitmap scans to the new representation.  Index deletion keeps all of
its corruption checks except that an entry may point at a HOT-indexed
heap-only tuple. Retain conservative fallbacks for catalog,
replication and expression-index input changes.

Cover repeated pruning, index scan order, direct anchors,
concurrent index builds and unique checks here. Add recovery checks
alongside the amcheck support they require in a later commit.

Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>
Expose the HOT-indexed counters maintained by the write path in
pg_stat_all_tables and pg_stat_all_indexes. Provide
pg_relation_hot_indexed_stats() to inspect chain composition, and
document the counters in monitoring.sgml.

Teach pg_surgery to handle HOT-indexed tuples and collapse stubs.
Cover that handling and concurrent update, vacuum and scan behavior
here. Add the pageinspect and recovery suites with the following
amcheck support, since those tests also verify heap integrity.
Recognize heap-only tuples with direct index anchors, xid-free
collapse stubs and multiple redirects to one live tuple as valid
heap representations, while continuing to check their chains.

For heapallindexed, fingerprint the stored leaf keys with the TID
of the snapshot-visible version reached through a valid anchor.
Reject anchors that cross changes to the index, including
away-and-back changes. Probe with the scanned heap version rather
than a rebuilt anchor, so different valid anchors do not report
missing entries and a broken chain cannot borrow coverage from
another tuple.

Use this path for heap handler aliases too. Validate raw TID
bounds, physical links, tuple headers and bitmap bounds under the
buffer share lock before walking visibility. In particular, never
let an invalid block number reach a read that could extend the
heap. Respect saved stub widths and ordinary off-page update links.

Cover rebuilt anchors, concurrent updates and vacuum, aliases, old
stub widths and corrupt keys, TIDs and chains. Check backend
survival and unchanged heap storage after corruption errors. Carry
selective chains and collapsed stubs through pg_upgrade as well.

Add the pageinspect and recovery suites here, after both statistics
and selective-aware amcheck are available. These include repeated
prune replay and visibility map initialization without an
applicable full-page image, with full-page writes and consistency
checking both on and off, and visibility map tail truncation.
Core disqualifies HOT-indexed updates unconditionally on the logical-replication
apply path: HeapUpdateHotAllowable returns HEAP_UPDATE_ALL_INDEXES whenever
IsLogicalWorker() is true, because a HOT-indexed update of a replica-identity
attribute leaves a stale index leaf that the apply worker's replica-identity
lookups would otherwise be exposed to.

This commit relaxes that unconditional bail-out with a per-subscription
hot_indexed_on_apply option (subhotindexedonapply: off / subset_only (default)
/ always).  HeapUpdateHotAllowable now consults it when running in an apply
worker: "off" keeps the unconditional bail-out, "subset_only" allows the
HOT-indexed path only when the relation's indexed attributes are a subset of
its primary-key attributes, and "always" applies no apply-path gating.  The
apply worker's replica-identity lookups (see RelationFindReplTupleByIndex) cope
with the stale leaf, but only when the indexed attributes are a subset of the
replica identity, which is what the default subset_only enforces.

Wire the option through CREATE/ALTER SUBSCRIPTION, pg_subscription, pg_dump,
and psql's \dRs+, and document it (create_subscription, alter_subscription,
catalogs).  Cover apply under each mode (039), apply under REPLICA IDENTITY
FULL and a non-PK USING INDEX whose key is cycled (040), and decoding of
HOT-indexed update chains (test_decoding).

Authored-by: Greg Burd <greg@burd.me>
src/test/modules/test_locator_tableam provides table AMs that store
rows exactly as heap does and differ only in the locator properties
they declare, so the parts of the locator contract that heap does
not exercise run in the regression tests:

  - heap_noretain declares old_version_retained = false.  A
    SnapshotAny fetch by locator follows the update chain to the
    row's current version, which is what an AM that updates in place
    hands back, so a caller that wrongly re-fetches an old row after
    an UPDATE gets the new one and the test fails.  It covers AFTER
    ROW UPDATE and DELETE triggers, OLD and NEW transition tables,
    deferred constraint triggers, foreign key CASCADE and RESTRICT,
    RETURNING OLD, MERGE, the trigger-copy spill under a small
    work_mem, temp_file_limit failures, savepoint rollback, and a
    cross-partition update.

  - heap_wide declares a 16-byte locator, so every path that builds
    an index on it must fail, and pg_indexam_has_property() reports
    can_var_locator false for every in-core index AM.

  - heap_bigoffset declares offsets up to MaxOffsetNumber, so the
    planner must not consider bitmap scans of it and GIN must refuse
    to index it.

  - heap_orrecheck sets bitmap_or_inexact, so BitmapOr consults it.

The executor's version-naming bits are not exercised, since all
these AMs use heap's lock and update code.  This is test scaffolding
for reviewers and is not meant to be committed.
A/B and single-variant benchmark scripts for HOT-indexed updates: build two
postgres variants, run pgbench workloads exercising classic-HOT, non-HOT, and
HOT-indexed paths, and a self-contained bloat probe that reports the skip count
(index writes avoided on unchanged indexes) and changed-index bounding.  Not
for merge; kept for evaluating the feature.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 OCR found 37 issue(s).

  • 25 inline, 12 in summary (inline capped at 25)

📄 src/test/benchmarks/siu/scripts/split_ab.sh (L67-L67)

sed -i.bak is not portable: BSD/macOS sed treats -i.bak differently from GNU sed (BSD needs -i '' or -i .bak as a separate argument). On BSD this either errors or consumes the next arg as the suffix, which would corrupt the in-place edit of a tracked source file. Since this edits heapam_handler.c in the working tree, a portability failure here risks leaving the tree mutated. Given the README's FreeBSD-portability claim, use a temp-file + mv approach instead of relying on -i's nonstandard semantics.

Confidence: high.


📄 src/test/benchmarks/siu/scripts/run.sh (L85-L85)

find ... -mindepth is a non-POSIX GNU/BSD extension (not available in a strict POSIX find, e.g. Solaris, which PostgreSQL targets). More simply, this whole dance to empty-and-recreate a directory can be replaced with rm -rf "$datadir"; mkdir -p "$datadir" (as split_ab.sh's own start_pg already does on line 87), which is both portable and simpler. The inconsistency between the two drivers is itself a maintainability smell.

Confidence: medium.

💡 Suggested change

Before:

  [ -d "$datadir" ] && find "$datadir" -mindepth 1 -delete && rmdir "$datadir"

After:

  rm -rf "$datadir"; mkdir -p "$datadir"

📄 src/test/benchmarks/siu/scripts/build.sh (L12-L12)

These developer benchmark harnesses live under src/test/benchmarks/ but are not wired into any Makefile or meson.build (none exists in this directory), so they are not run by the buildfarm/meson test harness and do not behave like the rest of src/test. They also hardcode absolute paths (/scratch/siu-bench, $HOME/ws/postgres/tepid), fixed ports (57480/57481/57503), sleep 2 as startup synchronization, and GNU-only tooling. For a pgsql-hackers submission, supporting benchmark artifacts for a performance claim are normally posted to the thread, not committed under the test tree; if they are to be committed, they belong somewhere that makes clear they are not part of the regression/TAP suite, and the portability/hardcoded-path issues should be addressed. Please confirm intent.

Confidence: medium.


📄 src/test/benchmarks/siu/scripts/bloat.sh (L24-L24)

README (line 17) documents the default BENCH as /scratch/tepid-bench, but every script defaults to /scratch/siu-bench. The docs and code disagree on the single most important path a user must know. Reconcile them.

Confidence: high.


📄 src/test/benchmarks/siu/scripts/run.sh (L101-L104)

Fragile synchronization: pg_ctl ... start is launched without -w, so it returns before the postmaster accepts connections, and readiness is then assumed via a fixed sleep 2. On a loaded/slow host 2s may be insufficient and the following setup_schemas/psql_as call fails; on a fast host it wastes time. bloat.sh in this same series already does this correctly with -w start. Use pg_ctl ... -w start (and drop the sleep) so startup blocks until the server is ready. soak.sh and split_ab.sh have the identical start + sleep 2 pattern and should be fixed the same way.

💡 Suggested change

Before:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
  sleep 2
}

After:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}

📄 src/test/benchmarks/siu/scripts/soak.sh (L47-L50)

Same fragile-startup pattern as run.sh: pg_ctl start without -w returns before the server accepts connections, and a fixed sleep 2 is used as the readiness barrier. Prefer pg_ctl ... -w start (as bloat.sh does) so startup blocks until ready and the sleep can be removed.

💡 Suggested change

Before:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
  sleep 2
}

After:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}

📄 src/test/benchmarks/siu/scripts/split_ab.sh (L99-L102)

Same fragile-startup pattern as run.sh/soak.sh: pg_ctl start without -w plus a fixed sleep 2 as the readiness barrier. Use pg_ctl ... -w start so startup blocks until the server accepts connections and the sleep can be dropped.

💡 Suggested change

Before:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" start >/dev/null
  sleep 2
}

After:

  LD_LIBRARY_PATH="$(LD_of "$v")" "$(bin_of "$v")/pg_ctl" -D "$datadir" \
    -o "-p $PORT" -l "$LOGDIR/pg_$v.log" -w start >/dev/null
}

📄 src/test/benchmarks/siu/scripts/run.sh (L29-L32)

No cleanup trap is installed. With set -e, any failing step (a bad psql, a missing binary, or Ctrl-C) aborts the script mid-loop and leaves a running postmaster holding $PORT and an un-removed _data_* dir, which then breaks the next invocation. bloat.sh and split_ab.sh both install an EXIT trap that runs pg_ctl stop; run.sh (and soak.sh) should do the same so a failed/interrupted run cleans up its cluster.


📄 src/test/benchmarks/siu/scripts/soak.sh (L5-L8)

No cleanup trap: under set -e, a failing command or Ctrl-C aborts before stop_pg runs, leaving a postmaster on $PORT and an un-removed data dir that breaks the next run. Install an EXIT trap that stops the running cluster (as bloat.sh/split_ab.sh do).


📄 src/test/regress/pg_regress.c (L1246-L1246)

This execl -> execlp switch is unrelated to the table-AM lock-facts feature this patch implements, and it carries a behavioral change: execlp resolves shellprog via $PATH when it is not an absolute path, whereas execl does not. SHELLPROG is $(SHELL) (make) / /bin/sh (meson); $(SHELL) can be overridden to a bare name, in which case this now silently changes which shell is executed. If this is needed for a non-standard build environment (e.g. the added Nix setup), split it into its own commit with a rationale; otherwise drop it to keep the diff minimal.


📄 src/test/regress/sql/tsearch.sql (L776-L777)

This is a comment-only change in tsearch.sql, a file otherwise unrelated to the table-AM lock-facts feature in this patch. PostgreSQL patch hygiene penalizes rewording comments in files not functionally touched by the change, since it adds churn to the diff and forces the matching .out update. The explanation about SET-clause tracking belongs with the feature's own code/tests, not retrofitted onto the tsearch regression test. Confidence: low (style/hygiene, not correctness).


📄 src/test/subscription/t/039_hot_indexed_apply.pl (L371-L373)

The comment "The old key's leaf entry is stale after a HOT-indexed update" / "may now be stale" is only accurate for the always iteration. For off and subset_only, HeapUpdateHotAllowable() demotes the UPDATE to HEAP_UPDATE_ALL_INDEXES (tab_uk's indexed attrs {id,payload,tag} are not a subset of the PK {id}), so the old (0, 'mode_$mode') leaf is properly removed - no stale entry exists. Those two iterations therefore pass trivially and do NOT exercise the _bt_check_unique stale-leaf recheck the comment describes; only the always pass does. Consider rewording so the comment does not overstate what the off/subset_only iterations cover (they only confirm the INSERT still succeeds, not that a recheck occurred).

Comment thread .clangd
Comment on lines +1 to +2
Diagnostics:
MissingIncludes: None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire file is local developer scaffolding unrelated to the HOT indexed updates feature. On pgsql-hackers a non-minimal diff that bundles unrelated tooling is a top rejection reason (patch does more than one thing). Additionally the committed config is non-portable: CompilationDatabase: build/ points at an uncommitted, local artifact (no compile_commands.json is tracked in the tree), and -I../../../../src/include only resolves from one specific subdirectory depth. This file should be dropped from the patch and kept local (e.g. via a personal gitignore).

Comment thread .gdbinit
# =========================================================================

# Main entry: heap_update
break heapam.c:3210

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is local debugging scaffolding unrelated to the committable feature and should be dropped from the patch. Worse, the absolute line-number breakpoints are already stale against the very patch they document. For example, heap_update() in the modified heapam.c starts at the TM_Result heap_update(...) signature near line 3313, not 3210; the HOT decision block (if (hot_mode != HEAP_UPDATE_ALL_INDEXES)) is near line 4207, not 4019/4024; HEAP_INDEXED_UPDATED is set near line 4343, not 4101; and there is no 'restore on old tuple' code near 4147. Any edit to these heavily-modified files (heapam.c +712/-174, pruneheap.c +696/-47) invalidates every numeric breakpoint, making this committed documentation wrong on arrival.

Comment thread .gdbinit
# =========================================================================

# Compute indexed_attrs for HOT indexed update chain following
break indexam.c:299

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breakpoint and its comment are fictional relative to the actual change. The only modification to indexam.c in this patch is a single line near 352 adding scan->xs_index_only = index_only_scan; inside index_beginscan_internal(); there is no 'indexed_attrs computation' at line 299. The comment describes code that does not exist here.

Comment thread pg-aliases.sh

# Clean commands
alias pg-clean='ninja -C "$PG_BUILD_DIR" clean'
alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Footgun: this alias runs rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR" with no guard against unset/empty variables. These vars are only exported by the Nix devShell hook (shell.nix commonHookTail); if this script is sourced directly outside that shell, both expand to empty and the command collapses to rm -rf "" "", which resolves relative to the current directory. Guard with ${PG_BUILD_DIR:?}/${PG_INSTALL_DIR:?} (or an explicit emptiness check) before destructive deletion. The same applies to pg-init below (rm -rf "$PG_DATA_DIR") and the broad pkill -f "postgres.*-D.*$PG_DATA_DIR" in pg-stop, which with an empty var matches every running postgres.

Comment thread flake.nix
Comment on lines +1 to +2
{
description = "PostgreSQL development environment";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flake is local developer-environment scaffolding unrelated to the HOT indexed updates feature. Bundling it into a feature patch is a non-minimal diff (the patch does more than one thing) and would draw a rejection on pgsql-hackers. It should be kept local, not committed as part of this series.

Comment on lines 2164 to +2167
if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber &&
PageGetMaxOffsetNumber(page) == FirstOffsetNumber)
PageGetMaxOffsetNumber(page) == FirstOffsetNumber &&
((PageHeader) page)->pd_upper ==
((PageHeader) page)->pd_special - MAXALIGN(heaptup->t_len))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This INIT_PAGE gating change (and the mirrored one in log_heap_update) fixes a pre-existing wal_consistency_checking hazard caused by vacuum's lp_truncate_only (PRUNE_VACUUM_CLEANUP) second pass leaving orphan tuple bytes behind an LP_UNUSED slot. Per the comment it is independent of the HOT-indexed/SIU feature this patch introduces. Bundling an unrelated crash/replay-consistency fix into a large feature patch hurts bisectability and reviewability; it should be split into its own commit (and ideally posted separately, since it is backpatchable on its own). (moderate confidence)


#include "storage/itemptr.h"
#include "utils/dsa.h"
#include "utils/relcache.h"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two new prototypes only need the Relation typedef, but this pulls the full relcache.h (and transitively access/tupdesc.h, common/relpath.h, nodes/bitmapset.h) into every includer of this fairly widely-included header. amlocator.h in this same patch deliberately avoids the analogous heavy include and forward-declares struct RelationData instead. Consider doing the same here (forward declare and use struct RelationData * in the prototypes) to keep the dependency minimal. Low confidence / minor.

Comment on lines +372 to +373
* This index was skipped because its key attributes did not
* change. When the overall update is a partial (selective-index)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says the index "was skipped because its key attributes did not change", but the controlling flag ii_IndexNeedsUpdate is the wide rule documented just above (key, INCLUDE, expression, AND partial-predicate columns). An index whose INCLUDE/expression/predicate column changed would set ii_IndexNeedsUpdate=true and not reach this skip path, so "key attributes" is too narrow here. Suggest "its referenced attributes did not change" to match ii_IndexNeedsUpdate's semantics.

Suggested change
* This index was skipped because its key attributes did not
* change. When the overall update is a partial (selective-index)
* This index was skipped because none of the attributes it
* references changed. When the overall update is a partial
* (selective-index)

Comment on lines +1179 to +1182
indexedattrs = RelationGetIndexedAttrs(indexDesc);
indexInfo->ii_IndexNeedsUpdate =
row_moved || bms_overlap(indexedattrs, modified_idx_attrs);
bms_free(indexedattrs);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance: ExecSetIndexUnchanged() runs once per updated row (from ExecUpdateEpilogue / ExecSimpleRelationUpdate), and here calls RelationGetIndexedAttrs(indexDesc) for every index. That function always returns a fresh bms_copy() of the cached rd_indattr (by design, since the cached pointer must not be borrowed across relcache invalidation). So a bulk UPDATE of N rows on a table with M indexes performs N*M palloc+copy+bms_free cycles of an invariant bitmap on the update hot path.

The old index_unchanged_by_update() cached its per-index result via ii_CheckedUnchanged and did the work once per statement. The indexedattrs contents do not vary across rows within a statement; only modified_idx_attrs does. Consider computing/caching the indexed-attrs bitmap once per index per statement (or carrying the overlap result in IndexInfo) instead of recomputing it per row. Confidence: high.

Comment on lines +134 to +135
tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1)
lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grep -oP uses PCRE, which is a GNU-grep extension and is not available in BSD/macOS grep. The README (lines 110-112) explicitly claims these scripts are "portable between Linux and FreeBSD", but this will fail on FreeBSD/macOS, silently producing empty tps/lat and taking the "FAILED (no tps/lat)" path on every iteration. Use a portable parser instead, e.g. awk '/tps = /{print $3}' "$log" | tail -1, consistent with how run.sh extracts TPS.

Confidence: high.

Suggested change
tps=$(grep -oP 'tps = \K[0-9.]+' "$log" | tail -1)
lat=$(grep -oP 'latency average = \K[0-9.]+' "$log" | tail -1)
tps=$(awk '/tps = /{print $3}' "$log" | tail -1)
lat=$(awk '/latency average = /{print $4}' "$log" | tail -1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant