Repository navigation
Add support for ARIES UNDO and constant time recovery #28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
ac48f3b
70cc154
942c59c
0fec969
0af85ae
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Weekly check for a newer Claude Opus on Bedrock; opens/updates one issue here | ||
| # when OCR_BEDROCK_MODEL is behind. Notify-only (GITHUB_TOKEN cannot write | ||
| # Actions variables). Logic lives in gburd/ci-workflows. | ||
|
|
||
| name: OCR model self-check | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 12 * * 1' # Mondays 12:00 UTC | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| check: | ||
| uses: gburd/ci-workflows/.github/workflows/ocr-model-check.yml@v1 | ||
| with: | ||
| aws_role_arn: ${{ vars.AWS_ROLE_ARN }} | ||
| aws_region: ${{ vars.AWS_REGION }} | ||
| bedrock_model: ${{ vars.OCR_BEDROCK_MODEL }} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # AI PR review (Claude Opus on Bedrock). Logic and review rules live in | ||
| # gburd/ci-workflows so this fork's master carries a stub instead of ~1000 | ||
| # lines that have to be rebased onto upstream forever. | ||
| # | ||
| # Needs repo variables AWS_ROLE_ARN, AWS_REGION, OCR_BEDROCK_MODEL. Auth is | ||
| # GitHub OIDC; no static AWS keys. See the ci-workflows README. | ||
|
|
||
| name: OCR AI Review | ||
|
|
||
| on: | ||
| pull_request: | ||
| # Note: no draft filter — drafts are reviewed too. | ||
| types: [opened, synchronize, reopened, ready_for_review] | ||
| issue_comment: | ||
| types: [created] | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| description: 'PR number to review' | ||
| required: true | ||
| type: number | ||
|
|
||
| # One review per PR; cancel superseded runs to save Bedrock spend. | ||
| concurrency: | ||
| group: ocr-review-${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| review: | ||
| # Comment events only when the comment is on a PR and starts with the | ||
| # trigger keyword; PR events and manual dispatch always. | ||
| if: | | ||
| github.event_name == 'pull_request' || | ||
| github.event_name == 'workflow_dispatch' || | ||
| (github.event_name == 'issue_comment' && github.event.issue.pull_request && | ||
| (startsWith(github.event.comment.body, '/open-code-review') || | ||
| startsWith(github.event.comment.body, '@open-code-review') || | ||
| startsWith(github.event.comment.body, '/pg-history'))) | ||
| uses: gburd/ci-workflows/.github/workflows/ocr-review.yml@v1 | ||
| with: | ||
| aws_role_arn: ${{ vars.AWS_ROLE_ARN }} | ||
| aws_region: ${{ vars.AWS_REGION }} | ||
| bedrock_model: ${{ vars.OCR_BEDROCK_MODEL }} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Keep this fork's master rebased on postgres/postgres master, carrying only | ||
| # the fork-owned CI stubs (.github/workflows/fork-*.yml). The workflows they | ||
| # call, and the OCR review rules, live in gburd/ci-workflows. | ||
| # | ||
| # Requires the SYNC_TOKEN secret: a PAT with repo+workflow scope. The default | ||
| # GITHUB_TOKEN is not allowed to push commits that touch .github/workflows/, | ||
| # and every commit we carry does. | ||
|
|
||
| name: Sync from upstream | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '17 */3 * * *' | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: fork-sync-upstream | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| sync: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| token: ${{ secrets.SYNC_TOKEN }} | ||
|
|
||
| - name: Rebase master onto upstream | ||
| run: | | ||
| set -euo pipefail | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "github-actions[bot]@users.noreply.github.com" | ||
| git remote add upstream https://github.com/postgres/postgres.git | ||
| git fetch --no-tags upstream master | ||
|
|
||
| # Master carries fork CI only. Anything else means work landed here | ||
| # that belongs on a branch; force-rebasing it on a timer would be wrong. | ||
| if git diff --name-only upstream/master...HEAD | | ||
| grep -v '^\.github/workflows/fork-'; then | ||
| echo "::error::master carries non-CI changes (listed above); move them to a branch" | ||
| exit 1 | ||
| fi | ||
|
|
||
| git rebase upstream/master | ||
| git push --force-with-lease |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| <!-- doc/src/sgml/undo.sgml --> | ||
|
|
||
| <chapter id="undo" xreflabel="UNDO Logging"> | ||
| <title>UNDO Logging</title> | ||
|
|
||
| <indexterm zone="undo"> | ||
| <primary>UNDO logging</primary> | ||
| </indexterm> | ||
|
|
||
| <para> | ||
| <productname>PostgreSQL</productname> provides UNDO logging infrastructure | ||
| that records the inverse of a change, so that the change can be reversed if | ||
| the transaction that made it rolls back or is still in progress when the | ||
| server crashes. UNDO records are written into the WAL stream, so they are | ||
| durable, replicated and archived like every other WAL record. | ||
| </para> | ||
|
|
||
| <para> | ||
| UNDO is not a cluster-wide switch. It is always-on infrastructure that is | ||
| used only by the subsystems that write UNDO records. The default | ||
| <literal>heap</literal> table access method does not, and ordinary heap and | ||
| index changes carry no UNDO overhead. The one user of UNDO in | ||
| <productname>PostgreSQL</productname> today is transactional filesystem | ||
| operations (FILEOPS), which make the directory and file changes done by | ||
| <command>CREATE DATABASE</command>, <command>ALTER DATABASE SET | ||
| TABLESPACE</command>, <command>CREATE TABLESPACE</command> and | ||
| <command>DROP TABLESPACE</command> roll back with the transaction. | ||
| </para> | ||
|
|
||
| <sect1 id="undo-fileops"> | ||
| <title>Transactional Filesystem Operations</title> | ||
|
|
||
| <para> | ||
| When a transaction creates, renames or writes a file, creates a directory, | ||
| changes a file's mode or owner, truncates a file, or creates a link, | ||
| FILEOPS makes the change and then writes a WAL record describing it and an | ||
| UNDO record describing how to reverse it. A change that fails is an | ||
| ordinary <literal>ERROR</literal> and leaves nothing in the WAL. If the | ||
| transaction commits, the UNDO record is never used. If it rolls back, the | ||
| abort record carries the inverses, and the backend applies them, newest | ||
| first, before <command>ROLLBACK</command> returns: a created file is | ||
| removed, a changed mode is restored, a new database directory is removed | ||
| with everything in it, and so on. Removing a file or directory is not | ||
| done immediately; it is deferred to commit and simply dropped on rollback. | ||
| </para> | ||
|
|
||
| <para> | ||
| Between the change and its WAL record there is a short window in which a | ||
| crash leaves the change in place with nothing to reverse it, as it would | ||
| without FILEOPS. Doing it the other way round, logging first, would turn | ||
| every ordinary filesystem error into a <literal>PANIC</literal>. | ||
| </para> | ||
|
|
||
| <para> | ||
| FILEOPS writes WAL and UNDO only when <xref linkend="guc-wal-level"/> is | ||
| <literal>replica</literal> or higher. At <literal>minimal</literal> it | ||
| writes neither, and a crash in the middle of a transaction can leave a | ||
| partly done filesystem change behind, as without FILEOPS. | ||
| </para> | ||
| </sect1> | ||
|
|
||
| <sect1 id="undo-crash-recovery"> | ||
| <title>Crash Recovery and Replication</title> | ||
|
|
||
| <para> | ||
| At the end of crash recovery, after redo, the startup process finds every | ||
| transaction that wrote UNDO records but neither committed nor aborted | ||
| before the crash, applies its UNDO records, and writes a WAL record of the | ||
| rollback, before the server accepts connections. Each checkpoint lists | ||
| the UNDO chains of transactions in progress, so this includes a | ||
| transaction whose UNDO records all precede the checkpoint redo starts | ||
| from. This step takes time proportional to | ||
| the number of UNDO records those transactions wrote. A prepared | ||
| transaction is left prepared; <command>ROLLBACK PREPARED</command> | ||
| applies its UNDO records later, and <command>COMMIT PREPARED</command> | ||
| discards them. | ||
| </para> | ||
|
|
||
| <para> | ||
| An UNDO record that cannot be applied, for example because the server no | ||
| longer has permission to change the file it describes, is reported with a | ||
| <literal>WARNING</literal> and is not retried, and the change it describes | ||
| stays in place. The end-of-recovery summary then also reports how many | ||
| records could not be applied. | ||
| </para> | ||
|
|
||
| <para> | ||
| A physical standby replays the WAL record of each filesystem change, and | ||
| the inverses carried by each abort record, so it ends up with the same | ||
| files as the primary. Crash recovery replays them the same way. A | ||
| transaction lost in a crash of the primary writes no abort record; the | ||
| primary's crash recovery writes the rollback record instead, and a | ||
| standby replays it. | ||
| </para> | ||
| </sect1> | ||
|
|
||
| <sect1 id="undo-space-management"> | ||
| <title>UNDO Space Management</title> | ||
|
|
||
| <para> | ||
| UNDO records live in the WAL stream as <literal>XLOG_UNDO_BATCH</literal> | ||
| records; there are no separate UNDO files. A checkpoint does not remove | ||
| WAL that still holds the UNDO records of a transaction that is in progress | ||
| or prepared, so a long-running or long-prepared transaction that wrote UNDO | ||
| holds back WAL removal until it ends, much as a replication slot does. | ||
| </para> | ||
| </sect1> | ||
|
|
||
| <sect1 id="undo-monitoring"> | ||
| <title>Monitoring</title> | ||
|
|
||
| <para> | ||
| The server log shows UNDO being applied during crash recovery: | ||
| </para> | ||
|
|
||
| <programlisting> | ||
| LOG: UNDO recovery: rolling back transaction 742 | ||
| LOG: UNDO recovery complete: 1 transactions rolled back, 3 records applied | ||
| </programlisting> | ||
|
|
||
| <para> | ||
| and, if some records could not be applied: | ||
| </para> | ||
|
|
||
| <programlisting> | ||
| WARNING: UNDO recovery complete: 1 transactions processed, 0 records applied, 1 records could not be applied | ||
| DETAIL: The changes those records describe were not rolled back. | ||
| </programlisting> | ||
|
|
||
| <para> | ||
| <application>pg_waldump</application> shows UNDO records with | ||
| <option>--rmgr=Undo</option> and the filesystem changes themselves with | ||
| <option>--rmgr=FileOps</option>. | ||
| </para> | ||
| </sect1> | ||
|
|
||
| </chapter> |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -22,6 +22,7 @@ SUBDIRS = \ | |||||||||||||||||||||||||
| sequence \ | ||||||||||||||||||||||||||
| table \ | ||||||||||||||||||||||||||
| tablesample \ | ||||||||||||||||||||||||||
| transam | ||||||||||||||||||||||||||
| transam \ | ||||||||||||||||||||||||||
| undo \ | ||||||||||||||||||||||||||
|
Comment on lines
+25
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The final SUBDIRS entry carries a dangling line-continuation backslash (
Suggested change
Comment on lines
+25
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Trailing backslash on the last list element is inconsistent with the tree's Makefile style (the previous last entry
Suggested change
Comment on lines
+25
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Style: the last SUBDIRS entry now has a dangling trailing backslash followed by a blank line. The PostgreSQL convention (see src/backend/Makefile, src/backend/storage/Makefile) is that the final SUBDIRS entry carries no continuation backslash. GNU Make tolerates this, but it deviates from the tree convention and leaves the diff reading non-idiomatically. Drop the trailing backslash after Confidence: high.
Suggested change
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| include $(top_srcdir)/src/backend/common.mk | ||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,3 +14,4 @@ subdir('spgist') | |
| subdir('table') | ||
| subdir('tablesample') | ||
| subdir('transam') | ||
| subdir('undo') | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last
SUBDIRSentry here carries a trailing\followed by a blank line. The tree convention is that the final element of aSUBDIRS/list has no line-continuation backslash (e.g.src/backend/Makefileends its list withjitand no\; this file previously ended withtransamand no\). GNU Make tolerates the dangling\+ empty line, but it is a needless deviation from the established style and leaves a stray empty continuation element. Drop the trailing backslash afterundo.