Skip to content

jpa/MPG-1254-1283-1531 : initiate.sh rewritten + add checksums + ship logs no matter what - #2625

Open
jpargudo wants to merge 28 commits into
developfrom
jpa/mpg-1253-enhance_pg_upgrade_scripts
Open

jpargudo wants to merge 28 commits into
developfrom
jpa/mpg-1253-enhance_pg_upgrade_scripts

Conversation

@jpargudo

@jpargudo jpargudo commented Nov 5, 2026

Copy link
Copy Markdown

What kind of change does this PR introduce?

  • hardening prepare.sh and
  • adding 2 features to initiate.sh:
    • taking the upgrade process as an opportunity to add checksums to the projects that don't have it enabled yet (few minutes downtime added)
    • better logging of operations

What is the current behavior?

  • prepare.sh was safe since it runs on a newly created VM, with no access except for us. But there was no logs and no "paranoid mode" on operations (stop PostgreSQL, umount /data, etc.)

  • initiate.sh

    • wasn't shipping the logs in some corner cases
    • wasn't enabling checksums before the upgrade: hence, an PG database with no checksums in a given version was migrated to the destination of PG without checksums enabled neither.

What is the new behavior?

  • prepare.sh has been strongly hardened, probably in "paranoid mode".

initiate.sh now

  • ships logs no matter what happens
  • enables checksums to projects with a size criteria, so that added operation is a great tradeoff: about 5 minutes for a 20 Gb database. A safeguard is added tho: the operation cannot last more than 10 minutes, otherwise the whole upgrade process will fail.

Notes on pg_checksums

The part around enabling checksums is not meant to survive since in PostgreSQL will one day have checksums enabled on-line with pg_enable_data_checksums() or removed the same way with pg_disable_data_checksums(). Unfortunately this feature was due for PG19, but has been removed from that version. We might then clean the code when PG20 is out, late 2027.

@jpargudo
jpargudo requested a review from a team as a code owner November 5, 2026 12:58
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Package version diff

Compares built package versions in legacyPackages between this PR and the tip of the base branch, per system.

aarch64-linux: Closure size: 372 -> 372 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).
<<< /nix/store/sn7f9ik15z493v700qvg4099j2s1hxdd-build-ami
>>> /nix/store/xd79532qaf4m1cd07l9yvci4a4m3ha8p-build-ami
No version or selection state changes.
Closure size: 98 -> 98 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).
<<< /nix/store/c8cq8llp3yhpb1ih5my4l2hkpa55jfgq-build-qemu-image
>>> /nix/store/301dshc5r3d4zbl5xqhwla8whwfpz435-build-qemu-image
No version or selection state changes.
Closure size: 372 -> 372 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).
x86_64-linux: Closure size: 370 -> 370 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).
<<< /nix/store/04vx0frwck76bc37gb471qa0yvg9b8l6-build-ami
>>> /nix/store/pydssfmidq2v5gm4dgxvzs85kbnc2r8f-build-ami
No version or selection state changes.
Closure size: 99 -> 99 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).
<<< /nix/store/dav0cx5d3ck62qjyv8drzsyhk077p7lh-build-qemu-image
>>> /nix/store/hv9qr16f5jbi0vbnh48lfgbir5das3f4-build-qemu-image
No version or selection state changes.
Closure size: 370 -> 370 (2 paths added, 2 paths removed, delta +0, disk usage +4.4KiB).

@encima

encima commented Nov 9, 2026

Copy link
Copy Markdown
Member

These failures are happening due to the pre-commit and treefmt. 2 Options:

  1. One quick, risky fix is to ask Copilot to resolve these
  2. Run nix develop --command pre-commit run --from-ref develop --to-ref HEAD and this will run the pre-commit checks for this branch so you can see and act on the failures.

We will go through these in a call today. It can be tedious but clicking through the job failures will show you which commands failed (it is only a few because the other failures are due to skips or the initial failures)

Co-authored-by: Márton Boros <martonboros@gmail.com>
@blacksmith-sh

This comment has been minimized.

jpargudo and others added 2 commits November 9, 2026 15:10
Co-authored-by: Márton Boros <martonboros@gmail.com>
Co-authored-by: Márton Boros <martonboros@gmail.com>
@blacksmith-sh

This comment has been minimized.

@blacksmith-sh

This comment has been minimized.

@jpargudo
jpargudo requested a review from brainrake November 10, 2026 09:25
jpargudo and others added 3 commits November 10, 2026 14:03
Co-authored-by: Márton Boros <martonboros@gmail.com>
Co-authored-by: Márton Boros <martonboros@gmail.com>
@jpargudo
jpargudo dismissed brainrake’s stale review November 10, 2026 14:49

I did apply all of your last comments, that I did resolve, and applied to 2 patches.
Thank you !!

@mmlb mmlb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few small QoL improvements

jpargudo and others added 7 commits November 12, 2026 07:34
I don't get much the change, since that's explained in the comment, but since it's the same for me, accepted!

Co-authored-by: Manuel Mendez <708570+mmlb@users.noreply.github.com>
I feel like you want the bash code to be more readable here. Good point. Accepted. Thanks!

Co-authored-by: Manuel Mendez <708570+mmlb@users.noreply.github.com>
Same remark as before: code is more readable this way. Accepted. Thanks!

Co-authored-by: Márton Boros <martonboros@gmail.com>
Yes, you're 100% right. I forgot to update the comment here in the hurry of applying changes... Thank you very much.

Co-authored-by: Márton Boros <martonboros@gmail.com>
Yes, it was in the case common.sh is not accessible, but.. You're right. Accepted. Thanks.

Co-authored-by: Márton Boros <martonboros@gmail.com>
Much better... Thank you very much!

Co-authored-by: Márton Boros <martonboros@gmail.com>
Co-authored-by: Manuel Mendez <708570+mmlb@users.noreply.github.com>
@jpargudo
jpargudo requested review from brainrake and mmlb November 12, 2026 08:39

@brainrake brainrake left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applying both STATUS=0 suggestions deleted MAX_DURATION and KILL_AFTER, so every checksum-eligible upgrade aborts under set -u. Also fixes the treefmt failure.

jpargudo and others added 2 commits November 12, 2026 16:46
Co-authored-by: Márton Boros <martonboros@gmail.com>
Co-authored-by: Márton Boros <martonboros@gmail.com>

@mmlb mmlb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm once @brainrake's feedback is addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants