From f442d8fc6a74323bc0d0180babe7b98c9866ff71 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 4 Aug 2026 10:20:52 +0000 Subject: [PATCH 1/3] docs(postgresql): add solution for Bus error (exit 135) on hugepage-enabled nodes PostgreSQL dies with SIGBUS during bootstrap on any node where vm.nr_hugepages is non-zero, because kubelet caps hugetlb at 0 on the pod slice of pods that do not request hugepages, and that limit is charged at page-fault time rather than at mmap. Documents the two-level cgroup diagnosis (the in-pod view is unable to show the refusal), the version-dependent fix for both the Zalando and CloudNativePG operators, and the hugepage-request alternative. Reproduced and all four remedies verified on ACP 4.2 / k8s 1.33.7 / cgroup v1. Co-Authored-By: Claude Opus 5 (1M context) --- ...xit_135_on_nodes_with_hugepages_enabled.md | 248 ++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md diff --git a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md new file mode 100644 index 000000000..efaff7b73 --- /dev/null +++ b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md @@ -0,0 +1,248 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - 4.1,4.2,4.3 +--- + +# PostgreSQL fails to start with "Bus error" (exit 135) on nodes with hugepages enabled + +## Issue + +After hugepages are reserved on a node (`vm.nr_hugepages` set to a non-zero value), PostgreSQL +instances scheduled onto that node stop starting. The pod log shows: + +```text +running bootstrap script ... Bus error (core dumped) +child process exited with exit code 135 +``` + +Exit code `135` is `128 + 7` = **SIGBUS**. Two details make this hard to recognise: + +- **The two "small default" lines are a symptom, not a setting.** A log containing + `selecting default "max_connections" ... 20` together with + `selecting default "shared_buffers" ... 400kB` means `initdb` tried a `postgres --boot` + child at every candidate size and **all of them crashed**, so it kept the floor values. + PostgreSQL did not choose a small configuration. +- **With the Zalando operator the failure is invisible at the Kubernetes level.** The pod + reports `1/1 Running` while Patroni loops: + `removing initialize key after failed attempt to bootstrap the cluster` → + `PatroniFatalException: Failed to bootstrap cluster`, and runit restarts it roughly every + 30 seconds. Never conclude the instance is healthy from pod readiness alone — connect with + `psql`. With CloudNativePG the `-initdb` Job goes to `Error` and retries visibly. + +The problem is not specific to PostgreSQL. Any containerised workload that opportunistically +uses hugepages — MySQL, MongoDB, JVM applications with `-XX:+UseLargePages` — is exposed on +the same node. + +## Environment + +Reproduced and fixed on: + +- Alauda Container Platform 4.2, Kubernetes v1.33.7, containerd 1.7.29, **cgroup v1** +- Worker node: amd64, kernel 3.10, **single NUMA node**, `cpuManagerPolicy` not enabled +- CloudNativePG operator v1.29.1-acp.1 with PostgreSQL 18.4, and the Zalando operator with + `spilo v4.3.0-beta.36` + +The mechanism is a property of kubelet plus the hugetlb cgroup controller, so it is not +limited to these versions. The exact cgroup paths below were verified on cgroup v1; on cgroup +v2 the equivalent file is `hugetlb..max` under the same pod slice. + +## Root cause + +Two independent facts combine: + +1. **kubelet caps hugepages at zero for pods that do not ask for them.** For every pod without + a `hugepages-` resource request, kubelet writes + `hugetlb..limit_in_bytes = 0` on the **pod-level** cgroup + (`kubepods--pod.slice`). +2. **hugetlb reservation and allocation happen at different times.** `mmap()` with + `MAP_HUGETLB` checks the reservation against the **global** pool and succeeds. The cgroup + limit is only charged when a page is **faulted in**. A refusal inside a page fault has no + errno to return, so the kernel delivers **SIGBUS**. + +PostgreSQL's default `huge_pages=try` only falls back to normal pages when `mmap()` *returns +an error*. Here `mmap()` succeeds, so PostgreSQL commits to hugepages and dies on first touch. + +This is why the failure appears only on nodes where hugepages exist. On a node with no pool at +all, `mmap(MAP_HUGETLB)` fails immediately with `ENOMEM` and PostgreSQL falls back silently. +**A node that advertises hugepages the pod cannot actually obtain is more dangerous than a +node with none.** + +## Diagnosis + +### Confirm it is hugepages + +Reproduce in seconds inside the container, without waiting for the operator to retry: + +```bash +kubectl -n exec -- initdb -D /tmp/hugetlb-check -U postgres +``` + +A `Bus error (core dumped)` confirms it. (Clean up `/tmp/hugetlb-check` afterwards.) + +### Read the cgroup counters at the correct level + +> **This is the step that is easy to get wrong.** The `/sys/fs/cgroup` view inside a pod is the +> **container** scope, which is *not* where kubelet writes the limit. It reads `unlimited` with +> `failcnt` permanently `0`, because a child cgroup cannot see refusals recorded by its parent. +> An in-pod reading of zero does **not** exonerate the cgroup. + +Read the **pod slice**, from the host: + +```bash +POD_UID=$(kubectl -n get pod -o jsonpath='{.metadata.uid}' | tr - _) +D=/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod${POD_UID}.slice +cat $D/hugetlb.2MB.limit_in_bytes $D/hugetlb.2MB.failcnt +``` + +Adjust `burstable` to `besteffort`, or drop that path element for Guaranteed pods. A `limit` of +`0` with a `failcnt` that increments on every crash confirms the diagnosis. Measured values +around a single failing `initdb`: + +| cgroup level | `limit_in_bytes` | `failcnt` before → after | +| --- | --- | --- | +| pod slice | `0` | `0` → `25` | +| container scope (visible inside the pod) | unlimited | `0` → `0` | + +`HugePages_Free` in `/proc/meminfo` does not move during the failure: the pages are reserved, +refused at fault time, and never allocated. + +### Check whether the node actually advertises hugepages + +```bash +kubectl get node -o jsonpath='{.status.allocatable.hugepages-2Mi}{"\n"}' +``` + +kubelet caches machine information, so a runtime `sysctl -w vm.nr_hugepages=N` is **not +visible to kubelet until it is restarted**. A node with a live pool but +`hugepages-2Mi: 0` is the worst state: pods are capped at zero **and** cannot request +hugepages, because the scheduler does not believe the resource exists. If the pool was enabled +online, restart kubelet on that node before using the "request hugepages" option below. + +## Resolution + +Pick a row. Do **not** disable the node's hugepage pool as a first move — on a shared node +those pages usually belong to another workload, and removing them has taken applications down. + +| Situation | Action | +| --- | --- | +| Instance already bootstrapped, only failing to restart | Set the `huge_pages: "off"` parameter. Works on every major version. | +| Fresh bootstrap needed, PostgreSQL **16 or later** | Set the parameter **and** pass `huge_pages=off` to `initdb`. | +| Fresh bootstrap needed, PostgreSQL **15 or earlier** | No `initdb` option exists. Request hugepages instead, or bootstrap on a node without a pool and move the instance afterwards. | +| PostgreSQL should genuinely use the hugepages | Request `hugepages-` in `resources` and set `huge_pages: "on"`. | + +### Why the parameter alone is not enough for a fresh bootstrap + +Both operators write `postgresql.conf` **after** `initdb` completes. The crash happens in +`initdb`'s bootstrap child, before that file exists. A cluster created with only +`huge_pages: "off"` still fails to bootstrap — verified. PostgreSQL 16 added +`initdb -c NAME=VALUE` (`--set`), and it is honoured by the bootstrap child, which gives a +configuration-only route on 16+. + +### Zalando operator + +```yaml +spec: + postgresql: + parameters: + huge_pages: "off" + patroni: + initdb: + set: "huge_pages=off" +``` + +Patroni renders each `{key: value}` entry as `--key=value`, and PostgreSQL 16's option is +`-c, --set`. The key must therefore be `set`; writing `c` produces an invalid `--c=...` flag. + +### CloudNativePG operator + +```yaml +spec: + postgresql: + parameters: + huge_pages: "off" + bootstrap: + initdb: + options: ["-c", "huge_pages=off"] +``` + +> **Caution.** `bootstrap.initdb.options` is a deprecated field, and setting it makes the +> operator ignore every explicit `initdb` setting — `dataChecksums`, `encoding`, +> `localeCollate`, `localeCType`, `walSegmentSize`. If you rely on any of them, restate them as +> raw flags in the same list, for example +> `["-c", "huge_pages=off", "--encoding=UTF8", "--lc-collate=C", "--lc-ctype=C", "-k"]`. + +### Option: let PostgreSQL use the hugepages + +If the pool is meant for the database, give the pod a real limit instead of removing the +request. This fixes both bootstrap and steady state: + +```yaml +spec: + postgresql: + parameters: + huge_pages: "on" + shared_buffers: 256MB + resources: + requests: { cpu: 200m, memory: 1Gi, hugepages-2Mi: 512Mi } + limits: { cpu: "1", memory: 1Gi, hugepages-2Mi: 512Mi } +``` + +Size the request from `postgres -C shared_memory_size_in_huge_pages` plus headroom. An +exact-fit limit means any later growth of the shared memory segment faults into SIGBUS instead +of failing cleanly at startup. The node must advertise `hugepages-2Mi` — see the kubelet +restart note above. + +## Verification + +Pod readiness is not a valid check, because the Zalando operator reports `1/1 Running` +throughout the failure. Use these instead. + +The instance is actually serving, and using the intended memory type: + +```bash +kubectl -n exec -- psql -U postgres -tAc \ + "select 'ALIVE', current_setting('huge_pages'), current_setting('shared_buffers')" +``` + +Expect `ALIVE|off|...` for the disable path, or `ALIVE|on|...` for the request path. + +No further cgroup refusals — `failcnt` stops incrementing on the pod slice (same command as in +Diagnosis). + +For the request path only, confirm the pages are genuinely held, on the node: + +```bash +grep -iE 'hugepages_total|hugepages_free|hugepages_rsvd' /proc/meminfo +``` + +Read **`HugePages_Rsvd`**, not `HugePages_Free`. Pages that are reserved but not yet touched +still count as free, so `HugePages_Free` understates usage and is not a reliable indicator. + +## Limitations and trade-offs + +- Setting `huge_pages: "off"` gives up the TLB benefit of hugepages for that instance. For most + ACP PostgreSQL workloads this is not measurable; if the database is the reason hugepages were + reserved, use the request path instead. +- The fix is per-cluster. Every PostgreSQL instance that may land on a hugepage-enabled node + needs it, and so does every other hugepage-capable workload on those nodes. +- Requesting hugepages does **not** protect against a node whose free pages are unreachable for + another reason, for example a restrictive `cpuset.mems` on a multi-NUMA machine with an + unevenly distributed pool. Disabling hugepages in the database does, because no + `MAP_HUGETLB` mapping is created at all. +- `vm.nr_hugepages=0` on the node is a last resort. Identify who consumes the pool first; the + database is the workload that should adapt, since it is the one opportunistically taking + pages it cannot fault in. + +## Related + +- `How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md` — moving an instance off an + affected node when no in-place option exists (PostgreSQL 15 and earlier). +- `CX_series_KubeVirt_VM_fails_to_start_with_FailedScheduling_on_dedicated_CPU_and_hugepages.md` + — the opposite failure mode for the same resource. A workload that *requests* + `hugepages-2Mi` and cannot get it stays `Pending` with `Insufficient hugepages-2Mi`, which is + an honest, visible failure. The SIGBUS described here happens precisely because the workload + never requested the resource, so nothing stopped it from being scheduled. From eb9c86d09eaf0b445bf3bd1892e0b112c2d43703 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 4 Aug 2026 10:46:37 +0000 Subject: [PATCH 2/3] docs(postgresql): apply review findings to the hugepages SIGBUS solution Static review (codex, passes 1+3), an independent domain cold-read, and my own pass-3 self-test of every recommended command. Confirmed findings only: - initdb.options discards 10 explicit fields, not the 5 originally listed; locale, localeProvider, icuLocale, icuRules and builtinLocale were missing, so ICU-configured clusters were the most exposed to silent loss. - shared_memory_size_in_huge_pages does not exist before PostgreSQL 15 (verified: 13/14 return "unrecognized configuration parameter"), yet the version table routed PG13/14 readers straight at it. Added a manual fallback. - The sizing command could not run as written: it requires -D/PGDATA, and no form works pre-bootstrap. It also returns a page COUNT, not a size - the same default shared_buffers reports 75 with 2MiB pages and 1 with 512MiB pages. - Page size is node-dependent. arm64 Kylin v10 defaults to 512MiB, so hugepages-2Mi is the wrong resource name there and the pod stays Pending. - The pod-UID snippet swallowed kubectl failure through a pipe (empty UID, status 0); added pipefail and an emptiness guard. - Scoped the cgroup paths to the systemd cgroup driver and gave the cgroupfs shape; noted that blank jsonpath output is a result, not an error. Co-Authored-By: Claude Opus 5 (1M context) --- ...xit_135_on_nodes_with_hugepages_enabled.md | 113 +++++++++++++++--- 1 file changed, 95 insertions(+), 18 deletions(-) diff --git a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md index efaff7b73..eac1a140b 100644 --- a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md +++ b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md @@ -46,9 +46,13 @@ Reproduced and fixed on: - CloudNativePG operator v1.29.1-acp.1 with PostgreSQL 18.4, and the Zalando operator with `spilo v4.3.0-beta.36` -The mechanism is a property of kubelet plus the hugetlb cgroup controller, so it is not -limited to these versions. The exact cgroup paths below were verified on cgroup v1; on cgroup -v2 the equivalent file is `hugetlb..max` under the same pod slice. +Everything asserted below was observed in that environment. The underlying mechanism is a +property of kubelet plus the hugetlb cgroup controller rather than of these specific versions, +so it is expected to generalise — but the concrete paths, filenames and numbers here are the +tested ones, and the cgroup layout in particular depends on the **systemd** cgroup driver +(`cgroupDriver: systemd`) and the default kubelet cgroup root. On cgroup v2 the equivalent file +is `hugetlb..max`; the filename is correct, but the unified hierarchy means the mount +point differs, and that combination was not tested here. ## Root cause @@ -83,6 +87,12 @@ kubectl -n exec -- initdb -D /tmp/hugetlb-check -U postgres A `Bus error (core dumped)` confirms it. (Clean up `/tmp/hugetlb-check` afterwards.) +`` can be **any** pod on the suspect node that contains PostgreSQL binaries — it does not +have to be the failing one, and often cannot be. With CloudNativePG the crashing entity is the +`-initdb` Job pod, which runs with `restartPolicy: Never` and may already be `Terminated` by +the time you look, so `exec` returns `container not running`. Start a throwaway pod pinned to +that node with the same image instead. + ### Read the cgroup counters at the correct level > **This is the step that is easy to get wrong.** The `/sys/fs/cgroup` view inside a pod is the @@ -93,14 +103,30 @@ A `Bus error (core dumped)` confirms it. (Clean up `/tmp/hugetlb-check` afterwar Read the **pod slice**, from the host: ```bash -POD_UID=$(kubectl -n get pod -o jsonpath='{.metadata.uid}' | tr - _) +set -o pipefail +POD_UID=$(kubectl -n get pod -o jsonpath='{.metadata.uid}' | tr - _) \ + || { echo "kubectl failed"; exit 1; } +[ -n "$POD_UID" ] || { echo "pod not found"; exit 1; } D=/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod${POD_UID}.slice -cat $D/hugetlb.2MB.limit_in_bytes $D/hugetlb.2MB.failcnt +cat "$D/hugetlb.2MB.limit_in_bytes" "$D/hugetlb.2MB.failcnt" ``` -Adjust `burstable` to `besteffort`, or drop that path element for Guaranteed pods. A `limit` of -`0` with a `failcnt` that increments on every crash confirms the diagnosis. Measured values -around a single failing `initdb`: +`set -o pipefail` matters: without it the assignment takes the exit status of `tr`, so a failed +`kubectl` yields status 0 and an empty UID, and you go on to read a path built from nothing. + +> This path is the **systemd** cgroup-driver layout, which is what the tested cluster uses +> (`cgroupDriver: systemd` in `/var/lib/kubelet/config.yaml`) and the common default. Under the +> `cgroupfs` driver the shape differs — `/sys/fs/cgroup/hugetlb/kubepods/burstable/pod/`, +> with the UID keeping its hyphens and no `.slice` suffix. Check `cgroupDriver` on the node +> before assuming either. A non-default `--cgroup-root` shifts the prefix as well. + +Adjust `burstable` to `besteffort`, or drop that path element entirely for Guaranteed pods — +only Burstable and BestEffort get a QoS-level cgroup. Substitute the node's default hugepage +size in the filenames: `hugetlb.2MB.*` on x86_64, `hugetlb.512MB.*` on arm64 builds that +default to 512 MiB pages (`grep Hugepagesize /proc/meminfo`). + +A `limit` of `0` with a `failcnt` that increments on every crash confirms the diagnosis. +Measured values around a single failing `initdb`: | cgroup level | `limit_in_bytes` | `failcnt` before → after | | --- | --- | --- | @@ -116,6 +142,10 @@ refused at fault time, and never allocated. kubectl get node -o jsonpath='{.status.allocatable.hugepages-2Mi}{"\n"}' ``` +(Use the resource name matching the node's default page size — `hugepages-512Mi` on arm64 +builds that default to 512 MiB pages.) **Blank output means the node is not advertising that +hugepage size** — the command still exits 0, so an empty line is a result, not a failure. + kubelet caches machine information, so a runtime `sysctl -w vm.nr_hugepages=N` is **not visible to kubelet until it is restarted**. A node with a live pool but `hugepages-2Mi: 0` is the worst state: pods are capped at zero **and** cannot request @@ -129,7 +159,7 @@ those pages usually belong to another workload, and removing them has taken appl | Situation | Action | | --- | --- | -| Instance already bootstrapped, only failing to restart | Set the `huge_pages: "off"` parameter. Works on every major version. | +| Instance already bootstrapped, only failing to restart | Set the `huge_pages: "off"` parameter. Applies to every major version in scope here (13-18). | | Fresh bootstrap needed, PostgreSQL **16 or later** | Set the parameter **and** pass `huge_pages=off` to `initdb`. | | Fresh bootstrap needed, PostgreSQL **15 or earlier** | No `initdb` option exists. Request hugepages instead, or bootstrap on a node without a pool and move the instance afterwards. | | PostgreSQL should genuinely use the hugepages | Request `hugepages-` in `resources` and set `huge_pages: "on"`. | @@ -169,11 +199,20 @@ spec: options: ["-c", "huge_pages=off"] ``` -> **Caution.** `bootstrap.initdb.options` is a deprecated field, and setting it makes the -> operator ignore every explicit `initdb` setting — `dataChecksums`, `encoding`, -> `localeCollate`, `localeCType`, `walSegmentSize`. If you rely on any of them, restate them as -> raw flags in the same list, for example -> `["-c", "huge_pages=off", "--encoding=UTF8", "--lc-collate=C", "--lc-ctype=C", "-k"]`. +> **Caution.** `bootstrap.initdb.options` is a deprecated field. Setting it makes +> `buildInitDBFlags` return early, so the operator ignores **every** explicit `initdb` setting: +> `dataChecksums`, `encoding`, `locale`, `localeCollate`, `localeCType`, `localeProvider`, +> `icuLocale`, `icuRules`, `builtinLocale`, and `walSegmentSize`. The cluster-level +> `logLevel` → `-d` side effect is skipped too. Diff your current `bootstrap.initdb` block +> before adding `options`, and restate everything you rely on as raw flags in the same list: +> +> ```yaml +> options: ["-c", "huge_pages=off", "--encoding=UTF8", "--lc-collate=C", "--lc-ctype=C", "-k"] +> ``` +> +> Verified in that argv order: `initdb` exits 0, `postgresql.conf` carries `huge_pages = off`, +> and the resulting cluster reports UTF8 / C / C with `data_checksums=on`. Clusters using ICU +> locale settings are the ones most at risk of silent loss here. ### Option: let PostgreSQL use the hugepages @@ -191,10 +230,48 @@ spec: limits: { cpu: "1", memory: 1Gi, hugepages-2Mi: 512Mi } ``` -Size the request from `postgres -C shared_memory_size_in_huge_pages` plus headroom. An -exact-fit limit means any later growth of the shared memory segment faults into SIGBUS instead -of failing cleanly at startup. The node must advertise `hugepages-2Mi` — see the kubelet -restart note above. +**PostgreSQL 15 and later** can estimate the requirement for you. The parameter needs a data +directory — it cannot be queried on an empty container: + +```bash +# against an existing instance +kubectl -n exec -- bash -c \ + 'postgres -D "$PGDATA" -c shared_buffers=256MB -C shared_memory_size_in_huge_pages' +``` + +**It returns a page count, not a size.** Multiply by the node's default hugepage size to get the +resource quantity. The same default `shared_buffers` returns `75` on a node with 2 MiB pages +and `1` on a node with 512 MiB pages — writing the raw number into +`hugepages-2Mi: Mi` under-provisions by the page-size factor. + +Measured on PostgreSQL 18.4 with 2 MiB pages: 75 pages at the 128 MB default, 543 at +`shared_buffers=1GB`, 4222 at `8GB` — roughly `shared_buffers` plus 10-15% of control +structures, so budget headroom rather than an exact fit. An exact-fit limit means any later +growth of the shared memory segment faults into SIGBUS instead of failing cleanly at startup. + +**PostgreSQL 14 and earlier do not have this parameter** — `postgres -C +shared_memory_size_in_huge_pages` returns `FATAL: unrecognized configuration parameter` +(verified on 13 and 14; present from 15). Size those manually: `shared_buffers` plus about 15%, +divided by the page size, rounded up. + +Before a first bootstrap there is no data directory on any version, so the query is unavailable +— there is no `--config-file` shortcut, `postgres` still demands `-D`. Either run `initdb` into +a throwaway directory purely to take the measurement, or use the manual estimate. + +> **Match the page size to the node.** The resource name and the page count both follow the +> node's *default* hugepage size, which is what `huge_pages=on` consumes +> (`huge_page_size = 0` means "use the system default"). Check it first: +> +> ```bash +> grep Hugepagesize /proc/meminfo +> ``` +> +> x86_64 nodes are normally `2048 kB` → request `hugepages-2Mi`. Some arm64 builds, including +> Kylin Linux Advanced Server V10 on 64 KiB pages, default to `524288 kB` → the resource is +> `hugepages-512Mi` and every count above is in 512 MiB units. Requesting `hugepages-2Mi` on +> such a node asks for a resource it does not advertise, and the pod stays `Pending`. + +The node must also advertise the resource at all — see the kubelet restart note above. ## Verification From 7a72b9545020a60ee896fcefa51b9d82e4372f54 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 4 Aug 2026 11:20:51 +0000 Subject: [PATCH 3/3] docs(postgresql): add tested recovery for an already-failed CNPG bootstrap The cold-read flagged that patching the Cluster stops working once the initdb Job exhausts its backoff. Confirmed, but the suggested remedy (delete the failed Job) does not work, so this documents what does, from an end-to-end run: - On v1.29.1-acp.1 the cluster gives no signal at all: status.phase stays "Setting up primary" while the reconciler hot-loops on "Selected PVC is not ready yet". Diagnose from the Job and status.danglingPVC instead. v1.29.2+ carries upstream #11035 and surfaces it in the phase; v1.29.1-acp.1 does not (verified by merge-base against the tags). - Patching the spec: no effect. Deleting the Job: no effect. - Deleting the PVC alone is actively harmful - it trips a second Unrecoverable branch ("no PersistentVolumeClaims exist ... restore from a recent backup"). - What works: delete and recreate the Cluster with the fix already in the spec, which is safe only because bootstrap never completed. Verified the recreated cluster reaches Running with huge_pages=off on a hugepage-enabled node. Co-Authored-By: Claude Opus 5 (1M context) --- ...xit_135_on_nodes_with_hugepages_enabled.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md index eac1a140b..53c980333 100644 --- a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md +++ b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md @@ -214,6 +214,51 @@ spec: > and the resulting cluster reports UTF8 / C / C with `data_checksums=on`. Clusters using ICU > locale settings are the ones most at risk of silent loss here. +### If a CloudNativePG bootstrap has already been failing for a while + +Patching the Cluster spec only works if the `-initdb` Job is still retrying. Once it has +exhausted its backoff, **the fix above has no effect on its own** — and the cluster gives you +almost no signal that this has happened. Sequence observed end-to-end on operator +`v1.29.1-acp.1`: + +1. The `-initdb` Job runs `restartPolicy: Never` with the default `backoffLimit: 6`, so it + stops after 7 pod attempts (about 10 minutes of exponential backoff) and is marked + `Failed / BackoffLimitExceeded`. +2. **`status.phase` stays `Setting up primary` regardless.** The reconciler is stuck earlier, + logging `Selected PVC is not ready yet, waiting for 1 second` in a tight loop, because the + PVC from the failed attempt never received its ready marker. Diagnose from the Job, not the + Cluster phase: + + ```bash + kubectl -n get jobs -l cnpg.io/cluster= + kubectl -n get cluster -o jsonpath='{.status.danglingPVC}{"\n"}' + ``` + + A `Failed` Job plus a non-empty `danglingPVC` is this state. (Operator v1.29.2 and later + include upstream #11035, "report failed instance creation jobs instead of waiting forever", + which surfaces it in the phase instead. `v1.29.1-acp.1` does not.) +3. Patching the spec does nothing — the failed Job is not replaced. +4. Deleting the Job does not help either; the PVC loop continues. +5. **Do not delete the PVC by itself.** That moves the cluster into a second and worse state: + `One or more instances were previously created, but no PersistentVolumeClaims (PVCs) exist. + The cluster is in an unrecoverable state. To resolve this, restore the cluster from a recent + backup.` + +**Recovery: delete the Cluster and recreate it with the fix already in the spec.** + +```bash +kubectl -n delete cluster +kubectl -n apply -f cluster-with-huge-pages-off.yaml +``` + +This is safe *specifically because bootstrap never completed* — there is no data in that PVC to +lose. Verified: the recreated cluster reached `Running` with `huge_pages=off` on the same +hugepage-enabled node. **Never do this to a cluster that bootstrapped successfully and later +failed to restart** — that one has data, and the GUC-only fix in the table above applies. + +The practical conclusion is to put the fix in the manifest *before* first apply on any node +that has, or might get, a hugepage pool. + ### Option: let PostgreSQL use the hugepages If the pool is meant for the database, give the pod a real limit instead of removing the