diff --git a/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md new file mode 100644 index 000000000..53c980333 --- /dev/null +++ b/docs/en/solutions/PostgreSQL_fails_to_start_with_Bus_error_exit_135_on_nodes_with_hugepages_enabled.md @@ -0,0 +1,370 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - 4.1,4.2,4.3 +--- + +# PostgreSQL fails to start with "Bus error" (exit 135) on nodes with hugepages enabled + +## Issue + +After hugepages are reserved on a node (`vm.nr_hugepages` set to a non-zero value), PostgreSQL +instances scheduled onto that node stop starting. The pod log shows: + +```text +running bootstrap script ... Bus error (core dumped) +child process exited with exit code 135 +``` + +Exit code `135` is `128 + 7` = **SIGBUS**. Two details make this hard to recognise: + +- **The two "small default" lines are a symptom, not a setting.** A log containing + `selecting default "max_connections" ... 20` together with + `selecting default "shared_buffers" ... 400kB` means `initdb` tried a `postgres --boot` + child at every candidate size and **all of them crashed**, so it kept the floor values. + PostgreSQL did not choose a small configuration. +- **With the Zalando operator the failure is invisible at the Kubernetes level.** The pod + reports `1/1 Running` while Patroni loops: + `removing initialize key after failed attempt to bootstrap the cluster` → + `PatroniFatalException: Failed to bootstrap cluster`, and runit restarts it roughly every + 30 seconds. Never conclude the instance is healthy from pod readiness alone — connect with + `psql`. With CloudNativePG the `-initdb` Job goes to `Error` and retries visibly. + +The problem is not specific to PostgreSQL. Any containerised workload that opportunistically +uses hugepages — MySQL, MongoDB, JVM applications with `-XX:+UseLargePages` — is exposed on +the same node. + +## Environment + +Reproduced and fixed on: + +- Alauda Container Platform 4.2, Kubernetes v1.33.7, containerd 1.7.29, **cgroup v1** +- Worker node: amd64, kernel 3.10, **single NUMA node**, `cpuManagerPolicy` not enabled +- CloudNativePG operator v1.29.1-acp.1 with PostgreSQL 18.4, and the Zalando operator with + `spilo v4.3.0-beta.36` + +Everything asserted below was observed in that environment. The underlying mechanism is a +property of kubelet plus the hugetlb cgroup controller rather than of these specific versions, +so it is expected to generalise — but the concrete paths, filenames and numbers here are the +tested ones, and the cgroup layout in particular depends on the **systemd** cgroup driver +(`cgroupDriver: systemd`) and the default kubelet cgroup root. On cgroup v2 the equivalent file +is `hugetlb..max`; the filename is correct, but the unified hierarchy means the mount +point differs, and that combination was not tested here. + +## Root cause + +Two independent facts combine: + +1. **kubelet caps hugepages at zero for pods that do not ask for them.** For every pod without + a `hugepages-` resource request, kubelet writes + `hugetlb..limit_in_bytes = 0` on the **pod-level** cgroup + (`kubepods--pod.slice`). +2. **hugetlb reservation and allocation happen at different times.** `mmap()` with + `MAP_HUGETLB` checks the reservation against the **global** pool and succeeds. The cgroup + limit is only charged when a page is **faulted in**. A refusal inside a page fault has no + errno to return, so the kernel delivers **SIGBUS**. + +PostgreSQL's default `huge_pages=try` only falls back to normal pages when `mmap()` *returns +an error*. Here `mmap()` succeeds, so PostgreSQL commits to hugepages and dies on first touch. + +This is why the failure appears only on nodes where hugepages exist. On a node with no pool at +all, `mmap(MAP_HUGETLB)` fails immediately with `ENOMEM` and PostgreSQL falls back silently. +**A node that advertises hugepages the pod cannot actually obtain is more dangerous than a +node with none.** + +## Diagnosis + +### Confirm it is hugepages + +Reproduce in seconds inside the container, without waiting for the operator to retry: + +```bash +kubectl -n exec -- initdb -D /tmp/hugetlb-check -U postgres +``` + +A `Bus error (core dumped)` confirms it. (Clean up `/tmp/hugetlb-check` afterwards.) + +`` can be **any** pod on the suspect node that contains PostgreSQL binaries — it does not +have to be the failing one, and often cannot be. With CloudNativePG the crashing entity is the +`-initdb` Job pod, which runs with `restartPolicy: Never` and may already be `Terminated` by +the time you look, so `exec` returns `container not running`. Start a throwaway pod pinned to +that node with the same image instead. + +### Read the cgroup counters at the correct level + +> **This is the step that is easy to get wrong.** The `/sys/fs/cgroup` view inside a pod is the +> **container** scope, which is *not* where kubelet writes the limit. It reads `unlimited` with +> `failcnt` permanently `0`, because a child cgroup cannot see refusals recorded by its parent. +> An in-pod reading of zero does **not** exonerate the cgroup. + +Read the **pod slice**, from the host: + +```bash +set -o pipefail +POD_UID=$(kubectl -n get pod -o jsonpath='{.metadata.uid}' | tr - _) \ + || { echo "kubectl failed"; exit 1; } +[ -n "$POD_UID" ] || { echo "pod not found"; exit 1; } +D=/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod${POD_UID}.slice +cat "$D/hugetlb.2MB.limit_in_bytes" "$D/hugetlb.2MB.failcnt" +``` + +`set -o pipefail` matters: without it the assignment takes the exit status of `tr`, so a failed +`kubectl` yields status 0 and an empty UID, and you go on to read a path built from nothing. + +> This path is the **systemd** cgroup-driver layout, which is what the tested cluster uses +> (`cgroupDriver: systemd` in `/var/lib/kubelet/config.yaml`) and the common default. Under the +> `cgroupfs` driver the shape differs — `/sys/fs/cgroup/hugetlb/kubepods/burstable/pod/`, +> with the UID keeping its hyphens and no `.slice` suffix. Check `cgroupDriver` on the node +> before assuming either. A non-default `--cgroup-root` shifts the prefix as well. + +Adjust `burstable` to `besteffort`, or drop that path element entirely for Guaranteed pods — +only Burstable and BestEffort get a QoS-level cgroup. Substitute the node's default hugepage +size in the filenames: `hugetlb.2MB.*` on x86_64, `hugetlb.512MB.*` on arm64 builds that +default to 512 MiB pages (`grep Hugepagesize /proc/meminfo`). + +A `limit` of `0` with a `failcnt` that increments on every crash confirms the diagnosis. +Measured values around a single failing `initdb`: + +| cgroup level | `limit_in_bytes` | `failcnt` before → after | +| --- | --- | --- | +| pod slice | `0` | `0` → `25` | +| container scope (visible inside the pod) | unlimited | `0` → `0` | + +`HugePages_Free` in `/proc/meminfo` does not move during the failure: the pages are reserved, +refused at fault time, and never allocated. + +### Check whether the node actually advertises hugepages + +```bash +kubectl get node -o jsonpath='{.status.allocatable.hugepages-2Mi}{"\n"}' +``` + +(Use the resource name matching the node's default page size — `hugepages-512Mi` on arm64 +builds that default to 512 MiB pages.) **Blank output means the node is not advertising that +hugepage size** — the command still exits 0, so an empty line is a result, not a failure. + +kubelet caches machine information, so a runtime `sysctl -w vm.nr_hugepages=N` is **not +visible to kubelet until it is restarted**. A node with a live pool but +`hugepages-2Mi: 0` is the worst state: pods are capped at zero **and** cannot request +hugepages, because the scheduler does not believe the resource exists. If the pool was enabled +online, restart kubelet on that node before using the "request hugepages" option below. + +## Resolution + +Pick a row. Do **not** disable the node's hugepage pool as a first move — on a shared node +those pages usually belong to another workload, and removing them has taken applications down. + +| Situation | Action | +| --- | --- | +| Instance already bootstrapped, only failing to restart | Set the `huge_pages: "off"` parameter. Applies to every major version in scope here (13-18). | +| Fresh bootstrap needed, PostgreSQL **16 or later** | Set the parameter **and** pass `huge_pages=off` to `initdb`. | +| Fresh bootstrap needed, PostgreSQL **15 or earlier** | No `initdb` option exists. Request hugepages instead, or bootstrap on a node without a pool and move the instance afterwards. | +| PostgreSQL should genuinely use the hugepages | Request `hugepages-` in `resources` and set `huge_pages: "on"`. | + +### Why the parameter alone is not enough for a fresh bootstrap + +Both operators write `postgresql.conf` **after** `initdb` completes. The crash happens in +`initdb`'s bootstrap child, before that file exists. A cluster created with only +`huge_pages: "off"` still fails to bootstrap — verified. PostgreSQL 16 added +`initdb -c NAME=VALUE` (`--set`), and it is honoured by the bootstrap child, which gives a +configuration-only route on 16+. + +### Zalando operator + +```yaml +spec: + postgresql: + parameters: + huge_pages: "off" + patroni: + initdb: + set: "huge_pages=off" +``` + +Patroni renders each `{key: value}` entry as `--key=value`, and PostgreSQL 16's option is +`-c, --set`. The key must therefore be `set`; writing `c` produces an invalid `--c=...` flag. + +### CloudNativePG operator + +```yaml +spec: + postgresql: + parameters: + huge_pages: "off" + bootstrap: + initdb: + options: ["-c", "huge_pages=off"] +``` + +> **Caution.** `bootstrap.initdb.options` is a deprecated field. Setting it makes +> `buildInitDBFlags` return early, so the operator ignores **every** explicit `initdb` setting: +> `dataChecksums`, `encoding`, `locale`, `localeCollate`, `localeCType`, `localeProvider`, +> `icuLocale`, `icuRules`, `builtinLocale`, and `walSegmentSize`. The cluster-level +> `logLevel` → `-d` side effect is skipped too. Diff your current `bootstrap.initdb` block +> before adding `options`, and restate everything you rely on as raw flags in the same list: +> +> ```yaml +> options: ["-c", "huge_pages=off", "--encoding=UTF8", "--lc-collate=C", "--lc-ctype=C", "-k"] +> ``` +> +> Verified in that argv order: `initdb` exits 0, `postgresql.conf` carries `huge_pages = off`, +> and the resulting cluster reports UTF8 / C / C with `data_checksums=on`. Clusters using ICU +> locale settings are the ones most at risk of silent loss here. + +### If a CloudNativePG bootstrap has already been failing for a while + +Patching the Cluster spec only works if the `-initdb` Job is still retrying. Once it has +exhausted its backoff, **the fix above has no effect on its own** — and the cluster gives you +almost no signal that this has happened. Sequence observed end-to-end on operator +`v1.29.1-acp.1`: + +1. The `-initdb` Job runs `restartPolicy: Never` with the default `backoffLimit: 6`, so it + stops after 7 pod attempts (about 10 minutes of exponential backoff) and is marked + `Failed / BackoffLimitExceeded`. +2. **`status.phase` stays `Setting up primary` regardless.** The reconciler is stuck earlier, + logging `Selected PVC is not ready yet, waiting for 1 second` in a tight loop, because the + PVC from the failed attempt never received its ready marker. Diagnose from the Job, not the + Cluster phase: + + ```bash + kubectl -n get jobs -l cnpg.io/cluster= + kubectl -n get cluster -o jsonpath='{.status.danglingPVC}{"\n"}' + ``` + + A `Failed` Job plus a non-empty `danglingPVC` is this state. (Operator v1.29.2 and later + include upstream #11035, "report failed instance creation jobs instead of waiting forever", + which surfaces it in the phase instead. `v1.29.1-acp.1` does not.) +3. Patching the spec does nothing — the failed Job is not replaced. +4. Deleting the Job does not help either; the PVC loop continues. +5. **Do not delete the PVC by itself.** That moves the cluster into a second and worse state: + `One or more instances were previously created, but no PersistentVolumeClaims (PVCs) exist. + The cluster is in an unrecoverable state. To resolve this, restore the cluster from a recent + backup.` + +**Recovery: delete the Cluster and recreate it with the fix already in the spec.** + +```bash +kubectl -n delete cluster +kubectl -n apply -f cluster-with-huge-pages-off.yaml +``` + +This is safe *specifically because bootstrap never completed* — there is no data in that PVC to +lose. Verified: the recreated cluster reached `Running` with `huge_pages=off` on the same +hugepage-enabled node. **Never do this to a cluster that bootstrapped successfully and later +failed to restart** — that one has data, and the GUC-only fix in the table above applies. + +The practical conclusion is to put the fix in the manifest *before* first apply on any node +that has, or might get, a hugepage pool. + +### Option: let PostgreSQL use the hugepages + +If the pool is meant for the database, give the pod a real limit instead of removing the +request. This fixes both bootstrap and steady state: + +```yaml +spec: + postgresql: + parameters: + huge_pages: "on" + shared_buffers: 256MB + resources: + requests: { cpu: 200m, memory: 1Gi, hugepages-2Mi: 512Mi } + limits: { cpu: "1", memory: 1Gi, hugepages-2Mi: 512Mi } +``` + +**PostgreSQL 15 and later** can estimate the requirement for you. The parameter needs a data +directory — it cannot be queried on an empty container: + +```bash +# against an existing instance +kubectl -n exec -- bash -c \ + 'postgres -D "$PGDATA" -c shared_buffers=256MB -C shared_memory_size_in_huge_pages' +``` + +**It returns a page count, not a size.** Multiply by the node's default hugepage size to get the +resource quantity. The same default `shared_buffers` returns `75` on a node with 2 MiB pages +and `1` on a node with 512 MiB pages — writing the raw number into +`hugepages-2Mi: Mi` under-provisions by the page-size factor. + +Measured on PostgreSQL 18.4 with 2 MiB pages: 75 pages at the 128 MB default, 543 at +`shared_buffers=1GB`, 4222 at `8GB` — roughly `shared_buffers` plus 10-15% of control +structures, so budget headroom rather than an exact fit. An exact-fit limit means any later +growth of the shared memory segment faults into SIGBUS instead of failing cleanly at startup. + +**PostgreSQL 14 and earlier do not have this parameter** — `postgres -C +shared_memory_size_in_huge_pages` returns `FATAL: unrecognized configuration parameter` +(verified on 13 and 14; present from 15). Size those manually: `shared_buffers` plus about 15%, +divided by the page size, rounded up. + +Before a first bootstrap there is no data directory on any version, so the query is unavailable +— there is no `--config-file` shortcut, `postgres` still demands `-D`. Either run `initdb` into +a throwaway directory purely to take the measurement, or use the manual estimate. + +> **Match the page size to the node.** The resource name and the page count both follow the +> node's *default* hugepage size, which is what `huge_pages=on` consumes +> (`huge_page_size = 0` means "use the system default"). Check it first: +> +> ```bash +> grep Hugepagesize /proc/meminfo +> ``` +> +> x86_64 nodes are normally `2048 kB` → request `hugepages-2Mi`. Some arm64 builds, including +> Kylin Linux Advanced Server V10 on 64 KiB pages, default to `524288 kB` → the resource is +> `hugepages-512Mi` and every count above is in 512 MiB units. Requesting `hugepages-2Mi` on +> such a node asks for a resource it does not advertise, and the pod stays `Pending`. + +The node must also advertise the resource at all — see the kubelet restart note above. + +## Verification + +Pod readiness is not a valid check, because the Zalando operator reports `1/1 Running` +throughout the failure. Use these instead. + +The instance is actually serving, and using the intended memory type: + +```bash +kubectl -n exec -- psql -U postgres -tAc \ + "select 'ALIVE', current_setting('huge_pages'), current_setting('shared_buffers')" +``` + +Expect `ALIVE|off|...` for the disable path, or `ALIVE|on|...` for the request path. + +No further cgroup refusals — `failcnt` stops incrementing on the pod slice (same command as in +Diagnosis). + +For the request path only, confirm the pages are genuinely held, on the node: + +```bash +grep -iE 'hugepages_total|hugepages_free|hugepages_rsvd' /proc/meminfo +``` + +Read **`HugePages_Rsvd`**, not `HugePages_Free`. Pages that are reserved but not yet touched +still count as free, so `HugePages_Free` understates usage and is not a reliable indicator. + +## Limitations and trade-offs + +- Setting `huge_pages: "off"` gives up the TLB benefit of hugepages for that instance. For most + ACP PostgreSQL workloads this is not measurable; if the database is the reason hugepages were + reserved, use the request path instead. +- The fix is per-cluster. Every PostgreSQL instance that may land on a hugepage-enabled node + needs it, and so does every other hugepage-capable workload on those nodes. +- Requesting hugepages does **not** protect against a node whose free pages are unreachable for + another reason, for example a restrictive `cpuset.mems` on a multi-NUMA machine with an + unevenly distributed pool. Disabling hugepages in the database does, because no + `MAP_HUGETLB` mapping is created at all. +- `vm.nr_hugepages=0` on the node is a last resort. Identify who consumes the pool first; the + database is the workload that should adapt, since it is the one opportunistically taking + pages it cannot fault in. + +## Related + +- `How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md` — moving an instance off an + affected node when no in-place option exists (PostgreSQL 15 and earlier). +- `CX_series_KubeVirt_VM_fails_to_start_with_FailedScheduling_on_dedicated_CPU_and_hugepages.md` + — the opposite failure mode for the same resource. A workload that *requests* + `hugepages-2Mi` and cannot get it stays `Pending` with `Insufficient hugepages-2Mi`, which is + an honest, visible failure. The SIGBUS described here happens precisely because the workload + never requested the resource, so nothing stopped it from being scheduled.