Skip to content

docs(postgresql): solution for Bus error (exit 135) on hugepage-enabled nodes - #825

Open
SuJinpei wants to merge 3 commits into
mainfrom
docs/pg-hugetlb-sigbus-solution
Open

docs(postgresql): solution for Bus error (exit 135) on hugepage-enabled nodes#825
SuJinpei wants to merge 3 commits into
mainfrom
docs/pg-hugetlb-sigbus-solution

Conversation

@SuJinpei

@SuJinpei SuJinpei commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Adds a Troubleshooting solution for PostgreSQL dying with Bus error / exit 135 on nodes where hugepages are enabled.

Why

Reported from a customer environment (Harbor's PostgreSQL crash-looping after vm.nr_hugepages was set for a business workload). The failure is not PostgreSQL-specific and not NUMA-specific: on any node with a hugepage pool, kubelet writes hugetlb.<size>.limit_in_bytes = 0 on the pod slice of every pod that does not request hugepages. Reservation is global so mmap(MAP_HUGETLB) succeeds; the limit is charged at page-fault time, where a refusal has no errno and the kernel sends SIGBUS. PostgreSQL's default huge_pages=try only falls back when mmap itself fails, so it dies.

The single most useful thing in here: the cgroup counters visible inside the pod cannot show this. The container scope reads unlimited with failcnt stuck at 0 while the parent pod slice refuses every charge. Reading the wrong level is what sent the original investigation down a false path for days.

Validation

Reproduced and every remedy run end-to-end on ACP 4.2 / k8s 1.33.7 / containerd 1.7.29 / cgroup v1, on a single-NUMA node with no CPU pinning — vm.nr_hugepages=512 alone is sufficient to break every PostgreSQL bootstrap on the node. Covered both the CloudNativePG operator (v1.29.1-acp.1, PG 18.4) and the Zalando operator (spilo v4.3.0-beta.36, PG 13-17).

Measured, not inferred:

  • pod slice limit=0, failcnt 0 → 25 across one failing initdb; container scope unlimited, 0 → 0
  • the huge_pages GUC alone does not fix a fresh bootstrap (the operator writes postgresql.conf after initdb) — a cluster with only that setting still crash-loops
  • initdb -c huge_pages=off works on PG 16/17/18 and does not exist on 13/15
  • requesting hugepages-2Mi works and PostgreSQL genuinely consumes the pages
  • the same default shared_buffers reports 75 pages on a 2 MiB-page node and 1 on a 512 MiB-page node — the sizing value is a page count, not a size

Test environment fully restored afterwards (pool back to 0, kubelet restarted, allocatable back to its original value).

Review

Ran the three-pass protocol before opening this:

  • Static (shell/YAML semantics) — caught a $(… | tr) that swallowed kubectl failure into an empty UID with exit status 0.
  • Independent domain cold-read — caught that shared_memory_size_in_huge_pages does not exist before PG 15 while the version table routed PG 13/14 readers straight at it.
  • Self-test of every recommended command — caught that the sizing command cannot run as written (needs -D/PGDATA, and has no pre-bootstrap form) and the page-count/page-size unit trap above.

The two review modalities caught disjoint defect classes; both independently found the bootstrap.initdb.options caution listed 5 discarded fields when it discards 10.

One reviewer recommendation was not taken: for a CloudNativePG bootstrap that has already exhausted its Job backoff, the suggested "delete the failed Job" does not recover the cluster. Testing showed patching does nothing, deleting the Job does nothing, and deleting the dangling PVC is actively harmful (it trips a second unrecoverable branch advising restore-from-backup). The documented recovery is delete-and-recreate the Cluster with the fix in the spec, which is safe only because bootstrap never completed.

Notes for reviewers

  • EN only — docs/zh/ is generated by the translation pipeline.
  • No id: field, so CI's add_id.sh assigns one from the H1 title. Please keep the title stable.
  • Version-specific claims are scoped to what was tested; the cgroup layout is explicitly scoped to the systemd cgroup driver, with the cgroupfs shape given as an alternative.

🤖 Generated with Claude Code

SuJinpei and others added 3 commits August 4, 2026 10:20
…nabled nodes

PostgreSQL dies with SIGBUS during bootstrap on any node where vm.nr_hugepages
is non-zero, because kubelet caps hugetlb at 0 on the pod slice of pods that do
not request hugepages, and that limit is charged at page-fault time rather than
at mmap. Documents the two-level cgroup diagnosis (the in-pod view is unable to
show the refusal), the version-dependent fix for both the Zalando and
CloudNativePG operators, and the hugepage-request alternative.

Reproduced and all four remedies verified on ACP 4.2 / k8s 1.33.7 / cgroup v1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Static review (codex, passes 1+3), an independent domain cold-read, and my own
pass-3 self-test of every recommended command. Confirmed findings only:

- initdb.options discards 10 explicit fields, not the 5 originally listed;
  locale, localeProvider, icuLocale, icuRules and builtinLocale were missing,
  so ICU-configured clusters were the most exposed to silent loss.
- shared_memory_size_in_huge_pages does not exist before PostgreSQL 15
  (verified: 13/14 return "unrecognized configuration parameter"), yet the
  version table routed PG13/14 readers straight at it. Added a manual fallback.
- The sizing command could not run as written: it requires -D/PGDATA, and no
  form works pre-bootstrap. It also returns a page COUNT, not a size - the same
  default shared_buffers reports 75 with 2MiB pages and 1 with 512MiB pages.
- Page size is node-dependent. arm64 Kylin v10 defaults to 512MiB, so
  hugepages-2Mi is the wrong resource name there and the pod stays Pending.
- The pod-UID snippet swallowed kubectl failure through a pipe (empty UID,
  status 0); added pipefail and an emptiness guard.
- Scoped the cgroup paths to the systemd cgroup driver and gave the cgroupfs
  shape; noted that blank jsonpath output is a result, not an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…strap

The cold-read flagged that patching the Cluster stops working once the initdb
Job exhausts its backoff. Confirmed, but the suggested remedy (delete the failed
Job) does not work, so this documents what does, from an end-to-end run:

- On v1.29.1-acp.1 the cluster gives no signal at all: status.phase stays
  "Setting up primary" while the reconciler hot-loops on "Selected PVC is not
  ready yet". Diagnose from the Job and status.danglingPVC instead. v1.29.2+
  carries upstream #11035 and surfaces it in the phase; v1.29.1-acp.1 does not
  (verified by merge-base against the tags).
- Patching the spec: no effect. Deleting the Job: no effect.
- Deleting the PVC alone is actively harmful - it trips a second Unrecoverable
  branch ("no PersistentVolumeClaims exist ... restore from a recent backup").
- What works: delete and recreate the Cluster with the fix already in the spec,
  which is safe only because bootstrap never completed. Verified the recreated
  cluster reaches Running with huge_pages=off on a hugepage-enabled node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SuJinpei
SuJinpei deployed to translate August 4, 2026 11:57 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant