From 304c4aa564c74c49cbd6983b555c485b872d50cc Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Fri, 14 Aug 2026 08:24:28 +0300 Subject: [PATCH 1/6] Document filesystem-smaller-than-partition failure mode for persistent disks When a persistent disk is resized, the partition resize can succeed while the filesystem grow fails (e.g. resize2fs: Permission denied due to pre-existing ext4 errors). Subsequent deploys then silently succeed, leaving the filesystem undersized with no visible error. --- content/persistent-disks.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index 863e7cbd..1baf1228 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -124,6 +124,33 @@ During the disk migration from one disk type and size to another, the Director c !!! note An IaaS might disallow attaching particular disk types and sizes to certain VM types. Consult your IaaS documentation for more information. +### Filesystem smaller than partition after a failed grow {: #filesystem-smaller-than-partition } + +When BOSH resizes a persistent disk it first extends the partition to fill the new block device, then grows the filesystem to fill the partition. If the partition resize succeeds but the filesystem grow fails (for example because `resize2fs` exits with `Permission denied` due to pre-existing ext4 filesystem errors), the two operations are left in an inconsistent state: + +- the partition already spans the full disk, so subsequent deploys take the "no resize needed" branch and never revisit the filesystem +- the deployment succeeds with no visible error, leaving the filesystem silently smaller than the partition + +The initial failure surfaces as: + +```text +Error: Action Failed get_task: Task result: Adjusting persistent disk partitioning: Failed to grow filesystem: Failed to grow Ext4 filesystem: Running command: 'resize2fs -f /dev/sdb1', stdout: 'Filesystem at /dev/sdb1 is mounted on /var/vcap/store; on-line resizing required +old_desc_blocks = 13, new_desc_blocks = 128 +', stderr: 'resize2fs 1.46.5 (30-Dec-2021) +resize2fs: Permission denied to resize filesystem +': exit status 1 +``` + +The underlying cause is visible in the kernel log: + +```text +EXT4-fs (sdb1): warning: mounting fs with errors, running e2fsck is recommended +EXT4-fs (sdb1): error count since last fsck: 494734 +EXT4-fs warning (device sdb1): ext4_resize_begin:82: There are errors in the filesystem, so online resizing is not allowed +``` + +The kernel refuses to grow a filesystem that has errors. Once the partition already spans the disk, `resize2fs` is never called again on subsequent deploys, so the error does not reappear and the undersized filesystem goes undetected. + --- ## Orphaned Disks {: #orphaned-disks } From 7d1fa280900fd459a34de0a100caf29c9c494dba Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Fri, 14 Aug 2026 08:31:18 +0300 Subject: [PATCH 2/6] Scope filesystem-smaller-than-partition to IaaS-native resize path --- content/persistent-disks.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index 1baf1228..b4da7824 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -126,11 +126,13 @@ During the disk migration from one disk type and size to another, the Director c ### Filesystem smaller than partition after a failed grow {: #filesystem-smaller-than-partition } -When BOSH resizes a persistent disk it first extends the partition to fill the new block device, then grows the filesystem to fill the partition. If the partition resize succeeds but the filesystem grow fails (for example because `resize2fs` exits with `Permission denied` due to pre-existing ext4 filesystem errors), the two operations are left in an inconsistent state: +When using [IaaS-native disk resize](cpi-api-v2-method/resize-disk.md), BOSH resizes the existing disk in place: the Director asks the IaaS to grow the block device, then the agent extends the partition to fill it and grows the filesystem. If the partition resize succeeds but the filesystem grow fails (for example because `resize2fs` exits with `Permission denied` due to pre-existing ext4 filesystem errors), the two operations are left in an inconsistent state: - the partition already spans the full disk, so subsequent deploys take the "no resize needed" branch and never revisit the filesystem - the deployment succeeds with no visible error, leaving the filesystem silently smaller than the partition +This failure mode does not apply to the fallback path (when native resize is disabled or the CPI does not support it): in that case the Director creates a new disk, copies data from the old one, and orphans it, so each deploy starts with a freshly partitioned and formatted disk. + The initial failure surfaces as: ```text @@ -149,7 +151,7 @@ EXT4-fs (sdb1): error count since last fsck: 494734 EXT4-fs warning (device sdb1): ext4_resize_begin:82: There are errors in the filesystem, so online resizing is not allowed ``` -The kernel refuses to grow a filesystem that has errors. Once the partition already spans the disk, `resize2fs` is never called again on subsequent deploys, so the error does not reappear and the undersized filesystem goes undetected. +The kernel refuses to grow a filesystem that has errors. Once the partition already spans the disk, the agent's `AdjustPersistentDiskPartitioning` takes the "no resize needed" branch on every subsequent deploy and never calls `resize2fs` again, so the error does not reappear and the undersized filesystem goes undetected. --- From c25d6e2b523d4ed82d6d1c4cad1c7fc980436ac7 Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Mon, 17 Aug 2026 11:42:34 +0300 Subject: [PATCH 3/6] Document detection and remediation for filesystem-smaller-than-partition --- content/persistent-disks.md | 44 +++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index b4da7824..e6e72c39 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -153,6 +153,50 @@ EXT4-fs warning (device sdb1): ext4_resize_begin:82: There are errors in the fil The kernel refuses to grow a filesystem that has errors. Once the partition already spans the disk, the agent's `AdjustPersistentDiskPartitioning` takes the "no resize needed" branch on every subsequent deploy and never calls `resize2fs` again, so the error does not reappear and the undersized filesystem goes undetected. +To check whether an instance is affected, SSH into the instance and compare the filesystem size against the partition size: + +```shell +bosh -d ssh / +``` + +```shell +df -BG /var/vcap/store +findmnt -n -o SOURCE /var/vcap/store +sudo blockdev --getsize64 +``` + +Replace `` with the device reported by `findmnt`. Note the partition path - it will be needed during remediation. If the filesystem size from `df` is significantly smaller than the partition size from `blockdev`, the instance is affected. + +#### Remediation + +Stop jobs from within the instance using `monit`, which keeps the disk attached to the VM throughout: + +```shell +bosh -d ssh / +``` + +```shell +# Note the partition — it is needed after jobs are stopped +findmnt -n -o SOURCE /var/vcap/store + +# Stop all jobs; this also tears down the process namespaces holding bind mounts on the disk +sudo monit stop all + +# Wait until all processes show 'not monitored' +sudo monit summary + +# Unmount, repair filesystem errors, grow to fill the partition +sudo umount /var/vcap/store +sudo e2fsck -fy +sudo resize2fs + +# Remount and restart jobs +sudo mount /var/vcap/store +sudo monit start all +``` + +Wait for all processes to reach `running` state before exiting the SSH session. If `resize2fs` reports `The filesystem is already N blocks long. Nothing to do!`, the filesystem already fills the partition and no grow is needed. + --- ## Orphaned Disks {: #orphaned-disks } From c507f28d9c1c3c5d2d99540fc100a78b25641990 Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Mon, 17 Aug 2026 11:51:06 +0300 Subject: [PATCH 4/6] Address CodeRabbit review comments on filesystem-smaller-than-partition --- content/persistent-disks.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index e6e72c39..fcc5fcb9 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -160,12 +160,12 @@ bosh -d ssh / ``` ```shell -df -BG /var/vcap/store findmnt -n -o SOURCE /var/vcap/store +df -h /var/vcap/store sudo blockdev --getsize64 ``` -Replace `` with the device reported by `findmnt`. Note the partition path - it will be needed during remediation. If the filesystem size from `df` is significantly smaller than the partition size from `blockdev`, the instance is affected. +Replace `` with the device reported by `findmnt`. Note the partition path — it will be needed during remediation. The `df` output shows the filesystem size in human-readable units; `blockdev` returns the partition size in bytes (1 GiB = 1,073,741,824 bytes). If the filesystem size is significantly smaller than the partition, the instance is affected. Confirm the filesystem type is ext4 with `blkid ` before proceeding — the remediation steps below apply to ext4 only. #### Remediation @@ -185,13 +185,17 @@ sudo monit stop all # Wait until all processes show 'not monitored' sudo monit summary -# Unmount, repair filesystem errors, grow to fill the partition +# Unmount the disk; the following command should produce no output if successful sudo umount /var/vcap/store +findmnt /var/vcap/store + +# Repair filesystem errors and grow to fill the partition sudo e2fsck -fy sudo resize2fs -# Remount and restart jobs +# Remount, verify the filesystem now fills the partition, then restart jobs sudo mount /var/vcap/store +df -h /var/vcap/store sudo monit start all ``` From 8d378df2f9976ef90b52ca23fbb94d755a071f33 Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Mon, 17 Aug 2026 11:57:19 +0300 Subject: [PATCH 5/6] Make the explanations clearer --- content/persistent-disks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index fcc5fcb9..3fec550a 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -161,11 +161,11 @@ bosh -d ssh / ```shell findmnt -n -o SOURCE /var/vcap/store -df -h /var/vcap/store +df -B1 /var/vcap/store sudo blockdev --getsize64 ``` -Replace `` with the device reported by `findmnt`. Note the partition path — it will be needed during remediation. The `df` output shows the filesystem size in human-readable units; `blockdev` returns the partition size in bytes (1 GiB = 1,073,741,824 bytes). If the filesystem size is significantly smaller than the partition, the instance is affected. Confirm the filesystem type is ext4 with `blkid ` before proceeding — the remediation steps below apply to ext4 only. +Replace `` with the device reported by `findmnt`. Note the partition path — it will be needed during remediation. Both values are in bytes; if the filesystem size from `df` is significantly smaller than the partition size from `blockdev`, the instance is affected. Confirm the filesystem type is ext4 with `blkid ` - the remediation steps below apply to ext4 only. #### Remediation From 307a74544852e62521dd095e62cbc91f76098529 Mon Sep 17 00:00:00 2001 From: Ned Petrov Date: Mon, 17 Aug 2026 11:59:19 +0300 Subject: [PATCH 6/6] Replace AI dashes --- content/persistent-disks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/persistent-disks.md b/content/persistent-disks.md index 3fec550a..5785baad 100644 --- a/content/persistent-disks.md +++ b/content/persistent-disks.md @@ -165,7 +165,7 @@ df -B1 /var/vcap/store sudo blockdev --getsize64 ``` -Replace `` with the device reported by `findmnt`. Note the partition path — it will be needed during remediation. Both values are in bytes; if the filesystem size from `df` is significantly smaller than the partition size from `blockdev`, the instance is affected. Confirm the filesystem type is ext4 with `blkid ` - the remediation steps below apply to ext4 only. +Replace `` with the device reported by `findmnt`. Note the partition path - it will be needed during remediation. Both values are in bytes; if the filesystem size from `df` is significantly smaller than the partition size from `blockdev`, the instance is affected. Confirm the filesystem type is ext4 with `blkid ` - the remediation steps below apply to ext4 only. #### Remediation @@ -176,7 +176,7 @@ bosh -d ssh / ``` ```shell -# Note the partition — it is needed after jobs are stopped +# Note the partition - it is needed after jobs are stopped findmnt -n -o SOURCE /var/vcap/store # Stop all jobs; this also tears down the process namespaces holding bind mounts on the disk