cloudbase · doccaz · Sep 19, 2026
diff --git a/README.md b/README.md
@@ -17,17 +17,30 @@ from VDDK 8 NBD traffic; see `docs/`.
 
 ## Status
 
-Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`
-(`nbd` is still available):
+Implemented against vCenter 8 / ESXi 8, including a standalone ESXi
+host with no vCenter. Default transport is `nbdssl` (`nbd` is still
+available):
 
-- `VixDiskLib_ConnectEx` (UID credentials)
+- `VixDiskLib_ConnectEx` (UID credentials; vCenter or direct ESXi)
 - `VixDiskLib_Open` (datastore path, read-only or read-write)
 - `VixDiskLib_Read` (optional ``skip_decompression`` packs FastLZ extras)
 - `VixDiskLib_Write`
-
-Not implemented: compression open flags other than FastLZ, CBT /
-allocated-block queries, disk geometry (`DDB_GET`), encrypted disks,
-and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`.
+- `VixDiskLib_GetInfo` (capacity and physical geometry from the `Open`
+  reply; `biosGeo`/`adapterType`/`uuid` from `DDB_GET`, matching real
+  VDDK's cost and behavior)
+- `VixDiskLib_QueryAllocatedBlocks` (allocated-block bitmap; see
+  `docs/nfc_read.md`)
+- Changed Block Tracking: `openvixdisklib.nfc_auth.enable_change_tracking`
+  / `disk_change_id` / `query_changed_disk_areas` (public VIM API, not
+  part of VixDiskLib itself; see `docs/cbt.md`)
+
+Reading/writing a snapshot delta file directly (and running
+`query_allocated_blocks` against it) already works — `NFC_DELTA_DISK`
+turned out to be an optional VMFS-only VDDK client optimization, not a
+correctness requirement (see `docs/reverse_engineering_procedure.md`).
+
+Not implemented: compression open flags other than FastLZ, and
+encrypted disks.
 
 Requires Python 3.10 or later.
 

diff --git a/docs/cbt.md b/docs/cbt.md
@@ -0,0 +1,110 @@
+# Changed Block Tracking (CBT)
+
+This is not a reverse-engineered NFC feature. VixDiskLib does not expose
+CBT itself: `VixDiskLib_QueryAllocatedBlocks` (implemented separately;
+see `docs/nfc_read.md`) reports which blocks are *allocated*
+(non-sparse) within a single NFC-opened disk, not which byte ranges
+*changed* between two points in time. Real backup tools
+get changed-range information from vSphere's public
+`VirtualMachine.QueryChangedDiskAreas` VIM call instead, used alongside
+VDDK/NFC reads for the actual bytes. `openvixdisklib.nfc_auth` wraps
+that public pyVmomi call directly — no capture, no wire format to
+document, per the project rule to reuse pyVmomi for anything it already
+exposes.
+
+## Workflow
+
+1. `nfc_auth.enable_change_tracking(vm)` — sets
+   `VirtualMachineConfigSpec.changeTrackingEnabled = True` via
+   `ReconfigVM_Task`. Takes effect for writes from that point forward;
+   it does not retroactively track earlier changes.
+2. Take a snapshot (or power-cycle the VM). A disk's `changeId` is
+   empty until this happens.
+3. `nfc_auth.disk_change_id(vm, device_key)` — reads the current
+   `changeId` off `VirtualDisk.backing.changeId` (for example
+   `"52 f3 b6 37 30 8d ea 3e-58 70 c0 fd 61 44 26 62/2"`).
+4. Do backup work (VDDK/NFC reads of the disk at that point).
+5. Later, take another snapshot.
+6. `nfc_auth.query_changed_disk_areas(vm, new_snapshot, device_key,
+   change_id_from_step_3)` — returns the byte ranges written between
+   the two snapshots.
+7. Read only those ranges via VDDK/NFC on the new snapshot's disk
+   chain for an incremental backup.
+
+For an initial full backup, pass `change_id="*"` in step 6 without a
+prior snapshot. **Correction from an earlier draft of this doc:**
+this does *not* report the entire disk as one changed extent — see
+"Wildcard `changeId='*'` reports allocated regions, not the whole
+disk" below.
+
+## Validated in this lab
+
+Confirmed end-to-end against a temporary VM on the standalone ESXi
+8.0.3 lab host (no vCenter): enabled CBT, snapshotted, wrote one
+sector via `openvixdisklib.openvixdisklib` at a known offset,
+snapshotted again, and called `query_changed_disk_areas` with the
+first snapshot's `changeId`. The single reported extent
+(`start=3932160, length=65536`, i.e. sectors 7680–7807) correctly
+covered the written sector (7777). Extents were 64 KiB-aligned in this
+lab's observations; that granularity is server-defined, not part of
+the function's contract.
+
+### Wildcard `changeId="*"` reports allocated regions, not the whole disk
+
+Tested `query_changed_disk_areas(vm, snapshot, device_key, "*")` (the
+initial-full-backup path, no prior snapshot needed) against a fresh
+10 GiB thin-provisioned temp-VM disk. `result.length` correctly
+reports the full declared virtual capacity (10737418240 bytes), but
+`result.changed_areas` only covered **1 MiB** total — not the whole
+disk. For a thin-provisioned disk, `"*"` reports the regions that are
+actually *allocated* (backed by real data on the datastore), not the
+full sparse virtual capacity; unwritten/unallocated regions have
+nothing to back up regardless. A backup tool doing an initial full
+backup with `"*"` should read exactly the reported extents, not assume
+it needs to read `result.length` bytes.
+
+### One large contiguous write is one extent; scattered writes are not
+
+Wrote a single 4 MiB contiguous region plus three separate one-sector
+writes at scattered offsets (same disk, one CBT interval), then
+queried changed areas:
+
+```
+4 extents reported:
+  start=    196608 length=     65536 (64 KiB)
+  start=  51183616 length=   4259840 (4160 KiB)  <-- covers the whole 4 MiB write as ONE extent
+  start= 460783616 length=     65536 (64 KiB)
+  start= 921567232 length=     65536 (64 KiB)
+```
+
+The 4 MiB write came back as a single extent (padded slightly beyond
+4 MiB — 4259840 bytes vs. the exact 4194304 written — to the 64 KiB
+tail-end granularity). Each scattered single-sector write produced its
+own separate 64 KiB extent. **The extent list scales with the number
+of discontiguous changed regions, not with the total volume of changed
+data.** A multi-hundred-GB sequential write is still one small extent
+record; thousands of scattered small writes (e.g. a busy database VM
+doing random I/O across a large disk) produce thousands of extent
+records in one `QueryChangedDiskAreas` response, since the API has no
+pagination. Real backup tools facing that scenario typically chunk the
+query with `start_offset` over fixed-size windows rather than querying
+the whole disk in one call — `query_changed_disk_areas`'s
+`start_offset` parameter exists for this, but nothing in this module
+does the chunking loop itself; that is caller responsibility.
+
+Disk-size scaling itself (e.g., whether extent granularity increases
+for very large disks) was not tested — only reasoned about above as an
+open question, not verified against a large ESXi 8 disk.
+
+## What this does not cover
+
+- `VixDiskLib_QueryAllocatedBlocks` (NFC-level allocated-block bitmap
+  within a single disk, useful for skipping sparse regions inside a
+  delta disk) — implemented separately, see `docs/nfc_read.md`. Pairs
+  naturally with CBT: `query_changed_disk_areas` says which byte
+  ranges changed, `query_allocated_blocks` says which parts of a
+  snapshot's delta disk are actually worth reading. Note its "same
+  still-open write handle" staleness gotcha in `docs/nfc_read.md` if
+  chaining a CBT-driven write with an allocation check.
+- `DDB_GET` fields (`biosGeo`, `adapterType`, `uuid`) — also
+  implemented, see `docs/nfc_open.md`.
diff --git a/docs/nfc_auth.md b/docs/nfc_auth.md
@@ -57,9 +57,9 @@ VDDK logs this as `Connected to VIM Server` / `Authenticating user` /
 `Logged in!`. OpenVixDiskLib keeps that `ServiceInstance` and
 its stub for the ticket call.
 
-Direct ESXi login is the same SOAP login against hostd, but the NFC
-moref and service name differ (`ha-nfc` instead of `nfcService` /
-`vpxa-nfc`). The lab path is vCenter-mediated.
+Direct ESXi login is the same SOAP login, this time against hostd
+instead of vCenter. The NFC moref and service/PROXY name differ; see
+"Direct ESXi (no vCenter)" below for the verified values.
 
 ## Stage 2: NFC ticket
 
@@ -253,6 +253,90 @@ are for local ESXi credentials. With a vCenter ticket:
 argument is not what VDDK sends. The SHA-1 value is for verifying the
 TLS certificate, not for the `THUMBPRINT_SHA2` command.
 
+## Direct ESXi (no vCenter)
+
+Captured against a standalone ESXi 8.0.3 host (`apiType: HostAgent`,
+no vCenter in the picture at all) with the same SSL-hook technique
+from `docs/ssl_hook.md`, using real VDDK 8.0.3 pointed straight at the
+host (`vmxSpec=moref=<N>`, `serverName=<esxi-ip>`). This corrects an
+earlier guess in this file that assumed the moref would be `ha-nfc`.
+
+Differences from the vCenter-mediated path above:
+
+| Item                            | vCenter-mediated       | Direct ESXi (verified)     |
+| ------------------------------- | ---------------------- | -------------------------- |
+| `NfcService` moref              | `nfcService`           | `ha-nfc-service`           |
+| `NfcGetVmFilesResponse.service` | `vpxa-nfc`             | `nfc`                      |
+| `NfcGetVmFilesResponse.host`    | present (ESXi address) | **absent** (omitted field) |
+| authd `PROXY` line              | `PROXY vpxa-nfc`       | `PROXY nfc`                |
+| authd success line              | `200 Connect ha-nfc`   | `200 Connect ha-nfc`       |
+
+The `NfcGetVmFiles` SOAP call itself is unchanged (`vm` argument only);
+only the `_this` moref and the response fields differ:
+
+```xml
+<NfcGetVmFiles xmlns="urn:vim25">
+  <_this type="NfcService">ha-nfc-service</_this>
+  <vm type="VirtualMachine">1</vm>
+</NfcGetVmFiles>
+```
+
+```xml
+<NfcGetVmFilesResponse xmlns="urn:vim25">
+  <returnval>
+    <port>902</port>
+    <sslThumbprint>...</sslThumbprint>
+    <service>nfc</service>
+    <serviceVersion>1.1</serviceVersion>
+    <sessionId>...</sessionId>
+  </returnval>
+</NfcGetVmFilesResponse>
+```
+
+Since `host` is absent, the client must already know where to dial
+authd: the same ESXi host it just logged into over VIM. A vCenter
+ticket always fills `host` because that ESXi address is not otherwise
+known to the client.
+
+### Finding the `ha-nfc-service` moref
+
+VDDK does not hardcode this moref either. Before the `NfcGetVmFiles`
+call, it issues an undocumented `RetrieveInternalContent` call on the
+same `ServiceInstance` moref used for the public
+`RetrieveServiceContent`:
+
+```xml
+<RetrieveInternalContent xmlns="urn:vim25">
+  <_this type="ServiceInstance">ServiceInstance</_this>
+</RetrieveInternalContent>
+```
+
+The response carries ~20 undocumented managed-object refs
+(`agentManager`, `llProvisioningManager`, `diskManager`,
+`nfcService`, `proxyService`, ...); only `nfcService` matters here.
+Its value was `nfcService` in the earlier vCenter capture and
+`ha-nfc-service` on this bare ESXi host — VDDK reads it from this
+response rather than assuming either name.
+
+### OpenVixDiskLib fix
+
+`openvixdisklib/nfc_auth.py` previously hardcoded
+`NFC_SERVICE_MOID = "nfcService"`, which fails outright against a bare
+ESXi host with `vmodl.fault.ManagedObjectNotFound`. It now resolves
+the moref the same way VDDK does: `_nfc_service_moid()` issues the
+`RetrieveInternalContent` SOAP call as raw XML over the existing
+authenticated stub connection (registering pyVmomi types for the full
+undocumented response schema wasn't worth it for one field) and
+regex-extracts `nfcService` from the reply.
+
+`connect_authd()` also gained a `fallback_host` parameter: when
+`ticket.host` is unset (the direct-ESXi case above), it dials the VIM
+connection's own host instead. `openvixdisklib.py` passes
+`conn.si._stub.host` for this.
+
+Validated end-to-end (`ConnectEx` + `Open` + `Read`, both `nbd` and
+`nbdssl` transports) against a live standalone ESXi 8.0.3 host.
+
 ## OpenVixDiskLib
 
 | Piece                | Module                                   | Reuses pyVmomi?                    |

diff --git a/docs/nfc_open.md b/docs/nfc_open.md
@@ -143,7 +143,7 @@ AIO types used for Open / Read / Close, correlated with the consecutive
 | 9    | `SET_SOCK_OPTS`      | 12           |                                               |
 | 22   | `SET_RES_POOL`       | 4            |                                               |
 | 4    | `OPEN_FILE`          | 60           | path string                                   |
-| 11   | `DDB_GET`            | 16           | key name (VDDK only)                          |
+| 11   | `DDB_GET`            | 16           | key name                                      |
 | 7    | `IO`                 | 44           | sector bytes (read reply / write request)     |
 | 5    | `CLOSE_FILE`         | 8            |                                               |
 | 3    | `CLOSE_SESSION`      | 4            |                                               |
@@ -156,6 +156,65 @@ VDDK Open also issues several `DDB_GET` queries (`resumeConsolidateSector`,
 (16 zero bytes) on this unencrypted disk. They are not required to
 obtain a file handle or to read sector 0.
 
+`VixDiskLib_GetInfo` (Step 14) triggers ~20 more `DDB_GET` calls right
+after `OPEN_FILE`, for these keys (captured in request-order, key name
+is the extra string after the 16-byte payload, no `ddb.` prefix on the
+wire): `resumeConsolidateSector`, `isDigest` (×3), `iofilters` (×2),
+`logicalSectorSize` (×2), `physicalSectorSize` (×2),
+`isNativeLinkedClone` (×2), `KMFilters`, `sidecars`, `adapterType`,
+`uuid`, `geometry.cylinders`, `geometry.heads`, `geometry.sectors`,
+`geometry.biosCylinders`, `geometry.biosHeads`, `geometry.biosSectors`.
+On this lab disk, `biosGeo` came back all zeros (key not found) and
+`logicalSectorSize`/`physicalSectorSize`/the non-bios `geometry.*` keys
+duplicate what OPEN_FILE already returned — only `adapterType` and
+`uuid` are genuinely new information from this burst.
+
+### `DDB_GET` request/reply layout
+
+Decoded from the same capture (request/reply pairs matched by `opId`
+across all ~28 calls seen in one `GetInfo`).
+
+Request: 16-byte fixed payload plus the key name as a raw ASCII extra
+(no NUL terminator, not counted in `size` — same convention as
+`OPEN_FILE`'s path):
+
+| Offset | Type     | Meaning                                 |
+| ------ | -------- | --------------------------------------- |
+| 0      | `uint64` | File handle (same value as `OPEN_FILE`) |
+| 8      | `uint32` | Key name length in bytes                |
+| 12     | `uint32` | 0                                       |
+| 16     | —        | Key name (ASCII, no `ddb.` prefix)      |
+
+Reply: 16 bytes plus a value extra, **not** padded (unlike
+`QueryAllocatedBlocks`'s bitmap — verified by decoding all 28 replies
+in sequence with no desync):
+
+| Offset | Type     | Meaning                                 |
+| ------ | -------- | --------------------------------------- |
+| 0-11   | —        | Zero/unused in every capture            |
+| 12     | `uint32` | Value length in bytes (`0` = not found) |
+| 16     | —        | Value (ASCII **text**, not binary)      |
+
+Values are ASCII text even for keys that sound numeric —
+`geometry.cylinders` comes back as the literal bytes `b"2088"`, not a
+binary `uint32`. This matches how a VMDK descriptor file's DDB (disk
+database) section stores keys as plain-text `ddb.<key> = "<value>"`
+lines; `adapterType` comes back as `b"lsilogic"` (a string), not
+VDDK's numeric `VIXDISKLIB_ADAPTER_SCSI_LSILOGIC` enum value — VDDK's
+own client does that string-to-enum mapping internally, which
+OpenVixDiskLib does not reproduce (`DiskInfo.adapter_type` is the raw
+DDB string).
+
+Implemented as `openvixdisklib.nfc_open.NfcDisk.ddb_get(key) -> str |
+None` and `NfcDisk.query_full_info() -> DiskInfo` (5 round trips:
+`geometry.biosCylinders`/`biosHeads`/`biosSectors`, `adapterType`,
+`uuid`), wired into `VixDiskLibHandle.get_info`, which now matches
+real VDDK's `VixDiskLib_GetInfo` exactly — capacity/physGeo free from
+`OPEN_FILE`, the rest costing the same 5 round trips VDDK itself pays.
+Validated against the live ESXi lab: matches native VDDK's `GetInfo`
+output on the same disk (`adapterType=3` ↔ `"lsilogic"`, same `uuid`
+string, same zeroed `biosGeo`).
+
 ### OPEN_SESSION / sockopts / resource pool
 
 `OPEN_SESSION` payload is 16 bytes, little-endian:
@@ -205,9 +264,19 @@ Reply payload (60 bytes), fields that matter:
 | 8      | `uint64` | File handle (opaque, per open)  |
 | 16     | `uint32` | File type (`2` = `NFC_DISK`)    |
 | 20     | `uint32` | Flags echoed (`0x1e` or `0x1a`) |
+| 28     | `uint64` | Disk capacity in **bytes**      |
 | 36     | `uint32` | Sector size (`512` on this VM)  |
-
-Later AIO messages pass that handle as a `uint64`.
+| 40     | `uint32` | Physical geometry cylinders     |
+| 44     | `uint32` | Physical geometry heads         |
+| 48     | `uint32` | Physical geometry sectors       |
+
+Later AIO messages pass that handle as a `uint64`. Offset 28 was found
+by capturing `VixDiskLib_GetInfo` (Step 14,
+`docs/reverse_engineering_procedure.md`): it matches
+`VixDiskLibInfo.capacity` converted to bytes, and offsets 40/44/48
+match `VixDiskLibInfo.physGeo` exactly — both already arrive with this
+reply, no separate `GetInfo` wire call exists. `biosGeo`, `adapterType`,
+and `uuid` are **not** here; VDDK gets those from `DDB_GET` (below).
 
 ### IO (read / write)
 
@@ -232,6 +301,8 @@ classic type 4 `NFC_SESSION_COMPLETE`.
 | Handshake + AIO + OPEN_FILE     | `openvixdisklib.nfc_open.open_disk`                       |
 | AIO extra size / pool count     | `open_disk(..., aio_buffer_size=, aio_buffer_count=)`     |
 | Sector read / write / close     | `openvixdisklib.nfc_open.NfcDisk`                         |
+| Full disk info (`GetInfo`)      | `openvixdisklib.openvixdisklib.VixDiskLibHandle.get_info` |
+| VMDK descriptor DDB lookup      | `openvixdisklib.nfc_open.NfcDisk.ddb_get`                 |
 
 Run:
 
@@ -246,10 +317,15 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
 
 ## What is still VDDK-only
 
-- `DDB_GET` / geometry / zlib and skipz compression / encryption keys
-- `NFC_DELTA_DISK`, change-block tracking
+- zlib and skipz compression / encryption keys (`DDB_GET` is
+  implemented for the plain, non-encrypted keys covered above)
 - Host-switch (`NFC_AIO_SWITCH_HOST_*`)
-- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
+
+Reading/writing a snapshot delta file directly, and running
+`query_allocated_blocks` against it, both already work with the
+existing implementation — `NFC_DELTA_DISK` turned out to be an
+optional VMFS-only VDDK client optimization, not a correctness
+requirement; see `docs/reverse_engineering_procedure.md`.
 
 Reads after open are in `docs/nfc_read.md`. Writes are in
 `docs/nfc_write.md`.