Fix direct-ESXi (no vCenter) connectivity

README.md

@@ -17,17 +17,17 @@ from VDDK 8 NBD traffic; see `docs/`.
 
 ## Status
 
-Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`
-(`nbd` is still available):
+Implemented against vCenter 8 / ESXi 8, including a standalone ESXi
+host with no vCenter. Default transport is `nbdssl` (`nbd` is still
+available):
 
-- `VixDiskLib_ConnectEx` (UID credentials)
+- `VixDiskLib_ConnectEx` (UID credentials; vCenter or direct ESXi)
 - `VixDiskLib_Open` (datastore path, read-only or read-write)
 - `VixDiskLib_Read` (optional ``skip_decompression`` packs FastLZ extras)
 - `VixDiskLib_Write`
 
 Not implemented: compression open flags other than FastLZ, CBT /
-allocated-block queries, disk geometry (`DDB_GET`), encrypted disks,
-and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`.
+allocated-block queries, disk geometry (`DDB_GET`), and encrypted disks.
 
 Requires Python 3.10 or later.
 
@@ -96,11 +96,17 @@ password: secret
 allow_untrusted: true
 datacenter: Datacenter
 datastore: datastore0
+esxi:
+  username: root
+  password: secret
 ```
 
 A session-scoped pytest fixture creates an empty VM with a 10 GiB thin
 disk on that datastore and tears it down when the session ends. Tests
-write known patterns and read them back.
+write known patterns and read them back. Direct-ESXi tests pick the lab
+VM's host from vCenter and log into hostd (default ``root`` and the
+vCenter password) so NFC uses ``ha-nfc-service`` instead of
+``nfcService``. They skip when lockdown is on or hostd login fails.
 
 ```bash
 tox -e integration

docs/nfc_auth.md

@@ -57,9 +57,9 @@ VDDK logs this as `Connected to VIM Server` / `Authenticating user` /
 `Logged in!`. OpenVixDiskLib keeps that `ServiceInstance` and
 its stub for the ticket call.
 
-Direct ESXi login is the same SOAP login against hostd, but the NFC
-moref and service name differ (`ha-nfc` instead of `nfcService` /
-`vpxa-nfc`). The lab path is vCenter-mediated.
+Direct ESXi login is the same SOAP login, this time against hostd
+instead of vCenter. The NFC moref and service/PROXY name differ; see
+"Direct ESXi (no vCenter)" below for the verified values.
 
 ## Stage 2: NFC ticket
 
@@ -253,6 +253,90 @@ are for local ESXi credentials. With a vCenter ticket:
 argument is not what VDDK sends. The SHA-1 value is for verifying the
 TLS certificate, not for the `THUMBPRINT_SHA2` command.
 
+## Direct ESXi (no vCenter)
+
+Captured against a standalone ESXi 8.0.3 host (`apiType: HostAgent`,
+no vCenter in the picture at all) with the same SSL-hook technique
+from `docs/ssl_hook.md`, using real VDDK 8.0.3 pointed straight at the
+host (`vmxSpec=moref=<N>`, `serverName=<esxi-ip>`). This corrects an
+earlier guess in this file that assumed the moref would be `ha-nfc`.
+
+Differences from the vCenter-mediated path above:
+
+| Item                            | vCenter-mediated       | Direct ESXi (verified)     |
+| ------------------------------- | ---------------------- | -------------------------- |
+| `NfcService` moref              | `nfcService`           | `ha-nfc-service`           |
+| `NfcGetVmFilesResponse.service` | `vpxa-nfc`             | `nfc`                      |
+| `NfcGetVmFilesResponse.host`    | present (ESXi address) | **absent** (omitted field) |
+| authd `PROXY` line              | `PROXY vpxa-nfc`       | `PROXY nfc`                |
+| authd success line              | `200 Connect ha-nfc`   | `200 Connect ha-nfc`       |
+
+The `NfcGetVmFiles` SOAP call itself is unchanged (`vm` argument only);
+only the `_this` moref and the response fields differ:
+
+```xml
+<NfcGetVmFiles xmlns="urn:vim25">
+  <_this type="NfcService">ha-nfc-service</_this>
+  <vm type="VirtualMachine">1</vm>
+</NfcGetVmFiles>
+```
+
+```xml
+<NfcGetVmFilesResponse xmlns="urn:vim25">
+  <returnval>
+    <port>902</port>
+    <sslThumbprint>...</sslThumbprint>
+    <service>nfc</service>
+    <serviceVersion>1.1</serviceVersion>
+    <sessionId>...</sessionId>
+  </returnval>
+</NfcGetVmFilesResponse>
+```
+
+Since `host` is absent, the client must already know where to dial
+authd: the same ESXi host it just logged into over VIM. A vCenter
+ticket always fills `host` because that ESXi address is not otherwise
+known to the client.
+
+### Finding the `ha-nfc-service` moref
+
+VDDK does not hardcode this moref either. Before the `NfcGetVmFiles`
+call, it issues an undocumented `RetrieveInternalContent` call on the
+same `ServiceInstance` moref used for the public
+`RetrieveServiceContent`:
+
+```xml
+<RetrieveInternalContent xmlns="urn:vim25">
+  <_this type="ServiceInstance">ServiceInstance</_this>
+</RetrieveInternalContent>
+```
+
+The response carries ~20 undocumented managed-object refs
+(`agentManager`, `llProvisioningManager`, `diskManager`,
+`nfcService`, `proxyService`, ...); only `nfcService` matters here.
+Its value was `nfcService` in the earlier vCenter capture and
+`ha-nfc-service` on this bare ESXi host — VDDK reads it from this
+response rather than assuming either name.
+
+### OpenVixDiskLib fix
+
+`openvixdisklib/nfc_auth.py` previously hardcoded
+`NFC_SERVICE_MOID = "nfcService"`, which fails outright against a bare
+ESXi host with `vmodl.fault.ManagedObjectNotFound`. It now resolves
+the moref the same way VDDK does: `_nfc_service_moid()` issues the
+`RetrieveInternalContent` SOAP call as raw XML over the existing
+authenticated stub connection (registering pyVmomi types for the full
+undocumented response schema wasn't worth it for one field) and
+regex-extracts `nfcService` from the reply.
+
+`connect_authd()` also gained a `fallback_host` parameter: when
+`ticket.host` is unset (the direct-ESXi case above), it dials the VIM
+connection's own host instead. `openvixdisklib.py` passes
+`conn.si._stub.host` for this.
+
+Validated end-to-end (`ConnectEx` + `Open` + `Read`, both `nbd` and
+`nbdssl` transports) against a live standalone ESXi 8.0.3 host.
+
 ## OpenVixDiskLib
 
 | Piece                | Module                                   | Reuses pyVmomi?                    |
@@ -273,10 +357,13 @@ Run:
 
 ```bash
 .venv/bin/pytest tests/integration/test_nfc_auth.py
+.venv/bin/pytest tests/integration/test_direct_esxi.py
 ```
 
-The test completes VIM login and the authd handshake (`200 Connect`)
-and asserts an established TLS socket on `ticket.host:ticket.port`.
+`test_nfc_auth.py` completes VIM login and the authd handshake against
+vCenter. `test_direct_esxi.py` picks the lab VM's ESXi host from
+inventory and repeats ConnectEx / Open / Read on hostd, where the
+ticket omits `host` and NfcService is `ha-nfc-service`.
 
 ## What comes after authentication
 

docs/nfc_open.md

@@ -249,7 +249,6 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
 - `DDB_GET` / geometry / zlib and skipz compression / encryption keys
 - `NFC_DELTA_DISK`, change-block tracking
 - Host-switch (`NFC_AIO_SWITCH_HOST_*`)
-- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
 
 Reads after open are in `docs/nfc_read.md`. Writes are in
 `docs/nfc_write.md`.

docs/reverse_engineering_procedure.md

@@ -9,7 +9,8 @@ NFC work can follow the same loop instead of rediscovering it.
 
 Scope so far: `VixDiskLib_ConnectEx` + `VixDiskLib_Open` +
 `VixDiskLib_Read` + `VixDiskLib_Write` against lab vCenter 8.0.1 /
-ESXi 8, transports `nbd` and `nbdssl`. Validation method:
+ESXi 8, transports `nbd` and `nbdssl`, plus a standalone ESXi 8.0.3
+host with no vCenter (Step 13). Validation method:
 `tests/integration/` (the session-scoped `lab` fixture creates a temporary
 empty VM with a 10 GiB disk and destroys it when the pytest session ends).
 
@@ -381,8 +382,33 @@ not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902
 Replay: pip `pyfastlz` via `openvixdisklib/fastlz.py` (NFC extra is
 raw FastLZ, without the wrapper's 4-byte length prefix) plus `NfcDisk`
 compression on each IO. Proof:
-`tests/integration/test_nfc_read_write.py` (`fastlz`) and
-`tests/perf/test_compare.py`.
+## Step 13 — Direct ESXi (`ha-nfc`) without vCenter
+
+Same SSL-hook technique (Step 4), this time pointing VDDK 8.0.3
+straight at a standalone ESXi 8.0.3 host (`vmxSpec=moref=<N>`,
+`serverName=<esxi-ip>`, no vCenter in the topology). Confirmed
+OpenVixDiskLib's hardcoded `NFC_SERVICE_MOID = "nfcService"` fails on
+this host with `vmodl.fault.ManagedObjectNotFound` *before* touching
+the capture — reproduced with plain `openvixdisklib` calls, no hook
+needed to see that failure.
+
+The capture showed VDDK does not hardcode the moref either: it calls
+an undocumented `RetrieveInternalContent` on the `ServiceInstance`
+moref first, and reads `nfcService` from the reply (`ha-nfc-service`
+on this host, vs. `nfcService` in the earlier vCenter capture).
+`NfcGetVmFilesResponse.service` was `nfc` (not `vpxa-nfc`), and its
+`host` field was **absent** — the authd endpoint is implicitly the
+same host already logged into. Full detail in `docs/nfc_auth.md`
+("Direct ESXi (no vCenter)").
+
+Fix: `_nfc_service_moid()` in `openvixdisklib/nfc_auth.py` issues the
+`RetrieveInternalContent` call as raw SOAP over the existing stub
+connection (its response schema has ~20 other undocumented morefs not
+worth registering with pyVmomi's type system for one field), and
+`connect_authd()` takes a `fallback_host` used when `ticket.host` is
+unset. Validated end-to-end (`ConnectEx`/`Open`/`Read`, `nbd` and
+`nbdssl`) against the live host.
+
 
 ## What to write down
 
@@ -408,4 +434,3 @@ Not yet reversed, same loop as above:
 - `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
 - `VixDiskLib_GetInfo` capacity
 - Host-switch AIO messages
-- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`

openvixdisklib/nfc_auth.py

@@ -19,16 +19,18 @@
 
 import contextlib
 import hashlib
+import re
 import socket
 import ssl
+import types
 
 from pyVim.connect import Disconnect, SmartConnect
 from pyVmomi import vim
 from pyVmomi.VmomiSupport import F_OPTIONAL, CreateManagedType, GetVmodlType
 
-NFC_SERVICE_MOID = "nfcService"
 AUTHD_DEFAULT_PORT = 902
 _NFC_TYPES_REGISTERED = False
+_NFC_SERVICE_MOID_RE = re.compile(r"<nfcService[^>]*>([^<]+)</nfcService>")
 
 
 def _ssl_client_context(verify: bool = True) -> ssl.SSLContext:
@@ -121,6 +123,44 @@ def _register_nfc_types() -> None:
     _NFC_TYPES_REGISTERED = True
 
 
+def _nfc_service_moid(si: vim.ServiceInstance) -> str:
+    """Return the NfcService moref via the internal ``RetrieveInternalContent`` call.
+
+    vCenter and a bare ESXi host disagree on this moref (``nfcService`` vs.
+    ``ha-nfc-service``); VDDK resolves it dynamically instead of assuming
+    vCenter's name, which is why OpenVixDiskLib must too. The response also
+    carries ~20 other undocumented managed-object refs (agent manager, disk
+    manager, and so on) that aren't worth registering with pyVmomi's type
+    system just to read one field, so the call is issued as raw SOAP over
+    the existing authenticated connection and only ``nfcService`` is pulled
+    out of the XML.
+    """
+    stub = si._stub
+    info = types.SimpleNamespace(
+        wsdlName="RetrieveInternalContent", version=stub.version, params=()
+    )
+    request = stub.SerializeRequest(si, info, ())
+    headers = {
+        "Cookie": stub.cookie,
+        "SOAPAction": stub.versionId,
+        "Content-Type": "text/xml; charset=utf-8",
+    }
+    conn = stub.GetConnection()
+    try:
+        conn.request("POST", stub.path, request, headers)
+        response = conn.getresponse()
+        body = response.read().decode("utf-8")
+    finally:
+        stub.ReturnConnection(conn)
+    match = _NFC_SERVICE_MOID_RE.search(body)
+    if response.status != 200 or not match:
+        raise RuntimeError(
+            f"RetrieveInternalContent (status {response.status}) "
+            "had no nfcService moref"
+        )
+    return match.group(1)
+
+
 def nfc_service(si: vim.ServiceInstance) -> vim.NfcService:
     """Return the vCenter/ESXi NfcService managed object on ``si``'s SOAP stub.
 
@@ -129,7 +169,7 @@ def nfc_service(si: vim.ServiceInstance) -> vim.NfcService:
     """
     _register_nfc_types()
     nfc_cls = GetVmodlType("vim.NfcService")
-    return nfc_cls(NFC_SERVICE_MOID, si._stub)
+    return nfc_cls(_nfc_service_moid(si), si._stub)
 
 
 def connect_vim(
@@ -315,6 +355,7 @@ def connect_authd(
     allow_untrusted: bool = False,
     timeout: float = 30.0,
     nfc_ssl: bool = True,
+    fallback_host: str | None = None,
 ) -> ssl.SSLSocket:
     """Complete the ESXi authd handshake using an NFC HostServiceTicket.
 
@@ -337,8 +378,12 @@ def connect_authd(
         timeout: Socket timeout in seconds.
         nfc_ssl: When True (the default), PROXY to the NFCSSL service
             used by nbdssl. Pass False for plaintext NFC (nbd).
+        fallback_host: Host to dial when ``ticket.host`` is unset. A ticket
+            issued directly by a bare ESXi host (no vCenter) omits ``host``
+            entirely, since the authd endpoint is that same host; pass the
+            VIM connection's host in that case.
     """
-    host = ticket.host
+    host = ticket.host or fallback_host
     port = ticket.port or AUTHD_DEFAULT_PORT
     raw = socket.create_connection((host, port), timeout=timeout)
     try:
@@ -463,7 +508,7 @@ def authenticate(
             read_only=read_only,
         )
         authd_sock = connect_authd(
-            ticket, allow_untrusted=allow_untrusted, nfc_ssl=nfc_ssl
+            ticket, allow_untrusted=allow_untrusted, nfc_ssl=nfc_ssl, fallback_host=host
         )
     except Exception:
         Disconnect(si)

openvixdisklib/openvixdisklib.py

@@ -305,7 +305,10 @@ def open(
             conn.si, vm, read_only=read_only, disk_path=None if read_only else disk_path
         )
         authd_sock = nfc_auth.connect_authd(
-            ticket, allow_untrusted=conn.allow_untrusted, nfc_ssl=nfc_ssl
+            ticket,
+            allow_untrusted=conn.allow_untrusted,
+            nfc_ssl=nfc_ssl,
+            fallback_host=conn.si._stub.host.rsplit(":", 1)[0],
         )
         session = nfc_auth.NfcAuthSession(conn.si, ticket, authd_sock, nfc_ssl=nfc_ssl)
         try:

tests/conftest.py

@@ -16,6 +16,7 @@
     destroy_lab_vm,
     ensure_vddk_library_path,
     require_vddk,
+    resolve_direct_esxi_lab,
 )
 
 
@@ -31,6 +32,12 @@ def lab() -> Iterator[LabEnv]:
         destroy_lab_vm(env)
 
 
+@pytest.fixture(scope="session")
+def esxi_lab(lab: LabEnv) -> LabEnv:
+    """The session lab VM addressed through its ESXi host, not vCenter."""
+    return resolve_direct_esxi_lab(lab)
+
+
 @pytest.fixture(scope="session")
 def vddk() -> None:
     """Skip VDDK-backed tests when ``libvixDiskLib`` cannot be loaded."""