Skip to content

Build deterministic physical HID fixture and capture lab - #1238

Draft
malpern wants to merge 107 commits into
masterfrom
codex/pico-hid-fixture
Draft

malpern wants to merge 107 commits into
masterfrom
codex/pico-hid-fixture

Conversation

@malpern

@malpern malpern commented Jul 28, 2026 •

Copy link
Copy Markdown
Owner

What changed

  • adds the ESP32-S3/Pico physical USB HID fixture, secure Wi-Fi control plane, dual-slot OTA, display health, animated device UI, safety releases, and QEMU coverage
  • adds the native KeyPath HID Capture Jig with a fail-closed focus oracle, exact event ledger, resource admission, responsive UI, and persisted JSON evidence
  • adds combined physical runners plus a Shift lead/hold/release diagnostic matrix correlated with bounded firmware traces
  • expands the installer/runtime lab, scenario campaign tooling, managed clone support, and lab documentation
  • routes CLI start/stop/restart service operations through the privileged helper instead of invoking system launchd control from an unprivileged process

Why

KeyPath needs repeatable end-to-end evidence that real installer/runtime builds correctly process physical keyboard input, including high-rate and resource-pressure regressions. Previous TextEdit and scheduler-only checks could not prove focus, exact output, or safe release. The new Jig and fixture preserve independent machine-readable evidence and fail closed when focus or host readiness is invalid.

Validation

  • Scripts/lab/pico-hid-fixture-tool test (host/core/client/runner/matrix plus ESP32-S3 QEMU smoke)
  • native HID Capture Jig suite: 26 tests
  • KeyPath privileged-helper and runtime-health regression suite: 56 focused tests
  • production firmware build and authenticated OTA verification on Waveshare ESP32-S3 build 4d1cb1cb54c4
  • connected-board status: valid OTA slot, USB mounted, Wi-Fi connected, splash complete, display heartbeat healthy
  • admitted three-cell physical Shift smoke matrix: baseline, 4 ms lead, and 8 ms lead all delivered exact 24/24 output with valid focus, released keys/modifiers, complete firmware traces, and zero late reports
  • full 5x5 physical Shift lead/release matrix: 25/25 valid cells, 1,500/1,500 characters captured, 4,000 firmware trace reports, valid focus and clean final releases throughout, and zero late reports
  • full-matrix diagnosis: 10 Shift demotions were confined to four of five zero-lead cells; all 20 cells and 1,200 characters with a 2 ms-or-greater modifier lead were exact

Pending physical acceptance

This remains a draft until the remaining person-dependent physical gates are complete. Real USB HID input cannot target a background app safely. Remaining gates are:

  • final visual confirmation that the repaired splash completes
  • USB disconnect/reconnect and intentional Wi-Fi-loss safe-abort checks

Excluded focus-, memory-, load-, and network-admission attempts are documented as harness admissions, not KeyPath outcomes.

Clean-VM resource admission correction

The capture Jig previously read kern.num_taskthreads as current thread inventory. That value is the per-process limit (4096 in the clean VM), which falsely blocked idle guests. The Jig now samples the live count through Apple's top reader off the UI thread, with a two-second timeout, one pending query, and a three-second freshness limit. Missing or stale inventory refuses arming; existing load thresholds are unchanged.

Validation: failing admission regression reproduced before the fix; all 37 Jig tests pass, including the production collector, timeout/failure/malformed output, pending-query exclusion, and stale-cache cases. The corrected Developer ID-signed Jig passed strict verification and its resource gate passed in the macOS 26 VM. Physical remap/restart validation remains in progress.

malpern added 30 commits July 24, 2026 20:14
@malpern

malpern commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

Post-update hardware control evidence: OTA build f92209f3ea52 verified healthy in valid slot ota_1. A 120-request no-HID soak passed 120/120 with 28.153 ms median, 93.715 ms p95, 105.845 ms max, zero request failures/resets/build changes/server restarts, max firmware handler latency 7.465 ms, and 8.33 MB free heap. Artifact: ~/.local/state/keypath-hid-fixture/control/control-soak-20260729T014347Z.json. Physical button/touch/demo-output validation remains intentionally deferred until an exclusive desktop window.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review scope note: the diff supplied to me was truncated after ~60k bytes (cut off partway through Scripts/lab/hid-capture-jig/...KeycapBurstModel.swift), so I did not see the changes to Sources/KeyPathCLI/.../PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, Sources/KeyPathCLISupport/CLIReportModels.swift, ServiceBootstrapper.swift, or the associated tests. Those should get a separate look.

Finding — Scripts/build-and-sign.sh: SKIP_DEPLOY=1 doesn't gate the website-publish step

The new SKIP_DEPLOY block wraps the "stop/replace/restart the installed app" section in if [ "${SKIP_DEPLOY:-0}" = "1" ]; then … else … fi, closing the fi right before GHPAGES_DIR=.... The stated intent (per the added comment) is that a matrix build "must not replace or restart the operator's installed app." But the subsequent "Publish help content to website" block is outside that if/else and keeps its original guard ([ -d "$GHPAGES_DIR" ] && [ "${SKIP_WEBSITE:-0}" != "1" ] && [ "${SKIP_NOTARIZE:-}" != "1" ]), so it still runs regardless of SKIP_DEPLOY.

Since matrix/lab builds want "the exact signed artifact" they presumably don't set SKIP_NOTARIZE=1, so on any machine where .worktrees/gh-pages happens to exist (e.g. a release engineer's box also used to produce matrix-test artifacts), running SKIP_DEPLOY=1 ./Scripts/build-and-sign.sh would still publish/copy help content toward the live website — an unintended side effect for a flag meant to make the build a no-op with respect to the operator's environment. Consider also gating the website-publish block on SKIP_DEPLOY, or explicitly documenting that it's intentionally excluded.

No other actionable correctness/security issues found in the reviewed portion (the new assert-runtime-state/assert-uninstalled-state/desktop-bootstrap lab scripts look reasonable; checksums in desktop-bootstrap are valid 64-hex-char SHA-256 values).

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated at 60000 bytes, cutting off before Sources/KeyPathCLI/.../PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, ServiceBootstrapper.swift, CLIReportModels.swift, Scripts/lab/keypath-lab, Scripts/lab/remote.sh, and most of HIDCaptureJig/main.swift — those are worth a separate pass, especially given AGENTS.md's postcondition-verification requirements for install/uninstall/service-lifecycle code).

One plausible flake in the portion I could review:

Scripts/lab/hid-capture-jig-tool demo subcommand — focus/arm race

"$client" reset >/dev/null
"$client" focus >/dev/null
"$client" arm --run-id offline-demo --expected "$expected" ...

focus and arm are sent as two independent synchronous file-RPC calls with nothing waited on between them. CaptureSession.arm() (Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift) hard-fails immediately if focused is false at the moment arm is processed:

guard !runID.isEmpty, runID.count <= 80, !expected.isEmpty,
      timeoutMs >= 250, timeoutMs <= 300_000,
      settleMs >= 50, settleMs <= 5000,
      focused
else { ... state = .failed; return false }

If the app's window/app-activation focus notification lands even one runloop tick after the focus RPC returns "ok", the immediately-following arm call will read stale focused=false and the whole demo run fails non-deterministically — independent of anything the operator does. Consider having arm poll/wait briefly for focus (or having the client poll status for focused: true before sending arm) instead of assuming synchronous ordering.

Everything else reviewed (build-and-sign.sh's new SKIP_DEPLOY guard, the new assert-* fixtures, desktop-bootstrap's checksum-verified tool installation) looks sound.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated after ~60KB, so remote.sh, physical-hid-capture-run, scenario-matrix-runner, and most of the Swift HID-capture-jig sources beyond KeycapBurstModel.swift weren't visible to me — worth a follow-up pass on those).

One actionable finding in the portion reviewed:

Scripts/lab/desktop-bootstrap — version/checksum pins are bypassed once a binary already exists. install_desktop_tools() gates each download on [[ ! -x "$local_bin/peekaboo" ]] (same pattern for mcporter, uv), not on the binary's actual version/hash. If a lab host is reused across script revisions (rather than being a freshly provisioned VM every time), a stale binary from a prior bootstrap run will silently stick around even after PEEKABOO_VERSION/PEEKABOO_SHA256 etc. are bumped in this file — the checksum verification in download_verified() never runs against it. Consider checking the installed binary's hash (or a version marker file) against the pinned constant and re-downloading on mismatch, so bumping the pinned version actually takes effect on existing hosts.

Everything else I could see — the SKIP_DEPLOY wrapping in build-and-sign.sh, the new assert-runtime-state/assert-uninstalled-state verifiers, and the hid-capture-jig-client/-tool command protocol — looked structurally sound; no other correctness/security issues stood out.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Correctness bug: CaptureSession.arm() failure isn't sticky, so a bad arm can later self-report as "passed"

Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift

In arm(), the validation-guard failure path only sets state = .failed — it never sets terminalFailure = true, and deadlineAtNs/settleNs are left at their defaults (deadlineAtNs = nil, settleNs = 250_000_000) because the success-path assignments are skipped:

guard !runID.isEmpty, runID.count <= 80, !expected.isEmpty,
      timeoutMs >= 250, timeoutMs <= 300_000,
      settleMs >= 50, settleMs <= 5000,
      focused
else {
    self.runID = runID
    self.expected = expected
    self.focused = focused
    state = .failed
    return false
}

If runID is non-empty in this failure case (e.g. arm rejected only because focused == false, or timeoutMs/settleMs out of range), subsequent record() calls pass the !runID.isEmpty guard and reach evaluate(). Since terminalFailure is still false, the early-out if terminalFailure { state = .failed; return } never fires, so evaluate() falls through to its normal pass/fail logic. Because deadlineAtNs is nil, the timeout branch never triggers, but the settle branch:

if received == expected, pressedKeyCodes.isEmpty, activeModifiers == 0,
   let lastEventAtNs, nowNs >= lastEventAtNs,
   nowNs - lastEventAtNs >= settleNs
{
    state = .passed
}

can still trigger once received catches up to expected and ~250ms (the default settleNs) elapses with no keys held — silently flipping a rejected/invalid arm from .failed to .passed. This would let a capture run that was armed with invalid parameters report a false PASS to callers polling status/wait in hid-capture-jig-client/hid-capture-jig-tool, undermining the acceptance signal this fixture exists to provide.

Suggested fix: set terminalFailure = true in the guard-else branch (or gate record()/evaluate() on a distinct invalidArm flag) so a failed arm() can never transition to .passed.


Note: this diff is very large (Swift capture-jig core, Python/zsh lab scripts, ESP32/Pico firmware in C, etc.) and was truncated in the review context after ~60KB. I reviewed the visible portion in depth (Swift CaptureSession, the Python assert-runtime-state/assert-uninstalled-state/hid-capture-jig-client scripts, and the desktop-bootstrap/build-and-sign.sh shell changes) and found no other high-confidence issues there. The remainder of the diff (firmware sources, additional test files, remote.sh, scenario-matrix-runner, etc.) was not visible to this review and should get a separate pass.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (note: it was truncated after Scripts/lab/hid-capture-jig/Sources/HIDCaptureCore/CaptureSession.swift, so I could not see the actual content of the product-code changes later in the file list — PrivilegedOperationsRouter.swift, SystemUninstallCommand.swift, ServiceBootstrapper.swift, CLIReportModels.swift, and the CLI test updates. Those are the highest-risk changes in this PR since everything else is disposable lab tooling; please make sure they get separate scrutiny, e.g. via ./Scripts/review-gate.sh.).

Within what I could see:

  • Scripts/build-and-sign.sh: the new SKIP_DEPLOY=1 branch wraps the "stop process / replace bundle / restart" block in an if/else/fi, with the closing fi inserted right before the "Publish help content to website" section. The bracing looks correct, but worth double-checking that nothing set inside that now-conditional block (e.g. a deploy path/version variable) is read later in the script when SKIP_DEPLOY=1 skips it — that portion of the file wasn't in the diff, so I couldn't confirm.

  • Scripts/lab/desktop-bootstrap: the old flow auto-clicked "Allow" on the privacy consent sheet; the new flow instead detects the consent dialog via JXA and exits with code 4 (a "checkpoint") for external handling. That's a real behavior change for any caller that expected desktop-bootstrap --install-tools to complete unattended — worth confirming keypath-lab/CI callers were updated to handle exit code 4 (not visible in the supplied diff).

Everything else reviewed (checksum-pinned downloads in desktop-bootstrap, assert-runtime-state, assert-uninstalled-state, CaptureSession.swift state machine) looked internally consistent with no obvious correctness bugs.

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (truncated at 60000 bytes out of a 144-file/~18.6k-line diff, so most of the new pico-hid-fixture C/ESP32 firmware and the HIDCaptureJig Swift package beyond CaptureSession.swift were not visible to this review).

Scripts/lab/damage-kanata-service — hard failure if the service is already unloaded

set -euo pipefail
...
if [[ -f "$plist" ]]; then
  /usr/bin/sudo -n /bin/launchctl bootout system "$plist"
else
  /usr/bin/sudo -n /bin/launchctl bootout system/com.keypath.kanata
fi

launchctl bootout exits non-zero when the job isn't currently loaded/registered (e.g. it was already booted out by a prior run, or install-runtime hasn't started it yet). Because the script runs under set -e, that failure kills the script immediately — it never reaches the polling loop below that would otherwise treat "no longer registered" as success (print "damage_fixture\tkanata-service-booted-out"). Net effect: this "damage" fixture can spuriously fail scenario runs purely due to fixture-script fragility rather than an actual runtime state problem. Consider tolerating a non-zero bootout exit (e.g. ... || true) and always falling through to the existing poll/verify loop to determine success/failure from actual launchd state.

No other actionable correctness/security issues found in the portion of the diff reviewed (build-and-sign.sh's new SKIP_DEPLOY if/else appears correctly scoped around just the stop/replace/restart block, and the checksum-pinned desktop-bootstrap downloads use full 64-char SHA-256 values as expected).

@claude

claude Bot commented Jul 29, 2026

Copy link
Copy Markdown

Reviewed the supplied diff (truncated after 60000 bytes — Swift/C sources for the HID capture jig, pico-hid-fixture firmware, scenario-matrix runner, and keypath-lab changes beyond that point were not covered).

1. Peekaboo/mcporter download source changed orgs (Scripts/lab/desktop-bootstrap)
install_desktop_tools() now pins and downloads from github.com/openclaw/Peekaboo and github.com/openclaw/mcporter, replacing the previous brew install steipete/tap/peekaboo / steipete/tap/mcporter install path referenced elsewhere in the repo (AGENTS.md still documents steipete/tap/peekaboo). SHA256 pinning mitigates tampering-in-transit, but this is a trust-boundary change (individual steipete identity → openclaw org) that isn't called out in the diff. Worth confirming openclaw is genuinely the canonical upstream for these tools before this lands, since a wrong/typo'd org would silently pin and execute an unrelated binary in the lab bootstrap path.

2. Scripts/build-and-sign.sh — verify the new SKIP_DEPLOY conditional scoping
The new if [ "${SKIP_DEPLOY:-0}" = "1" ]; then ... else <stop/replace/restart> fi wraps the local-deploy block, and the added closing fi lands right after the # Publish help content to website comment header but before the GHPAGES_DIR=... check. That means the website-publish logic sits outside the SKIP_DEPLOY guard (runs regardless), while the comment header describing it is now inside the guarded block. Given two fis appear back-to-back with a comment sandwiched between them, please double check both the SKIP_DEPLOY=1 and default paths execute exactly as intended (e.g. bash -n plus a dry run of both branches) — it's easy for a stray/misplaced fi in a script this size to silently change what block a later section belongs to.

3. Scripts/lab/desktop-bootstrap — consent-prompt auto-click removed
The previous implementation parsed Peekaboo's permission JSON and automatically clicked "Allow" on the notification-center consent sheet. The new implementation only detects a screen-capture or private-window-picker consent prompt via JXA and exits with code 4 plus a checkpoint JSON, without clicking anything. If this is an intentional shift to a manual/external-checkpoint model, please confirm the caller(s) (keypath-lab, scenario runners) actually handle exit code 4 by resuming after out-of-band approval — otherwise previously-unattended lab bootstrap runs could now stall or fail where they used to auto-proceed.

No other correctness/security issues found in the reviewed portion (the new assert-runtime-state, assert-uninstalled-state, hid-capture-jig-client, and hid-capture-jig-tool scripts look internally consistent).

The fixture could only send a boot-keyboard report, so it could not press
Mute, Volume or Play/Pause — the keys KeyPath's driverless session now reads
as inputs. Both targets now describe a consumer-control report (ID 2)
alongside the keyboard report. A script event line takes an optional ninth
field, the consumer usage held; eight-field scripts are unchanged. Each event
changes the keyboard state or the consumer state, never both, so it is
exactly one USB report; the cycle must end with everything released, and an
abort or unmount releases a held media key before the keyboard. The trace
and the HTTP trace JSON carry the consumer usage. The client gains
`compile-media`.

Tests: four host core tests (press/release and trace, eight-field scripts
never send consumer reports, unsafe consumer scripts refused, abort releases
the media key first) and two client tests.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@malpern

malpern commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Holding this open until malpern/vm-lab#4 is reviewed. Much of the lab tooling here has moved to vm-lab; once that lands, the plan is to split out what KeyPath itself still needs (fixture firmware/client, KeyPath-side scenarios) and close the rest. #1333's consumer-control support is already merged into this branch.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant