Skip to content

[agent] Hardware-testing fixes for the Swift client - #15

Merged
drycode merged 3 commits into
mainfrom
dy/swift-rewrite-integration
Aug 3, 2026
Merged

[agent] Hardware-testing fixes for the Swift client#15
drycode merged 3 commits into
mainfrom
dy/swift-rewrite-integration

Conversation

@drycode

@drycode drycode commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Four defects found by running #12 on real hardware — Studio, real dictation, against the existing Python server. It now works end to end. Every one of these passed both test suites, because they test what the code says and these are failures of what it reaches.

The hotkey misses almost every release

Hotkey.swift reconstructed the chord from keycode + modifier flags and applied that test to both edges. The flags describe the instant the event was generated, and releasing Ctrl+Alt+Space almost always lifts a modifier at or before the space bar — so the space key-UP arrives with the modifier bits already clear.

Measured with a bare session tap over 45 s of ordinary use:

DOWN events: 303      (auto-repeat while held)
UP events:     1
tap disabled:  0      (not the timeout path)

In the agent that left a capture that never ended, and since beginCapture() guards on state == .idle, every later press was ignored — presenting as "the hotkey stopped working".

This is the Phase 0 spike the design asked for, and it now has an answer. Switched to Carbon RegisterEventHotKey: pressed and released arrive as distinct events, the release is not conditional on modifier state, it consumes the chord so it does not also reach the focused app, and it needs no permission of its own — so no Input Monitoring grant is required. Accessibility is still needed for the ⌘V, so it is one grant either way.

The client could not reach any deployment

AgentController hardcoded http://127.0.0.1:<harkPort>/dictate with the key from ~/.config/hark/key — the server's deployment config used as the client's. That breaks the two-machine setup, and it also breaks a single machine whose server binds a tailnet address: measured here, 127.0.0.1:8911 refuses while 100.64.66.46:8911 answers 200.

Adds ClientConfig (~/.config/hark/client.json) and enforces the transport policy the design already specifies rather than assuming it — plain HTTP to loopback always; to a numeric IP only with an explicit allowPlaintext, because a tailnet is a defensible place for plaintext but it should be a stated decision; to a hostname never, since a MagicDNS name is a hostname. Absent config falls back to the previous behaviour.

Recorder accumulated across captures

samples and peak are instance properties on a long-lived Recorder, and neither start() nor stop() cleared them. Capture 2 transcribed 1+2, capture 3 transcribed 1+2+3, and the WAV grew without bound. Presented as the transcript repeating the previous dictation.

There was no presentation layer

private func alert(_ message: String) {
    log.info(message)
    NSWorkspace.shared.notificationCenter.post(name: .init("HarkAlert"), object: message)
}                                    // nothing observes "HarkAlert"

private func brief(_ message: String) { _ = message }

The whole diagnostic surface — 401, 415, 400, 503, transport failure, "heard nothing", "paste withheld (focus moved)" — was discarded. Added Overlay, a non-activating floating panel so it never pulls focus from what you are dictating into, and wired all three paths to it. The menu bar item stays.

Smaller

  • 200 ms ⌘V hold. Posting key-up immediately after key-down is a zero-duration keystroke that some apps drop. hs.eventtap.keyStroke holds 200 ms (local keyDelay = 200000); same tap, same flags, the gap was the only difference.
  • AXIsProcessTrusted re-checked at paste time, and on failure the user is told the transcript is on the clipboard. Everything before that point reports success either way, so an untrusted process was indistinguishable from a dropped event.
  • requestAccess waits for its answer instead of discarding it. An ungranted process is not refused the device — it receives full-length buffers of zeros.
  • status.json no longer reports a constant. hotkey was a hardcoded "registered" regardless of whether anything was bound, which is why the dead tap looked healthy from outside.
  • Trailing space on the transcript, so consecutive dictations do not run together. Client-side deliberately: the server should return the transcript, not presentation whitespace.
  • HARK_CLIENT_CONFIG overrides the config path, mirroring the server's HARK_CONFIG. Neither homeDirectoryForCurrentUser nor NSHomeDirectory() honours $HOME on macOS, so HOME=... hark agent silently reads the real config — which produced a round of "passing" policy checks all reading the same live file.

Verification

Studio: recording, transcription and paste all working; repeats gone; indicator visible; consecutive dictations separated. The 48 SwiftPM tests pass at every commit.

Not covered here, and still open: wiring install-client.sh onto hark agent, deleting the Lua client, and whether hark serve replaces the Python server — nothing here tested the server rewrite.

drycode added 3 commits August 3, 2026 16:40
Three fixes on top of the Swift rewrite, all found by running it against
a real deployment rather than by reading it.

CLIENT ADDRESSING. AgentController hardcoded
http://127.0.0.1:<harkPort>/dictate with the key from
~/.config/hark/key. That is the server's deployment config being used as
the client's, and it cannot work for the two-machine setup the project
exists to serve: the recording Mac has no server, no key file and
nothing on loopback. It also fails on a SINGLE machine whose server
binds a tailnet address - measured here, 127.0.0.1:8911 refuses while
100.64.66.46:8911 answers 200 - so loopback was wrong for every
deployment on hand.

Adds ClientConfig (~/.config/hark/client.json), and enforces the
transport policy the design doc already specifies rather than assuming
it: plain HTTP to loopback always; to a numeric IP only with an explicit
allowPlaintext, because a tailnet is a defensible place to send
plaintext but it should be a stated decision; to a hostname never, since
a name resolves through something and a MagicDNS name is a hostname.
Absent config falls back to the previous loopback behaviour, so an
existing same-machine install is untouched.

PASTE. down and up were posted back to back. A zero-duration keystroke
is silently dropped by some apps - the events arrive and nothing acts on
them. hs.eventtap.keyStroke, the implementation being replaced, holds
200 ms (`local keyDelay = 200000`). Same tap, same flags; the gap was
the only difference. Also re-checks AXIsProcessTrusted at paste time,
because a grant can be revoked or invalidated by a rebuild while the
process runs, and every step before this one reports success either way
- so an untrusted process was indistinguishable from a dropped event.

MICROPHONE. requestAccess discarded its answer, so the agent continued
without knowing. An ungranted process is not refused the device: it
receives full-length buffers of zeros, indistinguishable from a quiet
room until you look at the peak. Now waits for the answer, pumping the
run loop rather than blocking on a semaphore whose queue might be main.

Also: HARK_CLIENT_CONFIG overrides the config path, mirroring the server
side. Neither homeDirectoryForCurrentUser nor NSHomeDirectory honours
$HOME on macOS, so `HOME=... hark agent` silently reads the real config
- which produced a round of "passing" policy checks that were all
reading the same live file.

Solves: hark #2 — integration findings on top of #12
Tests: his 48 SwiftPM tests still pass; policy matrix verified against the running server
Three defects found by using the agent rather than by reading it. Each
one reported success while broken, and none is visible to a unit test.

HOTKEY: the CGEventTap missed nearly every release. A tap sees raw key
events, so the chord is reconstructed from keycode + modifier flags on
that event - and the flags describe the instant it was generated.
Releasing Ctrl+Alt+Space almost always lifts a modifier at or before the
space bar, so the space key-UP arrives with the modifier bits clear and
a chord test applied to both edges matches the press and misses the
release. Measured with a bare session tap over 45s of ordinary use: 303
key-downs, 1 key-up. In the agent that left a capture that never ended,
and since beginCapture() guards on state == .idle every later press was
ignored - presenting as "the hotkey stopped working".

Switched to Carbon RegisterEventHotKey, which the design left to a
hardware spike. That spike is now run. Carbon delivers pressed and
released as distinct events, does not condition the release on modifier
state, consumes the chord so it does not also reach the focused app, and
needs no permission of its own - which settles the Phase 0 question of
whether a tap would additionally require Input Monitoring. It is also
what hs.hotkey used underneath.

REPEATS: Recorder is long-lived and neither start() nor stop() cleared
`samples` or `peak`, so every capture appended to all previous audio.
Capture 2 transcribed 1+2, capture 3 transcribed 1+2+3, and the WAV grew
without bound. Reset at the top of start().

MUTE UI: there was no presentation layer. alert() posted an NSWorkspace
notification named "HarkAlert" that nothing observed, and brief() was
`{ _ = message }` - so 401/415/400/503, transport failures, "heard
nothing" and "paste withheld (focus moved)" were all discarded. The menu
bar title flipping to a dot was the only feedback of any kind. Added an
Overlay (non-activating floating panel, so it never steals focus from
what you are dictating into) and wired all three paths to it, keeping
the menu bar item.

Also: status.json reported a hardcoded hotkey: "registered" whether or
not anything was bound, so it could never surface the failure above. It
now reflects Hotkey.isRegistered.

Verified on hardware: recording, transcription and paste all working,
repeats gone, indicator visible.

Solves: hark #2 — integration findings on top of #12
Tests: his 48 SwiftPM tests still pass
Two dictations into the same field pasted as
"one, two, three.one, two, three." with nothing between them.

Not a regression from the rewrite: the server sanitiser ends with
.strip(), which removes Whisper's own leading space. That is correct for
an API — it should return the transcript, not presentation whitespace,
and a consumer that is not a paste target should not inherit padding —
so the fix belongs on the client. The Lua client had the same behaviour.

Trailing rather than leading: a leading space would open an empty field
with whitespace, and there is no reliable way to read what sits
immediately before the cursor to decide between them. Trailing is
occasionally redundant and never wrong.

The logged length stays the transcript's, not the padded string's.

Solves: hark #2 — reported during hardware testing
Tests: his 48 SwiftPM tests still pass
@drycode
drycode merged commit bb5d3e8 into main Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant