[agent] Hardware-testing fixes for the Swift client - #15
Merged
Conversation
Three fixes on top of the Swift rewrite, all found by running it against a real deployment rather than by reading it. CLIENT ADDRESSING. AgentController hardcoded http://127.0.0.1:<harkPort>/dictate with the key from ~/.config/hark/key. That is the server's deployment config being used as the client's, and it cannot work for the two-machine setup the project exists to serve: the recording Mac has no server, no key file and nothing on loopback. It also fails on a SINGLE machine whose server binds a tailnet address - measured here, 127.0.0.1:8911 refuses while 100.64.66.46:8911 answers 200 - so loopback was wrong for every deployment on hand. Adds ClientConfig (~/.config/hark/client.json), and enforces the transport policy the design doc already specifies rather than assuming it: plain HTTP to loopback always; to a numeric IP only with an explicit allowPlaintext, because a tailnet is a defensible place to send plaintext but it should be a stated decision; to a hostname never, since a name resolves through something and a MagicDNS name is a hostname. Absent config falls back to the previous loopback behaviour, so an existing same-machine install is untouched. PASTE. down and up were posted back to back. A zero-duration keystroke is silently dropped by some apps - the events arrive and nothing acts on them. hs.eventtap.keyStroke, the implementation being replaced, holds 200 ms (`local keyDelay = 200000`). Same tap, same flags; the gap was the only difference. Also re-checks AXIsProcessTrusted at paste time, because a grant can be revoked or invalidated by a rebuild while the process runs, and every step before this one reports success either way - so an untrusted process was indistinguishable from a dropped event. MICROPHONE. requestAccess discarded its answer, so the agent continued without knowing. An ungranted process is not refused the device: it receives full-length buffers of zeros, indistinguishable from a quiet room until you look at the peak. Now waits for the answer, pumping the run loop rather than blocking on a semaphore whose queue might be main. Also: HARK_CLIENT_CONFIG overrides the config path, mirroring the server side. Neither homeDirectoryForCurrentUser nor NSHomeDirectory honours $HOME on macOS, so `HOME=... hark agent` silently reads the real config - which produced a round of "passing" policy checks that were all reading the same live file. Solves: hark #2 — integration findings on top of #12 Tests: his 48 SwiftPM tests still pass; policy matrix verified against the running server
Three defects found by using the agent rather than by reading it. Each
one reported success while broken, and none is visible to a unit test.
HOTKEY: the CGEventTap missed nearly every release. A tap sees raw key
events, so the chord is reconstructed from keycode + modifier flags on
that event - and the flags describe the instant it was generated.
Releasing Ctrl+Alt+Space almost always lifts a modifier at or before the
space bar, so the space key-UP arrives with the modifier bits clear and
a chord test applied to both edges matches the press and misses the
release. Measured with a bare session tap over 45s of ordinary use: 303
key-downs, 1 key-up. In the agent that left a capture that never ended,
and since beginCapture() guards on state == .idle every later press was
ignored - presenting as "the hotkey stopped working".
Switched to Carbon RegisterEventHotKey, which the design left to a
hardware spike. That spike is now run. Carbon delivers pressed and
released as distinct events, does not condition the release on modifier
state, consumes the chord so it does not also reach the focused app, and
needs no permission of its own - which settles the Phase 0 question of
whether a tap would additionally require Input Monitoring. It is also
what hs.hotkey used underneath.
REPEATS: Recorder is long-lived and neither start() nor stop() cleared
`samples` or `peak`, so every capture appended to all previous audio.
Capture 2 transcribed 1+2, capture 3 transcribed 1+2+3, and the WAV grew
without bound. Reset at the top of start().
MUTE UI: there was no presentation layer. alert() posted an NSWorkspace
notification named "HarkAlert" that nothing observed, and brief() was
`{ _ = message }` - so 401/415/400/503, transport failures, "heard
nothing" and "paste withheld (focus moved)" were all discarded. The menu
bar title flipping to a dot was the only feedback of any kind. Added an
Overlay (non-activating floating panel, so it never steals focus from
what you are dictating into) and wired all three paths to it, keeping
the menu bar item.
Also: status.json reported a hardcoded hotkey: "registered" whether or
not anything was bound, so it could never surface the failure above. It
now reflects Hotkey.isRegistered.
Verified on hardware: recording, transcription and paste all working,
repeats gone, indicator visible.
Solves: hark #2 — integration findings on top of #12
Tests: his 48 SwiftPM tests still pass
Two dictations into the same field pasted as "one, two, three.one, two, three." with nothing between them. Not a regression from the rewrite: the server sanitiser ends with .strip(), which removes Whisper's own leading space. That is correct for an API — it should return the transcript, not presentation whitespace, and a consumer that is not a paste target should not inherit padding — so the fix belongs on the client. The Lua client had the same behaviour. Trailing rather than leading: a leading space would open an empty field with whitespace, and there is no reliable way to read what sits immediately before the cursor to decide between them. Trailing is occasionally redundant and never wrong. The logged length stays the transcript's, not the padded string's. Solves: hark #2 — reported during hardware testing Tests: his 48 SwiftPM tests still pass
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four defects found by running #12 on real hardware — Studio, real dictation, against the existing Python server. It now works end to end. Every one of these passed both test suites, because they test what the code says and these are failures of what it reaches.
The hotkey misses almost every release
Hotkey.swiftreconstructed the chord from keycode + modifier flags and applied that test to both edges. The flags describe the instant the event was generated, and releasing Ctrl+Alt+Space almost always lifts a modifier at or before the space bar — so the space key-UP arrives with the modifier bits already clear.Measured with a bare session tap over 45 s of ordinary use:
In the agent that left a capture that never ended, and since
beginCapture()guards onstate == .idle, every later press was ignored — presenting as "the hotkey stopped working".This is the Phase 0 spike the design asked for, and it now has an answer. Switched to Carbon
RegisterEventHotKey: pressed and released arrive as distinct events, the release is not conditional on modifier state, it consumes the chord so it does not also reach the focused app, and it needs no permission of its own — so no Input Monitoring grant is required. Accessibility is still needed for the ⌘V, so it is one grant either way.The client could not reach any deployment
AgentControllerhardcodedhttp://127.0.0.1:<harkPort>/dictatewith the key from~/.config/hark/key— the server's deployment config used as the client's. That breaks the two-machine setup, and it also breaks a single machine whose server binds a tailnet address: measured here,127.0.0.1:8911refuses while100.64.66.46:8911answers 200.Adds
ClientConfig(~/.config/hark/client.json) and enforces the transport policy the design already specifies rather than assuming it — plain HTTP to loopback always; to a numeric IP only with an explicitallowPlaintext, because a tailnet is a defensible place for plaintext but it should be a stated decision; to a hostname never, since a MagicDNS name is a hostname. Absent config falls back to the previous behaviour.Recorderaccumulated across capturessamplesandpeakare instance properties on a long-livedRecorder, and neitherstart()norstop()cleared them. Capture 2 transcribed 1+2, capture 3 transcribed 1+2+3, and the WAV grew without bound. Presented as the transcript repeating the previous dictation.There was no presentation layer
The whole diagnostic surface — 401, 415, 400, 503, transport failure, "heard nothing", "paste withheld (focus moved)" — was discarded. Added
Overlay, a non-activating floating panel so it never pulls focus from what you are dictating into, and wired all three paths to it. The menu bar item stays.Smaller
hs.eventtap.keyStrokeholds 200 ms (local keyDelay = 200000); same tap, same flags, the gap was the only difference.AXIsProcessTrustedre-checked at paste time, and on failure the user is told the transcript is on the clipboard. Everything before that point reports success either way, so an untrusted process was indistinguishable from a dropped event.requestAccesswaits for its answer instead of discarding it. An ungranted process is not refused the device — it receives full-length buffers of zeros.status.jsonno longer reports a constant.hotkeywas a hardcoded"registered"regardless of whether anything was bound, which is why the dead tap looked healthy from outside.HARK_CLIENT_CONFIGoverrides the config path, mirroring the server'sHARK_CONFIG. NeitherhomeDirectoryForCurrentUsernorNSHomeDirectory()honours$HOMEon macOS, soHOME=... hark agentsilently reads the real config — which produced a round of "passing" policy checks all reading the same live file.Verification
Studio: recording, transcription and paste all working; repeats gone; indicator visible; consecutive dictations separated. The 48 SwiftPM tests pass at every commit.
Not covered here, and still open: wiring
install-client.shontohark agent, deleting the Lua client, and whetherhark servereplaces the Python server — nothing here tested the server rewrite.