Skip to content

firmware, tools: the audio path reworked — /audio-in as a console page, partial saves, an extension that holds #475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
engmung merged 1 commit into from
Oct 4, 2026
+8,871 −6,742
Merged

firmware, tools: the audio path reworked — /audio-in as a console page, partial saves, an extension that holds #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/console-sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,11 @@ jobs:
python-version: "3.x"
- name: Headers match console/*.html
run: python firmware/toolchain/console_pages.py check
# audio-in.html is itself generated, from the browser extension's mapping
# editor. The step above proves the header matches the page; this proves
# the page matches the editor it was assembled from.
- name: audio-in.html matches the extension's editor
# audio-in.html is itself generated: from the browser extension's mapping
# editor and the console/_audio_in* files. The step above proves the
# header matches the page; this proves the page is what those sources
# assemble to, and that it is inside its size budget (BUDGET in the
# script: the stamped page, gzip -9, as the panel sends it). The step
# above reports both too; this one keeps them a line of their own.
- name: audio-in.html matches its sources and is inside its size budget
run: python firmware/toolchain/build_audio_in_page.py --check
185 changes: 136 additions & 49 deletions AUDIO_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,24 +41,42 @@ locally, and streams four knob lanes to the panel over your LAN.
**Install:** it's not on the Web Store — load it straight from this repo.
`chrome://extensions` → enable *Developer mode* → *Load unpacked* → pick
[`tools/patternflow-audio-extension/`](tools/patternflow-audio-extension/).
(Details and screenshots in its [README](tools/patternflow-audio-extension/README.md).)

**Use:** open a tab with sound → click the extension → **Start** (it captures
the active tab) → enter your panel's address (`patternflow.local` or its IP)
→ the knobs move with the music. (Make sure **Audio-React (AUD)** is turned on
at the top of the console's **Audio** page or via the panel's NETWORK screen.)
After pulling a newer copy, press *Reload* on its card there. Reload keeps the
mapping you already had, so the starting point described below is a new
install's. (Every control is described in its
[README](tools/patternflow-audio-extension/README.md), and so is how to get
that starting point on an older mapping.)

**Use:** open a tab with sound → click the extension → enter your panel's
address (`patternflow.local` or its IP) → **Start** (it captures the active
tab). The corner of the popup says **Live** once the panel is being driven,
and the knobs move with the music. If it says **Aud off**, the panel's
Audio-React switch is off and the panel is ignoring what arrives: press
**Turn on** in the popup. (The same switch is the `AUD` row of the panel's
NETWORK screen and the *Browser extension and phone app* row on the console's
**Audio** page.) If it says **Error**, the line under the address says what
is wrong: no panel at that address, or a panel that does not run the Audio
edition.

While the tab is silent or paused the extension holds the knobs, each where
its band rests, until you press **Stop**. A knob you turn by hand during a
pause goes back there about five seconds after you let go.

**Then open the editor** (*Mapping editor ↗* in the popup) — this is where it gets
good. Each of the four knobs is a **box drawn on the live spectrum**: the
box's width is the frequencies it listens to, its height the loudness window
it maps. Drag a box over the bass and knob 1 becomes a bass knob. Inside
each box a **response curve** (presets or a hand-dragged bezier) shapes how
it moves, an **output range** decides how far, and **attack/damping** set
the ballistics — fast rise, slow fall. Boxes start in **auto**, breathing
with the music; grab an edge to take manual control. A **preview** popup
shows how a test signal would ride your settings before you commit.

Boxes map to knobs 1:1 and that's fixed on purpose — box 2 *is* knob 2.
good. Each of the four knobs, **K1** to **K4**, is a **box drawn on the live
spectrum**: the box's width is the frequencies it listens to, its height the
loudness window it maps. Drag K1's box over the bass and knob 1 becomes a
bass knob. Pick a knob with the tabs under the plot, and its card has a
**response curve** (presets, or a bezier you drag by its two handles) that
shapes how it moves and an **output range** that decides how far; **Attack**
and **Damping**, above the plot, set the ballistics for all four — fast rise,
slow fall. A new install starts in **Auto range** with the Smooth curve: the
boxes follow the loudness of the music. Drag a box's top or bottom edge to
take the windows over by hand; switching Auto range off goes back to the
windows you set. **Preview** puts a scope where the plot is and shows how a
test signal would ride the selected knob's settings while you change them.

Boxes map to knobs 1:1 and that's fixed on purpose — K2 *is* knob 2.

## The on-board microphone

Expand Down Expand Up @@ -125,22 +143,37 @@ Stick the mic wherever sound reaches it. Done.

### Turn it on

Console → **Audio** page (`/audio-in`) → flip **Microphone** on. (**Audio-React (AUD)** beside it is the other input, the extension and the phone app.) That's the
whole switch: on means listening and driving the knobs, off releases the
hardware completely. The **gain** slider (1–16, default 8) is there if your
room runs quiet — PDM mics on this chip are famously low-amplitude, and gain
is the official answer.

The same box editor from the extension lives on this page, dark-themed —
same boxes, same curves. Mic settings are stored **on the panel** and
survive reboots and firmware updates; *Reset all* puts everything back.
Console → **Audio** page (`/audio-in`) → under **Sources**, flip
**Microphone** on. That's the whole switch: on means listening and driving
the knobs, off releases the hardware completely. The chip beside it reads
`listening` once samples are arriving, and the line at the top of the page
says who has the knobs. The **Input gain** slider (1–16, default 8) is there
if your room runs quiet — PDM mics on this chip are famously low-amplitude,
and gain is the official answer. (The row under it, *Browser extension and
phone app*, is the other input: the same switch as `AUD` on the panel's
NETWORK screen.)

The same box editor from the extension lives on this page, under
**Microphone mapping**, in the console's own look (light, or dark from the
toggle in the header) — same boxes K1 to K4, same curves, same Preview. Its
level axis is the microphone's, about −45 to +2 dB, and its frequencies stop
at 8 kHz. Changes save to the panel as you make them; mic settings are stored
**on the panel** and survive reboots and firmware updates. *Reset K1* puts the
selected knob back to the firmware's default. *Reset mapping* asks to be
pressed twice and then resets all four knobs, their curves, attack, damping
and input gain; it leaves the Microphone switch and Auto range as they are.

### If something's off

- **Silence, but everything looks healthy** → SEL isn't grounded. This is
the classic one.
- **Mic not detected** → the firmware notices an absent/miswired mic and
falls back to a synthetic test source, and says so on `/api/status`.
the classic one, and nothing can detect it: a mic with SEL tied high reads
as a quiet room.
- **The chip says `not detected`** → the firmware sees nothing on the data
pin: no microphone, or no DAT lead (DAT goes to RX). Nothing is driven.
- **The chip says `not answering`** → the microphone was there and has
stopped returning samples, or its driver did not start. The knobs are left
alone in this state too. `source` in `GET /api/audio-in` and in
`/api/status` has the firmware's own word for each of these.
- **The top band barely moves at normal volume** → physics, not a fault:
at room levels the highest octaves carry almost no energy. The auto range
is tuned around this.
Expand Down Expand Up @@ -203,9 +236,17 @@ microphone on any lane it's driving. The mic takes whatever's left. So you
can leave the mic on and still grab a knob whenever you want — it comes back
to the music a few seconds after you let go.

The extension keeps driving its knobs through a silent or paused tab, so with
both on, the microphone does not take over during a pause. Press **Stop** in
the extension to give the knobs to the room, or **Mute** a knob in the
extension's editor to give the microphone just that one. The phone app does
not hold yet: half a second into a pause its knobs go back to the encoders,
or to the microphone if that is on.

One thing to know: the **extension keeps its mapping in the browser**, while
the **mic and the phone app share the config stored on the panel**. Same
editor everywhere, two homes for the settings.
editor everywhere, two homes for the settings. The top of the console's Audio
page says which source has the knobs right now.

---

Expand All @@ -223,8 +264,8 @@ You were pointed here to work on Patternflow's audio path. The map:
| `feature_audio_in.h` | Descriptor, the sampling/analysis task, where levels enter the knob pipeline. |
| `core_audio_pdm.h` | PDM mic driver (GPIO 43/44). Its header comment is the authoritative wiring + driver-choice record. |
| `core_audio_fft.h` | FFT and spectrum buckets. |
| `core_audio_in_map.h` | Bands, curve LUTs (33-point, interpolate-only), attack/damping glide, auto-range, NVS persistence. |
| `core_audio_in_http.h` | `/audio-in` (editor page), `GET/POST /api/audio-in` (whole config), `POST /api/audio-in/reset`, the `?levels=1` live poll, and the frame endpoint the phone posts spectra to. |
| `core_audio_in_map.h` | Bands, curve LUTs (33-point, interpolate-only), attack/damping glide, auto-range, the defaults a reset restores, NVS persistence. Manual mode maps linearly in amplitude between a band's two edges; the page's dB axis is a conversion at the page's edge. |
| `core_audio_in_http.h` | `/audio-in` (editor page), `GET /api/audio-in` (the configuration, read once), `POST /api/audio-in` (any subset: a request changes only the fields it carries), `POST /api/audio-in/reset` (everything, or one band with `band=N`), the `?levels=1` live poll, and the frame endpoint the phone posts spectra to. The contract is [`docs/rest-api.md`](docs/rest-api.md#microphone-audio-edition); the phone app is a second client of it, so fields are added, never renamed. |
| `audio_in_index.h` | **Generated** — never edit (chain below). |

**Firmware — MIDI** ([`firmware/patternflow/features/midi/`](firmware/patternflow/features/midi/)):
Expand All @@ -237,37 +278,83 @@ AppleMIDI library. Contract: [`docs/midi-spec.md`](docs/midi-spec.md).
([`firmware/patternflow/features/audio/`](firmware/patternflow/features/audio/)):
`core_audio_ws.h` is a WebSocket server on port 81. The wire contract is
[`docs/audio-ws-spec.md`](docs/audio-ws-spec.md) — clients build against the
spec, not the source. Probe `GET /api/status` for `"audio"` in `caps`.
spec, not the source. Probe `GET /api/status` for `"audio"` in `caps`. A lane
is released 500 ms after the last message that set it, so a sender that means
to hold a value resends it; the spec says which senders in this tree do.
`feature_audio.h` has the Audio-React switch and its two routes
(`GET`/`POST /api/audio`).

**The editor is authored once and baked down.** Edit at the top, regenerate
downward; CI (`console-sync.yml`) fails on drift:
**The editor is authored once and assembled for the panel.** The panel's
`/audio-in` page is the extension's editor inside the panel's own frame, and
one command builds it; CI (`console-sync.yml`) fails when the generated files
are not what the sources assemble to, or the page is over its size budget:

```
tools/patternflow-audio-extension/editor.{js,html,css} ← edit here
→ python firmware/toolchain/build_audio_in_page.py (dark theme + device adapter)
→ firmware/patternflow/console/audio-in.html
→ python firmware/toolchain/console_pages.py build
→ firmware/patternflow/features/audio_in/audio_in_index.h
tools/patternflow-audio-extension/
editor.html the editor's body (between its EDITOR BODY lines)
editor.css, mapping.js down to their EXTENSION ONLY line
editor.js all of it
firmware/patternflow/console/
_audio_in_bar.html the page around the editor: heading, Sources, footer
_audio_in.css the console's look
_audio_in_adapter.js window.PFAdapter over /api/audio-in
↓ python firmware/toolchain/console_pages.py build
firmware/patternflow/console/audio-in.html generated
firmware/patternflow/features/audio_in/audio_in_index.h generated
```

Preview without building: `python firmware/toolchain/console_serve.py`
assembles `/audio-in` from those sources on every request and answers its API
the way the firmware does; `/mock?audio=music`, `silence`, `off`, `fresh`,
`nomic`, `stalled`, `phone`, `ext` and `noread` pick what the panel hears
([`console/README.md`](firmware/patternflow/console/README.md#the-page-that-is-assembled-audio-in)).
The page has a budget, `BUDGET` in `build_audio_in_page.py`: 17,500 bytes
gzipped as the panel sends it. It is the heaviest page in the console and
every byte crosses the panel's Wi-Fi, so whole-line comments and indentation
are dropped as it is assembled and what only the extension needs stays below
the `EXTENSION ONLY` lines.

The seam between the editor and whatever feeds it is `window.PFAdapter`. The
contract is the comment at the top of `editor.js`: the source owns its
limits, its level axis, its defaults and where a level sits in a window, and
the editor draws what it is told. The page's adapter follows the console's
rules for a one-connection server (`console/README.md`, "Writing a page"):
every request rides `PF.poll`/`PF.status`, nothing is saved until the
panel's mapping has been read, and a save sends only the fields that changed.

**Chrome extension**
([`tools/patternflow-audio-extension/`](tools/patternflow-audio-extension/)):
`editor.js` is the single shared editor module (adapter-isolated —
`editor-adapter.js` binds chrome storage/frames, and includes a synthetic
demo mode when opened outside the extension). `offscreen.js` does FFT →
band mapping → glide → WS send. `popup.*` is the capture console;
`background.js` owns capture lifecycle. Extension config lives in
`chrome.storage`, not on the panel.
`mapping.js` is the mapping model, one object (`PFMap`): defaults, the
curves, level → knob. `editor.js` is the single shared editor module, and
`editor-adapter.js` is its extension adapter (chrome storage, the capture's
frames), the editor's source bar, and a synthetic demo source when the page
is opened outside the extension. `offscreen.js` holds the capture: FFT →
`PFMap` → glide → WS send, the resend that holds a steady value, and the
state every page prints (`phase`). `popup.*` is the capture console and owns
the panel's address; `background.js` relays between the pages and the
offscreen document. The mapping lives in `chrome.storage`
(`patternflowAudioConfig`; the address under `patternflowAudioHost`), not on
the panel. The extension keeps its cream skin; only the panel's page wears
the console's.

**Android app**
([`tools/patternflow-audio-android/`](tools/patternflow-audio-android/)):
`Analyzer.kt` mirrors `offscreen.js`'s math; `DeviceLink.kt` speaks the WS
`Analyzer.kt` mirrors the extension's analysis and mapping (`offscreen.js`
and `mapping.js`); `DeviceLink.kt` speaks the WS
contract + syncs panel config over `/api/audio-in` + posts monitor frames;
`CaptureService.kt` is the playback-capture foreground service;
`EditorActivity.kt` is a WebView onto the panel's `/audio-in`. Build recipe
in its README.
in its README. Three things it carries by hand and that a change elsewhere
can leave behind: it sends lanes only when they change, so it does not hold a
steady value (the spec has the fix); it has its own copy of the three bezier
presets, by the names the page stores (`p:smooth`, `p:sharp`, `p:fall`); and
its WebView is set up without the part that shows a `window.confirm` dialog,
which is why the page's Reset mapping is a button pressed twice. Open the
app's editor once before releasing a change to `/audio-in`.

Rules of the road: firmware changes follow [FEATURE_GUIDE.md](FEATURE_GUIDE.md)
(hooks, checkers, `build.sh all`); protocol changes update the spec in the
same PR; editor changes regenerate the chain above and keep the extension
and console page identical — they are the same file, which is the point.
same PR; API changes extend [`docs/rest-api.md`](docs/rest-api.md) and never
rename what is there; editor changes run `console_pages.py build` so the
panel's page follows — the editor's body is one file for both surfaces, which
is the point.
Loading
Loading