From 639ef66f235958c4fab995a7fa8b4ffe72f37013 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Wed, 19 Aug 2026 14:46:31 +0200 Subject: [PATCH 1/4] docs: show the three-channel protocol split Input, output and prompt control are separate channels, which the prose buried in a bullet list: frames go up a WebSocket, come back down a plain HTTP response, and the prompt touches neither. A table names the channels and a sequence diagram shows the orchestrator only forwarding once a session is reserved. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 45 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1c70eb4..518df57 100644 --- a/README.md +++ b/README.md @@ -33,12 +33,45 @@ All the Livepeer integration therefore lives in [client.py](client.py). Grep `# 2. `ws_connect` — from here on it is the app's own protocol, over that url. 3. `stop_runner_session` — release the session, which settles payment on-chain. -The server's protocol, for reference: - -- `GET /api/queue` — liveness (the runner's `health_url`). -- `WS /api/ws/{uuid}` — input: control messages plus JPEG frames. -- `GET /api/stream/{uuid}` — output: MJPEG (`multipart/x-mixed-replace`). Opening it builds the pipeline and drives the per-frame pump. -- `POST /api/blending` — set the prompt. +### The protocol split + +Input, output and prompt control are **three separate channels**, which is easy to forget: frames go up a WebSocket, frames come back down a plain HTTP response, and the prompt is a third call that touches neither. + +| Channel | Endpoint | Carries | +| -------- | ------------------------ | ----------------------------------------------------- | +| Input | `WS /api/ws/{uuid}` | control messages plus input JPEG frames | +| Output | `GET /api/stream/{uuid}` | MJPEG out (`multipart/x-mixed-replace`) | +| Prompt | `POST /api/blending` | prompt updates, at any time | +| Liveness | `GET /api/queue` | the runner's `health_url`, polled by the orchestrator | + +Two consequences worth knowing. **Opening the output stream is what builds the pipeline** and drives the per-frame pump, so nothing happens until you `GET` it, and the first open compiles TensorRT engines. And the same `{uuid}` ties the two halves together, so input and output are one session in two directions, not a request and a response. + +```mermaid +sequenceDiagram + participant C as client.py + participant O as orchestrator + participant A as StreamDiffusion (port 7860) + + Note over O,A: static registration: runners.json names the app,
so the container needs no SDK + loop every few seconds + O->>A: GET /api/queue + end + + C->>O: reserve_session("livepeer/streamdiffusion") + O-->>C: proxied app_url, meter starts + + Note over C,A: from here it is the app's own protocol;
the orchestrator only forwards + C->>O: POST /api/blending + O->>A: POST /api/blending + C->>O: WS /api/ws/{uuid} + JPEG frames + O->>A: WS /api/ws/{uuid} + C->>O: GET /api/stream/{uuid} + O->>A: GET /api/stream/{uuid} + A-->>O: MJPEG + O-->>C: MJPEG + + C->>O: stop_runner_session, settles on-chain +``` The client reads MJPEG on stdin and writes MJPEG on stdout, so ffmpeg does capture and playback and the client stays a pipe stage. Input frames are kept drop-to-latest: a slow diffuser falls behind rather than building a backlog. From cb4635171f0833001e70541af43194538a781d7f Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Wed, 19 Aug 2026 15:17:58 +0200 Subject: [PATCH 2/4] fix(docs): drop the punctuation mermaid parses as syntax A semicolon ends a statement in mermaid, so the note text after it was read as a new one and the diagram refused to render. Line breaks and quotes go too, since none of them earn their risk in a note. Verified by rendering locally rather than by pushing and looking. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 518df57..48c74ce 100644 --- a/README.md +++ b/README.md @@ -52,18 +52,18 @@ sequenceDiagram participant O as orchestrator participant A as StreamDiffusion (port 7860) - Note over O,A: static registration: runners.json names the app,
so the container needs no SDK + Note over O,A: static registration, no SDK in the container loop every few seconds O->>A: GET /api/queue end - C->>O: reserve_session("livepeer/streamdiffusion") + C->>O: reserve_session(livepeer/streamdiffusion) O-->>C: proxied app_url, meter starts - Note over C,A: from here it is the app's own protocol;
the orchestrator only forwards + Note over C,A: from here it is the app's own protocol, the orchestrator only forwards C->>O: POST /api/blending O->>A: POST /api/blending - C->>O: WS /api/ws/{uuid} + JPEG frames + C->>O: WS /api/ws/{uuid} plus JPEG frames O->>A: WS /api/ws/{uuid} C->>O: GET /api/stream/{uuid} O->>A: GET /api/stream/{uuid} From 8f141c4315b72b691369d20e30541918aa7cdf86 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Wed, 19 Aug 2026 17:19:45 +0200 Subject: [PATCH 3/4] docs: name both transports in the summary table The row read "WebSocket + MJPEG", pairing a transport with a payload format and never mentioning HTTP, though two of the four endpoints are plain HTTP. Name what each carries, matching how the examples write it. Registration gains the health poll, which is half of what static means. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 48c74ce..cd0c01a 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,14 @@ ffmpeg -f v4l2 -input_format mjpeg -framerate 30 -video_size 640x480 -i /dev/vid | ffplay -f mjpeg -fflags nobuffer -flags low_delay -i - ``` -| | | -| ------------ | ------------------------------------------ | -| App id | `livepeer/streamdiffusion` | -| Runner mode | persistent (held-open session) | -| Registration | static (`runners.json`) | -| Transport | WebSocket + MJPEG (the app's own protocol) | -| Pricing | hour (metered per second while held) | -| Port | 7860 (the StreamDiffusion server) | +| | | +| ------------ | --------------------------------------------- | +| App id | `livepeer/streamdiffusion` | +| Runner mode | persistent (held-open session) | +| Registration | static (`runners.json` + health poll) | +| Transport | WebSocket + HTTP (JPEG frames up, MJPEG down) | +| Pricing | hour (metered per second while held) | +| Port | 7860 (the StreamDiffusion server) | **Requires an NVIDIA GPU.** You also need **Docker** (with the [NVIDIA container toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)), [**uv**](https://docs.astral.sh/uv/), and **ffmpeg** for capture and playback. From 3bb78e8b86f82feb485d54f3c1b7abbf54083d1b Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Wed, 19 Aug 2026 17:24:52 +0200 Subject: [PATCH 4/4] docs: give the offchain run a source that needs no webcam Every path shown needed a camera or a clip, so a machine with neither could not tell a broken stack from a missing /dev/video0. lavfi synthesises a source, which is also the quickest end-to-end check. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index cd0c01a..66f4687 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,14 @@ ffmpeg -f v4l2 -input_format mjpeg -framerate 30 -video_size 640x480 -i /dev/vid | ffplay -f mjpeg -fflags nobuffer -flags low_delay -i - ``` +**No webcam?** ffmpeg synthesises one, which is also the fastest way to prove the stack works: + +```sh +ffmpeg -f lavfi -i testsrc2=size=512x512:rate=15 -f image2pipe -c:v mjpeg -q:v 5 - \ + | uv run client.py --prompt 'van Gogh oil painting, vivid colors' \ + | ffplay -f mjpeg -fflags nobuffer -flags low_delay -i - +``` + Device numbers vary, so confirm your camera node first (`v4l2-ctl --list-devices`, `ffplay -f v4l2 -i /dev/videoN`). macOS: `-f avfoundation -i 0`. Windows: `-f dshow -i video=""`. Any MJPEG source works, so a file restyles too — but pace it with `-re`, or ffmpeg decodes the whole file at once and the run ends the moment the pipe closes: