Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Builds the docs site the way Read the Docs builds it from .readthedocs.yaml.
# -W turns a warning into a failure, as fail_on_warning does there.
name: docs

on:
push:
branches: [main]
paths:
- 'docs/**'
- '.readthedocs.yaml'
- '.github/workflows/docs.yml'
pull_request:
paths:
- 'docs/**'
- '.readthedocs.yaml'
- '.github/workflows/docs.yml'
workflow_dispatch:

permissions:
contents: read

jobs:
docs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: '3.13'
- name: Install docs requirements
run: pip install -r docs/requirements.txt
- name: Build docs
run: sphinx-build -W --keep-going -b html docs _site
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,5 @@
/.worktrees/
__pycache__/
/.micropython-build.lock
/_site/
/.venv-docs/
14 changes: 14 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
version: 2

build:
os: ubuntu-24.04
tools:
python: "3.13"

sphinx:
configuration: docs/conf.py
fail_on_warning: true

python:
install:
- requirements: docs/requirements.txt
119 changes: 119 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Contributing to micropython-pydevices

This repository holds our patches to one pinned MicroPython release, our
variants, the C modules that live only here, and the command that builds it all.
A documentation fix, a provenance update, or a narrowly scoped variant or module
fix is a good first contribution.

## Where things are

| Path | What's there |
|---|---|
| `build_mp.py` | The build command. |
| `UPSTREAM` | The MicroPython commit the patches apply to. |
| `modules.lock`, `deps.lock` | What gets fetched when it's missing, lvgl-bindings included. |
| `modules/` | The module manifest, `all/`, `tflite/` and `vision/` (esp-vision's), the modules linked or cloned, and the C modules that live only here: `audioif`, `cameraif`, `castif`, `displayif`, `h264enc`, `jpegio`, `mdns`, `pngio`, `ppa`, `tsmux`, `usbhost`. Each has its own README and tests. |
| `modules/.lvgl-bindings` | The lvgl-bindings lvgl-micropython compiles: the sibling checkout, linked, or `modules.lock`'s commit, fetched. |
| `patches/` | The patch series and `apply_patches.py`. |
| `variants/`, `boards/` | Our variants and CircuitPython boards. |
| `scripts/` | `exercise_build.sh`, `release_firmware.py`, `stage_firmware_page.py`, `serve_workbench.sh`, `freeze_list.py`, `update_provenance.py`. |
| `.devcontainer/` | One dev container per port. |
| `docs/` | The documentation site. |

The reference pages in [the docs](https://github.com/PyDevices/micropython-pydevices/blob/main/docs/index.md)
describe what each of these does.

## Rules

- Upstream is pinned. Moving `UPSTREAM` re-validates every patch.
- Never hand-edit a patched tree. Change the patch and prepare again.
- Patches are justified in their headers.
- `build_mp.py` names no version and no setting. Pins move in the lock files, and
esp32 settings live in sdkconfig fragments it finds by convention.
- Publishing binaries is a separate decision from this source repository.

## Run the tests

`build_mp.py`'s unit tests decide things before any build, such as when a build
directory has to be wiped. They need no toolchain:

```bash
python3 -m unittest test_build_mp test_freeze_list -v
python3 -m unittest test_provenance -v
python3 -m unittest test_release_firmware test_stage_firmware_page -v
```

Each module has its own workflow under `.github/workflows/`, named for it.

## Change a patch

Check a patch change with `patches/apply_patches.py --check` before you commit
it. After changing a patch or moving `UPSTREAM`, rewrite the record, because CI
fails on a stale file:

```bash
python3 scripts/update_provenance.py
```

## Exercise a build from a bare clone

`scripts/exercise_build.sh` builds every desktop port and two esp32 boards from
a bare clone, checks the failure paths, and writes a report of what should match
on any machine. It needs no board. For a report on builds you already have, name
their folders:

```bash
scripts/exercise_build.sh --report builds/rp2/RPI_PICO2/default ...
```

## Firmware sets

A firmware set is every board's flashable file, built from one commit of this
repository by the **Release firmware** workflow
([`release-firmware.yml`](.github/workflows/release-firmware.yml); the cells are
in [`scripts/release_firmware.py`](scripts/release_firmware.py), and
`release_firmware.py cells` lists them). Each cell is one `build_mp.py` call; the
`assemble` job checks every file (checksums, the ESP image magic, the app image
inside the flash image, UF2 blocks) and writes `firmware.json`, which records
every file's sha256 and size, the flash offset, and the pins the set was built
from.

### Build a set

Actions > Release firmware > Run workflow. A pull request that touches the
recipe, and a run every Monday, build the same cells and keep the files as a run
artifact for 14 days. They publish nothing.

### Put a set on a draft Release

Tick **draft_release** and give a **tag**. After `assemble`, the `draft` job
attests every file (`gh attestation verify <file> --repo
PyDevices/micropython-pydevices`) and creates a *draft* GitHub Release holding
the flash images, the `.elf` and `.map` of each build, and `firmware.json`. A
draft is seen only by people who can push to the repository, and GitHub makes no
tag until someone publishes it.

The tag is `v<MicroPython version>-<YYYYMMDD>`. A set built from a preview
MicroPython or CircuitPython only takes a tag that starts with `staged-`, so a
preview set can't be published by accident.

### The download page

[`scripts/stage_firmware_page.py`](scripts/stage_firmware_page.py) lays a set out
for the site's `firmware/` folder: `firmware.json`, the flash images the page
offers and the browser flasher fetches, an ESP Web Tools manifest per ESP32
build, and the desktop interpreters' `interpreters.json` from the newest
pydevices final that has one.

```bash
python3 scripts/stage_firmware_page.py <downloaded firmware artifact> <site checkout>/firmware \
--interpreters-from-release PyDevices/pydevices
```

## Build the docs

```bash
python3 -m venv .venv-docs
.venv-docs/bin/pip install -r docs/requirements.txt
.venv-docs/bin/sphinx-build -W --keep-going -b html docs _site
```
144 changes: 30 additions & 114 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,135 +1,51 @@
# micropython-pydevices

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/PyDevices/micropython-pydevices)

Build MicroPython firmware with PyDevices modules in it, for any port, with
one command:
micropython-pydevices builds firmware: MicroPython-compatible and
CircuitPython-compatible interpreters with the PyDevices C modules compiled in,
for boards, desktops and the browser, with one command.

```bash
./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI \
--flash 16MB --modules displayif,pygraphics,/home/you/earful
curl -fsSL https://pydevices.github.io/install.sh | sh
```

Boards are upstream's own, with a variant of ours where a board needs one; the
[newcomer's guide](docs/newcomers.md#boards-we-build-for) has the command
for each board we use. You name the modules you want: ours
by short name, anything else by its path, or `all`. Leave out `--port` and it
asks you for port, board and variant. Output lands in
`builds/<port>/[<board>/]<variant>/` (with no `--variant`, that last
directory is `default` on esp32 and rp2, `standard` on the desktop ports),
beside a record of every module's commit. [Flash it](docs/newcomers.md#flash-it)
says where `firmware.bin` goes on the board.
That clones this repository into `micropython-pydevices` in the current
directory (Linux, WSL, or Git Bash on Windows) and tells you which packages your
machine still needs. Or clone it yourself: `git clone
https://github.com/PyDevices/micropython-pydevices.git`. The build fetches
everything else.

Don't have it yet? This clones it into `micropython-pydevices` in the current
directory and checks your tools (Linux, WSL, or Git Bash on Windows):
Build MicroPython for your desktop with the pygraphics module inside:

```bash
curl -fsSL https://pydevices.github.io/install.sh | sh
./build_mp.py --port unix --modules pygraphics
```

It names the Debian packages each port still needs, with the `apt-get` line
that installs them; [What each port needs](docs/newcomers.md#what-each-port-needs)
is the same list.
Save this as `hello.py`:

Or build in a codespace, with nothing to install: the badge above opens one,
and [In a codespace](docs/newcomers.md#in-a-codespace) says how to pick your
port and get the file out.
```python
import sys
import pygraphics

On Windows you can build the windows port natively from PowerShell, with
`python build_mp.py --port windows --variant pydevices --modules all`, once
MSYS2 is installed: [On Windows](docs/newcomers.md#on-windows) has the
steps. Everything else builds from WSL. A Linux or WSL machine also
cross-builds that same `micropython.exe`, with the same command and the
MinGW packages from the list above; the SDL2 it links against is fetched.

The same command builds OmniPython CP (CircuitPython-compatible) firmware, which
is CircuitPython with our C modules compiled in, for CircuitPython's own
ports and boards:
fb = pygraphics.FrameBuffer(bytearray(160 * 80 * 2), 160, 80, pygraphics.RGB565)
area = fb.round_rect(8, 8, 64, 40, 8, 0xF800, f=True)
print(sys.implementation.name, pygraphics.implementation(), area)
```

```bash
./build_mp.py --interpreter circuitpython --port raspberrypi \
--board adafruit_feather_rp2040 --modules pygraphics
builds/unix/standard/micropython hello.py
```

It takes C modules only, so far; the
[build plan](docs/build-plan.md#circuitpython-compatible-builds) has how it
works.

New here? The [newcomer's guide](docs/newcomers.md) explains the model; the
[build plan](docs/build-plan.md) has the layout, the rules and the traps.

## What's in here

- [`build_mp.py`](build_mp.py) — the one command. It fills in `micropython/`, `modules/`
and `deps/` if they're missing, prepares the MicroPython tree, and runs
the port's own `make`.
- [`UPSTREAM`](UPSTREAM) — the MicroPython everything applies to (main at `b70ae1aca657`, v1.30.0-preview-160).
- [`modules.lock`](modules.lock), [`deps.lock`](deps.lock) — the commits and versions `build_mp.py`
fetches when a module or a toolchain isn't there. `deps.lock` also pins
CircuitPython, for `--interpreter circuitpython`. `modules.lock` also pins
lvgl-bindings, the generated binding and LVGL that lvgl-micropython
compiles: a build with lvgl-micropython fetches it into
`modules/.lvgl-bindings`. Beside sibling checkouts, these are links to those
checkouts instead.
- [`modules/`](modules/) — [`manifest.py`](modules/manifest.py) (the manifest every build freezes), `all/`
(every module), `tflite/` and `vision/` (esp-vision's modules, from the pin
in `.esp-vision/`; see [docs/esp-vision.md](docs/esp-vision.md)), the modules
themselves, linked or cloned, and the C modules that live only here:
- [`audioif/`](modules/audioif/): the audio hardware under the audio pump, with
CircuitPython's `audiobusio.I2SOut` and `audioi2sin.I2SIn` on top.
- [`cameraif/`](modules/cameraif/): a MIPI-CSI camera as a MicroPython object, on the ESP32-P4.
- [`castif/`](modules/castif/): casts an ESP32-P4's screen and sound to a TV over Wi-Fi
Display, or serves them as live HLS.
- [`displayif/`](modules/displayif/): the display buses and panels under `displaydev`
(SPI, I2C, i80, QSPI, dot-clock RGB, MIPI-DSI, PicoDVI, RGB matrix, SDL2); the
dot-clock, MIPI-DSI, PicoDVI and RGB matrix modules keep CircuitPython's names.
- [`h264enc/`](modules/h264enc/): RGB565 frames to H.264 on the ESP32-P4's hardware encoder.
- [`jpegio/`](modules/jpegio/): JPEG decode and encode, with CircuitPython's `jpegio` API.
- [`mdns/`](modules/mdns/): find and advertise services on the local network, with
CircuitPython's `mdns` API (esp32, rp2, unix, windows).
- [`pngio/`](modules/pngio/): PNG encode and decode, RGB565 in and out.
- [`ppa/`](modules/ppa/): scale, rotate, convert and fill pixel buffers, on the
ESP32-P4's Pixel Processing Accelerator or in portable C elsewhere.
- [`tsmux/`](modules/tsmux/): MPEG-TS and HLS segments from H.264.
- [`usbhost/`](modules/usbhost/): the USB host port and isochronous streams that
`usb.core` and the `usb.host.*` packages are built on (ESP32-S2, S3 and P4).
- [`patches/micropython/`](patches/micropython/) — our patch series (`0001-…` to `0042-…`; there
is no `0026`, whose fix moved into `patches/tinyusb/`), each justified
in its own header; [`patches/apply_patches.py`](patches/apply_patches.py) applies it, plus
each module's own MicroPython patches, once, as one local commit.
- [`patches/tinyusb/`](patches/tinyusb/) — patches to the TinyUSB the esp32 port builds on the
S2, S3 and P4, which `build_mp.py` applies to `deps/espressif__tinyusb`
(at its `deps.lock` tag) and builds in place of the port's own copy.
- [`patches/micropython-lib/`](patches/micropython-lib/) — micropython-lib's own fixes from
after the commit MicroPython's `lib/micropython-lib` points at, which `apply_patches.py`
puts on the submodule after every update. A firmware that freezes those packages, and
a board whose `/lib` you fill from the submodule, both get them.
- [`variants/`](variants/) — our variants: `unix/` and `windows/` (`pydevices`,
`vst3-engine`), and `webassembly/` (`pydevices`, with the `wasmbridge`
module and a Fetch-backed `requests`), and `esp32/`: the sdkconfig
fragments and partition table every esp32 or every chip gets, and the few
board variants of ours (`ESP32_GENERIC_S3` `LCD_4_3`, `LCD_7` and `T_EMBED`,
`ESP32_GENERIC_P4` `WIFI6_DEV_KIT` and `WIFI6_DEV_KIT_H2`); and `rp2/`
(`RPI_PICO2_W` `PIMORONI_PICO_PLUS2_W`, the Pimoroni Pico Plus 2 W with its
16 MB flash, 8 MB PSRAM and 48 GPIOs:
`--port rp2 --board RPI_PICO2_W --variant PIMORONI_PICO_PLUS2_W`).
- [`scripts/exercise_build.sh`](scripts/exercise_build.sh) — builds every desktop port and two esp32
boards from a bare clone, checks the failure paths, and writes a report of
what should match on any machine. Needs no board. For a report on builds you
already have, name their folders:
`scripts/exercise_build.sh --report builds/rp2/RPI_PICO2/default ...`.
- [`provenance.json`](provenance.json) — the patches' checksums and migration record. After changing a patch or moving `UPSTREAM`, run `python3 scripts/update_provenance.py`; CI fails on a stale file.
```text
micropython native_cmod Area(8, 8, 64, 40)
```

What's planned next is in [ROADMAP.md](ROADMAP.md).
For a board, name its port, board and variant, and the modules you want:

## Rules
```bash
./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI \
--flash 16MB --modules displayif,pygraphics
```

- Upstream is **pinned**. Moving `UPSTREAM` re-validates every patch.
- Patches are justified in their headers. Never hand-edit a patched tree:
change the patch and prepare again.
- `build_mp.py` names no version and no setting. Pins move in the lock files,
esp32 settings in sdkconfig fragments it finds by convention.
- Publishing binaries is a separate decision from this source repository.
[Docs](https://github.com/PyDevices/micropython-pydevices/blob/main/docs/index.md) · [Examples](https://pydevices.github.io/pydevices-examples/gallery/) · [Roadmap](ROADMAP.md)

Migrated from `PyDevices/cmods` (2026-08-29), which was archived on
2026-09-23.
Made by [PyDevices](https://pydevices.com).
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ or missing for you, open an
firmware page's "Install from a file", as the released images do.

More modules in CircuitPython-compatible builds
([what builds today](docs/build-plan.md#which-modules-build-on-circuitpython)):
([what builds today](docs/reference/modules.md#which-modules-build-on-circuitpython)):

- **castif casting on CircuitPython's ESP32-P4 builds.** castif and h264enc
build there and h264enc encodes, but the port has no network until
Expand Down
Loading
Loading