diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 00000000..1db3c30f --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,34 @@ +# Builds the docs site the way Read the Docs builds it from .readthedocs.yaml. +# -W turns a warning into a failure, as fail_on_warning does there. +name: docs + +on: + push: + branches: [main] + paths: + - 'docs/**' + - '.readthedocs.yaml' + - '.github/workflows/docs.yml' + pull_request: + paths: + - 'docs/**' + - '.readthedocs.yaml' + - '.github/workflows/docs.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + docs: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-python@v7 + with: + python-version: '3.13' + - name: Install docs requirements + run: pip install -r docs/requirements.txt + - name: Build docs + run: sphinx-build -W --keep-going -b html docs _site diff --git a/.gitignore b/.gitignore index 815aa374..bcad9c7d 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,5 @@ /.worktrees/ __pycache__/ /.micropython-build.lock +/_site/ +/.venv-docs/ diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..6a17e5fc --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,14 @@ +version: 2 + +build: + os: ubuntu-24.04 + tools: + python: "3.13" + +sphinx: + configuration: docs/conf.py + fail_on_warning: true + +python: + install: + - requirements: docs/requirements.txt diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..f84b8ec0 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,119 @@ +# Contributing to micropython-pydevices + +This repository holds our patches to one pinned MicroPython release, our +variants, the C modules that live only here, and the command that builds it all. +A documentation fix, a provenance update, or a narrowly scoped variant or module +fix is a good first contribution. + +## Where things are + +| Path | What's there | +|---|---| +| `build_mp.py` | The build command. | +| `UPSTREAM` | The MicroPython commit the patches apply to. | +| `modules.lock`, `deps.lock` | What gets fetched when it's missing, lvgl-bindings included. | +| `modules/` | The module manifest, `all/`, `tflite/` and `vision/` (esp-vision's), the modules linked or cloned, and the C modules that live only here: `audioif`, `cameraif`, `castif`, `displayif`, `h264enc`, `jpegio`, `mdns`, `pngio`, `ppa`, `tsmux`, `usbhost`. Each has its own README and tests. | +| `modules/.lvgl-bindings` | The lvgl-bindings lvgl-micropython compiles: the sibling checkout, linked, or `modules.lock`'s commit, fetched. | +| `patches/` | The patch series and `apply_patches.py`. | +| `variants/`, `boards/` | Our variants and CircuitPython boards. | +| `scripts/` | `exercise_build.sh`, `release_firmware.py`, `stage_firmware_page.py`, `serve_workbench.sh`, `freeze_list.py`, `update_provenance.py`. | +| `.devcontainer/` | One dev container per port. | +| `docs/` | The documentation site. | + +The reference pages in [the docs](https://github.com/PyDevices/micropython-pydevices/blob/main/docs/index.md) +describe what each of these does. + +## Rules + +- Upstream is pinned. Moving `UPSTREAM` re-validates every patch. +- Never hand-edit a patched tree. Change the patch and prepare again. +- Patches are justified in their headers. +- `build_mp.py` names no version and no setting. Pins move in the lock files, and + esp32 settings live in sdkconfig fragments it finds by convention. +- Publishing binaries is a separate decision from this source repository. + +## Run the tests + +`build_mp.py`'s unit tests decide things before any build, such as when a build +directory has to be wiped. They need no toolchain: + +```bash +python3 -m unittest test_build_mp test_freeze_list -v +python3 -m unittest test_provenance -v +python3 -m unittest test_release_firmware test_stage_firmware_page -v +``` + +Each module has its own workflow under `.github/workflows/`, named for it. + +## Change a patch + +Check a patch change with `patches/apply_patches.py --check` before you commit +it. After changing a patch or moving `UPSTREAM`, rewrite the record, because CI +fails on a stale file: + +```bash +python3 scripts/update_provenance.py +``` + +## Exercise a build from a bare clone + +`scripts/exercise_build.sh` builds every desktop port and two esp32 boards from +a bare clone, checks the failure paths, and writes a report of what should match +on any machine. It needs no board. For a report on builds you already have, name +their folders: + +```bash +scripts/exercise_build.sh --report builds/rp2/RPI_PICO2/default ... +``` + +## Firmware sets + +A firmware set is every board's flashable file, built from one commit of this +repository by the **Release firmware** workflow +([`release-firmware.yml`](.github/workflows/release-firmware.yml); the cells are +in [`scripts/release_firmware.py`](scripts/release_firmware.py), and +`release_firmware.py cells` lists them). Each cell is one `build_mp.py` call; the +`assemble` job checks every file (checksums, the ESP image magic, the app image +inside the flash image, UF2 blocks) and writes `firmware.json`, which records +every file's sha256 and size, the flash offset, and the pins the set was built +from. + +### Build a set + +Actions > Release firmware > Run workflow. A pull request that touches the +recipe, and a run every Monday, build the same cells and keep the files as a run +artifact for 14 days. They publish nothing. + +### Put a set on a draft Release + +Tick **draft_release** and give a **tag**. After `assemble`, the `draft` job +attests every file (`gh attestation verify --repo +PyDevices/micropython-pydevices`) and creates a *draft* GitHub Release holding +the flash images, the `.elf` and `.map` of each build, and `firmware.json`. A +draft is seen only by people who can push to the repository, and GitHub makes no +tag until someone publishes it. + +The tag is `v-`. A set built from a preview +MicroPython or CircuitPython only takes a tag that starts with `staged-`, so a +preview set can't be published by accident. + +### The download page + +[`scripts/stage_firmware_page.py`](scripts/stage_firmware_page.py) lays a set out +for the site's `firmware/` folder: `firmware.json`, the flash images the page +offers and the browser flasher fetches, an ESP Web Tools manifest per ESP32 +build, and the desktop interpreters' `interpreters.json` from the newest +pydevices final that has one. + +```bash +python3 scripts/stage_firmware_page.py /firmware \ + --interpreters-from-release PyDevices/pydevices +``` + +## Build the docs + +```bash +python3 -m venv .venv-docs +.venv-docs/bin/pip install -r docs/requirements.txt +.venv-docs/bin/sphinx-build -W --keep-going -b html docs _site +``` diff --git a/README.md b/README.md index 6814eba5..561080c4 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,51 @@ # micropython-pydevices -[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/PyDevices/micropython-pydevices) - -Build MicroPython firmware with PyDevices modules in it, for any port, with -one command: +micropython-pydevices builds firmware: MicroPython-compatible and +CircuitPython-compatible interpreters with the PyDevices C modules compiled in, +for boards, desktops and the browser, with one command. ```bash -./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI \ - --flash 16MB --modules displayif,pygraphics,/home/you/earful +curl -fsSL https://pydevices.github.io/install.sh | sh ``` -Boards are upstream's own, with a variant of ours where a board needs one; the -[newcomer's guide](docs/newcomers.md#boards-we-build-for) has the command -for each board we use. You name the modules you want: ours -by short name, anything else by its path, or `all`. Leave out `--port` and it -asks you for port, board and variant. Output lands in -`builds//[/]/` (with no `--variant`, that last -directory is `default` on esp32 and rp2, `standard` on the desktop ports), -beside a record of every module's commit. [Flash it](docs/newcomers.md#flash-it) -says where `firmware.bin` goes on the board. +That clones this repository into `micropython-pydevices` in the current +directory (Linux, WSL, or Git Bash on Windows) and tells you which packages your +machine still needs. Or clone it yourself: `git clone +https://github.com/PyDevices/micropython-pydevices.git`. The build fetches +everything else. -Don't have it yet? This clones it into `micropython-pydevices` in the current -directory and checks your tools (Linux, WSL, or Git Bash on Windows): +Build MicroPython for your desktop with the pygraphics module inside: ```bash -curl -fsSL https://pydevices.github.io/install.sh | sh +./build_mp.py --port unix --modules pygraphics ``` -It names the Debian packages each port still needs, with the `apt-get` line -that installs them; [What each port needs](docs/newcomers.md#what-each-port-needs) -is the same list. +Save this as `hello.py`: -Or build in a codespace, with nothing to install: the badge above opens one, -and [In a codespace](docs/newcomers.md#in-a-codespace) says how to pick your -port and get the file out. +```python +import sys +import pygraphics -On Windows you can build the windows port natively from PowerShell, with -`python build_mp.py --port windows --variant pydevices --modules all`, once -MSYS2 is installed: [On Windows](docs/newcomers.md#on-windows) has the -steps. Everything else builds from WSL. A Linux or WSL machine also -cross-builds that same `micropython.exe`, with the same command and the -MinGW packages from the list above; the SDL2 it links against is fetched. - -The same command builds OmniPython CP (CircuitPython-compatible) firmware, which -is CircuitPython with our C modules compiled in, for CircuitPython's own -ports and boards: +fb = pygraphics.FrameBuffer(bytearray(160 * 80 * 2), 160, 80, pygraphics.RGB565) +area = fb.round_rect(8, 8, 64, 40, 8, 0xF800, f=True) +print(sys.implementation.name, pygraphics.implementation(), area) +``` ```bash -./build_mp.py --interpreter circuitpython --port raspberrypi \ - --board adafruit_feather_rp2040 --modules pygraphics +builds/unix/standard/micropython hello.py ``` -It takes C modules only, so far; the -[build plan](docs/build-plan.md#circuitpython-compatible-builds) has how it -works. - -New here? The [newcomer's guide](docs/newcomers.md) explains the model; the -[build plan](docs/build-plan.md) has the layout, the rules and the traps. - -## What's in here - -- [`build_mp.py`](build_mp.py) — the one command. It fills in `micropython/`, `modules/` - and `deps/` if they're missing, prepares the MicroPython tree, and runs - the port's own `make`. -- [`UPSTREAM`](UPSTREAM) — the MicroPython everything applies to (main at `b70ae1aca657`, v1.30.0-preview-160). -- [`modules.lock`](modules.lock), [`deps.lock`](deps.lock) — the commits and versions `build_mp.py` - fetches when a module or a toolchain isn't there. `deps.lock` also pins - CircuitPython, for `--interpreter circuitpython`. `modules.lock` also pins - lvgl-bindings, the generated binding and LVGL that lvgl-micropython - compiles: a build with lvgl-micropython fetches it into - `modules/.lvgl-bindings`. Beside sibling checkouts, these are links to those - checkouts instead. -- [`modules/`](modules/) — [`manifest.py`](modules/manifest.py) (the manifest every build freezes), `all/` - (every module), `tflite/` and `vision/` (esp-vision's modules, from the pin - in `.esp-vision/`; see [docs/esp-vision.md](docs/esp-vision.md)), the modules - themselves, linked or cloned, and the C modules that live only here: - - [`audioif/`](modules/audioif/): the audio hardware under the audio pump, with - CircuitPython's `audiobusio.I2SOut` and `audioi2sin.I2SIn` on top. - - [`cameraif/`](modules/cameraif/): a MIPI-CSI camera as a MicroPython object, on the ESP32-P4. - - [`castif/`](modules/castif/): casts an ESP32-P4's screen and sound to a TV over Wi-Fi - Display, or serves them as live HLS. - - [`displayif/`](modules/displayif/): the display buses and panels under `displaydev` - (SPI, I2C, i80, QSPI, dot-clock RGB, MIPI-DSI, PicoDVI, RGB matrix, SDL2); the - dot-clock, MIPI-DSI, PicoDVI and RGB matrix modules keep CircuitPython's names. - - [`h264enc/`](modules/h264enc/): RGB565 frames to H.264 on the ESP32-P4's hardware encoder. - - [`jpegio/`](modules/jpegio/): JPEG decode and encode, with CircuitPython's `jpegio` API. - - [`mdns/`](modules/mdns/): find and advertise services on the local network, with - CircuitPython's `mdns` API (esp32, rp2, unix, windows). - - [`pngio/`](modules/pngio/): PNG encode and decode, RGB565 in and out. - - [`ppa/`](modules/ppa/): scale, rotate, convert and fill pixel buffers, on the - ESP32-P4's Pixel Processing Accelerator or in portable C elsewhere. - - [`tsmux/`](modules/tsmux/): MPEG-TS and HLS segments from H.264. - - [`usbhost/`](modules/usbhost/): the USB host port and isochronous streams that - `usb.core` and the `usb.host.*` packages are built on (ESP32-S2, S3 and P4). -- [`patches/micropython/`](patches/micropython/) — our patch series (`0001-…` to `0042-…`; there - is no `0026`, whose fix moved into `patches/tinyusb/`), each justified - in its own header; [`patches/apply_patches.py`](patches/apply_patches.py) applies it, plus - each module's own MicroPython patches, once, as one local commit. -- [`patches/tinyusb/`](patches/tinyusb/) — patches to the TinyUSB the esp32 port builds on the - S2, S3 and P4, which `build_mp.py` applies to `deps/espressif__tinyusb` - (at its `deps.lock` tag) and builds in place of the port's own copy. -- [`patches/micropython-lib/`](patches/micropython-lib/) — micropython-lib's own fixes from - after the commit MicroPython's `lib/micropython-lib` points at, which `apply_patches.py` - puts on the submodule after every update. A firmware that freezes those packages, and - a board whose `/lib` you fill from the submodule, both get them. -- [`variants/`](variants/) — our variants: `unix/` and `windows/` (`pydevices`, - `vst3-engine`), and `webassembly/` (`pydevices`, with the `wasmbridge` - module and a Fetch-backed `requests`), and `esp32/`: the sdkconfig - fragments and partition table every esp32 or every chip gets, and the few - board variants of ours (`ESP32_GENERIC_S3` `LCD_4_3`, `LCD_7` and `T_EMBED`, - `ESP32_GENERIC_P4` `WIFI6_DEV_KIT` and `WIFI6_DEV_KIT_H2`); and `rp2/` - (`RPI_PICO2_W` `PIMORONI_PICO_PLUS2_W`, the Pimoroni Pico Plus 2 W with its - 16 MB flash, 8 MB PSRAM and 48 GPIOs: - `--port rp2 --board RPI_PICO2_W --variant PIMORONI_PICO_PLUS2_W`). -- [`scripts/exercise_build.sh`](scripts/exercise_build.sh) — builds every desktop port and two esp32 - boards from a bare clone, checks the failure paths, and writes a report of - what should match on any machine. Needs no board. For a report on builds you - already have, name their folders: - `scripts/exercise_build.sh --report builds/rp2/RPI_PICO2/default ...`. -- [`provenance.json`](provenance.json) — the patches' checksums and migration record. After changing a patch or moving `UPSTREAM`, run `python3 scripts/update_provenance.py`; CI fails on a stale file. +```text +micropython native_cmod Area(8, 8, 64, 40) +``` -What's planned next is in [ROADMAP.md](ROADMAP.md). +For a board, name its port, board and variant, and the modules you want: -## Rules +```bash +./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI \ + --flash 16MB --modules displayif,pygraphics +``` -- Upstream is **pinned**. Moving `UPSTREAM` re-validates every patch. -- Patches are justified in their headers. Never hand-edit a patched tree: - change the patch and prepare again. -- `build_mp.py` names no version and no setting. Pins move in the lock files, - esp32 settings in sdkconfig fragments it finds by convention. -- Publishing binaries is a separate decision from this source repository. +[Docs](https://github.com/PyDevices/micropython-pydevices/blob/main/docs/index.md) · [Examples](https://pydevices.github.io/pydevices-examples/gallery/) · [Roadmap](ROADMAP.md) -Migrated from `PyDevices/cmods` (2026-08-29), which was archived on -2026-09-23. +Made by [PyDevices](https://pydevices.com). diff --git a/ROADMAP.md b/ROADMAP.md index bd19c533..49982e1f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,7 +9,7 @@ or missing for you, open an firmware page's "Install from a file", as the released images do. More modules in CircuitPython-compatible builds -([what builds today](docs/build-plan.md#which-modules-build-on-circuitpython)): +([what builds today](docs/reference/modules.md#which-modules-build-on-circuitpython)): - **castif casting on CircuitPython's ESP32-P4 builds.** castif and h264enc build there and h264enc encodes, but the port has no network until diff --git a/docs/build-plan.md b/docs/build-plan.md deleted file mode 100644 index 3dd295e7..00000000 --- a/docs/build-plan.md +++ /dev/null @@ -1,474 +0,0 @@ -# The build plan - -**Status:** built, 2026-10-05. Steps 1 to 5 are done and step 6 was dropped (see "Order of work"); this page now records how the build works and why. - -You build PyDevices firmware with one command: - -```bash -./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB --modules audiodsp,displayif,/home/you/earful -``` - -Leave out `--port` and it walks you through port, board and variant, the way -the old `build_mp.sh` did; leave out `--board` on a port that has boards -(esp32, rp2) and it asks for that. `--variant` is always optional, as with -upstream's `make`, and unix, windows and webassembly take no `--board`. Leave out `--modules` and it lists what's in `modules/` and -asks. Our modules go by short name, anything else by full path, and there are -no recipe files: what goes into a build is what you typed. Anything it doesn't -recognise goes straight to `make`. - -Boards come from upstream. You don't write a board definition for a new board; -you pick upstream's generic one, tell it the flash size, and, only where a -board really differs, add a small variant on top. If the image outgrows the -app partition, the build grows the partition and builds again. - -## The layout - -```text -micropython-pydevices/ -├── UPSTREAM the MicroPython commit we build on (b70ae1aca657, v1.30.0-preview-160) -├── VERSION our version (scheme still open, below) -├── build_mp.py the one command; Python so it runs anywhere -├── modules.lock one line per module: name, git URL, commit (and lvgl-bindings and omnipython-lib, below) -├── deps.lock one line per dependency: name, URL, version (ESP-IDF v5.5.4, emsdk 6.0.1, SDL2 2.30.10) -├── micropython/ ignored: symlink to a sibling checkout, or a clone of UPSTREAM -├── deps/ ignored: esp-idf/, emsdk/, SDL2/, symlinked or fetched on demand -├── modules/ -│ ├── manifest.py tracked: reads the module list build_mp.py sets, includes each, raises on a missing one -│ ├── all/manifest.py tracked: every sibling with a root manifest.py, except opt-in ones -│ ├── castif/ tracked: lives only here (moves from usermods/); ESP32-P4 only -│ ├── jpegio/ tracked: lives only here (moved from displayif, 2026-10-06); every port -│ ├── usbhost/ tracked: lives only here (moved from usbif, 2026-10-19, history kept); ESP32-S2, S3 and P4 -│ ├── mdns/ tracked: lives only here; esp32, rp2, unix, windows -│ ├── audioif/, cameraif/, displayif/ tracked: moved here 2026-10-19 from their own repos, history kept -│ ├── library.py tracked: registers omnipython-lib for the manifests that require from it -│ ├── pydevices/, palettes/, pdwidgets/, audiocomponents/ tracked: a manifest each, requiring the package(s) from omnipython-lib -│ ├── micropython-lib/ tracked: a manifest requiring the upstream packages our APIs lean on -│ ├── audiodsp -> ../../audiodsp ignored symlinks to sibling checkouts, -│ ├── lvgl-micropython, clones at the modules.lock commit anywhere else -│ ├── pygraphics, ulab -│ ├── .omnipython-lib ignored: the library repository, a link or a clone at its modules.lock commit -│ ├── .lvgl-bindings ignored: what lvgl-micropython compiles; a link, or a sparse fetch at its modules.lock commit -│ └── jpegio later: see "Open" -├── patches/ -│ ├── apply_patches.py the overlay and the modules' own patches, applied once as one local commit -│ ├── micropython/ 0001-…patch to 0038-…patch (no 0026) -│ └── tinyusb/ patches to the TinyUSB the esp32 port builds (S2, S3, P4) -├── variants/ only where a board truly differs from upstream -│ ├── unix/ windows/ pydevices/, vst3-engine/ (as today) -│ ├── webassembly/ pydevices/, and wasmbridge moves in here from usermods/ -│ └── esp32/ -│ ├── sdkconfig, sdkconfig. every esp32 build, then every build for that chip -│ └── // a board's own delta, only where one is left -├── builds/ ignored: /[/]/, the generated board dir inside -├── scripts/ maintainer scripts -├── docs/ -└── .devcontainer/ -``` - -When the repositories are already checked out side by side, the real -checkouts stay where they are, and `modules/` and `micropython/` are -symlinks to them. -Anywhere else, [`build_mp.py`](../build_mp.py) clones what's missing: modules at their -`modules.lock` commit, MicroPython at the `UPSTREAM` tag, and the toolchains -at their `deps.lock` version. Beside such checkouts, `deps/` links to the -sibling `esp-idf/`, `emsdk/` and `SDL2/`. A build fetches and checks only the toolchain -its port uses: ESP-IDF for esp32, emsdk for webassembly, SDL2 for windows when -displayif is selected, nothing for unix or rp2. If that one isn't the locked -version, the build refuses. Nothing is a git -submodule, so a module commit never dirties this repo and the overlay commit -never shows as modified. - -lvgl-bindings is pinned in `modules.lock` too, though it isn't a module: it -holds the generated binding and the LVGL that lvgl-micropython compiles. A -build with lvgl-micropython puts it at `modules/.lvgl-bindings`, a link to the -sibling checkout or, anywhere else, a fetch of its `modules.lock` commit with -only what gets compiled (tens of MB, not LVGL's history). The build hands it -to lvgl-micropython as `BINDINGS_DIR`, so lvgl-micropython's own -`LVGL_BINDINGS_COMMIT` and fetch never come into a build made here. A sibling -checkout is built as it is, like every other module; when it isn't -lvgl-micropython's pinned commit, the build prints a note. The build record -has a row for it, its commit and whether it was dirty, beside the modules'. - -## How a build runs - -`build_mp.py` fills in whatever is missing from `micropython/`, `modules/` and -`deps/`, then prepares the MicroPython tree if it isn't prepared already. It -puts your module list in an environment variable that [`modules/manifest.py`](../modules/manifest.py) -reads, and calls the port's own `make` with `FROZEN_MANIFEST=modules/manifest.py` -and, where you asked for one of ours, `VARIANT_DIR`. - -On esp32 it also writes a thin board dir into the build dir, the way cmods' -`build_mp.sh` did. That dir includes upstream's board and variant unchanged, -links the rest of the board's files through, and appends our sdkconfig -**last**: the flash size and partition table, then the fragments below. -Last matters, because kconfgen takes the last assignment, and a fragment -listed earlier is silently overridden. - -The fragments are found by convention, and each is applied only if it -exists. Every place that can carry them uses the same two names: `sdkconfig` -for every esp32 build, `sdkconfig.` (the chip read from the board's -`board.json`, such as `esp32p4`) for one chip. They stack in this order: - -1. [`variants/esp32/`](../variants/esp32/): PyDevices on any esp32, then on that chip. The S3's - Wi-Fi, lwIP and PSRAM settings and the P4's NimBLE-over-C6 and - cache-writeback settings live here. -2. Each selected module's root, in the order you listed them: what that - module needs. usbhost's USB host settings and cameraif's OV5647 driver live - in their own module directories. -3. `variants/esp32///`: what's left that really is the board, - such as the LCD-7's cache tuning. A variant of ours there carries an - `mpconfigvariant.cmake` that includes the upstream variant it builds on, - and may carry an `mpconfigboard.h`, included before the board's own, for - a C define the ESP-IDF components read too (they see the board header - but not `MICROPY_DEF_BOARD`; `WIFI6_DEV_KIT`'s `MICROPY_HW_USB_HS` is - one). It may also carry a `manifest.py`, which the build freezes instead - of `modules/manifest.py`: it includes that file and adds what only this - board freezes. `WIFI6_DEV_KIT_H2`'s adds pydevices' USB packages and - midiports, so its apps run without a mip install. - -On the S2, S3 and P4 the build also patches TinyUSB, which the port takes -from the IDF component manager and so the overlay can't reach. It checks out -`deps/espressif__tinyusb` at its `deps.lock` tag, which must be the one patch -0014 points the port's manifest at, applies [`patches/tinyusb/`](../patches/tinyusb/) as one local -commit, and adds a one-file component to the board dir whose manifest gives -`espressif/tinyusb` an `override_path` to that copy. The configure log says -`Using component placed at .../deps/espressif__tinyusb`, and the build record -names the TinyUSB it built. (A component of the same name in -`EXTRA_COMPONENT_DIRS` isn't enough: the managed copy is still fetched and -added after it, and wins.) - -A `partitions.csv` is found the same way, the most specific winning -(`//partitions.csv`, then `partitions..csv`, then -`partitions.csv`); without one, the board's own table is used. - -Nothing in `build_mp.py` names a version or a setting. A pin moves by editing -a lock file and a setting by editing a fragment, and the script never changes -for either. `make` gets -it as `BOARD_DIR=`. Upstream's tree is never edited, so the ports need no -patch for this. - -If the app doesn't fit its partition, the build reads the size out of the -error, grows the app partition to the image plus headroom (1/32 of the flash, -64 KB to 256 KB), rounded up to 64 KB, moves every partition after it, and builds once more against -that table, which lives only in the build dir. It then prints the new layout -and warns that the filesystem moved. This port puts the filesystem after the -last partition, so a board flashed with the new layout comes up with an empty -filesystem. Dev boards hold nothing we keep, so this is on by -default; `--no-autosize` refuses instead and prints the table that would fit. - -rp2 gets the same treatment. When the link overflows the `FLASH` region, the -build reads the linker's memory table, gives the firmware the image plus the -same headroom, and builds once more with the rest as the filesystem -(`MICROPY_HW_FLASH_STORAGE_BYTES`, recorded in the build record). - -Output lands in `builds//[/]/`. The last level is always -there, so one build never nests inside another: with no `--variant` it's the -name upstream's `make` uses, `standard` on unix, windows and webassembly and -`default` for a board. Each dir holds a record of what -went in: each module's path and commit. Build the same target with a -different module set, or with a module at another commit, and it wipes that -dir and rebuilds, and says which module moved and from where to where. -Uncommitted edits to a module build incrementally, as make and ninja handle -them. One dir -per target, never a stale cache (idf.py keeps `-D` values between runs), and -no pile of experiment dirs. - -`all` is a module like any other, [`modules/all/`](../modules/all/), so the logic of "everything" -lives in its own manifest, which we can change without touching the build -script. A module with an `OPT_IN` file is the one exception: `all` leaves it -out, and it's built only when named. tflite and vision are opt-in, for their -size ([docs/esp-vision.md](esp-vision.md)). - -A module never brings another module. ulab is a module like any other, -pinned only by `modules.lock`; audiodsp, which never used it, no longer names -its own copy, so nothing is compiled twice. Nothing checks dependencies -either: audiocomponents' instruments import ulab, and tflite and vision -compile against it, so name ulab with them, or use `all`. - -Our desktop and browser variants (`unix`, `windows` and `webassembly` -`pydevices`) freeze the universal desktop board config themselves, from -`modules/pydevices/board_configs/desktop`, so `--variant pydevices` is a -complete desktop runtime and `all` stays all. Board images never carry a board -config, and neither does mpvst's `vst3-engine`, which has its own. - -castif is the one module whose only home is this repo, so [`modules/castif/`](../modules/castif/) is -tracked here rather than linked. It's ESP32-P4 only. Its H.264 encoder moved -into [`modules/h264enc/`](../modules/h264enc/) (the media roadmap's Phase 3), an ESP-IDF component -that has to be added before `project()` runs, so the generated board dir adds -`h264enc/components/h264enc_esp_h264` to `EXTRA_COMPONENT_DIRS` whenever -h264enc is selected: any selected module's `components//` is added that -way, by convention. Its own glue skips every other port and chip, -the way audioif's skips non-esp32 ports, so `all` needs no exception for it. - -## Python-only packages come from omnipython-lib - -pydevices, palettes, pdwidgets and audiocomponents' two packages are Python -only, so they are not checkouts in `modules/`: they are packages of -omnipython-lib, the repository of our Python-only packages (the pydevices -repository, until its rename). `modules.lock` pins it like a module -(`omnipython-lib`), and a build that needs it asks its `tools/libbuild.py -build --tree-only` for the library tree: one directory per package with a -`manifest.py` beside its sources. `modules/library.py` registers that tree -with `add_library("omnipython-lib", ...)`, and a manifest then says -`require("palettes", library="omnipython-lib", opt=3)`. In the workspace the -checkout is a link to the sibling `pydevices`, which wins over the pin like -every other module; anywhere else it is a clone at the pinned commit. The tree -goes to `deps/omnipython-lib`, and the outside packages the library builds from -other repositories at a tag (audiocomponents' instruments and effects, -pygraphics) are cloned once into `deps/omnipython-lib-sources`. In the -workspace they come from the sibling checkouts instead, as modules do, so a -workspace build freezes the code beside it; the build says which. - -You still choose per build: name `pydevices`, `palettes`, `pdwidgets` or -`audiocomponents` in `--modules` and it's frozen (each is a small manifest in -`modules/` that requires its packages); leave it out and the board installs it -with mip. `all` takes them, because all means all. Any other package of the -library can be named the same way: `--modules appkit`, or a board config such -as `esp32-s3-touch-lcd-7` with its drivers, which the package requires. A -package with C stays a module (pygraphics, the device modules). - -`modules/pydevices` freezes what `lib/` always held: the library's `pydevices` -and `bledev`, plus the host-only files of `audiodev`, `bledev`, `displaydev` -and `multimer` that mip ships in `pydevices-desktop` (not its `mip.py`, -`pngio.py` or `mdns.py`, which would stand in front of the firmware's own). -Boards still carry those host-only files; leaving them out is a separate, -smaller change. - -`modules/micropython-lib` freezes the upstream packages the aligned APIs need -(`aioble`, `sdcard`, `bmi270`, `__future__`, `usb-device` and its device -classes) with `require()` from the micropython-lib checkout the MicroPython -tree carries, so an OmniPython firmware never asks anyone to `mip.install` -them. It is in `all`. - -CircuitPython-compatible builds freeze the same way. A module that names no -`c_module()` and has a `manifest.py` (palettes, pdwidgets, pydevices, -`lvgl-circuitpython`'s `fs_driver`) is frozen, and so is any library package -you name; `build_mp.py` writes one `frozen-manifest.py` into the build -directory and hands it to CircuitPython's make as `FROZEN_MANIFEST`. That -manifest starts with the unix port's coverage manifest on the unix port and -with the board's own `FROZEN_MPY_DIRS` content on a board that has some -(setting `FROZEN_MANIFEST` beats the manifest CircuitPython would make from -them, so the build freezes them itself). A build that freezes any of our -Python also freezes CircuitPython's `asyncio` and `adafruit_ticks` from its -`frozen/` submodules (unless the board already does), because `bledev` and the -rest `import asyncio` and no CircuitPython board freezes one. The build record -lists them as `frozen_libraries`, and a release cell fails if its image -doesn't hold them. C modules still reach CircuitPython -through `USER_C_MODULES`, and `micropython-lib` is MicroPython's own and is -left out. - -`scripts/freeze_list.py list` evaluates a build's manifest and writes every -file it freezes, by name, with the hash of its source and of what is frozen -(a required package's files get a `__version__` line, as mip installs them), -and the version of every package a `require()` brought in; `compare` says what -differs between two lists: run it on a build's manifest before and after a -change to see exactly what the change did to the firmware. - -## Order of work - -1. **Reorganise this repo**, on a branch in its own worktree. That covers the - tree above, `modules.lock`, `deps.lock`, the two module manifests, `apply_patches.py` - (replacing `apply.sh` and `tools/prepare-micropython.sh`, still applying - usbif's and cameraif's patches) and the `.gitignore`. The patch profiles - go: every build applies the whole series, and CI checks that the series - applies clean to the pin. `manifests/` goes: - the kitchen sink becomes `modules/all`, and the presets turn into - `--modules` lists. `usermods/castif` moves to `modules/castif`, and - audiocomponents gets the root `manifest.py` it lacks. -2. **Write `build_mp.py`**, with the generated esp32 board dir, `--flash`, - and autosize, ported from cmods' `build_mp.sh`. -3. **Prove the modules on stock upstream builds.** That means unix and - windows (stock variants), webassembly (our variant), `ESP32_GENERIC_S3` - `SPIRAM_OCT`, and `ESP32_GENERIC_P4` in both `C6_WIFI` (your DEV-KIT) and - `PRE_REV3_C6_WIFI` (the panel), with no delta but `--flash` and autosize. - The gate is that for the same module set, each build's module list matches - today's, and the S3 and the DEV-KIT boot and import what they carry. -4. **Repoint what calls the old way:** the interpreter build script, - mpvst's engine build (its modules listed by name plus - `/vstaudio,/vstui`; not `all`, which now freezes pydevices), - wokwi's stage script, earful's build, the README and [`newcomers.md`](newcomers.md). -5. **Retire `boards/`.** Done 2026-10-05. The four boards' sdkconfig lines - sorted into the three homes above: chip settings into [`variants/esp32/`](../variants/esp32/), - usbif's and cameraif's into their repos (usbif's board-header lines became - compile definitions in its own CMake), and two board variants of - `ESP32_GENERIC_S3`, not one: the LCD-7's cache tuning, and the T-Embed's - flash auto-suspend, which IDF supports only on particular flash chips. The - partition tables went; the S3 keeps a coredump partition through - [`variants/esp32/partitions.esp32s3.csv`](../variants/esp32/partitions.esp32s3.csv), found by the same convention as - the fragments. Flash core dumps are off in standard builds (2026-10-20): `--coredump` - layers [`sdkconfig.coredump`](../variants/esp32/sdkconfig.coredump) last, and refuses a board - whose table has no `coredump` partition (today everything but the S3). The S3 - table keeps its 64 KB slot unused in a standard build, so no filesystem moves. -6. **Dropped (2026-10-05): the VARIANT_DIR spike.** Teaching the esp32 and rp2 ports to - take a variant from outside the board dir would make the generated dir - tidier, not possible; it already works without it. It isn't planned as an - upstream PR. - -## CircuitPython-compatible builds - -The same command builds OmniPython CP (CircuitPython-compatible) firmware: -CircuitPython, with our C modules compiled in. - -```bash -./build_mp.py --interpreter circuitpython --port raspberrypi \ - --board adafruit_feather_rp2040 --modules pygraphics -./build_mp.py --interpreter circuitpython --port unix --modules pygraphics -./build_mp.py --interpreter circuitpython --port espressif \ - --board adafruit_qtpy_esp32_pico --modules audiodsp -``` - -Ports and boards are CircuitPython's own (`raspberrypi`, `atmel-samd`, -`espressif`, `unix` and the rest; boards by their directory names under `ports//boards/`). -Output lands in `builds/circuitpython///`, with the -same build record; unix builds CircuitPython's default variant, `coverage`, -unless you name another. Arguments for make, such as `CIRCUITPY_ULAB=0` to -free flash on a small board, go on the end and are part of the record: a -build with different ones starts from an empty build dir, because -CircuitPython's generated module table would otherwise keep the old set. - -It is a flag rather than a second command because everything around the -interpreter is shared: the module names and `modules.lock`, the workspace -links, the output layout and the build record. What differs is one function: - -- **The checkout.** CircuitPython is pinned in `deps.lock` like a toolchain - (`11.0.0-alpha.2`, whose core is MicroPython 1.29) and cloned into - `deps/circuitpython`, always our own clone, never a link to a sibling - checkout. A build fetches only the submodules its target needs, with - CircuitPython's own `tools/ci_fetch_deps.py` (a board's name, or `tests` - for unix), and its build tools go in `deps/circuitpython-venv` from its - `requirements-dev.txt`. -- **The modules go in as `USER_C_MODULES`**, not a manifest. From the 1.29 - merge on, CircuitPython's `py/py.mk` carries MicroPython 1.29's - `USER_C_MODULES` block unchanged, so a module's `micropython.mk` is the one - our MicroPython builds read. CircuitPython freezes Python its own way - (`FROZEN_MPY_DIRS`), so a module with no `micropython.mk` and no Python to - freeze is refused rather than left out quietly. Each module is handed to make through a - link named for it, in `usermods/` in the build dir: make puts a module's - objects under a directory named like the module's, and two worktrees - checked out under the same branch name would otherwise share one. -- **espressif brings its own ESP-IDF.** CircuitPython's espressif port builds - with the ESP-IDF in `ports/espressif/esp-idf` (6.1 for 11.0.0-alpha.2), not - the `esp-idf` in `deps.lock`. Before make, the build initialises that tree's - submodules shallowly (ESP-IDF's CMake would otherwise clone them with their - whole history, gigabytes for `esp32-wifi-lib` alone), runs its `install.sh` - for the board's chip (compilers go in `~/.espressif`, shared with any - ESP-IDF of the same version) and puts ESP-IDF's Python requirements in - `deps/circuitpython-venv`. -- **Our patches and boards go on first.** [`patches/circuitpython/*.patch`](../patches/circuitpython/) - are applied to `deps/circuitpython` and each `boards/circuitpython//` - is linked into the port's `boards/`, once each; the build record lists the - patches. The boards are ones CircuitPython has no definition for: - `feather_rp2040_lvgl` (room for LVGL), `waveshare_esp32s3_touch_lcd_7` - (its 800x480 panel as `board.DISPLAY`, one 4 MB app slot), - `waveshare_esp32s3_touch_lcd_43` (the same 800x480 panel as `board.DISPLAY` - and one 4 MB app slot, like the LCD-7), - `lilygo_twatch_s3_single_app` (the T-Watch S3 with one 4 MB app slot, so LVGL - fits), and the - ESP32-P4 boards `waveshare_esp32p4_wifi6_dev_kit` (a revision 3 chip, - console on USB Serial/JTAG) and `waveshare_esp32p4_wifi6_touch_lcd_4b` - (revision 1, console on the CH343 UART; its panel isn't set up yet), each - with one 6 MB app slot. -- **A module can ask espressif for ESP-IDF components.** CircuitPython's - espressif port builds and links a fixed list of ESP-IDF components, which - leaves out the camera's. With [`0006-espressif-esp-idf-components-a-user-c-module-asks-for.patch`](../patches/circuitpython/0006-espressif-esp-idf-components-a-user-c-module-asks-for.patch), - a module's `micropython.mk` adds names to `USER_ESP_IDF_COMPONENTS` - (cameraif: the CSI controller, the ISP and the sensor drivers; jpegio and - ppa: the P4's JPEG engine and PPA; h264enc: `esp_h264`), and the make line's - `USER_ESP_IDF_COMPONENT_DIRS` adds components ESP-IDF doesn't ship. - CircuitPython builds with the component manager off, so for cameraif the - build fetches `esp_cam_sensor`, `esp_sccb_intf` and `cmake_utilities` at - their `deps.lock` versions into `deps/` and passes them, and `esp_h264` the - same way for h264enc. On CircuitPython, - jpegio is only the encoder's C API (CircuitPython has its own `jpegio` - decoder module). The DEV-KIT's board sdkconfig enables the OV5647. -- **mpy-cross is built first, on its own**, for the same reason as on - MicroPython (the trap below). -- **`BUILD=` is a short link.** make is given `BUILD=build-`, a link in - the port directory to our build dir: raspberrypi's link step echoes every - object path in one shell argument, and with a long absolute path in front - of each it passes Linux's 128 KB limit on one argument. - -The banner is CircuitPython's, unchanged: `Adafruit CircuitPython -11.0.0-alpha.2 on ; with `. A module that has to -behave differently here can test `CIRCUITPY`, which every CircuitPython port -sets, unix included; pygraphics also passes its own define from its -`micropython.mk`, so it knows it came in as a user C module and not through -CircuitPython's shared-bindings. - -Two things a module's C has to absorb on CircuitPython's MCU ports: they -append `-Werror` warnings (`sign-compare`, `float-equal`, `cast-align` on -atmel-samd; `float-equal`, `double-promotion`, `cast-align` on espressif) -after the module's flags, so a `-Wno-` meant for them goes on -each object, where it lands last; and atmel-samd has no C heap, so -`malloc()` doesn't link there (use `m_malloc()`). - -Builds from Linux or WSL only. - -### Which modules build on CircuitPython - -What `--modules` can name on CircuitPython today, and why a module is missing -where it is. "Unix" is CircuitPython's unix port, built and imported in -[`circuitpython.yml`](../.github/workflows/circuitpython.yml); "boards" are the -five cells of [`release-firmware.yml`](../.github/workflows/release-firmware.yml). - -| Module | Unix | Boards | Notes | -|---|---|---|---| -| pygraphics | yes | all five | | -| audiodsp | yes | the four ESP32 cells | the modules CircuitPython lacks, `audiograph` and `audiopump.Ring` (Python-made audio that `audiobusio.I2SOut` plays) among them; its audiocore, synthio and the rest stay CircuitPython's. With all of it, a LilyGO T-Embed S3 build fits that board's 2 MB app slot only with `CIRCUITPY_ULAB=0` on the make line | -| displayif | yes (`usdl2`) | no, on purpose | boards have CircuitPython's own `displayio` and display buses. pydevices' `SDLDisplay` runs on top of `usdl2` | -| pngio | yes (was a build error) | all five | decodes with CircuitPython's older uzlib | -| tsmux | yes | all five | | -| ppa | yes (software) | ESP32-P4 | the hardware path on the P4 | -| jpegio | yes | ESP32-P4 | adds only the encoder's C API: CircuitPython has its own `jpegio` decoder | -| cameraif | yes (`available()` is False) | ESP32-P4 | | -| mdns | yes (was left out) | no, on purpose | CircuitPython's boards have an `mdns` of their own; its unix port had none | -| usbhost | yes, as `_usbhost`, with no host engine (was a build error) | ESP32-P4 | the host engine runs on ESP-IDF's USB host library, which from ESP-IDF 6 is Espressif's `usb` component: `build_mp.py` fetches it at the version `deps.lock` pins. The P4's boards leave its high-speed controller to us. Tethering is left out until the port has a network, and on the WIFI6 DEV-KIT the host has enumerated a webcam and a keyboard-and-mouse dongle, with drives and streams still to try ([roadmap](../ROADMAP.md)). The S3's one controller is CircuitPython's drive and REPL, so it gets the module without the engine. On the RP2040 and i.MX, CircuitPython's own `usb.core` is the host, and pydevices' HID, CDC and MSC drivers are planned for it | -| audioif | no | no | links against audiodsp's audio pump and `cp_compat`, which audiodsp's CircuitPython glue doesn't build. CircuitPython's audio samples are fed only from C, so audio made in Python goes through audiodsp's `audiopump.Ring`, the pump's C side shaped as a sample. It isn't needed there: audioif gives MicroPython the `audiobusio` and `audioi2sin` that CircuitPython boards already have | -| ulab | CircuitPython's own | CircuitPython's own | | -| castif | no: it is the ESP32-P4's | ESP32-P4, with no network yet | talks to lwIP directly, the lwIP under CircuitPython's own `socketpool`. CircuitPython's P4 port has no network until it drives the boards' ESP32-C6, so castif is built without one and `start()` says so ([roadmap](../ROADMAP.md)). Name tsmux, h264enc and ppa with it | -| h264enc | no: it is the ESP32-P4's encoder | ESP32-P4 | Espressif's `esp_h264`, which CircuitPython's port, built with the component manager off, can't fetch itself, is pinned in `deps.lock` and fetched by `build_mp.py`, the way cameraif's sensor components are. Name ppa with it | -| tflite, vision | no | no, on purpose | esp-vision's modules, for MicroPython only | -| lvgl-micropython | no | no | CircuitPython's binding is `lvgl-circuitpython`, passed by path | -| micropython-lib | no | no | MicroPython's own upstream packages | -| pydevices, palettes, pdwidgets | frozen | frozen | Python only. On unix, pydevices also freezes its desktop files (the boards drop them), and pdwidgets and palettes with it: together their names pass CircuitPython's 64 KB frozen-string pool, so `build_mp.py` builds the unix port with four-byte qstr offsets (patches 0001 and 0010). Boards keep two-byte offsets | - -## Traps we already know - -**`BUILD=` and the mpy-cross sub-make.** Passed on its own, `BUILD=` reaches -the mpy-cross sub-make and breaks an esp32 link (micropython#19667, still -open). Both build systems skip that sub-make when `MICROPY_MPYCROSS` names a -built mpy-cross (`py/mkenv.mk`, `py/mkrules.cmake`), so `build_mp.py` builds -mpy-cross on its own first and exports its path. No patch needed. - -**Upstream's generic esp32 boards give the app 1.94 MB** (`partitions-4MiBplus.csv`), -and the kitchen sink is about 3.4 MB. Autosize handles that, but only if the -flash size is right: gen_esp32part refuses a table larger than the configured -flash. Hence `--flash`. - -**A saved `sdkconfig` beats the defaults.** The IDF treats one left in the -build dir as your configuration, so one failed run can pin the wrong -partition table for every run after it. The build deletes it first, and -afterwards checks that the configured table is the one we chose, failing -loudly if it isn't. - -**`make clean` on esp32 is `idf.py fullclean`**, which deletes the whole -port's `managed_components/`. Never run it against a build dir that doesn't -exist yet. - -**Two esp32 builds at once in one port race** on `managed_components/`, so -`build_mp.py` builds esp32 one at a time. - -## Open - -Deferred, because the build script doesn't need them: - -- **The version scheme** and **what a release carries.** -- **A CI leg that builds the union** on one desktop port, so building only - what you ask for doesn't lose the cross-port check that caught usbif's - breaks in September. -- **jpegio** left displayif on 2026-10-06 for [`modules/jpegio`](../modules/jpegio/) here, not a repo of its own. Whether esp-vision's - sensor and H.264 replace cameraif (and castif's encoder) stays open. esp-vision's - `tflite` and image stack are in as modules; where `sensor` stands is in - [esp-vision.md](esp-vision.md). -- **CircuitPython-compatible builds** and the modules that don't build there - yet: [Which modules build on CircuitPython](#which-modules-build-on-circuitpython). diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 00000000..a043e86d --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,20 @@ +"""Sphinx configuration for the micropython-pydevices documentation. + +Everything shared by the PyDevices documentation projects (theme, top bar, +Markdown setup) comes from pydevices-docs. This repository has no Python +package to document, so the reference pages are written by hand. +""" + +from pydevices_docs import configure + +globals().update(configure( + "micropython-pydevices", __file__, + repo="PyDevices/micropython-pydevices", edit_path="docs", + version_file=None, +)) + +# Planning notes and upstream reports that live here until they move. + +# index.md is plain Markdown so the docs read on GitHub too; the site +# navigation lives in contents.md. +master_doc = "contents" diff --git a/docs/contents.md b/docs/contents.md new file mode 100644 index 00000000..cb68bd5f --- /dev/null +++ b/docs/contents.md @@ -0,0 +1,43 @@ +# Contents + +The site's navigation. On GitHub, start at [the docs index](index.md). + +```{toctree} +:hidden: +:caption: Get started + +index +get-started/install +get-started/first-program +``` + +```{toctree} +:hidden: +:caption: Guides + +guides/build-for-a-board +guides/choose-modules +guides/flash-a-board +guides/fit-firmware-on-a-small-flash +guides/build-in-a-codespace +guides/build-on-windows +guides/build-for-circuitpython +guides/add-a-board-variant +guides/build-a-troubleshooting-image +guides/use-esp-vision-modules +guides/read-build-warnings +``` + +```{toctree} +:hidden: +:caption: Reference + +reference/index +``` + +```{toctree} +:hidden: +:caption: How it works + +how-it-works/index +``` diff --git a/docs/esp-vision.md b/docs/esp-vision.md deleted file mode 100644 index f5a35f01..00000000 --- a/docs/esp-vision.md +++ /dev/null @@ -1,86 +0,0 @@ -# esp-vision's modules - -Two modules come from [espressif/esp-vision](https://github.com/espressif/esp-vision) -(Apache-2.0; its imlib is OpenMV's, MIT), compiled from esp-vision's own files -without changes. Both are ESP32-P4 and ESP32-S3 only, both need ulab, and both -are opt-in: `--modules all` leaves them out, so name them: - -```bash -./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB --modules ulab,tflite,vision -``` - -- **`tflite`**: TensorFlow Lite Micro inference (`tflite.Model`), the API OpenMV's `ml` module uses. -- **`vision`**: the OpenMV-style `image` API (`image`, `imageio`), ESP-DL inference (`espdl`), and on the P4 the hardware H.264 encoder (`h264`) and an RTSP server (`rtsp`). - -esp-vision's stubs (`stubs/*.pyi` in esp-vision) document the API. - -## How it is built - -[`modules/.esp-vision`](../modules/.esp-vision/) pins esp-vision at a full commit ([`ESP_VISION.lock`](../modules/.esp-vision/ESP_VISION.lock)). -Its [`fetch.sh`](../modules/.esp-vision/fetch.sh) links a sibling clone at that commit (`../esp-vision`), -or fetches that commit alone. The leading dot keeps it out of `--modules` and -`all`, because esp-vision's own `micropython.cmake` is its whole firmware -build, not a module. - -[`modules/tflite`](../modules/tflite/) and [`modules/vision`](../modules/vision/) are our glue around esp-vision's files: -the ESP-IDF components esp-vision uses, at its versions; the include paths and -definitions MicroPython's QSTR scan needs (it sees only a module's own paths, -never a linked component's); and two pieces of our own: - -- **imlib compiles inside the module**, not as an ESP-IDF component. As a - component it would wait for MicroPython's generated headers while being one - of their inputs (a ninja dependency cycle). esp-vision's own - `components/imlib` CMakeLists can't be used anyway: it finds its per-board - headers by `MICROPY_BOARD`, which names upstream's generic boards here. Ours - takes them from one of esp-vision's boards per chip - (`ESP32_P4X_FUNCTION_EV_BOARD`, `ESP32_S3_EYE`). -- **[`src/vision_preview_none.c`](../modules/vision/src/vision_preview_none.c)** stands in for esp-vision's preview, which - streams frames to its IDE over its own USB multiplexer. `image.flush()` - raises `OSError("preview flush failed: ESP_ERR_NOT_SUPPORTED")`. - -## Proven on hardware (2026-10-06) - -| Test | P4 DEV-KIT | S3, octal PSRAM | -|---|---|---| -| `tflite`: TFLM's own hello-world test (its int8 model, four golden inputs, 0.05 tolerance) | pass, worst error 0.016 | pass, outputs identical | -| `tflite`: esp-vision's sine model, per prediction | 104 µs | 397 µs | -| `vision`: [`examples/vision_smoke.py`](../modules/vision/examples/vision_smoke.py) (draw, read back, `find_blobs`, JPEG out and in, `flush`) | pass; JPEG 42 ms | pass; JPEG 84 ms | -| `espdl`: ESP-DL's pedestrian example photo, against ESP-DL's published boxes | pass, 68 ms, IoU ≥ 0.90 | pass, 176 ms, IoU ≥ 0.92 | -| `h264`: 30 frames decoded on the PC by ffmpeg | Constrained Baseline, frames as drawn | (P4 only) | -| `rtsp`: ffmpeg on the PC pulls the stream | 151 frames in 10 s at 15 fps | (P4 only) | - -Each test fails when it should: the hello-world test with its inputs offset, -the pedestrian test with one person blacked out. - -## What it costs - -Which is why they're opt-in (since 2026-10-06). Built into an `all` image on -2026-10-06 they added about 2.1 MB on the S3 (the LCD-7's app partition grew -from 0x440000 to 0x6b0000, leaving about 1.2 MB of filesystem on 8 MB of -flash) and about 3.1 MB on the P4 (`firmware.bin` 4.53 MB to 7.68 MB of -16 MB). Most of it is ESP-DL. The desktop ports skip both. - -## Not in yet - -- **`sensor`** (the camera). esp-vision's camera code is per board, and every - P4 board's `camera.c` uses Espressif's Board Manager (its YAML board - definitions). The other route: esp-vision's generic `platform/camera.c` - declares the camera functions weak, so our glue could provide them from - cameraif, which already drives the OV5647. That is a decision, not glue. -- **`display`**. esp-vision's display code also needs Board Manager, and - displayif already does displays here. -- **One `esp_h264`.** h264enc pins 1.4.1 and `vision` takes the same; - esp-vision uses 1.3.0. `sensor` would bring `esp_video`, which requires - 1.3.*, so a firmware with h264enc and a camera via `esp_video` doesn't - resolve as things stand. - -## Traps - -- An mpftp `exec` that outlasts mpftp's wait resets the board over USB - (`rst:0x17 CHIP_USB_UART_RESET`). Under a long-running script that looks like - the firmware crashing. Run such scripts as `main.py` or with nothing attached. -- The prebuilt RTSP library calls `ESP_ERROR_CHECK`'s handler, which nothing - else references with MicroPython's assertions off. `vision` links it with - `-u _esp_error_check_failed`, the way the port does for `abort_`. -- ESP-DL's headers are C++20. The module's C++ compiles with - `-std=gnu++2b`, esp-vision's own setting. diff --git a/docs/firmware-release.md b/docs/firmware-release.md deleted file mode 100644 index b9f3b304..00000000 --- a/docs/firmware-release.md +++ /dev/null @@ -1,41 +0,0 @@ -# Firmware sets - -A firmware set is every board's flashable file, built from one commit of this -repository by the **Release firmware** workflow -([`release-firmware.yml`](../.github/workflows/release-firmware.yml); the cells -are in [`scripts/release_firmware.py`](../scripts/release_firmware.py)). -Each cell is one `build_mp.py` call; the `assemble` job checks every file -(checksums, the ESP image magic, the app image inside the flash image, UF2 -blocks) and writes `firmware.json`, which records every file's sha256 and size, -the flash offset, and the pins the set was built from. - -## Build a set - -Actions > Release firmware > Run workflow. A pull request that touches the -recipe, and a run every Monday, build the same cells and keep the files as a -run artifact for 14 days. They publish nothing. - -## Put a set on a draft Release - -Tick **draft_release** and give a **tag**. After `assemble`, the `draft` job -attests every file (`gh attestation verify --repo PyDevices/micropython-pydevices`) -and creates a *draft* GitHub Release holding the flash images, the `.elf` and -`.map` of each build, and `firmware.json`. A draft is seen only by people who -can push to the repository, and GitHub makes no tag until someone publishes it. - -The tag is `v-`. A set built from a preview -MicroPython or CircuitPython only takes a tag that starts with `staged-`, so a -preview set can't be published by accident. - -## The download page - -[`scripts/stage_firmware_page.py`](../scripts/stage_firmware_page.py) lays a -set out for the site's `firmware/` folder: `firmware.json`, the flash images -the page offers and the browser flasher fetches, an ESP Web Tools manifest per -ESP32 build, and the desktop interpreters' `interpreters.json` from the newest -pydevices final that has one. - -``` -python3 scripts/stage_firmware_page.py /firmware \ - --interpreters-from-release PyDevices/pydevices -``` diff --git a/docs/get-started/first-program.md b/docs/get-started/first-program.md new file mode 100644 index 00000000..f6c2b069 --- /dev/null +++ b/docs/get-started/first-program.md @@ -0,0 +1,48 @@ +# Build your first firmware + +This builds MicroPython for your desktop with the pygraphics module compiled +in, then runs a program on it. It needs no board and only the unix packages +from [Install](install.md), and it proves the build works before you spend an +hour on an esp32 one. + +From the clone: + +```bash +./build_mp.py --port unix --modules pygraphics +``` + +The first run fetches MicroPython and the module. When it ends it prints where +the interpreter is: + +```text +Built into /home/you/micropython-pydevices/builds/unix/standard + /home/you/micropython-pydevices/builds/unix/standard/micropython +``` + +Save this as `hello.py`: + +```python +import sys +import pygraphics + +fb = pygraphics.FrameBuffer(bytearray(160 * 80 * 2), 160, 80, pygraphics.RGB565) +area = fb.round_rect(8, 8, 64, 40, 8, 0xF800, f=True) +print(sys.implementation.name, pygraphics.implementation(), area) +``` + +Run it with the interpreter you built: + +```bash +builds/unix/standard/micropython hello.py +``` + +```text +micropython native_cmod Area(8, 8, 64, 40) +``` + +`native_cmod` says pygraphics is the compiled module: the C code you named +with `--modules` is inside the interpreter. Beside the interpreter, the build +folder holds `pydevices-build.json`, a record of every module's commit, so +you can tell later what a firmware was made from. + +Now build for a board: [Build for a board](../guides/build-for-a-board.md). diff --git a/docs/get-started/install.md b/docs/get-started/install.md new file mode 100644 index 00000000..b52e488e --- /dev/null +++ b/docs/get-started/install.md @@ -0,0 +1,72 @@ +# Install + +You build firmware here, so "install" means getting the repository and the +packages your port's compiler needs. Nothing is published to PyPI. + +## Get the repository + +On Linux, WSL, or Git Bash on Windows: + +```bash +curl -fsSL https://pydevices.github.io/install.sh | sh +``` + +It clones the repository into `micropython-pydevices` in the current directory +and checks your tools. It prints only what your machine is missing, one +`apt-get install` line per port, so you can leave out ports you won't build. +It can install a port's packages for you, with `sh -s -- --deps rp2` (or +`all`) in place of the plain `sh`. + +Or clone it yourself: + +```bash +git clone https://github.com/PyDevices/micropython-pydevices.git +cd micropython-pydevices +``` + +Native Windows builds, from PowerShell, are in +[Build on Windows](../guides/build-on-windows.md). To build with nothing +installed at all, use [a codespace](../guides/build-in-a-codespace.md). + +## Install what each port needs + +`build_mp.py` fetches the big pieces itself: MicroPython, the modules, ESP-IDF, +emsdk, the pico-sdk and picotool, and SDL2 for MinGW. What it can't fetch are +your distribution's packages. On Debian or Ubuntu (WSL included): + +| Port | Packages | +|---|---| +| every port | `git make cmake python3 gcc g++` | +| unix | `pkg-config libsdl2-dev libffi-dev` (the desktop display) | +| esp32 | `ninja-build python3-venv libusb-1.0-0` (ESP-IDF's installer checks the openocd it fetches, which needs libusb) | +| rp2 | `gcc-arm-none-eabi libnewlib-arm-none-eabi` | +| windows, cross-built from Linux | `gcc-mingw-w64-x86-64 binutils-mingw-w64-x86-64 libtool libltdl-dev autoconf automake` (the binutils package has `windres`) | +| webassembly | `nodejs` | + +All of it in one line: + +```bash +sudo apt-get update && sudo apt-get install -y git make cmake ninja-build \ + python3 python3-venv python3-pip libusb-1.0-0 gcc g++ pkg-config libffi-dev \ + libsdl2-dev gcc-mingw-w64-x86-64 binutils-mingw-w64-x86-64 libtool \ + libltdl-dev autoconf automake \ + gcc-arm-none-eabi libnewlib-arm-none-eabi nodejs +``` + +Other distributions carry the same tools under their own names. A build for a +port that is still missing something stops and says so. + +## Room on disk + +The first run of each port fetches what it needs. Where the repositories are +already checked out side by side, `build_mp.py` links those sibling checkouts +instead of fetching. + +An esp32 build's first run clones ESP-IDF into the clone's `deps/` and +installs its toolchains, for every ESP chip, under `~/.espressif`, outside the +clone (set `IDF_TOOLS_PATH` to put them elsewhere). The toolchains stay put, +but ESP-IDF goes with the clone: move the clone, or clone again, and the next +esp32 build fetches ESP-IDF again. Each build folder takes room in the clone +too. + +Next: [build your first firmware](first-program.md). diff --git a/docs/guides/add-a-board-variant.md b/docs/guides/add-a-board-variant.md new file mode 100644 index 00000000..a7b48eae --- /dev/null +++ b/docs/guides/add-a-board-variant.md @@ -0,0 +1,68 @@ +# Add a board variant + +Most boards need no variant: pick the upstream board that matches the chip and +give `--flash` its flash size ([Build for a board](build-for-a-board.md)). Make +a variant only for settings that really are one board's, such as the LCD-7's +cache sizes and tick. + +## An esp32 board + +On esp32, `build_mp.py` appends our ESP-IDF settings after the board's, so ours +win. They come from files it finds by name, each used only if it exists, in +this order: + +1. `variants/esp32/sdkconfig`, then `variants/esp32/sdkconfig.` + (`esp32s3`, `esp32p4`): what every build, or every build for that chip, + wants. The S3's PSRAM, Wi-Fi and lwIP tuning and the P4's BLE-over-C6 + settings are here. +2. The same two names at the root of each module you selected: what that module + needs. usbhost's USB host settings and cameraif's OV5647 driver live in + their own module directories. +3. `variants/esp32///sdkconfig`: what's left that is one board's + alone. + +A variant directory of ours also holds an `mpconfigvariant.cmake` that includes +the upstream variant it builds on, which is how `LCD_7` and `T_EMBED` become +variants of `ESP32_GENERIC_S3`, and `WIFI6_DEV_KIT` one of `ESP32_GENERIC_P4`. + +It may also hold: + +- An `mpconfigboard.h`, included before the board's own, for C defines the + port's ESP-IDF components must see too. `WIFI6_DEV_KIT`'s sets + `MICROPY_HW_USB_HS (0)`, which moves the USB device to the P4's full-speed + controller, so other P4 boards keep theirs on the high-speed one. +- C sources, through `MICROPY_SOURCE_BOARD` in its `mpconfigvariant.cmake`, the + way upstream's `ESP32_GENERIC_C2` does. `WIFI6_DEV_KIT_H2` builds on + `WIFI6_DEV_KIT` and adds a board startup hook that swaps the P4's two + full-speed PHYs, so the USB device comes out on the kit's USB-C marked "USB". + +The partition table is found the same way, the most specific `partitions.csv` +winning ([`variants/esp32/partitions.esp32s3.csv`](https://github.com/PyDevices/micropython-pydevices/blob/main/variants/esp32/partitions.esp32s3.csv) +adds a coredump partition on the S3, which a standard build leaves unused). +Without one, the board's own is used. Either way, autosize grows the app +partition when the image doesn't fit. + +## An rp2 board + +An rp2 board of ours is a variant of one of upstream's, in +`variants/rp2///`. The port only knows variants that sit in the +board's own folder, so `build_mp.py` makes a folder of its own for the build, +`builds/rp2////`, links upstream's board into it +unchanged and adds what the variant holds: + +- `mpconfigvariant.cmake` (required): cmake settings, included after the + board's own default variant. The board's `PICO_BOARD`, the number of GPIOs, + PSRAM and the size of the filesystem go here. +- `mpconfigboard.h`: C defines, included after the board's own header, so it + can `#undef` what the board sets, such as its name. +- `pins.csv`: the pin names, in place of the board's. + +`PIMORONI_PICO_PLUS2_W` is the one rp2 variant of ours. It builds the Pico 2 W for the +Pimoroni board's 16 MB flash, 8 MB PSRAM, 48 GPIOs and its own pin names, and +its files say why each setting is there. + +## Desktop variants + +The unix and windows ports take our `pydevices` and `vst3-engine` variants, and +webassembly takes `pydevices`. [The variants table](../reference/boards.md#variants) +lists them. diff --git a/docs/guides/build-a-troubleshooting-image.md b/docs/guides/build-a-troubleshooting-image.md new file mode 100644 index 00000000..45e87652 --- /dev/null +++ b/docs/guides/build-a-troubleshooting-image.md @@ -0,0 +1,32 @@ +# Build a troubleshooting image with a core dump + +Standard builds, including every image on the download page, don't write a core +dump to flash when the firmware crashes. When you need one, to find out why a +board resets or hangs, add `--coredump` on esp32: + +```bash +./build_mp.py --port esp32 --board ESP32_GENERIC_S3 --variant SPIRAM_OCT --flash 8MB --modules all --coredump +``` + +It builds into its own folder, `builds/esp32//-coredump/`, so it +never replaces your standard image. + +## Read the dump + +After a crash the board keeps the dump in its `coredump` partition across the +reboot. Read it back with `esptool` and `esp-coredump` +(`esp_coredump info_corefile`) against that build's `micropython.elf`. The +decode only matches the ELF of the same commit and options, so build the +troubleshooting image from the commit your board runs. + +## Which boards can do it + +The dump needs a `coredump` partition, and only the ESP32-S3 table has one. A +standard S3 build leaves it unused, which is why the filesystem sits where it +always did, and a troubleshooting build uses the same table: you can flash its +app image (`micropython.bin`, at the app offset) onto a board running a +standard image and keep the files on it. + +On a board whose table has no such partition, such as the ESP32-P4, +`--coredump` stops with "this board's table has no coredump partition yet" +instead of moving your filesystem. diff --git a/docs/guides/build-for-a-board.md b/docs/guides/build-for-a-board.md new file mode 100644 index 00000000..551225fa --- /dev/null +++ b/docs/guides/build-for-a-board.md @@ -0,0 +1,41 @@ +# Build for a board + +Every board here is one of upstream's, with at most a small variant of ours on +top. Find yours in [the boards table](../reference/boards.md), which gives the +`--port`, `--board`, `--variant` and `--flash` to use, and add `--modules`: + +```bash +./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI \ + --flash 16MB --modules displayif,pygraphics,/home/you/earful +``` + +You name the modules you want: ours by short name, anything else by its path, +or `all`. [Choose modules](choose-modules.md) says what each name brings. Leave +out `--port` and the script asks for port, board, variant, flash size and +modules. + +Output lands in `builds//[/]/`, beside +`pydevices-build.json`, a record of every module's commit. With no +`--variant`, that last directory is `default` on esp32 and rp2, and `standard` +on the desktop ports. [Flash a board](flash-a-board.md) says where +`firmware.bin` goes. + +## A board that isn't in the table + +A new board usually needs no file at all. Pick the upstream board and variant +that match its chip and PSRAM, and give `--flash` its flash size. Make a +variant of your own only for settings that really are that board's: +[Add a board variant](add-a-board-variant.md). + +## When the image doesn't fit + +On esp32 and rp2 the build can grow the firmware's share of flash to fit the +image, which moves the filesystem. [Fit firmware on a small flash](fit-firmware-on-a-small-flash.md) +says when that happens and how to stop it. + +## When the first link fails + +A first link that fails on size, in red, is expected on esp32 and rp2 with +`--modules all`: the build grows the partition and builds again. It, and the +other output that looks wrong and isn't, is in +[Read build warnings](read-build-warnings.md). diff --git a/docs/guides/build-for-circuitpython.md b/docs/guides/build-for-circuitpython.md new file mode 100644 index 00000000..e4c1a0e6 --- /dev/null +++ b/docs/guides/build-for-circuitpython.md @@ -0,0 +1,42 @@ +# Build for CircuitPython + +The same command builds OmniPython CP (CircuitPython-compatible) firmware: +CircuitPython with our C modules compiled in, for CircuitPython's own ports and +boards. Add `--interpreter circuitpython`: + +```bash +./build_mp.py --interpreter circuitpython --port raspberrypi \ + --board adafruit_feather_rp2040 --modules pygraphics +./build_mp.py --interpreter circuitpython --port unix --modules pygraphics +``` + +Ports and boards are CircuitPython's own (`raspberrypi`, `atmel-samd`, +`espressif`, `unix` and the rest), with boards named by their directories +under `ports//boards/`. Boards CircuitPython has no definition for are +ours: [the boards table](../reference/boards.md#circuitpython-boards). + +Output lands in `builds/circuitpython///`, with the +same build record. The unix port builds CircuitPython's default variant, +`coverage`, unless you name another. + +It takes C modules only. [The modules table](../reference/modules.md) says +which build on CircuitPython and why a module is left out. + +## Make arguments + +Arguments the script doesn't recognise go to make. `CIRCUITPY_ULAB=0`, for +example, frees flash on a small board: + +```bash +./build_mp.py --interpreter circuitpython --port espressif \ + --board adafruit_qtpy_esp32_pico --modules audiodsp CIRCUITPY_ULAB=0 +``` + +They are part of the build record: a build with different ones starts from an +empty build directory, because CircuitPython's generated module table would +otherwise keep the old set. + +## Builds from Linux or WSL only + +A CircuitPython build runs on Linux or WSL. How it differs from a MicroPython +build is in [How a CircuitPython build differs](../how-it-works/circuitpython.md). diff --git a/docs/guides/build-in-a-codespace.md b/docs/guides/build-in-a-codespace.md new file mode 100644 index 00000000..d6c0299e --- /dev/null +++ b/docs/guides/build-in-a-codespace.md @@ -0,0 +1,43 @@ +# Build in a codespace + +You can build without installing anything, in a GitHub codespace or in a dev +container on your own machine. Each port has its own container, holding only +that port's packages: unix, webassembly, rp2, windows and esp32. + +[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/PyDevices/micropython-pydevices) + +## Open one + +On GitHub, open the repository's **Code** menu, then **Codespaces**, **...**, +**New with options**, and choose your port under "Dev container +configuration". With VS Code and Docker on your own machine, open your clone +and run **Dev Containers: Reopen in Container**, which asks for the port. + +## Build + +When the container is ready, its terminal prints the build line for its port, +and you build as you would anywhere else: + +```bash +./build_mp.py --port rp2 --board RPI_PICO2 --modules all +``` + +The first build fetches the port's toolchain into `deps/`, a volume that +outlives the container, so rebuilding the container doesn't fetch it again. +The esp32 container fetches ESP-IDF and its compilers while it's being +created, so your first build starts compiling; it asks for a 4-core machine. + +## Get the file out + +To flash, take the file to the machine your board is plugged into. In the +Explorer, open `builds/` down to your build, right-click `firmware.bin` (or +`firmware.uf2` on rp2) and choose **Download**. Then write it as +[Flash a board](flash-a-board.md) says. + +## Try a webassembly build in workbench + +The webassembly container opens your build in workbench, the browser IDE. +Build, then run `scripts/serve_workbench.sh`: it downloads the newest workbench +release, puts your `micropython.wasm` in it and serves it on port 8000, which +the container forwards and your browser opens. Click the simulator button (the +display icon) and the REPL that starts is your build. diff --git a/docs/guides/build-on-windows.md b/docs/guides/build-on-windows.md new file mode 100644 index 00000000..b01507df --- /dev/null +++ b/docs/guides/build-on-windows.md @@ -0,0 +1,52 @@ +# Build on Windows + +A Linux or WSL machine cross-builds the same `micropython.exe` with the same +command, `./build_mp.py --port windows --variant pydevices --modules all`, +once the MinGW packages in [Install](../get-started/install.md#install-what-each-port-needs) +are installed; the SDL2 it links against is fetched. This page is for building +on Windows itself. + +You can build the windows port natively, from PowerShell, with MSYS2's MinGW +toolchain. You need Python from python.org and Git for Windows; the rest is a +one-time install of MSYS2 and five of its packages: + +```powershell +winget install --id MSYS2.MSYS2 -e +C:\msys64\usr\bin\bash.exe -lc "pacman -Syu --noconfirm" +C:\msys64\usr\bin\bash.exe -lc "pacman -S --needed --noconfirm make mingw-w64-x86_64-gcc autoconf automake libtool" +``` + +If the first `pacman` line says it updated MSYS2's core and must close, run it +once more. Then, in any PowerShell: + +```powershell +git clone https://github.com/PyDevices/micropython-pydevices.git +cd micropython-pydevices +python build_mp.py --port windows --variant pydevices --modules all +.\builds\windows\pydevices\micropython.exe +``` + +That's all you type. `build_mp.py` finds MSYS2 in `C:\msys64` (set +`MSYS2_ROOT` if it's elsewhere), puts its tools first on the build's `PATH`, +and uses its `bash`. Left to `PATH`, `bash` from PowerShell is WSL's, which +would run the build inside Linux. Without MSYS2 it stops and prints the install +lines above. + +`micropython.exe` needs only Windows' own DLLs, so you can run it from any +shell, MSYS2 or not. + +## What differs from Linux + +- Type `python build_mp.py`, not `./build_mp.py`: Windows doesn't read the + `#!` line, and python.org's Python is `python` or `py`, never `python3`. If + `python` opens the Microsoft Store, turn off the `python.exe` and + `python3.exe` App Execution Aliases (Settings, Apps, Advanced app settings, + App execution aliases), or type `py` instead. +- Use `--variant pydevices` for the Windows display and audio backends + (`displaydev.windisplay`, `audiodev.win_audio`). They need `uwin32`, which + only that variant freezes, so on `dev` or the default they fail to import. +- Only the windows port builds natively. Build esp32, rp2, webassembly and + unix from WSL, as on Linux. +- The installer, `curl -fsSL https://pydevices.github.io/install.sh | sh`, runs + in Git Bash (Start menu, Git Bash), not PowerShell. A plain `git clone`, as + above, does the same job. diff --git a/docs/guides/choose-modules.md b/docs/guides/choose-modules.md new file mode 100644 index 00000000..7de6eb10 --- /dev/null +++ b/docs/guides/choose-modules.md @@ -0,0 +1,42 @@ +# Choose modules + +`--modules` takes a comma-separated list: + +```bash +./build_mp.py --port rp2 --board RPI_PICO2 --modules displayif,jpegio,pygraphics,palettes,pdwidgets +``` + +A name is one of three things: + +- **A short name** from [the modules table](../reference/modules.md): + `displayif`, `pygraphics`, `pydevices`. +- **A path**, to a module of your own: a directory with a `manifest.py`, or a + C module with its `micropython.mk` or `micropython.cmake`. +- **`all`**: every module except the opt-in ones. `tflite` and `vision` are + opt-in (each has an `OPT_IN` file saying why), so name them to get them: + [Use esp-vision's modules](use-esp-vision-modules.md). + +A module never brings another, and nothing checks dependencies. If you name +`audiocomponents`, `tflite` or `vision`, name `ulab` too, or use `all`. + +## Python packages from omnipython-lib + +Our Python-only packages come from omnipython-lib: `pydevices`, `palettes`, +`pdwidgets` and `audiocomponents` are modules whose manifest requires them from +it. Any other package of it, such as `appkit` or a board config, can be named +too. + +## The desktop board config + +The desktop `pydevices` variants freeze a board config that imports +`displaydev`. `displaydev` is frozen only when `--modules` names `pydevices` or +`all`, so without one of them `import board_config` fails: + +```bash +./build_mp.py --port unix --variant pydevices --modules pydevices,displayif,pygraphics +``` + +## A smaller image + +`all` is large. On a small flash, name only what your project uses: +[Fit firmware on a small flash](fit-firmware-on-a-small-flash.md). diff --git a/docs/guides/fit-firmware-on-a-small-flash.md b/docs/guides/fit-firmware-on-a-small-flash.md new file mode 100644 index 00000000..535904af --- /dev/null +++ b/docs/guides/fit-firmware-on-a-small-flash.md @@ -0,0 +1,37 @@ +# Fit firmware on a small flash + +On esp32 and rp2 the build can grow the firmware's share of flash to fit the +image: on esp32 the app partition, on rp2 the region before the filesystem. It +leaves 1/32 of the flash as headroom, 64 KB to 256 KB. That moves the +filesystem, so a board flashed with that image comes up with an empty one. + +To refuse the growth and stop instead, add `--no-autosize`. + +On esp32, autosize works only if the flash size is right: the partition table +tool refuses a table larger than the configured flash, so give `--flash`. + +## What you will see + +The build finds out by trying. The first link fails, loudly, and only then does +it build again. On esp32 that is a red `Error: app partition is too small for +binary micropython.bin size 0x…`, then `ninja: build stopped`, `make: *** +Error 1` and a Build-Troubleshooting link. On rp2 it is ``region `FLASH' +overflowed by … bytes``, `collect2: error: ld returned 1 exit status` and +nested `Error 2` lines. Then an `autosize:` line says the image didn't fit and +the build is running once more. With `all`, expect that second pass on every +esp32 and rp2 board; it roughly doubles the build time, because the whole esp32 +compile runs twice. + +## When nothing will fit + +How much fits depends on the flash. On a Pico 2 (4 MB), `all` builds and fits, +at 93 % of flash with a 1600 KB filesystem. On a Pico (2 MB) it can't: the +image is bigger than the whole flash, so the build stops at the first link with +the overflow error. Name what your project uses instead: + +```bash +./build_mp.py --port rp2 --board RPI_PICO --modules displayif,jpegio,pygraphics,palettes,pdwidgets +``` + +A display project's set like this one builds to about 540 KB and fits with the +Pico's default 1408 KB filesystem untouched. diff --git a/docs/guides/flash-a-board.md b/docs/guides/flash-a-board.md new file mode 100644 index 00000000..81de1725 --- /dev/null +++ b/docs/guides/flash-a-board.md @@ -0,0 +1,38 @@ +# Flash a board + +## ESP32 + +An esp32 build ends with ESP-IDF's `To flash, run: idf.py flash`. That isn't +the route here: `idf.py` lives in the ESP-IDF environment the build sets up, +not on your shell's `PATH`. Write `firmware.bin` with esptool (`pip install +esptool`) instead. It's the merged image, with bootloader, partition table and +app in one, so it goes where the bootloader goes: 0x2000 on the P4, 0x0 on the +S3. For another chip, the build folder's `flasher_args.json` has the +bootloader's offset. + +```bash +esptool --chip esp32p4 write-flash 0x2000 builds/esp32/ESP32_GENERIC_P4/WIFI6_DEV_KIT_H2/firmware.bin +esptool --chip esp32s3 write-flash 0x0 builds/esp32/ESP32_GENERIC_S3/SPIRAM_OCT/firmware.bin +``` + +## RP2 + +Hold BOOTSEL while you plug the board in, and copy `firmware.uf2` to the drive +it shows. + +## With mpftp + +mpftp writes either file for you, given the build's port and your board's +serial port: + +```bash +mpftp firmware flash --port esp32 --artifact firmware.bin -d /dev/ttyACM0 +mpftp firmware flash --port rp2 --uf2 --artifact firmware.uf2 +``` + +For a `.uf2`, mpftp copies it to the board's drive instead. + +## After flashing + +pydevices' [Bringing up a board](https://github.com/PyDevices/pydevices/blob/main/docs/board-bringup.md) +takes a flashed board to one drawing on its own panel. diff --git a/docs/guides/read-build-warnings.md b/docs/guides/read-build-warnings.md new file mode 100644 index 00000000..f50a85b0 --- /dev/null +++ b/docs/guides/read-build-warnings.md @@ -0,0 +1,52 @@ +# Read build warnings + +A build prints a few things that look wrong and aren't. + +## Every port + +Every esp32 and rp2 build with `all` fails its first link on size, then autosize +builds again: [Fit firmware on a small flash](fit-firmware-on-a-small-flash.md) +shows what that looks like. That failure's last line can read +`-e See [1;31mhttps://…/Build-Troubleshooting[0m`. Upstream's Makefile prints it +with `echo -e`, and where `/bin/sh` is dash, as on Debian and Ubuntu, the `-e` +comes out as text and a log shows the colour codes raw. Nothing is broken. + +## esp32 + +- `warning: unknown kconfig symbol 'XTAL_FREQ_AUTO'` and a run of + `CONFIG_… was replaced with CONFIG_…` lines: ESP-IDF renamed those settings + and still maps the old names. +- `Component lockfile contents have changed`, a CMake warning that asks you to + mention it in any issue. The build uses its own TinyUSB and a newer ESP-IDF + than upstream's lockfile records; `build_mp.py` puts the lockfile back + afterwards. +- cmake's `Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed` and `Could NOT find + Threads`. +- On the P4, `Detecting C compiler ABI info - failed` and the same for CXX, each + checked again a moment later and reported `done`. +- On the first build, `WARNING: Download failure: HTTP Error 502` from ESP-IDF's + tool installer, which tries again by itself. + +## rp2 + +- `No installed picotool with version 2.3.0 found - building from source`, once + per pass. +- On the first build, `PICO_TINYUSB_PATH specified, but does not contain + hw/bsp/rp2040; USB support will be unavailable`. It comes from the pass that + fetches submodules, before TinyUSB is checked out; the real build says + `TinyUSB available` and builds it. +- `PICO_BTSTACK_PATH specified but content not present` and the same for + `PICO_LWIP_PATH`, on boards without networking or Bluetooth, which don't + fetch those libraries. + +## windows + +- libffi's configure says `libtool.m4: error: problem compiling CXX test + program`; it is probing for a C++ compiler it doesn't need, and the build goes + on. +- Cross-built from Linux, `configure: WARNING: using cross tools not prefixed + with host triplet`: there's no MinGW-named `file` command, so it uses the + plain one. +- Autoconf's ``The macro `AC_TRY_COMPILE' is obsolete`` and ``The preprocessor + macro `STDC_HEADERS' is obsolete``, and four `… is deprecated + [-Wdeprecated-declarations]` warnings, all from libffi's own sources. diff --git a/docs/guides/use-esp-vision-modules.md b/docs/guides/use-esp-vision-modules.md new file mode 100644 index 00000000..680c1fe4 --- /dev/null +++ b/docs/guides/use-esp-vision-modules.md @@ -0,0 +1,35 @@ +# Use esp-vision's modules + +Two modules come from [espressif/esp-vision](https://github.com/espressif/esp-vision) +(Apache-2.0; its imlib is OpenMV's, MIT), compiled from esp-vision's own files +without changes. Both are ESP32-P4 and ESP32-S3 only, both need ulab, and both +are opt-in: `--modules all` leaves them out, so name them: + +```bash +./build_mp.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB --modules ulab,tflite,vision +``` + +- **`tflite`**: TensorFlow Lite Micro inference (`tflite.Model`), the API + OpenMV's `ml` module uses. +- **`vision`**: the OpenMV-style `image` API (`image`, `imageio`), ESP-DL + inference (`espdl`), and on the P4 the hardware H.264 encoder (`h264`) and an + RTSP server (`rtsp`). + +esp-vision's stubs (`stubs/*.pyi` in esp-vision) document the API. The desktop +ports skip both modules, and they build for MicroPython only. + +Built into an `all` image they add about 2 MB on the S3 and 3 MB on the P4, +most of it ESP-DL, so check [that the image fits](fit-firmware-on-a-small-flash.md). + +`image.flush()` raises `OSError("preview flush failed: ESP_ERR_NOT_SUPPORTED")`, +because esp-vision's preview streams frames to its own IDE and this build has no +such connection. + +## Run long scripts as `main.py` + +An mpftp `exec` that outlasts mpftp's wait resets the board over USB +(`rst:0x17 CHIP_USB_UART_RESET`). Under a long-running script that looks like +the firmware crashing. Run such scripts as `main.py`, or with nothing attached. + +How the modules are built into the firmware is in +[How esp-vision's modules are built](../how-it-works/esp-vision.md). diff --git a/docs/how-it-works/circuitpython.md b/docs/how-it-works/circuitpython.md new file mode 100644 index 00000000..b5400205 --- /dev/null +++ b/docs/how-it-works/circuitpython.md @@ -0,0 +1,66 @@ +# How a CircuitPython build differs + +`--interpreter circuitpython` is a flag rather than a second command because +everything around the interpreter is shared: the module names and +`modules.lock`, the workspace links, the output layout and the build record. +What differs is one function, and these are its parts. + +- **The checkout.** CircuitPython is pinned in `deps.lock` like a toolchain and + cloned into `deps/circuitpython`, always our own clone, never a link to a + sibling checkout. A build fetches only the submodules its target needs, with + CircuitPython's own `tools/ci_fetch_deps.py` (a board's name, or `tests` for + unix), and its build tools go in `deps/circuitpython-venv` from its + `requirements-dev.txt`. +- **The modules go in as `USER_C_MODULES`**, not a manifest. CircuitPython's + `py/py.mk` carries MicroPython's `USER_C_MODULES` block unchanged, so a + module's `micropython.mk` is the one our MicroPython builds read. CircuitPython + freezes Python its own way (`FROZEN_MPY_DIRS`), so a module with no + `micropython.mk` and no Python to freeze is refused rather than left out + quietly. Each module is handed to make through a link named for it, in + `usermods/` in the build directory: make puts a module's objects under a + directory named like the module's, and two worktrees checked out under the + same branch name would otherwise share one. +- **espressif brings its own ESP-IDF.** CircuitPython's espressif port builds + with the ESP-IDF in `ports/espressif/esp-idf`, not the `esp-idf` in + `deps.lock`. Before make, the build initialises that tree's submodules + shallowly (ESP-IDF's CMake would otherwise clone them with their whole + history), runs its `install.sh` for the board's chip (compilers go in + `~/.espressif`, shared with any ESP-IDF of the same version) and puts ESP-IDF's + Python requirements in `deps/circuitpython-venv`. +- **Our patches and boards go on first.** `patches/circuitpython/*.patch` are + applied to `deps/circuitpython` and each `boards/circuitpython//` + is linked into the port's `boards/`, once each; the build record lists the + patches. [The boards table](../reference/boards.md#circuitpython-boards) lists + the boards. +- **A module can ask espressif for ESP-IDF components.** CircuitPython's + espressif port builds and links a fixed list of ESP-IDF components, which + leaves out the camera's. With patch + `0006-espressif-esp-idf-components-a-user-c-module-asks-for.patch`, a module's + `micropython.mk` adds names to `USER_ESP_IDF_COMPONENTS` (cameraif: the CSI + controller, the ISP and the sensor drivers; jpegio and ppa: the P4's JPEG + engine and PPA; h264enc: `esp_h264`), and the make line's + `USER_ESP_IDF_COMPONENT_DIRS` adds components ESP-IDF doesn't ship. + CircuitPython builds with the component manager off, so for cameraif the build + fetches `esp_cam_sensor`, `esp_sccb_intf` and `cmake_utilities` at their + `deps.lock` versions into `deps/` and passes them, and `esp_h264` the same way + for h264enc. The DEV-KIT's board sdkconfig enables the OV5647. +- **mpy-cross is built first, on its own**, for the same reason as on + MicroPython ([the `BUILD=` trap](index.md#traps-the-build-already-handles)). +- **`BUILD=` is a short link.** make is given `BUILD=build-`, a link in + the port directory to our build directory: raspberrypi's link step echoes + every object path in one shell argument, and with a long absolute path in + front of each it passes Linux's 128 KB limit on one argument. + +## What a module sees + +The banner is CircuitPython's, unchanged: `Adafruit CircuitPython on +; with `. A module that has to behave differently here +can test `CIRCUITPY`, which every CircuitPython port sets, unix included. +pygraphics also passes its own define from its `micropython.mk`, so it knows it +came in as a user C module and not through CircuitPython's shared-bindings. + +Two things a module's C has to absorb on CircuitPython's MCU ports. They append +`-Werror` warnings (`sign-compare`, `float-equal`, `cast-align` on atmel-samd; +`float-equal`, `double-promotion`, `cast-align` on espressif) after the module's +flags, so a `-Wno-` meant for them goes on each object, where it lands last. And +atmel-samd has no C heap, so `malloc()` doesn't link there; use `m_malloc()`. diff --git a/docs/how-it-works/esp-vision.md b/docs/how-it-works/esp-vision.md new file mode 100644 index 00000000..cb4b8c20 --- /dev/null +++ b/docs/how-it-works/esp-vision.md @@ -0,0 +1,67 @@ +# How esp-vision's modules are built + +[Use esp-vision's modules](../guides/use-esp-vision-modules.md) says how to +build with `tflite` and `vision`. This page says what is behind them. + +## The pin and the glue + +[`modules/.esp-vision`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/.esp-vision) +pins esp-vision at a full commit in `ESP_VISION.lock`. Its `fetch.sh` links a +sibling clone at that commit (`../esp-vision`), or fetches that commit alone. +The leading dot keeps it out of `--modules` and `all`, because esp-vision's own +`micropython.cmake` is its whole firmware build, not a module. + +`modules/tflite` and `modules/vision` are our glue around esp-vision's files: +the ESP-IDF components esp-vision uses, at its versions; the include paths and +definitions MicroPython's QSTR scan needs (it sees only a module's own paths, +never a linked component's); and two pieces of our own. + +- **imlib compiles inside the module**, not as an ESP-IDF component. As a + component it would wait for MicroPython's generated headers while being one of + their inputs (a ninja dependency cycle). esp-vision's own `components/imlib` + CMakeLists can't be used anyway: it finds its per-board headers by + `MICROPY_BOARD`, which names upstream's generic boards here. Ours takes them + from one of esp-vision's boards per chip (`ESP32_P4X_FUNCTION_EV_BOARD`, + `ESP32_S3_EYE`). +- **`src/vision_preview_none.c`** stands in for esp-vision's preview, which + streams frames to its IDE over its own USB multiplexer. + +Two build details follow from esp-vision's code. The prebuilt RTSP library calls +`ESP_ERROR_CHECK`'s handler, which nothing else references with MicroPython's +assertions off, so `vision` links it with `-u _esp_error_check_failed`, the way +the port does for `abort_`. And ESP-DL's headers are C++20, so the module's C++ +compiles with `-std=gnu++2b`, esp-vision's own setting. + +## What was checked on hardware + +| Test | P4 DEV-KIT | S3, octal PSRAM | +|---|---|---| +| `tflite`: TFLM's own hello-world test (its int8 model, four golden inputs, 0.05 tolerance) | pass, worst error 0.016 | pass, outputs identical | +| `tflite`: esp-vision's sine model, per prediction | 104 µs | 397 µs | +| `vision`: [`examples/vision_smoke.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/modules/vision/examples/vision_smoke.py) (draw, read back, `find_blobs`, JPEG out and in, `flush`) | pass; JPEG 42 ms | pass; JPEG 84 ms | +| `espdl`: ESP-DL's pedestrian example photo, against ESP-DL's published boxes | pass, 68 ms, IoU ≥ 0.90 | pass, 176 ms, IoU ≥ 0.92 | +| `h264`: 30 frames decoded on the PC by ffmpeg | Constrained Baseline, frames as drawn | (P4 only) | +| `rtsp`: ffmpeg on the PC pulls the stream | 151 frames in 10 s at 15 fps | (P4 only) | + +Each test fails when it should: the hello-world test with its inputs offset, the +pedestrian test with one person blacked out. + +## What it costs + +Built into an `all` image, the two modules added about 2.1 MB on the S3 (the +LCD-7's app partition grew from 0x440000 to 0x6b0000, leaving about 1.2 MB of +filesystem on 8 MB of flash) and about 3.1 MB on the P4 (`firmware.bin` 4.53 MB +to 7.68 MB of 16 MB). Most of it is ESP-DL. That is why they are opt-in. + +## Not included + +- **`sensor`** (the camera). esp-vision's camera code is per board, and every P4 + board's `camera.c` uses Espressif's Board Manager (its YAML board definitions). + The other route: esp-vision's generic `platform/camera.c` declares the camera + functions weak, so our glue could provide them from cameraif, which already + drives the OV5647. +- **`display`**. esp-vision's display code also needs Board Manager, and + displayif already does displays here. +- **One `esp_h264`.** h264enc pins 1.4.1 and `vision` takes the same; esp-vision + uses 1.3.0. `sensor` would bring `esp_video`, which requires 1.3.*, so a + firmware with h264enc and a camera via `esp_video` doesn't resolve. diff --git a/docs/how-it-works/index.md b/docs/how-it-works/index.md new file mode 100644 index 00000000..ace01421 --- /dev/null +++ b/docs/how-it-works/index.md @@ -0,0 +1,85 @@ +# How it works + +This repository builds MicroPython-compatible and CircuitPython-compatible +firmware with the PyDevices C modules compiled in. It holds our patches to one +pinned MicroPython release, our variants and the build command. It is not a +fork, and it publishes no Python package. + +## The model + +```text +pinned MicroPython tag (UPSTREAM) + + patches/micropython/ and each module's own patches, one local commit + | + upstream board (esp32, rp2) or upstream / our variant (unix, windows, webassembly) + + on esp32: our sdkconfig, appended last (flash size, grown partition table, fragments) + | + modules/manifest.py: the port's own content, then the modules you named + | + the port's own make -> builds//[/]/ +``` + +Three choices make a build: the board (or not, on the desktop ports), the +variant (always optional, as with upstream's `make`), and the modules. A module +is a directory with a `manifest.py`, or a C module with its glue, like ulab. +[Choose modules](../guides/choose-modules.md) says how to name them. + +## Why the build tries and then grows + +On esp32 and rp2 the build can grow the firmware's share of flash to fit the +image. It finds out by trying: the first link fails, and only then does it +build again with a bigger partition. A saved `sdkconfig` in the build directory +would otherwise pin the wrong partition table for every run after a failed one, +so the build deletes it first and afterwards checks that the configured table is +the one it chose. [Fit firmware on a small flash](../guides/fit-firmware-on-a-small-flash.md) +is the user's side of this. + +## Rules + +- Upstream is **pinned**. Moving `UPSTREAM` re-validates every patch. +- Patches are justified in their headers. Never hand-edit a patched tree: change + the patch and prepare again. +- `build_mp.py` names no version and no setting. Pins move in the lock files, + and esp32 settings live in sdkconfig fragments it finds by convention. +- Publishing binaries is a separate decision from this source repository. + +## Traps the build already handles + +**`BUILD=` and the mpy-cross sub-make.** Passed on its own, `BUILD=` reaches the +mpy-cross sub-make and breaks an esp32 link +([micropython#19667](https://github.com/micropython/micropython/issues/19667)). +Both build systems skip that sub-make when `MICROPY_MPYCROSS` names a built +mpy-cross, so `build_mp.py` builds mpy-cross on its own first and exports its +path. + +**Upstream's generic esp32 boards give the app 1.94 MB** +(`partitions-4MiBplus.csv`), and the kitchen sink is about 3.4 MB. Autosize +handles that, but only if the flash size is right: gen_esp32part refuses a table +larger than the configured flash. Hence `--flash`. + +**`make clean` on esp32 is `idf.py fullclean`**, which deletes the whole port's +`managed_components/`. Never run it against a build directory that doesn't exist +yet. + +**Two esp32 builds at once in one port race** on `managed_components/`, so +`build_mp.py` builds esp32 one at a time. + +## The vst3-engine variants + +The `vst3-engine` variants are deliberately narrow. They set `MICROPY_PY_SOCKET`, +`MICROPY_PY_SSL` and `MICROPY_PY_FFI` to 0, so plugin content can't reach the +network or native libraries. A prepared tree carries the Windows networking and +FFI patches regardless; the variant is the guard. Don't swap it for a broader +desktop variant because that one builds. + +## More + +- [How a CircuitPython build differs](circuitpython.md) +- [How esp-vision's modules are built](esp-vision.md) + +```{toctree} +:hidden: + +circuitpython +esp-vision +``` diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..345b2014 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,32 @@ +# micropython-pydevices + +micropython-pydevices builds firmware: MicroPython-compatible and +CircuitPython-compatible interpreters with the PyDevices C modules compiled in, +for boards, desktops and the browser. One command, `build_mp.py`, does it. + +## Get started + +- [Install](get-started/install.md) +- [Build your first firmware](get-started/first-program.md) + +## Guides + +- [Build for a board](guides/build-for-a-board.md) +- [Choose modules](guides/choose-modules.md) +- [Flash a board](guides/flash-a-board.md) +- [Fit firmware on a small flash](guides/fit-firmware-on-a-small-flash.md) +- [Build in a codespace](guides/build-in-a-codespace.md) +- [Build on Windows](guides/build-on-windows.md) +- [Build for CircuitPython](guides/build-for-circuitpython.md) +- [Add a board variant](guides/add-a-board-variant.md) +- [Build a troubleshooting image with a core dump](guides/build-a-troubleshooting-image.md) +- [Use esp-vision's modules](guides/use-esp-vision-modules.md) +- [Read build warnings](guides/read-build-warnings.md) + +## Reference + +- [Reference](reference/index.md) + +## How it works + +- [How it works](how-it-works/index.md) diff --git a/docs/newcomers.md b/docs/newcomers.md deleted file mode 100644 index a37ce6dd..00000000 --- a/docs/newcomers.md +++ /dev/null @@ -1,422 +0,0 @@ -# Newcomer's guide to micropython-pydevices - -This repository builds MicroPython firmware with PyDevices modules in it. It -holds our patches to one pinned MicroPython release, our variants, and the -build command. It is not a fork, and it publishes no Python package. - -## Build something - -Get it, if you haven't: `curl -fsSL https://pydevices.github.io/install.sh | sh` -clones this repository into `micropython-pydevices` in the current directory -and checks your tools. On Windows, see [On Windows](#on-windows). -Then, from the clone: - -```bash -./build_mp.py --port unix --variant pydevices --modules pydevices,displayif,pygraphics -./build_mp.py --port esp32 --board ESP32_GENERIC_S3 --variant SPIRAM_OCT --flash 8MB --modules all -./build_mp.py # asks for port, board, variant, flash size and modules -``` - -The first run fetches what it needs. Where the repositories are already -checked out side by side, it links those sibling checkouts instead. An -esp32 build's first run also clones ESP-IDF into the clone's `deps/` (about -770 MB) and installs its toolchains, for every ESP chip, under -`~/.espressif`: about 4 GB, outside the clone (set `IDF_TOOLS_PATH` to put -them elsewhere), and a few minutes of the first esp32 build, more on a slow -connection. The toolchains stay put, but ESP-IDF goes with the clone: move it -or clone again and the next esp32 build fetches ESP-IDF again. Each build -folder takes room in the clone too; with `--modules all`, about 1 GB for a -P4, 700 MB for an S3, 250 MB for a Pico 2 and 80 MB for windows. Those four -builds leave the clone at about 3 GB, so plan on 7 GB with the toolchains. - -The desktop `pydevices` variants freeze a board config that imports -`displaydev`, which is frozen only when `--modules` names `pydevices` (the -repository whose `lib/` holds it) or `all`; without it `import board_config` -fails. - -## In a codespace - -You can build without installing anything, in a GitHub codespace or in a dev -container on your own machine. Each port has its own container, holding only -that port's packages: unix, webassembly, rp2, windows and esp32. - -On GitHub, open the repository's **Code** menu, then **Codespaces**, **...**, -**New with options**, and choose your port under "Dev container -configuration". With VS Code and Docker on your own machine, open your clone -and run **Dev Containers: Reopen in Container**, which asks for the port. - -When the container is ready, its terminal prints the build line for its port, -and you build as you would anywhere else: - -```bash -./build_mp.py --port rp2 --board RPI_PICO2 --modules all -``` - -The first build fetches the port's toolchain into `deps/`, a volume that -outlives the container, so rebuilding the container doesn't fetch it again. -The esp32 container fetches ESP-IDF and its compilers (about 4 GB) while it's -being created, so your first build starts compiling; it asks for a 4-core -machine. - -To flash, take the file to the machine your board is plugged into. In the -Explorer, open `builds/` down to your build, right-click `firmware.bin` (or -`firmware.uf2` on rp2) and choose **Download**. Then write it with mpftp, -giving the build's port with `--port` and your board's serial port with `-d`, -or with esptool as in [Flash it](#flash-it): - -```bash -mpftp firmware flash --port esp32 --artifact firmware.bin -d COM3 -``` - -For a `.uf2`, mpftp copies it to the board's drive instead: -`mpftp firmware flash --port rp2 --uf2 --artifact firmware.uf2`. - -The webassembly container opens your build in workbench, the browser IDE. -Build, then run `scripts/serve_workbench.sh`: it downloads the newest -workbench release, puts your `micropython.wasm` in it and serves it on port -8000, which the container forwards and your browser opens. Click the -simulator button (the display icon) and the REPL that starts is your build. - -## What each port needs - -[`build_mp.py`](../build_mp.py) fetches the big pieces itself: MicroPython, the modules, -ESP-IDF, emsdk, the pico-sdk and picotool, SDL2 for MinGW. What it can't -fetch are your distribution's packages. On Debian or Ubuntu (WSL included): - -| Port | Packages | -|---|---| -| every port | `git make cmake python3 gcc g++` | -| unix | `pkg-config libsdl2-dev libffi-dev` (the desktop display) | -| esp32 | `ninja-build python3-venv libusb-1.0-0` (ESP-IDF's installer checks the openocd it fetches, which needs libusb) | -| rp2 | `gcc-arm-none-eabi libnewlib-arm-none-eabi` | -| windows, cross-built from Linux | `gcc-mingw-w64-x86-64 binutils-mingw-w64-x86-64 libtool libltdl-dev autoconf automake` (the binutils package has `windres`) | -| webassembly | `nodejs` | - -All of it in one line: - -```bash -sudo apt-get update && sudo apt-get install -y git make cmake ninja-build \ - python3 python3-venv python3-pip libusb-1.0-0 gcc g++ pkg-config libffi-dev \ - libsdl2-dev gcc-mingw-w64-x86-64 binutils-mingw-w64-x86-64 libtool \ - libltdl-dev autoconf automake \ - gcc-arm-none-eabi libnewlib-arm-none-eabi nodejs -``` - -The installer prints only what your machine is missing, one `apt-get install` -line per port and marked with it, so you can leave out the ports you won't -build. It also says which ports build now, and it can install a port's -packages for you: `curl -fsSL https://pydevices.github.io/install.sh | sh -s --- --deps rp2` (or `all`). Other distributions carry the same -tools under their own names; a build for a port that's still missing -something stops and says so. - -### Warnings you can ignore - -A build prints a few things that look wrong and aren't: - -- Every esp32 and rp2 build with `all` fails its first link on size, then - autosize builds again. [The model](#the-model) says what that looks like. - That failure's last line can read `-e See [1;31mhttps://…/Build-Troubleshooting[0m`. - Upstream's Makefile prints it with `echo -e`, and where `/bin/sh` is dash, - as on Debian and Ubuntu, the `-e` comes out as text and a log shows the - colour codes raw. Nothing is broken. -- esp32: - - `warning: unknown kconfig symbol 'XTAL_FREQ_AUTO'` and a run of - `CONFIG_… was replaced with CONFIG_…` lines: ESP-IDF renamed those - settings and still maps the old names. - - `Component lockfile contents have changed`, a CMake warning that asks - you to mention it in any issue. The build uses its own TinyUSB and a - newer ESP-IDF than upstream's lockfile records; `build_mp.py` puts the - lockfile back afterwards. - - cmake's `Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed` and `Could - NOT find Threads`. - - On the P4, `Detecting C compiler ABI info - failed` and the same for - CXX, each checked again a moment later and reported `done`. - - On the first build, `WARNING: Download failure: HTTP Error 502` from - ESP-IDF's tool installer, which tries again by itself. -- rp2: - - `No installed picotool with version 2.3.0 found - building from - source`, once per pass. - - On the first build, `PICO_TINYUSB_PATH specified, but does not contain - hw/bsp/rp2040; USB support will be unavailable`. It comes from the pass - that fetches submodules, before TinyUSB is checked out; the real build - says `TinyUSB available` and builds it. - - `PICO_BTSTACK_PATH specified but content not present` and the same for - `PICO_LWIP_PATH`, on boards without networking or Bluetooth, which - don't fetch those libraries. -- windows: - - libffi's configure says `libtool.m4: error: problem compiling CXX test - program`; it is probing for a C++ compiler it doesn't need, and the - build goes on. - - Cross-built from Linux, `configure: WARNING: using cross tools not - prefixed with host triplet`: there's no MinGW-named `file` command, so - it uses the plain one. - - Autoconf's ``The macro `AC_TRY_COMPILE' is obsolete`` and ``The - preprocessor macro `STDC_HEADERS' is obsolete``, and four `… is - deprecated [-Wdeprecated-declarations]` warnings, all from libffi's own - sources. - -## On Windows - -This section is for building on Windows itself. A Linux or WSL machine -cross-builds the same `micropython.exe` with the same command, -`./build_mp.py --port windows --variant pydevices --modules all`, once the -MinGW packages in [What each port needs](#what-each-port-needs) are -installed; the SDL2 it links against is fetched. - -You can build the windows port natively, from PowerShell, with MSYS2's -MinGW toolchain. You need Python from python.org and Git for Windows; the -rest is a one-time install of MSYS2 and four of its packages: - -```powershell -winget install --id MSYS2.MSYS2 -e -C:\msys64\usr\bin\bash.exe -lc "pacman -Syu --noconfirm" -C:\msys64\usr\bin\bash.exe -lc "pacman -S --needed --noconfirm make mingw-w64-x86_64-gcc autoconf automake libtool" -``` - -If the first `pacman` line says it updated MSYS2's core and must close, run -it once more. Then, in any PowerShell: - -```powershell -git clone https://github.com/PyDevices/micropython-pydevices.git -cd micropython-pydevices -python build_mp.py --port windows --variant pydevices --modules all -.\builds\windows\pydevices\micropython.exe -``` - -That's all you type. `build_mp.py` finds MSYS2 in `C:\msys64` (set -`MSYS2_ROOT` if it's elsewhere), puts its tools first on the build's `PATH`, -and uses its `bash`. Left to `PATH`, `bash` from PowerShell is WSL's, which -would run the build inside Linux. Without MSYS2 it stops and prints the -install lines above. - -A few things differ from Linux: - -- Type `python build_mp.py`, not `./build_mp.py`: Windows doesn't read the - `#!` line, and python.org's Python is `python` or `py`, never `python3`. - If `python` opens the Microsoft Store, turn off the `python.exe` and - `python3.exe` App Execution Aliases (Settings, Apps, Advanced app - settings, App execution aliases), or type `py` instead. -- Use `--variant pydevices` for the Windows display and audio backends - (`displaydev.windisplay`, `audiodev.win_audio`). They need `uwin32`, which - only that variant freezes, so on `dev` or the default they fail to import. -- Only the windows port builds natively. Build esp32, rp2, webassembly and - unix from WSL, as on Linux. -- The installer, `curl -fsSL https://pydevices.github.io/install.sh | sh`, - runs in Git Bash (Start menu, Git Bash), not PowerShell. A plain - `git clone`, as above, does the same job. - -The `micropython.exe` it builds needs only Windows' own DLLs, so you can run -it from any shell, MSYS2 or not. - -## Boards we build for - -Every board here is one of upstream's, with at most a small variant of ours -on top. Add `--modules` to any of these: - -| Board | Command | -|---|---| -| Waveshare ESP32-P4-WIFI6-DEV-KIT, USB device on a USB-C wired to header P6 | `--port esp32 --board ESP32_GENERIC_P4 --variant WIFI6_DEV_KIT --flash 16MB` | -| Waveshare ESP32-P4-WIFI6-DEV-KIT, USB device on its USB-C marked "USB" | `--port esp32 --board ESP32_GENERIC_P4 --variant WIFI6_DEV_KIT_H2 --flash 16MB` | -| Waveshare ESP32-P4-WIFI6-Touch-LCD-4B | `--port esp32 --board ESP32_GENERIC_P4 --variant PRE_REV3_C6_WIFI --flash 32MB` | -| Another ESP32-P4 board, chip revision 3 or later, with an ESP32-C6 for Wi-Fi | `--port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB` | -| Waveshare ESP32-S3-Touch-LCD-7 | `--port esp32 --board ESP32_GENERIC_S3 --variant LCD_7 --flash 8MB` | -| Waveshare ESP32-S3-Touch-LCD-4.3 | `--port esp32 --board ESP32_GENERIC_S3 --variant LCD_4_3 --flash 8MB` | -| LilyGO T-Embed S3 | `--port esp32 --board ESP32_GENERIC_S3 --variant T_EMBED --flash 16MB` | -| WT32-SC01 Plus | `--port esp32 --board ESP32_GENERIC_S3 --flash 16MB` | -| Raspberry Pi Pico (RP2040) | `--port rp2 --board RPI_PICO` (not `--modules all`: see [The model](#the-model)) | -| Raspberry Pi Pico 2 (RP2350) | `--port rp2 --board RPI_PICO2` | -| Pimoroni Pico Plus 2 W (16 MB flash, 8 MB PSRAM, 48 GPIOs) | `--port rp2 --board RPI_PICO2_W --variant PIMORONI_PICO_PLUS2_W` | - -A new board usually needs no file at all: pick the upstream board and -variant that match its chip and PSRAM, and give `--flash` its flash size. Make -a variant of ours only for settings that really are that board's, like the -LCD-7's and LCD-4.3's cache sizes and tick; see [esp32 settings](#esp32-settings) below. - -## Flash it - -An esp32 build ends with ESP-IDF's `To flash, run: idf.py flash`. That isn't -the route here: `idf.py` lives in the ESP-IDF environment the build sets up, -not on your shell's `PATH`. Write `firmware.bin` with esptool (`pip install -esptool`) instead. It's the merged image, bootloader, partition table and app -in one, so it goes where the bootloader goes: 0x2000 on the P4, 0x0 on the -S3. For another chip, the build folder's `flasher_args.json` has the -bootloader's offset. - -```bash -esptool --chip esp32p4 write-flash 0x2000 builds/esp32/ESP32_GENERIC_P4/WIFI6_DEV_KIT_H2/firmware.bin -esptool --chip esp32s3 write-flash 0x0 builds/esp32/ESP32_GENERIC_S3/SPIRAM_OCT/firmware.bin -``` - -On rp2, hold BOOTSEL while you plug the board in, and copy `firmware.uf2` to -the drive it shows. From there, pydevices' -[Bringing up a board](https://github.com/PyDevices/pydevices/blob/main/docs/board-bringup.md) -takes a flashed board to one drawing on its own panel. - -## The model - -```text -pinned MicroPython tag (UPSTREAM) - + patches/micropython/ and each module's own patches, one local commit - | - upstream board (esp32, rp2) or upstream / our variant (unix, windows, webassembly) - + on esp32: our sdkconfig, appended last (flash size, grown partition table, fragments) - | - modules/manifest.py: the port's own content, then the modules you named - | - the port's own make -> builds//[/]/ -``` - -With no `--variant`, that last directory is `default` on esp32 and rp2, and -`standard`, upstream's default variant, on unix, windows and webassembly. - -Three choices make a build: the board (or not, on the desktop ports), the -variant (always optional, as with upstream's `make`), and the modules. A -module is a directory with a `manifest.py`, or a C module with its glue, -like ulab. Our Python-only packages come from omnipython-lib: `pydevices`, -`palettes`, `pdwidgets` and `audiocomponents` are modules whose manifest -requires them from it, and any other package of it (`appkit`, a board config) -can be named too. A module never brings another; nothing checks dependencies, so -name ulab when you name audiocomponents, tflite or vision, or use `all`. -`all` is every module except the opt-in ones (an `OPT_IN` file says why): -tflite and vision are built only when named. - -On esp32 and rp2 the build may grow the firmware's share of flash to fit the -image (on esp32 the app partition, on rp2 the region before the filesystem), -leaving 1/32 of the flash as headroom, 64 KB to 256 KB. That moves the -filesystem, so a board flashed with that image comes up with an empty one. -`--no-autosize` refuses instead. - -### A troubleshooting build with a core dump - -Standard builds, including every image on the download page, don't write a core -dump to flash when the firmware crashes. When you need one, to find out why a -board resets or hangs, add `--coredump` to esp32: - -```sh -./build_mp.py --port esp32 --board ESP32_GENERIC_S3 --variant SPIRAM_OCT --flash 8MB --modules all --coredump -``` - -It builds into its own folder, `builds/esp32//-coredump/`, so -it never replaces your standard image. After a crash the board keeps the dump -in its `coredump` partition across the reboot; read it back with `esptool` and -`esp-coredump` (`esp_coredump info_corefile`) against that build's -`micropython.elf`. The decode only matches the ELF of the same commit and -options, so build the troubleshooting image from the commit your board runs. - -The dump needs a `coredump` partition, and only the ESP32-S3 table has one -today. A standard S3 build leaves it unused, which is why the filesystem sits -where it always did, and a troubleshooting build uses the same table: you can -flash its app image (`micropython.bin`, at the app offset) onto a board -running a standard image and keep the files on it. On a board whose table has -no such partition, such as the ESP32-P4, `--coredump` stops with "this board's -table has no coredump partition yet" instead of moving your filesystem. - -It finds out by trying: the first link fails, loudly, and only then does it -build again. On esp32 that is a red `Error: app partition is too small for -binary micropython.bin size 0x…`, then `ninja: build stopped`, `make: *** -Error 1` and a Build-Troubleshooting link; on rp2 it is ``region `FLASH' -overflowed by … bytes``, `collect2: error: ld returned 1 exit status` and -nested `Error 2` lines. Then an `autosize:` line says it didn't fit and is -building once more. With `all`, expect that second pass on every board -above; it roughly doubles the build time, since the whole esp32 compile -runs twice. - -How much fits depends on the flash. On a Pico 2 (4 MB), `all` builds and -fits, at 93 % of flash with a 1600 KB filesystem. On a Pico (2 MB) it can't: -an image of that size (2.3 MB on the Pico 2) is bigger than the whole flash, -so the build stops at the first link with the overflow error. Name what your -project uses instead: `displayif,jpegio,pygraphics,palettes,pdwidgets`, a -display project's set, builds to about 540 KB and fits with the Pico's -default 1408 KB filesystem untouched. - -## esp32 settings - -On esp32, `build_mp.py` appends our ESP-IDF settings after the board's, so -ours win. They come from files it finds by name, each used only if it exists, -in this order: - -1. `variants/esp32/sdkconfig`, then `variants/esp32/sdkconfig.` - (`esp32s3`, `esp32p4`): what every build, or every build for that chip, - wants. The S3's PSRAM, Wi-Fi and lwIP tuning and the P4's BLE-over-C6 - settings are here. -2. The same two names at the root of each module you selected: what that - module needs. usbhost's USB host settings and cameraif's OV5647 driver live - in their own module directories. -3. `variants/esp32///sdkconfig`: what's left that is one - board's alone. A variant dir of ours also holds an `mpconfigvariant.cmake` - that includes the upstream variant it builds on, which is how `LCD_7` - and `T_EMBED` become variants of `ESP32_GENERIC_S3`, and `WIFI6_DEV_KIT` - one of `ESP32_GENERIC_P4`. A variant dir may also hold an - `mpconfigboard.h`, included before the board's own, for C defines the - port's ESP-IDF components must see too: `WIFI6_DEV_KIT`'s sets - `MICROPY_HW_USB_HS (0)`, which moves the USB device to the P4's - full-speed controller, so other P4 boards keep theirs on the high-speed - one. A variant dir may carry C sources too, through - `MICROPY_SOURCE_BOARD` in its `mpconfigvariant.cmake`, the way upstream's - `ESP32_GENERIC_C2` does: `WIFI6_DEV_KIT_H2` builds on `WIFI6_DEV_KIT` and - adds a board startup hook that swaps the P4's two full-speed PHYs, so the - USB device comes out on the kit's USB-C marked "USB". - -The partition table is found the same way, the most specific -`partitions.csv` winning ([`variants/esp32/partitions.esp32s3.csv`](../variants/esp32/partitions.esp32s3.csv) adds a -coredump partition on the S3, which a standard build leaves unused); -without one, the board's own is used. Either way autosize grows the app partition when the image doesn't fit. - -## rp2 settings - -An rp2 board of ours is a variant of one of upstream's, in -`variants/rp2///`. The port only knows variants that sit in -the board's own folder, so `build_mp.py` makes a folder of its own for the -build, `builds/rp2////`, links upstream's board into it -unchanged and adds what the variant holds: - -- `mpconfigvariant.cmake` (required): cmake settings, included after the - board's own default variant. The board's `PICO_BOARD`, the number of GPIOs, - PSRAM and the size of the filesystem go here. -- `mpconfigboard.h`: C defines, included after the board's own header, so it - can `#undef` what the board sets, such as its name. -- `pins.csv`: the pin names, in place of the board's. - -`PIMORONI_PICO_PLUS2_W` is the one so far. It builds the Pico 2 W for the -Pimoroni board's 16 MB flash, 8 MB PSRAM, 48 GPIOs and its own pin names, and -its files say why each setting is there. - -## Where things are - -| Path | What it is | -|---|---| -| [`build_mp.py`](../build_mp.py) | The build command. | -| [`UPSTREAM`](../UPSTREAM) | The MicroPython release the patches apply to. | -| [`modules.lock`](../modules.lock), [`deps.lock`](../deps.lock) | What gets fetched when it's missing, lvgl-bindings included. | -| `modules/.lvgl-bindings` | The lvgl-bindings lvgl-micropython compiles: the sibling checkout, linked, or `modules.lock`'s commit, fetched. | -| [`modules/`](../modules/) | The module manifest, `all/`, `tflite/` and `vision/` ([esp-vision's](esp-vision.md)), the rest linked or cloned, and the C modules that live only here, below. | -| [`modules/castif/`](../modules/castif/) | Casts an ESP32-P4's screen and sound to a TV over Wi-Fi Display, or serves them as live HLS. | -| [`modules/h264enc/`](../modules/h264enc/) | RGB565 frames to H.264 on the ESP32-P4's hardware encoder. | -| [`modules/jpegio/`](../modules/jpegio/) | JPEG decode and encode, with CircuitPython's `jpegio` API. | -| [`modules/mdns/`](../modules/mdns/) | Find and advertise services on the local network, with CircuitPython's `mdns` API. | -| [`modules/pngio/`](../modules/pngio/) | PNG encode and decode, RGB565 in and out. | -| [`modules/ppa/`](../modules/ppa/) | Scale, rotate, convert and fill pixel buffers, on the ESP32-P4's Pixel Processing Accelerator or in portable C elsewhere. | -| [`modules/tsmux/`](../modules/tsmux/) | MPEG-TS and HLS segments from H.264. | -| [`patches/`](../patches/) | The patch series and [`apply_patches.py`](../patches/apply_patches.py). | -| [`variants/`](../variants/) | Our unix, windows and webassembly variants, and the esp32 settings and board variants. | - -## Boundaries - -The upstream tag is a compatibility boundary: moving `UPSTREAM` means -re-validating every patch. Never hand-edit a patched checkout; change the -patch and prepare again ([`patches/apply_patches.py`](../patches/apply_patches.py)). - -The `vst3-engine` variants are deliberately narrow. They set -`MICROPY_PY_SOCKET`, `MICROPY_PY_SSL` and `MICROPY_PY_FFI` to 0, in the -variant's header as well as its makefile, so plugin content can't reach the -network or native libraries. A prepared tree carries the Windows networking -and FFI patches regardless; the variant is the guard. mpvst's -`mpvst_engine_capabilities` test fails if an engine can import any of them. -Don't swap it for a broader desktop variant because that one builds. - -## Safe first contributions - -Documentation, provenance, or a narrowly scoped variant or module fix. Check -a patch change with `patches/apply_patches.py --check` before you commit it. -The [build plan](build-plan.md) is the source of truth for the layout. diff --git a/docs/reference/boards.md b/docs/reference/boards.md new file mode 100644 index 00000000..b96e9a9b --- /dev/null +++ b/docs/reference/boards.md @@ -0,0 +1,55 @@ +# Boards and variants + +## Boards we build for + +Every board here is one of upstream's, with at most a small variant of ours on +top. Add `--modules` to any of these: + +| Board | Command | +|---|---| +| Waveshare ESP32-P4-WIFI6-DEV-KIT, USB device on a USB-C wired to header P6 | `--port esp32 --board ESP32_GENERIC_P4 --variant WIFI6_DEV_KIT --flash 16MB` | +| Waveshare ESP32-P4-WIFI6-DEV-KIT, USB device on its USB-C marked "USB" | `--port esp32 --board ESP32_GENERIC_P4 --variant WIFI6_DEV_KIT_H2 --flash 16MB` | +| Waveshare ESP32-P4-WIFI6-Touch-LCD-4B | `--port esp32 --board ESP32_GENERIC_P4 --variant PRE_REV3_C6_WIFI --flash 32MB` | +| Another ESP32-P4 board, chip revision 3 or later, with an ESP32-C6 for Wi-Fi | `--port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB` | +| Waveshare ESP32-S3-Touch-LCD-7 | `--port esp32 --board ESP32_GENERIC_S3 --variant LCD_7 --flash 8MB` | +| Waveshare ESP32-S3-Touch-LCD-4.3 | `--port esp32 --board ESP32_GENERIC_S3 --variant LCD_4_3 --flash 8MB` | +| LilyGO T-Embed S3 | `--port esp32 --board ESP32_GENERIC_S3 --variant T_EMBED --flash 16MB` | +| WT32-SC01 Plus | `--port esp32 --board ESP32_GENERIC_S3 --flash 16MB` | +| Raspberry Pi Pico (RP2040) | `--port rp2 --board RPI_PICO` (not `--modules all`: [it doesn't fit](../guides/fit-firmware-on-a-small-flash.md)) | +| Raspberry Pi Pico 2 (RP2350) | `--port rp2 --board RPI_PICO2` | +| Pimoroni Pico Plus 2 W (16 MB flash, 8 MB PSRAM, 48 GPIOs) | `--port rp2 --board RPI_PICO2_W --variant PIMORONI_PICO_PLUS2_W` | + +For a board that isn't listed, see [Build for a board](../guides/build-for-a-board.md#a-board-that-isnt-in-the-table). + +## Variants + +Our variants, in `variants/`. + +| Port | Variant | What it is | +|---|---|---| +| unix | `pydevices` | Upstream's standard variant plus `os.dupterm`, freezing the modules you named and the desktop board config. | +| unix | `vst3-engine` | The interpreter inside the micropython-vst3 plug-in: no sockets, no SSL, no FFI. | +| windows | `pydevices` | The same as unix `pydevices`, with the Windows display and audio backends (`displaydev.windisplay`, `audiodev.win_audio`). | +| windows | `vst3-engine` | The same as unix `vst3-engine`. | +| webassembly | `pydevices` | The browser runtime, with the `wasmbridge` module and a Fetch-backed `requests`. | +| esp32 | `ESP32_GENERIC_S3` `LCD_4_3`, `LCD_7`, `T_EMBED` | The S3 boards in the table above. | +| esp32 | `ESP32_GENERIC_P4` `WIFI6_DEV_KIT`, `WIFI6_DEV_KIT_H2` | The P4 DEV-KIT, with the USB device on the two different connectors in the table above. | +| rp2 | `RPI_PICO2_W` `PIMORONI_PICO_PLUS2_W` | The Pimoroni Pico Plus 2 W. | + +The esp32 `variants/esp32/` folder also holds the `sdkconfig` fragments and +partition table every esp32 or every chip gets: +[Add a board variant](../guides/add-a-board-variant.md) says how they combine. + +## CircuitPython boards + +With `--interpreter circuitpython`, boards are CircuitPython's own. These are +ours, in `boards/circuitpython/`, for boards CircuitPython has no definition for: + +| Board | What it is | +|---|---| +| `feather_rp2040_lvgl` | A Feather RP2040 with room for LVGL. | +| `waveshare_esp32s3_touch_lcd_7` | The 800x480 panel as `board.DISPLAY`, one 4 MB app slot. | +| `waveshare_esp32s3_touch_lcd_43` | The same 800x480 panel as `board.DISPLAY` and one 4 MB app slot, like the LCD-7. | +| `lilygo_twatch_s3_single_app` | The T-Watch S3 with one 4 MB app slot, so LVGL fits. | +| `waveshare_esp32p4_wifi6_dev_kit` | A revision 3 chip, console on USB Serial/JTAG, one 6 MB app slot. | +| `waveshare_esp32p4_wifi6_touch_lcd_4b` | Revision 1, console on the CH343 UART, one 6 MB app slot. Its panel isn't set up. | diff --git a/docs/reference/build-command.md b/docs/reference/build-command.md new file mode 100644 index 00000000..c14ed459 --- /dev/null +++ b/docs/reference/build-command.md @@ -0,0 +1,58 @@ +# The build command + +```text +./build_mp.py [--interpreter {micropython,circuitpython}] [--port PORT] + [--board BOARD] [--variant VARIANT] [--modules MODULES] + [--flash FLASH] [--no-autosize] [--coredump] [--clean] + [make arguments] +``` + +Leave out `--port` and it asks for port, board and variant. On a port that has +boards, leave out `--board` and it asks for that. `--variant` is always +optional. Leave out `--modules` and it lists `modules/` and asks. + +## Options + +| Option | Meaning | +|---|---| +| `--interpreter` | `micropython` (the default) or `circuitpython`: CircuitPython-compatible firmware from CircuitPython's own ports and boards. | +| `--port` | The port: `unix`, `windows`, `webassembly`, `esp32`, `rp2`. With `--interpreter circuitpython`, CircuitPython's own: `espressif`, `raspberrypi`, `unix` and the rest. | +| `--board` | The board, by upstream's directory name. | +| `--variant` | A variant of the board or port: upstream's, or one of ours in `variants/`. | +| `--modules` | A comma list of short names, full paths, or `all`. See [Modules](modules.md). | +| `--flash` | esp32 flash size, such as `16MB`. | +| `--no-autosize` | esp32 and rp2: refuse instead of growing the firmware's share of flash. | +| `--coredump` | esp32: a troubleshooting build that writes a core dump to flash on a crash. Standard builds don't. | +| `--clean` | Delete this target's build directory first. | + +Arguments it doesn't recognise go to make, such as `-j4` or `V=1`. On a +CircuitPython build they are part of the build record. + +## Environment + +| Variable | Default | Meaning | +|---|---|---| +| `OUT_DIR` | `builds/` | Where builds go. | +| `VARIANTS_DIR` | `variants/` | Where our variants are found. | +| `MODULES_DIR` | `modules/` | Where modules are found. | +| `JOBS` | every core | The parallelism handed to make. | +| `MSYS2_ROOT` | `C:\msys64` | On Windows, the MSYS2 whose make and MinGW gcc build the windows port. | +| `IDF_TOOLS_PATH` | `~/.espressif` | Where an esp32 build installs ESP-IDF's toolchains. | + +## Output + +Output lands in `builds//[/]/`, and in +`builds/circuitpython///` for CircuitPython. A +`--coredump` build adds `-coredump` to the variant's name. With no `--variant` +the last directory is: + +| Port | Directory | +|---|---| +| esp32, rp2 | `default` | +| unix, windows, webassembly | `standard`, upstream's default variant | +| CircuitPython unix | `coverage`, CircuitPython's default variant | + +Each build folder holds `pydevices-build.json`, a record of every module's +commit, and the files the build made: `firmware.bin` on esp32, `firmware.uf2` +on rp2, `micropython` on unix, `micropython.exe` on windows, `micropython.mjs` +and `micropython.wasm` on webassembly. diff --git a/docs/reference/files.md b/docs/reference/files.md new file mode 100644 index 00000000..dab44c1d --- /dev/null +++ b/docs/reference/files.md @@ -0,0 +1,18 @@ +# Files and pins + +| Path | What it is | +|---|---| +| [`build_mp.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/build_mp.py) | The build command: [every option](build-command.md). It fills in `micropython/`, `modules/` and `deps/` if they're missing, prepares the MicroPython tree and runs the port's own `make`. | +| [`UPSTREAM`](https://github.com/PyDevices/micropython-pydevices/blob/main/UPSTREAM) | The MicroPython commit everything applies to. | +| [`modules.lock`](https://github.com/PyDevices/micropython-pydevices/blob/main/modules.lock) | The commits `build_mp.py` fetches when a module isn't there. It also pins lvgl-bindings, the generated binding and LVGL that lvgl-micropython compiles: a build with lvgl-micropython fetches it into `modules/.lvgl-bindings`. Beside sibling checkouts, these are links to those checkouts instead. | +| [`deps.lock`](https://github.com/PyDevices/micropython-pydevices/blob/main/deps.lock) | The toolchains `build_mp.py` fetches: ESP-IDF, emsdk, SDL2 for MinGW, the TinyUSB the esp32 port builds on, and the ESP-IDF components CircuitPython builds ask for. It also pins CircuitPython, for `--interpreter circuitpython`. | +| [`provenance.json`](https://github.com/PyDevices/micropython-pydevices/blob/main/provenance.json) | The MicroPython pin and the checksum of every patch in `patches/micropython/` and `patches/tinyusb/`. | +| [`modules/`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules) | [`manifest.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/modules/manifest.py), the manifest every build freezes; `all/`; the modules, [linked or cloned](modules.md); and `.esp-vision/`, the pin for [esp-vision's modules](../how-it-works/esp-vision.md). | +| [`patches/micropython/`](https://github.com/PyDevices/micropython-pydevices/tree/main/patches/micropython) | Our patch series, `0001-…` to `0042-…` (there is no `0026`, whose fix moved into `patches/tinyusb/`), each justified in its own header. | +| [`patches/tinyusb/`](https://github.com/PyDevices/micropython-pydevices/tree/main/patches/tinyusb) | Patches to the TinyUSB the esp32 port builds on the S2, S3 and P4, which `build_mp.py` applies to `deps/espressif__tinyusb` (at its `deps.lock` tag) and builds in place of the port's own copy. | +| [`patches/micropython-lib/`](https://github.com/PyDevices/micropython-pydevices/tree/main/patches/micropython-lib) | micropython-lib's own fixes from after the commit MicroPython's `lib/micropython-lib` points at. A firmware that freezes those packages, and a board whose `/lib` you fill from the submodule, both get them. | +| [`patches/circuitpython/`](https://github.com/PyDevices/micropython-pydevices/tree/main/patches/circuitpython) | Our patches to CircuitPython, applied to `deps/circuitpython`. | +| [`patches/apply_patches.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/patches/apply_patches.py) | Applies the MicroPython series, plus each module's own MicroPython patches, once, as one local commit. `--check` tests a change without applying it. | +| [`variants/`](boards.md#variants) | Our variants and the esp32 settings. | +| [`boards/circuitpython/`](boards.md#circuitpython-boards) | Boards CircuitPython has no definition for. | +| [`scripts/`](https://github.com/PyDevices/micropython-pydevices/tree/main/scripts) | `exercise_build.sh`, `release_firmware.py`, `stage_firmware_page.py`, `serve_workbench.sh`, `freeze_list.py`, `update_provenance.py`. [Contributing](https://github.com/PyDevices/micropython-pydevices/blob/main/CONTRIBUTING.md) says what each is for. | diff --git a/docs/reference/index.md b/docs/reference/index.md new file mode 100644 index 00000000..b4f9fbae --- /dev/null +++ b/docs/reference/index.md @@ -0,0 +1,23 @@ +# Reference + +This repository has no Python package to document, so these pages are tables: +what `build_mp.py` takes, the modules you can name, the boards and variants +built, and the files that pin what a build uses. + +- [The build command](build-command.md): every option, environment variable and + where output goes. +- [Modules](modules.md): the short names `--modules` takes, and which build on + CircuitPython. +- [Boards and variants](boards.md): the command for each board, and every variant + in `variants/` and `boards/`. +- [Files and pins](files.md): `UPSTREAM`, the lock files, the patch series and + the scripts. + +```{toctree} +:hidden: + +build-command +modules +boards +files +``` diff --git a/docs/reference/modules.md b/docs/reference/modules.md new file mode 100644 index 00000000..2ff71b26 --- /dev/null +++ b/docs/reference/modules.md @@ -0,0 +1,64 @@ +# Modules + +The short names `--modules` takes. Each is a directory in `modules/`, either +tracked here or linked from a sibling checkout or cloned at the commit in +`modules.lock`. [Choose modules](../guides/choose-modules.md) says how to pick. + +## C modules that live here + +| Module | What it is | +|---|---| +| [`audioif`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/audioif) | The audio hardware under the audio pump, with CircuitPython's `audiobusio.I2SOut` and `audioi2sin.I2SIn` on top. | +| [`cameraif`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/cameraif) | A MIPI-CSI camera as a MicroPython object, on the ESP32-P4. | +| [`castif`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/castif) | Casts an ESP32-P4's screen and sound to a TV over Wi-Fi Display, or serves them as live HLS. | +| [`displayif`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/displayif) | The display buses and panels under `displaydev` (SPI, I2C, i80, QSPI, dot-clock RGB, MIPI-DSI, PicoDVI, RGB matrix, SDL2). The dot-clock, MIPI-DSI, PicoDVI and RGB matrix modules keep CircuitPython's names. | +| [`h264enc`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/h264enc) | RGB565 frames to H.264 on the ESP32-P4's hardware encoder. | +| [`jpegio`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/jpegio) | JPEG decode and encode, with CircuitPython's `jpegio` API. | +| [`mdns`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/mdns) | Find and advertise services on the local network, with CircuitPython's `mdns` API (esp32, rp2, unix, windows). | +| [`pngio`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/pngio) | PNG encode and decode, RGB565 in and out. | +| [`ppa`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/ppa) | Scale, rotate, convert and fill pixel buffers, on the ESP32-P4's Pixel Processing Accelerator or in portable C elsewhere. | +| [`tsmux`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/tsmux) | MPEG-TS and HLS segments from H.264. | +| [`usbhost`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/usbhost) | The USB host port and isochronous streams that `usb.core` and the `usb.host.*` packages are built on (ESP32-S2, S3 and P4). | + +## Modules from other repositories + +| Module | Where it comes from | +|---|---| +| `audiodsp`, `pygraphics`, `ulab`, `lvgl-micropython` | Cloned at the commit in `modules.lock` when `modules/` is missing, or linked from a sibling checkout. | +| `pydevices`, `palettes`, `pdwidgets`, `audiocomponents` | Manifests here that require the Python packages from omnipython-lib. | +| `micropython-lib` | MicroPython's own upstream packages, frozen from the checkout the MicroPython tree carries. | +| `tflite`, `vision` | [esp-vision's modules](../guides/use-esp-vision-modules.md), opt-in: `all` leaves them out. | + +`modules/all` is not a module but the rule behind `all`: every module beside it +except those with an `OPT_IN` file. + +## Which modules build on CircuitPython + +What `--modules` can name on CircuitPython, and why a module is missing where +it is. "Unix" is CircuitPython's unix port, built and imported in +[`circuitpython.yml`](https://github.com/PyDevices/micropython-pydevices/blob/main/.github/workflows/circuitpython.yml); +"boards" are the five CircuitPython cells of +[`release-firmware.yml`](https://github.com/PyDevices/micropython-pydevices/blob/main/.github/workflows/release-firmware.yml). + +| Module | Unix | Boards | Notes | +|---|---|---|---| +| pygraphics | yes | all five | | +| audiodsp | yes | the four ESP32 cells | The modules CircuitPython lacks, `audiograph` and `audiopump.Ring` (Python-made audio that `audiobusio.I2SOut` plays) among them; its audiocore, synthio and the rest stay CircuitPython's. With all of it, a LilyGO T-Embed S3 build fits that board's 2 MB app slot only with `CIRCUITPY_ULAB=0` on the make line. | +| displayif | yes (`usdl2`) | no, on purpose | Boards have CircuitPython's own `displayio` and display buses. pydevices' `SDLDisplay` runs on top of `usdl2`. | +| pngio | yes | all five | Decodes with CircuitPython's older uzlib. | +| tsmux | yes | all five | | +| ppa | yes (software) | ESP32-P4 | The hardware path on the P4. | +| jpegio | yes | ESP32-P4 | Adds only the encoder's C API: CircuitPython has its own `jpegio` decoder. | +| cameraif | yes (`available()` is False) | ESP32-P4 | | +| mdns | yes | no, on purpose | CircuitPython's boards have an `mdns` of their own; its unix port has none. | +| usbhost | yes, as `_usbhost`, with no host engine | ESP32-P4 | The host engine runs on ESP-IDF's USB host library, which from ESP-IDF 6 is Espressif's `usb` component: `build_mp.py` fetches it at the version `deps.lock` pins. The P4's boards leave its high-speed controller to us. Tethering is left out until the port has a network. The S3's one controller is CircuitPython's drive and REPL, so it gets the module without the engine. On the RP2040 and i.MX, CircuitPython's own `usb.core` is the host. | +| audioif | no | no | Links against audiodsp's audio pump and `cp_compat`, which audiodsp's CircuitPython glue doesn't build. It isn't needed there: audioif gives MicroPython the `audiobusio` and `audioi2sin` that CircuitPython boards already have. | +| ulab | CircuitPython's own | CircuitPython's own | | +| castif | no: it is the ESP32-P4's | ESP32-P4, with no network | Talks to lwIP directly, the lwIP under CircuitPython's own `socketpool`. CircuitPython's P4 port has no network until it drives the boards' ESP32-C6, so castif is built without one and `start()` says so. Name tsmux, h264enc and ppa with it. | +| h264enc | no: it is the ESP32-P4's encoder | ESP32-P4 | Espressif's `esp_h264`, which CircuitPython's port, built with the component manager off, can't fetch itself, is pinned in `deps.lock` and fetched by `build_mp.py`. Name ppa with it. | +| tflite, vision | no | no, on purpose | esp-vision's modules, for MicroPython only. | +| lvgl-micropython | no | no | CircuitPython's binding is `lvgl-circuitpython`, passed by path. | +| micropython-lib | no | no | MicroPython's own upstream packages. | +| pydevices, palettes, pdwidgets | frozen | frozen | Python only. On unix, pydevices also freezes its desktop files (the boards drop them), and pdwidgets and palettes with it: together their names pass CircuitPython's 64 KB frozen-string pool, so `build_mp.py` builds the unix port with four-byte qstr offsets (patches 0001 and 0010). Boards keep two-byte offsets. | + +What is planned for the rest is in the [roadmap](https://github.com/PyDevices/micropython-pydevices/blob/main/ROADMAP.md). diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 00000000..1cdafe01 --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,4 @@ +# Local preview: use the checkout of .github next to this repository, +# pip install -e ../.github/docs-config +# Read the Docs and CI install the tagged release: +pydevices-docs @ git+https://github.com/PyDevices/.github@docs-v0.1.0#subdirectory=docs-config diff --git a/docs/upstream/esp32-ble-mtu-before-connect.md b/docs/upstream/esp32-ble-mtu-before-connect.md deleted file mode 100644 index e2b0f9d9..00000000 --- a/docs/upstream/esp32-ble-mtu-before-connect.md +++ /dev/null @@ -1,127 +0,0 @@ -# Draft issue: esp32 BLE reports the MTU exchange before the connection - -A draft for micropython/micropython. Nothing here has been posted yet. We carry no patch for this: PyDevices' `bledev` works around it in -Python (`bledev.mpble` keeps an early MTU and applies it on the connect), in -[PyDevices/pydevices#81](https://github.com/PyDevices/pydevices/pull/81). - ---- - -**Title:** esp32: `_IRQ_MTU_EXCHANGED` can arrive before `_IRQ_CENTRAL_CONNECT`, and aioble then loses the MTU - -**Port, board, firmware** - -esp32 port, MicroPython v1.29.0 with ESP-IDF v5.5.4 and the IDF's NimBLE, on -ESP32-S3 boards: a Waveshare ESP32-S3-Touch-LCD-7 as the peripheral and a -LilyGo T-Embed as a second central. (Our builds carry a few local patches; -none of them touches Bluetooth.) The PC central is Windows 11 through -[bleak](https://github.com/hbldh/bleak) 3.0.2, and Windows exchanges the MTU -as soon as it connects. - -**Reproduction** - -On the board, a peripheral that logs the order of its IRQs: - -```python -import bluetooth, time -_CONNECT, _DISCONNECT, _MTU = 1, 2, 21 -ble = bluetooth.BLE() -ble.active(True) -ble.config(mtu=247) -ble.gatts_register_services(((bluetooth.UUID(0x181A), ((bluetooth.UUID(0x2A6E), 0x02),)),)) -ADV = b"\x02\x01\x06" + bytes((9, 0x09)) + b"mtu-test" -seen = [] -def irq(event, data): - if event == _CONNECT: - seen.append("connect") - print(time.ticks_ms(), "CENTRAL_CONNECT", data[0]) - elif event == _MTU: - seen.append("mtu") - print(time.ticks_ms(), "MTU_EXCHANGED", data[0], data[1]) - elif event == _DISCONNECT: - print(time.ticks_ms(), "CENTRAL_DISCONNECT", data[0], "order:", seen) - seen.clear() - ble.gap_advertise(100_000, ADV) -ble.irq(irq) -ble.gap_advertise(100_000, ADV) -``` - -On the PC, connect five times: - -```python -import asyncio -from bleak import BleakClient, BleakScanner - -async def main(): - device = await BleakScanner.find_device_by_name("mtu-test", timeout=10) - for i in range(5): - async with BleakClient(device) as client: - await asyncio.sleep(1) - print("bleak: connect", i, "mtu", client.mtu_size) - await asyncio.sleep(5) # Windows keeps an idle link a few seconds - -asyncio.run(main()) -``` - -The board prints this, five times out of five (2026-09-24): - -``` -169856 MTU_EXCHANGED 1 247 -169915 CENTRAL_CONNECT 1 -175795 CENTRAL_DISCONNECT 1 order: ['mtu', 'connect'] -``` - -The MTU exchange is reported 59-60 ms before the connection it belongs to, -every time. The five-second pause matters: Windows keeps an idle link open -for a few seconds after the last client closes it, and a reconnect inside -that window reuses the link instead of making a new one, so the board sees -nothing. - -With a second board as the central instead (MicroPython v1.29.0 on another -ESP32-S3, calling `gattc_exchange_mtu()` from inside its -`_IRQ_PERIPHERAL_CONNECT` handler), the order is normal, five times out of -five. - -**What happens** - -In the IDF's NimBLE (`components/bt/host/nimble/nimble/nimble/host/src/ble_gap.c` -in v5.5.4), `ble_gap_rx_conn_complete()` doesn't raise `BLE_GAP_EVENT_CONNECT`. -It inserts the connection into the host (`ble_hs_conn_insert()`, line 3394) -and, for a peripheral, asks the controller for the peer's version -(`ble_gap_rd_rem_ver_tx()`, line 3423). When that completes, -`ble_gap_rx_rd_rem_ver_info_complete()` reads the peer's features, and only -when those arrive does `ble_gap_rx_rd_rem_sup_feat_complete()` call -`ble_gap_event_connect_call()` (line 3512). That's two link-layer procedures, -several connection events. Meanwhile ATT is already live, so an MTU request -from the central is answered and `BLE_GAP_EVENT_MTU` goes out first. -`extmod/nimble/modbluetooth_nimble.c` passes both straight to Python in the -order it gets them (`commmon_gap_event_cb()`, line 414, and -`central_gap_event_cb()`, line 449). - -The NimBLE in `lib/mynewt-nimble` (1.4, used by the other NimBLE ports) -raises the connect event in `ble_gap_rx_conn_complete()` itself, so this is -specific to the esp32 port. - -A MicroPython central on the same port can't exchange the MTU early enough -to show it: its own connect IRQ is deferred the same way, until it has read -the peer's features and version, and by then the peripheral has reported the -connection. That's why two boards never show it and Windows always does. - -**Why it matters** - -aioble's `_device_irq` (`micropython-lib`, -`micropython/bluetooth/aioble/aioble/device.py`) looks the connection up in -`DeviceConnection._connected` and drops an MTU for a handle it doesn't know -yet. The link then runs at the negotiated MTU while the board believes 23, so -every notification the board sends carries 20 bytes. Against Windows that cut -our notification throughput from 49 to 19 KB/s. - -**Possible fixes** - -- In `modbluetooth_nimble.c`, hold an `BLE_GAP_EVENT_MTU` that arrives for a - handle whose connect hasn't been reported yet, and deliver it right after - the connect. That keeps the order the IRQ documentation implies on every - port. -- Or, in aioble, keep an MTU for an unknown handle and apply it when the - connection appears. That's what we do in our own code meanwhile. - -Happy to open a PR for either, whichever you'd prefer. diff --git a/docs/upstream/esp32-machine-timer-fire-during-reinit.md b/docs/upstream/esp32-machine-timer-fire-during-reinit.md deleted file mode 100644 index 3fb1cfb7..00000000 --- a/docs/upstream/esp32-machine-timer-fire-during-reinit.md +++ /dev/null @@ -1,99 +0,0 @@ -# Draft issue: esp32 machine.Timer re-armed while it fires calls NULL - -A draft for micropython/micropython. Nothing here has been posted yet. The fix we carry meanwhile is -[patches/0016](../../patches/micropython/0016-micropython-esp32-machine-timer-fire-during-reinit.patch), -and the tracking issue is micropython-pydevices#14. - ---- - -**Title:** esp32: re-initialising a machine.Timer while it fires panics (Instruction access fault, MEPC 0) - -**Port, board, firmware** - -esp32 port, MicroPython v1.29.0, ESP-IDF v5.5.4. Seen on an ESP32-P4 (the -Waveshare ESP32-P4-WIFI6-Touch-LCD-4B, pre-rev-3 silicon, C6 Wi-Fi -companion) and earlier on an ESP32-S3. The code involved is port-wide. - -**Reproduction** - -On a dual-core chip, re-arm a one-shot virtual timer at about the moment it -is due, over and over: - -```python -import machine, time - -fired = 0 - -def cb(t): - global fired - fired += 1 - -tim = machine.Timer(-1) -seed = 1 -while True: - tim.init(mode=machine.Timer.ONE_SHOT, period=1, callback=cb) - seed = (seed * 1103515245 + 12345) & 0x7FFFFFFF - t1 = time.ticks_us() - while time.ticks_diff(time.ticks_us(), t1) < 700 + seed % 600: - pass -``` - -On the ESP32-P4 this panicked in under two seconds in both of our runs. (Our -firmware is v1.29.0 plus a few local patches, none of which touch -`machine_timer.c` or esp_timer.) - -``` -Guru Meditation Error: Core 0 panic'ed (Instruction access fault). Exception was unhandled. -MEPC : 0x00000000 RA : 0x401e8602 SP : 0x4ff35c70 GP : 0x4ff1fa80 -``` - -We found it in a timing library that keeps one one-shot `machine.Timer` -and re-arms it to the next deadline of many software timers, so it calls -`init()` thousands of times a minute; an LVGL app on it rebooted the P4 -within seconds of starting to play audio. - -**What happens** - -A virtual timer is an `esp_timer` with `ESP_TIMER_TASK` dispatch -(`ports/esp32/machine_timer.c:148`). Its callback, -`machine_timer_isr_virtual`, runs on the esp_timer task, which is pinned -to core 0 (`CONFIG_ESP_TIMER_TASK_AFFINITY_CPU0=y` in the build), while the -interpreter runs on core 1 (`ports/esp32/mphalport.h:54`). esp_timer takes a timer off -its list under the list lock and then releases the lock before it calls the -callback (`components/esp_timer/src/esp_timer.c:434-435` in ESP-IDF v5.5.4), -so `esp_timer_stop()` (`machine_timer.c:208`) can return while a callback -that has already been dispatched has still to run. - -`Timer.init()` on a timer that is running calls `machine_timer_deinit()` -(`machine_timer.c:273`), which stops the timer and then sets -`self->handler = NULL` (`machine_timer.c:242`). The handler is set again only -at `machine_timer.c:298`, after the arguments are parsed. A dispatched -callback that reads the handler in that gap calls it unchecked -(`machine_timer.c:75`) and jumps to address 0. In the panic above, MEPC is 0 -and S1 holds `machine_timer_isr_virtual`. - -The hardware-timer ISR has the same unchecked call -(`machine_timer.c:65`), and `deinit()` has the same gap for `handler_ctx` -(`machine_timer.c:243`), which `machine_timer_handler` passes to -`mp_sched_schedule` (`machine_timer.c:83`). - -**A possible fix** - -What we carry, three changes in `ports/esp32/machine_timer.c`: - -1. Both callbacks read `self->handler` once and return if it is NULL, and - `machine_timer_handler` returns if `handler_ctx` is NULL. A fire in - flight during `deinit()` is dropped, as it would have been a moment later. -2. `init()` on a running timer stops it (`machine_timer_stop()`) instead of - deinitialising it, so the handler is never NULL on the way to being - replaced. `machine_timer_configure()` already tolerates an enabled - gptimer (`ESP_ERR_INVALID_STATE`) and a virtual timer already on the list. -3. `init()` sets `handler_ctx` before `handler`. - -A fire dispatched just before `init()` may still deliver the old callback -once, which is also what happens when the fire comes just before `init()` is -called. - -With these changes the loop above ran 60 s on the P4 (32693 fires) with no -panic, and the timing library that found it ran 10 minutes (288667 wakes). -Happy to open it as a PR if the approach is acceptable. diff --git a/docs/upstream/esp32-webrepl-ctrl-c.md b/docs/upstream/esp32-webrepl-ctrl-c.md deleted file mode 100644 index 8b97f6c5..00000000 --- a/docs/upstream/esp32-webrepl-ctrl-c.md +++ /dev/null @@ -1,70 +0,0 @@ -# Draft issue: esp32 WebREPL Ctrl-C can't break a loop that never waits - -A draft for micropython/micropython. Nothing here has been posted yet. The fix we carry meanwhile is -[patches/0011](../../patches/micropython/0011-micropython-esp32-webrepl-ctrl-c-in-tight-loops.patch). - ---- - -**Title:** esp32: Ctrl-C over WebREPL doesn't interrupt a loop that never waits, and blocks new WebREPL connections - -**Port, board, firmware** - -esp32 port, MicroPython v1.29.0. Seen on an ESP32-P4 board (C6 Wi-Fi -companion); the code involved is port-wide, so any esp32 build with WebREPL -should behave the same. - -**Reproduction** - -1. Start WebREPL (`import webrepl; webrepl.start()`) and connect with the - WebREPL client. -2. Run `while True: pass`. -3. Press Ctrl-C. Nothing happens; the loop keeps running. -4. Open a second WebREPL client and connect. It hangs instead of being - told "Concurrent WebREPL connection ... rejected". - -The same Ctrl-C over the USB/UART REPL stops the loop at once. Over WebREPL, -a loop that sleeps (`while True: time.sleep_ms(1)`) can be interrupted, so -this is specific to code that never waits. - -**What happens** - -Serial Ctrl-C arrives by interrupt: `ports/esp32/uart.c:116` and -`ports/esp32/usb_serial_jtag.c:58` call `mp_sched_keyboard_interrupt()` from -their ISRs, and the VM raises it at its next branch -(`py/vm.c:1341`, `pending_exception_check`). - -WebREPL's bytes sit in a socket until something reads them. On esp32 that is -`socket_events_handler()` (`ports/esp32/modsocket.c:123`), which selects on -the sockets registered with `setsockopt(SOL_SOCKET, 20, callback)` and runs -their callbacks. For the WebREPL client the callback is `os.dupterm_notify` -(`extmod/modos.c:134`), which reads through dupterm, where the interrupt -character is recognised and `mp_sched_keyboard_interrupt()` is called -(`extmod/os_dupterm.c:157-159`). - -The port calls `socket_events_handler()` only from `MICROPY_EVENT_POLL_HOOK` -(`ports/esp32/mpconfigport.h:343-371`) and from `mp_hal_delay_ms()` -(`ports/esp32/mphalport.c:203`), i.e. only while the program sleeps, waits -or sits at the REPL. A loop that never waits never reaches either, so the -Ctrl-C byte is never read. The listen socket's accept callback is starved the -same way, which is why a new connection hangs. - -**Related** - -#11536 was the same symptom on the Pico W, fixed for rp2 by #13200. I couldn't -find an esp32 issue. - -**A possible fix** - -Give the esp32 port a `MICROPY_VM_HOOK_LOOP` that, every N branches (a -decrement and test per branch), checks whether any socket has an events -callback and, rate-limited by `mp_hal_ticks_ms()` (say every 20 ms), schedules -the existing socket poll on a static scheduler node -(`mp_sched_schedule_node`). Scheduling rather than calling from the hook keeps -the callbacks, which can be Python (webrepl's accept handler is), in the same -context every scheduled callback gets: main thread, scheduler locked, -pending exceptions raised after they return. The cost with WebREPL off is a -decrement per branch and a call every N branches that returns at once. - -We carry this as a small patch on v1.29.0 and it builds for the ESP32-P4; -happy to open it as a PR if the approach is acceptable. Native and viper code -doesn't run VM hooks, so it wouldn't be covered, as on other ports. diff --git a/docs/upstream/fatfs-utf8-filenames.md b/docs/upstream/fatfs-utf8-filenames.md deleted file mode 100644 index 2db9b12d..00000000 --- a/docs/upstream/fatfs-utf8-filenames.md +++ /dev/null @@ -1,68 +0,0 @@ -# Draft issue: VfsFat names are code page 437, not UTF-8, so non-ASCII filenames fail - -A draft for micropython/micropython. Nothing here has been posted yet. The fix we carry meanwhile is -[patches/0038](../../patches/micropython/0038-micropython-lib-oofatfs-utf-8-filenames.patch). - ---- - -**Title:** extmod/vfs_fat: filenames go through FatFs as code page 437, so a card written by a PC can't be listed or opened by its non-ASCII names - -**Port, board, firmware** - -Every port that uses `VfsFat` with long filenames. Seen on the esp32 port -(MicroPython v1.30.0-preview, an ESP32-P4 with an SD card formatted and -filled on Windows) and reproduced on the unix port with `VfsFat` on a RAM -block device. `lib/oofatfs/ffconf.h` hard-codes `FF_LFN_UNICODE 0`, and no -`MICROPY_FATFS_*` option reaches it. - -**Reproduction** - -Copy a file named `Ravel - Boléro.mid` onto a FAT card from a PC, then on -the board: - -```python ->>> os.listdir(b"/sd") -[b'Ravel - Bol\x82ro.mid'] # code page 437 bytes, not UTF-8 ->>> os.listdir("/sd") -UnicodeError: ->>> open("/sd/Ravel - Boléro.mid", "rb") -OSError: [Errno 2] ENOENT -``` - -A name with a character code page 437 doesn't have, such as `Dvořák.mid`, -is listed only by its short name (`b'DVOR\xc1K~1.MID'`). The other -direction is wrong too: a file MicroPython creates as `Boléro.mid` appears -on the PC as `Bol├⌐ro.mid`, because its UTF-8 bytes were stored as two -code page 437 characters. - -**Cause** - -`FF_LFN_UNICODE 0` makes FatFs take and return names in the OEM code page -(`FF_CODE_PAGE`, 437 by default). MicroPython's `str` is UTF-8, and -`vfs_fat.c` passes `mp_obj_str_get_str()` to `f_open()` and builds names -from `FILINFO.fname` with `mp_obj_new_str()` unchanged, so every non-ASCII -name is mistranslated in both directions. - -**The fix** - -One line in `lib/oofatfs/ffconf.h`: `#define FF_LFN_UNICODE 2`, so the API -is UTF-8. It only takes effect where LFN is enabled, and `FF_CODE_PAGE` -still decides how short names are generated. Our patch makes it a -`MICROPY_FATFS_LFN_UNICODE` option defaulting to 2, so a port can keep the -old behaviour. - -With it, all seven accented names on the card above list by their real -names, open, stat and read whole, and on unix a round trip through a RAM -disk works for Latin, CJK and an emoji (a surrogate pair on disk). - -One limit: `FF_LFN_BUF` is 255 bytes, so a name whose UTF-8 form is longer -than that (FAT allows 255 UTF-16 units) is listed by its short name, as -names the code page couldn't hold were before. - -**Cost** - -No RAM: `FILINFO` and the LFN working buffer keep their sizes, and the -code page tables are already linked in with LFN. Code: +472 bytes on the -unix port (x86-64), and +400 bytes of firmware on an -ESP32-P4 (`ff.c` grows by 408 bytes of code; its 512-byte `LfnBuf` and its -read-only data are unchanged). diff --git a/docs/upstream/usb-device-midi-schedule-queue.md b/docs/upstream/usb-device-midi-schedule-queue.md deleted file mode 100644 index addfc001..00000000 --- a/docs/upstream/usb-device-midi-schedule-queue.md +++ /dev/null @@ -1,53 +0,0 @@ -# usb-device-midi: a burst fills the schedule queue and the device stops hearing MIDI - -Fixed upstream already, so there's nothing to report. micropython-lib fixed it on 2026-09-16, -in `90a3385` ("usb-device-midi: Queue at most one micropython.schedule() for rx") and `4fa59bd` -("usb-device-midi: Resume RX endpoint transfer after handling callback"), which make -`usb-device-midi` 0.1.1. MicroPython's `lib/micropython-lib` still points at `ee4bb8f` -(2026-08-17), which has 0.1.0, so a firmware that freezes `usb-device-midi` from the -submodule, or a board whose `/lib` is copied from it, gets the bug. `mip install -usb-device-midi` gets 0.1.1 from the index and is fine. - -We carry both commits unchanged as -[patches/micropython-lib/](../../patches/micropython-lib/). Once MicroPython's submodule -passes `4fa59bd`, `apply_patches.py` prints `already in micropython-lib` for each of -them, and the directory can go. - -## What it looks like - -With 0.1.0, a computer sends 200 note-ons back to back to an ESP32-P4 (high speed) or -ESP32-S3 (full speed), one 4-byte event per packet, about 0.3 ms apart. After 12 to -15 of them, the board prints: - -``` -Traceback (most recent call last): - File "/lib/usb/device/core.py", line 342, in _xfer_cb - File "/lib/usb/device/midi.py", line 152, in _rx_cb -RuntimeError: schedule queue full -``` - -and never receives MIDI again, because the device stops re-arming the OUT endpoint. The -computer's next send then blocks. - -TinyUSB's task runs as a scheduler node, so the scheduler stays locked while -`tud_task_ext()` handles every completion that lands during the pass. Each completion -re-arms the endpoint, the host's next packet completes before the pass ends, and each -one schedules another `_on_rx`. A probe that counted those calls saw all 8 slots of -`MICROPY_SCHEDULER_DEPTH` taken by `_on_rx` alone. In 0.1.0, `schedule()` comes before -the re-arm and raises, so the re-arm never happens. Catching that error isn't enough -on its own, either: the 64-byte receive buffer then fills with no `_on_rx` queued to -drain it and re-arm. - -0.1.1 queues at most one `_on_rx` at a time, re-arms in a `finally`, and has `_on_rx` -re-arm once it has drained the buffer. With it the same boards echo 200 of 200 in -order in a burst (0.51 s on the P4, 1.21 s on the S3) and 600 of 600 on the P4. - -## If you'd like MicroPython to move the submodule - -A comment for a MicroPython pull request or issue that bumps `lib/micropython-lib`: - -> `lib/micropython-lib` is at `ee4bb8f`, which predates micropython-lib `90a3385` and -> `4fa59bd` (usb-device-midi 0.1.1). With 0.1.0 frozen, a USB MIDI device that receives -> a fast burst raises `RuntimeError: schedule queue full` in `_rx_cb` and stops -> re-arming its OUT endpoint, so it never receives again. Moving the submodule past -> `4fa59bd` fixes it for frozen builds. diff --git a/docs/upstream/windows-repl-piped-stdin.md b/docs/upstream/windows-repl-piped-stdin.md deleted file mode 100644 index 4a15064a..00000000 --- a/docs/upstream/windows-repl-piped-stdin.md +++ /dev/null @@ -1,44 +0,0 @@ -# Draft issue: windows `micropython -i` with piped stdin prints prompts forever - -A draft for micropython/micropython. Nothing here has been posted yet. The fix we carry meanwhile is -[patches/0013](../../patches/micropython/0013-micropython-windows-read-piped-stdin-in-the-repl.patch). - ---- - -**Title:** windows: `micropython -i` with stdin from a pipe prints empty prompts in a tight loop - -**Port, build** - -windows port, MicroPython v1.29.0, built with mingw-w64 (the `dev` variant). -The code involved is the same for the MSVC build. - -**Reproduction** - -``` -printf 'print(6*7)\r\n' | micropython.exe -i -``` - -It never prints `42`. It prints `>>> ` followed by a newline, over and over, -until it is killed (about 6 MB of prompts in five seconds). Without `-i` the -same pipe prints `42`, and CPython's `python.exe -i` reads the pipe fine. - -**What happens** - -`ports/unix/main.c:702`, which the windows port also builds, enters the REPL -when stdin is not a tty if `-i` or `MICROPYINSPECT` is given. The windows -`mp_hal_stdin_rx_chr` (`ports/windows/windows_mphal.c:189`) reads keys with -`ReadConsoleInput` (line 203), which only works on a console handle. On a -pipe it fails, and the function returns `CHAR_CTRL_C` (line 206, commented -"EOF, ctrl-D"). The REPL treats Ctrl-C as a cancelled line, prints a new -prompt and reads again, and the read fails again. - -The unix port reads stdin with `read()` and returns Ctrl-D at end of input -(`ports/unix/unix_mphal.c:180`), so the same pipe works there. - -**Suggested fix** - -When `GetConsoleMode` fails on the stdin handle, read bytes with `ReadFile` -the way the unix port uses `read()`: CR LF or a lone LF is Enter, and end of -input returns Ctrl-D, which ends the REPL. A failed console read should -return Ctrl-D too, not Ctrl-C. We carry this as a patch and can open a PR -with it if that's welcome. diff --git a/modules/vision/manifest.py b/modules/vision/manifest.py index 79a6906c..7c681b96 100644 --- a/modules/vision/manifest.py +++ b/modules/vision/manifest.py @@ -1,7 +1,7 @@ # vision: esp-vision's OpenMV-style image stack, compiled as it is # (modules/.esp-vision). C only; ESP32-P4 and ESP32-S3. image, imageio, # espdl, and on the P4 h264 and rtsp; not sensor or display (see -# docs/esp-vision.md). +# docs/guides/use-esp-vision-modules.md). # # image works with ulab ndarrays: name ulab with it, or use all (a module # never brings another).