Support recent zigpy and Python 3.14 - #63
Conversation
`asyncio.get_event_loop()` no longer creates an event loop when none is set: as of Python 3.14 it raises `RuntimeError` instead. Every `click_coroutine` callback ran through it, so all `zigpy radio ...` commands failed immediately with "There is no current event loop in thread 'MainThread'". Create the loop explicitly instead. It has to be a single shared loop rather than a per-invocation `asyncio.run()`: the `radio` group callback constructs the `ControllerApplication`, the subcommand then uses it and `radio_cleanup` shuts it down, and those objects are bound to the loop they were created on.
`radio restore` uses `zigpy.backups.NetworkBackup`, but the module was never imported: the attribute only resolved because `zigpy.application` happens to import it, which is not something to rely on. Import it directly and drop `zigpy.state`/`zigpy.zdo`, which are unused. Ruff could not flag those as unused imports since they only bind the name `zigpy`, which is what hid the missing import in the first place.
zigpy has required Python 3.11 since 0.85.0, so `requires-python = ">=3.8"` was already unsatisfiable — it also made `uv sync` fail outright, since uv resolves for the whole declared Python range. The dependency floors had drifted just as far: resolving them (`uv lock --resolution lowest-direct`) picked up radio libraries predating the current zigpy APIs, and unpinned `scapy` resolved back to 2.3.1, a Python 2-era release that fails to build at all. Pin each one to a release that works with recent zigpy and supports Python 3.11+.
The shared zigpy CI workflow prefers `uv sync --group ci` and only falls back to `requirements_test.txt` when that fails, so declare the groups the way zigpy itself does and drop the fallback file. The `testing` group gains `pre-commit` and `coverage`, and `ci` adds `pytest-xdist` and `pytest-github-actions-annotate-failures`. The fallback path installed all of these itself, so without them the pre-commit and coverage CI jobs would break once the modern path starts being used. Every entry gets a floor, for the same reason the runtime dependencies did: unpinned, `coverage` and `pytest-xdist` resolve to releases too old to import, and `pytest` 7.1.2 uses `ast.Str`, removed in Python 3.12, so the previous floors could not run on the 3.13/3.14 matrix entries at all. `uv.lock` is ignored rather than committed, matching zigpy.
zigpy-zboss 2.x pins `zigpy<2`, so depending on it held the whole install back: the resolver had to fall to zboss 1.2.0 (which declares an unbounded `zigpy>=0.60.2`) to allow a recent zigpy, and zboss 1.2.0 emits deprecation warnings against it. Drop the radio until upstream relaxes the cap. `zboss` is no longer a valid `radio` argument, so it now fails on the `click.Choice` with the list of supported radios rather than at import time.
`asyncio.set_event_loop()` was only called on the branch that creates the loop, so the registration could go stale if anything cleared the thread's current loop in between commands. Nothing in the CLI does that today, but bellows (`uart.py`) and zigpy-deconz (`api.py`) both call `asyncio.get_event_loop()` at runtime, and on Python 3.14 that raises when no loop is set. Re-register unconditionally instead; it is idempotent when already current.
Teardown restored `_LOOP` to whatever it was before the test but cleared the thread's current loop unconditionally, so the two could disagree: `_LOOP` pointing at one loop while `asyncio.get_event_loop()` saw another. `_LOOP` is always `None` outside the tests, so nothing was actually wrong, but the save/restore bought nothing and would mislead once async tests exist. Reset both to `None` at each end instead, and clear the current loop on setup too so a test never inherits one from elsewhere.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #63 +/- ##
========================================
+ Coverage 2.61% 7.61% +5.00%
========================================
Files 9 9
Lines 613 617 +4
========================================
+ Hits 16 47 +31
+ Misses 597 570 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
zigpy-xbee 0.22.0 ships zigpy/zigpy-xbee#179, which migrates off the quirks API zigpy moved out to zha-device-handlers. Before it, `zigpy radio xbee` failed at import with `ModuleNotFoundError: No module named 'zhaquirks'` on any install without zha-quirks present. 0.22.0 requires `zigpy>=2.0.0`, which becomes the real floor for the whole project, so declare that directly rather than leaving a `zigpy>=0.85.0` that no longer describes anything installable.
puddly
left a comment
puddly
left a comment
There was a problem hiding this comment.
Thanks!! zigpy-cli has gotten a bit crusty 😅
|
@puddly Any opinion on v1.3.0 or v2.0.0 release for this? (zigpy-zboss is technically a breaking change but zigpy-cli doesn't seem to have followed semantic versioning very well so far, so I'm not sure it's worth the jump 😅). I guess zigpy is already at 2.0.0, so maybe we can justify it to keep up with that as well. 😄 |
|
It's a development tool either way, I think any version is good. 2.0.0 seems good. We can throw in zigpy-ziggurat in 2.1.0 or something. |
zigpy-review-bot
left a comment
zigpy-review-bot
left a comment
There was a problem hiding this comment.
Reviewed the full branch at d9328c2 in a clean worktree, on Python 3.11 and 3.14, at both the declared floors and the latest resolvable set. Approving — the changes are correct and, unusually, the description matches what the code actually does on every load-bearing claim I checked. Everything below is non-blocking.
What I verified
The 3.14 event-loop fix is correct, and the shared loop is genuinely required.
Confirmed on 3.14.5 that asyncio.get_event_loop() raises RuntimeError: There is no current event loop in thread 'MainThread' when no loop is set, so the old click_coroutine did break every zigpy radio ... command.
I then instrumented the three callbacks with a stub ControllerApplication driven through click's CliRunner: the group callback's __init__, the subcommand's connect()/load_network_info(), and radio_cleanup's shutdown() all report the same loop id — and shutdown() still runs when the subcommand raises, since click fires ctx.call_on_close during context teardown either way. So the single shared loop is load-bearing rather than incidental, and a per-invocation asyncio.run() really would have broken cleanup.
Also confirmed under -W error::DeprecationWarning that asyncio.set_event_loop() is not deprecated on 3.14, so re-registering on every lookup costs nothing.
End-to-end against a pty: zigpy -vv radio znp <pty> info reaches zigpy_znp.api._skip_bootloader and fails with a clean TimeoutError — group callback, subcommand and teardown all executed on the shared loop.
The missing-import class of bug is now fully gone, not just fixed in one spot.
Rather than spot-checking, I ran an AST pass over every module in zigpy_cli/, comparing each zigpy.<submodule> attribute chain against what that module actually imports. Only radio.py (zigpy.backups, zigpy.types) and database.py (zigpy.appdb) reach through the bare zigpy name, and after this PR both are directly imported — no other latent dependency on zigpy.application's import side effects remains. Dropping zigpy.state, zigpy.zdo and zigpy.zdo.types is safe; nothing in the package references them.
The floors are honest.
uv sync --group ci --resolution lowest-direct resolves and installs on both 3.11 and 3.14 (zigpy 2.0.0, bellows 0.47.0, zigpy-deconz 0.25.0, zigpy-xbee 0.22.0, zigpy-zigate 0.14.0, zigpy-znp 1.0.0, scapy 2.5.0, click 8.1.0), and at those floors all five <radio>.zigbee.application modules plus every zigpy_cli module import cleanly on both versions — including scapy.all at 2.5.0 on 3.14, which was the one I expected to break. The suite passes at the floors on 3.11 and 3.14, and at latest on 3.14.
Every zigpy symbol the package touches also resolves at the 2.0.0 floor, not just at 2.1.0: appdb.DB_VERSION, backups.NetworkBackup.from_dict, types.Channels, ota.image.{ElementTagId,HueSBLOTAImage,parse_ota_image}, ota.validators.validate_ota_image, types.named._hex_string_to_bytes.
The zboss removal is justified by current upstream state, not just as of when you wrote it.
zigpy-zboss's latest release (2.0.5) still declares zigpy<2,>=0.92.0, so it does cap the whole install below zigpy 2.x — the reasoning holds today. No dangling references either: RADIO_TO_PYPI is derived from RADIO_TO_PACKAGE, RADIO_LOGGING_CONFIGS is updated in step, and the README never mentioned zboss.
The packaging move matches zigpy, and CI is demonstrably taking the modern path.
The shared workflow's if ! uv sync --group ci fallback is what previously installed pre-commit, pytest-xdist and pytest-github-actions-annotate-failures, so folding those into the groups was required rather than cosmetic. All four Prepare base dependencies jobs are green here, which means uv sync --group ci succeeded and the fallback never ran — worth noting that the fallback is now a dead end (no requirements_test.txt), so if that path is ever taken again it will fail loudly rather than recover. zigpy is in exactly the same position, so this is the org convention, not a regression.
The added tests are the right ones. I checked that test_get_or_create_event_loop_reregisters_cleared_loop fails against create-only registration, and that test_click_coroutine_without_running_loop fails against the old asyncio.get_event_loop() on 3.14 — both genuinely guard their fix rather than only exercising the new code.
A second-opinion static pass with another model over the same diff returned no findings, independently reaching the same conclusions on the loop refactor, the removed imports, the floors and the zboss removal.
Two optional notes
asyncio_mode is the one thing not carried over from zigpy's pyproject — inline on the pytest-asyncio line. No correctness risk, purely ergonomic.
Loop teardown is now explicitly owned by this code, which turns async-generator finalization into a visible loose end. _LOOP is created here but never closed, and loop.shutdown_asyncgens() is never called. The old get_event_loop() path did exactly the same, so this is not a regression and nothing in this PR needs to change — but radio packet-capture consumes app.packet_capture(...) as an async generator inside a TaskGroup, so on Ctrl-C that generator's finally may never run. Now that the loop's lifetime is explicit and in one place, that is a cheap follow-up if it ever misbehaves in practice.
On the version question already settled in the comments: 2.0.0 looks right to me too — dropping 3.8–3.10, removing the testing extra, and removing a radio are three separate user-visible breaks landing in one release.
| "coverage[toml]>=7.0", | ||
| "pre-commit>=3.5", | ||
| "pytest>=8.4", | ||
| "pytest-asyncio>=1.0", |
There was a problem hiding this comment.
Optional, for consistency with zigpy: zigpy's own pyproject.toml pairs pytest-asyncio with
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"and this PR adds the dependency without either. There is no correctness risk in leaving it out — I checked that both pytest 8.4.0 (the declared floor) and 9.1.1 fail an unmarked async def test rather than silently skipping it, so nothing can pass unnoticed. It just means the first async test added here will need an explicit @pytest.mark.asyncio, and setting the fixture loop scope now avoids a behaviour change later. Entirely reasonable to skip if you would rather keep the diff focused.
There was a problem hiding this comment.
Doesn't matter now but I think we can add it for consistency. Did so with: 1a8706a (though the second part of the commit message with function being the default is not correct – doesn't really matter as we're squash-merging and I edit out the commit body anyway (or have Refined GitHub on to auto do that)).
`pytest-asyncio` was already a test dependency but had no configuration, which leaves `asyncio_mode` at `strict`. In strict mode a plain `@pytest.fixture` async fixture is not handled at all — pytest errors out with "requested an async fixture ... with no plugin or hook that handled it" even when the test itself carries `@pytest.mark.asyncio`. `auto` makes async tests and fixtures work without markers, as they do in zigpy. `asyncio_default_fixture_loop_scope` silences no warning on pytest-asyncio 1.x, where `function` is already the default, but pinning it keeps the fixture and its test on one loop if that default ever changes.
| # uv | ||
| uv.lock |
There was a problem hiding this comment.
note: uv.lock is added to .gitignore for now, but we may want to track that in the future. We can look at that in a future PR.
3 participants
Brings zigpy-cli up to date with recent zigpy (2.1.0) and the 3.11–3.14 Python matrix.
zigpy radiowas completely broken on Python 3.14, anduv syncfailed outright, so this fixes both along with the packaging metadata that had drifted out of date.Every
zigpy radio ...command crashed on Python 3.14click_coroutinecalledasyncio.get_event_loop()from a sync context. As of Python 3.14 that raises instead of creating a loop, so every async command died before doing anything:The loop is now created explicitly. It has to be a single shared loop rather than a per-invocation
asyncio.run(), because the work is split across three callbacks: theradiogroup callback constructs theControllerApplication, the subcommand awaitsconnect()on it, andradio_cleanup(registered viactx.call_on_close) later awaitsshutdown().connect()creates the transport and background tasks on whichever loop is running at the time, andshutdown()has to tear them down on that same loop.asyncio.set_event_loop()is called on every lookup rather than only when the loop is created. The concrete reason is thatasyncio.run()clears the thread's current-loop registration when it exits, so a create-only registration could be left stale for the rest of the process; re-registering is idempotent and self-heals. Radio libraries do callasyncio.get_event_loop()at runtime (bellows/uart.py,zigpy_deconz/api.py), though those particular calls happen inside a running loop, where the running loop takes precedence — so this is defensive rather than a fix for an observed crash.zigpy.backupswas used without being importedradio restorecallszigpy.backups.NetworkBackup.from_dict(), butradio.pynever importedzigpy.backups— it only resolved becausezigpy.applicationhappens to import it. Now imported directly.zigpy.state,zigpy.zdoandzigpy.zdo.typeswere imported but unused, and are dropped. Ruff could not flag either problem: those imports only bind the namezigpy, which is what hid the missing import in the first place.Packaging metadata
requires-pythonwas>=3.8, which was already false. zigpy has required 3.11 since 0.85.0, andradio.pyusesasyncio.TaskGroup(3.11+). It also madeuv syncfail, since uv resolves across the whole declared range — the resolver names the>=3.8range and zigpy's>=3.11as the conflict. Now>=3.11.uv lock --resolution lowest-directpicked radio libraries predating the zigpy APIs in use, and unpinnedscapyresolved to 2.3.1 — a Python 2-era release that fails to build (os.chmod(fname,0755)). Floors are now set to releases that work with recent zigpy on 3.11+; all five radio libraries were checked to import at their declared floors.[dependency-groups](testing+ci), matching zigpy, andrequirements_test.txtis removed. The shared CI workflow triesuv sync --group cifirst and only falls back torequirements_test.txt; the fallback also installedpre-commit,pytest-xdistandpytest-github-actions-annotate-failuresitself, so those had to move into the groups or the pre-commit job and the-n autopytest runs would break once the modern path is taken. The test floors were stale too —pytest7.1.2 usesast.Str, removed in Python 3.14, so resolving at the declared floors could not run on the 3.14 matrix entry at all. (CI itself was not broken by this, since the fallback installedpytestunpinned.)uv.lockis gitignored rather than committed, matching zigpy.Breaking changes
zigpy>=0.75.0floor itself required 3.9+. On 3.9/3.10 the CLI could only ever pair with zigpy ≤0.84, andradio packet-capturealready required 3.11 (asyncio.TaskGroup).testingextra is gone. Dependency groups are not extras, sopip install zigpy-cli[testing]no longer installs test dependencies — useuv sync --group testing, or pip 25.1+--group. Nothing in the repo or CI referenced the extra.zigpy-zboss support removed
zigpy-zboss 2.x pins
zigpy<2, so depending on it held the whole install back: the resolver had to fall to zboss 1.2.0 (which declares an unboundedzigpy>=0.60.2) to allow a recent zigpy, and 1.2.0 emits deprecation warnings against it. Dropped for now — happy to restore it once the cap upstream is relaxed.zbossis no longer a validradioargument, so it now fails on theclick.Choicewith the list of supported radios.zigpy radio xbeeneeded zigpy-xbee 0.22.0zigpy moved the quirks API out of the library (zigpy#1846), leaving
zigpy.quirksas a shim that forwards tozhaquirks, and zigpy-xbee still subclassedzigpy.quirks.CustomDevice. Any install without zha-quirks present therefore failed at import:zigpy/zigpy-xbee#179 fixed this and shipped in zigpy-xbee 0.22.0, so the floor here is set to that release. It also fixed a
serialxincompatibility in zigpy-xbee's baudrate setter, whichinit_api_mode()reaches only once entering AT command mode at the configured baudrate has failed and it starts scanning the other known baudrates.0.22.0 requires
zigpy>=2.0.0, which makes 2.0.0 the real floor for the whole project, so that is now declared directly instead of azigpy>=0.85.0that no longer described anything installable.Testing
pre-commit run --all-files(black + ruff) pass on 3.14.asyncio.get_event_loop()raises on 3.14, and the create-only registration fails the re-registration test.uv sync --resolution lowest-direct) and the latest resolvable set (zigpy 2.1.0); the suite passes on both.zigpy radio znp <pty> infoandzigpy radio xbee <pty> infoexercised against a pty: group callback, subcommand and cleanup all run on the shared loop, and the failure propagates cleanly. Both reach the radio protocol layer.Not tested against real hardware — radio verification used a pty, which reaches the protocol layer but not a physical coordinator.