Skip to content

Prototype PyMC sampling through an Emscripten Nutpie extension - #348

Draft
twiecki wants to merge 2 commits into
pymc-devs:mainfrom
twiecki:codex/emscripten-lib-probe
Draft

twiecki wants to merge 2 commits into
pymc-devs:mainfrom
twiecki:codex/emscripten-lib-probe

Conversation

@twiecki

@twiecki twiecki commented Sep 11, 2026

Copy link
Copy Markdown
Member

No description provided.

@twiecki
twiecki marked this pull request as draft September 11, 2026 13:00
@twiecki

twiecki commented Sep 11, 2026 •

Copy link
Copy Markdown
Member Author

Following up on the concern about reimplementing Nutpie in the browser: this draft explores compiling the real nutpie._lib as an Emscripten side module, so we can reuse the existing PyMC compiler, Numba callbacks, PyMcModel, transforms/data ownership and nuts-rs sampler in the Xeus runtime.

The import problem in #347 is therefore not fundamental. The blockers encountered were native-only dependencies (initially Tokio/mio), the callback ABI (c_int/usize, rather than fixed 64-bit signatures), and Python imports that pulled in Arrow/storage too early.

I also extracted the existing final ArviZ/xarray conversion from the Arrow decoder. Both native sampling and the WASM path now call the same conversion function. The browser supports nutpie.sample(compiled_model, ...) returning a DataTree with named dimensions, string coordinates, posterior and sample stats, plus optional unconstrained draws. PyArrow is not required for that conversion.

Validation so far:

  • Browser checks for densities/gradients, transformations, deterministic expansion, mutable float32/int32 data, callback lifetime, runtime memory growth and an MMM.
  • An ABI-tagged wheel installed with unmodified pip in a fresh Xeus kernel; the public xarray result and data-update checks pass.
  • Six existing native PyMC/Numba tests plus a focused transformed/shared-data regression pass after the shared-code changes.
  • Release benchmark on the same 15-parameter MMM and runtime, five seeds, two chains × (750 warmup + 500 draws): median sampling/materialization was 7.71 s for the existing browser adapter, 7.90 s through _lib using the same compiled graph, and 6.92 s using Nutpie's compiler. Compilation medians were 14.31 s vs 9.69 s (three builds). This does not establish a benefit from removing the JS boundary: output delivery differs, and median minimum ESS/s was essentially unchanged (22.55 vs 22.31). Zero divergences, but some R-hat values exceeded 1.01 (max 1.0302).

This remains a bounded prototype: a small synchronous chain loop and raw-result decoder are still new; it does not yet reuse all native execution/storage machinery. No progress/live samples, cooperative cancellation, warmup storage, parallel chains, Stan or flow adaptation. The intention is to agree on a reusable upstream execution/output interface and remove the separate browser model/sampler implementation, rather than grow another complete sampling stack.

Details, timing scopes, limitations and reproduction instructions: wasm/evaluation.md. I'd particularly appreciate feedback on the feature boundary and how best to share the synchronous execution/storage interface.

The tested clean-commit wheel is now available as an experimental fork prerelease, with a hash-pinned requirements file, runtime manifest and validation results. It can be used as a Python dependency in the pinned runtime; integrating Mixlab's worker/progress interface is still separate work.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant