Drop tensorflow_probability so scenvi imports on current jax - #93
Open
Marius1311 wants to merge 1 commit into
Open
Drop tensorflow_probability so scenvi imports on current jax#93Marius1311 wants to merge 1 commit into
Marius1311 wants to merge 1 commit into
Conversation
`pip install scenvi` currently produces a package that cannot be imported at
all. tensorflow_probability is pinned to ^0.22.0, i.e. <0.23, and tfp 0.22 fails
against any recent jax:
AttributeError: module 'jax.interpreters.xla' has no attribute
'pytype_aval_mappings'
Re-pinning does not fix this for long: tfp's last release is 0.25.0 (November
2024) and it is unmaintained, while jax is at 0.11. The dependency is also far
larger than what is used -- four log-densities and one matrix helper, all
closed-form. So they are written out against jax directly and tfp is dropped.
* scenvi/_dists.py implements Poisson, negative binomial, zero-inflated
negative binomial and unit-variance normal log-densities with
jax.scipy.special.gammaln / jax.nn.log_sigmoid, plus fill_triangular.
* tests/test_dists.py checks every one of them against the tfp original to
rtol 1e-11 in float64, including saturated logits and large counts. It skips
where tfp is absent, which after this commit is everywhere except a
deliberately pinned environment; it is the record that the replacements
reproduce what they replaced.
Two details worth flagging, both caught by that differential test:
* tfp builds Inflated's mixture from categorical logits [d, -d], whose
difference is 2d. The zero-inflation weight is therefore sigmoid(2d), not
sigmoid(d). Using the latter silently rescales the parameter.
* fill_triangular's fill order is not the obvious one: the last m - n entries
are laid down first, then the whole vector reversed on top, then the lower
triangle taken.
Removing tfp is necessary but not sufficient, because the flax pin has gone the
same way: ^0.10.4 caps below 0.11, and flax 0.10.7 calls
jax.core.get_opaque_trace_state, which jax 0.11 removed. flax, optax and clu
therefore lose their upper caps, and jax becomes a direct dependency rather than
one inherited from flax. The bounds are written >= rather than ^ deliberately --
poetry's caret on a 0.x version caps at the next minor, which is how these pins
became stale in the first place.
Verified on a clean install resolving to jax 0.11.0, flax 0.12.8, optax 0.2.8,
clu 0.0.12 and no tfp: `import scenvi` and `scenvi.ENVI` both work, and the
suite passes. Also still passes against jax 0.4.23 with tfp 0.22 installed,
where the differential tests run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 5, 2026
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
pip install scenvicurrently gives you a package that cannot be imported at all.tensorflow_probabilityis pinned to^0.22.0— i.e.<0.23— and tfp 0.22 failsagainst any recent jax:
This is what is failing every
buildjob on this repo right now, including on#91 and #92, where collection errors out before any test runs.
Re-pinning buys little: tfp's last release is 0.25.0 (November 2024) and it is
unmaintained, while jax is at 0.11. The dependency is also much larger than what
scenvi uses — four log-densities and one matrix helper, all closed-form — so this
removes it rather than chasing a compatible window.
What this does
scenvi/_dists.pyimplements Poisson, negative binomial, zero-inflated negativebinomial and unit-variance normal log-densities against
jax.scipy.special.gammaln/jax.nn.log_sigmoid, plusfill_triangular.scenvi/ENVI.pyloses itstensorflow_probabilityimport and calls the localfill_triangular.tests/test_dists.pychecks each replacement against the tfp original tortol=1e-11in float64, including saturated logits and large counts. It skipswhere tfp is absent — after this PR, everywhere except a deliberately pinned
environment — and exists as the record that the replacements reproduce what they
replaced.
Two details that the differential test caught, and that are easy to get wrong:
Inflated's mixture from categorical logits[d, -d], whosedifference is
2d. The zero-inflation weight is thereforesigmoid(2d), notsigmoid(d)— using the latter silently rescales the parameter.fill_triangular's fill order is not the obvious one: the lastm - nentriesare laid down first, then the whole vector reversed on top, then the lower
triangle taken.
[1..6]becomes[[4,0,0],[6,5,0],[3,2,1]].The flax pin has gone the same way
Removing tfp is necessary but not sufficient.
flax = "^0.10.4"caps below 0.11,and flax 0.10.7 calls
jax.core.get_opaque_trace_state, which jax 0.11 removed —so a fresh install still fails, just later and for a different reason. flax, optax
and clu therefore lose their upper caps, and jax becomes a direct dependency
instead of one inherited from flax.
The new bounds are written
>=rather than^deliberately: poetry's caret on a0.xversion caps at the next minor, which is precisely how these pins becamestale.
^0.22.0meant<0.23;^0.10.4means<0.11.Testing
import scenviandscenvi.ENVIwork; 8 passed, 1 skippedThe first row is the case that is broken on
maintoday.