chore(rust): replace LEECH_DISABLE_RUST with --backend on data prepare - #208
Open
jayhesselberth wants to merge 1 commit into
Open
chore(rust): replace LEECH_DISABLE_RUST with --backend on data prepare#208jayhesselberth wants to merge 1 commit into
jayhesselberth wants to merge 1 commit into
Conversation
LEECH_DISABLE_RUST was a temporary workaround for #176, added because "Rust is available" and "Rust is faster here" are different claims and nothing could measure the second. #176 is fixed (#178), so the hatch has outlived its reason (#177). Deleting it outright would have cost the diagnostic: --backend existed only on predict, so the env var was the only way to force the Python path on prepare, and docs/troubleshooting.md said to use it for exactly that. So it moves to the seam #169 built instead of just disappearing. - data prepare gains --backend auto|rust|python, matching predict. - One resolver, _select_prepare_backend(), ANDs the choice with availability and rust_prepare_unsupported_reason(config). Scoped to prepare: it no longer kills leech_core process-wide for move-table parsing and inference, where the extension is a real win. - --backend rust raises rather than falling back, whether the cause is a missing leech_core, an unsupported config, or --workers 1 (the sequential path is Python-only). A forced run that quietly took the other path measures nothing. - LEECH_DISABLE_RUST and the DISABLE_RUST branch are gone from _rust_accel.py; both docs references now say --backend python. Closes #177
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #177.
LEECH_DISABLE_RUSTwas a temporary workaround for #176, added because "Rust is available" and "Rust is faster here" are different claims and nothing could measure the second. They came apart badly — 13 reads/s against 130–234 on the Python pool. #176 is fixed (#178: concurrent batch dispatch, the process-global POD5 reader cache, andpy.detachover the I/O), so the hatch has outlived its reason.Why this isn't a straight delete
#177 offered three options. (a) "delete it" turned out to need a piece of (b) first:
--backendexisted only onpredict, so the env var was the only way to force the Python path onprepare— anddocs/troubleshooting.mdtold people to use it for exactly that, in both the throughput and read-yield sections. Removing it would have taken the diagnostic with it.So the hatch moves to the seam #169 built rather than disappearing.
Changes
data preparegains--backend auto|rust|python, matchingpredict.preparation.parallel._select_prepare_backend(), ANDs the choice with availability andrust_prepare_unsupported_reason(config). One place decides, so the startup log line and the dispatch cannot disagree.prepare. The old variable raised inside thetry:in_rust_accel.py, so it killedleech_coreprocess-wide — move-table parsing and inference included, where the extension is a real win. That was far broader than the one step it was meant to switch.--backend rustraises instead of falling back, whether the cause is a missingleech_core, an unsupported config, or--workers 1(the sequential path is Python-only). A forced run that quietly took the other path measures nothing, which is the whole reason the switch exists.LEECH_DISABLE_RUSTand theDISABLE_RUSTbranch are gone from_rust_accel.py(theosimport with them); both docs references now say--backend python.The two backends produce identical chunks — that is what
tests/test_backend_parity.pyenforces field by field — so this flag only ever changes throughput. The help text says so.Tests
Seven new cases in
tests/test_prepare_dispatch.py::TestBackendSelection, covering each choice against available/unavailable and supported/unsupported configs, plus the raise-don't-fall-back property. They need no POD5/BAM fixtures, matching the rest of that file.1015 passed, 27 skipped.ruff format --check,ruff check, andty checkclean.Downstream
rnabioco/escapepod-models—workflow/rules/charging.smk, rulecharging_leech_prepare— setsLEECH_DISABLE_RUST=1. That should now be dropped outright, since #176 is fixed and the default path is the fast one. If a pin is still wanted, it is--backend python.🤖 Generated with Claude Code