Skip to content

add backends - #4108

Open
Intron7 wants to merge 10 commits into
mainfrom
add-scverse-backends
Open

add backends#4108
Intron7 wants to merge 10 commits into
mainfrom
add-scverse-backends

Conversation

@Intron7

@Intron7 Intron7 commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

  • Add Scanpy backend dispatch for RAPIDS single-cell across preprocessing, tools, and sc.get.aggregate.
  • Require scverse-backends>=0.0.3 and exercise its lazy entry-point discovery and automatic backend-parameter signature injection.
  • Rebase the branch onto current main, preserving upstream API changes alongside dispatch decorators.

Validation

  • uv run --group test pytest -q tests/test_backends.py tests/test_package_structure.py::test_sig_conventions
  • uv run --group test pytest -q tests/test_aggregated.py tests/test_pca.py tests/test_embedding.py tests/test_rank_genes_groups.py

@codecov

codecov Bot commented May 7, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2047 2 2045 1076
View the top 2 failed test(s) by shortest run time
tests/plotting/legacy/embedded/test_spatial.py::test_manual_equivalency[na_color.transparent]
Stack Traces | 1.06s run time
self = <tempfile._TemporaryFileCloser object at 0x7f1ed9630050>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92m__del__#x1B[39;49;00m(#x1B[96mself#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
        close_called = #x1B[96mself#x1B[39;49;00m.close_called#x1B[90m#x1B[39;49;00m
        #x1B[96mself#x1B[39;49;00m.cleanup()#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m close_called:#x1B[90m#x1B[39;49;00m
>           _warnings.warn(#x1B[96mself#x1B[39;49;00m.warn_message, #x1B[96mResourceWarning#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           ResourceWarning: Implicitly cleaning up <HTTPError 404: 'Not Found'>#x1B[0m

#x1B[1m#x1B[31m../...../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/tempfile.py#x1B[0m:484: ResourceWarning

#x1B[33mThe above exception was the direct cause of the following exception:#x1B[0m

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x7f1ed9e5a6c0>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    #x1B[0m#x1B[37m@classmethod#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mfrom_call#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mcls#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        func: Callable[[], TResult],#x1B[90m#x1B[39;49;00m
        when: Literal[#x1B[33m"#x1B[39;49;00m#x1B[33mcollect#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33msetup#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mcall#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mteardown#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
        reraise: #x1B[96mtype#x1B[39;49;00m[#x1B[96mBaseException#x1B[39;49;00m] | #x1B[96mtuple#x1B[39;49;00m[#x1B[96mtype#x1B[39;49;00m[#x1B[96mBaseException#x1B[39;49;00m], ...] | #x1B[94mNone#x1B[39;49;00m = #x1B[94mNone#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
    ) -> CallInfo[TResult]:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Call func, wrapping the result in a CallInfo.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    :param func:#x1B[39;49;00m
    #x1B[33m        The function to call. Called without arguments.#x1B[39;49;00m
    #x1B[33m    :type func: Callable[[], _pytest.runner.TResult]#x1B[39;49;00m
    #x1B[33m    :param when:#x1B[39;49;00m
    #x1B[33m        The phase in which the function is called.#x1B[39;49;00m
    #x1B[33m    :param reraise:#x1B[39;49;00m
    #x1B[33m        Exception or exceptions that shall propagate if raised by the#x1B[39;49;00m
    #x1B[33m        function, instead of being wrapped in the CallInfo.#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        excinfo = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        instant = timing.Instant()#x1B[90m#x1B[39;49;00m
        #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
>           result: TResult | #x1B[94mNone#x1B[39;49;00m = func()#x1B[90m#x1B[39;49;00m
                                     ^^^^^^#x1B[90m#x1B[39;49;00m

#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14....../site-packages/_pytest/runner.py#x1B[0m:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14....../site-packages/_pytest/runner.py#x1B[0m:250: in <lambda>
    #x1B[0m#x1B[94mlambda#x1B[39;49;00m: runtest_hook(item=item, **kwds),#x1B[90m#x1B[39;49;00m
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14.../site-packages/pluggy/_hooks.py#x1B[0m:512: in __call__
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._hookexec(#x1B[96mself#x1B[39;49;00m.name, #x1B[96mself#x1B[39;49;00m._hookimpls.copy(), kwargs, firstresult)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14.../site-packages/pluggy/_manager.py#x1B[0m:120: in _hookexec
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._inner_hookexec(hook_name, methods, kwargs, firstresult)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14.../site-packages/_pytest/logging.py#x1B[0m:865: in pytest_runtest_call
    #x1B[0m#x1B[94myield#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14.../site-packages/_pytest/capture.py#x1B[0m:900: in pytest_runtest_call
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m (#x1B[94myield#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            ^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14.../site-packages/_pytest/skipping.py#x1B[0m:268: in pytest_runtest_call
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m (#x1B[94myield#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            ^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:183: in pytest_runtest_call
    #x1B[0mcollect_unraisable(item.config)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:79: in collect_unraisable
    #x1B[0m#x1B[94mraise#x1B[39;49;00m errors[#x1B[94m0#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config = <_pytest.config.Config object at 0x7f1f53e32f90>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mcollect_unraisable#x1B[39;49;00m(config: Config) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        pop_unraisable = config.stash[unraisable_exceptions].pop#x1B[90m#x1B[39;49;00m
        errors: #x1B[96mlist#x1B[39;49;00m[pytest.PytestUnraisableExceptionWarning | #x1B[96mRuntimeError#x1B[39;49;00m] = []#x1B[90m#x1B[39;49;00m
        meta = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        hook_error = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            #x1B[94mwhile#x1B[39;49;00m #x1B[94mTrue#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                    meta = pop_unraisable()#x1B[90m#x1B[39;49;00m
                #x1B[94mexcept#x1B[39;49;00m #x1B[96mIndexError#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                    #x1B[94mbreak#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                #x1B[94mif#x1B[39;49;00m #x1B[96misinstance#x1B[39;49;00m(meta, #x1B[96mBaseException#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
                    hook_error = #x1B[96mRuntimeError#x1B[39;49;00m(#x1B[33m"#x1B[39;49;00m#x1B[33mFailed to process unraisable exception#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
                    hook_error.__cause__ = meta#x1B[90m#x1B[39;49;00m
                    errors.append(hook_error)#x1B[90m#x1B[39;49;00m
                    #x1B[94mcontinue#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                msg = meta.msg#x1B[90m#x1B[39;49;00m
                #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE                   pytest.PytestUnraisableExceptionWarning: Exception ignored while calling deallocator <function _TemporaryFileCloser.__del__ at 0x7f1f5816f740>: None#x1B[0m

#x1B[1m#x1B[31m../../../..../scanpy/B9PcT7QG/hatch-test.few-extras/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:67: PytestUnraisableExceptionWarning
tests/test_datasets.py::test_ebi_expression_atlas
Stack Traces | 2.86s run time
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f1ed0572660>

    #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.internet#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_ebi_expression_atlas#x1B[39;49;00m(monkeypatch: pytest.MonkeyPatch):#x1B[90m#x1B[39;49;00m
        #x1B[94mfrom#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[04m#x1B[96mscanpy#x1B[39;49;00m#x1B[04m#x1B[96m.#x1B[39;49;00m#x1B[04m#x1B[96mdatasets#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[94mimport#x1B[39;49;00m _ebi_expression_atlas #x1B[94mas#x1B[39;49;00m ea_mod#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[90m# make sure we use chunks when testing.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[90m# This dataset has <8M entries, so 4M entries/chunk = 2 chunks#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94massert#x1B[39;49;00m #x1B[96mhasattr#x1B[39;49;00m(ea_mod, #x1B[33m"#x1B[39;49;00m#x1B[33mCHUNK_SIZE#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
        monkeypatch.setattr(ea_mod, #x1B[33m"#x1B[39;49;00m#x1B[33mCHUNK_SIZE#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[96mint#x1B[39;49;00m(#x1B[94m4e6#x1B[39;49;00m))#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
>       adata = sc.datasets.ebi_expression_atlas(#x1B[33m"#x1B[39;49;00m#x1B[33mE-MTAB-4888#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m

#x1B[1m#x1B[31mtests/test_datasets.py#x1B[0m:85: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[.../scanpy/datasets/_ebi_expression_atlas.py#x1B[0m:159: in ebi_expression_atlas
    #x1B[0mdownload_experiment(accession)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[.../scanpy/datasets/_utils.py#x1B[0m:16: in wrapper
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m f(*args, **kwargs)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[.../scanpy/datasets/_ebi_expression_atlas.py#x1B[0m:48: in download_experiment
    #x1B[0msniff_url(accession)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[.../scanpy/datasets/_ebi_expression_atlas.py#x1B[0m:39: in sniff_url
    #x1B[0m#x1B[94mwith#x1B[39;49;00m urlopen(base_url):  #x1B[90m# Check if server up/ dataset exists#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
         ^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:187: in urlopen
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m opener.open(url, data, timeout)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:493: in open
    #x1B[0mresponse = meth(req, response)#x1B[90m#x1B[39;49;00m
               ^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:602: in http_response
    #x1B[0mresponse = #x1B[96mself#x1B[39;49;00m.parent.error(#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:531: in error
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._call_chain(*args)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:464: in _call_chain
    #x1B[0mresult = func(*args)#x1B[90m#x1B[39;49;00m
             ^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f1eb81069c0>
req = <urllib.request.Request object at 0x7f1ed0423ad0>
fp = <http.client.HTTPResponse object at 0x7f1ed9686fb0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f1efcfe2350>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mhttp_error_default#x1B[39;49;00m(#x1B[96mself#x1B[39;49;00m, req, fp, code, msg, hdrs):#x1B[90m#x1B[39;49;00m
>       #x1B[94mraise#x1B[39;49;00m HTTPError(req.full_url, code, msg, hdrs, fp)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       urllib.error.HTTPError: HTTP Error 404: Not Found#x1B[0m
#x1B[1m#x1B[31mE       https://www.ebi.ac..../sc/experiments/E-MTAB-4888/#x1B[0m

#x1B[1m#x1B[31m../..................../_temp/uv-python-dir/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/urllib/request.py#x1B[0m:611: HTTPError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Intron7 added 3 commits July 23, 2026 18:06
# Conflicts:
#	pyproject.toml
#	src/scanpy/get/_aggregated.py
#	src/scanpy/preprocessing/_pca/__init__.py
#	src/scanpy/tools/_diffmap.py
#	src/scanpy/tools/_embedding_density.py
#	src/scanpy/tools/_rank_genes_groups.py

@flying-sheep flying-sheep left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! I didn’t look at the test yet (I will once you mark it ready for review)

Comment thread src/scanpy/_settings/__init__.py Outdated
Comment on lines +186 to +190
def use_backend(self, backend: str):
"""Temporarily set the active computational backend."""
from .._backends import settings as backend_settings

return backend_settings.use_backend(backend)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be removed, sc.settings.override(backend=...) exists, and if it doesn’t work for some reason, we should fix that

Comment thread src/scanpy/_settings/__init__.py Outdated
Comment on lines +198 to +202
def get_backend(self, name: str):
"""Look up an installed computational backend by name or alias."""
from .._backends import settings as backend_settings

return backend_settings.get_backend(name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this exist? Does the user ever need to access the backend object?

Comment thread src/scanpy/_backends.py Outdated
Comment thread tests/test_backends.py Outdated
return FakeRapidsBackend


DISPATCHED_FUNCTIONS = [

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t want to keep this in sync. Can’t the decorator just populate a list that we can access here?

One thing to note is that it probably can’t just put its function argument in the list, as other decorators might be applied after this one.

Comment thread pyproject.toml Outdated
"pynndescent>=0.5.13",
"scikit-learn>=1.6",
"scipy>=1.15",
"scverse-backends>=0.0.3,<0.1",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this has no dependencies right?

@Intron7 Intron7 Jul 31, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No None. I'll also remove the pin

Intron7 and others added 2 commits July 31, 2026 09:58
@Intron7 Intron7 added this to the 1.13.0 milestone Jul 31, 2026
@Intron7
Intron7 marked this pull request as ready for review August 13, 2026 09:08
Comment thread src/scanpy/_backends.py
host_name="scanpy",
trusted_backends={
"rapids-singlecell": {
"aliases": ["cuda", "rapids", "rapids_singlecell"],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to have that many aliases that mean the same thing? Could be confusing.
I think aliases are mostly useful for backwards-compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants