Skip to content

Add subject ID discovery to Dandiset APIs - #1917

Open
AtomicGlance wants to merge 4 commits into
dandi:masterfrom
AtomicGlance:feat/get-subject-ids
Open

Add subject ID discovery to Dandiset APIs#1917
AtomicGlance wants to merge 4 commits into
dandi:masterfrom
AtomicGlance:feat/get-subject-ids

Conversation

@AtomicGlance

Copy link
Copy Markdown
Contributor

Fixes #1457

Add get_subject_ids() to both local Dandiset and remote RemoteDandiset.

The local implementation inspects only immediate child directories named
sub-*. The remote implementation streams asset records ordered by path and
extracts only nested top-level sub-* prefixes, without requesting asset
payloads or metadata. Both methods deduplicate and return stable sorted labels;
root-level files and malformed sub- names are ignored.

The API reference now documents both methods and includes local and remote
usage examples.

Tests:

  • local empty/malformed/root-level cases
  • remote duplicate subjects and path-shape filtering, including verification
    that the API is called with order="path"

Local validation: flake8, syntax checks, and both focused tests pass. The local
mypy invocation is blocked by the checkout's missing types-requests stubs.

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.42%. Comparing base (d91a041) to head (77c39ba).

Files with missing lines Patch % Lines
dandi/dandiset.py 80.00% 2 Missing ⚠️
dandi/dandiapi.py 83.33% 1 Missing ⚠️
dandi/utils.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1917      +/-   ##
==========================================
+ Coverage   77.35%   77.42%   +0.07%     
==========================================
  Files          89       89              
  Lines       13325    13391      +66     
==========================================
+ Hits        10307    10368      +61     
- Misses       3018     3023       +5     
Flag Coverage Δ
unittests 77.42% <94.11%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yarikoptic

Copy link
Copy Markdown
Member

The remote implementation streams asset records ordered by path and
extracts only nested top-level sub-* prefixes

this sounds too heavy (try on 000026) -- there is a dedicated API endpoint
image

so you can quickly get them

time curl --silent -X 'GET' 'https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/paths/' -H 'accept: application/json' | jq -r '.results' | grep path
    "path": "README",
    "path": "dataset_description.json",
    "path": "derivatives",
    "path": "participants.tsv",
    "path": "samples.tsv",
    "path": "sub-EXC022",
    "path": "sub-HCPA1",
    "path": "sub-I38",
    "path": "sub-I41",
    "path": "sub-I45",
    "path": "sub-I46",
    "path": "sub-I48",
    "path": "sub-I52",
    "path": "sub-I53",
    "path": "sub-I55",
    "path": "sub-I56",
    "path": "sub-I57",
    "path": "sub-I58",
    "path": "sub-I59",
    "path": "sub-I60",
    "path": "sub-I61",
    "path": "sub-I62",
    "path": "sub-I63",
    "path": "sub-I64",
    "path": "sub-KC001",
noglob curl --silent -X 'GET'  -H 'accept: application/json'  0.00s user 0.01s system 3% cpu 0.357 total

so check if we interface that endpoint already and use that instead of listing all assets

@AtomicGlance

Copy link
Copy Markdown
Contributor Author

Good catch — I switched the remote implementation to /assets/paths/, so it now reads only the immediate children of the Dandiset root instead of streaming every asset. It also distinguishes directory entries from root-level assets before parsing sub-* names.

I added coverage for the endpoint call and missing-version handling. The six focused subject-discovery tests, lint, and typing pass locally, and I verified the method against production Dandiset 000026 (20 subject IDs). The fix is in 77c39ba0; CI is rerunning now.

@AtomicGlance

Copy link
Copy Markdown
Contributor Author

The matrix has finished: 35 checks pass, including lint, typing, docs, CodeQL, both Codecov checks, and all other test environments. The Windows/Python 3.12 job's only failure was the existing test_get_content_url timing out while reading dandiarchive.s3.amazonaws.com; the subject-discovery tests passed in that job. I tried to rerun the failed job, but GitHub limits upstream workflow reruns to repository maintainers. Could someone rerun that job when convenient?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

request: dandiset.get_subjects() in the Python API

2 participants