Repository navigation
refactor: check DocDB v2 locations against Code Ocean data assets - #68
Draft
seanmcculloch wants to merge 1 commit into
Draft
seanmcculloch wants to merge 1 commit into
seanmcculloch wants to merge 1 commit into
Conversation
seanmcculloch
added this pull request to stack #67
October 1, 2026 19:04
Collaborator
Author
|
restrict this to aind-open-data |
seanmcculloch
changed the title
seanmcculloch
force-pushed
the
record-consistency-code-ocean-location
branch
from
October 1, 2026 19:38
0cfe07b to
e78570f
Compare
dbirman
force-pushed
the
record-consistency-code-ocean-location
branch
2 times, most recently
from
October 7, 2026 05:28
8d55805 to
e2109b0
Compare
seanmcculloch
force-pushed
the
record-consistency-code-ocean-location
branch
2 times, most recently
from
October 13, 2026 22:40
ab2309f to
da33b72
Compare
seanmcculloch
force-pushed
the
record-consistency-code-ocean-location
branch
from
October 13, 2026 22:40
da33b72 to
441f471
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Add a
code_ocean_data_assetssource and a check that fails every DocDB v2 record whoselocationdisagrees with any visible Code Ocean data asset listed in itsother_identifiers. Checks may now returnNonefor records they cannot judge (no row, counted in a newskipped_count), and sources may pass extra fields such ascode_ocean_idsto checks without storing them.Review notes
On today's data, 12 of 121,237 compared records fail; 4,799 are skipped (no Code Ocean ID, or none visible to the token). Every failure is a record whose
locationis a private or ephys bucket while its Code Ocean asset reads fromaind-open-data.codeoceanis pinned below 0.17 because 0.17 requires Code Ocean server 4.8 and ours is 4.7.3. The source reads its token fromCUSTOM_KEY; the catalog sweep takes about 100 seconds.