Repository navigation
Fix resubmit_job_to_watched_files publishing bare filename instead of archive path - #700
Merged
Merged
Conversation
Run.filename only ever stores the bare filename, not a path, so
resubmit_job_to_watched_files was publishing an unusable filename to
the watched-files queue instead of a real file path.
Add find_file_in_archive() which resolves an instrument + filename to
its real path: a flat lookup under IMAT_DIR for IMAT, or a cycle-aware
lookup under ARCHIVE_DIR/NDX{INSTRUMENT}/Instrument/data for every
other instrument (using the run's start date to guess the likely cycle
folder before falling back to a recursive search). Wire this into
resubmit_job_to_watched_files, raising JobRequestError when the file
cannot be located.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #700 +/- ##
==========================================
+ Coverage 96.76% 96.81% +0.04%
==========================================
Files 50 50
Lines 2227 2259 +32
==========================================
+ Hits 2155 2187 +32
Misses 72 72 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Real archive data shows some years have up to 6+ sub-cycles (e.g. cycle_02_6), so widen the year-guessed fast-path candidates from 1-5 to 1-8 to reduce reliance on the recursive fallback search.
find_file_in_archive's recursive fallback delegated to _safe_find_file_in_dir, which raises AuthError when the target directory doesn't exist at all. This meant resubmit_job_to_watched_files surfaced a 403 Forbidden instead of the intended 400 JobRequestError whenever an instrument's archive data directory wasn't present, as seen failing in CI's e2e-job-maker test_resubmit_job_file_not_found. Guard the fallback so a missing instrument data directory is treated as a normal not-found result.
Pasarus
marked this pull request as ready for review
September 30, 2026 11:05
Dagonite
reviewed
Sep 30, 2026
|
|
||
| for year in (run_start.year, run_start.year - 1): | ||
| for cycle_name in _cycle_candidate_names(year): | ||
| candidate = instrument_dir / cycle_name / filename |
Collaborator
There was a problem hiding this comment.
With this line and 193, could they return 400 for valid files? Run.filename doesn’t include the extension because the job controller uses .stem. Lookups would search for MAR123456 instead of MAR123456.nxs (for example).
The tests added include extensions.
Member
Author
There was a problem hiding this comment.
You are entirely correct, I've also decided that I hate the recursive archive search and how long it takes so we are only doing "fast" searches.
find_file_in_archive now appends .nxs to filename if missing, since Run.filename is not guaranteed to include the extension even though raw NeXus files on the archive always carry it. Also removes the recursive fallback search of the instrument's data directory, since a full scan is too expensive to run on every lookup - a miss on every year-guessed cycle candidate is now treated as not found.
Dagonite
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Run.filenameonly ever stores the bare filename (not a path), soresubmit_job_to_watched_fileswas publishing an unusable filename to thewatched-filesqueue instead of a real file path.This adds
find_file_in_archive()infia_api/core/utility.py, which resolves an instrument + filename to a real path:IMAT_DIR(default/imat).ARCHIVE_DIR/NDX{INSTRUMENT}/Instrument/data(defaultARCHIVE_DIR=/archive), using the run'srun_startto guess the likely cycle folder (fast path) before falling back to a recursive search of the instrument's data directory.resubmit_job_to_watched_filesnow resolves and publishes the full path, raisingJobRequestError(400) if the file cannot be located anywhere.Draft PR for review before finalising.