Refactor DICOM tests and enhance hybrid mode functionality

[NicholasLeotta99]

This pull request enhances the DICOM preprocessing pipeline by improving how removed scans are tracked, checkpointed, and aggregated, as well as increasing robustness and memory efficiency. The main changes include more consistent handling of removed scan records, improved checkpointing for better resumability, and several bug fixes and code cleanups.

Improvements to removed scan tracking and aggregation:

• Changed the structure of removed records to always store lists of DataFrames, ensuring consistent tracking of all removed scans throughout filtering and splitting steps. This includes updating how removed scans are appended and aggregated in _filter_worker, _split_worker, and _aggregate_removed. [1] [2] [3] [4] [5]
• Updated the split worker to always return a DataFrame of removed scans if all scans in a session are discarded, ensuring that no lost scans are missed in the removal log.

Checkpointing and resumability enhancements:

• Added new helper functions to load checkpoint data for filter, split, and order steps, and updated checkpoint save/load logic to include removed scans and redirections. This allows for more robust resumption and memory management during long preprocessing runs. [1] [2] [3] [4] [5] [6]
• Modified batch processing in the main pipeline to reload existing checkpoint data before saving, ensuring that all results and removed scans are preserved and memory usage is capped. [1] [2]

Robustness and bug fixes:

• Improved handling of malformed relocation commands in _relocate_worker, skipping and warning about invalid entries instead of failing.
• Fixed issues where previously some removed scans could be dropped or overwritten, by always appending and aggregating lists of DataFrames instead of single DataFrames.

Interface and argument consistency:

• Updated CLI argument names in 02_parseDicom.py from --batch-size and --min-free-gb to --batch_size and --min_free_gb for consistency.
• Updated shell script 00_preprocess.sh to pass --save_dir and --load_table arguments to the parsing script if SAVE_DIR is set.

General code cleanup and documentation:

• Added and improved docstrings for worker functions, clarified return values, and cleaned up unused or redundant code paths. [1] [2]

These changes collectively make the preprocessing pipeline more reliable, easier to resume, and better at tracking all data modifications throughout the workflow.