ML4H Submission Final: Code + Experiment Run Results - #52
Open
will-pang wants to merge 65 commits into
Open
Conversation
* rename arg name for chefer * Initial attempts to fix the interpretability target_class_idx * Support negative prediction for interpretability metric. * Fix tests * Fix more tests * Revert "Support negative prediction for interpretability metric." This reverts commit fe8c8ad. * Reapply "Support all samples for interpretability metric" * Initial attempt for the filter * Fixup * Fix sample_class handling * fixup * fix test * Fix arg name * Add example * fix docs
…sunlabuiuc#927) * small fix + bump to pyproject.toml ver. for bug fixed release on pypi * We don't really have someone qualified for a second review, and this broken CI is leading to a lot of issues here. Will revert if it doesn't resolve here.
Add test_rnn.py with 12 test cases covering: TestRNN (8 tests): - Model initialization with correct attributes - Forward pass output structure and shapes - Backward pass gradient propagation - Embedding extraction via embed=True - Custom hyperparameters (embedding_dim, hidden_dim) - LSTM cell type variant - Vanilla RNN cell type variant - Bidirectional RNN variant TestMultimodalRNN (4 tests): - Initialization with correct sequential/non-sequential classification - Forward pass with mixed modalities (sequence + multi_hot + tensor) - Backward pass gradient propagation - Embedding extraction with correct mixed-modality dimensions Follows the established test pattern from test_mlp.py and test_tcn.py using create_sample_dataset with synthetic data. Ref sunlabuiuc#425
* Fixed repo to be able to run TUEV/TUAB + updated example scripts * Args need to be passed correctly * Minor fixes and precomputed STFT logic * Fix the test files to reflect codebase changes * Args update * test script fixes * dataset path update * fix contrawr - small change * divide by 0 error * Incorporate tfm logic * Fix label stuff * tuab fixes * fix metrics * aggregate alphas * Fix splitting and add tfm weights * fix tfm+tuab * updates scripts and haoyu splitter * fix conflict * Remove weightfiles from tracking and add to .gitignore Weight files are large binaries distributed separately; untrack all existing .pth files under weightfiles/ and add weightfiles/ to .gitignore so they are excluded from future commits and the PR. Made-with: Cursor * normalization = 95% * temporarily re-add weight files * 16 workers * tuab sanity check * consistent log outputs * test tuab * change back to multiclass * update conformal scripts * remove weightfiles * oops * fix tests
* feat: migrate GRASP model from PyHealth 1.0 to 2.0 API Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * feat: add GRASP mortality prediction notebook and fix cluster_num Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * Restore code_mapping support in SequenceProcessor for PyHealth 2.0 Adds optional code_mapping parameter to SequenceProcessor that maps granular medical codes to grouped vocabularies (e.g. ICD9CM→CCSCM) before building the embedding table. Resolves the functional gap from the 1.x→2.0 rewrite where code_mapping was removed. Ref sunlabuiuc#535 Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> * Add RNN baseline and code_mapping comparison notebooks for MIMIC-III Two identical notebooks for A/B testing code_mapping impact on mortality prediction. Only difference is the schema override in Step 2. Both use seed=42 for reproducible splits. Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> * fix(tasks): extract NDC codes instead of drug names for prescription mapping event.drug returns drug names (e.g. "Aspirin") which produce zero matches in CrossMap NDC→ATC; event.ndc returns actual NDC codes enabling 3/3 feature mapping for mortality and readmission tasks. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * test(tasks): add tests verifying NDC extraction in drug tasks Checks that mortality and readmission task processors build vocabulary from NDC codes (numeric strings) rather than drug names (e.g. "Aspirin"), confirming the event.drug -> event.ndc fix works correctly. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * fix(tasks): fix missed MortalityPredictionMIMIC4 event.drug and update docs - Fix event.drug -> event.ndc in MortalityPredictionMIMIC4 (line 282) - Update readmission task docstrings to reflect NDC extraction Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * fix(tasks): fix DrugRecommendationMIMIC3 to extract NDC codes DrugRecommendationMIMIC3 used prescriptions/drug (drug names) via Polars column select; changed to prescriptions/ndc to match MIMIC-4 variant and enable NDC->ATC code mapping. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * fix(models): guard RNNLayer and ConCare against zero-length sequences RNNLayer: clamp sequence lengths to min 1 so pack_padded_sequence does not crash on all-zero masks, matching TCNLayer (tcn.py:186). ConCare: guard covariance divisor with max(n-1, 1) to prevent ZeroDivisionError when attention produces single-element features. Both edge cases are triggered when code_mapping collapses vocabularies and some patients have all codes map to <unk>, producing all-zero embeddings and all-zero masks. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * docs: add docstrings to SequenceProcessor class and fit method Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * docs: add docstrings, type hints, and fix test dims for GRASP module Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * feat: add GRASP mortality prediction notebooks for baseline and code_mapping Baseline notebook runs GRASP with raw ICD-9/NDC codes. Code_mapping notebook collapses vocab via ICD9CM→CCSCM, ICD9PROC→CCSPROC, NDC→ATC for trainable embeddings on full MIMIC-III. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * fix(models): guard ConCare and GRASP against batch_size=1 crashes - ConCare FinalAttentionQKV: bare .squeeze() removed batch dim when batch_size=1, causing IndexError in softmax. Use .squeeze(-1) and .squeeze(1) to target only the intended dimensions. - ConCare cov(): division by zero when x.size(1)==1. Guard with max(). - GRASP grasp_encoder: remove stale torch.squeeze(hidden_t, 0) that collapsed [1, hidden] to [hidden] with batch_size=1. Both RNNLayer and ConCareLayer already return [batch, hidden]. - GRASP random_init: clamp num_centers to num_points to prevent ValueError when cluster_num > batch_size. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * feat: add GRASP mortality prediction notebooks for baseline and code_mapping Baseline notebook runs GRASP with raw ICD-9/NDC codes. Code_mapping notebook collapses vocab via ICD9CM→CCSCM, ICD9PROC→CCSPROC, NDC→ATC for trainable embeddings on full MIMIC-III. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * Add code_mapping as task __init__ argument Allow tasks to accept a code_mapping dict that upgrades input_schema entries so SequenceProcessor maps raw codes (e.g. ICD9CM) to grouped vocabularies (e.g. CCSCM) at fit/process time. This avoids manual schema manipulation after task construction. - Add code_mapping parameter to BaseTask.__init__() - Thread **kwargs + super().__init__() through all task subclasses with existing __init__ methods (4 readmission tasks, 1 multimodal mortality task) - Add 17 tests covering SequenceProcessor mapping and task-level code_mapping initialization Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * Update code_mapping notebook to use task init argument Replace manual task.input_schema override with the new code_mapping parameter on MortalityPredictionMIMIC3(). Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * feat(examples): add ConCare hyperparameter grid sweep script Mirrors the GRASP+ConCare mortality notebook pipeline exactly (same tables, split, seed, metrics) but sweeps 72 configurations of embedding_dim, hidden_dim, cluster_num, lr, and weight_decay. Results are logged to sweep_results.csv. Supports --root for pointing at local MIMIC-III, --code-mapping, --dev, and --monitor. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * chore(sweep): increase early stopping patience from 10 to 15 epochs Smaller ConCare configs (embedding_dim=8/16) may learn slower and need more epochs before plateauing. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * Initial plan * fix: filter falsy NDCs, guard None tokens in process(), fix NDC regex Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-authored-by: ddhangdd <43976109+ddhangdd@users.noreply.github.com> * refactor(sweep): rename and generalize sweep script for all backbones Rename sweep_concare_grasp.py → sweep_grasp.py. Now supports --block GRU|ConCare|LSTM with per-backbone default grids, --resume for crash recovery, --grid JSON override, auto-dated output dirs (sweep/{BLOCK}_{YYYYMMDD}_{HHMMSS}_{mapping}/), and config.json saved alongside results for reproducibility. Co-Authored-By: Colton Loew <colton.loew@gmail.com> Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com> Co-Authored-By: christiana-beard <christyanamarie116@gmail.com> Co-Authored-By: ddhangdd <dfung2@wisc.edu> * test(sweep): add unit and integration tests for sweep_grasp utilities Covers grid building, combo hashing, CSV resume parsing, output directory naming, and end-to-end single-config runs for GRU and ConCare on synthetic data (13 tests, all passing). Co-Authored-By: Colton Loew <loewcx@illinois.edu> Co-Authored-By: lookman-olowo <lookman-olowo@github.com> Co-Authored-By: christiana-beard <christiana-beard@github.com> Co-Authored-By: ddhangdd <ddhangdd@github.com> * docs(sweep): add tmux copy-paste instructions for each paper run Co-Authored-By: Colton Loew <loewcx@illinois.edu> Co-Authored-By: lookman-olowo <lookman-olowo@github.com> Co-Authored-By: christiana-beard <christiana-beard@github.com> Co-Authored-By: ddhangdd <ddhangdd@github.com> * chore(examples): adds cleans examples, removes util script * Delete tests/core/test_grasp.py we removed grasp script from examples, dropped test * Revert "Delete tests/core/test_grasp.py" This reverts commit 0d95758. * fix: remove orphaned sweep test, restore grasp tests * feat(grasp): add static_key support for demographic features with tests * fix(test): add valid NDC to test prescriptions so readmit test produces both labels --------- Co-authored-by: lookman-olowo <lookmanolowo@hotmail.com> Co-authored-by: christiana-beard <christyanamarie116@gmail.com> Co-authored-by: ddhangdd <dfung2@wisc.edu> Co-authored-by: Lookman Olowo <42081779+lookman-olowo@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ddhangdd <43976109+ddhangdd@users.noreply.github.com> Co-authored-by: ddhangdd <desmondfung123@gmail.com> Co-authored-by: Colton Loew <loewcx@illinois.edu> Co-authored-by: lookman-olowo <lookman-olowo@github.com> Co-authored-by: christiana-beard <christiana-beard@github.com> Co-authored-by: ddhangdd <ddhangdd@github.com> Co-authored-by: lookman-olowo <lookman-olowo@users.noreply.github.com>
* dl4h final project kobeguo2 - CaliForest * Update CaliForest to require explicit fit before inference * Remove unused logit_scale from CaliForest
* Fix Drug Recommandation NDC/ATC3 code * Fix padding behaviour * remove .codex file * Change test from FakePatient to demo dataset
…IUC purge (sunlabuiuc#1143) literally just updating the examples/ no need to waste reviewer time.
* add back backups of original tutorials * Backup lost tutorials * generate new tokenizer tutorial * update with pip install d4rl install pyhealth and rename * update colab references
* Add synthetic-EHR generative evaluation metrics Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic EHR data along privacy, utility, and statistical-fidelity axes: - privacy.py: NNAAR, membership inference attack, discriminator privacy - utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence similarity (R2, Pearson, RMSE) - utils.py: shared data prep, an LSTM classifier, and a random-forest baseline - evaluate_synthetic_ehr(): convenience orchestrator for the full suite These functions are ported from a standalone evaluation script. The MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any flat EHR dataframe. Public functions are re-exported from pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py and Sphinx docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add synthetic-EHR generative evaluation metrics Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic EHR data along privacy, utility, and statistical-fidelity axes: - privacy.py: NNAAR, membership inference attack, discriminator privacy - utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence similarity (R2, Pearson, RMSE) - utils.py: shared data prep, an LSTM classifier, and a random-forest baseline - evaluate_synthetic_ehr(): convenience orchestrator for the full suite These functions are ported from a standalone evaluation script. The MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any flat EHR dataframe. Public functions are re-exported from pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py and Sphinx docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * add baselines * removed halo save file and updated promptehr to be more paper accurate * update docs * update docs * Update pyhealth.models.HALO.rst --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* transfer FHIR pipeline to branch * fix * fix unit test using fast json readers * Replace editdistance with rapidfuzz for Python 3.13 compatibility editdistance 0.8.1 only ships cp311 wheels and has no Python 3.13 binary, causing CI installs to fail on Linux. rapidfuzz>=3.0.0 ships wheels for all major platforms including cp313 and provides an equivalent Levenshtein.distance() API. https://claude.ai/code/session_01L5qHpvAZQSgmZyc6tMTX6d * copilot fixes * revert ignore error change --------- Co-authored-by: Claude <noreply@anthropic.com>
sunlabuiuc#1003) * Add MedFuse multimodal model for EHR+CXR fusion * Address PR feedback: docstring, mask doc, test cleanup, paper citations
…unlabuiuc#1158) * Attention Rollout skeleton * attention_rollout.py done * tests/core/test_attention_rollout.py done * attention rollout integrated into example scripts * attention rollout docs * attention rollout docs/interpret/pyhealth.interpret.methods.attention_rollout.rst added * Stip trailing whitespace and rename example keys to rollout * attention rollout: doc and style changes * attention rollout: module docstring header
…nlabuiuc#1176) Any PR touching pyhealth/**/*.py must also update docs/ and examples/, keep added/modified lines free of ruff lint violations, and give new or modified top-level public classes/functions a '>>>' docstring example. Co-authored-by: Claude <noreply@anthropic.com>
* feat: add EEGBCI helper functions * feat: add EEGBCI dataset * feat: add EEGBCI tasks * test: add opt-in EEGBCI real-data smoke test * docs: add EEGBCI pattern discovery example * docs: add EEGBCI API docs * chore: record EEGBCI verification * docs: refine EEGBCI moment report design * Add EEGBCI moment report constants * Add EEGBCI rest baseline helpers * Add EEGBCI state scoring helpers * Add EEGBCI task state quality helpers * Add EEGBCI moment row annotation * Add EEGBCI representative windows * Render EEGBCI moment summary * Wire EEGBCI moment report main flow * Document EEGBCI moment report outputs * Fix EEGBCI moment report review findings * Polish EEGBCI report artifact * Exclude EEG pattern discovery notes * fix: address EEGBCI review feedback * fix: address EEGBCI review feedback * fix: satisfy PR contribution rules
* Covariate CP fixes * small edits to pass checks
* Add missing citations for TFMTokenizer, EHRMambaCEHR, CEHR embeddings, comprehensiveness/sufficiency metrics, MLE, and NNAAR * to pass the pr checks
…labuiuc#1179) * feat(datasets): add Parquet scan path to BaseDataset Route .parquet/.pq files, globs, and directories through a typed _scan_parquet scanner; keep CSV/TSV(.gz) on the existing path. Add a datetime fast-path in load_table that skips the string round-trip and casts to datetime64[ms], preserving NaT for static events. * feat(datasets): add MEDSDataset for the Medical Event Data Standard Declarative YAML wrapper over the shared Parquet scan path, with split_source subset selection (metadata or directory layout), distinct processing caches per subset, and a construction-time Parquet footer schema guard that rejects missing, non-timestamp, or timezone-aware time columns. * test(datasets): add MEDS synthetic and demo smoke tests Deterministic sharded Parquet fixtures cover nested splits, subset filtering, cache isolation, set_task smoke, and construction-time schema-guard TypeErrors. Demo smoke stays skip-gated behind MEDS_DEMO_ROOT / test-resources/meds_demo (gitignored). * docs(examples): add MEDS example and API docs Document MEDSDataset in the API reference and add an end-to-end examples/meds_demo.py against the public PhysioNet MIMIC-IV MEDS demo. * feat(tasks): add InHospitalMortalityMEDS MEDS-native in-hospital mortality task: one sample per completed stay, reconstructed by joining HOSPITAL_ADMISSION/HOSPITAL_DISCHARGE events on hadm_id. Half-open [admit, prediction_time) observation window (full_stay default; first_hours early-warning variant), label from the HOSPITAL_DISCHARGE//DIED discharge code. Discharge and MEDS_DEATH events are excluded from features to prevent label leakage. hadm_id is dataset-specific (not part of the core MEDS schema), so it is exposed via a bundled configs/meds_with_hadm.yaml rather than the default config. Verified on the public MIMIC-IV demo in MEDS: 12 positive / 238 stays (rate 0.0504); set_task sample count 238. - pyhealth/tasks/in_hospital_mortality_meds.py (+ __init__ export) - pyhealth/datasets/configs/meds_with_hadm.yaml - tests/core/test_in_hospital_mortality_meds.py - examples/verify_meds_mortality.py - docs/api/tasks/pyhealth.tasks.InHospitalMortalityMEDS.rst (+ tasks.rst toctree) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tasks): re-export InHospitalMortalityMEDS with explicit alias Satisfies ruff F401 on the newly added __init__ line under tools/check_pr_rules scoped lint (same pattern as eegbci). Co-authored-by: Cursor <cursoragent@cursor.com> * docs: link MEDS schema docs for subject_splits mapping Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: rename _reconstruct_stays to _group_stays and clarify summarize docstring Co-authored-by: Cursor <cursoragent@cursor.com> * docs: add end-to-end RNN training to MEDS demo Co-authored-by: Cursor <cursoragent@cursor.com> * docs: link subject_splits in MEDSDataset API page; qualify demo metrics output Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: drop summarize helper from InHospitalMortalityMEDS Co-authored-by: Cursor <cursoragent@cursor.com> * style: modernize typing annotations, drop unused noqa (UP006/UP035/UP045/RUF100) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…buiuc#1189) * Implement real APS and add dynamic score_type to conformal methods Adds pyhealth/calib/predictionset/scores.py, a shared score module implementing genuine Adaptive Prediction Sets (Romano, Sesia, and Candes 2020): nonconformity score = cumulative sum of predicted probabilities for classes ranked above the target, plus a randomized U*p(target) term (U ~ Uniform(0,1), one draw per example, shared across all candidate classes). Previously, BaseConformal's score_type="aps" was silently just an alias for "threshold" and did not implement APS at all. Threads a new score_type parameter ("threshold" [default, unchanged behavior] or "aps") through BaseConformal, LABEL, ClusterLabel, CovariateLabel, and NeighborhoodLabel, each with an optional random_state for reproducible APS randomization. SCRIB and FavMac are intentionally excluded since their calibration isn't a score-then- quantile pattern. Verified via numpy-only synthetic tests: both score types hit ~90% empirical coverage at alpha=0.1, for marginal and class-conditional coverage, in both nonconformity and conformity sign conventions. * Add scores.py doctests, aps usage examples, tests, and docs Adds >>> usage examples to the 4 public functions in pyhealth/calib/predictionset/scores.py (verified against real computed output). Adds a score_type="aps" usage example to the docstrings of BaseConformal, LABEL, ClusterLabel, CovariateLabel, and NeighborhoodLabel. Adds tests/core/test_scores.py covering both score types: threshold backward-compatibility, the APS formula's hand-computable non-randomized case, monotonicity, reproducibility under a seeded RNG, nonconformity/ conformity complementarity, and empirical marginal coverage at the target alpha. Extends test_cluster_label.py, test_covariate_label.py, and test_neighborhood_label.py with score_type="aps" end-to-end cases. Documents the score_type argument and adds the previously-missing BaseConformal entry to docs/api/calib/pyhealth.calib.predictionset.rst. * Fix ruff lint violations flagged by CI (UP/RUF rules) CI's ruff install (pip install 'ruff~=0.15') resolves to the latest 0.x release under PEP 440 compatible-release semantics, which enabled more pyupgrade/ruff-specific default rules than the older cached ruff used for local verification. Fixes all 13 flagged violations: Optional[X]/Union[X, Y] -> X | Y, typing.Dict -> dict (including the now-modernized pre-existing forward() return annotations this forced), an unused unpacked variable, and an unsorted __all__. Verified by reproducing the CI's exact environment: a clean venv with `pip install 'ruff~=0.15'` (which also resolves to 0.16.3), confirming `tools/check_pr_rules.py` now passes.
…/Multimodal-PyHealth/PyHealth into ml4h-merge-tranche-1-wp-20260825
…HR, BottleneckTransformer) with results. Claude-Session: https://claude.ai/code/session_01U3rt6DmCQ2NiPmfwujzoje
…erflow-path fix. Claude-Session: https://claude.ai/code/session_01U3rt6DmCQ2NiPmfwujzoje
…/Multimodal-PyHealth/PyHealth into ml4h-merge-tranche-1-wp-20260825
…hunked note rows.
Replace the trailing -1 in .view()/.reshape() calls for the text and image branches with the known embedding dim (or an explicit empty mask), since torch can't infer -1 for a 0-element tensor and raises "cannot reshape tensor of 0 elements ... dimension size -1 is ambiguous" when a batch has zero note or image slots.
…single per-sample clock. Previously, time offsets were computed inconsistently across modalities within a multi-admission sample: - ICD codes: hours since the previous admission (a delta, reset to 0 for the first admission). - Labs, notes, CXR: hours since that admission's own start (reset to 0 at every admission). Since labs, admission_note_times, and cxr_image_times concatenate events from every admission into one sequence per patient, the per-admission reset caused values from different admissions to collide — e.g., a lab drawn 6h into stay 2 sorted identically to one drawn 6h into stay 1, even though the two are actually days apart.
Five small changes on top of cdeb3e0, plus the paper launchers. 1. lab_standardizer.py + wiring. Per-feature z-score fit on the training split only; missing values stay missing. UnifiedMultimodalEmbeddingModel already accepted numeric_standardizers, so this is only the fit + the hand-off. --no-lab-standardization runs raw labs as an ablation, so the default is a choice you can turn off rather than a commitment. Measured on EHRMamba/labs+notes/seed 1: 0.7473 without, 0.8011 with. 2. write_run_config. metrics_history.json records what a run scored but not the conditions that produced it, and it records resolved values rather than raw flags -- that distinction is what surfaced a per-model optimizer override where run_config stored adam_eps: null while the optimizer used 1e-6. Also records source_sha256 so a table can be shown to come from one build. 3. eval_split. The inference loader fell back test-or-val-or-train, so a run without a test split reported TRAINING metrics as test with nothing saying so. The split is now named, warned about, and recorded. 4. exp_name includes the task. It was {model}_seed{seed}, so labs and notes_labs at one seed wrote to the same directory and the second run silently destroyed the first. This matters immediately: the plan is 36 paired cells. 5. Restore emitted_data_version. cdeb3e0 removed it. It is part of vars(task), which is what the task-cache uuid5 key is built from, so without it a cache built before an emitted-data change is silently reused -- and cdeb3e0 changes every event timestamp, which is exactly when the bump is needed. Deliberately not included: the time_origin fix. cdeb3e0 already does it, and by inspection it is identical to ours (same _hours_since helper, same anchor on admissions_to_process[0].timestamp). No need to revert it. scripts/paper: common.sh holds the protocol; will.sh and rian.sh add only the data roots and the CPU tuning for their machine. rian.sh pins OMP threads and uses loader workers because those nodes run several cells at once -- unpinned, four concurrent cells put ~800 threads on 128 cores and epoch time went 191s to 8600s with the GPUs at 0-1%. will.sh keeps num_workers=4 and no pinning. Verified on a dev split before and after: output dir goes mlp_seed1 -> notes_labs_mlp_seed1, run_config.json appears with eval_split=test, and the fitted mean/std/count buffers land in the checkpoint with --no-lab- standardization correctly removing them.
… onto a single per-sample clock." This reverts commit cdeb3e0.
…imic4.py onto a single per-sample clock."" This reverts commit 6ad4726.
will-pang
changed the title
…ough --loader-num-workers and --persistent-workers do not exist on this branch, so every rian cell died at argparse before doing any work. This runner has no dataloader-worker control at all: --num-workers feeds the dataset build only, and thread pinning is what actually keeps concurrent cells off each other. Also forward "$@" so callers can add flags (--wandb, --observation-window-hours) without editing the launcher.
Four small fixes found while running the Tranche 1 sweep. 1. Test evaluation was gated on wandb. `if wandb_logger.enabled and test_loader is not None` meant a run without --wandb never computed test metrics at all -- not merely unlogged, never calculated. Ungated. 2. Test metrics are now written to test_metrics.json. metrics_history.json carries validation only and log.txt has no test lines, so the numbers that go in a paper previously lived nowhere on disk: only in stdout and W&B, recoverable afterwards only by re-scoring predictions_*.csv by hand. 3. Per-epoch CPU accounting alongside the existing VRAM and epoch_time_s: train_cpu_seconds and train_cpu_util_pct. Counts dataloader workers, since self-only time badly understates a data-loading-bound run. psutil is already present via wandb, with a resource fallback. 4. exp_name and the W&B run name now include the observation window. An observation-window arm is a different experiment from the full-stay run at the same task/model/seed, but both resolved to the same name -- so they shared an output directory and collided in W&B. Runs also now set W&B group (arm) and job_type (backbone) so a many-cell sweep is navigable. Also: create_directory used `if not exists: makedirs`, which two processes importing pyhealth for the first time can both pass, leaving one to die on FileExistsError. Seen on a shared cluster home with two concurrent jobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Overview of Changes
Results
Seed 1
Labs (Rian)
Labs + Notes (Rian)
Labs + Notes + CXR (Rian)
Seed 2
Labs (Will)
Labs + Notes (Will)
Labs + Notes + CXR (Rian)
Seed 3
Labs (Rian)
Labs + Notes (Rian)
Labs + Notes + CXR (Rian)