Persist test metrics, log CPU usage, and make run identity window-aware

[Rian354]

Four small fixes found while running the Tranche 1 sweep. None of them change what a model computes — no numerics move.

1. Test evaluation was gated on wandb

if wandb_logger.enabled and test_loader is not None:
    test_scores = trainer.evaluate(test_loader)

A run without --wandb never computed test metrics at all — not merely unlogged, never calculated. Now ungated; the logger only decides whether they get shipped.

2. Test metrics are written to disk

metrics_history.json holds validation only and log.txt has no test lines, so the numbers that go in a paper lived nowhere on disk — only in stdout and W&B. If a log scrolled away, the only route back was re-scoring predictions_*.csv by hand, which is exactly what I ended up doing to build the results table.

Now test_metrics.json lands next to the predictions, with eval_split recorded alongside so it is unambiguous which split produced them.

3. Per-epoch CPU accounting

The trainer already recorded train_vram_allocated_mb, train_vram_peak_mb, and epoch_time_s. Added train_cpu_seconds and train_cpu_util_pct next to them.

Counts dataloader workers, not just the parent: workers are separate processes, so self-only CPU time badly understates a run whose cost is data loading — which is the interesting case for this toolkit. psutil is already present via wandb; there is a resource fallback, which reads low while persistent workers are still alive because it only sees reaped children.

Utilisation above 100% is expected and means several cores are busy.

4. Run identity now includes the observation window

exp_name was {task}_{model}_seed{seed}. An observation-window arm is a different experiment from the full-stay run at the same task/model/seed, but both resolved to the same name — so they shared an output directory and collided in W&B under one run name. Two runs in our project were genuinely indistinguishable except by tag.

Now suffixed _w48 when a window is set. Runs also set W&B group (arm) and job_type (backbone), so a many-cell sweep is navigable instead of one flat list — 37 runs with group=None is not a project you can read.

Also: create_directory race

if not os.path.exists(directory):
    os.makedirs(directory)

Two processes importing pyhealth for the first time both pass the check and one dies with FileExistsError. Hit this on a shared cluster home when two jobs started together and raced on ~/.cache/pyhealth/medcode/. Now exist_ok=True.

Verification

Dev split, two cells on the same tree:

	A (no --wandb)	B (--observation-window-hours 48)
output dir	labs_mlp_seed1	labs_mlp_seed1_w48
test_metrics.json	present	present
train_cpu_seconds / train_cpu_util_pct	present	present

Cell A is the one that matters: before this change it would have produced no test metrics at all.