Repository navigation
Description
Describe the bug
get_confusion_matrix() (the backbone of ConfusionMatrixMetric) handles a valid ignore_index by zeroing the ignored channel in y_pred and y. Zeroing a channel does not remove those voxels from the other channels' counts, so a voxel of the ignored class that the model assigned to another class still counts as a false positive for that class.
DiceMetric and MeanIoU instead build a spatial mask with create_ignore_mask() (#9134 fixed compute_iou() for exactly this inconsistency), so ConfusionMatrixMetric disagrees with them for the same input and the same setting.
Two related facets of the same root cause:
- With a single-channel binary target and
ignore_index=0,include_background=Trueadditionally zeroes the only channel, hiding a valid false positive at a non-ignored location (specificity 1.0 instead of 0.0). - Boolean one-hot targets crash in
create_ignore_mask()(1.0 - bool_tensorraisesRuntimeError).
To Reproduce
import torch
from monai.metrics import get_confusion_matrix
# 4 voxels, 3 one-hot classes; voxel 1 belongs to the ignored class 1
y = torch.tensor([[[1., 0, 0, 1], [0, 1, 0, 0], [0, 0, 1, 0]]])
# a perfect prediction except the ignored voxel is called class 0
y_pred = torch.tensor([[[1., 1, 0, 1], [0, 0, 0, 0], [0, 0, 1, 0]]])
counts = get_confusion_matrix(y_pred, y, include_background=True, ignore_index=1)
counts[0, 0].tolist() # [2.0, 1.0, 1.0, 0.0] — TP, FP, TN, FN for class 0Single-channel case:
import torch
from monai.metrics import ConfusionMatrixMetric
y = torch.tensor([[[1., 1., 0.]]]) # single-channel binary target
metric = ConfusionMatrixMetric(metric_name="specificity", reduction="none", ignore_index=0)
metric(y_pred=torch.ones_like(y), y=y)
metric.aggregate() # specificity 1.0 — the false positive at the only valid voxel is hiddenExpected behavior
counts[0, 0].tolist() # [2.0, 0.0, 1.0, 0.0] — the ignored voxel is excluded from every class's countsClass-0 precision on the valid voxels is 1.0, matching DiceMetric/MeanIoU for the same input. In the single-channel case, specificity is 0.0: the false positive at the only valid voxel is retained regardless of include_background.
Environment
Validated on dev @ bbebca6f17cabf98ea5fb8ec5776eeacd10aa60d, Windows, Python 3.14.1, PyTorch 2.12.1+cpu, NumPy 2.3.5.
Additional context
PR #9195 fixes this by excluding the ignored voxels spatially (reusing create_ignore_mask() semantics), while preserving the existing channel exclusion for multi-channel targets, retaining valid single-channel errors regardless of include_background, and supporting boolean targets. Related to #9134, which fixed the same inconsistency for MeanIoU.
Activity
fix(metrics): exclude ignored voxels from confusion matrix scores (Pr…