ClusterLabel audit + tests - #1221
Open
lehendo wants to merge 2 commits into
Open
Conversation
…fied K-means design choice, add coverage tests ClusterLabel had no paper citation at all, despite being a legitimate, correctly-implemented instance of Mondrian conformal prediction (Vovk, Lindsay, Nouretdinov, and Gammerman, 'Mondrian confidence machine,' 2003) using K-means clusters as the category function. Added that citation plus Vovk/Gammerman/Shafer 2005, and documented the actual guarantee this class provides: per-cluster coverage (P(Y not in C(X) | cluster=c) <= alpha for every cluster), which is strictly stronger than plain marginal coverage -- the previous docstring didn't call this out. Investigated a theoretical concern by analogy to the NeighborhoodLabel self-inclusion bug fixed earlier this session: calibrate() fits KMeans on train+cal embeddings combined, so calibration points influence the very cluster centroids used to assign their own threshold, unlike a strict split-conformal setup where the category function would be fit on data disjoint from calibration. Verified via Monte Carlo simulation (including a calibration-set-dominated stress test, 10 train vs 300 cal points) that this does NOT introduce measurable coverage bias, unlike k-NN's self- inclusion (a hard, always-occurring artifact) -- a single point's leverage on a K-means centroid, an average over many points, is negligible. Documented this as a deliberate, verified design choice rather than leaving it unexamined, and added regression tests locking in both the per-cluster coverage guarantee and the train+cal-vs-train-only equivalence finding.
Contributor
|
The documentation promises coverage for each cluster separately, but the clusters are built using the same calibration patients that are later used to set each cluster's threshold. That means the calibration data helps define the groups and choose their thresholds, so the usual finite-sample guarantee does not apply. Fit K-means on training embeddings only, then assign calibration patients with |
calibrate() fit K-means on concatenate([train_embeddings, cal_embeddings]) and read calibration points' cluster assignments off kmeans_model.labels_ -- i.e. in-sample. Each calibration point's own presence in the fit could shift the cluster boundary used to assign its own threshold, breaking the Mondrian conformal guarantee's requirement (Vovk, Lindsay, Nouretdinov, and Gammerman 2003) that the category function be independent of the calibration data it's evaluated against. A prior commit on this branch (b1964a3) investigated this exact concern via Monte Carlo simulation and found no measurable coverage bias, given K-means centroids are a smooth average over many points (unlike NeighborhoodLabel's k-NN self-match, which is an exact, always-occurring distance-0 hit). That's a reasonable empirical argument, but the textbook-correct fix -- fit K-means on train_embeddings only, assign calibration/test points via out-of-sample .predict() -- is simpler code (no concatenation bookkeeping) and needs no ongoing empirical justification, so there's no reason to keep the weaker guarantee. Replaced the "combined vs. train-only fit" Monte Carlo comparison test (whose premise no longer applies, since only train-only fitting remains) with a regression test spying on KMeans.fit/.predict to directly verify calibrate() fits on train_embeddings only and assigns calibration points out-of-sample.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title is self explanatory.