From dc8f6b03edb48124cfd12a5d791b18a2d2e160a1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 8 Jul 2026 20:13:28 +0900 Subject: [PATCH 1/3] test: pin FIPC linking formulas and audit performance refactors Formula-integrity audit of the fixed item parameter calibration (FIPC) linking contract implemented by autoFIPC(). The estimation mathematics lives in mirt; this package only orchestrates the Kim (2006) contract of fixing old-form anchor parameters while non-anchor new-form parameters are estimated on the base scale. Reviewed every merged performance refactor that touches the linking path (#48/#52 fscores hoist, #51 grep->match anchor lookup, #56 category-count guard, #99 vectorized IPD anchor extraction) and confirmed each is mathematically equivalent - no term, margin, or constant changed. No formula damage found; nothing to restore. - Add tests/testthat/test-optimization-equivalence.R pinning the two formula-bearing expressions (#56 category count, #99 unlist extraction) to independent hand-computed reference values. - Strengthen test-fixed-parameter-calibration.R to assert non-anchor new-form items stay free (est == TRUE), pinning the second half of the Kim (2006) invariant. - Document the canonical linking equation, the refactor audit, and source DOIs (docs/fixed-parameter-item-calibration.md, docs/papers/README.md). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P --- docs/fixed-parameter-item-calibration.md | 46 +++++++++++ docs/papers/README.md | 46 +++++++++++ .../test-fixed-parameter-calibration.R | 10 +++ .../testthat/test-optimization-equivalence.R | 80 +++++++++++++++++++ 4 files changed, 182 insertions(+) create mode 100644 docs/papers/README.md create mode 100644 tests/testthat/test-optimization-equivalence.R diff --git a/docs/fixed-parameter-item-calibration.md b/docs/fixed-parameter-item-calibration.md index a511f33..ccdcbbe 100644 --- a/docs/fixed-parameter-item-calibration.md +++ b/docs/fixed-parameter-item-calibration.md @@ -20,6 +20,52 @@ contract with generated 2PL data: 5. Assert that the old-form estimates recover the generating common-item parameters closely enough for a small deterministic regression test. +## Canonical linking contract + +For each declared anchor pair `(x_j, y_j)` where `x_j` is a new-form item and +`y_j` its old-form counterpart, fixed item parameter calibration sets the +new-form item parameter vector to the old-form estimate and holds it fixed +during the linked calibration: + +```text +theta_hat(x_j) := theta_hat(y_j) and est(x_j) := FALSE +``` + +while every non-anchor new-form parameter stays free and is estimated on the +base scale defined by the fixed anchors (Kim, 2006). An anchor pair is only +eligible when both items share the same number of scored response categories, +i.e. `n_cat(x_j) == n_cat(y_j)` where `n_cat` counts distinct non-missing +responses. + +## Formula-integrity audit of performance refactors + +The estimation mathematics (item-response probabilities, the MML-EM cycles, +`fscores`, `expected.test`, and the DIF/IPD statistics) live in `mirt`; this +package only orchestrates the linking contract above. The following merged +performance refactors were reviewed against that contract and confirmed +**mathematically equivalent** (no term, margin, or constant changed): + +- **#48 / #52** (`82fa77d`, `762b8a9`): hoist `fscores(..., method = 'MAP')` + into a variable reused by `expected.test` instead of recomputing it. MAP + scoring is deterministic for a fixed fitted model, so the reused value is + identical to the second call. +- **#51** (`07e94d2`): replace anchored-regex `grep('^name$', cols)` anchor + lookups with `match(name, cols)`. For unique column names both return the + single matching index; `match` is exact-string and avoids regex + metacharacter hazards. The linked parameter values are unchanged. +- **#56** (`fc8bbfb`): category-count guard rewritten from + `length(levels(as.factor(x)))` to `length(na.omit(unique(x)))`. Both count + distinct non-missing response categories; factor levels already exclude `NA`. +- **#99** (`d73adbd`): vectorize IPD anchor extraction from a per-column loop + to `as.character(unlist(IPDItemList[row, cols]))`. Row 1 (old form), row 2 + (new form), and the screened column order are all preserved exactly. + +Regression guards for the two formula-bearing expressions (#56 and #99) are +pinned to hand-computed reference values in +`tests/testthat/test-optimization-equivalence.R`. The end-to-end linking +contract (anchors fixed to old-form values, non-anchors left free) is pinned in +`tests/testthat/test-fixed-parameter-calibration.R`. + ## References - Kim, S. (2006). A comparative study of IRT fixed parameter calibration diff --git a/docs/papers/README.md b/docs/papers/README.md new file mode 100644 index 0000000..75b6272 --- /dev/null +++ b/docs/papers/README.md @@ -0,0 +1,46 @@ +# Source papers for the calibration and linking mathematics + +`autoFIPC()` implements the fixed item parameter calibration (FIPC) linking +contract; the underlying IRT estimation is delegated to `mirt`. The canonical +equations this package must match are documented in the sources below. + +No PDFs are committed here: the two primary FIPC references (Kim, 2006; Kim & +Kolen, 2010) are published in the *Journal of Educational Measurement* and are +not open access, so they are cited by DOI rather than redistributed, to respect +copyright. The `mirt` reference is open access. + +## Primary source — the FIPC linking contract + +- **Kim, S. (2006). A comparative study of IRT fixed parameter calibration + methods.** *Journal of Educational Measurement, 43*(4), 355-381. + DOI: + - Canonical rule implemented: old-form (anchor) item parameters are treated as + known and held fixed while the new form is calibrated directly onto the + established base scale. + +- **Kim, S., & Kolen, M. J. (2010). Linking item parameters to a base scale.** + *Journal of Educational Measurement, 47*(2), 164-181. + DOI: + - Basis for treating the old-form scale as the fixed base onto which new-form + parameters are placed. + +## Estimation engine (open access) + +- **Chalmers, R. P. (2012). mirt: A Multidimensional Item Response Theory + Package for the R Environment.** *Journal of Statistical Software, 48*(6), + 1-29. DOI: (open access) + - Provides the MML-EM estimation, `fscores` (MAP), `expected.test`, and the + `multipleGroup`/`DIF` machinery used for item parameter drift screening. + - `?mirt::fixedCalib` documents fixed-item calibration methods based on + Kim (2006). + +## Supporting reference + +- **Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of + item parameters: Application of an EM algorithm.** *Psychometrika, 46*(4), + 443-459. DOI: + - The MML-EM algorithm underlying `mirt`'s `method = "EM"` estimation. + +See `../fixed-parameter-item-calibration.md` for the equation restatement, the +formula-integrity audit of the performance refactors, and the tests that pin +these formulas to reference values. diff --git a/tests/testthat/test-fixed-parameter-calibration.R b/tests/testthat/test-fixed-parameter-calibration.R index 03923f5..7cdbe5f 100644 --- a/tests/testthat/test-fixed-parameter-calibration.R +++ b/tests/testthat/test-fixed-parameter-calibration.R @@ -78,6 +78,16 @@ test_that("autoFIPC fixes common-item parameters on the old-form scale", { expect_false(any(linked_fixed$est)) } + # Kim (2006) invariant, second half: parameters of the NON-common new-form + # items must stay free (estimated) so they move onto the fixed base scale, + # rather than being frozen along with the anchors. + new_unique_items <- setdiff(new_item_names, new_common_items) + free_new <- linked_values[ + linked_values$item %in% new_unique_items & linked_values$name %in% c("a1", "d"), + "est" + ] + expect_true(all(free_new)) + old_estimates <- old_values[ old_values$item %in% old_common_items & old_values$name %in% c("a1", "d"), "value" diff --git a/tests/testthat/test-optimization-equivalence.R b/tests/testthat/test-optimization-equivalence.R new file mode 100644 index 0000000..02ce2f7 --- /dev/null +++ b/tests/testthat/test-optimization-equivalence.R @@ -0,0 +1,80 @@ +# Formula-integrity regression guards for performance refactors. +# +# These tests pin the two formula-bearing expressions that recent "Bolt" +# performance refactors rewrote, so any future re-optimization that silently +# changes their meaning is caught. Values below are hand-computed references, +# not a re-encoding of the current implementation. +# +# Audited refactors: +# * #56 (fc8bbfb): response-category count guard rewritten from +# length(levels(as.factor(x))) -> length(na.omit(unique(x))) +# Both count DISTINCT NON-MISSING response categories. This guard decides +# whether an old/new common-item pair may be linked (Kim, 2006: an anchor +# item must share the same response structure on both forms). +# * #99 (d73adbd): IPD common-item extraction rewritten from a per-column +# for-loop over IPDItemList[cols][row, i] +# to a vectorized +# as.character(unlist(IPDItemList[row, cols])). +# Row 1 = old-form anchor names, row 2 = new-form anchor names, restricted +# to the columns that survived IPD screening (CommonItemList_NOIPD). + +test_that("category-count guard counts distinct non-missing categories (#56)", { + vecs <- list( + dichotomous = c(0, 1, 0, 1, 1, 0), + trichotomous_w_na = c(0, 1, 2, NA, 2, 1, 0), + constant = c(0, 0, 0, 0), + four_category_w_na = c(0, 1, 2, 3, 3, NA, 1) + ) + + # Independent hand-computed reference (distinct non-missing categories). + expected <- c( + dichotomous = 2L, + trichotomous_w_na = 3L, + constant = 1L, + four_category_w_na = 4L + ) + + new_idiom <- vapply( + vecs, + function(x) length(na.omit(unique(x))), + integer(1) + ) + legacy_idiom <- vapply( + vecs, + function(x) length(levels(as.factor(x))), + integer(1) + ) + + expect_equal(new_idiom, expected) + # The refactor must remain equivalent to the pre-#56 expression. + expect_equal(unname(new_idiom), unname(legacy_idiom)) +}) + +test_that("IPD anchor extraction keeps old/new rows and screened columns (#99)", { + old_anchor_names <- c("old_1", "old_2", "old_3") + new_anchor_names <- c("new_1", "new_2", "new_3") + + # Mirror how autoFIPC() builds IPDItemList and names its columns. + IPDItemList <- data.frame(rbind(old_anchor_names, new_anchor_names)) + colnames(IPDItemList) <- paste0("X", seq_along(old_anchor_names)) + + # Item X2 is flagged as showing drift and dropped from the anchor set. + CommonItemList_NOIPD <- c("X1", "X3") + + actual_old <- as.character(unlist(IPDItemList[1, CommonItemList_NOIPD])) + actual_new <- as.character(unlist(IPDItemList[2, CommonItemList_NOIPD])) + + # Independent hand-computed reference. + expect_equal(actual_old, c("old_1", "old_3")) + expect_equal(actual_new, c("new_1", "new_3")) + + # The vectorized refactor must match the pre-#99 element-wise loop. + legacy_old <- character(length(CommonItemList_NOIPD)) + legacy_new <- character(length(CommonItemList_NOIPD)) + for (i in seq_along(CommonItemList_NOIPD)) { + legacy_old[i] <- as.character(IPDItemList[CommonItemList_NOIPD][1, i]) + legacy_new[i] <- as.character(IPDItemList[CommonItemList_NOIPD][2, i]) + } + expect_identical(actual_old, legacy_old) + expect_identical(actual_new, legacy_new) +}) From ab20ac96f994b0f5c68c6b1368e0448927d23cbd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 9 Jul 2026 15:42:01 +0900 Subject: [PATCH 2/3] docs: use item-parameter notation for FIPC contract --- docs/fixed-parameter-item-calibration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/fixed-parameter-item-calibration.md b/docs/fixed-parameter-item-calibration.md index ccdcbbe..66e1b1b 100644 --- a/docs/fixed-parameter-item-calibration.md +++ b/docs/fixed-parameter-item-calibration.md @@ -28,7 +28,7 @@ new-form item parameter vector to the old-form estimate and holds it fixed during the linked calibration: ```text -theta_hat(x_j) := theta_hat(y_j) and est(x_j) := FALSE +item_parms(x_j) := item_parms(y_j) and est(x_j) := FALSE ``` while every non-anchor new-form parameter stays free and is estimated on the From c926033f817b3d07796cdb300398090dee63c756 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 9 Jul 2026 21:13:10 +0900 Subject: [PATCH 3/3] test: cover missing and boundary response rows --- docs/fixed-parameter-item-calibration.md | 2 ++ tests/testthat/test-fixed-parameter-calibration.R | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/docs/fixed-parameter-item-calibration.md b/docs/fixed-parameter-item-calibration.md index 66e1b1b..7cd1726 100644 --- a/docs/fixed-parameter-item-calibration.md +++ b/docs/fixed-parameter-item-calibration.md @@ -13,6 +13,8 @@ scale. The package test `test-fixed-parameter-calibration.R` reproduces this contract with generated 2PL data: 1. Generate old-form and new-form responses from known true item parameters. + The generated forms include all-zero and all-one response rows plus missing + responses on a common item. 2. Fit separate old-form and new-form `mirt` models. 3. Run `autoFIPC()` with the shared items declared as common items. 4. Assert that linked common-item `a1` and `d` values equal the old-form values diff --git a/tests/testthat/test-fixed-parameter-calibration.R b/tests/testthat/test-fixed-parameter-calibration.R index 7cdbe5f..78deff3 100644 --- a/tests/testthat/test-fixed-parameter-calibration.R +++ b/tests/testthat/test-fixed-parameter-calibration.R @@ -25,6 +25,12 @@ test_that("autoFIPC fixes common-item parameters on the old-form scale", { )) names(old_data) <- old_item_names names(new_data) <- new_item_names + old_data[1, ] <- 0 + old_data[2, ] <- 1 + new_data[1, ] <- 0 + new_data[2, ] <- 1 + old_data[3, old_common_items[1]] <- NA + new_data[3, new_common_items[1]] <- NA old_model <- mirt::mirt( old_data,