WAS wealth imputation: student_loan_balance allocated by positional index, and QRF trained on raw negative sentinels (hbedrmr8 ~96% missing)

[vahid-ahmadi]

Two defects in the WAS round-8 wealth imputation, both found during the Microcosm migration's E5 port (microcosm#714) and verified against this repo's code at current main. Filing them together because they live in the same stage and would sensibly fix together.

1. student_loan_balance is allocated to the wrong households (positional index read as household ids)

datasets/imputations/wealth.py, at the predict site:

output_df = model.predict(input_df)
...
dataset.person[column] = _allocate_student_loan_balance_to_people(
    household_balances=output_df[column].clip(lower=0),
    ...
)

output_df comes straight from model.predict and carries a 0-based RangeIndex. But _allocate_student_loan_balance_to_people iterates household_balances.items() treating that index as person_household_id:

group_indices = person.groupby("person_household_id").indices
for household_id, household_balance in household_balances.items():
    if household_balance <= 0 or household_id not in group_indices:
        continue
    idx = np.asarray(group_indices[household_id], dtype=int)

group_indices is keyed by real (1-based, sernum-derived) household ids. So the household with id k receives the balance predicted for the household at position k — its positional neighbor — and predictions at positions that match no real id are silently dropped (the not in group_indices guard makes the misalignment invisible). Because FRS sernums are near-sequential, most lookups succeed, which is why nothing crashes and test_student_loan_balance.py passes: the tier cascade behaves plausibly on whichever balance arrives.

The fix is to key the balances by actual household id before allocation (e.g. set the index from the recipient frame's household_id alongside model.predict, or pass ids explicitly). Microcosm's port allocates by entity id and pins non-positional alignment with a regression test — happy to point at it as a reference implementation.

2. The QRF trains on raw UKDS negative sentinels; hbedrmr8 is −8 for ~96% of donor rows

generate_was_table applies a blanket fillna(0) but never recodes UKDS negative sentinel codes, so the QRF fits on them as if they were values. An audit of all 33 columns the model consumes (run for microcosm#714's adjudication, on the round-8 household EUL tab) found:

• hbedrmr8: 95.8% of rows carry −8 — the bedrooms question is effectively unasked in this tab, so the number-of-bedrooms predictor the model trains on is a near-constant corrupt column, not a bedroom count.
• vcarnr8: 2 rows of −8 surviving into the vehicles target.

(dvprirntr8's −9 is structural not-applicable and its mapping is already correct; legitimate negative domains — net wealth positions — are untouched by this finding.)

Microcosm's port recodes the two nonnegative-domain sentinel columns to zero before fitting and measured the effect on the imputed surface: most outputs move <2%, well inside the seed-realization noise band, with the largest move (student_loan_balance, +12%) going toward the administrative register. So the fix is low-risk on the output surface — the main gain is that the bedrooms predictor stops being noise shaped like a constant.

---

Context: this is the third production finding out of the migration's port-fidelity reviews, after the benunit sort scramble (#461) and the EDUCQUAL inversion (#459) — the pattern of verifying each ported derivation against the raw data keeps paying for itself, so these are offered in the same spirit. cc @juaristi22, whose E5 adjudication receipts carry the full measurements.