feat(measurement): add RAT-Bench reidentification W&B metrics

[lipikaramaswamy]

Related Issue

No linked issue required: maintainer-owned benchmark measurement integration for RAT-Bench re-identification metrics.

Plan Document

No plan required: scoped measurement schema/W&B ingestion extension with regression coverage.

Summary

Adds RAT-Bench re-identification as a first-class measurement record for Anonymizer benchmark ingestion and W&B aggregation.

• Adds rat_bench_reidentification to measurement ingress validation.
• Adds the matching W&B table row model and outbound field policies.
• Registers scalar aggregation fields for row counts, re-identification rates, coverage, attacker metadata, thresholds, and summary scores.
• Treats reid_threshold as a last-value configuration scalar rather than an averaged metric.
• Bounds RAT-Bench percentage fields to [0, 100] at ingress and W&B table-model validation.
• Adds regression coverage for measurement ingress and W&B scalar aggregation.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring
• CI, release, or contributor workflow update

Contributor Checklist

• PR title follows Conventional Commits, for example fix: handle empty entity list
• Related issue is linked, or a maintainer-owned no-issue reason is documented above
• For non-trivial changes, a plan document is linked above, or the no-plan reason is documented above
• Public API impact checked; skills/anonymizer/SKILL.md updated if needed
• No real PII added to tests, docs, notebooks, fixtures, or artifacts
• No API keys, service tokens, private keys, credentials, or real endpoint secrets added

Validation

• Commands run:
  • uv run pytest tests/tools/test_measurement_ingress_completion.py::test_wandb_ingress_accepts_rat_bench_reidentification_summary tests/tools/test_measurement_ingress_completion.py::test_wandb_ingress_rejects_rat_bench_percentages_above_100 tests/tools/test_measurement_wandb_logging.py::test_wandb_aggregates_rat_bench_reidentification_record tests/tools/test_measurement_wandb_logging.py::test_wandb_scalar_registry_matches_package_field_catalog
• Skipped checks or known failures:
  • Full test suite not run locally; this PR is scoped to measurement ingress/W&B aggregation and targeted coverage passed.

Documentation and Artifacts

• Docs updated, or not needed
• If docs changed: make docs-build passes locally
• If tutorial sources changed: notebooks regenerated with make convert-notebooks
• If e2e, benchmark, or model-provider behavior changed: relevant validation is listed above

[greptile-apps]

Greptile Summary

This PR adds rat_bench_reidentification as a first-class measurement type in the Anonymizer W&B ingestion pipeline. It introduces the ingress validation model, the W&B table row model, outbound field policies, and scalar aggregation registration — along with regression coverage for all three concerns.

• Adds RatBenchReidentificationMeasurement (ingress) and RatBenchReidentificationTableRow (W&B table) with a new bounded Percentage type (ge=0, le=100) applied to percentage fields, and a validated RatBenchAttackerEndpointKind literal for endpoint metadata.
• Registers row-count fields in SCALAR_ADDITIVE_FIELDS, rate/score fields in SCALAR_AVERAGED_FIELDS, and attacker_endpoint_kind/reid_threshold in SCALAR_LAST_VALUE_FIELDS; prior thread issues around reid_threshold classification and missing upper bounds on percentage fields are addressed.

Confidence Score: 5/5

Safe to merge; changes are well-scoped schema additions that follow existing codebase patterns.

The new measurement type is a purely additive schema extension with no changes to existing code paths. Ingress validation is strict (bounded Percentage type, discriminated union, endpoint kind literal), outbound field policies cover all model fields, and the previous thread issues are resolved. The two observations are cosmetic/data-quality concerns that do not affect runtime correctness.

No files require special attention.

Important Files Changed

Filename	Overview
src/anonymizer/measurement/fields.py	Adds 10 additive count fields, 3 averaged rate/score fields, and 2 last-value configuration fields for RAT-Bench; reid_threshold is correctly placed in SCALAR_LAST_VALUE_FIELDS.
tools/measurement/measurement_tools/validation.py	Adds Percentage type (ge=0, le=100) and RatBenchAttackerEndpointKind literal; minimal, correctly scoped additions.
tools/measurement/measurement_tools/wandb_ingress.py	Adds RatBenchReidentificationMeasurement with Percentage bounds on rate/coverage fields, and registers it in the MeasurementRecord discriminated union.
tools/measurement/measurement_tools/wandb_models.py	Adds RatBenchReidentificationTableRow extending _MetricTableRow, registers it in WandbTableRow union and WANDB_TABLE_ROW_MODELS, and provides a complete OUTBOUND_FIELD_POLICIES entry covering all model fields.
tests/tools/test_measurement_ingress_completion.py	Adds three ingress tests: acceptance of a valid record, rejection of percentages above 100, and rejection of an unknown endpoint kind.
tests/tools/test_measurement_wandb_logging.py	Adds W&B scalar aggregation test including positive assertion for reid_threshold last-value and negative assertion that reid_threshold_mean is absent.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Producer["RAT-Bench producer\n(JSONL measurement record)"]
    Ingress["RatBenchReidentificationMeasurement\nwandb_ingress.py"]
    Discriminator["MeasurementRecord discriminated union"]
    ScalarExtract["extract_scalar_metrics"]
    AddFields["SCALAR_ADDITIVE_FIELDS -> SUM"]
    LastFields["SCALAR_LAST_VALUE_FIELDS -> LAST"]
    AvgFields["SCALAR_AVERAGED_FIELDS -> MEAN"]
    WBScalars["W&B history / summary"]
    TableRow["RatBenchReidentificationTableRow"]
    WBTable["W&B table"]
    Producer -->|JSONL line| Ingress
    Ingress --> Discriminator
    Discriminator --> ScalarExtract
    ScalarExtract --> AddFields --> WBScalars
    ScalarExtract --> LastFields --> WBScalars
    ScalarExtract --> AvgFields --> WBScalars
    Discriminator --> TableRow --> WBTable

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    Producer["RAT-Bench producer\n(JSONL measurement record)"]
    Ingress["RatBenchReidentificationMeasurement\nwandb_ingress.py"]
    Discriminator["MeasurementRecord discriminated union"]
    ScalarExtract["extract_scalar_metrics"]
    AddFields["SCALAR_ADDITIVE_FIELDS -> SUM"]
    LastFields["SCALAR_LAST_VALUE_FIELDS -> LAST"]
    AvgFields["SCALAR_AVERAGED_FIELDS -> MEAN"]
    WBScalars["W&B history / summary"]
    TableRow["RatBenchReidentificationTableRow"]
    WBTable["W&B table"]
    Producer -->|JSONL line| Ingress
    Ingress --> Discriminator
    Discriminator --> ScalarExtract
    ScalarExtract --> AddFields --> WBScalars
    ScalarExtract --> LastFields --> WBScalars
    ScalarExtract --> AvgFields --> WBScalars
    Discriminator --> TableRow --> WBTable

Loading

_{Reviews (3): Last reviewed commit: "fix(measurement): sanitize RAT-Bench att..." | Re-trigger Greptile}

[andreatnvidia]

These fields are included in SCALAR_LAST_VALUE_FIELDS, so arbitrary attacker_model and attacker_endpoint_kind values reach aggregate W&B metrics unchanged. Could we constrain attacker_endpoint_kind to known values and omit attacker_model from aggregate metrics, or validate it against an explicit allowlist? A canary test would help preserve the sanitized publication boundary.

[lipikaramaswamy]

Thanks, agreed. I pushed 6a21815 to tighten this boundary:

• attacker_endpoint_kind is now constrained to the known producer values: bigiron and nvidia_inference_fallback.
• attacker_model is still preserved on the row-level RAT-Bench re-ID record/table for auditability, but is no longer emitted as an aggregate W&B scalar.
• Added regression coverage for rejecting unknown endpoint kinds and for ensuring attacker_model does not appear in aggregate metrics.