feat: holdout support — normalized models, assignment precedence, exposure fields (ISS-61)#12
feat: holdout support — normalized models, assignment precedence, exposure fields (ISS-61)#12marcio-absmartly wants to merge 7 commits into
Conversation
…osure fields, tests Squashed restore of feat/holdouts work (original commits lost to a tooling accident; content fully preserved): - ExperimentHoldout json model; ContextData.holdouts[] + Experiment.holdoutIds[] (normalized payload: definitions not duplicated per experiment) - Context: holdout check after override, before audience/traffic/variant (Override > Holdout > Audience > Traffic > Variant); id->def map with graceful skip of unknown ids; cache invalidation on holdoutIds and resolved definition changes - Exposure heldOut + holdoutId, serializer round-trip - ContextHoldoutTest matrix + serializer/deserializer coverage
Add missing exposure assertion for the not-held-out-but-has-holdouts path (heldOut=false/holdoutId=0 after holdouts are actually evaluated), plus empty-holdoutIds and empty-split branch coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reuse the already-bound unitType local in the traffic-split branch instead of re-reading experiment.data.unitType, matching the new holdout branch. Behaviour-preserving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughThis change adds holdout support across the SDK. New JSON models and fields cover experiment holdouts in context data, experiment definitions, and exposures. Estimated code review effort: 4 (Complex) | ~60 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java (1)
223-250: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRepeated publish-assertion boilerplate.
The
PublishEvent/when(eventHandler.publish...)/verify(...)sequence is duplicated acrossskipsHoldoutWhenUnitMissingForUnitType,exposureCarriesHeldOutAndHoldoutId, andexposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Consider extracting a small helper (e.g.assertPublishedExposure(context, expectedExposures)) to reduce repetition.Also applies to: 358-414
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java` around lines 223 - 250, The publish assertion setup is duplicated in ContextHoldoutTest across multiple holdout tests, including skipsHoldoutWhenUnitMissingForUnitType, exposureCarriesHeldOutAndHoldoutId, and exposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Extract the repeated PublishEvent construction, when(eventHandler.publish(...)) stubbing, and verify(...) timeout assertion into a small helper such as assertPublishedExposure or assertPublishEvent so the tests stay focused on the scenario-specific expectations.core-api/src/main/java/com/absmartly/sdk/Context.java (1)
786-865: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider de-duplicating the unit lookup in the holdout/traffic-split branches.
The
uid = units_.get(unitType)lookup (plus the follow-ongetUnitHash/getVariantAssignercalls) is repeated once for holdout evaluation and again for traffic-split evaluation. Hoisting theuidlookup once at the top of this block would remove the duplication and slightly reduce the complexity of an already dense method (holdout precedence, audience matching, full-on/traffic logic are all interleaved here).♻️ Suggested simplification
if (experiment != null) { final String unitType = experiment.data.unitType; + final String uid = units_.get(unitType); assignment.holdoutIds = experiment.data.holdoutIds; assignment.holdouts = experiment.holdouts; - if (experiment.holdouts != null && experiment.holdouts.length > 0) { - final String uid = units_.get(unitType); - if (uid != null) { - final byte[] unitHash = Context.this.getUnitHash(unitType, uid); - final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); - for (final ExperimentHoldout holdout : experiment.holdouts) { + if (uid != null && experiment.holdouts != null && experiment.holdouts.length > 0) { + final byte[] unitHash = Context.this.getUnitHash(unitType, uid); + final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); + for (final ExperimentHoldout holdout : experiment.holdouts) { if (assigner.assign(holdout.split, holdout.seedHi, holdout.seedLo) == 0) { assignment.heldOut = true; assignment.holdoutId = holdout.id; assignment.variant = 0; assignment.assigned = true; break; } - } - } + } } if (!assignment.heldOut) { ... } else if (experiment.data.fullOnVariant == 0) { - final String uid = units_.get(unitType); if (uid != null) { final byte[] unitHash = Context.this.getUnitHash(unitType, uid); final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); ... } } else { ... } } ... }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core-api/src/main/java/com/absmartly/sdk/Context.java` around lines 786 - 865, The unit lookup and assigner setup in the experiment assignment flow is duplicated in both the holdout and traffic-split branches. In Context’s assignment block, hoist the units_.get(unitType) lookup (and, if available, the derived getUnitHash/getVariantAssigner setup) once before the conditional branches, then reuse it for holdout and eligibility evaluation. Keep the existing holdout precedence and audience/full-on behavior unchanged while simplifying the repeated logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@core-api/src/main/java/com/absmartly/sdk/Context.java`:
- Around line 786-865: The unit lookup and assigner setup in the experiment
assignment flow is duplicated in both the holdout and traffic-split branches. In
Context’s assignment block, hoist the units_.get(unitType) lookup (and, if
available, the derived getUnitHash/getVariantAssigner setup) once before the
conditional branches, then reuse it for holdout and eligibility evaluation. Keep
the existing holdout precedence and audience/full-on behavior unchanged while
simplifying the repeated logic.
In `@core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java`:
- Around line 223-250: The publish assertion setup is duplicated in
ContextHoldoutTest across multiple holdout tests, including
skipsHoldoutWhenUnitMissingForUnitType, exposureCarriesHeldOutAndHoldoutId, and
exposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Extract the repeated
PublishEvent construction, when(eventHandler.publish(...)) stubbing, and
verify(...) timeout assertion into a small helper such as
assertPublishedExposure or assertPublishEvent so the tests stay focused on the
scenario-specific expectations.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 00fd9122-82f1-4830-b808-f01c239e7192
📒 Files selected for processing (12)
core-api/src/main/java/com/absmartly/sdk/Context.javacore-api/src/main/java/com/absmartly/sdk/json/ContextData.javacore-api/src/main/java/com/absmartly/sdk/json/Experiment.javacore-api/src/main/java/com/absmartly/sdk/json/ExperimentHoldout.javacore-api/src/main/java/com/absmartly/sdk/json/Exposure.javacore-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.javacore-api/src/test/java/com/absmartly/sdk/ContextTest.javacore-api/src/test/java/com/absmartly/sdk/DefaultContextDataDeserializerTest.javacore-api/src/test/java/com/absmartly/sdk/DefaultContextEventSerializerTest.javacore-api/src/test/java/com/absmartly/sdk/json/ContextDataTest.javacore-api/src/test/java/com/absmartly/sdk/json/ExperimentHoldoutTest.javacore-api/src/test/resources/holdouts_context.json
- Assignment cache: treat held-out assignments as up-to-date even when a custom assignment is set. Holdout wins over custom assignment, so the cached variant (0) can never equal the custom variant; the old condition forced a cache miss and a fresh Assignment (exposed=false) on every call, queueing a duplicate exposure per getTreatment. - Tests: cached held-out assignment with custom assignment set; holdout precedence over fullOnVariant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a custom assignment is set but the cached variant was forced by a higher-precedence rule (holdout, full-on, traffic ineligibility, or a strict audience mismatch), the custom value can never equal that variant. The old cache-validity check compared them directly, so it forced a cache miss and a fresh Assignment (exposed=false) on every getTreatment call, queueing a duplicate exposure each time. Introduce variantForcedRegardlessOfCustom() covering all forced-variant cases and treat them as cache-valid. 5af036a fixed the holdout case only; this generalizes the same fix to the audience-strict-mismatch and traffic-ineligible cases, which are pre-existing on main. Tests: cached forced-variant assignments with a custom assignment set for both the strict audience mismatch and traffic ineligible cases (stable pending count across repeated calls). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assignment.holdoutIds/holdouts alias the arrays owned by ContextExperiment instead of copying them. This is safe only because experiment data is treated as immutable after setData and the arrays are read-only here. Note the invariant at the assignment site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Summary
Client-side holdout support (ISS-61, Phase 3) — companion to absmartly/abs#4539 (server side).
ContextData.holdouts[](definitions:{id, seedHi, seedLo, split}) +Experiment.holdoutIds[]references — definitions are not duplicated per experiment.assigned=true; overrides bypass holdouts (debugging escape hatch).VariantAssignerwith the holdout's own seed — same verdict across all experiments referencing the holdout.holdoutIdsor a referenced definition (seed/split) invalidates cached assignments.heldOut+holdoutIdfor downstream analysis.holdoutIdsand malformed definitions (null/short split) are skipped gracefully.Verification
:core-api:test), findbugsMain cleanNotes
Summary by CodeRabbit
heldOutandholdoutId.