-
Notifications
You must be signed in to change notification settings - Fork 0
docs: add troubleshooting section for common SRM causes and SDK mistakes (FT-1878) #277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
ee95811
docs: add troubleshooting section for common SRM causes and SDK mistakes
calthejuggler c4efc60
docs: fix JS API name and publish ordering in context best practices
calthejuggler 71e52b8
docs: simplify troubleshooting examples to TypeScript
calthejuggler cf054d4
docs: link Events > Explore and Metrics from goal undercounting diagn…
calthejuggler 055c287
docs: fix multiple-contexts cause to describe duplicate exposures
calthejuggler File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
6 changes: 6 additions & 0 deletions
6
docs/APIs-and-SDKs/SDK-Documentation/troubleshooting/_category_.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "label": "Troubleshooting", | ||
| "position": 11, | ||
| "collapsible": true, | ||
| "collapsed": true | ||
| } |
47 changes: 47 additions & 0 deletions
47
docs/APIs-and-SDKs/SDK-Documentation/troubleshooting/goal-undercounting.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| --- | ||
| sidebar_position: 2 | ||
| --- | ||
|
|
||
| # Goal undercounting | ||
|
|
||
| Your experiment results show fewer goal events than you expect, or goals that should be common look rare. The usual causes are ordering (the goal fires before exposure) and metric misconfiguration (the time or property filters on the metric exclude the events you wanted to count). | ||
|
|
||
| ## Goal tracked before exposure | ||
|
|
||
| A goal event must be recorded after the exposure for the SDK to attribute it to the experiment. When the goal fires first, the SDK has no exposure to tie it to, and the event is not counted against the experiment. | ||
|
|
||
| The most common version of this is on a page where the user lands and the goal event (a page view, a purchase, a sign-up) fires before your experiment code calls `treatment()`. | ||
|
|
||
| **Wrong** - the goal event fires before exposure: | ||
|
|
||
| ```ts | ||
| analytics.track("checkout_complete", { orderId: 123 }); | ||
| // Too late: the goal already fired. | ||
| const variant = context.treatment("checkout_flow"); | ||
| ``` | ||
|
|
||
| **Right** - expose first, then track the goal: | ||
|
|
||
| ```ts | ||
| const variant = context.treatment("checkout_flow"); | ||
| // Now the SDK has recorded the exposure. | ||
| context.track("checkout_complete"); | ||
| ``` | ||
|
|
||
| ## Metric time or property filters misconfigured | ||
|
|
||
| A metric is more than a goal name. When you configure a metric in the web console, you can restrict which events count by time window (relative to exposure) or by event properties. If those filters are wrong, events that you expect to count are excluded. | ||
|
|
||
| ### Time window too narrow | ||
|
|
||
| A purchase goal with a 1-hour window will not count purchases that happen the next day. If your conversion cycle is longer than the window, undercounting is guaranteed. | ||
|
|
||
| ### Property filter excludes valid events | ||
|
|
||
| A goal configured to only count events where `plan == "pro"` will miss every `plan == "enterprise"` conversion. The filter discards it even though the event fires. | ||
|
|
||
| ### How to diagnose | ||
|
|
||
| 1. Open the metric in the web console and check the time window and property filters against the events you expect to count. See [Metrics](/docs/web-console-docs/goals-and-metrics/metrics/overview) for how metrics are configured. | ||
| 2. Look at the raw event stream for a user you know converted in [Events > Explore](/docs/web-console-docs/Events/the-events-page). If the event is present but the metric does not count it, the filter is the cause. | ||
| 3. Widen the time window or relax the property filter and re-check. |
49 changes: 49 additions & 0 deletions
49
.../APIs-and-SDKs/SDK-Documentation/troubleshooting/sdk-context-best-practices.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| --- | ||
| sidebar_position: 3 | ||
| --- | ||
|
|
||
| # Common SDK mistakes | ||
|
|
||
| The symptoms below show up as data quality problems in the web console: SRM warnings, assignment conflicts, or participants counted under the wrong user. Each entry describes the symptom, the cause, and the fix. | ||
|
|
||
| ## Symptom: SRM warning from duplicate exposures | ||
|
|
||
| You see an SRM alert on the experiment dashboard, and the raw event stream shows the same user exposed multiple times. | ||
|
|
||
| **Cause: multiple contexts for one user.** A context is scoped to one user. If you create a new context for every treatment check, the SDK records a separate exposure each time. Variant assignment is deterministic from the unit ID and experiment name, so the user sees the same variant on every request. But each context emits its own exposure event, so the analysis over-counts that user and the variant split skews. On a server, each request gets its own context; on a client, the context should live for the session. | ||
|
|
||
| **Fix: one context per user lifecycle.** Create the context once at the start of the request or app session, reuse it for every treatment check and goal track, and finalize it before the response returns or the session ends. | ||
|
|
||
| **Wrong** - new context per treatment check: | ||
|
|
||
| ```ts | ||
| export function handleRequest(req: Request, res: Response) { | ||
| // New context each call. The user gets the same variant, | ||
| // but the SDK records a separate exposure each time. | ||
| const context = sdk.createContext({ units: { user_id: req.userId } }); | ||
| const variant = context.treatment("checkout_flow"); | ||
| res.render(variant === 1 ? "new" : "old"); | ||
| } | ||
| ``` | ||
|
|
||
| **Right** - one context per request, finalized before the response: | ||
|
|
||
| ```ts | ||
| export async function handleRequest(req: Request, res: Response) { | ||
| const context = sdk.createContext({ units: { user_id: req.userId } }); | ||
| const variant = context.treatment("checkout_flow"); | ||
| await context.publish(); | ||
| context.finalize(); | ||
| res.render(variant === 1 ? "new" : "old"); | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| ## Symptom: participants counted under the wrong user after login or logout | ||
|
|
||
| After a user logs out or a different user logs in, the next participant is counted under the previous user's identity. | ||
|
|
||
| **Cause: context not reset on user change.** The current context belongs to the old user. Reusing it leaks the old user's units and attributes into the new session. | ||
|
|
||
| **Fix: finalize the old context and create a new one.** On logout, finalize the existing context and create a new one with a fresh `anonymous_id`. On login, add the `user_id` to the existing context (or create a new one if you did not keep the anonymous session). The React SDK's `resetContext` does this for you. | ||
|
|
||
| For the full lifecycle (create, publish, finalize, refresh, reset), see [Managing the context lifecycle](/docs/APIs-and-SDKs/SDK-Documentation/managing-the-context-lifecycle). | ||
111 changes: 111 additions & 0 deletions
111
docs/APIs-and-SDKs/SDK-Documentation/troubleshooting/srm-and-exposure-issues.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| --- | ||
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| # SRM and exposure issues | ||
|
|
||
| A [Sample Ratio Mismatch (SRM)](/docs/glossary#srm-sample-ratio-mismatch) warning means the split of participants across your variants does not match the configured allocation. ABsmartly shows an SRM alert on the experiment dashboard when the mismatch is statistically significant. See [Experiment health checks](/docs/web-console-docs/experiments/Experiment-health-checks) for how the alert surfaces. | ||
|
|
||
| The mistakes below are the most common causes. Each one shows the wrong code first, then the fix. | ||
|
|
||
| ## Conditional exposure | ||
|
|
||
| Your variants call `treatment()` at different points in the code path, so some users are exposed and counted in one variant but never reach the call in another. | ||
|
|
||
| :::tip Rule of thumb | ||
| Call `treatment()` exactly once, in the same place in the request or page lifecycle, for every user who should be in the experiment. Branch on the value you get back, not on whether to call it. | ||
| ::: | ||
|
|
||
| **Wrong** - `treatment()` called only on one code path: | ||
|
|
||
| ```ts | ||
| // Only users who see the new checkout call treatment(). | ||
| if (user.featureEnabled) { | ||
| const variant = context.treatment("checkout_flow"); | ||
| if (variant === 1) { | ||
| renderNewCheckout(); | ||
| } else { | ||
| renderOldCheckout(); | ||
| } | ||
| } else { | ||
| // These users never call treatment(), so they are | ||
| // never exposed or counted in the experiment. | ||
| renderOldCheckout(); | ||
| } | ||
| ``` | ||
|
|
||
| **Right** - call `treatment()` once, before any branching: | ||
|
|
||
| ```ts | ||
| const variant = context.treatment("checkout_flow"); | ||
|
|
||
| // Exposure is recorded for everyone at the same point. | ||
| if (variant === 1) { | ||
| renderNewCheckout(); | ||
| } else { | ||
| renderOldCheckout(); | ||
| } | ||
| ``` | ||
|
|
||
| ## Audience filtering before treatment | ||
|
|
||
| You check an attribute in your own code and skip `treatment()` for users who should not be in the experiment. But the experiment's audience is configured in the web console for a reason: when you filter in code, the users who never call `treatment()` are invisible to the analysis. If they behave differently across variants, the remaining counts skew. | ||
|
|
||
| The fix is to let the SDK apply the audience. Call `treatment()` for everyone who hits the experiment surface. The SDK returns variant 0 (control) for users outside the audience, and the exposure is recorded consistently. | ||
|
|
||
| **Wrong** - filtering in code before calling `treatment()`: | ||
|
|
||
| ```ts | ||
| // You filter out mobile users yourself, so they never call treatment(). | ||
| if (context.attributes().platform === "mobile") { | ||
| return renderDefault(); | ||
| } | ||
|
|
||
| const variant = context.treatment("checkout_flow"); | ||
| ``` | ||
|
|
||
| **Right** - call `treatment()` for everyone, let the audience do the filtering: | ||
|
|
||
| ```ts | ||
| const variant = context.treatment("checkout_flow"); | ||
|
|
||
| // The SDK returns 0 for users outside the audience. | ||
| if (variant === 1) { | ||
| renderNewCheckout(); | ||
| } else { | ||
| renderOldCheckout(); | ||
| } | ||
| ``` | ||
|
|
||
| ## Not awaiting publish before navigation | ||
|
|
||
| `publish()` sends exposure and goal events to the collector. If the page navigates or the server returns a response before the request completes, events are lost. The exposure was recorded locally but never delivered, so the analysis sees fewer participants than expected and the split skews. | ||
|
|
||
| Always await the promise (or call the synchronous blocking variant) before redirecting or returning the response. See [Managing the context lifecycle](/docs/APIs-and-SDKs/SDK-Documentation/managing-the-context-lifecycle) for the full picture. | ||
|
|
||
| **Wrong** - fire and forget before navigation: | ||
|
|
||
| ```ts | ||
| context.publish(); | ||
| window.location.replace("/checkout"); | ||
| // The publish request is still in flight when the page unloads. | ||
| ``` | ||
|
|
||
| **Right** - await the promise: | ||
|
|
||
| ```ts | ||
| await context.publish(); | ||
| window.location.replace("/checkout"); | ||
| ``` | ||
|
|
||
| On the server, call `publish()` before sending the response: | ||
|
|
||
| ```ts | ||
| export async function handleRequest(req: Request, res: Response) { | ||
| const context = sdk.createContext({ units: { user_id: req.userId } }); | ||
| const variant = context.treatment("checkout_flow"); | ||
| await context.publish(); | ||
| context.finalize(); | ||
| res.render(variant === 1 ? "new" : "old"); | ||
| } | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.