Skip to content

Dual-path retry: exponential backoff + CDN-driven httpConfig support - #143

Closed
MichaelGHSeg wants to merge 4 commits into
mainfrom
status-response-update
Closed

Dual-path retry: exponential backoff + CDN-driven httpConfig support#143
MichaelGHSeg wants to merge 4 commits into
mainfrom
status-response-update

Conversation

@MichaelGHSeg

Copy link
Copy Markdown
Contributor

Summary

Rewrites HTTPClient.Upload() with a structured dual-path retry system and adds support for reading retry configuration from CDN settings (integrations["Segment.io"].httpConfig).

Retry logic (HTTPClient.cs)

  • 429 + Retry-After header: sleep for the specified duration, does NOT consume the retry budget. Bounded by MaxRateLimitRetries and MaxRateLimitDuration (default 12h). Retry-After: 0 is treated as a 1-second floor to prevent busy-loops.
  • Other retryable errors (5xx, 408, 410, 460): counted exponential backoff (base 500ms, 2×, cap 60s). Bounded by MaxRetries and MaxTotalBackoffDuration (default 12h).
  • Non-retryable errors: discard immediately.
  • Returns true on all terminal failure paths so EventPipeline removes the batch file instead of re-uploading indefinitely.
  • New configurable properties: MaxRetries, MaxRateLimitRetries, BackoffEnabled, RateLimitEnabled, BaseBackoffMs, MaxBackoffMs, MaxRetryAfterCapSeconds, StatusCodeOverrides.
  • Adds X-Retry-Count header on retry attempts.

CDN-driven config (SegmentDestination.cs)

  • Reads httpConfig.backoffConfig and httpConfig.rateLimitConfig from CDN settings and applies them to the HTTP client after settings arrive.
  • statusCodeOverrides uses dynamic key enumeration (matching the analytics-kotlin approach) instead of a hardcoded status code list — any valid HTTP status code (100–599) in the CDN config is respected.
  • Uses CultureInfo.InvariantCulture for double.TryParse to handle non-English locales correctly.
  • Maps maxRetryInterval to MaxRetryAfterCapSeconds (per-request sleep cap, not total budget).

E2E cli (e2e-cli/)

  • CapturingLogger filters to final-failure messages only ("Retries exhausted", "Max total backoff") to avoid false failures from transient per-attempt errors.
  • Uses enrichment callbacks for userId injection instead of pre-calling Identify.
  • Adds 2s wait after new Analytics() when AUTO_SETTINGS=true so CDN settings arrive before the first upload.
  • Enables retry and retry-settings test suites.

Test plan

  • dotnet test passes
  • E2E basic,retry,retry-settings suites pass (68/68)

- HTTPClient: dual-path retry loop (429+Retry-After vs counted exponential backoff), configurable retry properties (MaxRetries, MaxTotalBackoffDuration, MaxRateLimitDuration, BackoffEnabled, RateLimitEnabled, BaseBackoffMs, MaxBackoffMs, StatusCodeOverrides), X-Retry-Count header on retries
- Configuration: add MaxRetries, MaxTotalBackoffDuration, MaxRateLimitDuration constructor params
- EventPipeline: wire config retry params into HTTPClient at construction; expose _httpClient as internal
- SegmentDestination: parse httpConfig from CDN settings response and apply to HTTPClient at runtime
- UnityHTTPClient: update DoPost signature to match new abstract base
- Tests.csproj: add net10.0 target for local dev (net6.0 runtime no longer installed)
HTTPClient.cs:
- Rewrite Upload() with dual-path retry: 429+Retry-After uses uncounted
  rate-limit path; other retryable errors use counted exponential backoff
- Add configurable properties: MaxRetries, MaxRateLimitRetries,
  BackoffEnabled, RateLimitEnabled, BaseBackoffMs, MaxBackoffMs,
  MaxRetryAfterCapSeconds, StatusCodeOverrides
- Return true (discard) on all terminal failure paths so EventPipeline
  removes the batch file instead of re-uploading indefinitely
- Accept Retry-After: 0 as valid (sleep 0ms, still counts as rate-limit)

SegmentDestination.cs:
- Read httpConfig from CDN settings and apply backoffConfig/rateLimitConfig
- Use CultureInfo.InvariantCulture for double parsing
- Map maxRetryInterval to MaxRetryAfterCapSeconds (per-request cap)

e2e-cli:
- Add CapturingLogger that only captures final-failure messages
- Use enrichment callbacks for userId instead of pre-identify
- Add AUTO_SETTINGS 2s wait for settings to arrive before first upload
- Enable retry-settings test suite
…ides

- Clamp ParseRetryAfter minimum to 1 second to prevent busy-loop when
  server sends Retry-After: 0 or MaxRetryAfterCapSeconds is misconfigured
- Replace hardcoded status code list in ParseStatusCodeOverrides with
  dynamic key enumeration (matches analytics-kotlin approach), so CDN
  overrides for any valid HTTP status code (100-599) are respected
@MichaelGHSeg

Copy link
Copy Markdown
Contributor Author

Closing this as superseded, with the surviving piece carried forward in #147.

Context. This branch and #144 were never stacked — both branched independently off 85f025e ("Add e2e-cli for E2E testing (#142)", May 11). None of this PR's commits (bb0e925 May 15 → b50d4de May 21) appear in #144's history (6732ff6 May 28 → 11198cc Jun 4). They were parallel implementations of the same goal, and #144 landed first.

What #144 already covers. #144 reimplemented this ground more thoroughly (1,929 insertions vs 326 here): the full Retry/ package — RetryStateMachine, RetryConfig, RetryState, RetryStateStorage, HttpConfigParser, TimeProvider — plus EventPipeline/SyncEventPipeline.UpdateHttpConfig, e2e-cli/Program.cs, and tests. e2e-cli/e2e-config.json is byte-identical between the two branches. The Unity DoPost signature change here is no longer needed either: #144 added a virtual 3-arg DoPost that delegates to the abstract 2-arg one, so the sample still compiles.

What was still missing, and where it went. #144 left no user-facing entry point — RateLimitConfig, BackoffConfig, RetryConfig and HttpConfig were all internal, so retry behaviour could only come from CDN settings. This PR fixed that with individual MaxRetries / MaxTotalBackoffDuration / MaxRateLimitDuration knobs on Configuration. That shape matches analytics-python, but not Kotlin (Configuration.httpConfig: HttpConfig?) or Swift (public func httpConfig(_:)) — the SDKs #144 was written to align with.

#147 closes the gap in the Kotlin/Swift shape instead: it makes HttpConfig and friends public and adds Configuration.HttpConfig. #148 then adds generic Retry-After/529 handling on top.

Nothing here is lost — the branch remains if we want to revisit the per-knob API separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant