Skip to content

fix: quiet false-positive alerts and improve slow-upstream handling - #1938

Open
paustint wants to merge 1 commit into
mainfrom
fix/csp-reporting-and-slow-upstream-handling
Open

fix: quiet false-positive alerts and improve slow-upstream handling#1938
paustint wants to merge 1 commit into
mainfrom
fix/csp-reporting-and-slow-upstream-handling

Conversation

@paustint

Copy link
Copy Markdown
Contributor

Four unrelated issues found while triaging a day of Better Stack alerts. None were a server fault — the API served ~370k requests that day with 2 failures — but each produced misleading signal or a poor user outcome.

Disable Zod's JIT probe in browsers (~3k CSP reports/day)
Zod probes new Function to decide whether to JIT-compile validators.
Our CSP has no 'unsafe-eval', so the probe always fails; Zod swallows
the throw, but the browser still reports a script-src violation on
every page load. jitless short-circuits the probe before it runs, at
no runtime cost since CSP had already forced the interpreted path.
Called from browser entry points only — the server keeps the JIT path.

Ignore Monaco blob: worker load failures
Monaco builds editor workers from a blob: bootstrap that importScripts
the real worker file. When that fetch is blocked by an extension or
proxy, the existing /js/monaco/vs/ ignore rule cannot match, because
the frame is the blob URL. Monaco falls back to the main thread, so it
is not actionable — and each occurrence carries a fresh blob UUID, so
every one groups as a brand-new error and re-alerts. One user produced
~10 alerts this way. Kept narrow: a raw error still needs a blob: frame,
so an identically worded app fetch failure (the wording is Firefox's)
is reported as before.

Bound deferred responses via DEFERRED_RESPONSE_MAX_DURATION_MS (10m)
The keepalive loop ran for as long as upstream stayed silent, holding a
socket open long after the client and Cloudflare had given up.
Salesforce calls over fetch are bounded by undici's 300s headersTimeout;
nothing bounded the other paths. Backstop only — does not fire on
current traffic.

Stop leaking undici's opaque "fetch failed" to users
Node's fetch collapses every transport failure into TypeError: fetch failed, which reached users verbatim ("Error saving permissions: fetch
failed"). Now unwrapped via error.cause. The timeout copy deliberately
does not say the operation failed: the request did reach Salesforce, so
it may well have been applied with only the response lost, and the user
needs to re-check rather than blindly retry.

Adds specs for the error-tracker ignore rules, the deferred backstop, and the fetch-failure mapping.

Copilot AI lite review requested due to automatic review settings August 13, 2026 13:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses several sources of noisy/low-signal alerts and improves user-facing behavior when upstream (Salesforce) is slow or unreachable across Jetstream’s browser clients and API server.

Changes:

  • Disable Zod’s browser JIT probe under strict CSP via a shared helper invoked from browser entry points.
  • Expand error-tracker ignore rules (with specs) to suppress Monaco blob: worker load failure alert storms.
  • Add a bounded deferred-response max duration backstop and map opaque Node/undici fetch failed transport errors to clearer user-facing copy (with specs).

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
libs/shared/utils/src/lib/configure-zod.ts Adds disableZodJitForStrictCsp helper to avoid CSP noise from Zod’s JIT probe.
libs/shared/utils/src/index.ts Exports the new Zod configuration helper from shared utils.
libs/shared/ui-utils/src/lib/errorTracker.ts Adds Monaco blob: worker-load ignore rule and exports shouldIgnore for testing.
libs/shared/ui-utils/src/lib/tests/errorTracker.spec.ts Adds coverage for new and existing error-tracker ignore rules.
libs/shared/constants/src/lib/shared-constants.ts Introduces user-facing upstream timeout/unreachable messages.
libs/api-config/src/lib/env-config.ts Adds DEFERRED_RESPONSE_MAX_DURATION_MS env var schema entry and updates defaults comment.
apps/landing/pages/_app.js Calls disableZodJitForStrictCsp during landing app boot.
apps/jetstream/src/main.tsx Calls disableZodJitForStrictCsp during web app boot.
apps/jetstream-web-extension/src/utils/web-extension.utils.ts Calls disableZodJitForStrictCsp in extension render entrypoint.
apps/jetstream-desktop-client/src/main.tsx Calls disableZodJitForStrictCsp during desktop client boot.
apps/jetstream-canvas/src/main.tsx Calls disableZodJitForStrictCsp during canvas app boot.
apps/api/src/app/utils/error-handler.ts Unwraps undici fetch failed (via cause.code) into clearer user-facing messages.
apps/api/src/app/utils/deferred-response.middleware.ts Adds a max-duration backstop timer that abandons stalled deferred responses with an error body.
apps/api/src/app/utils/tests/error-handler.spec.ts Adds specs for fetch failed → user-facing message mapping.
apps/api/src/app/utils/tests/deferred-response.middleware.spec.ts Adds specs for the deferred-response max-duration backstop and timer cleanup.
apps/api/src/app/types/route.types.ts Extends deferred response state to track maxDurationTimer.
.env.example Documents the new DEFERRED_RESPONSE_MAX_DURATION_MS setting.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/api/src/app/utils/error-handler.ts
Comment thread apps/api/src/app/utils/error-handler.ts Outdated
Comment thread apps/api/src/app/utils/__tests__/error-handler.spec.ts Outdated
Comment thread libs/shared/utils/src/lib/configure-zod.ts Outdated
Copilot AI review requested due to automatic review settings August 13, 2026 14:19
@paustint
paustint force-pushed the fix/csp-reporting-and-slow-upstream-handling branch from a18ec2f to 14c6cd5 Compare August 13, 2026 14:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (3)

apps/api/src/app/utils/response.handlers.ts:271

  • This comment still references the old 45s deferred threshold; the middleware default is now 75s. Updating it avoids misleading future maintainers when interpreting logs and timeouts.
    // Clear the deferred timer on fast responses to avoid holding req/res references for 45s
    if (deferred) {
      clearDeferredTimers(deferred);
    }

libs/shared/utils/src/lib/tests/zod-eval-probe.utils.ts:33

  • hasProbedForEval currently returns true if Object.getOwnPropertyDescriptor(...) returns undefined (because undefined === undefined). That would incorrectly report that Zod has probed even when the descriptor can't be inspected (e.g. if Zod changes internals), potentially masking a real regression.
export function hasProbedForEval(): boolean {
  return Object.getOwnPropertyDescriptor(util.allowsEval, 'value')?.get === undefined;
}

apps/api/src/app/utils/response.handlers.ts:92

  • This comment still says the deferred timer holds req/res references for 45s, but the middleware threshold default is now 75s (and max duration also exists). Keeping this accurate helps future debugging and alert triage.

This issue also appears on line 268 of the same file.

  // Clear the deferred timer on fast responses to avoid holding req/res references for 45s
  if (deferred) {
    clearDeferredTimers(deferred);
  }

Copilot AI review requested due to automatic review settings August 13, 2026 14:45
@paustint
paustint force-pushed the fix/csp-reporting-and-slow-upstream-handling branch from 14c6cd5 to 6fb4ce3 Compare August 13, 2026 14:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (1)

libs/shared/utils/src/lib/tests/configure-zod-timing.spec.ts:45

  • vi.resetModules() can make ../configure-zod import a fresh zod module instance, while this test still asserts against the pre-reset z imported at file scope. That can let the test pass even if configure-zod mutates the fresh instance.

Re-import z after resetModules (or use vi.isolateModules) so the assertion observes the same instance configure-zod configures.

    z.config({ jitless: false });
    vi.stubGlobal('window', undefined);
    vi.resetModules();

    await import('../configure-zod');

Four unrelated issues found while triaging a day of Better Stack alerts.
None were a server fault — the API served ~370k requests that day with 2
failures — but each produced misleading signal or a poor user outcome.

Disable Zod's JIT probe in browsers (~3k CSP reports/day)
  Zod probes `new Function` to decide whether to JIT-compile validators.
  Our CSP has no 'unsafe-eval', so the probe always fails; Zod swallows
  the throw, but the browser still reports a script-src violation on
  every page load. `jitless` short-circuits the probe before it runs, at
  no runtime cost since CSP had already forced the interpreted path.
  No-ops outside the browser, so the server keeps the JIT path.

  Winning that race takes two things. Zod probes when the first schema is
  *constructed* — the `z.object({...})` at module scope in
  @jetstream/types and friends — not when one is parsed, and ES modules
  evaluate every import before the importing module's body, so this has
  to be a side-effect module imported first rather than a function called
  from an entry point. Position alone is still not enough under Vite:
  merged into the entry chunk it runs after every chunk that entry
  imports, schemas included, so each app also gives it its own chunk.
  Verified by loading the production builds under a production-like CSP —
  the violation is reported without either half, and gone with both.

Ignore Monaco blob: worker load failures
  Monaco builds editor workers from a blob: bootstrap that importScripts
  the real worker file. When that fetch is blocked by an extension or
  proxy, the existing `/js/monaco/vs/` ignore rule cannot match, because
  the frame is the blob URL. Monaco falls back to the main thread, so it
  is not actionable — and each occurrence carries a fresh blob UUID, so
  every one groups as a brand-new error and re-alerts. One user produced
  ~10 alerts this way. Kept narrow: a raw error still needs a blob: frame,
  so an identically worded app fetch failure (the wording is Firefox's)
  is reported as before.

Bound deferred responses via DEFERRED_RESPONSE_MAX_DURATION_MS (10m)
  The keepalive loop ran for as long as upstream stayed silent, holding a
  socket open long after the client and Cloudflare had given up.
  Salesforce calls over fetch are bounded by undici's 300s headersTimeout;
  nothing bounded the other paths. The budget is measured from when the
  request arrived, so it can be compared directly against upstream
  timeouts. Once abandoned, the response is flagged, so the controller
  finishing later is logged as the expected tail rather than reported as
  an unhandled response. Backstop only — does not fire on current traffic.

Stop leaking undici's opaque "fetch failed" to users
  Node's fetch collapses every transport failure into `TypeError: fetch
  failed`, which reached users verbatim ("Error saving permissions: fetch
  failed"). Now unwrapped via error.cause. The timeout copy deliberately
  does not say the operation failed: the request did reach Salesforce, so
  it may well have been applied with only the response lost, and the user
  needs to re-check rather than blindly retry. Failures that never got
  that far, connect timeouts included, are reported as unreachable so the
  user can simply retry.

Adds specs for the error-tracker ignore rules, the deferred backstop, the
fetch-failure mapping, and Zod's probe timing plus the import-order and
chunking contract that depends on it.
Copilot AI review requested due to automatic review settings August 13, 2026 23:47
@paustint
paustint force-pushed the fix/csp-reporting-and-slow-upstream-handling branch from 6fb4ce3 to 53f69c8 Compare August 13, 2026 23:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (2)

apps/api/src/app/utils/response.handlers.ts:98

  • The comment about clearing the deferred timer still mentions “45s”, but the deferred threshold default is now 75s (and configurable). This is likely to mislead future readers when debugging deferred-response behavior.
  if (res.headersSent) {
    // The max duration backstop already ended this response and logged why. The late body is the
    // expected tail of that, so it is not reported as an unhandled response.
    if (deferred?.abandoned) {
      getLogger().info(

libs/shared/utils/src/lib/tests/configure-zod-timing.spec.ts:45

  • This test uses vi.resetModules() and then asserts against the statically imported z. After a module reset, ../configure-zod will use a fresh zod module instance, so this assertion can pass even if the newly-imported instance was configured. Re-import zod after the reset (or after importing ../configure-zod) and assert against that instance instead.
  it('should leave the JIT path alone outside the browser', async () => {
    z.config({ jitless: false });
    vi.stubGlobal('window', undefined);
    vi.resetModules();

    await import('../configure-zod');

    expect(z.config().jitless).toBe(false);
    vi.unstubAllGlobals();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants