fix: quiet false-positive alerts and improve slow-upstream handling - #1938
fix: quiet false-positive alerts and improve slow-upstream handling#1938paustint wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses several sources of noisy/low-signal alerts and improves user-facing behavior when upstream (Salesforce) is slow or unreachable across Jetstream’s browser clients and API server.
Changes:
- Disable Zod’s browser JIT probe under strict CSP via a shared helper invoked from browser entry points.
- Expand error-tracker ignore rules (with specs) to suppress Monaco
blob:worker load failure alert storms. - Add a bounded deferred-response max duration backstop and map opaque Node/undici
fetch failedtransport errors to clearer user-facing copy (with specs).
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/shared/utils/src/lib/configure-zod.ts | Adds disableZodJitForStrictCsp helper to avoid CSP noise from Zod’s JIT probe. |
| libs/shared/utils/src/index.ts | Exports the new Zod configuration helper from shared utils. |
| libs/shared/ui-utils/src/lib/errorTracker.ts | Adds Monaco blob: worker-load ignore rule and exports shouldIgnore for testing. |
| libs/shared/ui-utils/src/lib/tests/errorTracker.spec.ts | Adds coverage for new and existing error-tracker ignore rules. |
| libs/shared/constants/src/lib/shared-constants.ts | Introduces user-facing upstream timeout/unreachable messages. |
| libs/api-config/src/lib/env-config.ts | Adds DEFERRED_RESPONSE_MAX_DURATION_MS env var schema entry and updates defaults comment. |
| apps/landing/pages/_app.js | Calls disableZodJitForStrictCsp during landing app boot. |
| apps/jetstream/src/main.tsx | Calls disableZodJitForStrictCsp during web app boot. |
| apps/jetstream-web-extension/src/utils/web-extension.utils.ts | Calls disableZodJitForStrictCsp in extension render entrypoint. |
| apps/jetstream-desktop-client/src/main.tsx | Calls disableZodJitForStrictCsp during desktop client boot. |
| apps/jetstream-canvas/src/main.tsx | Calls disableZodJitForStrictCsp during canvas app boot. |
| apps/api/src/app/utils/error-handler.ts | Unwraps undici fetch failed (via cause.code) into clearer user-facing messages. |
| apps/api/src/app/utils/deferred-response.middleware.ts | Adds a max-duration backstop timer that abandons stalled deferred responses with an error body. |
| apps/api/src/app/utils/tests/error-handler.spec.ts | Adds specs for fetch failed → user-facing message mapping. |
| apps/api/src/app/utils/tests/deferred-response.middleware.spec.ts | Adds specs for the deferred-response max-duration backstop and timer cleanup. |
| apps/api/src/app/types/route.types.ts | Extends deferred response state to track maxDurationTimer. |
| .env.example | Documents the new DEFERRED_RESPONSE_MAX_DURATION_MS setting. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
a18ec2f to
14c6cd5
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.
Suppressed comments (3)
apps/api/src/app/utils/response.handlers.ts:271
- This comment still references the old 45s deferred threshold; the middleware default is now 75s. Updating it avoids misleading future maintainers when interpreting logs and timeouts.
// Clear the deferred timer on fast responses to avoid holding req/res references for 45s
if (deferred) {
clearDeferredTimers(deferred);
}
libs/shared/utils/src/lib/tests/zod-eval-probe.utils.ts:33
hasProbedForEvalcurrently returnstrueifObject.getOwnPropertyDescriptor(...)returnsundefined(becauseundefined === undefined). That would incorrectly report that Zod has probed even when the descriptor can't be inspected (e.g. if Zod changes internals), potentially masking a real regression.
export function hasProbedForEval(): boolean {
return Object.getOwnPropertyDescriptor(util.allowsEval, 'value')?.get === undefined;
}
apps/api/src/app/utils/response.handlers.ts:92
- This comment still says the deferred timer holds req/res references for 45s, but the middleware threshold default is now 75s (and max duration also exists). Keeping this accurate helps future debugging and alert triage.
This issue also appears on line 268 of the same file.
// Clear the deferred timer on fast responses to avoid holding req/res references for 45s
if (deferred) {
clearDeferredTimers(deferred);
}
14c6cd5 to
6fb4ce3
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.
Suppressed comments (1)
libs/shared/utils/src/lib/tests/configure-zod-timing.spec.ts:45
vi.resetModules()can make../configure-zodimport a freshzodmodule instance, while this test still asserts against the pre-resetzimported at file scope. That can let the test pass even ifconfigure-zodmutates the fresh instance.
Re-import z after resetModules (or use vi.isolateModules) so the assertion observes the same instance configure-zod configures.
z.config({ jitless: false });
vi.stubGlobal('window', undefined);
vi.resetModules();
await import('../configure-zod');
Four unrelated issues found while triaging a day of Better Stack alerts.
None were a server fault — the API served ~370k requests that day with 2
failures — but each produced misleading signal or a poor user outcome.
Disable Zod's JIT probe in browsers (~3k CSP reports/day)
Zod probes `new Function` to decide whether to JIT-compile validators.
Our CSP has no 'unsafe-eval', so the probe always fails; Zod swallows
the throw, but the browser still reports a script-src violation on
every page load. `jitless` short-circuits the probe before it runs, at
no runtime cost since CSP had already forced the interpreted path.
No-ops outside the browser, so the server keeps the JIT path.
Winning that race takes two things. Zod probes when the first schema is
*constructed* — the `z.object({...})` at module scope in
@jetstream/types and friends — not when one is parsed, and ES modules
evaluate every import before the importing module's body, so this has
to be a side-effect module imported first rather than a function called
from an entry point. Position alone is still not enough under Vite:
merged into the entry chunk it runs after every chunk that entry
imports, schemas included, so each app also gives it its own chunk.
Verified by loading the production builds under a production-like CSP —
the violation is reported without either half, and gone with both.
Ignore Monaco blob: worker load failures
Monaco builds editor workers from a blob: bootstrap that importScripts
the real worker file. When that fetch is blocked by an extension or
proxy, the existing `/js/monaco/vs/` ignore rule cannot match, because
the frame is the blob URL. Monaco falls back to the main thread, so it
is not actionable — and each occurrence carries a fresh blob UUID, so
every one groups as a brand-new error and re-alerts. One user produced
~10 alerts this way. Kept narrow: a raw error still needs a blob: frame,
so an identically worded app fetch failure (the wording is Firefox's)
is reported as before.
Bound deferred responses via DEFERRED_RESPONSE_MAX_DURATION_MS (10m)
The keepalive loop ran for as long as upstream stayed silent, holding a
socket open long after the client and Cloudflare had given up.
Salesforce calls over fetch are bounded by undici's 300s headersTimeout;
nothing bounded the other paths. The budget is measured from when the
request arrived, so it can be compared directly against upstream
timeouts. Once abandoned, the response is flagged, so the controller
finishing later is logged as the expected tail rather than reported as
an unhandled response. Backstop only — does not fire on current traffic.
Stop leaking undici's opaque "fetch failed" to users
Node's fetch collapses every transport failure into `TypeError: fetch
failed`, which reached users verbatim ("Error saving permissions: fetch
failed"). Now unwrapped via error.cause. The timeout copy deliberately
does not say the operation failed: the request did reach Salesforce, so
it may well have been applied with only the response lost, and the user
needs to re-check rather than blindly retry. Failures that never got
that far, connect timeouts included, are reported as unreachable so the
user can simply retry.
Adds specs for the error-tracker ignore rules, the deferred backstop, the
fetch-failure mapping, and Zod's probe timing plus the import-order and
chunking contract that depends on it.
6fb4ce3 to
53f69c8
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.
Suppressed comments (2)
apps/api/src/app/utils/response.handlers.ts:98
- The comment about clearing the deferred timer still mentions “45s”, but the deferred threshold default is now 75s (and configurable). This is likely to mislead future readers when debugging deferred-response behavior.
if (res.headersSent) {
// The max duration backstop already ended this response and logged why. The late body is the
// expected tail of that, so it is not reported as an unhandled response.
if (deferred?.abandoned) {
getLogger().info(
libs/shared/utils/src/lib/tests/configure-zod-timing.spec.ts:45
- This test uses
vi.resetModules()and then asserts against the statically importedz. After a module reset,../configure-zodwill use a freshzodmodule instance, so this assertion can pass even if the newly-imported instance was configured. Re-importzodafter the reset (or after importing../configure-zod) and assert against that instance instead.
it('should leave the JIT path alone outside the browser', async () => {
z.config({ jitless: false });
vi.stubGlobal('window', undefined);
vi.resetModules();
await import('../configure-zod');
expect(z.config().jitless).toBe(false);
vi.unstubAllGlobals();
Four unrelated issues found while triaging a day of Better Stack alerts. None were a server fault — the API served ~370k requests that day with 2 failures — but each produced misleading signal or a poor user outcome.
Disable Zod's JIT probe in browsers (~3k CSP reports/day)
Zod probes
new Functionto decide whether to JIT-compile validators.Our CSP has no 'unsafe-eval', so the probe always fails; Zod swallows
the throw, but the browser still reports a script-src violation on
every page load.
jitlessshort-circuits the probe before it runs, atno runtime cost since CSP had already forced the interpreted path.
Called from browser entry points only — the server keeps the JIT path.
Ignore Monaco blob: worker load failures
Monaco builds editor workers from a blob: bootstrap that importScripts
the real worker file. When that fetch is blocked by an extension or
proxy, the existing
/js/monaco/vs/ignore rule cannot match, becausethe frame is the blob URL. Monaco falls back to the main thread, so it
is not actionable — and each occurrence carries a fresh blob UUID, so
every one groups as a brand-new error and re-alerts. One user produced
~10 alerts this way. Kept narrow: a raw error still needs a blob: frame,
so an identically worded app fetch failure (the wording is Firefox's)
is reported as before.
Bound deferred responses via DEFERRED_RESPONSE_MAX_DURATION_MS (10m)
The keepalive loop ran for as long as upstream stayed silent, holding a
socket open long after the client and Cloudflare had given up.
Salesforce calls over fetch are bounded by undici's 300s headersTimeout;
nothing bounded the other paths. Backstop only — does not fire on
current traffic.
Stop leaking undici's opaque "fetch failed" to users
Node's fetch collapses every transport failure into
TypeError: fetch failed, which reached users verbatim ("Error saving permissions: fetchfailed"). Now unwrapped via error.cause. The timeout copy deliberately
does not say the operation failed: the request did reach Salesforce, so
it may well have been applied with only the response lost, and the user
needs to re-check rather than blindly retry.
Adds specs for the error-tracker ignore rules, the deferred backstop, and the fetch-failure mapping.