Skip to content

fix(api): replay a gateway error only for reads, so a write is never sent twice - #1891

Merged
dawsontoth merged 19 commits into
claude/1283-freeform-operationsfrom
claude/1879-safe-gateway-retry
Oct 13, 2026
Merged

dawsontoth merged 19 commits into
claude/1283-freeform-operationsfrom
claude/1879-safe-gateway-retry

Conversation

@dawsontoth

@dawsontoth dawsontoth commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

A pigeon re-sending questions but holding back a parcel at a broken bridge

Stacked on #1882 (claude/1283-freeform-operations). It reuses that PR's isReadOperation and skipGatewayRetry, so it targets that branch. Retarget it to stage once #1882 merges.

⊙ Problem

Every instance client replayed a 502, 503 or 504 up to three times (5s, 10s, 20s apart), whatever the request. None of those statuses means the request went unapplied, so a write whose answer a gateway lost could run again: an insert storing its records twice, a drop_table that worked and then reported "does not exist", a second deploy_component or restart. Closes #1879.

❓ Your call: Writes now get no automatic retry on any gateway status, 503 included. The issue suggested also replaying a write's 503 ("usually refused before processing"). I left that out: Harper itself answers 503 after it has handed an operation to a worker that then exits or times out (harper server/serverHelpers/registeredOperations.ts, The worker thread executing this operation exited / Timed out waiting for worker thread…). Adding a 503 exemption back would be a one-line change.

💡 Solution

The gateway retry now replays only requests the restart gate already treats as reads: a GET, or a POST whose operations-API body names a read by Harper's naming. A write is sent once, and its first 5xx reaches the caller the same way the last failed retry used to. The predicate moves out of the restart gate into isReadOperation.ts as isReadRequest, so "safe to send late" and "safe to send twice" are one rule with one list. Like the gate, it parses a body that axios has already serialized. That matters here: by the time a response fails, axios 1.20.0 has replaced config.data with its JSON string (lib/core/dispatchRequest.js:46). Anything it does not recognize counts as a write. skipGatewayRetry stays, and the operations console still sets it, because a component can register a get_… operation that writes.

Callers that lose the replay, checked one by one:

  • Restarts (restart / restart_service): a 502/504 now fails the click straight away instead of sending a second restart 5s later. The tracker is still released in finally. The user_info probes in waitUntilAnsweringAfterRestart are reads and keep their retry.
  • Rolling set_configuration: the user_info probe still retries. A 5xx on set_configuration or restart_service counts that member as failed, and the loop moves on to the existing "Only N of M restarted" toast. No member restarts twice.
  • Deploys: the live (SSE) deploy uses fetch and is unaffected. A 504 on the buffered deploy_component now shows "Failed to import" instead of deploying a second time.
  • Logouts: sign-out's logout client already drops every response interceptor (fix(auth): send the sign-out logout to the instance, and stop its failure reaching Error Tracking #1680). Disconnect swallows the error and clears local state anyway; it now finishes up to 35s sooner.
  • Token minting (create_authentication_tokens, refresh_operation_token): neither is a read by name, on purpose (see the restart gate). A gateway status on the route guard's mint now sends the user to sign-in at once instead of holding navigation for up to 35s. A gateway status on a refresh falls back to re-minting through the proxy at once.
  • Polls (get_job, get_status, get_deployment, get_analytics, cluster_status, GET /health, GET /api/openapi/rest): reads, so nothing changes.

❓ Your call: Token minting gives up its retry together with every other write. Both token operations would be safe to replay, as traced in harper security/tokenAuthentication.ts: create_authentication_tokens overwrites hdb_user.refresh_token, so the latest mint wins, and refresh_operation_token writes nothing. I kept the rule as plain "reads only" rather than adding a per-call opt-in, because these failures already fall back to something the user can act on. If a transient gateway blip on first visit should not land people on sign-in, the alternative is a gatewayRetry: true opt-in on those two calls. It is easy to add and easy to remove.

⚖️ Alternatives

  • Set skipGatewayRetry on every write call site. Rejected: the replay stays the default, there are about 40 distinct write operations, and any new write that forgets the flag brings the bug back.
  • Opt reads in at call sites. Rejected: reads are most of the roughly 80 files that use an instance client, and every read that was not touched would silently lose its retry. The body already names the operation, so the shared client can classify it, as the restart gate does.
  • Remove the retry. Rejected: reads outside React Query (the route guard's user_info, the rolling-config probes, the restart probes) have no other retry.
  • Use the console's isBuiltInRead (naming plus the role-editor catalog). Not needed: only the console sends operations Studio did not write itself, and it opts out entirely. Pulling the catalog into integrations/api would also make the integration layer depend on a feature.
  • Server-side dedupe (idempotency keys). Neither Harper's operations API nor central manager's proxy has one.
  • Export the gate's isSafeToDelay as it was and call it from the retry. That is nearly what this does. The only additions are the non-string method guard and the POST-only rule, and neither changes a request Studio sends today.
  • Fix the caller copy here too. Kept in fix(errors): say a write's outcome is unknown when it got no verdict, not that it failed #1897, stacked on this branch, so this PR stays a transport change. Until fix(errors): say a write's outcome is unknown when it got no verdict, not that it failed #1897 lands, a write's gateway status reads as a plain failure. That is the copy that was already there.

The planning review (codex) returned Framing-Verdict: chosen-approach-sound.

🔧 Changes

Filed while tracing callers: #1889. The live deploy's fallback sends deploy_component again after a gateway status or a dropped connection, which is the same double send through fetch, outside this interceptor. #1890. A write's gateway status still reads as a plain failure, which invites a manual retry. This PR stops the automatic replays and leaves that copy as it was. Residual, not filed: a read's retry on a proxied or cookie client re-sends with whatever credentials are current after the 5s–35s wait. Direct-connect clients already refuse that through the connection-generation stamp. This PR narrows that exposure to reads; it does not change it.

✅ Verification

  • End to end, through the real interceptor chain: src/integrations/api/retryGatewayErrors.integration.test.ts builds a real getInstanceClient client whose adapter rejects with an AxiosError carrying 502/503/504, then counts adapter calls. A write (insert, drop_table, deploy_component, set_configuration, restart, create_authentication_tokens) is sent once. A read (describe_all) is sent 4 times, sleeping 5s/10s/20s. A recovering get_job resolves on its replay, a GET replays, and a read with skipGatewayRetry is sent once.
  • Fails on base: on aaca36e38 (feat(instance): add an Operations console for sending any operations-API request #1882's head) the same file fails 8 of 14 tests, all of them write cases, with the bug's signature: expected [ …(4) ] to deeply equal [ { method: 'post', … } ].
  • Mutation checks: dropping && isReadRequest(config) turns 13 tests red. Ignoring a serialized body turns 9 red, among them the end-to-end read replays and the restart gate's existing serialized-body test. Dropping the GET branch turns 4 red. Every mutation was reverted.
  • Unit tests: retryGatewayErrors.test.ts (the retry tests now use a serialized read; new write cases cover object, serialized, token-mint, non-JSON and missing bodies across all three statuses) and isReadOperation.test.ts (isReadRequest). Its non-POST write cases (a DELETE, PUT or PATCH with a read-named body, a read body with no method or a non-string one) all fail against the previous predicate (5 red) and pass now.
  • Gate: the pre-commit hook's full vitest run passed 402 files / 3,797 tests at 77ef3b02. npx tsc -b exit 0, npx oxlint --format stylish . exit 0, npx dprint check exit 0.
  • Not verified in a browser: a gateway status cannot be produced on demand against a live instance, so the adapter-level end-to-end test is the route.
  • Cross-model review: one full round, codex (graded) and gemini. Codex's one finding was the misleading failure copy, now A write's gateway status or timeout reads as a plain failure, though the write may have been applied #1890. Gemini's two were rejected on evidence. Its "backoff applies only to 503" blocker is contradicted by the passing end-to-end tests, which assert 5s/10s/20s for 502 and 504 too. Its "JSON.parse of a large failed write" cost runs only on a gateway status, and it is the same order of work as the JSON.stringify axios already did to send that body. The Cursor legs failed no-receipt (the 1Password SSH agent refused git fetch), and the Harper domain leg failed auth (expired OAuth session), so no adjudication ran in that round.
  • Second round, a full one at c2dc6593 (the POST-only rule): codex (graded), gemini and the Harper domain leg all ran, and the domain leg adjudicated. Nothing above minor survived. Its two minors are already on this page: the "Failed to import" copy (A write's gateway status or timeout reads as a plain failure, though the write may have been applied #1890, fixed by fix(errors): say a write's outcome is unknown when it got no verdict, not that it failed #1897, which is stacked on this branch) and token minting losing its retry (the second Your call above). It kept the JSON.parse cost as a disputed nit: the parse runs only after a gateway status or while the restart gate holds requests, a success never reaches it, and the suggested regex would match an "operation" key inside nested data. Its comment nit on the new isReadRequest JSDoc is fixed in 77ef3b02.

gemini-code-assist feedback

  • Read an operation only from a POST body: fixed in c2dc6593, with the non-POST test cases above.
  • Treat a missing method as GET: declined. axios sets a lowercase config.method, defaulting to 'get', before any interceptor runs (axios 1.20.0 lib/core/Axios.js:148-152), so no real request arrives without one. For a config built outside axios, reading it as GET would make an unknown request count as a read. The tests pin the opposite.

🤖 Generated with Claude Code (Claude Opus 5.5); posted via @dawsontoth.

Related PRs: #1882 overlaps (stack base — this PR builds on its isReadOperation and skipGatewayRetry and must land after it), #1897 overlaps (stacked on this branch; it extends the read list and owns the outcome-unknown copy for a write's gateway status)
Complexity: medium

Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; blocked=cursor-grok(no-receipt),cursor-composer(no-receipt); declined=cursor-kimi,cursor-muse; rounds=2; full=1 @ 77ef3b0

Review-Attention: read ~8m (sensitive: describeAuthFailure.ts; decisions: name-based-replay, first-5xx-final-for-every-write, post-only-operation, transport-before-callers, do-less-alternative) @ 77ef3b0

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the gateway retry mechanism to ensure that only read requests (GET requests or POST requests with read operations) are replayed, and integrates this unified check into the restart gate. The reviewer feedback suggests making the isReadRequest utility more robust by defaulting undefined methods to 'get' and restricting operation-based read checks specifically to 'post' requests, as well as adding corresponding test cases to verify these edge cases.

Comment on lines +32 to +34
export function isReadRequest({ method, data }: { method?: unknown; data?: unknown }): boolean {
return (typeof method === 'string' && method.toLowerCase() === 'get') || isReadOperation(operationOf(data));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of isReadRequest has two potential issues:

  1. If method is omitted/undefined, it will not default to 'get', even though Axios defaults to 'get' for requests. This would cause default GET requests to be incorrectly classified as writes.
  2. If the HTTP method is a write/mutation method other than POST (such as DELETE, PUT, or PATCH) but the request body happens to contain an operation field matching a read pattern, it will be incorrectly classified as a read request.

We can make this more robust by defaulting method to 'get' when it is undefined, and explicitly restricting the operation-based read check to POST requests.

Suggested change
export function isReadRequest({ method, data }: { method?: unknown; data?: unknown }): boolean {
return (typeof method === 'string' && method.toLowerCase() === 'get') || isReadOperation(operationOf(data));
}
export function isReadRequest({ method, data }: { method?: unknown; data?: unknown }): boolean {
const normalizedMethod = method === undefined ? 'get' : typeof method === 'string' ? method.toLowerCase() : '';
if (normalizedMethod === 'get') {
return true;
}
if (normalizedMethod === 'post') {
return isReadOperation(operationOf(data));
}
return false;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Point 2 fixed in c2dc659: only a POST body's operation counts now, so a DELETE, PUT or PATCH carrying a read-named field is a write. Harper takes operations only on POST / (server/operationsServer.ts:241), and central manager's proxy takes them only from a POST body (HDBInstance.js:292).

Point 1 declined. A missing method never reaches either caller: axios sets config.method to a lowercase string, defaulting to 'get', before any interceptor runs (axios 1.20.0 lib/core/Axios.js:148-152). Reading a missing method as GET would only change configs built outside axios, and there it would make an unknown request count as a read, against the predicate's rule that anything unrecognized is a write. The new tests pin the opposite: a read body with no method, or with a non-string method, counts as a write.

🤖 Generated by Claude Code (Claude Opus 5.5); posted via @dawsontoth.

Comment on lines +48 to +53
it.each([
['a GET', { method: 'get' }],
['an upper-case GET', { method: 'GET' }],
['a read body', { method: 'post', data: { operation: 'describe_all' } }],
['a read body axios already serialized', { method: 'post', data: JSON.stringify({ operation: 'get_job' }) }],
])('reads %s as a read', (_, config) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a test case to verify that isReadRequest correctly defaults to 'get' when the method property is omitted/undefined.

Suggested change
it.each([
['a GET', { method: 'get' }],
['an upper-case GET', { method: 'GET' }],
['a read body', { method: 'post', data: { operation: 'describe_all' } }],
['a read body axios already serialized', { method: 'post', data: JSON.stringify({ operation: 'get_job' }) }],
])('reads %s as a read', (_, config) => {
it.each([
['a GET', { method: 'get' }],
['an upper-case GET', { method: 'GET' }],
['an omitted method (defaults to GET)', {}],
['a read body', { method: 'post', data: { operation: 'describe_all' } }],
['a read body axios already serialized', { method: 'post', data: JSON.stringify({ operation: 'get_job' }) }],
])('reads %s as a read', (_, config) => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined. Axios always sets config.method before interceptors run (axios 1.20.0 lib/core/Axios.js:148-152), so no real request reaches isReadRequest without one. Where a config does lack it, treating it as a read would break the predicate's rule that anything unrecognized is a write. c2dc659 pins the opposite instead: a read body with no method is in the write cases.

🤖 Generated by Claude Code (Claude Opus 5.5); posted via @dawsontoth.

Comment thread src/integrations/api/isReadOperation.test.ts
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 77.37% 13288 / 17173
🔵 Statements 77.33% 14107 / 18242
🔵 Functions 72.24% 3499 / 4843
🔵 Branches 72.26% 10426 / 14428
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/features/auth/describeAuthFailure.ts 100% 100% 100% 100%
src/features/cluster/domains/Management.tsx 91.08% 87.69% 90% 92.78% 100, 127, 264-275, 283, 289-290
src/features/clusters/components/ClusterCard.tsx 54.02% 82.97% 25.92% 54.76% 107-119, 123-124, 127, 130-163, 267-296, 382, 457-496
src/features/instance/applications/components/Chat/tools/describeToolError.ts 100% 50% 100% 100%
src/features/instance/applications/components/Chat/tools/createApp/execute.ts 75% 50% 50% 75% 21-23, 75-91
src/features/instance/applications/components/Chat/tools/deleteTableRecords/execute.ts 0% 100% 0% 0% 12-29
src/features/instance/applications/components/Chat/tools/dropComponentFile/execute.ts 0% 100% 0% 0% 11-24
src/features/instance/applications/components/Chat/tools/insertTableRecords/execute.ts 0% 100% 0% 0% 12-29
src/features/instance/applications/components/Chat/tools/restartHTTPService/execute.ts 77.77% 100% 100% 77.77% 42-52
src/features/instance/applications/components/Chat/tools/setComponentFile/execute.ts 0% 100% 0% 0% 13-32
src/features/instance/applications/components/Chat/tools/updateTableRecords/execute.ts 0% 100% 0% 0% 12-29
src/features/instance/applications/components/NewApplication/useImportApplication.ts 91.78% 82.6% 100% 91.54% 112-116, 118-122, 137-140, 174-175
src/features/instance/applications/hooks/useRenameFiles.tsx 56.66% 35% 40% 55.93% 27, 36, 76, 88, 97-100, 111, 117-169
src/features/instance/applications/modals/RedeployApplicationModal.tsx 75.22% 51.94% 91.66% 74.77% 103, 126-141, 162-169, 173, 179, 197, 211-213, 225-234, 246, 334-344
src/features/instance/applications/modals/RenameFileModal.tsx 37.03% 14.28% 50% 36.53% 40, 47-48, 82-143, 148-149
src/features/instance/applications/modals/reportDeployHealth.ts 84.61% 73.58% 100% 84% 63-67, 72-91
src/features/instance/config/deployments/components/ActivateStagedRelease.tsx 100% 81.25% 100% 100%
src/features/instance/config/secrets/SecretGrantsEditor.tsx 0% 0% 0% 0% 30-78
src/features/instance/databases/components/DatabaseTableView.tsx 72.33% 73.84% 64.91% 71.95% 269, 402-405, 450-459, 636-650, 695-711, 715-723, 732-740, 746-774, 785, 809, 835-850, 857-861, 864-891, 907, 911, 941-942, 1083-1093
src/features/instance/operations/index.tsx 98.71% 88.15% 100% 98.7% 96
src/features/instance/secrets/SecretModals.tsx 88.88% 85.96% 92.3% 88.88% 130-132, 287-288, 308-309
src/features/organization/roles/modals/AddOrganizationRoleModal.tsx 65.62% 69.23% 50% 65.62% 73, 82, 92-98, 152, 170-198
src/features/organization/roles/modals/EditOrganizationRoleModal.tsx 45.61% 56.86% 33.33% 46.42% 43, 63-67, 102-118, 144, 151-187, 280-310
src/features/organization/settings/index.tsx 72.6% 66.66% 64.7% 74.64% 85-86, 126, 127, 139-143, 150-151, 156-157, 162-165, 204, 268-374, 409-410, 484-485
src/features/organization/users/modals/EditUserModal.tsx 0% 0% 0% 0% 36-172
src/hooks/useApplyLicensesClick.tsx 93.02% 84.84% 50% 92.85% 122, 133, 154
src/hooks/useRestartClusterClick.tsx 93.15% 96.15% 63.63% 92.85% 38, 84, 206, 217, 240
src/hooks/useRestartInstanceClick.ts 89.65% 73.07% 50% 89.65% 43, 84, 98
src/hooks/useRollingConfigUpdate.tsx 95.65% 98.61% 71.42% 95.5% 118, 260, 271, 295
src/integrations/api/isReadOperation.ts 100% 100% 100% 100%
src/integrations/api/retryGatewayErrors.ts 100% 100% 100% 100%
src/integrations/api/unansweredWrite.ts 100% 91.66% 100% 100%
src/integrations/api/instance/applications/packageComponent.ts 66.66% 100% 100% 66.66% 34
src/integrations/api/instance/secrets/secrets.ts 91.3% 89.47% 85% 90.9% 139-146
src/integrations/api/instance/status/restartInstance.ts 100% 100% 100% 100%
src/integrations/api/instance/status/restartUnconfirmed.ts 100% 96.29% 100% 100%
src/integrations/api/sse/resolveInstanceConnection.ts 100% 91.3% 100% 100%
src/lib/restart/restartGate.ts 97.56% 90.9% 100% 97.29% 94
src/react-query/queryClient.ts 98.18% 100% 85.71% 98.18% 121
Generated in workflow #2323 for commit 6cd6fbc by the Vitest Coverage Report Action

dawsontoth and others added 8 commits October 12, 2026 20:51
…sent twice

Every instance client replayed a 502/503/504 up to three times whatever
the request, and none of those statuses means the request went
unapplied: a proxy answers for an upstream that may already have it, and
Harper itself answers 503 after handing an operation to a worker that
exits or overruns. So an insert could store its records twice, a drop
could succeed and then report "does not exist", and a deploy or restart
could run again.

The retry now replays only what the restart gate already treats as a
read — a GET, or a body whose operation is a read by Harper's naming.
That predicate moves out of the gate into isReadOperation.ts as
isReadRequest, so the two share one rule. A write is sent once and its
first 5xx is final. skipGatewayRetry stays for the operations console,
whose component operations can carry a read-looking name.

Closes #1879

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
isReadRequest took the operation name from any request's body, so a DELETE, PUT or PATCH whose body happened to carry a read-named operation field counted as a read: the restart gate would hold it and the gateway retry would replay it. Harper takes an operation only from POST / (server/operationsServer.ts) and central manager's proxy forwards one only from a POST body (HDBInstance.js post), so on any other method that field is data and the request is a write. A missing or non-string method stays a write too; axios always sets a lowercase method before interceptors run (lib/core/Axios.js), so no real request reaches the predicate without one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cross-model round flagged the JSDoc's first sentence as restating the predicate's two branches. Keep only what the code cannot say: what the answer is used for, the fail-closed default, and why only a POST body names an operation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… not that it failed

A write answered 502/503/504, or not at all (a timeout, ERR_NETWORK), may
already have been applied, but Studio showed it as an ordinary failure: the
gateway's body or "Request failed with status code 504", and callers' own
"Failed to restart …" / "Failed to import …" on top. Retrying by hand
repeated the side effect.

isUnansweredWrite (new) recognizes it from the request (isReadRequest) and
the status, never the body. describeError checks it first and returns
"Outcome unknown" copy, so the global mutation toast and every describeError
caller get it. Callers that add their own failure toast to a mutation ask
shouldToastOwnFailure first; rename and the chat tools, which have no
mutation behind them, show the same copy; the cluster restart, rolling
config and license loops count these apart from failures. secrets.ts no
longer re-wraps such an error, so the predicate can still see it.
package_component is classified as the read it is, and skips the gateway
replay because each attempt rebuilds the archive.

Closes #1890

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e TXT record

Validating pending domains posts one request per domain and, on any failure,
said "Validation failed! Please make sure the TXT record has been put in
place." A validation central manager answered 502/503/504, or never
answered, may have succeeded, so it is now counted apart: the toast says the
outcome is unknown and the list is refetched to show what changed.

Also from review: isUnansweredWrite checks the status before it parses a
serialized body, so only an ambiguous failure pays for the parse; trims
comments that only restated their identifiers; guards a test teardown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…answers again

A restart answered 502/503/504, or not at all, may have run: central
manager's proxy answers 504 after forwarding it, and Harper can go down
mid-reply. The Restart button still released its restart mark at once on
that failure, so the tracker's refetches hit an instance going down and
their connectivity errors toasted beside "Outcome unknown". The cluster
loop also released the member and restarted the next one while it might
still be down, which defeats the one-at-a-time restart.

Both now hold the mark until waitUntilAnsweringAfterRestart sees the
target again, as the operations console already does: the button in the
background, after showing the warning; the loop before it moves on.

From the adjudicated cross-model review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sh after an unanswered deploy

The license loop's "Outcome unknown" toast dropped the licenses that were
refused outright, so a run with one of each left the refused one
unaccounted for. It now adds "N other licenses failed to apply", as the
domain validation toast does.

A redeploy or import that got no answer is told to check whether it took
effect, but nothing refreshed what the user would check. The redeploy
modal now refetches the components list on that failure, live progress or
not, and the import refetches the instance and reloads the file tree.

Also trims the isUnansweredWrite JSDoc to the part the code doesn't say.

From the adjudicated cross-model review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h gives up

The note read as though the mark held indefinitely and the cluster loop never
moved past a member that never answers. Both release when
waitUntilAnsweringAfterRestart gives up, as they do after an accepted restart.

From the adjudicated cross-model review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dawsontoth
dawsontoth force-pushed the claude/1879-safe-gateway-retry branch from 77ef3b0 to e7478e9 Compare October 13, 2026 00:52
dawsontoth and others added 11 commits October 12, 2026 21:23
…ailed

The rebase onto stage brought in a write surface this PR's caller audit
predates: activating a staged release (#1876's chain). Its mutation opts
out of the global toast, like import and redeploy since #1871, so an
activation that timed out or got a 504 showed "Failed to activate <app>"
with axios's own text and no warning that it may have gone through.

deployFailureTitle (new, beside the deploy reporters) is the one place a
deploy's failure title is decided: "Couldn't confirm <app> was activated"
for a write with no verdict, "Failed to activate <app>" otherwise. The
import form and the redeploy modal, which the rebase had given their own
copies of that rule, now use it too. reportDeployFailure shows the
unknown-outcome copy, and the activation's progress panel says so as
well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eat their neighbours

Review round after the rebase (codex nit, kept by the Harper domain
adjudicator): DeployVerb had no consumer outside reportDeployHealth.ts,
since every caller passes a literal. The redeploy modal's second refresh
comment repeated the first, and the skipGatewayRetry type doc listed
package_component, whose own call site already says why it opts out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at it failed

restartInstance sent the restart and then watched for the target with
user_info, and both steps rejected the same promise, so a target slow to
answer again read as a failed restart everywhere: "Failed to restart
instance.", "Only N of M instances restarted", and success: false to the
chat agent. Each of those invites a second restart of a target that
already went down once.

Harper's answer to the restart request is its only verdict: `restart`
answers before it tears down (bin/restart.ts schedules the teardown 50ms
out), and `restart_service` once its job has started (executeJob). So a
probe failure after a 2xx now rejects with RestartUnconfirmedError
(cause: the last probe error), shown as "Not back yet ... check its
status before you restart it again".

- The watch keeps going past the usual recovery time (a second probe
  schedule, no second settle) and calls onSlowToReturn once in between,
  so the Restart button and both loops say "still waiting" instead of
  giving up. Every restart mark's TTL comes from the same constants, and
  the probes skip the gateway replay, so no mark lapses mid-watch.
- The cluster restart and rolling config update stop at a member that
  never comes back instead of restarting the next one, which could leave
  no member serving, and report the members they left alone.
- createApp keeps success: true (the app exists) and restartHTTPService
  returns copy without the "Error:" prefix telling the agent not to
  restart again. Both now replace their progress toast on any failure,
  which used to keep spinning.

Closes #1895

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…members too

Review of the previous commit:

- A Cancel clicked while the loop waited on a member that never came back
  replaced the "Not back yet, check before you restart it again" toast
  with "Cancelled", hiding the one warning that matters. The uncertain
  outcome now wins; Cancelled still shows when nothing is uncertain.
- The uncertain-outcome toast counted the members that refused only
  implicitly. It now says "N instances failed to restart", from one
  helper both loops share (describeUncertainRollingOutcome).
- onSlowToReturn is queued rather than called inline, so a callback that
  throws cannot cut the watch short or become its cause.

Not adopted: counting a 401/403 probe as the target being back. Through
central manager's proxy a 403 is CM refusing the caller before it
contacts the instance (notAllowed() in HDBInstance.js), so it says
nothing about the target; recorded in DESIGN.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hat it never answered

Review round 2:

- A probe can fail with an answer (a 401/403, a Harper 500), so "hasn't
  answered since" was not always true. Every surface now says Studio
  couldn't confirm the target came back, which is what a failed watch
  proves. The watch itself still counts only a successful user_info as
  back: through central manager's proxy a 403 is CM refusing the caller
  (notAllowed() in HDBInstance.js) before it contacts the instance.
- A throwing onSlowToReturn is caught and logged instead of escaping as
  an uncaught microtask exception; the watch carries on either way.
- RollingOutcome is no longer exported; it has no consumer outside its
  module.

Not adopted: dismissing the button's loading toast for an unconfirmed
restart. It already is: the catch dismisses it before deciding whether to
add its own toast. A test now pins that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ever answered

Review round 3: DESIGN.md and restartInstance's doc comment still said the
target "never answers", which contradicts the rule that only a successful
user_info counts as back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efused config-update members

Mutation checks after the rebase onto the new restart loops found three
guards no test held: the restart request's explicit 60s timeout (the
client default is also 60s, so it only shows with that default cleared),
RESTART_MARK_TTL_MS on the Restart button and both cluster-loop marks
(restoring the old five minutes stayed green, since sleep is mocked), and
the rolling config update's failed count.

- restartInstance: the request's bound, every probe's timeout and every
  wait add up to less than RESTART_MARK_TTL_MS.
- The Restart button and the cluster loop still hold their marks six
  minutes in, past the old TTL, while the watch runs.
- A config update names a member that refused alongside the one not back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fig fan-out

After the rebase onto #1861's rolling config update, the stop rule also
decides the fan-out: a canary that is not back never proved it can run
the new configuration, so it is never fanned out from, and a stop after
the fan-out keeps the note that the members left alone may already have
the configuration saved. Both are pinned in useRollingConfigUpdate.test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay what Cancel left

Review round 1 after the port (codex major, kept by the Harper domain
adjudicator): the cluster loop waits on a member whose restart got no
verdict, but when that watch also ran out it discarded the rejection and
restarted the next member. On a two-member cluster, a 504 after the
restart ran plus a member that stays down left no member serving, which
is exactly what the stop rule exists to prevent. The loop now stops
there too, and the report says Studio couldn't confirm that member came
back either, instead of claiming it accepted the restart.

The adjudicator's nit: a Cancel after a no-verdict member now yields the
"Outcome unknown" report, which never said Cancel skipped the rest. Both
loops now name the members Cancel left alone in that report.

Also from the round: drop the test comments that only narrated their
setup, and trim restartInstance's JSDoc to its contract (DESIGN.md
carries the reasoning).

Not taken: Cancel during set_configuration still restarts that member;
that check is #1867's, unchanged here. The watch outliving the progress
toasts' durations is not a problem: sonner never starts an auto-close
timer for a loading toast (sonner 2.0.8, dist/index.mjs:644).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onor Cancel mid-write

Review round 2 after the port (codex major twice; the Harper domain
adjudicator kept both as minor and said the base stack could take them):

- An unanswered restart_service moved on to the next member after 3s.
  If the restart ran onto a configuration that broke the member, the
  next member got the same configuration. The config update now waits on
  that member, as the cluster restart does, and stops there if the watch
  gives up too. An unanswered set_configuration does not wait: nothing
  restarted that member.
- A Cancel clicked while a member's set_configuration was in flight still
  restarted it. The loop now stops before that restart, and the Cancelled
  report says the configuration may be pending there, as after a fan-out.
- After a Cancel, the "Waiting On Instance" toast that onSlowToReturn
  brings back offered a second Cancel, as if the first had not landed.
  Both loops now show it without the action and say no other instance
  will be restarted or updated.

Also drops the test helper comments the round flagged as narration.

Rejected on evidence: gemini's "axiosRetry resends the restart POST"
(the POST is sent once; axiosRetry wraps only the probes, and #1891's
replay never repeats a write), and "a Cancel aborts the watch into
RestartUnconfirmedError" (Cancel only sets a flag; no signal reaches the
watch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l was clicked

Review round 3 (codex minor, kept by the Harper domain adjudicator):
round 2 took the Cancel action off the slow-return toast, but the
no-verdict follow-up toast round 2 added to the config update, the
cluster loop's one, and both loops' "carry on momentarily" toasts still
came back with a Cancel button after Cancel had closed the toast, and
promised to carry on for a watch of up to about seven minutes.

One helper per loop now builds every progress toast after the first: the
usual Cancel action and copy, or, once Cancel was clicked, neither, and a
note that no other instance will be restarted or updated. Also drops a
comment that only restated its flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dawsontoth
dawsontoth marked this pull request as ready for review October 13, 2026 02:49
@dawsontoth
dawsontoth requested a review from a team as a code owner October 13, 2026 02:49
@dawsontoth
dawsontoth merged commit 6cd6fbc into stage Oct 13, 2026
8 checks passed
@dawsontoth
dawsontoth deleted the claude/1879-safe-gateway-retry branch October 13, 2026 02:49
dawsontoth added a commit that referenced this pull request Oct 13, 2026
…onor Cancel mid-write

Review round 2 after the port (codex major twice; the Harper domain
adjudicator kept both as minor and said the base stack could take them):

- An unanswered restart_service moved on to the next member after 3s.
  If the restart ran onto a configuration that broke the member, the
  next member got the same configuration. The config update now waits on
  that member, as the cluster restart does, and stops there if the watch
  gives up too. An unanswered set_configuration does not wait: nothing
  restarted that member.
- A Cancel clicked while a member's set_configuration was in flight still
  restarted it. The loop now stops before that restart, and the Cancelled
  report says the configuration may be pending there, as after a fan-out.
- After a Cancel, the "Waiting On Instance" toast that onSlowToReturn
  brings back offered a second Cancel, as if the first had not landed.
  Both loops now show it without the action and say no other instance
  will be restarted or updated.

Also drops the test helper comments the round flagged as narration.

Rejected on evidence: gemini's "axiosRetry resends the restart POST"
(the POST is sent once; axiosRetry wraps only the probes, and #1891's
replay never repeats a write), and "a Cancel aborts the watch into
RestartUnconfirmedError" (Cancel only sets a flag; no signal reaches the
watch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Instance clients replay a 502/503/504 for every operation, so a write the gateway lost the answer to runs again

1 participant