fix(errors): say a write's outcome is unknown when it got no verdict, not that it failed

DESIGN.md

@@ -66,6 +66,25 @@ The wide hero artwork spans both columns beneath the form card in three independ
 
 Every `getInstanceClient` client replays a 502/503/504 up to three times (5s/10s/20s), but only for a request [`isReadRequest`](src/integrations/api/isReadOperation.ts) accepts: a GET, or a POST whose body's `operation` is a read by Harper's naming — the restart gate's rule, one predicate for both ([`retryGatewayErrors.ts`](src/integrations/api/retryGatewayErrors.ts)). A write is sent once and its first 5xx is final, because no gateway status proves the write was not applied: Harper itself answers 503 after handing an operation to a worker that exits or overruns. Do not add an exemption for a "usually safe" status or a write that looks idempotent; a caller whose write is safe to repeat retries it itself, where it can say why.
 
+## A write with no verdict never reads as failed
+
+The same rule decides what a failure says. [`isUnansweredWrite`](src/integrations/api/unansweredWrite.ts) is a request `isReadRequest` rejects, answered 502/503/504 or not at all (`ERR_NETWORK`, a timeout). [`describeError`](src/react-query/queryClient.ts) checks it before reading any body and returns "Outcome unknown … check whether it took effect before you try again", so the global mutation toast and every `describeError` caller say so with no copy of their own. Gate on status, never body (a gateway's body describes the gateway), and keep it to display: rewriting `error.message` would defeat `shouldKeepEvent`, which recognises timeouts and `Network Error` by message.
+
+- **A caller that adds its own failure toast to a mutation** asks [`shouldToastOwnFailure`](src/react-query/queryClient.ts) first, so "Failed to …" never sits beside the global warning. A caller with no mutation behind it (rename, the chat tools), or one whose mutation opts out of the global toast (the deploy mutation behind import, redeploy and staged-release activation), shows `describeError`'s text itself, under a title from [`deployFailureTitle`](src/features/instance/applications/modals/reportDeployHealth.ts) for a deploy ("Couldn't confirm … was redeployed", never "Failed to …"), and a loop that tallies one write per target (cluster restart, rolling config, licenses) counts these apart from failures.
+- **Do not re-wrap an AxiosError into a plain `Error`** for a write: the predicate needs its `config`. `secrets.ts` passes an unanswered one through as is.
+- **A restart with no verdict stays marked until its target answers again or the watch gives up**, as an accepted one does, because a gateway can lose the answer to a restart that ran. The Restart button holds the mark while it watches, and the cluster loop waits on that member before restarting the next.
+- **These toasts are never collapsed** onto a shared id: each warns about a different write, and a read timing out alongside one must not replace it through `request-timeout`.
+- **A read that is costly to repeat** stays a read here and opts out of the replay instead (`package_component`, `skipGatewayRetry`).
+
+## A restart Harper accepted never reads as failed
+
+Harper's answer to the restart request is its only verdict on the restart: `restart` answers before it tears down (harper `bin/restart.ts` schedules the teardown 50ms out so the reply can go first) and `restart_service` once its job has started (`executeJob`). So [`restartInstance`](src/integrations/api/instance/status/restartInstance.ts) rejects with the request's own error only when that request failed; after a 2xx, a target Studio never sees answer the `user_info` watch successfully rejects with [`RestartUnconfirmedError`](src/integrations/api/instance/status/restartUnconfirmed.ts), which reads "Not back yet", never "Failed", and is never a reason to send the restart again.
+
+- **The watch outlasts the usual recovery time** (one settle, then two probe schedules), and `onSlowToReturn` fires once between them so a caller can say it is still waiting. Every restart mark uses `RESTART_MARK_TTL_MS`, derived from the same constants, so no mark lapses while its watch still runs; the probes skip the gateway replay for the same reason.
+- **Only a successful probe counts as back, never a 401/403.** Through central manager's operations proxy a 403 is CM refusing the caller before it contacts the instance (`notAllowed()` in its `HDBInstance.js`), so it says nothing about the target; reading it as "back" would report a restart Studio never saw finish. That is also why the copy says Studio "couldn't confirm it came back", not that the target never answered.
+- **A one-at-a-time loop stops at a member that is not back** (cluster restart, rolling config update): restarting the next one could leave no member serving. Both loops also wait on a member whose restart got no verdict, and stop if that follow-up watch gives up too: it may have restarted and still be down, or, in a config update, restarted onto a configuration that broke it. Either report names the members it left alone, and it, like the no-verdict warning, outranks Cancel, which would otherwise hide the warning not to restart again; the report then says which members Cancel left. A Cancel clicked during a member's configuration write stops before its restart, and the report says the configuration may be pending there. On a 5.2+ cluster the config update never fans out from a canary that is not back, since it never proved it can run the configuration; a stop after the fan-out keeps the note that the members it left may already have the new configuration saved.
+- **The chat tools return `success: true`** for it: their action was carried out, and `false` invites the agent to repeat it.
+
 ## Cluster upgrade picker — which versions an existing cluster is offered
 
 [`buildUpgradeVersionOptions`](src/features/clusters/upsert/lib/buildUpgradeVersionOptions.ts) is the one place that decides, and its test file is the spec. Two rules, and they differ on purpose:

src/features/auth/describeAuthFailure.ts

@@ -1,3 +1,4 @@
+import { NO_RESPONSE_CODES } from '@/integrations/api/unansweredWrite';
 import { describeError } from '@/react-query/queryClient';
 import { AxiosError } from 'axios';
 
@@ -24,15 +25,6 @@ const TRANSIENT_STATUSES: ReadonlySet<number> = new Set([503]);
 // safe once you know it did not). Repeating blindly turns a completed sign-up into a 409 with the
 // verification mail already sent (#1668).
 const OUTCOME_UNKNOWN_STATUSES: ReadonlySet<number> = new Set([502, 504]);
-// Axios reports a request that got no response as ERR_NETWORK — including a CORS rejection, and
-// including a connection that dropped *after* the POST was applied — so it cannot be described as
-// "we couldn't reach the server". It is the same unknown outcome. ECONNABORTED/ETIMEDOUT are the
-// timeout pair; which one axios uses depends on `transitional.clarifyTimeoutError`.
-const OUTCOME_UNKNOWN_CODES: ReadonlySet<string> = new Set([
-	AxiosError.ERR_NETWORK,
-	AxiosError.ETIMEDOUT,
-	AxiosError.ECONNABORTED,
-]);
 
 /**
  * Our own copy for a failure the server's own words shouldn't answer, or `undefined` to defer.
@@ -64,7 +56,7 @@ export function describeRetryableAuthFailure(rawErr: unknown, outcomeUnknownReco
 	// ERR_CANCELED is a local abort that reached nothing, and a config fault (ERR_INVALID_URL,
 	// ERR_BAD_OPTION) never left the browser — none of those is ours to describe.
 	const code = axiosErr?.code;
-	return typeof code === 'string' && OUTCOME_UNKNOWN_CODES.has(code) ? outcomeUnknown : undefined;
+	return typeof code === 'string' && NO_RESPONSE_CODES.has(code) ? outcomeUnknown : undefined;
 }
 
 /** Display text for an auth-form rejection, preferring whatever the server said. */

src/features/cluster/domains/Management.test.tsx

@@ -3,7 +3,7 @@
  */
 import { MutationCache, onlineManager, QueryCache, QueryClient, QueryClientProvider } from '@tanstack/react-query';
 import { act, fireEvent, render, waitFor, within } from '@testing-library/react';
-import { AxiosError } from 'axios';
+import { AxiosError, type InternalAxiosRequestConfig } from 'axios';
 import { PropsWithChildren, Suspense } from 'react';
 import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
 
@@ -30,6 +30,7 @@ vi.mock('@/hooks/usePermissions', () => ({
 const { instancePost } = vi.hoisted(() => ({ instancePost: vi.fn() }));
 vi.mock('@/config/getInstanceClient', () => ({ getInstanceClient: () => ({ post: instancePost }) }));
 
+import { describeUnansweredWrites, OUTCOME_UNKNOWN_TITLE } from '@/integrations/api/unansweredWrite';
 import { mutationErrorHandler, queryErrorHandler } from '@/react-query/queryClient';
 import { toast } from 'sonner';
 
@@ -1121,3 +1122,49 @@ describe('DomainsManagement — the Remove Domain tooltip', () => {
 		expect(pendingCell.querySelectorAll('[tabindex]')).toHaveLength(0);
 	});
 });
+
+describe('validating pending domains', () => {
+	beforeEach(() => {
+		vi.clearAllMocks();
+	});
+
+	it('says a validation that got no answer may have gone through, and refreshes the list to show it', async () => {
+		get.mockImplementation((url: string) =>
+			url.startsWith('/Domain/')
+				? Promise.resolve({
+					data: [
+						{ id: 'dom-1', domain: 'a.example.com', status: 'PENDING_VALIDATION' },
+						{ id: 'dom-2', domain: 'b.example.com', status: 'PENDING_VALIDATION' },
+					],
+				})
+				: Promise.resolve({ data: { id: 'clu-1', organizationId: 'org-1', status: 'RUNNING', domainIds: [] } })
+		);
+		post.mockImplementation((url: string) => {
+			const config = { method: 'post', url } as InternalAxiosRequestConfig;
+			return Promise.reject(
+				url.includes('dom-1')
+					? new AxiosError('timeout of 60000ms exceeded', AxiosError.ECONNABORTED, config)
+					: new AxiosError('Request failed with status code 400', AxiosError.ERR_BAD_REQUEST, config, undefined, {
+						status: 400,
+						statusText: '',
+						data: 'TXT record not found',
+						headers: {},
+						config,
+					}),
+			);
+		});
+		const { container } = renderManagement();
+		const validateButton = () => container.querySelector<HTMLButtonElement>('button[accesskey="r"]');
+		await waitFor(() => expect(validateButton()?.disabled).toBe(false));
+		const fetchesBefore = domainListFetches();
+
+		fireEvent.click(validateButton()!);
+
+		await waitFor(() => expect(toast.error).toHaveBeenCalled());
+		expect(toast.error).toHaveBeenCalledWith(OUTCOME_UNKNOWN_TITLE, {
+			description: `${describeUnansweredWrites(1)} 1 other domain failed to validate.`,
+			id: 'validatingDomains',
+		});
+		expect(domainListFetches()).toBeGreaterThan(fetchesBefore);
+	});
+});

src/features/cluster/domains/Management.tsx

@@ -40,6 +40,7 @@ import { getOrganizationDomainsQueryOptions } from '@/features/organization/quer
 import { useOrganizationRolePermissions } from '@/hooks/usePermissions';
 import { useRefreshClick } from '@/hooks/useRefreshClick';
 import { SchemaOrganizationDomain } from '@/integrations/api/api.patch';
+import { describeUnansweredWrites, isUnansweredWrite, OUTCOME_UNKNOWN_TITLE } from '@/integrations/api/unansweredWrite';
 import { pluralize } from '@/lib/pluralize';
 import { describeError } from '@/react-query/queryClient';
 import { zodResolver } from '@hookform/resolvers/zod';
@@ -235,18 +236,32 @@ export function DomainsManagement() {
 		const id = 'validatingDomains';
 		let checked = 0;
 		let failed = 0;
+		let unanswered = 0;
 		for (const pendingDomain of pendingDomains) {
 			try {
 				toast.loading(message, {
 					description: `${checked++} of ${pendingDomains.length} checked`,
 					id,
 				});
 				await validateDomainInOrganization(pendingDomain.id);
-			} catch {
-				failed += 1;
+			} catch (error) {
+				if (isUnansweredWrite(error)) {
+					unanswered += 1;
+				} else {
+					failed += 1;
+				}
 			}
 		}
-		if (failed > 0) {
+		if (unanswered > 0) {
+			await refetch();
+			toast.error(OUTCOME_UNKNOWN_TITLE, {
+				description: [
+					describeUnansweredWrites(unanswered),
+					failed > 0 && `${pluralize(failed, 'other domain', 'other domains')} failed to validate.`,
+				].filter(Boolean).join(' '),
+				id,
+			});
+		} else if (failed > 0) {
 			toast.error('Validation failed!', {
 				description:
 					`Please make sure the TXT record has been put in place. You may need to wait a bit for the DNS change to propagate.`,

src/features/clusters/components/ClusterCard.tsx

@@ -28,6 +28,7 @@ import { excludeFalsy } from '@/lib/arrays/excludeFalsy';
 import { LocalStorageKeys } from '@/lib/storage/localStorageKeys';
 import { buildAbsoluteLinkToPage } from '@/lib/urls/buildAbsoluteLinkToPage';
 import { getOperationsUrlForCluster } from '@/lib/urls/getOperationsUrlForCluster';
+import { shouldToastOwnFailure } from '@/react-query/queryClient';
 import { useQueryClient } from '@tanstack/react-query';
 import { Link, useRouter } from '@tanstack/react-router';
 import {
@@ -146,17 +147,19 @@ export function ClusterCard({ item: summary }: { item: ClusterListItem }) {
 				});
 				setIsTerminateClusterModalOpen(false);
 			},
-			onError: () => {
-				toast.error('Error', {
-					description: isSelfManaged
-						? `Failed to remove cluster: ${cluster.name}`
-						: `Failed to terminate cluster: ${cluster.name}.`,
-					duration: 5000,
-					action: {
-						label: 'Dismiss',
-						onClick: () => toast.dismiss(),
-					},
-				});
+			onError: (error) => {
+				if (shouldToastOwnFailure(error)) {
+					toast.error('Error', {
+						description: isSelfManaged
+							? `Failed to remove cluster: ${cluster.name}`
+							: `Failed to terminate cluster: ${cluster.name}.`,
+						duration: 5000,
+						action: {
+							label: 'Dismiss',
+							onClick: () => toast.dismiss(),
+						},
+					});
+				}
 				setIsTerminateClusterModalOpen(false);
 			},
 		});

src/features/instance/applications/components/Chat/tools/createApp/execute.ts

@@ -1,13 +1,15 @@
 import { templates } from '@/features/instance/applications/components/NewApplication/templates';
 import { addComponent } from '@/integrations/api/instance/applications/addComponent';
 import { restartInstance } from '@/integrations/api/instance/status/restartInstance';
+import { RestartUnconfirmedError } from '@/integrations/api/instance/status/restartUnconfirmed';
 import { findBy } from '@/lib/arrays/findBy';
 import { emitToListeners } from '@/lib/events/listener';
 import { toKebabCase } from '@/lib/string/to-kebab-case';
 import { inputSchema } from '@harperfast/agent-tools/tools/createApp/inputSchema';
 import { toast } from 'sonner';
 import { z } from 'zod';
 import { ExecuteParams } from '../../types/executeParams';
+import { describeToolError, toastToolFailure } from '../describeToolError';
 import { Output } from './output';
 
 export async function execute(
@@ -37,9 +39,10 @@ export async function execute(
 			template: template.npm || template.githubUrl,
 		});
 	} catch (err) {
+		toastToolFailure(toastId, err);
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}
 
@@ -58,9 +61,20 @@ export async function execute(
 			replicated: instanceClientParams.entityType === 'cluster',
 		});
 	} catch (err) {
+		toastToolFailure(toastId, err);
+		if (err instanceof RestartUnconfirmedError) {
+			// The app exists either way; `success: false` would invite creating it again.
+			return {
+				success: true,
+				message: `App "${name}" created. Harper accepted the restart that loads it, but Studio couldn't confirm `
+					+ "the instance came back, so it may still be starting. Don't create the app or restart again until "
+					+ 'its status shows it is back.',
+				webURL: baseURL,
+			};
+		}
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}
 

src/features/instance/applications/components/Chat/tools/deleteTableRecords/execute.ts

@@ -3,6 +3,7 @@ import { queryClient } from '@/react-query/queryClient';
 import { inputSchema } from '@harperfast/agent-tools/tools/deleteTableRecords/inputSchema';
 import { z } from 'zod';
 import { ExecuteParams } from '../../types/executeParams';
+import { describeToolError } from '../describeToolError';
 import { Output } from './output';
 
 export async function execute(
@@ -24,7 +25,7 @@ export async function execute(
 	} catch (err) {
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}
 }

src/features/instance/applications/components/Chat/tools/describeToolError.ts

@@ -0,0 +1,13 @@
+import { isUnansweredWrite, UNANSWERED_WRITE_MESSAGE } from '@/integrations/api/unansweredWrite';
+import { describeError } from '@/react-query/queryClient';
+import { toast } from 'sonner';
+
+export function describeToolError(err: unknown): string {
+	return isUnansweredWrite(err) ? UNANSWERED_WRITE_MESSAGE : String(err);
+}
+
+/** A loading toast never dismisses itself, so a failed tool must replace its progress toast. */
+export function toastToolFailure(toastId: string | number, err: unknown) {
+	const { title, description } = describeError(err);
+	toast.error(title, { id: toastId, description, duration: 10_000 });
+}

src/features/instance/applications/components/Chat/tools/dropComponentFile/execute.ts

@@ -2,6 +2,7 @@ import { dropComponent } from '@/integrations/api/instance/applications/dropComp
 import { inputSchema } from '@harperfast/agent-tools/tools/dropComponentFile/inputSchema';
 import { z } from 'zod';
 import { ExecuteParams } from '../../types/executeParams';
+import { describeToolError } from '../describeToolError';
 import { Output } from './output';
 
 export async function execute(
@@ -19,7 +20,7 @@ export async function execute(
 	} catch (err) {
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}
 }

src/features/instance/applications/components/Chat/tools/insertTableRecords/execute.ts

@@ -3,6 +3,7 @@ import { queryClient } from '@/react-query/queryClient';
 import { inputSchema } from '@harperfast/agent-tools/tools/insertTableRecords/inputSchema';
 import { z } from 'zod';
 import { ExecuteParams } from '../../types/executeParams';
+import { describeToolError } from '../describeToolError';
 import { Output } from './output';
 
 export async function execute(
@@ -24,7 +25,7 @@ export async function execute(
 	} catch (err) {
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}
 }

src/features/instance/applications/components/Chat/tools/restartHTTPService/execute.ts

@@ -1,8 +1,10 @@
 import { restartInstance } from '@/integrations/api/instance/status/restartInstance';
+import { RestartUnconfirmedError } from '@/integrations/api/instance/status/restartUnconfirmed';
 import { inputSchema } from '@harperfast/agent-tools/tools/restartHTTPService/inputSchema';
 import { toast } from 'sonner';
 import { z } from 'zod';
 import { ExecuteParams } from '../../types/executeParams';
+import { describeToolError, toastToolFailure } from '../describeToolError';
 import { Output } from './output';
 
 export async function execute(
@@ -20,9 +22,20 @@ export async function execute(
 			replicated: instanceClientParams.entityType === 'cluster',
 		});
 	} catch (err) {
+		toastToolFailure(toastId, err);
+		if (err instanceof RestartUnconfirmedError) {
+			// `success` is whether the tool's own action was carried out, and Harper accepted the restart.
+			return {
+				success: true,
+				message: "Harper accepted the restart, but Studio couldn't confirm the HTTP service came back, so it may "
+					+ "still be starting. Don't restart it again until its status shows it is back, or it goes down a "
+					+ 'second time.',
+				webURL: baseURL,
+			};
+		}
 		return {
 			success: false,
-			message: `Error: ${err}`,
+			message: `Error: ${describeToolError(err)}`,
 		};
 	}