feat(deploy): follow a rolling deploy's restart job, and show canary certification

DESIGN.md

@@ -12,6 +12,7 @@ This file collects architecture and design notes for studio that aren't otherwis
 
 - [`src/features/instance/databases/DESIGN.md`](src/features/instance/databases/DESIGN.md) — when the browse grid may show its spinner, and why a table without a primary key can't be listed.
 
+- [`src/features/instance/applications/DESIGN.md`](src/features/instance/applications/DESIGN.md) — when a rolling deploy is done: following its restart job, and why a job Studio can't follow is never a success; and when a live deploy may be sent again.
 - [`src/features/instance/config/roles/DESIGN.md`](src/features/instance/config/roles/DESIGN.md) — why the role editor's JSON text is the only source of truth, what forces the JSON view, and what Save changes on purpose.
 - [`src/router/DESIGN.md`](src/router/DESIGN.md) — why a failed route load is reported and shown only by its route, and why a preload still toasts.
 

src/features/instance/applications/DESIGN.md

@@ -0,0 +1,57 @@
+# Applications — design notes
+
+## When a deploy is done
+
+Studio deploys with `restart: 'rolling'`. The node that receives the deploy answers once its own part is done and hands
+the other nodes to a `restart_service` job, named in the response's `restartJobId` (Harper 4.6+). The deployment record
+does not help: only the receiving node writes it, so it reads `success` while that job runs and after it fails.
+
+So [`deployAndFollowComponent`](../../../integrations/api/instance/applications/deployComponent.ts) follows the job with
+`get_job` before the mutation settles ([`restartJob.ts`](../../../integrations/api/instance/applications/restartJob.ts)):
+
+- **A failed job rejects with `DeployRolloutError`.** The release is live on the receiving node; the error and the
+  progress panel name each node that did not take it. Its message quotes those nodes' errors, so Datadog withholds it
+  like `SSEOperationError`.
+- **A job Studio could not follow resolves, as `rollout.state === 'unfollowable'`, and is never reported as success.**
+  That covers a job that ended with a result in neither shape below. [`reportDeployOutcome`](modals/reportDeployHealth.ts)
+  skips the post-deploy health check for it, because that check reads the one node Studio reaches, which can be healthy
+  on the release a rejecting node kept; `reportDeployResult` is the same policy for a caller that does not health-check.
+  Any new caller of the deploy mutation reports through one of the two.
+- **The job may not be there.** `hdb_job` is a non-replicating system table, so only the receiving node can answer for
+  it. Central manager's cluster proxy sends every operation to the first running instance, so over the proxy the deploy
+  and its polls reach the same node; a direct connection to a cluster's address can reach another node, which answers
+  `[]`. A row missing for 20 seconds counts as `not-found`.
+- **Every node's outcome arrives at once.** Harper writes the job's row when it starts and when it ends, never in between.
+
+The job comes in two shapes. Harper 5.4's certified rollout activates the staged release on each other node, and each
+certifies it with a canary of its own: `result.activated` on success, `message: { error, activated }` on failure. Older
+versions, and a 5.4 deploy whose release was not armed for a canary, run the plain restart: `result.replicated`, where a
+node the job could not reach is listed with its error and the job still completes, so that node reads `unconfirmed`.
+
+Canary certification is shown only when the response carries it, which only 5.4+ does. There is no version check on
+the deploy path: `registration_info` is denied to a role with an operations allowlist, which is exactly the kind of
+role that may be allowed to deploy, so a version gate there would hide the result from it.
+
+## Staging and activation are gated the other way
+
+Stage only (`activate: false`) and Activate (`deployment_id`) are offered only when `registration_info` reads 5.3.0 or
+later (`useStagedDeploymentsAvailable`), and an unreadable version offers neither. The difference from certification is
+what happens when the guess is wrong: a node older than 5.3.0 accepts `activate: false` as an unknown field and makes
+the release live, which is the opposite of what the person asked for. Certification only adds a field to a response.
+
+All three kinds of request are built in one place, [`deployRequestBody`](../../../integrations/api/instance/applications/deployComponent.ts),
+because Harper's validator refuses the wrong mix outright: a stage with `restart`, or an activation carrying `package`,
+install options or `activate`. A stage succeeds only once every other node confirmed staging it, so a stage restarts
+nothing and is reported without a health check. Rollback to a kept release (#1232) is the same activation call.
+
+## A live deploy is sent once
+
+A live (SSE) deploy that gets no event stream goes out again as the buffered POST only when its status proves Harper
+never started it: a 4xx. Harper refuses a request before the operation starts (authentication, permissions,
+validation) with a 4xx, and answers one it started with a 200 event stream at once. Everything else can follow a deploy
+that is already running: a gateway's 5xx after forwarding, a 2xx that is not a stream, a dropped connection, the idle
+watchdog firing before headers. Harper's own request-queue 503 never started anything either, but status alone cannot
+tell it from a gateway's. So [`onDeployComponentSubmit`](../../../integrations/api/instance/applications/deployComponent.ts)
+turns those into `SSEInconclusiveError`, which every deploy caller already answers by checking rather than resending.
+`streamOperation` stays neutral: it puts the `status` on `SSEUnsupportedError` and leaves the decision to the caller,
+because a read (`get_deployment`) may fall back on anything.

src/features/instance/applications/components/ContentActions.tsx

@@ -16,10 +16,16 @@ import { useEditorFileContent } from '@/features/instance/applications/context/e
 import { isDirectory } from '@/features/instance/applications/context/isDirectory';
 import { useEditorView } from '@/features/instance/applications/hooks/useEditorView';
 import { useEntryActions } from '@/features/instance/applications/hooks/useEntryActions';
-import { useInstanceBrowseManagePermission } from '@/hooks/usePermissions';
+import {
+	useDeploymentsAvailable,
+	useStagedDeploymentsAvailable,
+} from '@/features/instance/applications/hooks/useSupportsDeploymentSSE';
+import { useInstanceBrowseManagePermission, useInstanceManagePermission } from '@/hooks/usePermissions';
 import { useRestartInstanceClick } from '@/hooks/useRestartInstanceClick';
 import { emitToListeners, useEmitToListeners } from '@/lib/events/listener';
 import { useSetWatchedValue, useWatchedValue } from '@/lib/events/watcher';
+import { buildAbsoluteLinkToPage } from '@/lib/urls/buildAbsoluteLinkToPage';
+import { useNavigate, useParams } from '@tanstack/react-router';
 import {
 	ArrowLeftIcon,
 	ArrowRightIcon,
@@ -28,12 +34,14 @@ import {
 	DownloadIcon,
 	FilePlusIcon,
 	FolderPlusIcon,
+	HistoryIcon,
 	LockIcon,
 	PackageIcon,
 	PanelRightCloseIcon,
 	PanelRightOpenIcon,
 	PencilIcon,
 	PlusIcon,
+	RocketIcon,
 	RotateCcwIcon,
 	SaveIcon,
 	TrashIcon,
@@ -73,6 +81,14 @@ export function ContentActions({
 	const { openedEntryContents, openedEntry, isSavingFile, restrictPackageModification } = useEditorView();
 	const { content: updatedFileContent } = useEditorFileContent(openedEntry?.path);
 	const canManageBrowseInstance = useInstanceBrowseManagePermission();
+	const canManageInstance = useInstanceManagePermission();
+	const deploymentsAvailable = useDeploymentsAvailable();
+	const stagingAvailable = useStagedDeploymentsAvailable();
+	const canViewDeployments = canManageInstance && deploymentsAvailable;
+	const navigate = useNavigate();
+	const routeParams: { clusterId?: string; instanceId?: string; organizationId?: string } = useParams({
+		strict: false,
+	});
 	const { onRestartClick, isRestartPending } = useRestartInstanceClick({
 		operation: 'restart_service',
 		instanceClient: instanceParams.instanceClient,
@@ -133,7 +149,8 @@ export function ContentActions({
 		useEntryActions(openedEntry);
 
 	const showFileMenu = canEditFile || canRename || canAddEntries || canAddTable || canDeleteEntry;
-	const showApplicationMenu = canDownload || canManageBrowseInstance || canRedeploy;
+	const deploymentsProject = canViewDeployments && openedEntry?.project ? openedEntry.project : undefined;
+	const showApplicationMenu = canDownload || canManageBrowseInstance || canRedeploy || !!deploymentsProject;
 	// The "File" menu acts on whatever is selected, so name it for that thing.
 	const fileMenuLabel = openedEntry && isDirectory(openedEntry) ? 'Directory' : 'File';
 
@@ -389,6 +406,30 @@ export function ContentActions({
 										Redeploy Package
 									</DropdownMenuItem>
 								)}
+								{deploymentsProject && (
+									<DropdownMenuItem
+										onSelect={() =>
+											void navigate({
+												to: buildAbsoluteLinkToPage(routeParams, 'config/deployments'),
+												search: { project: deploymentsProject },
+											})}
+									>
+										<HistoryIcon />
+										View Deployments
+									</DropdownMenuItem>
+								)}
+								{deploymentsProject && stagingAvailable && (
+									<DropdownMenuItem
+										onSelect={() =>
+											void navigate({
+												to: buildAbsoluteLinkToPage(routeParams, 'config/deployments'),
+												search: { project: deploymentsProject, status: 'staged' },
+											})}
+									>
+										<RocketIcon />
+										Staged Releases
+									</DropdownMenuItem>
+								)}
 							</DropdownMenuContent>
 						</DropdownMenu>
 					)}

src/features/instance/applications/components/DeployProgress/DeployProgress.test.tsx

@@ -50,3 +50,150 @@ describe('DeployProgress', () => {
 		expect(screen.queryByRole('link', { name: 'Manage SSH keys' })).toBeNull();
 	});
 });
+
+const DONE: DeploymentStreamState['phases'] = {
+	prepare: 'done',
+	load: 'done',
+	replicate: 'done',
+	restart: 'done',
+	success: 'done',
+};
+
+function deployedState(extra: Partial<DeploymentStreamState>): DeploymentStreamState {
+	return { lifecycle: 'success', phases: DONE, installLog: [], peers: [], ...extra };
+}
+
+describe('DeployProgress canary certification (Harper 5.4+)', () => {
+	it('says a certified release was checked before it served', () => {
+		render(<DeployProgress state={deployedState({ certification: 'certified' })} />);
+		expect(screen.getByText('Certified')).toBeTruthy();
+		expect(screen.getByText(/canary loaded the release before it served/)).toBeTruthy();
+	});
+
+	it.each([
+		['uncertified', /went out unchecked: the component ran nothing as it loaded/],
+		['unavailable', /restarted unchecked: no worker on this node loads the component/],
+	])('says why an %s release went out unchecked', (certification, why) => {
+		render(<DeployProgress state={deployedState({ certification })} />);
+		expect(screen.getByText('Not certified')).toBeTruthy();
+		expect(screen.getByText(why)).toBeTruthy();
+	});
+
+	it('renders a status it does not know rather than hiding it', () => {
+		render(<DeployProgress state={deployedState({ certification: 'withdrawn' })} />);
+		expect(screen.getByText('Not certified (withdrawn)')).toBeTruthy();
+	});
+
+	it('shows nothing about certification for a version that does not report it', () => {
+		render(<DeployProgress state={deployedState({})} />);
+		expect(screen.queryByText(/certified/i)).toBeNull();
+	});
+
+	it('shows a rejection as the decision, with the release that went back, in place of the raw message', () => {
+		render(
+			<DeployProgress
+				state={{
+					...failedState('web was not deployed: release 9b2e failed to load in its canary worker: …'),
+					decision: { status: 'rejected', reason: 'web.jsResource: boom', restored: 'a3f8', failedClosed: false },
+				}}
+			/>,
+		);
+		expect(screen.getByText(/canary rejected this release on the node that received the deploy/)).toBeTruthy();
+		expect(screen.getByText('web.jsResource: boom')).toBeTruthy();
+		expect(screen.getByText('Deployment a3f8, the release it replaced, is live there again.')).toBeTruthy();
+		expect(screen.getByText('No other node received it.')).toBeTruthy();
+		expect(screen.queryByText(/web was not deployed/)).toBeNull();
+	});
+
+	it('says a rejected first release is failed closed', () => {
+		render(
+			<DeployProgress
+				state={{
+					...failedState('boom'),
+					decision: { status: 'interrupted', reason: 'shutting down', restored: null, failedClosed: true },
+				}}
+			/>,
+		);
+		expect(screen.getByText(/restart stopped before the canary decided/)).toBeTruthy();
+		expect(screen.getByText(/failed closed there until it is deployed again/)).toBeTruthy();
+	});
+});
+
+describe('DeployProgress staging (Harper 5.3+)', () => {
+	it('ends a stage at Staged, with no restart', () => {
+		render(
+			<DeployProgress
+				staging
+				state={deployedState({ phases: { prepare: 'done', replicate: 'done', staged: 'done' } })}
+			/>,
+		);
+		expect(screen.getByText('Staged')).toBeTruthy();
+		expect(screen.queryByText('Restart')).toBeNull();
+		expect(screen.queryByText('Finish')).toBeNull();
+	});
+});
+
+describe('DeployProgress warnings and rollout', () => {
+	it('keeps the warnings the deploy streamed', () => {
+		render(<DeployProgress state={deployedState({ warnings: ['Install fingerprints differ on node-b'] })} />);
+		expect(screen.getByText('Warnings')).toBeTruthy();
+		expect(screen.getByText('Install fingerprints differ on node-b')).toBeTruthy();
+	});
+
+	it('says the deploy is still rolling out while the job runs', () => {
+		render(
+			<DeployProgress
+				state={deployedState({ lifecycle: 'streaming', rollout: { jobId: 'j', state: 'following', nodes: [] } })}
+			/>,
+		);
+		expect(screen.getByText(/Rolling out to the other nodes/)).toBeTruthy();
+	});
+
+	it("names each node and how it took the release, including the one that didn't", () => {
+		render(
+			<DeployProgress
+				state={{
+					...failedState(''),
+					rollout: {
+						jobId: 'j',
+						state: 'failed',
+						nodes: [
+							{ node: 'node-b', outcome: 'activated', certification: 'certified' },
+							{ node: 'node-c', outcome: 'failed', detail: 'web was not deployed on this node: boom' },
+						],
+						message: 'Deployment 9b2e of web was not activated on 1 of 2 peer node(s): node-c (boom)',
+					},
+				}}
+			/>,
+		);
+		expect(screen.getByText('node-b')).toBeTruthy();
+		expect(screen.getByText('Certified')).toBeTruthy();
+		expect(screen.getByText('node-c')).toBeTruthy();
+		expect(screen.getByText('Did not take the release')).toBeTruthy();
+		expect(screen.getByText('web was not deployed on this node: boom')).toBeTruthy();
+		expect(screen.getByText(/was not activated on 1 of 2 peer node/)).toBeTruthy();
+	});
+
+	it("says it can't tell, rather than claiming success, when it could not follow the job", () => {
+		render(
+			<DeployProgress
+				state={deployedState({ rollout: { jobId: 'j', state: 'unfollowable', nodes: [], unfollowable: 'not-found' } })}
+			/>,
+		);
+		expect(screen.getByText(/couldn't find this deploy's restart job on the node it reached/)).toBeTruthy();
+	});
+
+	it('shows nothing for a single instance, which has no other nodes', () => {
+		render(<DeployProgress state={deployedState({ rollout: { jobId: 'j', state: 'complete', nodes: [] } })} />);
+		expect(screen.queryByText('Other nodes')).toBeNull();
+	});
+
+	it("links to the deployment's record, closing a hosting modal first", () => {
+		const onNavigateAway = vi.fn();
+		render(<DeployProgress state={deployedState({ deploymentId: 'd1' })} onNavigateAway={onNavigateAway} />);
+		const link = screen.getByRole('link', { name: "View this deployment's record" });
+		expect(link.getAttribute('href')).toBe('/org-1/instance/ins-1/config/deployments/d1');
+		fireEvent.click(link);
+		expect(onNavigateAway).toHaveBeenCalledOnce();
+	});
+});