diff --git a/DESIGN.md b/DESIGN.md index fd52bd729..9a656f28f 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -12,6 +12,7 @@ This file collects architecture and design notes for studio that aren't otherwis - [`src/features/instance/databases/DESIGN.md`](src/features/instance/databases/DESIGN.md) — when the browse grid may show its spinner, and why a table without a primary key can't be listed. +- [`src/features/instance/applications/DESIGN.md`](src/features/instance/applications/DESIGN.md) — when a rolling deploy is done: following its restart job, and why a job Studio can't follow is never a success; and when a live deploy may be sent again. - [`src/features/instance/config/roles/DESIGN.md`](src/features/instance/config/roles/DESIGN.md) — why the role editor's JSON text is the only source of truth, what forces the JSON view, and what Save changes on purpose. - [`src/router/DESIGN.md`](src/router/DESIGN.md) — why a failed route load is reported and shown only by its route, and why a preload still toasts. diff --git a/src/features/instance/applications/DESIGN.md b/src/features/instance/applications/DESIGN.md new file mode 100644 index 000000000..e58317fc0 --- /dev/null +++ b/src/features/instance/applications/DESIGN.md @@ -0,0 +1,57 @@ +# Applications — design notes + +## When a deploy is done + +Studio deploys with `restart: 'rolling'`. The node that receives the deploy answers once its own part is done and hands +the other nodes to a `restart_service` job, named in the response's `restartJobId` (Harper 4.6+). The deployment record +does not help: only the receiving node writes it, so it reads `success` while that job runs and after it fails. + +So [`deployAndFollowComponent`](../../../integrations/api/instance/applications/deployComponent.ts) follows the job with +`get_job` before the mutation settles ([`restartJob.ts`](../../../integrations/api/instance/applications/restartJob.ts)): + +- **A failed job rejects with `DeployRolloutError`.** The release is live on the receiving node; the error and the + progress panel name each node that did not take it. Its message quotes those nodes' errors, so Datadog withholds it + like `SSEOperationError`. +- **A job Studio could not follow resolves, as `rollout.state === 'unfollowable'`, and is never reported as success.** + That covers a job that ended with a result in neither shape below. [`reportDeployOutcome`](modals/reportDeployHealth.ts) + skips the post-deploy health check for it, because that check reads the one node Studio reaches, which can be healthy + on the release a rejecting node kept; `reportDeployResult` is the same policy for a caller that does not health-check. + Any new caller of the deploy mutation reports through one of the two. +- **The job may not be there.** `hdb_job` is a non-replicating system table, so only the receiving node can answer for + it. Central manager's cluster proxy sends every operation to the first running instance, so over the proxy the deploy + and its polls reach the same node; a direct connection to a cluster's address can reach another node, which answers + `[]`. A row missing for 20 seconds counts as `not-found`. +- **Every node's outcome arrives at once.** Harper writes the job's row when it starts and when it ends, never in between. + +The job comes in two shapes. Harper 5.4's certified rollout activates the staged release on each other node, and each +certifies it with a canary of its own: `result.activated` on success, `message: { error, activated }` on failure. Older +versions, and a 5.4 deploy whose release was not armed for a canary, run the plain restart: `result.replicated`, where a +node the job could not reach is listed with its error and the job still completes, so that node reads `unconfirmed`. + +Canary certification is shown only when the response carries it, which only 5.4+ does. There is no version check on +the deploy path: `registration_info` is denied to a role with an operations allowlist, which is exactly the kind of +role that may be allowed to deploy, so a version gate there would hide the result from it. + +## Staging and activation are gated the other way + +Stage only (`activate: false`) and Activate (`deployment_id`) are offered only when `registration_info` reads 5.3.0 or +later (`useStagedDeploymentsAvailable`), and an unreadable version offers neither. The difference from certification is +what happens when the guess is wrong: a node older than 5.3.0 accepts `activate: false` as an unknown field and makes +the release live, which is the opposite of what the person asked for. Certification only adds a field to a response. + +All three kinds of request are built in one place, [`deployRequestBody`](../../../integrations/api/instance/applications/deployComponent.ts), +because Harper's validator refuses the wrong mix outright: a stage with `restart`, or an activation carrying `package`, +install options or `activate`. A stage succeeds only once every other node confirmed staging it, so a stage restarts +nothing and is reported without a health check. Rollback to a kept release (#1232) is the same activation call. + +## A live deploy is sent once + +A live (SSE) deploy that gets no event stream goes out again as the buffered POST only when its status proves Harper +never started it: a 4xx. Harper refuses a request before the operation starts (authentication, permissions, +validation) with a 4xx, and answers one it started with a 200 event stream at once. Everything else can follow a deploy +that is already running: a gateway's 5xx after forwarding, a 2xx that is not a stream, a dropped connection, the idle +watchdog firing before headers. Harper's own request-queue 503 never started anything either, but status alone cannot +tell it from a gateway's. So [`onDeployComponentSubmit`](../../../integrations/api/instance/applications/deployComponent.ts) +turns those into `SSEInconclusiveError`, which every deploy caller already answers by checking rather than resending. +`streamOperation` stays neutral: it puts the `status` on `SSEUnsupportedError` and leaves the decision to the caller, +because a read (`get_deployment`) may fall back on anything. diff --git a/src/features/instance/applications/components/ContentActions.tsx b/src/features/instance/applications/components/ContentActions.tsx index 6638d13dd..39fd53629 100644 --- a/src/features/instance/applications/components/ContentActions.tsx +++ b/src/features/instance/applications/components/ContentActions.tsx @@ -16,10 +16,16 @@ import { useEditorFileContent } from '@/features/instance/applications/context/e import { isDirectory } from '@/features/instance/applications/context/isDirectory'; import { useEditorView } from '@/features/instance/applications/hooks/useEditorView'; import { useEntryActions } from '@/features/instance/applications/hooks/useEntryActions'; -import { useInstanceBrowseManagePermission } from '@/hooks/usePermissions'; +import { + useDeploymentsAvailable, + useStagedDeploymentsAvailable, +} from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; +import { useInstanceBrowseManagePermission, useInstanceManagePermission } from '@/hooks/usePermissions'; import { useRestartInstanceClick } from '@/hooks/useRestartInstanceClick'; import { emitToListeners, useEmitToListeners } from '@/lib/events/listener'; import { useSetWatchedValue, useWatchedValue } from '@/lib/events/watcher'; +import { buildAbsoluteLinkToPage } from '@/lib/urls/buildAbsoluteLinkToPage'; +import { useNavigate, useParams } from '@tanstack/react-router'; import { ArrowLeftIcon, ArrowRightIcon, @@ -28,12 +34,14 @@ import { DownloadIcon, FilePlusIcon, FolderPlusIcon, + HistoryIcon, LockIcon, PackageIcon, PanelRightCloseIcon, PanelRightOpenIcon, PencilIcon, PlusIcon, + RocketIcon, RotateCcwIcon, SaveIcon, TrashIcon, @@ -73,6 +81,14 @@ export function ContentActions({ const { openedEntryContents, openedEntry, isSavingFile, restrictPackageModification } = useEditorView(); const { content: updatedFileContent } = useEditorFileContent(openedEntry?.path); const canManageBrowseInstance = useInstanceBrowseManagePermission(); + const canManageInstance = useInstanceManagePermission(); + const deploymentsAvailable = useDeploymentsAvailable(); + const stagingAvailable = useStagedDeploymentsAvailable(); + const canViewDeployments = canManageInstance && deploymentsAvailable; + const navigate = useNavigate(); + const routeParams: { clusterId?: string; instanceId?: string; organizationId?: string } = useParams({ + strict: false, + }); const { onRestartClick, isRestartPending } = useRestartInstanceClick({ operation: 'restart_service', instanceClient: instanceParams.instanceClient, @@ -133,7 +149,8 @@ export function ContentActions({ useEntryActions(openedEntry); const showFileMenu = canEditFile || canRename || canAddEntries || canAddTable || canDeleteEntry; - const showApplicationMenu = canDownload || canManageBrowseInstance || canRedeploy; + const deploymentsProject = canViewDeployments && openedEntry?.project ? openedEntry.project : undefined; + const showApplicationMenu = canDownload || canManageBrowseInstance || canRedeploy || !!deploymentsProject; // The "File" menu acts on whatever is selected, so name it for that thing. const fileMenuLabel = openedEntry && isDirectory(openedEntry) ? 'Directory' : 'File'; @@ -389,6 +406,30 @@ export function ContentActions({ Redeploy Package )} + {deploymentsProject && ( + + void navigate({ + to: buildAbsoluteLinkToPage(routeParams, 'config/deployments'), + search: { project: deploymentsProject }, + })} + > + + View Deployments + + )} + {deploymentsProject && stagingAvailable && ( + + void navigate({ + to: buildAbsoluteLinkToPage(routeParams, 'config/deployments'), + search: { project: deploymentsProject, status: 'staged' }, + })} + > + + Staged Releases + + )} )} diff --git a/src/features/instance/applications/components/DeployProgress/DeployProgress.test.tsx b/src/features/instance/applications/components/DeployProgress/DeployProgress.test.tsx index 3cbd14dbc..fb867318e 100644 --- a/src/features/instance/applications/components/DeployProgress/DeployProgress.test.tsx +++ b/src/features/instance/applications/components/DeployProgress/DeployProgress.test.tsx @@ -50,3 +50,150 @@ describe('DeployProgress', () => { expect(screen.queryByRole('link', { name: 'Manage SSH keys' })).toBeNull(); }); }); + +const DONE: DeploymentStreamState['phases'] = { + prepare: 'done', + load: 'done', + replicate: 'done', + restart: 'done', + success: 'done', +}; + +function deployedState(extra: Partial): DeploymentStreamState { + return { lifecycle: 'success', phases: DONE, installLog: [], peers: [], ...extra }; +} + +describe('DeployProgress canary certification (Harper 5.4+)', () => { + it('says a certified release was checked before it served', () => { + render(); + expect(screen.getByText('Certified')).toBeTruthy(); + expect(screen.getByText(/canary loaded the release before it served/)).toBeTruthy(); + }); + + it.each([ + ['uncertified', /went out unchecked: the component ran nothing as it loaded/], + ['unavailable', /restarted unchecked: no worker on this node loads the component/], + ])('says why an %s release went out unchecked', (certification, why) => { + render(); + expect(screen.getByText('Not certified')).toBeTruthy(); + expect(screen.getByText(why)).toBeTruthy(); + }); + + it('renders a status it does not know rather than hiding it', () => { + render(); + expect(screen.getByText('Not certified (withdrawn)')).toBeTruthy(); + }); + + it('shows nothing about certification for a version that does not report it', () => { + render(); + expect(screen.queryByText(/certified/i)).toBeNull(); + }); + + it('shows a rejection as the decision, with the release that went back, in place of the raw message', () => { + render( + , + ); + expect(screen.getByText(/canary rejected this release on the node that received the deploy/)).toBeTruthy(); + expect(screen.getByText('web.jsResource: boom')).toBeTruthy(); + expect(screen.getByText('Deployment a3f8, the release it replaced, is live there again.')).toBeTruthy(); + expect(screen.getByText('No other node received it.')).toBeTruthy(); + expect(screen.queryByText(/web was not deployed/)).toBeNull(); + }); + + it('says a rejected first release is failed closed', () => { + render( + , + ); + expect(screen.getByText(/restart stopped before the canary decided/)).toBeTruthy(); + expect(screen.getByText(/failed closed there until it is deployed again/)).toBeTruthy(); + }); +}); + +describe('DeployProgress staging (Harper 5.3+)', () => { + it('ends a stage at Staged, with no restart', () => { + render( + , + ); + expect(screen.getByText('Staged')).toBeTruthy(); + expect(screen.queryByText('Restart')).toBeNull(); + expect(screen.queryByText('Finish')).toBeNull(); + }); +}); + +describe('DeployProgress warnings and rollout', () => { + it('keeps the warnings the deploy streamed', () => { + render(); + expect(screen.getByText('Warnings')).toBeTruthy(); + expect(screen.getByText('Install fingerprints differ on node-b')).toBeTruthy(); + }); + + it('says the deploy is still rolling out while the job runs', () => { + render( + , + ); + expect(screen.getByText(/Rolling out to the other nodes/)).toBeTruthy(); + }); + + it("names each node and how it took the release, including the one that didn't", () => { + render( + , + ); + expect(screen.getByText('node-b')).toBeTruthy(); + expect(screen.getByText('Certified')).toBeTruthy(); + expect(screen.getByText('node-c')).toBeTruthy(); + expect(screen.getByText('Did not take the release')).toBeTruthy(); + expect(screen.getByText('web was not deployed on this node: boom')).toBeTruthy(); + expect(screen.getByText(/was not activated on 1 of 2 peer node/)).toBeTruthy(); + }); + + it("says it can't tell, rather than claiming success, when it could not follow the job", () => { + render( + , + ); + expect(screen.getByText(/couldn't find this deploy's restart job on the node it reached/)).toBeTruthy(); + }); + + it('shows nothing for a single instance, which has no other nodes', () => { + render(); + expect(screen.queryByText('Other nodes')).toBeNull(); + }); + + it("links to the deployment's record, closing a hosting modal first", () => { + const onNavigateAway = vi.fn(); + render(); + const link = screen.getByRole('link', { name: "View this deployment's record" }); + expect(link.getAttribute('href')).toBe('/org-1/instance/ins-1/config/deployments/d1'); + fireEvent.click(link); + expect(onNavigateAway).toHaveBeenCalledOnce(); + }); +}); diff --git a/src/features/instance/applications/components/DeployProgress/DeployProgress.tsx b/src/features/instance/applications/components/DeployProgress/DeployProgress.tsx index 56d15da16..bec4f3674 100644 --- a/src/features/instance/applications/components/DeployProgress/DeployProgress.tsx +++ b/src/features/instance/applications/components/DeployProgress/DeployProgress.tsx @@ -1,13 +1,30 @@ import { Badge } from '@/components/ui/badge'; import { ScrollArea } from '@/components/ui/scrollArea'; import { Separator } from '@/components/ui/separator'; -import { DEPLOY_PHASE_ORDER } from '@/integrations/api/instance/applications/deployComponentStream'; +import { deploymentRecordPath } from '@/features/instance/applications/hooks/useDeploymentRecordAction'; +import { DEPLOY_PHASE_ORDER, STAGE_PHASE_ORDER } from '@/integrations/api/instance/applications/deployComponentStream'; +import { CertificationDecision } from '@/integrations/api/instance/applications/deployOutcome'; +import { RolloutNode, RolloutSnapshot } from '@/integrations/api/instance/applications/restartJob'; import { cn } from '@/lib/cn'; import { errorText } from '@/lib/errorText'; import { buildAbsoluteLinkToPage } from '@/lib/urls/buildAbsoluteLinkToPage'; import { Link, useParams } from '@tanstack/react-router'; -import { CheckIcon, CircleDashedIcon, CircleIcon, Loader2Icon, XIcon } from 'lucide-react'; +import { + CheckIcon, + CircleDashedIcon, + CircleIcon, + Loader2Icon, + ShieldCheckIcon, + TriangleAlertIcon, + XIcon, +} from 'lucide-react'; import { useEffect, useRef } from 'react'; +import { + describeCertification, + describeDecision, + describeRolloutNode, + describeUnfollowable, +} from './describeDeployOutcome'; import { DeploymentStreamState, PhaseStatus } from './useDeploymentStream'; const PHASE_LABELS: Record = { @@ -16,6 +33,7 @@ const PHASE_LABELS: Record = { replicate: 'Replicate', restart: 'Restart', success: 'Finish', + staged: 'Staged', }; /** @@ -45,10 +63,12 @@ function PhaseIcon({ status, isCurrent }: { status?: PhaseStatus; isCurrent: boo export function DeployProgress({ state, onNavigateAway, + staging = false, }: { state: DeploymentStreamState; /** Called when a guidance link navigates away — lets a hosting modal close itself first. */ onNavigateAway?: () => void; + staging?: boolean; }) { const logEndRef = useRef(null); const params: { clusterId?: string; instanceId?: string; organizationId?: string } = useParams({ strict: false }); @@ -62,7 +82,7 @@ export function DeployProgress({ return (
    - {DEPLOY_PHASE_ORDER.map((phase) => { + {(staging ? STAGE_PHASE_ORDER : DEPLOY_PHASE_ORDER).map((phase) => { const status = state.phases[phase]; const isCurrent = state.currentPhase === phase && status !== 'done'; return ( @@ -122,7 +142,28 @@ export function DeployProgress({
)} - {phasesFailed && state.error && ( + {!!state.warnings?.length && ( +
+ +
Warnings
+
    + {state.warnings.map((warning, i) => ( +
  • + + {warning} +
  • + ))} +
+
+ )} + + {state.certification && } + + {state.rollout && } + + {phasesFailed && state.decision && } + + {phasesFailed && state.error && !state.decision && (
@@ -140,11 +181,107 @@ export function DeployProgress({
)} - {state.lifecycle === 'inconclusive' && ( + {state.lifecycle === 'inconclusive' && !state.rollout && ( Live updates ended early — verifying… )} + + {state.deploymentId && ( + + View this deployment's record + + )} +
+ ); +} + +function CertificationLine({ certification }: { certification: string }) { + const { label, detail, certified } = describeCertification(certification); + return ( +
+ {certified + ? + : } + + {label} + {detail && {' '}{detail}} +
); } + +function DecisionBlock({ decision }: { decision: CertificationDecision }) { + const { title, reason, outcome } = describeDecision(decision); + return ( +
+ {title} + {reason && {reason}} + {outcome && {outcome}} + No other node received it. +
+ ); +} + +function RolloutSection({ rollout }: { rollout: RolloutSnapshot }) { + const unfollowable = describeUnfollowable(rollout); + if (rollout.state === 'complete' && rollout.nodes.length === 0) { + return null; + } + return ( +
+ +
Other nodes
+ {rollout.state === 'following' && ( +
+ + Rolling out to the other nodes, one at a time… +
+ )} + {!!rollout.nodes.length && ( +
    + {rollout.nodes.map((node, i) => { + const { label, detail } = describeRolloutNode(node); + return ( +
  • + + + {node.node} + {label} + + {detail && ( + + {detail} + + )} +
  • + ); + })} +
+ )} + {rollout.state === 'failed' && rollout.message && ( +
{rollout.message}
+ )} + {unfollowable && ( +
+ + {unfollowable} +
+ )} +
+ ); +} + +function RolloutNodeIcon({ outcome }: { outcome: RolloutNode['outcome'] }) { + if (outcome === 'failed') { + return ; + } + if (outcome === 'unconfirmed') { + return ; + } + return ; +} diff --git a/src/features/instance/applications/components/DeployProgress/describeDeployOutcome.ts b/src/features/instance/applications/components/DeployProgress/describeDeployOutcome.ts new file mode 100644 index 000000000..b2bfd3261 --- /dev/null +++ b/src/features/instance/applications/components/DeployProgress/describeDeployOutcome.ts @@ -0,0 +1,85 @@ +import { CertificationDecision } from '@/integrations/api/instance/applications/deployOutcome'; +import { RolloutNode, RolloutSnapshot } from '@/integrations/api/instance/applications/restartJob'; + +interface CertificationSummary { + label: string; + /** Why it went out unchecked, when it did. */ + detail?: string; + certified: boolean; +} + +// The reasons are Harper's own (documentation `reference/operations-api/operations.md`, "Certifying a release in a +// canary worker"; harper `components/operations.js:815-817`). +export function describeCertification(certification: string): CertificationSummary { + switch (certification) { + case 'certified': + return { label: 'Certified', detail: 'Its canary loaded the release before it served.', certified: true }; + case 'uncertified': + return { + label: 'Not certified', + detail: 'It went out unchecked: the component ran nothing as it loaded (dev-only loading, an absent ' + + 'if-installed component, or safe mode), or no worker was replaced.', + certified: false, + }; + case 'unavailable': + return { + label: 'Not certified', + detail: 'It restarted unchecked: no worker on this node loads the component, its isolated setting ' + + 'changed with this deploy, it was already live, or it was deployed from a local directory.', + certified: false, + }; + case 'not-requested': + return { label: 'Not certified', detail: 'The deploy did not restart, so nothing checked it.', certified: false }; + default: + return { label: `Not certified (${certification})`, certified: false }; + } +} + +export function describeDecision( + decision: CertificationDecision, +): { title: string; reason?: string; outcome?: string } { + const title = decision.status === 'rejected' + ? 'Its canary rejected this release on the node that received the deploy.' + : decision.status === 'interrupted' + ? 'Its restart stopped before the canary decided, on the node that received the deploy.' + : `It was not certified (${decision.status}) on the node that received the deploy.`; + const outcome = decision.restored + ? `Deployment ${decision.restored}, the release it replaced, is live there again.` + : decision.failedClosed + ? 'There was no earlier release to restore, so the component is failed closed there until it is deployed again.' + : undefined; + return { title, reason: decision.reason, outcome }; +} + +export function describeUnfollowable(rollout: RolloutSnapshot): string | undefined { + switch (rollout.unfollowable) { + case 'not-found': + return "Studio couldn't find this deploy's restart job on the node it reached, so it can't tell how the " + + 'other nodes took the release. Check their status before relying on it.'; + case 'refused': + return "Studio couldn't read this deploy's restart job, so it can't tell how the other nodes took the " + + 'release. Check their status before relying on it.'; + case 'timed-out': + return 'The other nodes were still restarting after 30 minutes, so Studio stopped waiting. The restart may ' + + 'still finish; check their status before relying on it.'; + case 'unrecognized': + return "This deploy's restart job finished, but Studio couldn't read how the other nodes took the release. " + + 'Check their status before relying on it.'; + } + return undefined; +} + +export function describeRolloutNode(node: RolloutNode): { label: string; detail?: string } { + switch (node.outcome) { + case 'activated': + return node.certification + ? { label: describeCertification(node.certification).label } + : { label: 'Took the release' }; + case 'restarted': + return { label: 'Restarted' }; + case 'failed': + return { label: 'Did not take the release', detail: node.detail }; + case 'unconfirmed': + return { label: 'Restart not confirmed', detail: node.detail }; + } +} diff --git a/src/features/instance/applications/components/DeployProgress/useDeploymentStream.test.ts b/src/features/instance/applications/components/DeployProgress/useDeploymentStream.test.ts new file mode 100644 index 000000000..4983c956d --- /dev/null +++ b/src/features/instance/applications/components/DeployProgress/useDeploymentStream.test.ts @@ -0,0 +1,80 @@ +/** @vitest-environment jsdom */ +import { SSEOperationError } from '@/integrations/api/sse/errors'; +import { act, renderHook } from '@testing-library/react'; +import { describe, expect, it } from 'vitest'; +import { useDeploymentStream } from './useDeploymentStream'; + +describe('useDeploymentStream', () => { + it('keeps streamed warnings and the latest rollout snapshot', () => { + const { result } = renderHook(() => useDeploymentStream()); + act(() => { + result.current.markStarted(); + result.current.onEvent({ type: 'warning', data: { message: 'node-b installed npm:web@1.5.0' } }); + result.current.onEvent({ type: 'warning', data: {} }); + result.current.onEvent({ type: 'rollout', data: { jobId: 'j', state: 'following', nodes: [] } }); + result.current.onEvent({ type: 'rollout', data: { jobId: 'j', state: 'complete', nodes: [] } }); + }); + expect(result.current.state.warnings).toEqual(['node-b installed npm:web@1.5.0']); + expect(result.current.state.rollout?.state).toBe('complete'); + }); + + it("takes the receiving node's certification and the record id from a deploy that answered", () => { + const { result } = renderHook(() => useDeploymentStream()); + act(() => { + result.current.markStarted(); + result.current.markSettled('success', undefined, { + result: { message: 'ok', deployment_id: 'd1', certification: 'certified' }, + }); + }); + expect(result.current.state).toMatchObject({ + lifecycle: 'success', + certification: 'certified', + deploymentId: 'd1', + }); + }); + + it("takes the canary's decision and the record id from a deploy it rejected", () => { + const { result } = renderHook(() => useDeploymentStream()); + const failure = new SSEOperationError('web was not deployed: …', { + deploymentId: 'd1', + certification: { status: 'rejected', reason: 'boom', restored: 'd0', failed_closed: false }, + }); + act(() => { + result.current.markStarted(); + result.current.markSettled('error', failure.message, { failure }); + }); + expect(result.current.state).toMatchObject({ + lifecycle: 'error', + error: 'web was not deployed: …', + deploymentId: 'd1', + decision: { status: 'rejected', reason: 'boom', restored: 'd0', failedClosed: false }, + }); + }); + + it('keeps the warnings a result carries when none streamed, without repeating streamed ones', () => { + const { result } = renderHook(() => useDeploymentStream()); + act(() => { + result.current.markStarted(); + result.current.markSettled('success', undefined, { result: { message: 'ok', warnings: ['drift'] } }); + }); + expect(result.current.state.warnings).toEqual(['drift']); + + act(() => { + result.current.markStarted(); + result.current.onEvent({ type: 'warning', data: { message: 'drift' } }); + result.current.markSettled('success', undefined, { result: { message: 'ok', warnings: ['drift'] } }); + }); + expect(result.current.state.warnings).toEqual(['drift']); + }); + + it('starts each deploy clean', () => { + const { result } = renderHook(() => useDeploymentStream()); + act(() => { + result.current.onEvent({ type: 'warning', data: { message: 'old' } }); + result.current.markSettled('success', undefined, { result: { message: 'ok', deployment_id: 'd1' } }); + result.current.markStarted(); + }); + expect(result.current.state.warnings).toBeUndefined(); + expect(result.current.state.deploymentId).toBeUndefined(); + }); +}); diff --git a/src/features/instance/applications/components/DeployProgress/useDeploymentStream.ts b/src/features/instance/applications/components/DeployProgress/useDeploymentStream.ts index af6b80954..78db60f37 100644 --- a/src/features/instance/applications/components/DeployProgress/useDeploymentStream.ts +++ b/src/features/instance/applications/components/DeployProgress/useDeploymentStream.ts @@ -1,4 +1,14 @@ -import { DeployPeerEvent, DeployStreamEvent } from '@/integrations/api/instance/applications/deployComponentStream'; +import { + DeployComponentResult, + DeployPeerEvent, + DeployStreamEvent, +} from '@/integrations/api/instance/applications/deployComponentStream'; +import { + CertificationDecision, + DeployCertification, + deployFailureDetails, +} from '@/integrations/api/instance/applications/deployOutcome'; +import { RolloutSnapshot } from '@/integrations/api/instance/applications/restartJob'; import { useCallback, useMemo, useReducer } from 'react'; export type DeploymentLifecycle = 'idle' | 'streaming' | 'success' | 'error' | 'inconclusive'; @@ -18,6 +28,13 @@ export interface DeploymentStreamState { currentPhase?: string; installLog: InstallLine[]; peers: DeployPeerEvent[]; + warnings?: string[]; + rollout?: RolloutSnapshot; + /** How the node that received the deploy checked the release, once it answered. */ + certification?: DeployCertification; + /** Why that node's canary refused the release. */ + decision?: CertificationDecision; + deploymentId?: string; error?: string; } @@ -32,11 +49,39 @@ const INITIAL_STATE: DeploymentStreamState = { // limit; the server already aggregates, this is a UI safety net. const MAX_INSTALL_LINES = 1000; +type Settled = Extract; + type Action = | { kind: 'reset' } | { kind: 'start' } | { kind: 'event'; event: DeployStreamEvent } - | { kind: 'settled'; lifecycle: Extract; error?: string }; + | { kind: 'settled'; lifecycle: Settled; error?: string; outcome?: SettledOutcome }; + +interface SettledOutcome { + result?: DeployComponentResult; + failure?: unknown; +} + +function settledDetails( + state: DeploymentStreamState, + outcome: SettledOutcome | undefined, +): Partial { + if (outcome?.result) { + const { certification, deployment_id, rollout, warnings } = outcome.result; + return { + certification: typeof certification === 'string' ? certification : undefined, + deploymentId: deployment_id, + ...(rollout ? { rollout } : {}), + // The streamed events already put them here; a result that did not stream still carries them. + ...(!state.warnings?.length && warnings?.length ? { warnings } : {}), + }; + } + if (outcome?.failure !== undefined) { + const { deploymentId, certification } = deployFailureDetails(outcome.failure); + return { deploymentId, decision: certification }; + } + return {}; +} function reducer(state: DeploymentStreamState, action: Action): DeploymentStreamState { switch (action.kind) { @@ -64,11 +109,20 @@ function reducer(state: DeploymentStreamState, action: Action): DeploymentStream installLog: next.length > MAX_INSTALL_LINES ? next.slice(-MAX_INSTALL_LINES) : next, }; } + if (event.type === 'warning') { + const { message } = event.data; + return typeof message === 'string' && message + ? { ...state, warnings: (state.warnings ?? []).concat(message) } + : state; + } + if (event.type === 'rollout') { + return { ...state, rollout: event.data }; + } // peer return { ...state, peers: state.peers.concat(event.data) }; } case 'settled': - return { ...state, lifecycle: action.lifecycle, error: action.error }; + return { ...state, ...settledDetails(state, action.outcome), lifecycle: action.lifecycle, error: action.error }; } } @@ -87,8 +141,8 @@ export function useDeploymentStream() { const markStarted = useCallback(() => dispatch({ kind: 'start' }), []); const reset = useCallback(() => dispatch({ kind: 'reset' }), []); const markSettled = useCallback( - (lifecycle: 'success' | 'error' | 'inconclusive', error?: string) => - dispatch({ kind: 'settled', lifecycle, error }), + (lifecycle: Settled, error?: string, outcome?: SettledOutcome) => + dispatch({ kind: 'settled', lifecycle, error, outcome }), [], ); diff --git a/src/features/instance/applications/components/NewApplication/ImportInstructions.tsx b/src/features/instance/applications/components/NewApplication/ImportInstructions.tsx index 4399f8c89..a63381c2d 100644 --- a/src/features/instance/applications/components/NewApplication/ImportInstructions.tsx +++ b/src/features/instance/applications/components/NewApplication/ImportInstructions.tsx @@ -1,13 +1,16 @@ import { Button } from '@/components/ui/button'; import { CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card'; import { FormControl } from '@/components/ui/form/FormControl'; +import { FormDescription } from '@/components/ui/form/FormDescription'; import { FormField } from '@/components/ui/form/FormField'; import { FormItem } from '@/components/ui/form/FormItem'; import { FormLabel } from '@/components/ui/form/FormLabel'; import { FormMessage } from '@/components/ui/form/FormMessage'; import { Input } from '@/components/ui/input'; import { Separator } from '@/components/ui/separator'; +import { Switch } from '@/components/ui/switch'; import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs'; +import { useStagedDeploymentsAvailable } from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; import { GitMergeIcon, LinkIcon, PackageIcon, RocketIcon } from 'lucide-react'; import { useCallback } from 'react'; import { Control, FormState, UseFormSetValue, UseFormWatch } from 'react-hook-form'; @@ -29,6 +32,7 @@ export function ImportInstructions({ watch: UseFormWatch>; }) { const importSource = watch('contents.source'); + const canStage = useStagedDeploymentsAvailable(); const setImportSource = useCallback((source: string) => { setValue('contents.source', source as 'git' | 'npm' | 'tarball'); }, [setValue]); @@ -154,6 +158,26 @@ export function ImportInstructions({ {importSource === 'git' && } + {canStage && ( + ( + + + + +
+ Stage only + + Install it on every node without making it live. Activate it later from Deployments. + +
+
+ )} + /> + )} + diff --git a/src/features/instance/applications/components/NewApplication/NewApplication.test.tsx b/src/features/instance/applications/components/NewApplication/NewApplication.test.tsx index b2d2f39bf..fb44778fe 100644 --- a/src/features/instance/applications/components/NewApplication/NewApplication.test.tsx +++ b/src/features/instance/applications/components/NewApplication/NewApplication.test.tsx @@ -21,6 +21,9 @@ vi.mock('./useCheckCLI', () => ({ useCheckCLI: () => ({ checkCLI: vi.fn() }) })) // These read the instance's SSH keys and auth, which aren't what these tests exercise. vi.mock('./ImportAuthorization', () => ({ ImportAuthorization: () => null })); vi.mock('./useCLISteps', () => ({ useCLISteps: () => [] })); +vi.mock('@/features/instance/applications/hooks/useSupportsDeploymentSSE', () => ({ + useStagedDeploymentsAvailable: () => false, +})); import { NewApplication } from './index'; diff --git a/src/features/instance/applications/components/NewApplication/schema.ts b/src/features/instance/applications/components/NewApplication/schema.ts index 70c1d1d9f..d55d7da44 100644 --- a/src/features/instance/applications/components/NewApplication/schema.ts +++ b/src/features/instance/applications/components/NewApplication/schema.ts @@ -20,6 +20,7 @@ export const ImportSchema = z.object({ // The SSH key the import URL is expected to use. Form-only state used to verify the URL's host // — it is not sent to deploy_component (the backend resolves the key from the URL host alias). sshKeyName: z.string(), + stageOnly: z.boolean(), }); export const defaultImportOptions: z.infer = { @@ -29,6 +30,7 @@ export const defaultImportOptions: z.infer = { requiresAuth: false, installCommand: '', sshKeyName: '', + stageOnly: false, }; export const CLISchema = z.object({ diff --git a/src/features/instance/applications/components/NewApplication/useImportApplication.ts b/src/features/instance/applications/components/NewApplication/useImportApplication.ts index 23d738864..156156d32 100644 --- a/src/features/instance/applications/components/NewApplication/useImportApplication.ts +++ b/src/features/instance/applications/components/NewApplication/useImportApplication.ts @@ -1,9 +1,15 @@ import { useInstanceClientIdParams } from '@/config/useInstanceClient'; import { useComponentHealthCheck } from '@/features/instance/applications/hooks/useComponentHealthCheck'; +import { useDeploymentRecordAction } from '@/features/instance/applications/hooks/useDeploymentRecordAction'; import { useEditorView } from '@/features/instance/applications/hooks/useEditorView'; -import { useSupportsDeploymentSSE } from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; -import { reportDeployHealth } from '@/features/instance/applications/modals/reportDeployHealth'; +import { + useStagedDeploymentsAvailable, + useSupportsDeploymentSSE, +} from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; +import { reportDeployOutcome, reportStaged } from '@/features/instance/applications/modals/reportDeployHealth'; import { useDeployComponentMutation } from '@/integrations/api/instance/applications/deployComponent'; +import { deployFailureDetails } from '@/integrations/api/instance/applications/deployOutcome'; +import { DeployRolloutError } from '@/integrations/api/instance/applications/restartJob'; import { SSEInconclusiveError } from '@/integrations/api/sse/errors'; import { invalidateEntityQueries } from '@/react-query/invalidateEntityQueries'; import { useQueryClient } from '@tanstack/react-query'; @@ -28,7 +34,9 @@ export function useImportApplication( const queryClient = useQueryClient(); const instanceParams = useInstanceClientIdParams(); const sseDeploy = useSupportsDeploymentSSE(); + const canStage = useStagedDeploymentsAvailable(); const runHealthCheck = useComponentHealthCheck(instanceParams); + const recordAction = useDeploymentRecordAction(); const { reloadRootEntries, setFocusedItem, setExpandedItems, setSelectedItems } = useEditorView(); const onImported = useCallback((project: string) => { @@ -55,39 +63,57 @@ export function useImportApplication( contents: z.infer; project: string; }) => { - const toastId = toast.loading(`Importing application...`, { + const stage = canStage && contents.stageOnly; + const toastId = toast.loading(stage ? 'Staging application...' : 'Importing application...', { description: 'This may take a bit.', duration: 300_000, }); mutate({ - applicationName: project, - applicationUrl: contents.ref, - installCommand: contents.installCommand, + request: { + kind: stage ? 'stage' : 'deploy', + project, + package: contents.ref, + installCommand: contents.installCommand, + }, useSSE: sseDeploy, onEvent: (event) => { // Stream live progress into the loading toast: phase transitions, and the latest // install output line during the install/load phase (so npm output shows through). if (event.type === 'phase' && event.data.status === 'start') { - toast.loading(`Importing ${project}…`, { + toast.loading(`${stage ? 'Staging' : 'Importing'} ${project}…`, { id: toastId, description: PHASE_LABELS[event.data.phase] ?? event.data.phase, duration: 300_000, }); } else if (event.type === 'install' && event.data.line?.trim()) { - toast.loading(`Importing ${project}…`, { + toast.loading(`${stage ? 'Staging' : 'Importing'} ${project}…`, { id: toastId, description: event.data.line.trim().slice(0, 140), duration: 300_000, }); + } else if (event.type === 'rollout' && event.data.state === 'following') { + toast.loading(`${stage ? 'Staging' : 'Importing'} ${project}…`, { + id: toastId, + description: 'Rolling out to the other nodes…', + duration: Infinity, + }); } }, ...instanceParams, }, { - onSuccess: async () => { + onSuccess: async (result) => { toast.dismiss(toastId); + if (stage) { + // Nothing went live, so there is no new application in the tree to open. + reportStaged(project, result, { action: recordAction(result?.deployment_id) }); + return; + } onImported(project); - if (sseDeploy) { - reportDeployHealth(project, await runHealthCheck(project)); + // A followed rollout restarted the other nodes, so there is a health to check however the deploy was sent. + if (sseDeploy || result?.rollout) { + await reportDeployOutcome(project, result, runHealthCheck, { + action: recordAction(result?.deployment_id), + }); } else { toast.success(`Imported successfully`, { description: `${project} will be available once you restart your ${instanceParams.entityType}!`, @@ -98,17 +124,30 @@ export function useImportApplication( onError: async (error) => { toast.dismiss(toastId); // Live stream dropped after the deploy started — verify rather than assume failure. + if (error instanceof SSEInconclusiveError && stage) { + toast.warning(`${project} may not be staged`, { + description: 'Live updates ended before the stage finished. Check Deployments for its record.', + }); + return; + } if (error instanceof SSEInconclusiveError) { onImported(project); - reportDeployHealth(project, await runHealthCheck(project)); + await reportDeployOutcome(project, undefined, runHealthCheck); + return; + } + const action = recordAction(deployFailureDetails(error).deploymentId); + if (error instanceof DeployRolloutError) { + onImported(project); + toast.error(`${project} did not reach every node`, { description: error.message, action }); return; } - toast.error(`Failed to import ${project}`, { + toast.error(`Failed to ${stage ? 'stage' : 'import'} ${project}`, { description: error instanceof Error ? error.message : undefined, + action, }); }, }); - }, [mutate, sseDeploy, instanceParams, runHealthCheck, onImported]); + }, [mutate, sseDeploy, canStage, instanceParams, runHealthCheck, onImported, recordAction]); return useMemo(() => ({ isImportingApplication, diff --git a/src/features/instance/applications/hooks/useDeploymentRecordAction.ts b/src/features/instance/applications/hooks/useDeploymentRecordAction.ts new file mode 100644 index 000000000..eb549e48e --- /dev/null +++ b/src/features/instance/applications/hooks/useDeploymentRecordAction.ts @@ -0,0 +1,23 @@ +import { buildAbsoluteLinkToPage } from '@/lib/urls/buildAbsoluteLinkToPage'; +import { useNavigate, useParams } from '@tanstack/react-router'; +import { useCallback } from 'react'; + +export function deploymentRecordPath( + params: { clusterId?: string; instanceId?: string; organizationId?: string }, + deploymentId: string, +): string { + return buildAbsoluteLinkToPage(params, `config/deployments/${encodeURIComponent(deploymentId)}`); +} + +/** A toast action that opens a deployment's record, for a deploy whose result or failure named one. */ +export function useDeploymentRecordAction() { + const navigate = useNavigate(); + const params: { clusterId?: string; instanceId?: string; organizationId?: string } = useParams({ strict: false }); + return useCallback( + (deploymentId: string | undefined) => + deploymentId + ? { label: 'View record', onClick: () => void navigate({ to: deploymentRecordPath(params, deploymentId) }) } + : undefined, + [navigate, params], + ); +} diff --git a/src/features/instance/applications/hooks/useSupportsDeploymentSSE.test.ts b/src/features/instance/applications/hooks/useSupportsDeploymentSSE.test.ts new file mode 100644 index 000000000..5fb026b9d --- /dev/null +++ b/src/features/instance/applications/hooks/useSupportsDeploymentSSE.test.ts @@ -0,0 +1,20 @@ +/** @vitest-environment jsdom */ +import { describe, expect, it } from 'vitest'; +import { supportsStagedDeployments } from './useSupportsDeploymentSSE'; + +describe('supportsStagedDeployments', () => { + it.each([ + ['5.3.0', true], + ['5.3.2', true], + ['5.4.0', true], + ['6.0.0', true], + ['5.2.15', false], + ['4.7.9', false], + // An older node reads `activate: false` as an ordinary deploy, so anything short of a release is refused. + ['5.3.0-beta.4', false], + [undefined, false], + ['not a version', false], + ])('%s -> %s', (version, expected) => { + expect(supportsStagedDeployments(version)).toBe(expected); + }); +}); diff --git a/src/features/instance/applications/hooks/useSupportsDeploymentSSE.ts b/src/features/instance/applications/hooks/useSupportsDeploymentSSE.ts index deedf2c6b..4bdbf78a6 100644 --- a/src/features/instance/applications/hooks/useSupportsDeploymentSSE.ts +++ b/src/features/instance/applications/hooks/useSupportsDeploymentSSE.ts @@ -15,6 +15,26 @@ export function supportsDeployments(harperVersion: string | undefined): boolean return !!harperVersion && wasAReleasedBeforeB(MIN_DEPLOYMENT_SSE_VERSION, harperVersion); } +/** + * Staging (`activate: false`), activation by `deployment_id`, and the `staged` status ship in 5.3.0 (harper + * `v5.3.0:components/operations.js:638`). A 5.3.0 prerelease is refused, as `supportsPutOperation` refuses one. + */ +export const MIN_STAGED_DEPLOYMENT_VERSION = '5.3.0'; + +export function supportsStagedDeployments(harperVersion: string | undefined): boolean { + return !!harperVersion && wasAReleasedBeforeB(MIN_STAGED_DEPLOYMENT_VERSION, harperVersion); +} + +/** + * Whether to offer staging and activation. Unlike a put, an unreadable version must not offer them: a node older + * than 5.3.0 accepts `activate: false` as an unknown field and deploys the release live. + */ +export function useStagedDeploymentsAvailable(): boolean { + const params = useInstanceClientIdParams(); + const { data } = useQuery(getRegistrationInfoQueryOptions(params)); + return supportsStagedDeployments(data?.version); +} + /** * True when the instance/cluster is Harper >= 5.1.0, so the deployment history/detail * feature (`list_deployments` / `get_deployment`) is available. This gates feature diff --git a/src/features/instance/applications/modals/RedeployApplicationModal.tsx b/src/features/instance/applications/modals/RedeployApplicationModal.tsx index 1e9fe6d65..2047b14a1 100644 --- a/src/features/instance/applications/modals/RedeployApplicationModal.tsx +++ b/src/features/instance/applications/modals/RedeployApplicationModal.tsx @@ -2,27 +2,36 @@ import { Button } from '@/components/ui/button'; import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle } from '@/components/ui/dialog'; import { Form } from '@/components/ui/form/Form'; import { FormControl } from '@/components/ui/form/FormControl'; +import { FormDescription } from '@/components/ui/form/FormDescription'; import { FormField } from '@/components/ui/form/FormField'; import { FormItem } from '@/components/ui/form/FormItem'; import { FormLabel } from '@/components/ui/form/FormLabel'; import { FormMessage } from '@/components/ui/form/FormMessage'; import { Input } from '@/components/ui/input'; +import { Switch } from '@/components/ui/switch'; import { useInstanceClientIdParams } from '@/config/useInstanceClient'; import { DeployProgress } from '@/features/instance/applications/components/DeployProgress/DeployProgress'; import { useDeploymentStream } from '@/features/instance/applications/components/DeployProgress/useDeploymentStream'; import { useComponentHealthCheck } from '@/features/instance/applications/hooks/useComponentHealthCheck'; +import { useDeploymentRecordAction } from '@/features/instance/applications/hooks/useDeploymentRecordAction'; import { useEditorView } from '@/features/instance/applications/hooks/useEditorView'; -import { useSupportsDeploymentSSE } from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; +import { + useStagedDeploymentsAvailable, + useSupportsDeploymentSSE, +} from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; import { useDeployComponentMutation } from '@/integrations/api/instance/applications/deployComponent'; +import { deployFailureDetails } from '@/integrations/api/instance/applications/deployOutcome'; +import { DeployRolloutError } from '@/integrations/api/instance/applications/restartJob'; import { SSEInconclusiveError } from '@/integrations/api/sse/errors'; import { attemptToRestoreFocus } from '@/lib/attemptToRestoreFocus'; import { setWatchedValue, useWatchedValue } from '@/lib/events/watcher'; +import { describeError } from '@/react-query/queryClient'; import { useQueryClient } from '@tanstack/react-query'; import { RefreshCwIcon } from 'lucide-react'; import { useCallback, useEffect, useState } from 'react'; import { useForm } from 'react-hook-form'; import { toast } from 'sonner'; -import { reportDeployHealth } from './reportDeployHealth'; +import { deployNotes, reportDeployOutcome, reportDeployResult, reportStaged } from './reportDeployHealth'; export function RedeployApplicationModal() { const { value: isModalOpen, trigger } = useWatchedValue('ShowRedeployApplicationModal', false); @@ -30,33 +39,39 @@ export function RedeployApplicationModal() { const queryClient = useQueryClient(); const instanceParams = useInstanceClientIdParams(); const sseDeploy = useSupportsDeploymentSSE(); + const canStage = useStagedDeploymentsAvailable(); const stream = useDeploymentStream(); const runHealthCheck = useComponentHealthCheck(instanceParams); + const recordAction = useDeploymentRecordAction(); const { openedEntry } = useEditorView(); const packageUrl = openedEntry?.package; const { mutate: reDeployApplication, isPending } = useDeployComponentMutation(); const [isStreaming, setIsStreaming] = useState(false); + const [isStaging, setIsStaging] = useState(false); const methods = useForm({ defaultValues: { applicationUrl: packageUrl, installCommand: '', + stageOnly: false, }, }); - const { control, handleSubmit, reset } = methods; + const { control, handleSubmit, reset, watch } = methods; + const stageOnlyChecked = canStage && watch('stageOnly'); useEffect(() => { - reset({ applicationUrl: packageUrl }); + reset({ applicationUrl: packageUrl, stageOnly: false }); }, [reset, packageUrl]); const closeModal = useCallback(() => { setWatchedValue('ShowRedeployApplicationModal', false); attemptToRestoreFocus(trigger); - reset({ applicationUrl: packageUrl }); + reset({ applicationUrl: packageUrl, stageOnly: false }); setIsStreaming(false); + setIsStaging(false); stream.reset(); }, [reset, packageUrl, trigger, stream]); @@ -68,7 +83,7 @@ export function RedeployApplicationModal() { }, [queryClient, instanceParams.entityId]); const redeployWithProgress = useCallback( - (applicationUrl: string, installCommand: string | undefined) => { + (applicationUrl: string, installCommand: string | undefined, stage: boolean) => { if (!openedEntry) { return; } @@ -76,80 +91,157 @@ export function RedeployApplicationModal() { stream.reset(); stream.markStarted(); setIsStreaming(true); + setIsStaging(stage); reDeployApplication( { - applicationName: project, - applicationUrl, - installCommand, + request: { kind: stage ? 'stage' : 'deploy', project, package: applicationUrl, installCommand }, useSSE: true, onEvent: stream.onEvent, ...instanceParams, }, { - onSuccess: async () => { - stream.markSettled('success'); + onSuccess: async (result) => { + if (stage) { + stream.markSettled('success', undefined, { result }); + reportStaged(project, result, { action: recordAction(result?.deployment_id) }); + if (!deployNotes(result).warn) { + closeModal(); + } + return; + } + const known = result?.rollout?.state !== 'unfollowable'; + stream.markSettled(known ? 'success' : 'inconclusive', undefined, { result }); invalidateComponents(); - const health = await runHealthCheck(project); - reportDeployHealth(project, health); - closeModal(); + await reportDeployOutcome(project, result, runHealthCheck, { + action: recordAction(result?.deployment_id), + }); + // Left open when there is something to read in it: an outcome Studio couldn't confirm, or a warning. + if (known && !deployNotes(result).warn) { + closeModal(); + } }, onError: async (error) => { // The deploy started but the live stream dropped — verify via health check // instead of reporting a failure we don't actually know happened. if (error instanceof SSEInconclusiveError) { stream.markSettled('inconclusive'); + if (stage) { + // A stage changes nothing that is running, so the component's health says nothing about it. + toast.warning(`${project} may not be staged`, { + description: 'Live updates ended before the stage finished. Check Deployments for its record.', + }); + closeModal(); + return; + } invalidateComponents(); - const health = await runHealthCheck(project); - reportDeployHealth(project, health); + await reportDeployOutcome(project, undefined, runHealthCheck); closeModal(); return; } - stream.markSettled('error', error instanceof Error ? error.message : String(error)); - toast.error(`Failed to redeploy ${project}`, { + const action = recordAction(deployFailureDetails(error).deploymentId); + if (error instanceof DeployRolloutError) { + // Live on the node that received it; the rollout section names the nodes that did not take it. + stream.markSettled('error', undefined, { failure: error }); + invalidateComponents(); + toast.error(`${project} did not reach every node`, { description: error.message, action }); + return; + } + stream.markSettled('error', error instanceof Error ? error.message : String(error), { + failure: error, + }); + toast.error(`Failed to ${stage ? 'stage' : 'redeploy'} ${project}`, { description: error instanceof Error ? error.message : undefined, + action, }); }, }, ); }, - [reDeployApplication, openedEntry, instanceParams, stream, invalidateComponents, runHealthCheck, closeModal], + [ + reDeployApplication, + openedEntry, + instanceParams, + stream, + invalidateComponents, + runHealthCheck, + closeModal, + recordAction, + ], ); const redeployBuffered = useCallback( - (applicationUrl: string, installCommand: string | undefined) => { + (applicationUrl: string, installCommand: string | undefined, stage: boolean) => { if (!openedEntry) { return; } closeModal(); - const toastId = toast.loading('Redeploying...'); + const toastId = toast.loading(stage ? 'Staging...' : 'Redeploying...'); reDeployApplication( - { applicationName: openedEntry.project, applicationUrl, installCommand, ...instanceParams }, { - onSuccess: () => { - toast.success(`Application ${openedEntry.project} redeployed successfully`, { id: toastId }); + request: { + kind: stage ? 'stage' : 'deploy', + project: openedEntry.project, + package: applicationUrl, + installCommand, + }, + ...instanceParams, + }, + { + onSuccess: (result) => { + if (stage) { + reportStaged(openedEntry.project, result, { id: toastId, action: recordAction(result?.deployment_id) }); + return; + } + reportDeployResult( + openedEntry.project, + result, + { title: `Application ${openedEntry.project} redeployed successfully` }, + { id: toastId, action: recordAction(result?.deployment_id) }, + ); invalidateComponents(); }, - onError: () => { + onError: (error) => { + // The modal is already closed, so a toast is the only place left to link the record from. + const action = recordAction(deployFailureDetails(error).deploymentId); + if (error instanceof DeployRolloutError) { + toast.error(`${openedEntry.project} did not reach every node`, { + id: toastId, + description: error.message, + action, + }); + invalidateComponents(); + return; + } + if (action) { + toast.error(`Failed to ${stage ? 'stage' : 'redeploy'} ${openedEntry.project}`, { + id: toastId, + description: describeError(error).message, + action, + }); + return; + } toast.dismiss(toastId); }, }, ); }, - [reDeployApplication, openedEntry, instanceParams, invalidateComponents, closeModal], + [reDeployApplication, openedEntry, instanceParams, invalidateComponents, closeModal, recordAction], ); - const submitForm = ({ applicationUrl, installCommand }: { + const submitForm = ({ applicationUrl, installCommand, stageOnly }: { applicationUrl: string | undefined; installCommand: string | undefined; + stageOnly?: boolean; }) => { if (!applicationUrl) { return; } + const stage = canStage && !!stageOnly; if (sseDeploy) { - redeployWithProgress(applicationUrl, installCommand); + redeployWithProgress(applicationUrl, installCommand, stage); } else { - redeployBuffered(applicationUrl, installCommand); + redeployBuffered(applicationUrl, installCommand, stage); } }; @@ -162,14 +254,16 @@ export function RedeployApplicationModal() { Redeploy Package - {isStreaming ? `Redeploying ${openedEntry?.project ?? 'package'}…` : 'Redeploy this package?'} + {isStreaming + ? `${isStaging ? 'Staging' : 'Redeploying'} ${openedEntry?.project ?? 'package'}…` + : 'Redeploy this package?'} {isStreaming ? (
- + + {started && ( + <> + + + + )} + + + + Activate this release of {project}? + + Deployment {deploymentId}{' '} + goes live on every node, which restart one at a time. Nothing is installed: each node swaps in the release + it staged. + + + + Cancel + void activate()}>Activate + + + + + ); +} diff --git a/src/features/instance/config/deployments/components/DeploymentDetail.tsx b/src/features/instance/config/deployments/components/DeploymentDetail.tsx index 2f68691b5..01c45ec6e 100644 --- a/src/features/instance/config/deployments/components/DeploymentDetail.tsx +++ b/src/features/instance/config/deployments/components/DeploymentDetail.tsx @@ -3,14 +3,19 @@ import { Separator } from '@/components/ui/separator'; import { useInstanceClientIdParams } from '@/config/useInstanceClient'; import { DeployProgress } from '@/features/instance/applications/components/DeployProgress/DeployProgress'; import { useDeploymentStream } from '@/features/instance/applications/components/DeployProgress/useDeploymentStream'; +import { deploymentRecordPath } from '@/features/instance/applications/hooks/useDeploymentRecordAction'; import { useSupportsDeploymentSSE } from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; import { getDeploymentQueryOptions, getDeploymentStream } from '@/integrations/api/instance/deployments/getDeployment'; import { Deployment, isTerminalDeploymentStatus } from '@/integrations/api/instance/deployments/types'; +import { cn } from '@/lib/cn'; import { errorText } from '@/lib/errorText'; import { humanFileSize } from '@/lib/humanFileSize'; import { translateSecondsToAgo } from '@/lib/translateSecondsToAgo'; import { useQuery, useQueryClient } from '@tanstack/react-query'; +import { Link, useParams } from '@tanstack/react-router'; import { useEffect } from 'react'; +import { describePeerInstall } from '../functions/describePeerInstall'; +import { ActivateStagedRelease } from './ActivateStagedRelease'; import { DeploymentStatusBadge } from './deploymentStatusBadge'; function Field({ label, value }: { label: string; value: React.ReactNode }) { @@ -31,6 +36,7 @@ function timestamp(ms?: number) { export function DeploymentDetail({ deploymentId }: { deploymentId: string }) { const instanceParams = useInstanceClientIdParams(); + const params: { clusterId?: string; instanceId?: string; organizationId?: string } = useParams({ strict: false }); const sseSupported = useSupportsDeploymentSSE(); const queryClient = useQueryClient(); const stream = useDeploymentStream(); @@ -113,17 +119,35 @@ export function DeploymentDetail({ deploymentId }: { deploymentId: string }) { : undefined} /> {deployment.install_fingerprint.source} + : undefined} /> {deployment.rollback_of} + label="Activation of" + value={deployment.activated_from + ? ( + + {deployment.activated_from} + + ) : undefined} />
+ {deployment.status === 'staged' && ( + <> +
+ Staged: built and installed, but not live. Activating it makes it live with nothing to install. +
+ + + )} + {showLiveProgress && ( <> @@ -152,17 +176,30 @@ export function DeploymentDetail({ deploymentId }: { deploymentId: string }) { <>
-
Replication
-
    +
    Other nodes
    +
      {deployment.peer_results.map((peer, i) => { const peerError = errorText(peer.error); + const install = describePeerInstall(peer); return ( -
    • - {peer.node} - - {peer.status ?? 'ok'} - {peerError ? `: ${peerError}` : ''} +
    • + + {peer.node} + + {peer.status ?? 'ok'} + {peerError ? `: ${peerError}` : ''} + + {install && ( + + {install.label} + + )}
    • ); })} diff --git a/src/features/instance/config/deployments/components/deploymentStatusBadge.tsx b/src/features/instance/config/deployments/components/deploymentStatusBadge.tsx index eb4f0ff4b..7033913c3 100644 --- a/src/features/instance/config/deployments/components/deploymentStatusBadge.tsx +++ b/src/features/instance/config/deployments/components/deploymentStatusBadge.tsx @@ -13,9 +13,10 @@ const STATUS_VARIANT: Record = { success: 'success', failed: 'destructive', rolled_back: 'destructive', + staged: 'secondary', }; -const STATUS_LABEL: Record = { +export const DEPLOYMENT_STATUS_LABEL: Record = { pending: 'Pending', extracting: 'Extracting', installing: 'Installing', @@ -25,8 +26,9 @@ const STATUS_LABEL: Record = { success: 'Success', failed: 'Failed', rolled_back: 'Rolled back', + staged: 'Staged', }; export function DeploymentStatusBadge({ status }: { status: DeploymentStatus }) { - return {STATUS_LABEL[status] ?? status}; + return {DEPLOYMENT_STATUS_LABEL[status] ?? status}; } diff --git a/src/features/instance/config/deployments/constants/tableDefinition.tsx b/src/features/instance/config/deployments/constants/tableDefinition.tsx index 81a03dfb7..83cfc3446 100644 --- a/src/features/instance/config/deployments/constants/tableDefinition.tsx +++ b/src/features/instance/config/deployments/constants/tableDefinition.tsx @@ -2,6 +2,7 @@ import { Deployment } from '@/integrations/api/instance/deployments/types'; import { ColumnDef, createColumnHelper } from '@/lib/table'; import { translateSecondsToAgo } from '@/lib/translateSecondsToAgo'; import { DeploymentStatusBadge } from '../components/deploymentStatusBadge'; +import { deploymentSource, shortDeploymentId } from '../functions/deploymentSource'; const columnHelper = createColumnHelper(); @@ -10,8 +11,18 @@ function ago(ms?: number) { } export const deploymentColumns: Array> = [ + columnHelper.display({ + header: 'Deployment', + id: 'deployment_id', + enableSorting: false, + cell: (props) => ( + + {shortDeploymentId(props.row.original.deployment_id)} + + ), + }), { - header: 'Project', + header: 'Component', accessorKey: 'project', enableSorting: false, }, @@ -19,28 +30,35 @@ export const deploymentColumns: Array> = [ header: 'Status', id: 'status', enableSorting: false, - cell: (props) => , - }), - columnHelper.display({ - header: 'Started', - id: 'started_at', - enableSorting: false, - cell: (props) => ago(props.row.original.started_at), + cell: (props) => ( + + + {props.row.original.status === 'staged' && ( + waiting to be activated + )} + + ), }), columnHelper.display({ - header: 'Completed', - id: 'completed_at', + header: 'Source', + id: 'source', enableSorting: false, - cell: (props) => ago(props.row.original.completed_at), + cell: (props) => { + const source = deploymentSource(props.row.original); + return source + ? {source.label} + : '—'; + }, }), { header: 'User', accessorKey: 'user', enableSorting: false, }, - { - header: 'Origin', - accessorKey: 'origin_node', + columnHelper.display({ + header: 'Started', + id: 'started_at', enableSorting: false, - }, + cell: (props) => ago(props.row.original.started_at), + }), ]; diff --git a/src/features/instance/config/deployments/deploymentsSearch.test.ts b/src/features/instance/config/deployments/deploymentsSearch.test.ts new file mode 100644 index 000000000..347e1e639 --- /dev/null +++ b/src/features/instance/config/deployments/deploymentsSearch.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest'; +import { filterableStatuses, validateDeploymentsSearch } from './deploymentsSearch'; + +describe('validateDeploymentsSearch', () => { + it('keeps a component, a known status and a later page', () => { + expect(validateDeploymentsSearch({ project: ' web ', status: 'staged', page: '3' })) + .toEqual({ project: 'web', status: 'staged', page: 3 }); + }); + + it.each([ + ['an empty component', { project: ' ' }], + ['a status Harper does not filter on', { status: 'bogus' }], + ['the first page, which is the default', { page: 1 }], + ['a page that is not a whole number', { page: '2.5' }], + ['a negative page', { page: -2 }], + ['a component that is not text', { project: ['web'] }], + ])('drops %s', (_label, search) => { + expect(validateDeploymentsSearch(search)).toEqual({ project: undefined, status: undefined, page: undefined }); + }); +}); + +describe('filterableStatuses', () => { + it('offers staged only where staging exists (Harper 5.3.0+)', () => { + expect(filterableStatuses(true)).toContain('staged'); + expect(filterableStatuses(false)).not.toContain('staged'); + expect(filterableStatuses(false)).toContain('success'); + }); +}); diff --git a/src/features/instance/config/deployments/deploymentsSearch.ts b/src/features/instance/config/deployments/deploymentsSearch.ts new file mode 100644 index 000000000..084369a24 --- /dev/null +++ b/src/features/instance/config/deployments/deploymentsSearch.ts @@ -0,0 +1,38 @@ +import { DeploymentStatus } from '@/integrations/api/instance/deployments/types'; + +export interface DeploymentsSearch { + project?: string; + status?: DeploymentStatus; + /** 1-based. */ + page?: number; +} + +// `rolled_back` is left out: no Harper version writes it, so the filter could only ever come back empty. +export const FILTERABLE_STATUSES: readonly DeploymentStatus[] = [ + 'pending', + 'extracting', + 'installing', + 'loading', + 'replicating', + 'restarting', + 'success', + 'failed', + 'staged', +]; + +export function filterableStatuses(stagingSupported: boolean): readonly DeploymentStatus[] { + return stagingSupported ? FILTERABLE_STATUSES : FILTERABLE_STATUSES.filter((status) => status !== 'staged'); +} + +export const DEPLOYMENTS_PAGE_SIZE = 25; + +export function validateDeploymentsSearch(search: Record): DeploymentsSearch { + const project = typeof search.project === 'string' ? search.project.trim() : ''; + const status = FILTERABLE_STATUSES.find((candidate) => candidate === search.status); + const page = Number(search.page); + return { + project: project || undefined, + status, + page: Number.isInteger(page) && page > 1 ? page : undefined, + }; +} diff --git a/src/features/instance/config/deployments/functions/deploymentDescriptions.test.ts b/src/features/instance/config/deployments/functions/deploymentDescriptions.test.ts new file mode 100644 index 000000000..f702becd9 --- /dev/null +++ b/src/features/instance/config/deployments/functions/deploymentDescriptions.test.ts @@ -0,0 +1,52 @@ +import { Deployment } from '@/integrations/api/instance/deployments/types'; +import { describe, expect, it } from 'vitest'; +import { deploymentSource } from './deploymentSource'; +import { describePeerInstall } from './describePeerInstall'; + +const base = { deployment_id: 'd', project: 'web', status: 'success' } as Deployment; + +describe('deploymentSource', () => { + it('names the package a deploy came from', () => { + expect(deploymentSource({ ...base, package_identifier: 'github:acme/web#v2' })) + .toEqual({ label: 'github:acme/web#v2', title: 'github:acme/web#v2' }); + }); + + it("names a payload deploy by its tarball's hash", () => { + expect(deploymentSource({ ...base, package_identifier: null, payload_hash: 'abcdef0123456789abcdef' })) + .toEqual({ label: 'Payload abcdef012345', title: 'abcdef0123456789abcdef' }); + }); + + it('names the release an activation made live, which built nothing', () => { + expect(deploymentSource({ ...base, activated_from: 'a1b2c3d4-0000-4000-8000-000000000001' })) + .toEqual({ label: 'Activation of a1b2c3d4', title: 'a1b2c3d4-0000-4000-8000-000000000001' }); + }); + + it('names nothing for a record that says nothing', () => { + expect(deploymentSource(base)).toBeUndefined(); + }); +}); + +// The comparison fields Harper records per peer: harper `components/deploymentRecorder.ts` installComparisonFields. +describe('describePeerInstall', () => { + it('says nothing for a record from before installs were compared (pre-5.3.1)', () => { + expect(describePeerInstall({ node: 'node-b', status: 'success' })).toBeUndefined(); + }); + + it('names what a peer installed differently', () => { + expect(describePeerInstall({ + node: 'node-b', + install: { source: 'npm:web@1.5.0', lockfiles: {} }, + install_matches: false, + install_differs: ['source', 'package-lock.json'], + })).toEqual({ label: 'Installed differently: source npm:web@1.5.0, package-lock.json.', matches: false }); + }); + + it('says a matching install matched', () => { + expect(describePeerInstall({ node: 'node-b', install_matches: true, install_differs: [] })?.matches).toBe(true); + }); + + it('does not read missing evidence as a match', () => { + expect(describePeerInstall({ node: 'node-b', install: null, install_matches: null, install_differs: [] })) + .toEqual({ label: "Couldn't compare its install: a fingerprint is missing or unidentified.", matches: null }); + }); +}); diff --git a/src/features/instance/config/deployments/functions/deploymentSource.ts b/src/features/instance/config/deployments/functions/deploymentSource.ts new file mode 100644 index 000000000..b7641e005 --- /dev/null +++ b/src/features/instance/config/deployments/functions/deploymentSource.ts @@ -0,0 +1,25 @@ +import { Deployment } from '@/integrations/api/instance/deployments/types'; + +export function shortDeploymentId(id: string): string { + return id.length > 8 ? id.slice(0, 8) : id; +} + +/** + * Where a release came from. A package deploy records its reference and a payload deploy its tarball's hash; an + * activation (5.3.0+) builds nothing, so it names the deployment whose release it made live. + */ +export function deploymentSource(deployment: Deployment): { label: string; title: string } | undefined { + if (deployment.activated_from) { + return { + label: `Activation of ${shortDeploymentId(deployment.activated_from)}`, + title: deployment.activated_from, + }; + } + if (deployment.package_identifier) { + return { label: deployment.package_identifier, title: deployment.package_identifier }; + } + if (deployment.payload_hash) { + return { label: `Payload ${deployment.payload_hash.slice(0, 12)}`, title: deployment.payload_hash }; + } + return undefined; +} diff --git a/src/features/instance/config/deployments/functions/describePeerInstall.ts b/src/features/instance/config/deployments/functions/describePeerInstall.ts new file mode 100644 index 000000000..c66ef2378 --- /dev/null +++ b/src/features/instance/config/deployments/functions/describePeerInstall.ts @@ -0,0 +1,26 @@ +import { DeploymentPeerResult } from '@/integrations/api/instance/deployments/types'; + +/** + * Whether a node installed what the node that received the deploy did (5.3.1+). Harper adds the comparison only + * on a deploy or a stage that compared fingerprints, so a record without it says nothing either way. + */ +export function describePeerInstall( + peer: DeploymentPeerResult, +): { label: string; matches: boolean | null } | undefined { + if (!('install_matches' in peer)) { + return undefined; + } + if (peer.install_matches === true) { + return { label: 'Installed the same as the node that received the deploy.', matches: true }; + } + if (peer.install_matches === false) { + const differs = (peer.install_differs ?? []).map((field) => + field === 'source' && peer.install?.source ? `source ${peer.install.source}` : field + ); + return { + label: `Installed differently: ${differs.length ? differs.join(', ') : 'see the deploy log'}.`, + matches: false, + }; + } + return { label: "Couldn't compare its install: a fingerprint is missing or unidentified.", matches: null }; +} diff --git a/src/features/instance/config/deployments/index.test.tsx b/src/features/instance/config/deployments/index.test.tsx index 4a621a354..c210bc520 100644 --- a/src/features/instance/config/deployments/index.test.tsx +++ b/src/features/instance/config/deployments/index.test.tsx @@ -1,6 +1,6 @@ /** @vitest-environment jsdom */ import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; -import { cleanup, render, screen } from '@testing-library/react'; +import { cleanup, fireEvent, render, screen } from '@testing-library/react'; import { afterEach, describe, expect, it, vi } from 'vitest'; import { ConfigDeploymentsIndex } from './index'; @@ -8,8 +8,16 @@ const state = vi.hoisted(() => ({ version: undefined as string | undefined, registrationFails: false, listEnabled: [] as (boolean | undefined)[], + listParams: [] as Record[], + listData: undefined as { deployments: object[]; total: number } | undefined, + search: {} as Record, + navigate: vi.fn(), +})); +vi.mock('@tanstack/react-router', () => ({ + useNavigate: () => state.navigate, + useParams: () => ({}), + useSearch: () => state.search, })); -vi.mock('@tanstack/react-router', () => ({ useNavigate: () => vi.fn(), useParams: () => ({}) })); vi.mock('@/hooks/usePermissions', () => ({ useInstanceManagePermission: () => true })); vi.mock('@/config/useInstanceClient', () => ({ useInstanceClientIdParams: () => ({ entityId: 'ins-a' }) })); vi.mock('@/integrations/api/instance/status/getRegistrationInfo', () => ({ @@ -21,9 +29,15 @@ vi.mock('@/integrations/api/instance/status/getRegistrationInfo', () => ({ }), })); vi.mock('@/integrations/api/instance/deployments/listDeployments', () => ({ - getDeploymentsQueryOptions: ({ enabled }: { enabled?: boolean }) => { - state.listEnabled.push(enabled); - return { queryKey: ['list_deployments'], queryFn: () => ({ deployments: [], total: 0 }), enabled: false }; + getDeploymentsQueryOptions: (params: { enabled?: boolean }) => { + state.listEnabled.push(params.enabled); + state.listParams.push(params); + return { + queryKey: ['list_deployments'], + queryFn: () => ({ deployments: [], total: 0 }), + enabled: false, + ...(state.listData ? { initialData: state.listData } : {}), + }; }, })); vi.mock('./modals/DeploymentDetailModal', () => ({ DeploymentDetailModal: () => null })); @@ -33,6 +47,10 @@ afterEach(() => { state.version = undefined; state.registrationFails = false; state.listEnabled = []; + state.listParams = []; + state.listData = undefined; + state.search = {}; + state.navigate.mockClear(); }); function renderPage(version: string | undefined) { @@ -71,3 +89,89 @@ describe('ConfigDeploymentsIndex version gate', () => { expect(screen.queryByText('No deployments yet.')).toBeNull(); }); }); + +const row = (id: string, extra: object = {}) => ({ + deployment_id: id, + project: 'web', + status: 'success', + started_at: 1, + ...extra, +}); + +describe('ConfigDeploymentsIndex filters and paging', () => { + it('asks list_deployments for the filtered page the URL names', () => { + state.search = { project: 'web', status: 'staged', page: 3 }; + renderPage('5.3.0'); + expect(state.listParams.at(-1)).toMatchObject({ project: 'web', status: 'staged', limit: 25, offset: 50 }); + }); + + it('does not ask a pre-5.3 instance for staged records a saved link names', () => { + state.search = { status: 'staged' }; + renderPage('5.2.15'); + expect(state.listParams.at(-1)).toMatchObject({ status: undefined }); + // …and drops it from the URL, which would otherwise claim a filter the list is not applying. + const update = state.navigate.mock.calls.at(-1)?.[0] as { search: (prev: object) => object; replace: boolean }; + expect(update.replace).toBe(true); + expect(update.search({ status: 'staged', project: 'web' })).toEqual({ status: undefined, project: 'web' }); + }); + + it('keeps a staged filter where staging exists', () => { + state.search = { status: 'staged' }; + renderPage('5.3.0'); + expect(state.listParams.at(-1)).toMatchObject({ status: 'staged' }); + expect(state.navigate).not.toHaveBeenCalled(); + }); + + it('applies the component filter on Enter only, so leaving the field does not move the page', () => { + state.search = { page: 2 }; + renderPage('5.3.0'); + const input = screen.getByRole('textbox', { name: 'Component' }); + fireEvent.change(input, { target: { value: 'web' } }); + fireEvent.blur(input); + expect(state.navigate).not.toHaveBeenCalled(); + }); + + it('pages by the total the server reports, not by the rows it sent', () => { + state.search = { page: 2 }; + state.listData = { deployments: Array.from({ length: 25 }, (_, i) => row(`d-${i}`)), total: 60 }; + renderPage('5.3.0'); + expect(screen.getByText('26–50 of 60')).toBeTruthy(); + fireEvent.click(screen.getByRole('button', { name: 'Next page' })); + const update = state.navigate.mock.calls.at(-1)?.[0] as { search: (prev: object) => object }; + expect(update.search({ page: 2, project: 'web' })).toEqual({ page: 3, project: 'web' }); + fireEvent.click(screen.getByRole('button', { name: 'Previous page' })); + const back = state.navigate.mock.calls.at(-1)?.[0] as { search: (prev: object) => object }; + expect(back.search({ page: 2 })).toEqual({ page: undefined }); + }); + + it('filters by component on Enter, and starts again from the first page', () => { + state.search = { page: 4 }; + renderPage('5.3.0'); + const input = screen.getByRole('textbox', { name: 'Component' }); + fireEvent.change(input, { target: { value: ' web ' } }); + fireEvent.submit(input.closest('form')!); + const update = state.navigate.mock.calls.at(-1)?.[0] as { search: (prev: object) => object }; + expect(update.search({ page: 4 })).toEqual({ page: undefined, project: 'web' }); + }); + + it('says the filters matched nothing rather than that there are no deployments', () => { + state.search = { project: 'nope' }; + state.listData = { deployments: [], total: 0 }; + renderPage('5.3.0'); + expect(screen.getByText('No deployments match these filters.')).toBeTruthy(); + }); + + it('shows a staged release as waiting, and an activation by the release it made live', () => { + state.listData = { + deployments: [ + row('a1b2c3d4-0000-4000-8000-000000000001', { status: 'staged', package_identifier: 'npm:web@2.0.0' }), + row('e5f6a7b8-0000-4000-8000-000000000002', { activated_from: 'a1b2c3d4-0000-4000-8000-000000000001' }), + ], + total: 2, + }; + renderPage('5.3.0'); + expect(screen.getByText('waiting to be activated')).toBeTruthy(); + expect(screen.getByText('npm:web@2.0.0')).toBeTruthy(); + expect(screen.getByText('Activation of a1b2c3d4')).toBeTruthy(); + }); +}); diff --git a/src/features/instance/config/deployments/index.tsx b/src/features/instance/config/deployments/index.tsx index ec984b3ab..8ba5e698b 100644 --- a/src/features/instance/config/deployments/index.tsx +++ b/src/features/instance/config/deployments/index.tsx @@ -1,24 +1,34 @@ import { SimpleBrowseDataTable } from '@/components/SimpleBrowseDataTable'; import { Button } from '@/components/ui/button'; +import { Input } from '@/components/ui/input'; +import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select'; import { useInstanceClientIdParams } from '@/config/useInstanceClient'; -import { supportsDeployments } from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; +import { + supportsDeployments, + supportsStagedDeployments, +} from '@/features/instance/applications/hooks/useSupportsDeploymentSSE'; import { useInstanceManagePermission } from '@/hooks/usePermissions'; import { useRefreshClick } from '@/hooks/useRefreshClick'; import { getDeploymentsQueryOptions } from '@/integrations/api/instance/deployments/listDeployments'; -import { Deployment } from '@/integrations/api/instance/deployments/types'; +import { Deployment, DeploymentStatus } from '@/integrations/api/instance/deployments/types'; import { getRegistrationInfoQueryOptions } from '@/integrations/api/instance/status/getRegistrationInfo'; import { Row } from '@/lib/table'; import { useQuery } from '@tanstack/react-query'; -import { useNavigate, useParams } from '@tanstack/react-router'; -import { RefreshCwIcon } from 'lucide-react'; -import { useCallback, useMemo } from 'react'; +import { useNavigate, useParams, useSearch } from '@tanstack/react-router'; +import { ChevronLeftIcon, ChevronRightIcon, RefreshCwIcon } from 'lucide-react'; +import { useCallback, useEffect, useMemo, useRef } from 'react'; +import { DEPLOYMENT_STATUS_LABEL } from './components/deploymentStatusBadge'; import { deploymentColumns } from './constants/tableDefinition'; +import { DEPLOYMENTS_PAGE_SIZE, DeploymentsSearch, filterableStatuses } from './deploymentsSearch'; import { DeploymentDetailModal } from './modals/DeploymentDetailModal'; +const ALL_STATUSES = 'all'; + export function ConfigDeploymentsIndex() { const navigate = useNavigate(); const canManage = useInstanceManagePermission(); const { deploymentId }: { deploymentId?: string } = useParams({ strict: false }); + const { project, status: requestedStatus, page = 1 }: DeploymentsSearch = useSearch({ strict: false }); const instanceParams = useInstanceClientIdParams(); // The nav hides this page below Harper 5.1, but a bookmark or back-navigation still lands here. const { data: registrationInfo, isError: registrationFailed } = useQuery( @@ -27,27 +37,70 @@ export function ConfigDeploymentsIndex() { const versionKnown = registrationInfo !== undefined; const versionUnavailable = !versionKnown && registrationFailed; const deploymentsAvailable = supportsDeployments(registrationInfo?.version); + const statuses = filterableStatuses(supportsStagedDeployments(registrationInfo?.version)); + const status = requestedStatus && statuses.includes(requestedStatus) ? requestedStatus : undefined; + const unsupportedStatus = versionKnown && !!requestedStatus && !status; + useEffect(() => { + // Otherwise the URL would keep claiming a filter the list is not applying. + if (unsupportedStatus) { + void navigate({ + to: '.', + search: (previous: DeploymentsSearch) => ({ ...previous, status: undefined }), + replace: true, + }); + } + }, [unsupportedStatus, navigate]); const { data, isFetching, isRefetching, refetch, error } = useQuery( getDeploymentsQueryOptions({ ...instanceParams, - limit: 100, + project, + status, + limit: DEPLOYMENTS_PAGE_SIZE, + offset: (page - 1) * DEPLOYMENTS_PAGE_SIZE, enabled: canManage && deploymentsAvailable, pollWhileOpen: true, }), ); const deployments = useMemo(() => data?.deployments ?? [], [data]); + const total = data?.total ?? 0; + const firstShown = total === 0 ? 0 : (page - 1) * DEPLOYMENTS_PAGE_SIZE + 1; + const lastShown = Math.min(total, (page - 1) * DEPLOYMENTS_PAGE_SIZE + deployments.length); + const filtered = !!project || !!status; + + const setSearch = useCallback( + (patch: DeploymentsSearch) => { + void navigate({ to: '.', search: (previous: DeploymentsSearch) => ({ ...previous, ...patch }) }); + }, + [navigate], + ); + const applyProject = useCallback( + (value: string) => { + const next = value.trim() || undefined; + if (next !== project) { + setSearch({ project: next, page: undefined }); + } + }, + [project, setSearch], + ); const onSelect = useCallback( (next: string | undefined) => { const parts = [deploymentId ? '..' : '', next].filter(Boolean); - void navigate({ to: parts.join('/') || '.' }); + void navigate({ to: parts.join('/') || '.', search: true }); }, [deploymentId, navigate], ); const onRowClick = useCallback((row: Row) => onSelect(row.original.deployment_id), [onSelect]); const closeModal = useCallback(() => onSelect(undefined), [onSelect]); const onRefreshClick = useRefreshClick(refetch); + const projectInput = useRef(null); + useEffect(() => { + // Follows a filter set from outside the field, such as the editor's link, without remounting it. + if (projectInput.current && projectInput.current.value.trim() !== (project ?? '')) { + projectInput.current.value = project ?? ''; + } + }, [project]); if (!canManage) { return ( @@ -73,6 +126,37 @@ export function ConfigDeploymentsIndex() { columns={deploymentColumns} onRowClick={onRowClick} > +
      { + event.preventDefault(); + applyProject(String(new FormData(event.currentTarget).get('project') ?? '')); + }} + > + + +
      + + ) + : filtered + ? 'No deployments match these filters.' + : 'No deployments yet.'} +
+ )} + {deployments.length > 0 && ( +
+ + {firstShown}–{lastShown} of {total} + + + +
)} {!!deploymentId && ( diff --git a/src/features/instance/config/routes.ts b/src/features/instance/config/routes.ts index ee2b40bef..55713d9a2 100644 --- a/src/features/instance/config/routes.ts +++ b/src/features/instance/config/routes.ts @@ -1,5 +1,6 @@ import { ConfigCertificatesIndex } from '@/features/instance/config/certificates'; import { ConfigDeploymentsIndex } from '@/features/instance/config/deployments'; +import { validateDeploymentsSearch } from '@/features/instance/config/deployments/deploymentsSearch'; import { ConfigDomainsIndex } from '@/features/instance/config/domains'; import { ConfigIndex } from '@/features/instance/config/index'; import { ConfigOverviewIndex } from '@/features/instance/config/overview'; @@ -76,12 +77,14 @@ export function createConfigRouteTree(instanceLayoutRoute: ReturnType instanceConfigRoute, path: 'deployments', + validateSearch: validateDeploymentsSearch, head: () => ({ meta: [{ title: 'Deployments — Harper Fabric' }] }), component: ConfigDeploymentsIndex, }); const instanceConfigDeploymentRoute = createRoute({ getParentRoute: () => instanceConfigRoute, path: 'deployments/$deploymentId', + validateSearch: validateDeploymentsSearch, head: () => ({ meta: [{ title: 'Deployments — Harper Fabric' }] }), component: ConfigDeploymentsIndex, }); diff --git a/src/integrations/api/instance/applications/deployComponent.fallback.test.ts b/src/integrations/api/instance/applications/deployComponent.fallback.test.ts new file mode 100644 index 000000000..7eaed9539 --- /dev/null +++ b/src/integrations/api/instance/applications/deployComponent.fallback.test.ts @@ -0,0 +1,103 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +vi.mock('@/integrations/api/sse/resolveInstanceConnection', () => ({ + resolveInstanceConnection: () => ({ + url: 'https://ins-1.example.com:9925/', + headers: { 'Content-Type': 'application/json', Accept: 'text/event-stream' }, + credentials: 'same-origin' as RequestCredentials, + mode: 'direct', + }), +})); + +import { SSEInconclusiveError } from '@/integrations/api/sse/errors'; +import { onDeployComponentSubmit } from './deployComponent'; + +const live = { + request: { kind: 'deploy' as const, project: 'my-app', package: 'npm:my-app' }, + entityId: 'ins-1' as const, + entityType: 'instance' as const, + useSSE: true, +}; + +function bufferedClient() { + return { post: vi.fn().mockResolvedValue({ data: { message: 'Successfully deployed: my-app' } }) }; +} + +function deploy(client: ReturnType) { + return onDeployComponentSubmit({ ...live, instanceClient: client as never }); +} + +afterEach(() => { + vi.unstubAllGlobals(); + vi.useRealTimers(); +}); + +describe('the live deploy, through the real stream', () => { + it.each([502, 503, 504])( + 'is not sent again after a %i from a gateway, which may have forwarded it', + async (status) => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue( + new Response('Bad Gateway', { + status, + headers: { 'content-type': 'text/html' }, + }), + ), + ); + const client = bufferedClient(); + + await expect(deploy(client)).rejects.toBeInstanceOf(SSEInconclusiveError); + expect(client.post).not.toHaveBeenCalled(); + }, + ); + + it('is not sent again after a 2xx that is not a stream, which means it ran', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue(Response.json({ message: 'Successfully deployed: my-app' })), + ); + const client = bufferedClient(); + + await expect(deploy(client)).rejects.toBeInstanceOf(SSEInconclusiveError); + expect(client.post).not.toHaveBeenCalled(); + }); + + it('is not sent again when the connection drops after the request went out', async () => { + vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new TypeError('Failed to fetch'))); + const client = bufferedClient(); + + await expect(deploy(client)).rejects.toBeInstanceOf(SSEInconclusiveError); + expect(client.post).not.toHaveBeenCalled(); + }); + + it('is not sent again when no answer arrives within the idle window', async () => { + vi.useFakeTimers(); + vi.stubGlobal( + 'fetch', + vi.fn((_url: string, init: RequestInit) => + new Promise((_resolve, reject) => { + init.signal?.addEventListener('abort', () => reject(new DOMException('aborted', 'AbortError'))); + }) + ), + ); + const client = bufferedClient(); + + const outcome = deploy(client).catch((e: unknown) => e); + await vi.advanceTimersByTimeAsync(120_000); + + expect(await outcome).toBeInstanceOf(SSEInconclusiveError); + expect(client.post).not.toHaveBeenCalled(); + }); + + it.each([401, 403])('goes out on the buffered path when Harper refused the stream with a %i', async (status) => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue(Response.json({ error: 'Login failed' }, { status })), + ); + const client = bufferedClient(); + + await expect(deploy(client)).resolves.toEqual({ message: 'Successfully deployed: my-app' }); + expect(client.post).toHaveBeenCalledOnce(); + }); +}); diff --git a/src/integrations/api/instance/applications/deployComponent.test.ts b/src/integrations/api/instance/applications/deployComponent.test.ts index 0d655d978..199ca4f79 100644 --- a/src/integrations/api/instance/applications/deployComponent.test.ts +++ b/src/integrations/api/instance/applications/deployComponent.test.ts @@ -1,12 +1,13 @@ -import { afterEach, describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; vi.mock('./deployComponentStream', () => ({ deployComponentStream: vi.fn(), })); -import { SSEOperationError, SSEUnsupportedError } from '@/integrations/api/sse/errors'; -import { onDeployComponentSubmit } from './deployComponent'; +import { SSEInconclusiveError, SSEOperationError, SSEUnsupportedError } from '@/integrations/api/sse/errors'; +import { deployAndFollowComponent, deployRequestBody, onDeployComponentSubmit } from './deployComponent'; import { deployComponentStream } from './deployComponentStream'; +import { DeployRolloutError } from './restartJob'; const mockStream = vi.mocked(deployComponentStream); @@ -15,8 +16,7 @@ function fakeClient(data: unknown) { } const baseArgs = { - applicationName: 'my-app', - applicationUrl: 'npm:my-app', + request: { kind: 'deploy' as const, project: 'my-app', package: 'npm:my-app' }, entityId: 'ins-1' as const, entityType: 'instance' as const, }; @@ -25,7 +25,78 @@ afterEach(() => { vi.clearAllMocks(); }); +// Harper's validator decides these shapes: a stage refuses `restart` (harper `v5.3.0:components/operationsValidation.js:517-524`), +// and an activation refuses `package`, install options and `activate` alongside `deployment_id` (`:613-616`). +describe('deployRequestBody', () => { + it('deploys with a rolling restart', () => { + expect(deployRequestBody({ kind: 'deploy', project: 'web', package: 'npm:web', installCommand: 'npm ci' }, true)) + .toEqual({ + operation: 'deploy_component', + package: 'npm:web', + project: 'web', + replicated: true, + install_command: 'npm ci', + restart: 'rolling', + }); + }); + + it('stages with activate: false and no restart', () => { + const body = deployRequestBody({ kind: 'stage', project: 'web', package: 'npm:web' }, true); + expect(body).toEqual({ + operation: 'deploy_component', + package: 'npm:web', + project: 'web', + replicated: true, + install_command: undefined, + activate: false, + }); + expect(body).not.toHaveProperty('restart'); + }); + + it('activates by id alone, with a rolling restart and nothing to install', () => { + const body = deployRequestBody({ kind: 'activate', project: 'web', deploymentId: 'a3f8c2d1' }, false); + expect(body).toEqual({ + operation: 'deploy_component', + project: 'web', + deployment_id: 'a3f8c2d1', + replicated: false, + restart: 'rolling', + }); + expect(JSON.parse(JSON.stringify(body))).not.toHaveProperty('package'); + }); +}); + describe('onDeployComponentSubmit', () => { + it('stages through the buffered path exactly as built', async () => { + const client = fakeClient({ message: 'Staged: my-app', staged: true, deployment_id: 'd1' }); + await onDeployComponentSubmit({ + ...baseArgs, + request: { kind: 'stage', project: 'my-app', package: 'npm:my-app' }, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: client as any, + }); + expect(client.post).toHaveBeenCalledWith( + '/', + expect.objectContaining({ activate: false, project: 'my-app', replicated: false }), + { timeout: 300_000 }, + ); + expect(client.post.mock.calls[0][1]).not.toHaveProperty('restart'); + }); + + it('streams the same body it would post', async () => { + mockStream.mockResolvedValue({ message: 'ok' }); + await onDeployComponentSubmit({ + ...baseArgs, + request: { kind: 'activate', project: 'my-app', deploymentId: 'd1' }, + useSSE: true, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: fakeClient({}) as any, + }); + expect(mockStream).toHaveBeenCalledWith(expect.objectContaining({ + body: deployRequestBody({ kind: 'activate', project: 'my-app', deploymentId: 'd1' }, false), + })); + }); + it('uses the buffered axios path when SSE is not requested', async () => { const client = fakeClient({ message: 'buffered' }); const result = await onDeployComponentSubmit({ @@ -52,17 +123,44 @@ describe('onDeployComponentSubmit', () => { expect(client.post).not.toHaveBeenCalled(); }); - it('falls back to the buffered path when the stream is unsupported', async () => { - mockStream.mockRejectedValue(new SSEUnsupportedError('no stream')); + it.each([400, 401, 403, 404, 406, 413, 429])( + 'falls back to the buffered path when the stream was refused with a %i', + async (status) => { + mockStream.mockRejectedValue(new SSEUnsupportedError('no stream', { status })); + const client = fakeClient({ message: 'buffered' }); + const result = await onDeployComponentSubmit({ + ...baseArgs, + useSSE: true, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: client as any, + }); + expect(result).toEqual({ message: 'buffered' }); + expect(client.post).toHaveBeenCalledOnce(); + }, + ); + + it.each([ + ['a gateway 502', 502], + ['a gateway 503', 503], + ['a gateway 504', 504], + ['a Harper 500', 500], + ['a 2xx that is not a stream', 200], + ['no response at all', undefined], + ])('never sends the deploy again after %s, which can follow a deploy already running', async (_, status) => { + const unsupported = new SSEUnsupportedError('no stream', { status }); + mockStream.mockRejectedValue(unsupported); const client = fakeClient({ message: 'buffered' }); - const result = await onDeployComponentSubmit({ + + const error = await onDeployComponentSubmit({ ...baseArgs, useSSE: true, // eslint-disable-next-line @typescript-eslint/no-explicit-any instanceClient: client as any, - }); - expect(result).toEqual({ message: 'buffered' }); - expect(client.post).toHaveBeenCalledOnce(); + }).catch((e: unknown) => e); + + expect(error).toBeInstanceOf(SSEInconclusiveError); + expect((error as Error).cause).toBe(unsupported); + expect(client.post).not.toHaveBeenCalled(); }); it('rethrows a deploy failure without re-POSTing (no double deploy)', async () => { @@ -79,3 +177,110 @@ describe('onDeployComponentSubmit', () => { expect(client.post).not.toHaveBeenCalled(); }); }); + +describe('deployAndFollowComponent', () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + function rolloutClient(deployed: object, jobRows: object[][]) { + let polls = 0; + return vi.fn((_url: string, body: { operation: string }) => { + if (body.operation === 'deploy_component') { + return Promise.resolve({ data: deployed }); + } + const rows = jobRows[Math.min(polls, jobRows.length - 1)]; + polls += 1; + return Promise.resolve({ data: rows }); + }); + } + + it('does not poll anything for a deploy that handed no restart to a job', async () => { + const post = rolloutClient({ message: 'Successfully deployed: my-app' }, []); + const onEvent = vi.fn(); + const result = await deployAndFollowComponent({ + ...baseArgs, + onEvent, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: { post } as any, + }); + expect(result).toEqual({ message: 'Successfully deployed: my-app' }); + expect(post).toHaveBeenCalledOnce(); + expect(onEvent).not.toHaveBeenCalled(); + expect(vi.getTimerCount()).toBe(0); + }); + + it('is not done until the restart job says how the other nodes took the release', async () => { + const post = rolloutClient( + { message: 'Successfully deployed: my-app, activating it', deployment_id: 'd1', restartJobId: 'job-1' }, + [ + [{ id: 'job-1', status: 'IN_PROGRESS' }], + [{ id: 'job-1', status: 'COMPLETE', result: { activated: [{ node: 'node-b', certification: 'certified' }] } }], + ], + ); + const onEvent = vi.fn(); + let settled = false; + const pending = deployAndFollowComponent({ + ...baseArgs, + onEvent, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: { post } as any, + }).finally(() => (settled = true)); + await vi.advanceTimersByTimeAsync(0); + expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(5_000); + const result = await pending; + expect(result.rollout).toMatchObject({ state: 'complete', nodes: [{ node: 'node-b', outcome: 'activated' }] }); + expect(post).toHaveBeenCalledWith('/', { operation: 'get_job', id: 'job-1' }, undefined); + expect(onEvent.mock.calls.map(([event]) => [event.type, event.data.state])).toEqual([ + ['rollout', 'following'], + ['rollout', 'complete'], + ]); + }); + + it("fails as a rollout failure, keeping the deployment's id, when another node rejects the release", async () => { + const post = rolloutClient({ message: 'ok', deployment_id: 'd1', restartJobId: 'job-1' }, [ + [{ + id: 'job-1', + status: 'ERROR', + message: { error: 'not activated on node-c', activated: [{ node: 'node-c', error: 'boom' }] }, + }], + ]); + const pending = deployAndFollowComponent({ + ...baseArgs, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: { post } as any, + }); + pending.catch(() => {}); + await vi.advanceTimersByTimeAsync(1_000); + await expect(pending).rejects.toBeInstanceOf(DeployRolloutError); + await expect(pending).rejects.toMatchObject({ message: 'not activated on node-c', deploymentId: 'd1' }); + // Following the job never deploys again. + expect(post.mock.calls.filter(([, body]) => body.operation === 'deploy_component')).toHaveLength(1); + }); + + it('resolves an unfollowable rollout as unknown rather than as a success or a failure', async () => { + const post = rolloutClient({ message: 'ok', restartJobId: 'job-1' }, [[]]); + const pending = deployAndFollowComponent({ + ...baseArgs, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: { post } as any, + }); + await vi.advanceTimersByTimeAsync(30_000); + expect((await pending).rollout).toMatchObject({ state: 'unfollowable', unfollowable: 'not-found' }); + }); + + it('follows a streamed deploy the same way', async () => { + mockStream.mockResolvedValue({ message: 'ok', restartJobId: 'job-1', certification: 'certified' }); + const post = rolloutClient({}, [[{ id: 'job-1', status: 'COMPLETE', result: { replicated: [] } }]]); + const pending = deployAndFollowComponent({ + ...baseArgs, + useSSE: true, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + instanceClient: { post } as any, + }); + await vi.advanceTimersByTimeAsync(1_000); + const result = await pending; + expect(result).toMatchObject({ certification: 'certified', rollout: { state: 'complete' } }); + expect(post.mock.calls.map(([, body]) => body.operation)).toEqual(['get_job']); + }); +}); diff --git a/src/integrations/api/instance/applications/deployComponent.ts b/src/integrations/api/instance/applications/deployComponent.ts index 1bad65093..d5c3e0ec9 100644 --- a/src/integrations/api/instance/applications/deployComponent.ts +++ b/src/integrations/api/instance/applications/deployComponent.ts @@ -1,17 +1,44 @@ import { InstanceClientConfig, InstanceTypeConfig } from '@/config/instanceClientConfig'; import { EntityIds } from '@/features/auth/store/authStore'; import { ReplicatedResponse } from '@/integrations/api/replication'; -import { SSEUnsupportedError } from '@/integrations/api/sse/errors'; +import { SSEInconclusiveError, SSEUnsupportedError } from '@/integrations/api/sse/errors'; import { useMutation } from '@tanstack/react-query'; import { DeployComponentResult, deployComponentStream, DeployStreamEvent } from './deployComponentStream'; +import { DeployRolloutError, followRestartJob } from './restartJob'; -export interface DeployComponentFormData { - applicationName: string; - applicationUrl: string; - installCommand?: string; +/** + * What a `deploy_component` call does. `stage` builds and installs the release on every node without making it + * live, and takes no `restart` (Harper refuses one); `activate` makes a staged or kept release live by its id and + * installs nothing. Both need Harper 5.3.0 on every node — an older node reads a stage as an ordinary deploy and + * serves it — so a caller offers them only on a version it could read. + */ +export type DeployRequest = + | { kind: 'deploy'; project: string; package: string; installCommand?: string } + | { kind: 'stage'; project: string; package: string; installCommand?: string } + | { kind: 'activate'; project: string; deploymentId: string }; + +export function deployRequestBody(request: DeployRequest, replicated: boolean): Record { + if (request.kind === 'activate') { + return { + operation: 'deploy_component', + project: request.project, + deployment_id: request.deploymentId, + replicated, + restart: 'rolling', + }; + } + return { + operation: 'deploy_component', + package: request.package, + project: request.project, + replicated, + install_command: request.installCommand || undefined, + ...(request.kind === 'stage' ? { activate: false } : { restart: 'rolling' }), + }; } -export interface DeployComponentArgs extends DeployComponentFormData, InstanceClientConfig, InstanceTypeConfig { +export interface DeployComponentArgs extends InstanceClientConfig, InstanceTypeConfig { + request: DeployRequest; /** Entity id, used to resolve the SSE connection. Present on `useInstanceClientIdParams`. */ entityId?: EntityIds; /** Opt into the live SSE deploy (only when the instance is Harper >= 5.1.0). */ @@ -22,10 +49,13 @@ export interface DeployComponentArgs extends DeployComponentFormData, InstanceCl signal?: AbortSignal; } +/** Only a 4xx proves Harper never started the deploy: applications/DESIGN.md, "A live deploy is sent once". */ +function refusedBeforeRunning({ status }: SSEUnsupportedError): boolean { + return status !== undefined && status >= 400 && status < 500; +} + export async function onDeployComponentSubmit({ - applicationName, - applicationUrl, - installCommand, + request, entityType, entityId, instanceClient, @@ -33,49 +63,60 @@ export async function onDeployComponentSubmit({ onEvent, signal, }: DeployComponentArgs): Promise { - const replicated = entityType === 'cluster'; + const body = deployRequestBody(request, entityType === 'cluster'); if (useSSE && entityId) { try { - return await deployComponentStream({ - connection: { id: entityId }, - package: applicationUrl, - project: applicationName, - replicated, - install_command: installCommand, - restart: 'rolling', - signal, - onEvent, - }); + return await deployComponentStream({ connection: { id: entityId }, body, signal, onEvent }); } catch (error) { - // Couldn't establish a stream (bad response/content-type/connection) and no deploy - // has started yet — safe to fall back to the proven buffered path below. if (!(error instanceof SSEUnsupportedError)) { // A terminal error, an inconclusive stream, or a deliberate abort. The deploy has // already started server-side, so we must NOT re-POST it (that would deploy // twice). Let the caller handle it (show the error, or poll on inconclusive). throw error; } + if (!refusedBeforeRunning(error)) { + throw new SSEInconclusiveError( + 'The live deploy did not stream and may already have started, so it was not sent again.', + { cause: error }, + ); + } } } - const { data } = await instanceClient.post( - '/', - { - operation: 'deploy_component', - package: applicationUrl, - project: applicationName, - replicated, - install_command: installCommand || undefined, - restart: 'rolling', - }, - { timeout: 300_000 }, - ); + const { data } = await instanceClient.post('/', body, { timeout: 300_000 }); return data as ReplicatedResponse; } -export function useDeployComponentMutation() { +/** + * Deploy, then — when the deploy handed its rolling restart to a job — follow that job, so the + * deploy is not done until every other node has taken the release, or said why it did not. + * + * A failed job rejects with {@link DeployRolloutError}. A job Studio could not follow resolves with + * `rollout.state === 'unfollowable'`, which no caller may report as success. + */ +export async function deployAndFollowComponent(args: DeployComponentArgs): Promise { + const result = await onDeployComponentSubmit(args); + const jobId = result?.restartJobId; + if (typeof jobId !== 'string' || !jobId) { + return result; + } + const rollout = await followRestartJob({ + instanceClient: args.instanceClient, + jobId, + signal: args.signal, + onUpdate: (snapshot) => args.onEvent?.({ type: 'rollout', data: snapshot }), + }); + if (rollout.state === 'failed') { + throw new DeployRolloutError(rollout, result.deployment_id); + } + return { ...result, rollout }; +} + +export function useDeployComponentMutation(mutationKey?: readonly unknown[], meta?: Record) { return useMutation({ - mutationFn: onDeployComponentSubmit, + mutationKey, + meta, + mutationFn: deployAndFollowComponent, }); } diff --git a/src/integrations/api/instance/applications/deployComponentStream.ts b/src/integrations/api/instance/applications/deployComponentStream.ts index 57cfe4bec..2d06e1406 100644 --- a/src/integrations/api/instance/applications/deployComponentStream.ts +++ b/src/integrations/api/instance/applications/deployComponentStream.ts @@ -1,11 +1,13 @@ import { ReplicatedResponse } from '@/integrations/api/replication'; -import { parseSSEData } from '@/integrations/api/sse/parseSSEStream'; +import { parseSSEData, SSEMessage } from '@/integrations/api/sse/parseSSEStream'; import { ResolveInstanceConnectionParams } from '@/integrations/api/sse/resolveInstanceConnection'; import { streamOperation } from '@/integrations/api/sse/streamOperation'; +import { DeployCertification } from './deployOutcome'; +import { RolloutSnapshot } from './restartJob'; /** Lifecycle phases Harper's `deploy_component` emits, in order. */ export const DEPLOY_PHASE_ORDER = ['prepare', 'load', 'replicate', 'restart', 'success'] as const; -export type DeployPhase = (typeof DEPLOY_PHASE_ORDER)[number]; +export const STAGE_PHASE_ORDER = ['prepare', 'load', 'replicate', 'staged'] as const; export interface DeployPhaseEvent { phase: string; @@ -29,76 +31,95 @@ export interface DeployPeerEvent { [key: string]: unknown; } -/** A live deploy event, discriminated by `type`, surfaced to the UI via `onEvent`. */ +/** Something the deploy wants a person to know without failing, e.g. a peer that installed different code (5.3.1+). */ +interface DeployWarningEvent { + message?: string; +} + +/** + * A live deploy event, discriminated by `type`, surfaced to the UI via `onEvent`. `rollout` is Studio's + * own: how the other nodes are taking the release, read from the restart job after the deploy answered. + */ export type DeployStreamEvent = | { type: 'phase'; data: DeployPhaseEvent } | { type: 'install'; data: DeployInstallEvent } - | { type: 'peer'; data: DeployPeerEvent }; + | { type: 'peer'; data: DeployPeerEvent } + | { type: 'warning'; data: DeployWarningEvent } + | { type: 'rollout'; data: RolloutSnapshot }; export interface DeployComponentResult extends ReplicatedResponse { deployment_id?: string; phase?: string; + certification?: DeployCertification; + staged?: boolean; + /** The `restart_service` job a rolling restart hands the other nodes to. */ + restartJobId?: string; + /** The `warning` events the deploy streamed. */ + warnings?: string[]; + /** How the other nodes took the release, once Studio followed `restartJobId`. */ + rollout?: RolloutSnapshot; +} + +export function toDeployStreamEvent(message: SSEMessage): DeployStreamEvent | undefined { + switch (message.event) { + case 'phase': + return { type: 'phase', data: parseSSEData(message) as DeployPhaseEvent }; + case 'install': + return { type: 'install', data: parseSSEData(message) as DeployInstallEvent }; + case 'peer': + return { type: 'peer', data: parseSSEData(message) as DeployPeerEvent }; + case 'warning': { + const data = parseSSEData(message); + return { + type: 'warning', + data: typeof data === 'string' ? { message: data } : (data as DeployWarningEvent | undefined) ?? {}, + }; + } + } + return undefined; } export interface DeployComponentStreamParams { connection: ResolveInstanceConnectionParams; - package: string; - project: string; - replicated: boolean; - install_command?: string; - restart?: string; + body: Record; signal?: AbortSignal; /** Called for each live deploy event so the UI can render progress. */ onEvent?: (event: DeployStreamEvent) => void; } /** - * Run `deploy_component` over SSE, forwarding live phase/install/peer events to `onEvent` + * Run `deploy_component` over SSE, forwarding live phase/install/peer/warning events to `onEvent` * and resolving with the terminal `done` result. * * Throws the transport errors from `@/integrations/api/sse/errors`: - * - `SSEUnsupportedError` — the server/proxy didn't stream; caller should retry buffered. + * - `SSEUnsupportedError` — the server/proxy didn't stream. Resend buffered only when its `status` is a 4xx + * refusal (`onDeployComponentSubmit` decides); anything else may follow a deploy already running. * - `SSEOperationError` — the deploy failed (terminal `error` event). * - `SSEInconclusiveError` — the stream ended without a verdict; caller must poll. */ export async function deployComponentStream({ connection, - package: packageRef, - project, - replicated, - install_command, - restart, + body, signal, onEvent, }: DeployComponentStreamParams): Promise { + const warnings: string[] = []; const result = await streamOperation({ connection, - body: { - operation: 'deploy_component', - package: packageRef, - project, - replicated, - install_command: install_command || undefined, - restart, - }, + body, signal, onMessage: (message) => { - if (!onEvent) { + const event = toDeployStreamEvent(message); + if (!event) { return; } - switch (message.event) { - case 'phase': - onEvent({ type: 'phase', data: parseSSEData(message) as DeployPhaseEvent }); - break; - case 'install': - onEvent({ type: 'install', data: parseSSEData(message) as DeployInstallEvent }); - break; - case 'peer': - onEvent({ type: 'peer', data: parseSSEData(message) as DeployPeerEvent }); - break; + if (event.type === 'warning' && typeof event.data.message === 'string' && event.data.message) { + warnings.push(event.data.message); } + onEvent?.(event); }, }); - return (result ?? {}) as DeployComponentResult; + const deployed = (result ?? {}) as DeployComponentResult; + return warnings.length ? { ...deployed, warnings } : deployed; } diff --git a/src/integrations/api/instance/applications/deployOutcome.test.ts b/src/integrations/api/instance/applications/deployOutcome.test.ts new file mode 100644 index 000000000..73deb7064 --- /dev/null +++ b/src/integrations/api/instance/applications/deployOutcome.test.ts @@ -0,0 +1,137 @@ +import { AxiosError } from 'axios'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +vi.mock('@/integrations/api/sse/resolveInstanceConnection', () => ({ + resolveInstanceConnection: () => ({ + url: 'https://host:9925/', + headers: { Accept: 'text/event-stream' }, + credentials: 'include' as RequestCredentials, + }), +})); + +import { deployRequestBody } from './deployComponent'; +import { deployComponentStream } from './deployComponentStream'; +import { deployFailureDetails, readCertificationDecision } from './deployOutcome'; +import { DeployRolloutError } from './restartJob'; + +// The decision a rejected deploy carries on both transports: harper `components/operations.js:1094` +// (buffered body) and `:1101-1109` (streamed `error` event), built by `certificationFailure` (:1222-1247). +const REJECTION = { + status: 'rejected', + reason: 'web.jsResource: Cannot find module ./missing.js', + failures: [{ key: 'web.jsResource', name: 'Error', message: 'Cannot find module ./missing.js', stack: '...' }], + restored: 'a3f8c2d1-0000-4000-8000-000000000000', + failed_closed: false, +}; +const MESSAGE = 'web was not deployed: release 9b2e failed to load in its canary worker: web.jsResource: ' + + 'Cannot find module ./missing.js. Deployment a3f8c2d1, the release it replaced, is live again. ' + + 'No other node received it. Deploy a fix, or activate deployment 9b2e again once the cause is fixed.'; + +function sseResponse(body: string): Response { + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(body)); + controller.close(); + }, + }); + return new Response(stream, { status: 200, headers: { 'content-type': 'text/event-stream' } }); +} + +function streamDeploy() { + return deployComponentStream({ + connection: { id: 'ins-1' }, + body: deployRequestBody({ kind: 'deploy', project: 'web', package: 'npm:web' }, true), + }); +} + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +describe('deployFailureDetails', () => { + it("reads the canary's decision off a streamed rejection", async () => { + const event = { message: MESSAGE, code: 400, phase: 'load', deployment_id: '9b2e', certification: REJECTION }; + vi.stubGlobal('fetch', vi.fn().mockResolvedValue(sseResponse(`event: error\ndata: ${JSON.stringify(event)}\n\n`))); + + const failure = await streamDeploy().catch((error: unknown) => error); + expect(deployFailureDetails(failure)).toEqual({ + deploymentId: '9b2e', + certification: { + status: 'rejected', + reason: REJECTION.reason, + restored: REJECTION.restored, + failedClosed: false, + }, + }); + }); + + it('reads the same decision off a buffered rejection', () => { + const error = new AxiosError('Request failed with status code 400'); + error.response = { + status: 400, + data: { error: MESSAGE, phase: 'load', deployment_id: '9b2e', certification: REJECTION }, + } as AxiosError['response']; + expect(deployFailureDetails(error)).toEqual({ + deploymentId: '9b2e', + certification: { + status: 'rejected', + reason: REJECTION.reason, + restored: REJECTION.restored, + failedClosed: false, + }, + }); + }); + + it('keeps the record id of a deploy whose restart job failed', () => { + const error = new DeployRolloutError({ jobId: 'job-1', state: 'failed', nodes: [] }, 'd1'); + expect(deployFailureDetails(error)).toEqual({ deploymentId: 'd1' }); + }); + + it('reads nothing from a failure that carries neither', () => { + expect(deployFailureDetails(new Error('boom'))).toEqual({}); + expect(deployFailureDetails('Request failed')).toEqual({}); + expect(deployFailureDetails(undefined)).toEqual({}); + }); +}); + +describe('readCertificationDecision', () => { + it('reads a release with nothing to go back to as failed closed', () => { + expect(readCertificationDecision({ status: 'rejected', reason: 'boom', restored: null, failed_closed: true })) + .toEqual({ status: 'rejected', reason: 'boom', restored: null, failedClosed: true }); + }); + + it.each([ + ['the success string', 'certified'], + ['no status', { reason: 'boom' }], + ['an array', [REJECTION]], + ['null', null], + ])('is not fooled by %s', (_label, value) => { + expect(readCertificationDecision(value)).toBeUndefined(); + }); +}); + +describe('deployComponentStream', () => { + it('forwards and keeps the warnings the deploy streamed', async () => { + const drift = "Install fingerprints differ from this node's on 1 of 2 peer node(s): node-b (source npm:web@1.5.0)."; + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue( + sseResponse( + `event: warning\ndata: ${JSON.stringify({ message: drift })}\n\n` + + `event: done\ndata: ${ + JSON.stringify({ result: { message: `Successfully deployed: web ${drift}`, certification: 'certified' } }) + }\n\n`, + ), + ), + ); + + const events: unknown[] = []; + const result = await deployComponentStream({ + connection: { id: 'ins-1' }, + body: deployRequestBody({ kind: 'deploy', project: 'web', package: 'npm:web' }, true), + onEvent: (event) => events.push(event), + }); + expect(events).toEqual([{ type: 'warning', data: { message: drift } }]); + expect(result).toMatchObject({ certification: 'certified', warnings: [drift] }); + }); +}); diff --git a/src/integrations/api/instance/applications/deployOutcome.ts b/src/integrations/api/instance/applications/deployOutcome.ts new file mode 100644 index 000000000..fd14761db --- /dev/null +++ b/src/integrations/api/instance/applications/deployOutcome.ts @@ -0,0 +1,61 @@ +import { SSEOperationError } from '@/integrations/api/sse/errors'; +import { DeployRolloutError } from './restartJob'; + +/** + * How a successful, unstaged restarting deploy was checked, from Harper 5.4.0. Open-ended: Harper can + * also decide `withdrawn`, and a newer version may add more. + */ +export type DeployCertification = 'certified' | 'uncertified' | 'unavailable' | 'not-requested' | (string & {}); + +/** What a canary decided about a release it refused. A rejected deploy carries it in place of the string. */ +export interface CertificationDecision { + /** `rejected`, or `interrupted` for a restart that stopped before the canary decided. */ + status: string; + reason?: string; + /** The release put back in its place; `null` when there was none to put back. */ + restored: string | null; + /** No earlier release to restore, so no worker on that node loads the component until it is deployed again. */ + failedClosed: boolean; +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +export function readCertificationDecision(value: unknown): CertificationDecision | undefined { + if (!isRecord(value) || typeof value.status !== 'string' || !value.status) { + return undefined; + } + return { + status: value.status, + reason: typeof value.reason === 'string' && value.reason ? value.reason : undefined, + restored: typeof value.restored === 'string' && value.restored ? value.restored : null, + failedClosed: value.failed_closed === true, + }; +} + +interface DeployFailureDetails { + deploymentId?: string; + certification?: CertificationDecision; +} + +/** + * The record id and canary decision a failed deploy carries, from whichever transport ran it: the + * streamed `error` event, the buffered 4xx/5xx body, or a restart job that failed after the deploy. + */ +export function deployFailureDetails(error: unknown): DeployFailureDetails { + if (error instanceof SSEOperationError) { + return { deploymentId: error.deploymentId, certification: readCertificationDecision(error.certification) }; + } + if (error instanceof DeployRolloutError) { + return { deploymentId: error.deploymentId }; + } + const data: unknown = (error as { response?: { data?: unknown } } | undefined)?.response?.data; + if (!isRecord(data)) { + return {}; + } + return { + deploymentId: typeof data.deployment_id === 'string' && data.deployment_id ? data.deployment_id : undefined, + certification: readCertificationDecision(data.certification), + }; +} diff --git a/src/integrations/api/instance/applications/restartJob.test.ts b/src/integrations/api/instance/applications/restartJob.test.ts new file mode 100644 index 000000000..294c956d4 --- /dev/null +++ b/src/integrations/api/instance/applications/restartJob.test.ts @@ -0,0 +1,314 @@ +import { AxiosError } from 'axios'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + DeployRolloutError, + followRestartJob, + readRestartJob, + rolloutPollDelayMs, + RolloutSnapshot, +} from './restartJob'; + +type InstanceClient = Parameters[0]['instanceClient']; + +// Fixtures follow the rows Harper writes: harper `server/jobs/jobProcess.ts:72-87` records the job, and +// `bin/restart.ts` builds both result shapes — `activated` (5.4+, :232/:235/:258/:260) and `replicated` (:363/:385/:397). +const certifiedActivation = { + id: 'job-1', + status: 'COMPLETE', + message: 'Successfully completed job: job-1', + result: { + activated: [ + { node: 'node-b', certification: 'certified', message: 'Successfully deployed: web, restarting Harper' }, + { node: 'node-c', certification: 'uncertified', message: 'Successfully deployed: web, restarting Harper' }, + ], + }, +}; + +const PEER_REJECTION = 'web was not deployed on this node: release 9b2e failed to load in its canary worker: ' + + 'web.jsResource: Cannot find module ./missing.js. Deployment a3f8, the release it replaced, is live again. ' + + 'Deploy a fix, or activate deployment 9b2e again once the cause is fixed.'; + +const rejectedActivation = { + id: 'job-1', + status: 'ERROR', + message: { + error: `Deployment 9b2e of web was not activated on 1 of 2 peer node(s): node-c (${PEER_REJECTION})`, + activated: [ + { node: 'node-b', certification: 'certified', message: 'Successfully deployed: web, restarting Harper' }, + { node: 'node-c', error: PEER_REJECTION }, + ], + }, +}; + +const legacyRestart = { + id: 'job-1', + status: 'COMPLETE', + message: 'Successfully completed job: job-1', + result: { + replicated: [ + { node: 'node-b', message: 'Successfully completed job: peer-job-b' }, + { node: 'node-c', message: 'connect ECONNREFUSED 10.0.0.3:9933' }, + ], + }, +}; + +describe('readRestartJob', () => { + it('keeps following a job that has not ended', () => { + expect(readRestartJob('job-1', { id: 'job-1', status: 'CREATED' }).state).toBe('following'); + expect(readRestartJob('job-1', { id: 'job-1', status: 'IN_PROGRESS' }).state).toBe('following'); + }); + + it("reads each node's own certification from a 5.4 activation job", () => { + expect(readRestartJob('job-1', certifiedActivation)).toEqual({ + jobId: 'job-1', + state: 'complete', + nodes: [ + { node: 'node-b', outcome: 'activated', certification: 'certified' }, + { node: 'node-c', outcome: 'activated', certification: 'uncertified' }, + ], + }); + }); + + it('names the node that rejected the release, with its reason, when the activation job fails', () => { + const snapshot = readRestartJob('job-1', rejectedActivation); + expect(snapshot.state).toBe('failed'); + expect(snapshot.nodes).toEqual([ + { node: 'node-b', outcome: 'activated', certification: 'certified' }, + { node: 'node-c', outcome: 'failed', detail: PEER_REJECTION }, + ]); + expect(snapshot.message).toBe(rejectedActivation.message.error); + }); + + it('does not read a node the legacy restart job could not reach as restarted, though the job completes', () => { + expect(readRestartJob('job-1', legacyRestart)).toEqual({ + jobId: 'job-1', + state: 'complete', + nodes: [ + { node: 'node-b', outcome: 'restarted' }, + { node: 'node-c', outcome: 'unconfirmed', detail: 'connect ECONNREFUSED 10.0.0.3:9933' }, + ], + }); + }); + + it("fails with the legacy job's text when a peer's restart failed", () => { + expect( + readRestartJob('job-1', { + id: 'job-1', + status: 'ERROR', + message: 'Timed out waiting for restart job to complete', + }), + ) + .toEqual({ + jobId: 'job-1', + state: 'failed', + nodes: [], + message: 'Timed out waiting for restart job to complete', + }); + }); + + it('reads a single instance, which has no other nodes, as complete', () => { + const snapshot = readRestartJob('job-1', { id: 'job-1', status: 'COMPLETE', result: { replicated: [] } }); + expect(snapshot).toEqual({ jobId: 'job-1', state: 'complete', nodes: [] }); + }); + + it.each([ + ['entries that are not objects', { status: 'COMPLETE', result: { activated: [null, 3, 'x'] } }, 'complete'], + ['an ERROR with no message', { status: 'ERROR' }, 'failed'], + ['a status in another case', { status: 'complete', result: { replicated: [] } }, 'complete'], + ['no status at all', {}, 'following'], + ])('never throws on %s', (_label, job, state) => { + expect(readRestartJob('job-1', { id: 'job-1', ...job } as never).state).toBe(state); + }); + + it.each([ + ['no result', { status: 'COMPLETE', message: 'Successfully completed job: job-1' }], + ['a result that is not an object', { status: 'COMPLETE', result: 'done' }], + ['a result in neither shape', { status: 'COMPLETE', result: { restarted: ['node-b'] } }], + ['a non-array activated list', { status: 'COMPLETE', result: { activated: 'node-b' } }], + ])('does not read a finished job with %s as a successful rollout', (_label, job) => { + expect(readRestartJob('job-1', { id: 'job-1', ...job } as never)).toEqual({ + jobId: 'job-1', + state: 'unfollowable', + nodes: [], + unfollowable: 'unrecognized', + }); + }); + + it('names an entry with no node rather than dropping it', () => { + const snapshot = readRestartJob('job-1', { + id: 'job-1', + status: 'ERROR', + message: { activated: [{ error: 'boom' }] }, + }); + expect(snapshot.nodes).toEqual([{ node: 'unknown node', outcome: 'failed', detail: 'boom' }]); + expect(snapshot.message).toBe('The restart job failed.'); + }); +}); + +function httpError(status: number) { + const error = new AxiosError(`Request failed with status code ${status}`); + error.response = { status } as AxiosError['response']; + return error; +} + +type Answer = { job: object } | { rows: [] } | { reject: unknown }; + +function makeClient(answers: Answer[]) { + let call = 0; + const post = vi.fn(() => { + const answer = answers[Math.min(call, answers.length - 1)]; + call += 1; + if ('reject' in answer) { + return Promise.reject(answer.reject); + } + return Promise.resolve({ data: 'rows' in answer ? [] : [answer.job] }); + }); + return { post, client: { post } as unknown as InstanceClient }; +} + +describe('followRestartJob', () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + it('polls get_job for that id until the job ends, reporting each snapshot', async () => { + const { post, client } = makeClient([ + { job: { id: 'job-1', status: 'IN_PROGRESS' } }, + { job: { id: 'job-1', status: 'IN_PROGRESS' } }, + { job: certifiedActivation }, + ]); + const updates: RolloutSnapshot[] = []; + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1', onUpdate: (s) => updates.push(s) }); + await vi.advanceTimersByTimeAsync(10_000); + const snapshot = await pending; + expect(snapshot.state).toBe('complete'); + expect(post).toHaveBeenCalledTimes(3); + expect(post).toHaveBeenCalledWith('/', { operation: 'get_job', id: 'job-1' }, undefined); + expect(updates.map((s) => s.state)).toEqual(['following', 'complete']); + }); + + it('keeps polling through dropped connections and 5xx while a node restarts', async () => { + const { client } = makeClient([ + { reject: new AxiosError('Network Error', 'ERR_NETWORK') }, + { reject: httpError(502) }, + { reject: httpError(503) }, + { job: legacyRestart }, + ]); + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1' }); + await vi.advanceTimersByTimeAsync(10_000); + expect((await pending).state).toBe('complete'); + }); + + it.each([400, 403])('stops following on a %s, which it would only get again', async (status) => { + const { post, client } = makeClient([{ reject: httpError(status) }]); + const snapshot = await followRestartJob({ instanceClient: client, jobId: 'job-1' }); + expect(snapshot).toMatchObject({ state: 'unfollowable', unfollowable: 'refused' }); + expect(post).toHaveBeenCalledOnce(); + }); + + it('gives up on a job the node it reaches does not have, but only once it has stayed missing', async () => { + const { client } = makeClient([{ rows: [] }]); + let settled: RolloutSnapshot | undefined; + void followRestartJob({ instanceClient: client, jobId: 'job-1' }).then((s) => (settled = s)); + await vi.advanceTimersByTimeAsync(18_000); + expect(settled).toBeUndefined(); + await vi.advanceTimersByTimeAsync(4_000); + expect(settled).toMatchObject({ state: 'unfollowable', unfollowable: 'not-found' }); + }); + + it('forgives a miss once the node that has the job answers again', async () => { + const { client } = makeClient([ + { rows: [] }, + ...Array.from({ length: 8 }, () => ({ rows: [] }) as Answer), + { job: { id: 'job-1', status: 'IN_PROGRESS' } }, + ...Array.from({ length: 8 }, () => ({ rows: [] }) as Answer), + { job: certifiedActivation }, + ]); + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1' }); + await vi.advanceTimersByTimeAsync(60_000); + expect((await pending).state).toBe('complete'); + }); + + it('does not count the time no node answered toward a missing job', async () => { + const { client } = makeClient([ + { rows: [] }, + ...Array.from({ length: 12 }, () => ({ reject: httpError(503) }) as Answer), + { rows: [] }, + { job: certifiedActivation }, + ]); + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1' }); + await vi.advanceTimersByTimeAsync(40_000); + expect((await pending).state).toBe('complete'); + }); + + it('stops at its ceiling without claiming the job ended', async () => { + const { client } = makeClient([{ job: { id: 'job-1', status: 'IN_PROGRESS' } }]); + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1', timeoutMs: 120_000 }); + await vi.advanceTimersByTimeAsync(130_000); + expect(await pending).toMatchObject({ state: 'unfollowable', unfollowable: 'timed-out' }); + }); + + it('slows down after the first minute instead of polling every 2s for half an hour', async () => { + expect(rolloutPollDelayMs(0, 2_000)).toBe(2_000); + expect(rolloutPollDelayMs(59_999, 2_000)).toBe(2_000); + expect(rolloutPollDelayMs(60_000, 2_000)).toBe(5_000); + expect(rolloutPollDelayMs(5 * 60_000, 2_000)).toBe(15_000); + + const { post, client } = makeClient([{ job: { id: 'job-1', status: 'IN_PROGRESS' } }]); + void followRestartJob({ instanceClient: client, jobId: 'job-1' }); + await vi.advanceTimersByTimeAsync(30 * 60_000); + // 30 in the first minute, 48 over the next four, 100 over the last 25. + expect(post.mock.calls.length).toBeLessThan(200); + }); + + it('rejects, without reporting the outcome, when its signal aborts while a poll is in flight', async () => { + let answer: (value: { data: object[] }) => void = () => {}; + const post = vi.fn((_url: string, _body: object, config?: { signal?: AbortSignal }) => { + expect(config?.signal).toBeInstanceOf(AbortSignal); + return new Promise<{ data: object[] }>((resolve) => (answer = resolve)); + }); + const controller = new AbortController(); + const updates: RolloutSnapshot[] = []; + const pending = followRestartJob({ + instanceClient: { post } as unknown as InstanceClient, + jobId: 'job-1', + signal: controller.signal, + onUpdate: (s) => updates.push(s), + }); + pending.catch(() => {}); + await vi.advanceTimersByTimeAsync(0); + controller.abort(new Error('cancelled')); + answer({ data: [certifiedActivation] }); + await expect(pending).rejects.toThrow('cancelled'); + expect(updates.map((s) => s.state)).toEqual(['following']); + }); + + it('rejects when its signal aborts', async () => { + const { client } = makeClient([{ job: { id: 'job-1', status: 'IN_PROGRESS' } }]); + const controller = new AbortController(); + const pending = followRestartJob({ instanceClient: client, jobId: 'job-1', signal: controller.signal }); + pending.catch(() => {}); + await vi.advanceTimersByTimeAsync(3_000); + controller.abort(new Error('cancelled')); + await vi.advanceTimersByTimeAsync(3_000); + await expect(pending).rejects.toThrow('cancelled'); + }); +}); + +describe('DeployRolloutError', () => { + it("carries the job's own text, which names each node that did not take the release", () => { + const rollout = readRestartJob('job-1', rejectedActivation); + const error = new DeployRolloutError(rollout, '9b2e'); + expect(error.message).toBe(rejectedActivation.message.error); + expect(error.deploymentId).toBe('9b2e'); + expect(error.rollout).toBe(rollout); + }); + + it('names the failed nodes itself when the job left no text', () => { + const error = new DeployRolloutError({ + jobId: 'job-1', + state: 'failed', + nodes: [{ node: 'node-c', outcome: 'failed', detail: 'boom' }], + }); + expect(error.message).toBe('Not every node took this release: node-c.'); + }); +}); diff --git a/src/integrations/api/instance/applications/restartJob.ts b/src/integrations/api/instance/applications/restartJob.ts new file mode 100644 index 000000000..d054e7c53 --- /dev/null +++ b/src/integrations/api/instance/applications/restartJob.ts @@ -0,0 +1,205 @@ +import { InstanceClientConfig } from '@/config/instanceClientConfig'; +import { getJob, HarperJob } from '@/integrations/api/instance/database/getJob'; +import { errorText } from '@/lib/errorText'; +import { sleep } from '@/lib/sleep'; +import { isDeterministicRejection } from '@/react-query/pollUnlessForbidden'; + +/** + * How a rolling deploy's other nodes took the release, read from the `restart_service` job the + * receiving node starts and names in `restartJobId`. + * + * `unfollowable` means Studio stopped without learning how the job ended, so it claims nothing. + */ +type RolloutState = 'following' | 'complete' | 'failed' | 'unfollowable'; + +/** + * `not-found`: the node Studio reached has no such job. `hdb_job` does not replicate, so only the node + * that received the deploy can answer for it, and a cluster's address may reach another node. + * `unrecognized`: the job ended with a result in neither shape Harper writes, so it says nothing about the nodes. + */ +type UnfollowableReason = 'not-found' | 'refused' | 'timed-out' | 'unrecognized'; + +/** + * `activated` and `failed` come from a Harper 5.4+ activation job, where each node certifies the + * release itself. `restarted` and `unconfirmed` come from the plain restart job older versions run: + * there a node the job could not reach is listed with the error, and the job still completes. + */ +type RolloutNodeOutcome = 'activated' | 'restarted' | 'failed' | 'unconfirmed'; + +export interface RolloutNode { + node: string; + outcome: RolloutNodeOutcome; + /** That node's own canary verdict (`certified`, `uncertified`, ...), on an activation job. */ + certification?: string; + detail?: string; +} + +export interface RolloutSnapshot { + jobId: string; + state: RolloutState; + nodes: RolloutNode[]; + message?: string; + unfollowable?: UnfollowableReason; +} + +// Harper's job worker records an object result under this message; a peer's restart job returns no +// result, so a peer that restarted reads exactly this, and one the job could not reach reads the error. +const PEER_RESTARTED = /^Successfully completed job\b/; + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function nodeName(entry: Record): string { + return typeof entry.node === 'string' && entry.node ? entry.node : 'unknown node'; +} + +function activatedNodes(entries: unknown[]): RolloutNode[] { + return entries.filter(isRecord).map((entry) => { + const error = errorText(entry.error); + if (error) { + return { node: nodeName(entry), outcome: 'failed', detail: error }; + } + return { + node: nodeName(entry), + outcome: 'activated', + certification: typeof entry.certification === 'string' ? entry.certification : undefined, + }; + }); +} + +function restartedNodes(entries: unknown[]): RolloutNode[] { + return entries.filter(isRecord).map((entry) => { + const message = errorText(entry.message); + return message && PEER_RESTARTED.test(message) + ? { node: nodeName(entry), outcome: 'restarted' } + : { node: nodeName(entry), outcome: 'unconfirmed', detail: message }; + }); +} + +export function readRestartJob(jobId: string, job: HarperJob): RolloutSnapshot { + const status = typeof job.status === 'string' ? job.status.toUpperCase() : undefined; + if (status === 'COMPLETE') { + const result = isRecord(job.result) ? job.result : undefined; + const nodes = Array.isArray(result?.activated) + ? activatedNodes(result.activated) + : Array.isArray(result?.replicated) + ? restartedNodes(result.replicated) + : undefined; + if (!nodes) { + return { jobId, state: 'unfollowable', nodes: [], unfollowable: 'unrecognized' }; + } + return { jobId, state: nodes.some((node) => node.outcome === 'failed') ? 'failed' : 'complete', nodes }; + } + if (status === 'ERROR') { + // A 5.4+ activation job fails with its report as the message: `{ error, activated }`. + const report = isRecord(job.message) ? job.message : undefined; + return { + jobId, + state: 'failed', + nodes: Array.isArray(report?.activated) ? activatedNodes(report.activated) : [], + message: errorText(report ? report.error : job.message) ?? 'The restart job failed.', + }; + } + return { jobId, state: 'following', nodes: [] }; +} + +// Fast while a rollout is most likely to finish, then slower, so a long rollout — or a node that is +// down — costs a few requests a minute rather than thirty. +export function rolloutPollDelayMs(elapsedMs: number, fastMs: number): number { + if (elapsedMs < 60_000) { + return fastMs; + } + return elapsedMs < 5 * 60_000 ? 5_000 : 15_000; +} + +// A cluster's address can reach a node other than the one that received the deploy, and reach that one +// again on the next request, so a missing row only counts once the job has stayed missing for a while. +const NOT_FOUND_GRACE_MS = 20_000; + +interface FollowRestartJobParams extends InstanceClientConfig { + jobId: string; + onUpdate?: (snapshot: RolloutSnapshot) => void; + signal?: AbortSignal; + pollIntervalMs?: number; + timeoutMs?: number; +} + +/** + * Poll `get_job` until the restart job ends, and say how each other node took the release. + * + * Harper writes the job's row when it starts and when it ends, so every node's outcome arrives at once. + * Rejects only when `signal` aborts: a job Studio could not follow resolves `unfollowable`. + */ +export async function followRestartJob({ + instanceClient, + jobId, + onUpdate, + signal, + pollIntervalMs = 2_000, + timeoutMs = 30 * 60_000, +}: FollowRestartJobParams): Promise { + const settle = (snapshot: RolloutSnapshot) => { + onUpdate?.(snapshot); + return snapshot; + }; + const unfollowable = (reason: UnfollowableReason) => + settle({ jobId, state: 'unfollowable', nodes: [], unfollowable: reason }); + const startedAt = Date.now(); + let missingSince: number | undefined; + onUpdate?.({ jobId, state: 'following', nodes: [] }); + for (;;) { + signal?.throwIfAborted(); + try { + const job = await getJob({ jobId, instanceClient, signal }); + signal?.throwIfAborted(); + if (job) { + missingSince = undefined; + const snapshot = readRestartJob(jobId, job); + if (snapshot.state !== 'following') { + return settle(snapshot); + } + } else { + missingSince ??= Date.now(); + if (Date.now() - missingSince >= NOT_FOUND_GRACE_MS) { + return unfollowable('not-found'); + } + } + } catch (error) { + signal?.throwIfAborted(); + // Anything else — a dropped connection, a 5xx while that node's own workers restart — passes. + if (isDeterministicRejection(error)) { + return unfollowable('refused'); + } + // An answer that never came is not a miss, so the grace needs an unbroken run of misses. The + // receiving node restarts its own workers first, while another node can still answer. + missingSince = undefined; + } + const elapsed = Date.now() - startedAt; + if (elapsed >= timeoutMs) { + return unfollowable('timed-out'); + } + await sleep(rolloutPollDelayMs(elapsed, pollIntervalMs)); + } +} + +/** + * A rolling deploy whose release is live on the node that received it, but whose restart job failed. + * Datadog withholds its message by this `name`: it quotes each node's error, which can carry the + * component's own load failure. + */ +export class DeployRolloutError extends Error { + readonly rollout: RolloutSnapshot; + readonly deploymentId?: string; + + constructor(rollout: RolloutSnapshot, deploymentId?: string) { + const failed = rollout.nodes.filter((node) => node.outcome === 'failed').map((node) => node.node); + super( + rollout.message + ?? (failed.length ? `Not every node took this release: ${failed.join(', ')}.` : 'The restart job failed.'), + ); + this.name = 'DeployRolloutError'; + this.rollout = rollout; + this.deploymentId = deploymentId; + } +} diff --git a/src/integrations/api/instance/database/getJob.test.ts b/src/integrations/api/instance/database/getJob.test.ts index b5fe44b10..f5ebdf970 100644 --- a/src/integrations/api/instance/database/getJob.test.ts +++ b/src/integrations/api/instance/database/getJob.test.ts @@ -3,7 +3,7 @@ import { ImportJobError, waitForJob } from './getJob'; type InstanceClient = Parameters[0]['instanceClient']; -function makeClient(responses: ({ status: string; message?: string } | { reject: true })[]) { +function makeClient(responses: ({ status: string; message?: unknown } | { reject: true })[]) { let call = 0; const post = vi.fn(() => { const job = responses[Math.min(call, responses.length - 1)]; @@ -57,6 +57,19 @@ describe('waitForJob', () => { await expect(pending).rejects.toBeInstanceOf(ImportJobError); }); + // harper `server/jobs/jobProcess.ts:86` (5.3.0+) records a refused load's permission report as the message. + it('reads a structured job message as text, never "[object Object]"', async () => { + const report = { + error: 'This operation is not authorized due to role restrictions and/or invalid database items', + unauthorized_access: [{ database: 'data', table: 'dog', required_table_permissions: ['insert'] }], + }; + const instanceClient = makeClient([{ status: 'ERROR', message: report }]); + const pending = waitForJob({ jobId: 'job-1', instanceClient, pollIntervalMs: 100 }); + pending.catch(() => {}); + await vi.advanceTimersByTimeAsync(100); + await expect(pending).rejects.toThrow(JSON.stringify(report)); + }); + it('tolerates transient get_job failures and keeps polling', async () => { const instanceClient = makeClient([ { status: 'IN_PROGRESS' }, diff --git a/src/integrations/api/instance/database/getJob.ts b/src/integrations/api/instance/database/getJob.ts index 8265aa1a5..62c2d566b 100644 --- a/src/integrations/api/instance/database/getJob.ts +++ b/src/integrations/api/instance/database/getJob.ts @@ -1,18 +1,27 @@ import { InstanceClientConfig } from '@/config/instanceClientConfig'; +import { errorText } from '@/lib/errorText'; export interface HarperJob { id: string; /** Lifecycle: CREATED → IN_PROGRESS → COMPLETE | ERROR. */ status: 'CREATED' | 'IN_PROGRESS' | 'COMPLETE' | 'ERROR' | string; - message?: string; + /** + * Usually text. From Harper 5.3.0 a job that fails with a structured report — a refused bulk load's + * permission report, a rolling activation's per-node outcomes — records the report object here. + */ + message?: unknown; + /** What a job that returned an object finished with; its `message` is then a generic sentence. */ + result?: unknown; job_body?: unknown; } -export async function getJob({ jobId, instanceClient }: { jobId: string } & InstanceClientConfig) { +export async function getJob( + { jobId, instanceClient, signal }: { jobId: string; signal?: AbortSignal } & InstanceClientConfig, +) { const { data } = await instanceClient.post('/', { operation: 'get_job', id: jobId, - }); + }, signal ? { signal } : undefined); return data?.[0]; } @@ -60,7 +69,7 @@ export async function waitForJob({ // CREATED (no worker picked it up yet), so anything else means keep polling. const status = job?.status?.toUpperCase(); if (status === 'ERROR') { - throw new ImportJobError(job?.message || 'The import job failed.'); + throw new ImportJobError(errorText(job?.message) ?? 'The import job failed.'); } if (status === 'COMPLETE' && job) { return job; diff --git a/src/integrations/api/instance/database/importData.ts b/src/integrations/api/instance/database/importData.ts index 29bfe73f9..359fecaa3 100644 --- a/src/integrations/api/instance/database/importData.ts +++ b/src/integrations/api/instance/database/importData.ts @@ -1,4 +1,5 @@ import { InstanceClientConfig } from '@/config/instanceClientConfig'; +import { errorText } from '@/lib/errorText'; import { useMutation } from '@tanstack/react-query'; import { AxiosError } from 'axios'; import { onAddCSVDataSubmit } from './addCSVData'; @@ -67,7 +68,7 @@ export async function importData({ case 'csv-data': { const { job_id } = await onAddCSVDataSubmit({ database, table, fileData: source.data, instanceClient }); const job = await waitForJob({ jobId: job_id, instanceClient }); - return { message: job.message || 'CSV data imported successfully' }; + return { message: errorText(job.message) ?? 'CSV data imported successfully' }; } case 'csv-url': { const { data } = await instanceClient.post<{ message: string; job_id: string }>('/', { @@ -78,7 +79,7 @@ export async function importData({ action: 'insert', }); const job = await waitForJob({ jobId: data.job_id, instanceClient }); - return { message: job.message || 'CSV data imported successfully' }; + return { message: errorText(job.message) ?? 'CSV data imported successfully' }; } case 'json-records': { const result = await insertTableRecords({ diff --git a/src/integrations/api/instance/deployments/deploymentPolling.test.ts b/src/integrations/api/instance/deployments/deploymentPolling.test.ts index 3ba1e9f83..3ceab67e9 100644 --- a/src/integrations/api/instance/deployments/deploymentPolling.test.ts +++ b/src/integrations/api/instance/deployments/deploymentPolling.test.ts @@ -36,6 +36,7 @@ describe('get_deployment polling', () => { const poll = detailInterval(); expect(poll({ state: { error: null, data: deployment('installing') } })).toBe(2_000); expect(poll({ state: { error: null, data: deployment('success') } })).toBe(false); + expect(poll({ state: { error: null, data: deployment('staged') } })).toBe(false); }); it('does not poll a detail view that never asked to', () => { @@ -62,6 +63,7 @@ describe('list_deployments polling', () => { const poll = listInterval() as IntervalFn; expect(poll({ state: { error: null, data: list('success', 'installing') } })).toBe(2_000); expect(poll({ state: { error: null, data: list('success', 'failed') } })).toBe(10_000); + expect(poll({ state: { error: null, data: list('success', 'staged') } })).toBe(10_000); expect(poll({ state: { error: null } })).toBe(10_000); }); diff --git a/src/integrations/api/instance/deployments/getDeployment.ts b/src/integrations/api/instance/deployments/getDeployment.ts index 8b3cb2b76..6249a6530 100644 --- a/src/integrations/api/instance/deployments/getDeployment.ts +++ b/src/integrations/api/instance/deployments/getDeployment.ts @@ -1,12 +1,6 @@ import { InstanceClientIdConfig } from '@/config/instanceClientConfig'; import { EntityIds } from '@/features/auth/store/authStore'; -import { - DeployInstallEvent, - DeployPeerEvent, - DeployPhaseEvent, - DeployStreamEvent, -} from '@/integrations/api/instance/applications/deployComponentStream'; -import { parseSSEData } from '@/integrations/api/sse/parseSSEStream'; +import { DeployStreamEvent, toDeployStreamEvent } from '@/integrations/api/instance/applications/deployComponentStream'; import { ResolveInstanceConnectionParams } from '@/integrations/api/sse/resolveInstanceConnection'; import { streamOperation } from '@/integrations/api/sse/streamOperation'; import { pollUnlessForbidden } from '@/react-query/pollUnlessForbidden'; @@ -70,19 +64,9 @@ export async function getDeploymentStream({ body: { operation: 'get_deployment', deployment_id: deploymentId }, signal, onMessage: (message) => { - if (!onEvent) { - return; - } - switch (message.event) { - case 'phase': - onEvent({ type: 'phase', data: parseSSEData(message) as DeployPhaseEvent }); - break; - case 'install': - onEvent({ type: 'install', data: parseSSEData(message) as DeployInstallEvent }); - break; - case 'peer': - onEvent({ type: 'peer', data: parseSSEData(message) as DeployPeerEvent }); - break; + const event = onEvent && toDeployStreamEvent(message); + if (event) { + onEvent(event); } }, }); diff --git a/src/integrations/api/instance/deployments/types.test.ts b/src/integrations/api/instance/deployments/types.test.ts index 3c2dd27ef..3c21e25ca 100644 --- a/src/integrations/api/instance/deployments/types.test.ts +++ b/src/integrations/api/instance/deployments/types.test.ts @@ -12,6 +12,7 @@ describe('isTerminalDeploymentStatus', () => { ['success', true], ['failed', true], ['rolled_back', true], + ['staged', true], ])('%s -> terminal=%s', (status, expected) => { expect(isTerminalDeploymentStatus(status)).toBe(expected); }); diff --git a/src/integrations/api/instance/deployments/types.ts b/src/integrations/api/instance/deployments/types.ts index f86693480..9c9254c8b 100644 --- a/src/integrations/api/instance/deployments/types.ts +++ b/src/integrations/api/instance/deployments/types.ts @@ -8,12 +8,16 @@ export type DeploymentStatus = | 'restarting' | 'success' | 'failed' - | 'rolled_back'; + | 'rolled_back' + /** Built and installed on every node, not live (5.3.0+). Activating it writes a record of its own. */ + | 'staged'; +// Harper's own set (harper `components/deploymentOperations.ts:19`): a staged record never changes again. export const TERMINAL_DEPLOYMENT_STATUSES: ReadonlySet = new Set([ 'success', 'failed', 'rolled_back', + 'staged', ]); export function isTerminalDeploymentStatus(status: DeploymentStatus | undefined): boolean { @@ -27,6 +31,15 @@ export interface DeploymentEventLogEntry { data: unknown; } +/** + * What an install came from (5.3.1+): `npm:@`, `git:`, `integrity:` or `unidentified`, + * absent for a payload; and the sha256 of each root lockfile, or why it could not be read. + */ +export interface InstallFingerprint { + source?: string; + lockfiles?: Record; +} + export interface DeploymentPeerResult { node: string; status?: string; @@ -34,6 +47,12 @@ export interface DeploymentPeerResult { error?: string | DeploymentPeerError | null; started_at?: number; completed_at?: number; + /** Present once the origin compared this peer's install with its own (5.3.1+). */ + install?: InstallFingerprint | null; + /** `false` when anything differs, `null` when the evidence is missing or unidentified, `true` when it all matches. */ + install_matches?: boolean | null; + /** `source` and/or the lockfile names that differ. */ + install_differs?: string[]; } export interface DeploymentPeerError { @@ -68,8 +87,10 @@ export interface Deployment { started_at?: number; completed_at?: number; user?: string; - restorable?: boolean; - rollback_of?: string | null; + /** On an activation (5.3.0+), the deployment whose staged or kept release it made live. */ + activated_from?: string | null; + /** The origin's own install, on a deploy or a stage (5.3.1+); an activation installs nothing. */ + install_fingerprint?: InstallFingerprint | null; error?: DeploymentError | null; } diff --git a/src/integrations/api/sse/errors.ts b/src/integrations/api/sse/errors.ts index e67f10611..66911be20 100644 --- a/src/integrations/api/sse/errors.ts +++ b/src/integrations/api/sse/errors.ts @@ -8,12 +8,16 @@ * The server did not (or could not) speak SSE for this request: the response was not OK, * arrived without a `text/event-stream` content-type (e.g. an older Harper that ignores * the `Accept` header, or a proxy that rewrote it), or the connection failed before any - * bytes streamed. The caller should fall back to the classic buffered operation. + * bytes streamed. A read can fall back to the classic buffered operation; a write only when + * `status` shows the server refused it (see `onDeployComponentSubmit`). */ export class SSEUnsupportedError extends Error { - constructor(message: string, options?: { cause?: unknown }) { + readonly status?: number; + + constructor(message: string, options?: { cause?: unknown; status?: number }) { super(message, options); this.name = 'SSEUnsupportedError'; + this.status = options?.status; } } @@ -38,12 +42,18 @@ export class SSEOperationError extends Error { readonly code?: string | number; readonly phase?: string; readonly deploymentId?: string; + /** A deploy its canary refused carries the decision here, as sent (Harper 5.4.0+). */ + readonly certification?: unknown; - constructor(message: string, details?: { code?: string | number; phase?: string; deploymentId?: string }) { + constructor( + message: string, + details?: { code?: string | number; phase?: string; deploymentId?: string; certification?: unknown }, + ) { super(message); this.name = 'SSEOperationError'; this.code = details?.code; this.phase = details?.phase; this.deploymentId = details?.deploymentId; + this.certification = details?.certification; } } diff --git a/src/integrations/api/sse/streamOperation.test.ts b/src/integrations/api/sse/streamOperation.test.ts index 734733250..2c667f123 100644 --- a/src/integrations/api/sse/streamOperation.test.ts +++ b/src/integrations/api/sse/streamOperation.test.ts @@ -109,23 +109,39 @@ describe('streamOperation', () => { ).rejects.toMatchObject({ name: 'SSEOperationError', message: 'The operation failed.' }); }); - it('falls back (SSEUnsupportedError) when the response is not an event stream', async () => { + it('falls back (SSEUnsupportedError) when the response is not an event stream, with its status', async () => { vi.stubGlobal( 'fetch', vi.fn().mockResolvedValue(sseResponse('{"message":"ok"}', { contentType: 'application/json' })), ); - await expect( - streamOperation({ connection: { id: 'ins-1' }, body: { operation: 'deploy_component' } }), - ).rejects.toBeInstanceOf(SSEUnsupportedError); + const error = await streamOperation({ connection: { id: 'ins-1' }, body: { operation: 'deploy_component' } }) + .catch((e: unknown) => e); + + expect(error).toBeInstanceOf(SSEUnsupportedError); + expect((error as SSEUnsupportedError).status).toBe(200); }); - it('falls back (SSEUnsupportedError) when fetch rejects before streaming', async () => { + it('reports the status of a response that refused the stream', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue(sseResponse('{"error":"Gateway Timeout"}', { status: 504, contentType: 'text/html' })), + ); + + const error = await streamOperation({ connection: { id: 'ins-1' }, body: { operation: 'deploy_component' } }) + .catch((e: unknown) => e); + + expect((error as SSEUnsupportedError).status).toBe(504); + }); + + it('falls back (SSEUnsupportedError) when fetch rejects before streaming, with no status', async () => { vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new TypeError('network down'))); - await expect( - streamOperation({ connection: { id: 'ins-1' }, body: { operation: 'deploy_component' } }), - ).rejects.toBeInstanceOf(SSEUnsupportedError); + const error = await streamOperation({ connection: { id: 'ins-1' }, body: { operation: 'deploy_component' } }) + .catch((e: unknown) => e); + + expect(error).toBeInstanceOf(SSEUnsupportedError); + expect((error as SSEUnsupportedError).status).toBeUndefined(); }); it('throws SSEInconclusiveError when the stream ends without a terminal event', async () => { diff --git a/src/integrations/api/sse/streamOperation.ts b/src/integrations/api/sse/streamOperation.ts index d0917a7b3..206c1051d 100644 --- a/src/integrations/api/sse/streamOperation.ts +++ b/src/integrations/api/sse/streamOperation.ts @@ -25,8 +25,8 @@ export interface StreamOperationOptions { * * Failure modes are explicit (see `./errors`): * - {@link SSEUnsupportedError} — couldn't establish a stream (bad response, wrong - * content-type, or connection error before any bytes). Caller should fall back to the - * buffered axios path. + * content-type, or connection error before any bytes). A read can fall back to the buffered + * axios path; the request may still have reached the server, so a write must check `status`. * - {@link SSEOperationError} — a terminal `error` event arrived. * - {@link SSEInconclusiveError} — the stream ended (or idled out) without a terminal * event. The outcome is unknown; caller must poll to determine it, never assume success. @@ -70,8 +70,8 @@ export async function streamOperation({ if (signal?.aborted) { throw error; } - // Anything else (network error, DNS, idle before headers) means we never got a - // stream going; let the caller fall back to the buffered request. + // Anything else (network error, DNS, idle before headers) means we never got a stream + // going, though the request may have been sent: there is no `status` to say otherwise. throw new SSEUnsupportedError('Failed to open the SSE stream.', { cause: error }); } @@ -81,6 +81,7 @@ export async function streamOperation({ await response.body?.cancel().catch(() => {}); throw new SSEUnsupportedError( `Server did not return an event stream (status ${response.status}, content-type "${contentType}").`, + { status: response.status }, ); } @@ -104,7 +105,13 @@ export async function streamOperation({ if (!data || typeof data !== 'object') { throw new SSEOperationError(typeof data === 'string' && data ? data : 'The operation failed.'); } - const err = data as { message?: string; code?: string | number; phase?: string; deployment_id?: string }; + const err = data as { + message?: string; + code?: string | number; + phase?: string; + deployment_id?: string; + certification?: unknown; + }; // A non-string message (a structured/nested error object) would stringify to // "[object Object]" in every toast that renders it — extract its nested // message (or JSON) instead of discarding the detail (#1426). @@ -112,6 +119,7 @@ export async function streamOperation({ code: err.code, phase: err.phase, deploymentId: err.deployment_id, + certification: err.certification, }); } } diff --git a/src/integrations/datadog/redactRelayedMessage.test.ts b/src/integrations/datadog/redactRelayedMessage.test.ts index 1f18a070d..da530c0aa 100644 --- a/src/integrations/datadog/redactRelayedMessage.test.ts +++ b/src/integrations/datadog/redactRelayedMessage.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'vitest'; +import { DeployRolloutError } from '../api/instance/applications/restartJob'; import { ImportJobError } from '../api/instance/database/getJob'; import { SSEOperationError } from '../api/sse/errors'; import { redactRelayedMessage, redactRelayedStack } from './redactRelayedMessage'; @@ -64,6 +65,21 @@ describe('redactRelayedMessage', () => { .toBe(WITHHELD + frames); }); + it("withholds a rolling deploy's failed restart job, whose message quotes each node's error", () => { + const error = new DeployRolloutError({ + jobId: 'job-1', + state: 'failed', + nodes: [{ node: 'node-b', outcome: 'failed', detail: 'acme-billing.jsResource: Cannot find module' }], + message: 'Deployment d1 of acme-billing was not activated on 1 of 1 peer node(s): node-b (acme-billing...)', + }); + expect(error.name).toBe('DeployRolloutError'); + expect(redactRelayedMessage(error.name, error.message)).toBe(WITHHELD); + const frames = '\n at deployAndFollowComponent @ https://fabric.harper.fast/assets/index-A1b2C3d4.js:5:1234'; + const redacted = redactRelayedStack(error.name, error.message, `DeployRolloutError: ${error.message}${frames}`); + expect(redacted).not.toContain('acme-billing'); + expect(redacted).toBe(WITHHELD + frames); + }); + it('removes exactly the message and keeps what follows', () => { const message = [ 'SSEOperationError: Failed to clone package github:gh repo clone acme-corp/billing: fatal: Too many arguments.', diff --git a/src/integrations/datadog/redactRelayedMessage.ts b/src/integrations/datadog/redactRelayedMessage.ts index 98f548e5b..d279926cc 100644 --- a/src/integrations/datadog/redactRelayedMessage.ts +++ b/src/integrations/datadog/redactRelayedMessage.ts @@ -19,9 +19,10 @@ * `SSEOperationError`'s `code` and `phase`, which reach the browser but are not on the RUM event; * reporting them deliberately would restore that detail without the message. * - * `ImportJobError` is the same relay for a failed load job, whose text quotes the rejected rows. + * `ImportJobError` is the same relay for a failed load job, whose text quotes the rejected rows, and + * `DeployRolloutError` for a rolling deploy's failed restart job, whose text quotes each node's error. */ -const RELAYED_ERROR_TYPES = new Set(['SSEOperationError', 'ImportJobError']); +const RELAYED_ERROR_TYPES = new Set(['SSEOperationError', 'ImportJobError', 'DeployRolloutError']); const WITHHELD = 'Harper reported an operation failure (server message withheld).';