Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/rule-poll-transient-failures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@taskless/cli": patch
---

`rule create` and `rule improve` no longer give up on the first transient failure while waiting for a generation request or fetching the rules it produced. A network failure, `408`, `429`, or `5xx` is retried on the next 15-second poll, and the command fails only after 8 in a row. A rejected token, a documented error, or a response the CLI cannot read still fails immediately.

Both commands now take `--resume <requestId>` to pick up a request a previous run submitted, instead of submitting a new one. When a command gives up, its message names the request id, says the request may still complete on the service, and gives the `--resume` command to run.
80 changes: 80 additions & 0 deletions openspec/specs/cli-rules/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,86 @@ capability requirements.
- **WHEN** `taskless rule create` is waiting on the API
- **THEN** the CLI SHALL report progress rather than appearing to hang

### Requirement: Rule generation tolerates transient service failures

While `taskless rule create` or `taskless rule improve` polls a submitted request, and while it
fetches the rules that request produced, an `unavailable` outcome marked `retryable` (a network
failure, `408`, `429`, or a `5xx`) SHALL NOT end the command. The CLI SHALL report the failure as
progress and try again after the poll interval, and SHALL give up only after 8 such outcomes in a
row. Any other answer SHALL reset that count. A non-retryable `unavailable` (an undocumented `4xx`
or a malformed body), a documented `error`, a refusal, or a `401` SHALL still fail on the first
occurrence. When the CLI gives up, the message SHALL name the request id, SHALL say the request
was not cancelled and may still complete, and SHALL name the `--resume` command that picks it up
again. All rules SHALL still be fetched before any is written, so a fetch that gives up writes
nothing.

#### Scenario: A transient run is ridden out

- **WHEN** polling answers `503`, `429`, and a network failure before reaching `generated`
- **THEN** the CLI SHALL keep polling and deliver the rules
- **AND** SHALL have submitted exactly one request

#### Scenario: Persistent failure gives up without implying a resubmit

- **WHEN** polling answers a retryable failure 8 times in a row
- **THEN** the CLI SHALL fail with `NETWORK_ERROR`
- **AND** the message SHALL name the request id, say it may still complete, and name
`taskless rule <create|improve> --resume <requestId>`

#### Scenario: An answer in between resets the count

- **WHEN** polling answers 7 retryable failures, then `building`, then 7 more, then `generated`
- **THEN** the CLI SHALL deliver the rules

#### Scenario: A non-retryable failure fails at once

- **WHEN** polling answers an undocumented `4xx`, or `401`
- **THEN** the CLI SHALL fail on that answer without polling again

#### Scenario: Fetching a generated rule is retried the same way

- **WHEN** `GET /cli/api/v2/rule/{ruleId}` answers a retryable failure 8 times in a row
- **THEN** the CLI SHALL fail with `NETWORK_ERROR`, write no rules, and name the `--resume` command

### Requirement: Rules create and improve resume a submitted request

`taskless rule create` and `taskless rule improve` SHALL accept `--resume <requestId>`, the request
id a previous run of the same command printed. With it, the CLI SHALL NOT submit anything: it SHALL
poll that request and fetch, verify, and write what it produced exactly as it would have after
submitting it, and SHALL report the same output, with `requestId` set to the resumed id.
`--resume` SHALL NOT be combined with `--from`, and its value SHALL be a UUID; either mistake SHALL
fail with `INVALID_INPUT` before any service call.
A `requestId` the service returns from submit or iterate SHALL also be a UUID, since the CLI prints
it inside a `--resume` command an agent is told to run; any other value SHALL be treated as an
invalid response body and SHALL NOT be printed.

#### Scenario: A resumed request is not resubmitted

- **WHEN** a user runs `taskless rule create --resume <requestId>` after a run that gave up on it
- **THEN** the CLI SHALL poll `GET /cli/api/v2/request/{requestId}` and write the produced rules
- **AND** SHALL NOT call `POST /cli/api/v2/request`

#### Scenario: Improve resumes without iterating again

- **WHEN** a user runs `taskless rule improve --resume <requestId>`
- **THEN** the CLI SHALL NOT call `POST /cli/api/v2/rule/{ruleId}/iterate`

#### Scenario: --resume with --from is refused

- **WHEN** a user runs `taskless rule create --resume <requestId> --from req.json`
- **THEN** the CLI SHALL fail with `INVALID_INPUT` without calling the service

#### Scenario: A server-authored request id is never put on a command line

- **WHEN** submit answers `200` with a `requestId` that is not a UUID, such as `x; rm -rf ~`
- **THEN** the CLI SHALL fail with `NETWORK_ERROR` as an invalid response body
- **AND** SHALL NOT print that value or poll it

#### Scenario: A rule id is not a request id

- **WHEN** a user runs `taskless rule improve --resume no-eval-3fa9c21b`
- **THEN** the CLI SHALL fail with `INVALID_INPUT` without calling the service

### Requirement: Rules improve reads request from file

`taskless rule improve` SHALL accept a `--from <file>` flag specifying a JSON file containing the
Expand Down
9 changes: 5 additions & 4 deletions packages/cli/src/agent/create-remote-rule.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Topic: create-remote-rule (CLI v%(CLI_VERSION)s / topic v4)
# Topic: create-remote-rule (CLI v%(CLI_VERSION)s / topic v5)

## You are here
This is `create-remote-rule`. It helps you have the Taskless service
Expand Down Expand Up @@ -137,8 +137,9 @@ Two ways to legitimately be here:
the id is the rule's directory name (for example
`no-eval-3fa9c21b`). It is what `rule improve`, `rule restore`, and
`rule rollback` take, and it is on disk, so nothing needs recording.
`requestId` names the generation request only; no command takes it
back, and passing it to `rule improve` fails with `RULE_NOT_FOUND`.
`requestId` names the generation request only. The one command that
takes it back is `rule create --resume <requestId>` (see Errors);
passing it to `rule improve` as a rule id fails with `RULE_NOT_FOUND`.

**Read `notices` if it is present.** It is an optional array of
advisory messages about a delivery that was written anyway. A rule
Expand Down Expand Up @@ -181,7 +182,7 @@ With `--json`, failures emit `{ ok: false, code, message }`:
| `NO_ORIGIN_REMOTE` | git repository, no `origin` | route to a local recipe; `auth login` cannot fix it |
| `UNSUPPORTED_REMOTE_HOST`| `origin` is not GitHub | route to a local recipe; `auth login` cannot fix it |
| `INVALID_INPUT` | `--from` JSON failed validation | re-read the input schema, fix, retry |
| `NETWORK_ERROR` | submit/poll failed | report and suggest retry |
| `NETWORK_ERROR` | submit/poll failed | the CLI already retried transient failures. If the message names a `--resume` command, run that to keep waiting on the same request; re-running with `--from` submits a NEW request, so confirm with the user first |
| `RULE_GENERATION_FAILED` | the service failed to generate | report the message; suggest enriching prompt |
| `RULE_UNSUPPORTED` | plan lacks this generation type | tell the user to enable it; do not retry |

Expand Down
4 changes: 2 additions & 2 deletions packages/cli/src/agent/improve-rule.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Topic: improve-rule (CLI v%(CLI_VERSION)s / topic v6)
# Topic: improve-rule (CLI v%(CLI_VERSION)s / topic v7)

## Goal
Iterate on an existing Taskless rule. The CLI submits the user's
Expand Down Expand Up @@ -128,7 +128,7 @@ When `--json` is set, failures emit `{ ok: false, code, message }`:
| `UNSUPPORTED_REMOTE_HOST`| `origin` is not GitHub | tell the user; `auth login` cannot fix it |
| `INVALID_INPUT` | `--from` JSON failed validation | re-read input schema, fix, retry |
| `RULE_NOT_FOUND` | the service did not issue this rule | re-check the directory name; a local or pre-0.12.0 rule needs the anonymous flow |
| `NETWORK_ERROR` | API submit/poll failed | report and suggest retry |
| `NETWORK_ERROR` | API submit/poll failed | the CLI already retried transient failures. If the message names a `--resume` command, run that to keep waiting on the same request; re-running with `--from` submits a NEW request, so confirm with the user first |
| `RULE_GENERATION_FAILED` | API returned a generation failure | report; suggest enriching guidance/references |
| `RULE_UNSUPPORTED` | plan lacks this generation type | tell the user to enable it; do not retry |

Expand Down
32 changes: 30 additions & 2 deletions packages/cli/src/api/v2.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import createClient from "openapi-fetch";
import { z } from "zod";

import type { paths } from "../generated/api-v2";
import { getApiBaseUrl } from "./config";
Expand Down Expand Up @@ -431,6 +432,33 @@ export type RequestBody = NonNullable<

export type RequestAccepted = OkBody<"/cli/api/v2/request", "post">;

/**
* Whether `value` has the shape of a request id, which the service documents
* as a UUID. `guid`, not `uuid`: the shape is what matters, not the RFC
* version bits.
*
* A request id is printed inside a `--resume` command that an agent is told
* to run, so one that is anything else is refused rather than passed on: it
* would put server-authored text on a command line.
*/
export function isRequestId(value: unknown): value is string {
return z.guid().safeParse(value).success;
}

/** Accept a submit or iterate body only when it names a well-formed request. */
function acceptRequest<C extends string>(
data: unknown
): V2Outcome<RequestAccepted, C> {
if (!isRecord(data) || !isRequestId(data.requestId)) {
return {
status: "unavailable",
reason: "invalid response body",
retryable: false,
};
}
return { status: "ok", data: data as RequestAccepted };
}

export type RequestCode = ErrorCode<"/cli/api/v2/request", "post">;

const REQUEST_CODES = errorCodes<ErrorCode<"/cli/api/v2/request", "post">>()([
Expand All @@ -448,7 +476,7 @@ export function submitRequest(
return settle<RequestAccepted, RequestCode>(
() => client.POST("/cli/api/v2/request", { body }),
REQUEST_CODES,
acceptObject
acceptRequest
);
}

Expand Down Expand Up @@ -520,7 +548,7 @@ export function iterateRule(
body,
}),
ITERATE_CODES,
acceptObject
acceptRequest
);
}

Expand Down
Loading
Loading