Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions platform/deploy-operate/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,46 @@ No migration. `[governance.opa] enabled` stays `false` by default. See [Trino on
- **Behaviour:** an OPA request whose body does not arrive within `body_read_timeout_ms` is answered 408 and counted in
the new `egret_opa_unread_total`. Restrict `/opa/` to the engines' network: it is now required.

### Workflow outputs: Loop outputs, Loop retry and flow outputs (platform #840)

One migration adds columns for execution and task-run outputs; it only adds columns and applies at boot. There is no backfill: an
execution that ended before the upgrade has no outputs. **Behaviour change for stored flows:** a flow saved earlier can give a
different answer, because each change makes egret do what Kestra does. See [Flow control](../workflows/flows/flow-control.md#loop-outputs)
and [Flow outputs](../workflows/flows/inputs-and-outputs.md#flow-outputs).

| Flow | Before | Now |
|---|---|---|
| A `Loop` with `outputs:` | Accepted, never evaluated; `{{ outputs.loop.outputs }}` failed to render | Evaluated at the end of each iteration and collected in `outputs.<loop>.outputs` |
| Any `Loop` run | No counters | `iterationCount`, `runningIterations` and `terminatedIterations` |
| A `Loop` output that fails to render or convert | Ignored | The iteration gets `FAILED`; with `transmitFailed: true` the `Loop` fails |
| `fetchType: STORE` (or `AUTO` over a `kestra://`, `file://` or `nsfile://` `values:`) | Ignored | One `outputs.ion` file per iteration; the list holds `{uri}` |
| A `retry:` on a `Loop` or on an ancestor of it | Dropped for the tasks inside each iteration | Inherited by every task and `Subflow` of each iteration |
| A failed flowable task under an ancestor's or the flow's `retry:` | Its parent stayed `RUNNING` forever | Its parent ends `FAILED` (bug fix) |
| A flow-level `outputs:` | Never evaluated | Evaluated when the execution ends; a failure ends it `FAILED` |
| `{{ execution.outputs }}` and a `Subflow` parent's `outputs.<id>.outputs` | Always `{}` | The evaluated outputs; a child whose declared outputs were not stored fails the `Subflow` run |
| `{{ trigger.outputs }}` in a Flow trigger's `inputs:` | Failed (`trigger` unknown) | The triggering execution's outputs |
| `GET` of one execution | No outputs | An `outputs` field (`SECRET` values are `******`); the list does not carry it |
| A parent reading its `Subflow` child's `FILE` output with `read()` | Refused | Allowed, for the files the child published |

Limits that exist in Egret only (Kestra has none):

- One `Loop` iteration's outputs may serialise to at most 1 MiB and a `Loop`'s whole list to 16 MiB; a flow's outputs to 16 MiB per
execution. Beyond a limit the iteration or execution fails with a message that points to `fetchType: STORE` or to a `FILE` output.
- A `type: FILE` output must name a file of the execution itself or of its `Loop` family; anything else is refused with
`egret refuses output <id>: <uri> is not a file of this execution`. A parent cannot re-publish its `Subflow` child's file.
- A `SECRET` output with `fetchType: STORE` is refused, when you save and when it runs. Other `SECRET` outputs are sealed with
`secrets.master_key` and shown as `******`; without a master key the output fails and names the setting.
- Duplicate output ids inside a `Loop`'s `outputs:` are refused when you save (`duplicate_output_id`), and `secret()` is refused in an
output's `value:`.
- A `dependsOn` Flow trigger does not fire on an event whose merged `trigger.outputs` would pass 16 MiB; the log names the sizes
and the window is kept.

Known differences from Kestra: the parent flow's `outputs:` are not evaluated inside each `Loop` iteration; a STORE file is
named `outputs.ion` and `nextOffset` is not written; a `Subflow` child that ended before it ran and declares only literal outputs
fails the `Subflow` run; a `retry:` with `behavior: CREATE_NEW_EXECUTION` that reaches a task inside a `Loop` is refused when it
runs. A `SECRET` value that an author renders into a `Subflow`'s `inputs:` or a `STRING` flow output is stored in clear and
returned by the API; keep `SECRET` values in `SECRET` outputs and inputs.

## Rolling back

There is no supported in-place rollback. Restore the database from the backup taken in step 2, then start the previous
Expand Down
2 changes: 1 addition & 1 deletion platform/known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ Collected while writing the detailed pages for each area (October 2026), checked
| KI-114 | workflows | Gap | S2 | `concurrency`, `checks`, `sla`, `quotas` only warn; two runs of a `limit: 1` flow ran together. *Steps:* docs.demo/one-at-a-time. |
| KI-115 | workflows | Gap | S2 | `trigger.*` (date, body) unreadable in task expressions; `{{ trigger.date }}` in a scheduled flow fails with "undefined value". *Steps:* nightly-heartbeat first runs. |
| KI-116 | workflows | Gap | S2 | API/schedule/webhook starts do not validate required inputs, types or SELECT values; only the run form does. *Steps:* docs.demo/required-inputs executions. |
| KI-117 | workflows | Gap | S2 | Flow `outputs:` are not computed; Subflow `outputs` always `{}`. *Steps:* call-subflow / add-numbers. |
| KI-117 | workflows | Gap | S2 | Flow `outputs:` are not computed; Subflow `outputs` always `{}`. *Steps:* call-subflow / add-numbers. *Status:* fixed in platform #840 (November 2026, workflow package #6): a flow's `outputs:` are evaluated when the execution ends and stored on it, `{{ execution.outputs }}` and a `Subflow` parent's `outputs.<id>.outputs` carry them, and `GET /executions/{id}` returns them as `outputs`. |
| KI-118 | workflows | Gap | S3 | YAML input defaults are not parsed into an object in expressions. *Steps:* docs.demo/input-types-tour. |
| KI-119 | workflows | Bug | S2 | Deleting a flow keeps its namespace/id reserved ("a trashed flow still holds"), with no way to restore or purge. *Steps:* DELETE a flow then POST same id. *Status:* fixed in platform #747 (October 2026): **Flows → Trash** (`GET /flows?trashed=true`) lists the trash. **Restore** (`POST /flows/{id}/restore`) brings a flow back and recreates its schedules from their next tick. **Delete permanently** (`DELETE /flows/{id}/permanent`, type-to-confirm) works only on a flow in the trash: it deletes the flow, its versions and its trigger state, frees its namespace and id, and keeps its executions. Both need `workflow:manage` and are audited. The 409 on a taken name now says *restore it, or delete it permanently to free the name*. |
| KI-120 | workflows | Bug | S3 | Served schema's `DependsOn` lists `condition`/`inputs`, but the engine reads namespace/flowId/states/labels/when. *Steps:* GET /workflows/schema. *Status:* closed, does not reproduce (October 2026, checked for platform #747): the served `io.kestra.core.models.flows.DependsOn` (`inputs`, `condition`) is the **input** `dependsOn`, used only by the input definitions, and that is what the engine reads for an input. A Flow trigger's `dependsOn` items point to `Flow-Dependency` (`namespace`, `flowId`, `states`, `labels`, `when`), which is what the engine reads for a trigger. |
Expand Down
5 changes: 5 additions & 0 deletions platform/reference/rest-api/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ Operation id: `get_execution`. Auth: Session cookie. Forbidden (403): Missing wo
| `state` | string | yes | Kestra's wire spelling — `RUNNING`, `SUCCESS`, `FAILED`, … |
| `history` | string[] | yes | The state timeline, as stored: an array of state NAMES, e.g. `["CREATED","RUNNING","SUCCESS"]`. It carries no timestamps — see [`TaskRunDto::history`], which is written by the same function. |
| `inputs` | object | yes | |
| `outputs` | object | no | The flow's `outputs:` as evaluated when the execution ended; absent when none were. A `SECRET` value is `******`. Only the single-execution `GET` fills it; the executions list does not carry it. |
| `seq` | integer (int64) | yes | |
| `execution_trigger` | object, nullable | no | The trigger record, `NULL` for an API start. |
| `trigger_execution_id` | string (uuid), nullable | no | |
Expand Down Expand Up @@ -289,6 +290,7 @@ Tasks retrying, paused or not yet started end `KILLED` at once; a running task i
| `state` | string | yes | Kestra's wire spelling — `RUNNING`, `SUCCESS`, `FAILED`, … |
| `history` | string[] | yes | The state timeline, as stored: an array of state NAMES, e.g. `["CREATED","RUNNING","SUCCESS"]`. It carries no timestamps — see [`TaskRunDto::history`], which is written by the same function. |
| `inputs` | object | yes | |
| `outputs` | object | no | The flow's `outputs:` as evaluated when the execution ended; absent when none were. A `SECRET` value is `******`. Only the single-execution `GET` fills it; the executions list does not carry it. |
| `seq` | integer (int64) | yes | |
| `execution_trigger` | object, nullable | no | The trigger record, `NULL` for an API start. |
| `trigger_execution_id` | string (uuid), nullable | no | |
Expand Down Expand Up @@ -384,6 +386,7 @@ Run a finished execution again.
| `state` | string | yes | Kestra's wire spelling — `RUNNING`, `SUCCESS`, `FAILED`, … |
| `history` | string[] | yes | The state timeline, as stored: an array of state NAMES, e.g. `["CREATED","RUNNING","SUCCESS"]`. It carries no timestamps — see [`TaskRunDto::history`], which is written by the same function. |
| `inputs` | object | yes | |
| `outputs` | object | no | The flow's `outputs:` as evaluated when the execution ended; absent when none were. A `SECRET` value is `******`. Only the single-execution `GET` fills it; the executions list does not carry it. |
| `seq` | integer (int64) | yes | |
| `execution_trigger` | object, nullable | no | The trigger record, `NULL` for an API start. |
| `trigger_execution_id` | string (uuid), nullable | no | |
Expand Down Expand Up @@ -442,6 +445,7 @@ W7b Task 3 fix round 1, the user's ruling of 2026-10-11, after review found the
| `state` | string | yes | Kestra's wire spelling — `RUNNING`, `SUCCESS`, `FAILED`, … |
| `history` | string[] | yes | The state timeline, as stored: an array of state NAMES, e.g. `["CREATED","RUNNING","SUCCESS"]`. It carries no timestamps — see [`TaskRunDto::history`], which is written by the same function. |
| `inputs` | object | yes | |
| `outputs` | object | no | The flow's `outputs:` as evaluated when the execution ended; absent when none were. A `SECRET` value is `******`. Only the single-execution `GET` fills it; the executions list does not carry it. |
| `seq` | integer (int64) | yes | |
| `execution_trigger` | object, nullable | no | The trigger record, `NULL` for an API start. |
| `trigger_execution_id` | string (uuid), nullable | no | |
Expand Down Expand Up @@ -759,6 +763,7 @@ Operation id: `start_flow_execution`. Auth: Session cookie and `X-Egret-CSRF` he
| `state` | string | yes | Kestra's wire spelling — `RUNNING`, `SUCCESS`, `FAILED`, … |
| `history` | string[] | yes | The state timeline, as stored: an array of state NAMES, e.g. `["CREATED","RUNNING","SUCCESS"]`. It carries no timestamps — see [`TaskRunDto::history`], which is written by the same function. |
| `inputs` | object | yes | |
| `outputs` | object | no | The flow's `outputs:` as evaluated when the execution ended; absent when none were. A `SECRET` value is `******`. Only the single-execution `GET` fills it; the executions list does not carry it. |
| `seq` | integer (int64) | yes | |
| `execution_trigger` | object, nullable | no | The trigger record, `NULL` for an API start. |
| `trigger_execution_id` | string (uuid), nullable | no | |
Expand Down
4 changes: 2 additions & 2 deletions platform/workflows/executions/detail.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,13 @@ The **Outputs** tab lists the task runs that produced outputs. Select one to see

![The Outputs tab showing the Return task's value and its expression](/img/platform/workflows/execution-outputs.png)

*No outputs* means no task run of the execution produced any. Flow-level outputs are not shown because they are not computed yet.
*No outputs* means no task run of the execution produced any. The tab lists task runs only: the flow's own [outputs](../flows/inputs-and-outputs.md#flow-outputs) are not shown here. `GET /api/ws/{workspace}/workflows/executions/{id}` returns them in an `outputs` field (a `SECRET` value is `******`); the executions list does not carry it.

## Limits and known issues

- The **Overview** tab does not list inputs that came from `defaults:`.
- Tasks inside a `Loop` have task runs in the child executions, not in the parent. Open a child to see them.
- Flow `outputs:` are not shown anywhere.
- Flow `outputs:` are not shown in the UI; read them from the execution API.

## Further reading

Expand Down
2 changes: 2 additions & 0 deletions platform/workflows/flows/error-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ The task above fails three times, one second apart. The **Gantt** tab draws each

The three retry types, and `behavior`, are in the [YAML reference](./yaml-reference.md#task-retry). `warningOnRetry` is accepted and has no effect; the editor warns `ignored_property` when you save.

A `retry:` on a flowable task (a `Sequential`, a `Loop`, a `Parallel` and so on) is inherited by the tasks inside it; inside a `Loop`, by the tasks of each iteration. The nearest policy wins: the task's own, then its ancestors', then the flow's. See [Loop retry](./flow-control.md#loop-retry). When a task fails for good under such a policy, its parent now ends `FAILED`; before this release it stayed `RUNNING` forever.

### Retry the whole flow

A flow-level `retry:` starts a new execution when the flow fails:
Expand Down
Loading
Loading