From dc78928e96b97063883997f2268771508357083c Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:25:55 -0800 Subject: [PATCH 1/9] Add the plan for golden tests of tasks Thirty tests of Lua's `tasks` table, each a small prompt beside the log that running it through the Harness must produce. The plan lists the 29 behaviors under test, the 30 tests, the runner, and the format of a stored log. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- plan/2026-11-18-1-tasks-golden-tests.plan.md | 272 +++++++++++++++++++ 1 file changed, 272 insertions(+) create mode 100644 plan/2026-11-18-1-tasks-golden-tests.plan.md diff --git a/plan/2026-11-18-1-tasks-golden-tests.plan.md b/plan/2026-11-18-1-tasks-golden-tests.plan.md new file mode 100644 index 000000000..d6275c374 --- /dev/null +++ b/plan/2026-11-18-1-tasks-golden-tests.plan.md @@ -0,0 +1,272 @@ +--- +name: Golden tests for tasks +overview: Add 30 tests of Lua's `tasks` table to the `runvil` crate's suite. Each test is a small prompt file beside the log that running it through the Harness must produce. One new runner runs a prompt on a Host with a mock model and a paused clock, renders the run's whole record as text, and compares it with the stored log. +todos: + - id: plan-copy + content: "Copy this plan to plan/2026-11-18-1-tasks-golden-tests.plan.md on branch tasks-golden-tests and commit it alone" + status: pending + - id: runner + content: "Add the runner, the mock model, and test 1, and show that a changed log line fails the test" + status: pending + - id: cases + content: "Add tests 2 to 30, one commit per area, checking each generated log against its row in the table" + status: pending + - id: overlap + content: "List which existing tests already cover each of the 30, for the PR description" + status: pending + - id: gist + content: "Reword Test #29 in the gist to drop \"the same seed\"" + status: pending + - id: verify + content: "Run every verification command and vibelight" + status: pending + - id: pr + content: "Push the branch to the fork and open one PR against cppalliance/runvil-harness" + status: pending +--- + +# Golden tests for tasks + + + +## Product Requirements + +Lua's `tasks` table lets a prompt launch, watch, steer, and stop tasks. It must be airtight before the model is allowed to launch tasks, and other people must be able to read how it behaves. This plan adds 30 tests that each pair the smallest prompt for one behavior with the log that the run must produce. The tests run through the Harness's public API, as a Host runs a prompt. All 30 arrive in one pull request with this plan. + +- Problem and users: + - The roughly 175 existing tests of tasks are Rust tests that step the Engine. A reader has to know the test support code to see what a prompt did and what came out. + - No test of tasks runs through the Harness with a model that a Host supplied. + - The users are the engineers who will build on tasks, and reviewers who need to see each behavior as a prompt and its outcome. +- Goals: + - One test per row of the table in the Functional Specification, 30 in total, covering all 29 behaviors listed there. + - A test is two files: a prompt and its expected log. Adding a test needs no new Rust beyond one line that names it. + - A failing test says which line of the log differed, with the expected and the actual line. +- Non-goals: + - Changing any production code, or any existing test. + - Moving the tasks code into a module or a crate of its own. + - Testing a fixed seed. The Harness draws a new seed for every run. +- Success criteria: + - All 30 tests pass on two consecutive runs with no change to a stored log. + - Changing one line of a stored log makes that test, and only that test, fail. + - Every verification command in the Testing Plan passes. +- Constraints: + - Tests reach the Harness only through the `runvil` crate, the one crate a Host depends on. + - No real model, no network, and no real sleeping. The whole suite runs in seconds. + - A stored log is never changed to match a run when the run contradicts the table. The disagreement is reported first. +- Open questions: None + +## Functional Specification + +A developer writes a prompt file, runs the suite once in bless mode to write its log, reads the log against what the prompt should do, and commits both. From then on the suite runs the prompt and fails if the log differs. The prompt itself tells the mock model what to answer, so nothing else describes a test. + +- Actors and workflows: + - Add a test: write `NN-name.md`, add one line naming it to the suite, run with `RUNVIL_BLESS=1`, review `NN-name.log`, commit. + - Run the tests: the workspace test command runs them with every other test. + - Change behavior on purpose: run with `RUNVIL_BLESS=1` and review the diff of the logs in the commit. +- Inputs and outputs: + - Input: a prompt file, an ordinary Runvil prompt. A test whose prompt calls `execute` keeps the second prompt in a `helpers/` folder beside the tests. + - Output: the expected log. One line per record that the recorder received, in the order received: the task id, the record's position in its task, `effect`, `answer`, or `event`, the effect's number or `-`, and the payload as JSON with sorted keys. One last line gives the run's result: `result ok` and the text, or `result error`, the error code, and the message. + - The mock model reads its instruction from the newest user message of the request: + - `reply ` answers `` at once. + - `after reply ` answers after that many seconds on the paused clock. + - `call ` asks for one tool call. Steps joined by ` then ` are taken one per round, so `call echo {} then reply done` is a two-round tool loop. + - Any other text fails the round with an error that quotes the text. +- States and validation: + - A guard test fails when a prompt file has no log, a log has no prompt, or a prompt file is not named in the suite. + - A missing log outside bless mode fails the test and names the bless command. +- Errors and recovery: + - A mismatch reports the file, the first differing line number, the expected line, and the actual line. + - A prompt that fails to parse fails its test with the parser's error. +- Security and privacy behavior: None +- Acceptance criteria: + - Each test in the table below exists as a prompt and a log, and its log shows the stated outcome. + - Each of the 29 behaviors is covered by at least one test, as the last column records. + +### The 29 behaviors under test + +1. **Chain.** A chain is one walk through a prompt's sections. +2. **Task.** A task is a section that `tasks.spawn` starts as its own chain. The chain that spawned it owns it. +3. **Lifetime.** A task lives as long as its owner chain. +4. **Turns.** Chains take turns on one thread, so only one chain's Lua runs at a time. +5. **Launching.** `tasks.spawn(target, opts)` returns a handle, `{ task = id }`, before the task runs. +6. **Inputs.** `opts` can set the task's `args`, `item`, and `index`. `var` and the model are copied at the spawn. +7. **Start.** A task first runs once its owner parks and a slot is free. +8. **Limit.** At most 8 tasks run at once by default. `tasks.concurrency(n)` changes a chain's limit. +9. **Result.** A task's result is the text its section returns. An error fails only that task. +10. **Join any.** `tasks.join_any(set)` waits for one task to end or reply. It returns the handle, `ok`, and the result. +11. **Join.** `tasks.join(set)` waits for every task and returns one entry per task, in the order listed. +12. **Errors.** A wait returns a failed task's error as data and never raises it. +13. **Once.** Each result goes to one wait only. A second wait for it raises `task_consumed`. +14. **Store.** Once a wait delivers a task, the task's store writes are visible to its owner. +15. **Timeout.** `opts.timeout` ends a wait after that many seconds and leaves the tasks running. +16. **Checking.** `tasks.pending()` lists the caller's live tasks. `tasks.ready(task)` says if one has ended or holds a reply no wait has taken. +17. **Status.** `tasks.status(task)` gives a task's state and what it waits on. `tasks.note(text)` sets a progress note. +18. **Canceling.** `tasks.cancel(task)` stops a task and every task it owns. A wait on it returns kind `canceled`. +19. **Steering.** Every task has an inbox of text messages. Only its owner fills it, with `tasks.send`. +20. **Receive.** `tasks.receive()` returns the oldest message, waiting if there is none. +21. **Reply.** `tasks.reply(value)` hands the owner a result, then waits for the next message, so the task stays open. +22. **Ownership.** Only the owner may wait on, check, send to, or cancel a task, though a task may read its own status. Others get `task_not_owned`. +23. **Correctness.** Ownership protects program correctness, not security: a task id is a guessable string. +24. **Endings.** A chain that ends normally while its task is still running fails with `tasks_live`. A chain that fails keeps its own error, and a task waiting after a reply does not count. +25. **Abandoned.** A task still live when its owner ends is `abandoned`, with every task under it. +26. **Ids.** The main walk is task `0`, and a task's id adds a dot and its owner's next child number: `0.0`, then `0.0.0` under it. A `call`, an `execute`, or a timed wait takes a number too. +27. **Events.** Each task that starts reports one `TaskStarted` event and exactly one terminal event. +28. **Repeatability.** The same seed, start time, and answers in the same order give the same run. +29. **Fanout.** `tasks.fanout` is Lua over `spawn`, `join_any`, and `cancel`. Each arm is an ordinary task. + +### The 30 tests + +| # | File stem | The prompt | The log must show | Behaviors | +|---|---|---|---|---| +| 1 | `01-spawn-and-join` | Spawns one task and joins it. | One start, one success, and the result. | 2, 5 | +| 2 | `02-survives-fall-through` | Spawns a task, falls through to the next section, joins it there. | The task succeeds after the section change. | 1, 3 | +| 3 | `03-queued-until-owner-waits` | Spawns a task, then polls `tasks.status` in a loop that never waits. | Every read is `queued`. | 4, 5, 7 | +| 4 | `04-inputs-copied-at-spawn` | Spawns with `args`, `item`, and `index`, then changes `var` and the model. | The task sees the values from the spawn. | 6 | +| 5 | `05-limit-of-one` | Sets the limit to 1 and spawns two tasks. | The second starts after the first succeeds. | 7, 8 | +| 6 | `06-default-limit-of-eight` | Spawns nine tasks that each wait on the mock model. | Eight start, and the ninth starts when one ends. | 8 | +| 7 | `07-join-two-tool-loops` | Launches two tasks that each block in a tool loop, and joins both. | Results in the order listed. | 11 | +| 8 | `08-join-any-first-to-end` | One task waits on the mock model and one returns at once. | `join_any` returns the second's handle, `ok`, and result. | 10 | +| 9 | `09-result-and-error` | One task returns text and one raises. | The owner gets the text, and `ok = false` with the error. | 9, 12 | +| 10 | `10-wait-twice` | Waits twice on one task. | The second wait raises `task_consumed`. | 13 | +| 11 | `11-store-after-join` | A task writes a store file. | Its owner reads it after the join. | 14 | +| 12 | `12-timed-join` | Time-limits a `join` while one task waits on the mock model. | It times out, and a second `join` collects the task. | 15 | +| 13 | `13-pending` | Spawns three tasks and joins the second. | `tasks.pending()` lists the first and third. | 16 | +| 14 | `14-ready-after-end` | Reads `tasks.ready` before and after a task ends. | `false`, then `true`. | 16 | +| 15 | `15-ready-with-reply` | A task replies and stays open. | `tasks.ready` reads `true` until a wait takes the reply, then `false`. | 16 | +| 16 | `16-status-through-a-model-call` | Reads `tasks.status` before, during, and after a task's model call. | `queued`, `chat`, then `done`. | 17 | +| 17 | `17-note` | A task sets two notes. | Its owner's `tasks.status` shows the second. | 17 | +| 18 | `18-cancel` | Cancels a queued task, one waiting on the mock model, and one that owns a task. | Each wait returns kind `canceled`. | 18 | +| 19 | `19-steer-with-replies` | A worker replies, is sent "shorten it", replies again, is sent "done". | The worker returns. | 19, 21 | +| 20 | `20-receive-oldest-first` | Sends a task two messages before it runs. | `tasks.receive()` returns them oldest first, and a third call waits. | 19, 20 | +| 21 | `21-sibling-not-owned` | A task tries to wait on, check, send to, and cancel its sibling. | Each raises `task_not_owned`. | 22 | +| 22 | `22-self-and-unknown-not-owned` | A task tries to cancel itself and the made-up id `9.9`. | Each raises `task_not_owned`. | 22, 23 | +| 23 | `23-own-status` | A task reads `tasks.status(sys.taskid)`. | Its own status, with state `running`. | 22 | +| 24 | `24-return-with-live-task` | Returns while a task that owns a task runs. | The run fails with `tasks_live` and both end `abandoned`. | 24, 25 | +| 25 | `25-error-with-live-task` | Raises an error while a task runs. | The run fails with that error, and the task ends `abandoned`. | 24 | +| 26 | `26-return-with-waiting-task` | Returns while a task waits after a reply. | The run ends normally and the task ends `abandoned`. | 24 | +| 27 | `27-nested-ids` | A task spawns a task. | Ids `0.0` and `0.0.0`, each with one start and one terminal event. | 26, 27 | +| 28 | `28-ids-skip-other-children` | Makes a `call` and an `execute`, spawns a task, waits on it with a time limit, spawns another. | The tasks are `0.2` and `0.4`. | 26 | +| 29 | `29-same-log-twice` | Runs one prompt twice. | The two logs are identical. | 28 | +| 30 | `30a-fanout`, `30b-spawns-by-hand` | Fans out over three members, and spawns three tasks by hand. | The task events of the two logs match. | 29 | + + + + +## Technical Design + +The change is test code and test data in the `runvil` crate, plus one dev-dependency feature. No production code and no facade listing changes. The runner builds one Host for every test, so a test differs from another only in its prompt. + +- Architecture: + - The runner is a module of the crate's existing `suite` test target, beside the modules already there (`crates/runvil/tests/suite/main.rs`). + - Each run goes `Host::parse`, `Harness::new`, `Harness::run`, as the suite's `launch` does today (`crates/runvil/tests/suite/support.rs`), with a `MemoryRecorder` whose records become the log. + - The clock is tokio's, paused. A sleep completes in the order of its deadline with no real waiting, which fixes the order in which timers and delayed model replies answer. +- Modules and interfaces: + - `golden`: builds the Host and the `RunSetup`, runs a prompt by file stem, renders the record, and compares or blesses. Its Host finds prompts by file name in the test folder, sleeps on the paused clock, and stamps every run's start as the Unix epoch. + - `mock_model`: an `InferenceBroker` that lists two models, `mock-model` and `mock-model-2`, and answers by the instruction grammar in the Functional Specification. + - `tasks`: one line per test naming its file stem, the guard test, and the two tests that compare logs with each other (29 and 30). + - One fixture Pack with a single `echo` tool, for the tool loops of test 7, registered as `crates/runvil/tests/suite/vfs.rs` registers its `acme/desk` Pack. +- File and public API changes: + - New: `crates/runvil/tests/suite/golden.rs`, `mock_model.rs`, `tasks.rs`, and three `mod` lines in `main.rs`. + - New: `crates/runvil/tests/golden/tasks/`, holding 31 prompts, 31 logs, and `helpers/` with the prompt that test 28 executes. + - Changed: `crates/runvil/Cargo.toml`, where the tokio dev-dependency gains the `test-util` feature for the paused clock. + - New: `plan/2026-11-18-1-tasks-golden-tests.plan.md`, this plan. + - Public API: none. `crates/runvil/public-api.txt` and `crates/runvil-engine/public-api.txt` stay as they are. +- Data, persistence, failure, security, and privacy constraints: + - A stored log is read from the source tree at test time, and written only when `RUNVIL_BLESS=1` is set. + - A prompt in this suite uses no randomness, because the seed differs on every run. + + + + +## Testing Plan + +The 30 tests are the deliverable, so the plan for testing is about trusting the runner and the stored logs. Each log is generated once and then read line by line against its row in the table before it is committed. + +- Unit: + - One test of the comparison: two texts that differ on one line produce a message with that line number, the expected line, and the actual line. +- Integration and end-to-end: + - The 30 tests of the table, through the Harness's public API. + - The guard test over the test folder. +- Regression, security, and performance: + - `cargo nextest run --locked --workspace --all-features` + - `cargo clippy --workspace --all-targets --all-features` with `CARGO_BUILD_WARNINGS=deny` + - `cargo fmt --all --check` + - `cargo doc --workspace --no-deps --all-features` and `cargo doc -p runvil-engine --no-deps`, both with `RUSTDOCFLAGS="-D warnings"` + - `cargo + xtask api --check`, with the nightly named in `crates/build-xtask/src/api/toolchain.rs` + - `cargo dylint --all -- --locked --workspace --all-targets --all-features` + - `cargo test -p build-xtask` + - `node --test tools/docs-claims.mjs` + - `vibelight` in the repository root +- Exit criteria: + - Every command above passes. + - The suite passes twice in a row with no stored log changed. + - A hand edit of one stored log fails exactly that test, and the edit is then reverted. + - Every stored log has been read against its row of the table, and every disagreement is written into the Decision Record. + + + + +## Decision Record + +- Decisions: + - One pull request holds the plan, the runner, and all 30 tests. The standing rule is "Keep PRs small... Split big work into smaller steps, each with its own plan". Asked which governs here, Andrew chose "One PR with all 30". + - The tests go through the Harness, not the Engine alone. Andrew's note from the session of 2026-11-17: "test the tasks through the harness". Vinnie, in the huddle of 2026-11-18, described the mock model as an inference broker passed to the Harness that returns canned results. + - A log holds the whole record: every effect, answer, and event. Andrew chose "Whole record". It shows timers and what the mock model was sent, and it is the same record that replay will compare. + - Test 29 runs one prompt twice and compares the logs. The Harness draws the seed with `rand::random()` (`crates/harness-internal/runner/src/prepare.rs`), and `RunSetup` has no seed field (`crates/harness-internal/runner/src/setup.rs`), so a test here cannot hold the seed fixed. Andrew chose "Run twice, compare logs". Its wording in the shared list of tests drops "the same seed". + - The mock model's instructions sit in the prompt's own messages. A test stays two files, and a reader sees the model's answer where the prompt asks for it. + - The clock is paused, not real. Tests of timeouts and of "while a task waits on the model" need a fixed order of answers, and real sleeps would make the order depend on the machine. + - The tests join the existing `suite` target. A new test target or crate would add a link step and nothing else. + - The existing tests stay untouched, and a new test may cover what an existing one covers. The new suite's purpose is a readable prompt and outcome for each behavior. The PR description lists the overlap. +- Rejected alternatives: + - A runner on the Engine with its scripted model. It could fix the seed, but it would not go through the Harness or a Host's broker. Revisit if a test needs a fixed seed. + - A log of events only. It is about a third shorter, and it hides timers and model requests. Revisit if the logs prove too noisy to review. + - A third file of canned replies beside each prompt. It splits one test across three files. Revisit if the instruction grammar grows past the four forms. + - One test function that loops over the folder. A failure would not name its test in the test report. +- Assumptions, risks, and notes: + - The 29 behaviors were checked by reading the code at commit `4afed45fe`, not by running it. A generated log may show that a row of the table is wrong. That is a finding to report, and the row or the code changes before the log is stored. + - In a trial at commit `4afed45fe`, one prompt with two tasks, a timed join, and a delayed model reply gave the same 30-line record on two runs with different seeds. + - A change to any event's fields changes every stored log. Bless mode rewrites them, and the commit's diff shows what changed. + - Two sleeps with the same deadline could complete in either order. Prompts give concurrent model replies different delays. + - Justin's command-line Host and Kirill's replay need the same answer to what a stored log holds. This plan's format is a proposal to them, not a decision for them. + +### Deferred and Out of Scope + +- Deferred: a test that holds the seed fixed, until a test of the Engine or a replay driver can supply one. +- Deferred: removing an existing test that a new one makes redundant, until mutation testing shows the breakages it catches are still caught. +- Out of scope: moving the tasks code into its own module or crate. +- Out of scope: reading a task's conversation from outside the task, which no `tasks` function does. +- Out of scope: tasks that the model launches. + + + + +## Project Survey + +- The `runvil` crate's suite already runs prompts through the public API: `launch`, `no_packs`, an offline broker, and a tokio clock in `crates/runvil/tests/suite/support.rs`. +- A scripted broker that answers from the last message exists in `crates/runvil/tests/suite/broker.rs`, and brokers that return tool calls exist in `crates/harness-internal/runner/tests/it/scripted.rs`. +- A fixture Pack registered through the facade exists in `crates/runvil/tests/suite/vfs.rs`. +- `MemoryRecorder::records` returns each record's task id, position, kind, effect number, and JSON payload (`crates/harness-internal/runner/src/recorder.rs`, `recorder_memory.rs`). +- The Host's `Timer` supplies both `sleep` and `now`, and the Harness stamps a run's start with `now` (`crates/harness-internal/runner/src/performers.rs`, `harness.rs`). +- The run's default limit on tasks running at once is 8 (`crates/runvil-internal/runtime/src/execute/params.rs`). +- No golden-file runner exists in the repository. The existing tests of tasks are under `crates/runvil-internal/runtime/src/execute/tests/` and `execute/run/`. + + + + +## Execution Instructions + +- Work items: + 1. On branch `tasks-golden-tests`, cut from upstream `master` at `4afed45fe`, copy this plan to `plan/2026-11-18-1-tasks-golden-tests.plan.md` and commit it alone. + 2. Add `golden`, `mock_model`, the `echo` Pack, the `test-util` feature, and test 1. Edit test 1's stored log by hand, confirm the failure message, and revert the edit. + 3. Add tests 2 to 30 in the order of the table, one commit per area: launching and lifetime (1 to 6), waiting and results (7 to 12), finding out what is running (13 to 17), canceling (18), steering (19 and 20), ownership (21 to 23), endings (24 to 26), ids and repeatability (27 to 29), fanout (30). + 4. For each test, generate the log with `RUNVIL_BLESS=1`, read it against the row of the table, and stop to report any disagreement before storing it. + 5. Search the existing tests for each of the 30 behaviors and write the overlap into the PR description. + 6. Reword Test #29 in Andrew's gist of the list to "Run one prompt twice. The two logs are identical." + 7. Run every command of the Testing Plan, then `vibelight`. + 8. Push the branch to the fork and open one pull request against `cppalliance/runvil-harness` `master`. Andrew tags the human reviewers and posts the link in the channel. +- Dependencies and verification: + - Item 2 comes before item 3, because every test needs the runner. + - Item 7 runs after the last code change, and again after any fix it prompts. + - Commits use the GitHub no-reply address, and nothing is pushed to the cppalliance repository. + + From 25265ef63b104ec5d09501408193700930a9eba7 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:29:18 -0800 Subject: [PATCH 2/9] Add the golden runner and the first test of tasks A golden test is a prompt file beside the log its run must produce. The runner runs the prompt through a Harness on a Host with a mock model and a paused clock, renders every record the recorder received and the run's result as text, and compares the text with the stored log. `RUNVIL_BLESS=1` stores the run's log instead. - The mock model answers each round from an instruction in the request's newest user message: `reply`, `after reply`, or `call `, with steps joined by ` then `. - The Host holds the built-in Packs and `acme/echo`, whose one tool says back its `text`. - tokio's `test-util` feature joins the crate's dev-dependencies for the paused clock. - Test 1 spawns one task and joins it. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- crates/runvil/Cargo.toml | 5 +- .../tests/golden/tasks/01-spawn-and-join.log | 17 ++ .../tests/golden/tasks/01-spawn-and-join.md | 19 ++ crates/runvil/tests/suite/golden.rs | 232 ++++++++++++++++++ crates/runvil/tests/suite/main.rs | 3 + crates/runvil/tests/suite/mock_model.rs | 221 +++++++++++++++++ crates/runvil/tests/suite/tasks.rs | 11 + 7 files changed, 506 insertions(+), 2 deletions(-) create mode 100644 crates/runvil/tests/golden/tasks/01-spawn-and-join.log create mode 100644 crates/runvil/tests/golden/tasks/01-spawn-and-join.md create mode 100644 crates/runvil/tests/suite/golden.rs create mode 100644 crates/runvil/tests/suite/mock_model.rs create mode 100644 crates/runvil/tests/suite/tasks.rs diff --git a/crates/runvil/Cargo.toml b/crates/runvil/Cargo.toml index 7929c705c..90cc02061 100644 --- a/crates/runvil/Cargo.toml +++ b/crates/runvil/Cargo.toml @@ -17,10 +17,11 @@ runvil-engine.workspace = true runvil-plugin.workspace = true # The suite and the `run-prompt` example run a Host on tokio, as a real -# Host does; the facade's own items name no tokio type. +# Host does; the facade's own items name no tokio type. `test-util` is the +# paused clock the golden tests run on. [dev-dependencies] serde_json.workspace = true -tokio = { workspace = true, features = ["macros", "rt", "signal", "time"] } +tokio = { workspace = true, features = ["macros", "rt", "signal", "test-util", "time"] } [lints] workspace = true diff --git a/crates/runvil/tests/golden/tasks/01-spawn-and-join.log b/crates/runvil/tests/golden/tasks/01-spawn-and-join.log new file mode 100644 index 000000000..7d43a1a6a --- /dev/null +++ b/crates/runvil/tests/golden/tasks/01-spawn-and-join.log @@ -0,0 +1,17 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"spawn-and-join"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"spawn-and-join"} +result ok "done" diff --git a/crates/runvil/tests/golden/tasks/01-spawn-and-join.md b/crates/runvil/tests/golden/tasks/01-spawn-and-join.md new file mode 100644 index 000000000..af653bc4e --- /dev/null +++ b/crates/runvil/tests/golden/tasks/01-spawn-and-join.md @@ -0,0 +1,19 @@ +--- +name: spawn-and-join +description: Spawns one task and joins it +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local results = tasks.join({ worker }) +return results[1].result +``` + +## Worker + +```lua +return 'done' +``` diff --git a/crates/runvil/tests/suite/golden.rs b/crates/runvil/tests/suite/golden.rs new file mode 100644 index 000000000..e95492129 --- /dev/null +++ b/crates/runvil/tests/suite/golden.rs @@ -0,0 +1,232 @@ +//! The golden runner: it runs a prompt file through a Harness and compares +//! the run's log with the log stored beside the prompt. +//! +//! A golden test is two files in `tests/golden/tasks/`: `.md`, an +//! ordinary prompt, and `.log`, the log its run must produce. The log +//! holds one line per record the run's recorder received, in the order +//! received: the record's task id, its position in that task, its kind, +//! the number of its effect or `-`, and its payload as JSON. One last line +//! holds the run's result. +//! +//! Every golden run has the same Host and the same setup, so a test +//! differs from another only in its prompt: the Host finds prompts by file +//! name in the tests' folder, holds the built-in +//! Packs and [`ECHO`], sleeps on tokio's clock, which each test pauses, and +//! stamps every run's start as the Unix epoch. The model is the +//! [`MockModel`]. The Harness draws a new seed for every run, so a golden +//! prompt uses no randomness. +//! +//! Setting [`BLESS`] in the environment stores each run's log in place of +//! comparing it. + +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +use runvil::harness::{MemoryRecorder, Packs, PromptSet, Record, Timer}; +use runvil::packs::PackId; +use runvil::plugin::{Pack, PackInfo, PackRegistry, ToolContext}; +use runvil::{ + BoxFuture, Error, ErrorCode, Harness, Host, RunSetup, Timestamp, ToolId, ToolInfo, ToolOutput, + make_tool_error, +}; +use serde_json::{Value, json}; + +use crate::mock_model::{MODEL, MockModel}; + +/// The environment variable that, when set, makes [`check`] store each +/// run's log instead of comparing it. +const BLESS: &str = "RUNVIL_BLESS"; + +/// The Pack `acme/echo`, whose one tool, `echo/say`, answers the `text` it +/// is given: the tool a golden prompt's model loop calls. +const ECHO: PackInfo = PackInfo::new("acme/echo").description("Says back what it is told"); + +/// One run's instance of [`ECHO`]. +struct Echo { + say: ToolInfo, +} + +impl Pack for Echo { + fn tools(&self) -> Vec { + vec![self.say.clone()] + } + + fn call<'a>( + &'a self, + cx: ToolContext<'a>, + args: Value, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + match args.get("text").and_then(Value::as_str) { + Some(text) => Ok(ToolOutput::trusted(text)), + None => Err(make_tool_error( + ErrorCode::ToolInvalidArguments, + format!("{} requires a string `text`, got {args}", cx.tool()), + )), + } + }) + } +} + +/// Makes one run's [`Echo`], whose tool sits under the Pack id `id`. +fn new_echo(id: &PackId) -> Result, Error> { + let say = ToolInfo::new( + ToolId::parse(&format!("{id}/say"))?, + "Says back the text it is given.", + json!({ + "type": "object", + "properties": { "text": { "type": "string" } }, + "required": ["text"], + }), + ); + Ok(Arc::new(Echo { say })) +} + +/// Sleeps on tokio's clock and reads every run's start as the Unix epoch. +struct PausedClock; + +impl Timer for PausedClock { + fn sleep(&self, duration: Duration) -> BoxFuture<'static, ()> { + Box::pin(tokio::time::sleep(duration)) + } + + fn now(&self) -> Timestamp { + Timestamp::UNIX_EPOCH + } +} + +/// The folder that holds the golden prompts and their logs. +pub(crate) fn folder() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/golden/tasks") +} + +/// The Host every golden run shares the shape of. +fn host() -> Arc { + let mut registry = PackRegistry::new(); + if let Err(error) = registry.insert(ECHO, new_echo) { + panic!("the registry takes the echo Pack: {error}"); + } + let mut prompts = PromptSet::new(); + if let Err(error) = prompts.insert_dir(folder()) { + panic!("the golden folder is listed: {error}"); + } + Arc::new(Host { + packs: Packs::new(registry), + prompts: Arc::new(prompts), + timer: Arc::new(PausedClock), + }) +} + +/// Runs the prompt `.md` through a Harness and returns its log. +async fn run(stem: &str) -> String { + let host = host(); + let prompt = match host.parse(&format!("{stem}.md")) { + Ok(prompt) => prompt, + Err(error) => panic!("{stem}.md does not parse: {error}"), + }; + let model = match MockModel::new() { + Ok(model) => model, + Err(error) => panic!("the mock model's catalog builds: {error}"), + }; + let recorder = Arc::new(MemoryRecorder::new()); + let setup = RunSetup { + selected_model: Some(MODEL.to_owned()), + ..RunSetup::new(recorder.clone(), Arc::new(model)) + }; + match Harness::new(host, prompt, setup).run().await { + Ok(report) => render(&recorder.records(report.run_id), &report.result), + Err(error) => panic!("the memory recorder takes every write of {stem}.md: {error}"), + } +} + +/// The log of a run: one line per record, then the run's result. +fn render(records: &[Record], result: &Result) -> String { + let mut lines: Vec = records + .iter() + .map(|record| { + let effect = record + .effect_id + .map_or_else(|| "-".to_owned(), |id| id.to_string()); + format!( + "{} {} {} {effect} {}", + record.task_id, + record.task_seq, + record.kind.as_str(), + record.payload + ) + }) + .collect(); + lines.push(match result { + Ok(text) => format!("result ok {}", json!(text)), + Err(error) => format!("result error {} {}", error.code.tag(), json!(error.message)), + }); + lines.join("\n") + "\n" +} + +/// Where `actual` first differs from `expected`, as the line number with +/// the expected and the actual line, or `None` when the two are the same. +fn difference(expected: &str, actual: &str) -> Option { + let show = |line: Option<&str>| line.map_or_else(|| "no line".to_owned(), str::to_owned); + let (mut expected, mut actual) = (expected.lines(), actual.lines()); + let mut number = 0_usize; + loop { + number += 1; + let (wanted, got) = (expected.next(), actual.next()); + if wanted.is_none() && got.is_none() { + return None; + } + if wanted != got { + return Some(format!( + "line {number}\n expected: {}\n actual: {}", + show(wanted), + show(got) + )); + } + } +} + +/// Runs the prompt `.md` and requires its log to be the one stored +/// in `.log`, then returns the log. With [`BLESS`] set, it stores the +/// run's log there instead. +pub(crate) async fn check(stem: &str) -> String { + let actual = run(stem).await; + let path = folder().join(format!("{stem}.log")); + if std::env::var_os(BLESS).is_some() { + if let Err(error) = std::fs::write(&path, &actual) { + panic!("cannot store {}: {error}", path.display()); + } + return actual; + } + let expected = match std::fs::read_to_string(&path) { + Ok(expected) => expected, + Err(error) => panic!( + "cannot read the stored log {}: {error}; run with {BLESS}=1 to store the run's log", + path.display() + ), + }; + if let Some(difference) = difference(&expected, &actual) { + panic!( + "the run of {stem}.md differs from {stem}.log at {difference}\n\ + run with {BLESS}=1 to store the run's log if the change is intended" + ); + } + actual +} + +#[test] +fn a_difference_names_the_first_line_that_differs_with_both_lines() { + assert_eq!(difference("a\nb\n", "a\nb\n"), None); + assert_eq!( + difference("a\nb\nc\n", "a\nB\nC\n"), + Some("line 2\n expected: b\n actual: B".to_owned()) + ); + assert_eq!( + difference("a\n", "a\nb\n"), + Some("line 2\n expected: no line\n actual: b".to_owned()) + ); + assert_eq!( + difference("a\nb\n", "a\n"), + Some("line 2\n expected: b\n actual: no line".to_owned()) + ); +} diff --git a/crates/runvil/tests/suite/main.rs b/crates/runvil/tests/suite/main.rs index 9abd0a19d..db923bb85 100644 --- a/crates/runvil/tests/suite/main.rs +++ b/crates/runvil/tests/suite/main.rs @@ -2,7 +2,10 @@ //! the public API alone. mod broker; +mod golden; mod host; mod launch; +mod mock_model; mod support; +mod tasks; mod vfs; diff --git a/crates/runvil/tests/suite/mock_model.rs b/crates/runvil/tests/suite/mock_model.rs new file mode 100644 index 000000000..6bc710395 --- /dev/null +++ b/crates/runvil/tests/suite/mock_model.rs @@ -0,0 +1,221 @@ +//! The mock model of the golden tests: an inference broker that answers +//! each round from an instruction in the request's own messages, so a +//! prompt says what its model replies and a test needs no script beside +//! it. +//! +//! The instruction is the newest user message of the request. It holds one +//! or more steps joined by ` then `, and the broker takes one step per +//! round: the step numbered by the assistant messages that follow the +//! instruction. A step is one of: +//! +//! - `reply ` answers `` at once. +//! - `after reply ` answers `` once that many seconds +//! have passed on tokio's clock, which the golden tests pause. +//! - `call ` asks for one call of the tool with that wire +//! name, passing the JSON object as its arguments. +//! +//! Any other request fails its round with an error that quotes what the +//! broker read, so a prompt with a misspelled instruction fails its test. + +use std::num::NonZeroU32; +use std::time::Duration; + +use runvil::effect::Round; +use runvil::harness::InferenceBroker; +use runvil::model::{ + Completion, CompletionOptions, CompletionResult, Message, ModelBinding, ModelCatalog, ModelId, + ModelInfo, Role, ThinkingMode, ToolCall, ToolSchema, make_completion_error, +}; +use runvil::{BoxFuture, Error, ErrorCode}; +use serde_json::Value; + +/// The model a golden run selects. +pub(crate) const MODEL: &str = "mock-model"; + +/// A second model, for a prompt that switches models. +const SECOND_MODEL: &str = "mock-model-2"; + +/// The context window both models list: 131,072 tokens. +const WINDOW: NonZeroU32 = NonZeroU32::MIN.saturating_add(131_071); + +/// What separates an instruction's steps. +const THEN: &str = " then "; + +/// One step of an instruction: what the model does in one round. +#[derive(Debug, PartialEq)] +enum Step { + /// Answer `text` once `after` has passed. + Reply { after: Duration, text: String }, + /// Ask for one call of `tool` with `args`. + Call { tool: String, args: Value }, +} + +/// Lists two models and answers each round by its request's instruction. +pub(crate) struct MockModel { + catalog: ModelCatalog, +} + +impl MockModel { + /// A mock model listing [`MODEL`] and a second model. + pub(crate) fn new() -> Result { + let mut models = Vec::new(); + for name in [MODEL, SECOND_MODEL] { + let id = ModelId::gateway(name)?; + models.push(ModelInfo::new( + id, + "a mock model", + WINDOW, + ThinkingMode::Never, + )); + } + Ok(MockModel { + catalog: ModelCatalog::new(models)?, + }) + } +} + +impl InferenceBroker for MockModel { + fn models(&self) -> BoxFuture<'static, Result> { + let catalog = self.catalog.clone(); + Box::pin(async move { Ok(catalog) }) + } + + fn chat( + &self, + binding: ModelBinding, + messages: Vec, + _tools: Vec, + _options: CompletionOptions, + round: Round, + ) -> BoxFuture<'static, Result, Error>> { + let model = binding.id().name().to_owned(); + let step = step_of(&messages); + Box::pin(async move { + let result = match step.map_err(refusal)? { + Step::Reply { after, text } => { + tokio::time::sleep(after).await; + CompletionResult::Text(text) + } + Step::Call { tool, args } => { + let id = format!("call-{}", round.id.get()); + CompletionResult::ToolCalls(vec![ToolCall::from_parts(id, tool, args)?]) + } + }; + Completion::from_result(result, model).map(Box::new) + }) + } +} + +/// The round's failure for a request the mock model cannot answer. +fn refusal(why: String) -> Error { + make_completion_error(ErrorCode::Unavailable, why) +} + +/// The step `messages` ask for: the step of the newest user message that +/// the assistant messages after it have not taken yet. +fn step_of(messages: &[Message]) -> Result { + let Some(at) = messages + .iter() + .rposition(|message| message.role() == Role::User) + else { + return Err("the mock model needs a user message to read its instruction from".to_owned()); + }; + let instruction = messages[at].content(); + let taken = messages[at..] + .iter() + .filter(|message| message.role() == Role::Assistant) + .count(); + let steps: Vec<&str> = instruction.split(THEN).collect(); + let Some(step) = steps.get(taken) else { + return Err(format!( + "the mock model's instruction {instruction:?} has {} steps, and round {} needs one more", + steps.len(), + taken + 1 + )); + }; + parse_step(step) +} + +/// One step's text as a [`Step`], or what the text should have been. +fn parse_step(step: &str) -> Result { + let unknown = || { + format!( + "the mock model cannot read the step {step:?}: required `reply `, \ + `after reply `, or `call `" + ) + }; + if let Some(text) = step.strip_prefix("reply ") { + return Ok(Step::Reply { + after: Duration::ZERO, + text: text.to_owned(), + }); + } + if let Some(rest) = step.strip_prefix("after ") { + let (seconds, text) = rest.split_once(" reply ").ok_or_else(unknown)?; + let seconds: u64 = seconds.parse().map_err(|_| unknown())?; + return Ok(Step::Reply { + after: Duration::from_secs(seconds), + text: text.to_owned(), + }); + } + if let Some(rest) = step.strip_prefix("call ") { + let (tool, json) = rest.split_once(' ').ok_or_else(unknown)?; + let args = serde_json::from_str(json).map_err(|_| unknown())?; + return Ok(Step::Call { + tool: tool.to_owned(), + args, + }); + } + Err(unknown()) +} + +#[test] +fn a_step_is_the_one_the_assistant_messages_after_the_instruction_have_not_taken() { + let instruction = Message::user("call echo_say {\"text\":\"a\"} then after 5 reply done"); + let first = step_of(std::slice::from_ref(&instruction)); + assert_eq!( + first, + Ok(Step::Call { + tool: "echo_say".to_owned(), + args: serde_json::json!({ "text": "a" }), + }) + ); + let second = step_of(&[ + Message::user("reply an older instruction"), + instruction.clone(), + Message::assistant(""), + Message::tool("call-0", "a"), + ]); + assert_eq!( + second, + Ok(Step::Reply { + after: Duration::from_secs(5), + text: "done".to_owned(), + }) + ); + let third = step_of(&[instruction, Message::assistant(""), Message::assistant("")]); + assert!( + third + .as_ref() + .is_err_and(|why| why.contains("has 2 steps, and round 3 needs one more")), + "{third:?}" + ); +} + +#[test] +fn a_step_the_mock_model_cannot_read_is_refused_with_the_forms_it_takes() { + for step in [ + "say hello", + "after soon reply x", + "call echo_say", + "call echo_say {", + ] { + let refused = parse_step(step); + assert!( + refused + .as_ref() + .is_err_and(|why| why.contains(step) && why.contains("required `reply `")), + "{step}: {refused:?}" + ); + } +} diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs new file mode 100644 index 000000000..977896873 --- /dev/null +++ b/crates/runvil/tests/suite/tasks.rs @@ -0,0 +1,11 @@ +//! The golden tests of Lua's `tasks` table: each test runs one prompt of +//! `tests/golden/tasks/` through a Harness and requires the log stored +//! beside it. The plan `plan/2026-11-18-1-tasks-golden-tests.plan.md` +//! lists what each prompt does and what its log must show. + +use crate::golden::check; + +#[tokio::test(start_paused = true)] +async fn a_spawned_task_is_joined() { + check("01-spawn-and-join").await; +} From f70627cffff39f27c88d4d967d133a0b4df6e900 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:30:39 -0800 Subject: [PATCH 3/9] Test launching and lifetime of tasks Golden tests 2 to 6: a task survives its owner's move to the next section, stays queued while its owner's Lua runs, sees its inputs as they were at the spawn, waits for a slot under a limit of one, and waits for a slot under the default limit of eight. The mock model lists one model, since a model switch shows in the label alone. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- .../golden/tasks/02-survives-fall-through.log | 30 +++++ .../golden/tasks/02-survives-fall-through.md | 26 ++++ .../tasks/03-queued-until-owner-waits.log | 20 +++ .../tasks/03-queued-until-owner-waits.md | 22 ++++ .../tasks/04-inputs-copied-at-spawn.log | 17 +++ .../golden/tasks/04-inputs-copied-at-spawn.md | 22 ++++ .../tests/golden/tasks/05-limit-of-one.log | 33 +++++ .../tests/golden/tasks/05-limit-of-one.md | 21 ++++ .../tasks/06-default-limit-of-eight.log | 117 ++++++++++++++++++ .../golden/tasks/06-default-limit-of-eight.md | 22 ++++ crates/runvil/tests/suite/mock_model.rs | 23 +--- crates/runvil/tests/suite/tasks.rs | 25 ++++ 12 files changed, 361 insertions(+), 17 deletions(-) create mode 100644 crates/runvil/tests/golden/tasks/02-survives-fall-through.log create mode 100644 crates/runvil/tests/golden/tasks/02-survives-fall-through.md create mode 100644 crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.log create mode 100644 crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.md create mode 100644 crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.log create mode 100644 crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.md create mode 100644 crates/runvil/tests/golden/tasks/05-limit-of-one.log create mode 100644 crates/runvil/tests/golden/tasks/05-limit-of-one.md create mode 100644 crates/runvil/tests/golden/tasks/06-default-limit-of-eight.log create mode 100644 crates/runvil/tests/golden/tasks/06-default-limit-of-eight.md diff --git a/crates/runvil/tests/golden/tasks/02-survives-fall-through.log b/crates/runvil/tests/golden/tasks/02-survives-fall-through.log new file mode 100644 index 000000000..1400db8f7 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/02-survives-fall-through.log @@ -0,0 +1,30 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"survives-fall-through"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"First"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"First"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"First","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua","message":"timed out in First: true","provenance":{"seq":5,"task":"0"},"section":"First"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"First"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"First"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"First"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"First"} +0 10 event - {"kind":"section_started","provenance":{"seq":10,"task":"0"},"section":"Second"} +0 11 event - {"kind":"lua_chunk_started","provenance":{"seq":11,"task":"0"},"section":"Second"} +0.0 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"done","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"done","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 12 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":12,"task":"0"},"section":"Second"} +0 13 event - {"kind":"lua_teardown_started","provenance":{"seq":13,"task":"0"},"section":"Second"} +0 14 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":14,"task":"0"},"section":"Second"} +0 15 event - {"kind":"section_finished","provenance":{"seq":15,"task":"0"},"section":"Second"} +0 16 event - {"kind":"run_succeeded","provenance":{"seq":16,"task":"0"},"section":"survives-fall-through"} +result ok "done" diff --git a/crates/runvil/tests/golden/tasks/02-survives-fall-through.md b/crates/runvil/tests/golden/tasks/02-survives-fall-through.md new file mode 100644 index 000000000..50c3f8727 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/02-survives-fall-through.md @@ -0,0 +1,26 @@ +--- +name: survives-fall-through +description: Joins a task in the section after the one that spawned it +runvil: 0 +--- + +## First + +```lua +var.worker = tasks.spawn('../Worker') +local _, timed_out = tasks.join({ var.worker }, { timeout = 1 }) +log('timed out in First: ' .. tostring(timed_out)) +``` + +## Second + +```lua +local results = tasks.join({ var.worker }) +return results[1].result +``` + +## Worker + +```lua +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.log b/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.log new file mode 100644 index 000000000..6147b1b37 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.log @@ -0,0 +1,20 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"queued-until-owner-waits"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"kind":"lua","message":"queued","provenance":{"seq":3,"task":"0"},"section":"Main"} +0 4 event - {"kind":"lua","message":"queued","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua","message":"queued","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":6,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"run_succeeded","provenance":{"seq":11,"task":"0"},"section":"queued-until-owner-waits"} +result ok "done" diff --git a/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.md b/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.md new file mode 100644 index 000000000..6af186751 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.md @@ -0,0 +1,22 @@ +--- +name: queued-until-owner-waits +description: Reads a task's status in a loop that never waits +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +for _ = 1, 3 do + log(tasks.status(worker).blocked) +end +local results = tasks.join({ worker }) +return results[1].result +``` + +## Worker + +```lua +return 'done' +``` diff --git a/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.log b/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.log new file mode 100644 index 000000000..32dd74a2f --- /dev/null +++ b/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.log @@ -0,0 +1,17 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"inputs-copied-at-spawn"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":{"name":"ada"},"index":7,"item":"apple","kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{"color":"red"}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"inputs-copied-at-spawn"} +result ok "ada apple 7 red default | owner: blue strong" diff --git a/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.md b/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.md new file mode 100644 index 000000000..75c5a3e01 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.md @@ -0,0 +1,22 @@ +--- +name: inputs-copied-at-spawn +description: Changes var and the model after a spawn +runvil: 0 +--- + +## Main + +```lua +var.color = 'red' +local worker = tasks.spawn('../Worker', { args = { name = 'ada' }, item = 'apple', index = 7 }) +var.color = 'blue' +models.use('strong') +local results = tasks.join({ worker }) +return results[1].result .. ' | owner: ' .. var.color .. ' ' .. models.current().label +``` + +## Worker + +```lua +return args.name .. ' ' .. item .. ' ' .. sys.index .. ' ' .. var.color .. ' ' .. models.current().label +``` diff --git a/crates/runvil/tests/golden/tasks/05-limit-of-one.log b/crates/runvil/tests/golden/tasks/05-limit-of-one.log new file mode 100644 index 000000000..5a0486a8c --- /dev/null +++ b/crates/runvil/tests/golden/tasks/05-limit-of-one.log @@ -0,0 +1,33 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"limit-of-one"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":"first","kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 effect 0 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 1 reply first","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.0 2 answer 0 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"first","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"first","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 4 event - {"args":null,"index":null,"item":"second","kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.1","var":{}} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Worker"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Worker"} +0.1 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 1 reply second","role":"user"}],"round":1,"temperature":null,"thinking":null,"tools":[]}} +0.1 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"second","tool_calls":[]}}} +0.1 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.1"},"section":"Worker"} +0.1 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.1"},"round":1,"section":"Worker","text":"second","turn":1} +0.1 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.1"},"section":"Worker"} +0.1 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.1"},"section":"Worker"} +0.1 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.1"},"section":"Worker"} +0.1 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.1"},"section":"Worker"} +0.1 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.1"},"section":"Worker","task":"0.1"} +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"run_succeeded","provenance":{"seq":9,"task":"0"},"section":"limit-of-one"} +result ok "first second" diff --git a/crates/runvil/tests/golden/tasks/05-limit-of-one.md b/crates/runvil/tests/golden/tasks/05-limit-of-one.md new file mode 100644 index 000000000..3693755df --- /dev/null +++ b/crates/runvil/tests/golden/tasks/05-limit-of-one.md @@ -0,0 +1,21 @@ +--- +name: limit-of-one +description: Spawns two tasks under a limit of one +runvil: 0 +--- + +## Main + +```lua +tasks.concurrency(1) +local first = tasks.spawn('../Worker', { item = 'first' }) +local second = tasks.spawn('../Worker', { item = 'second' }) +local results = tasks.join({ first, second }) +return results[1].result .. ' ' .. results[2].result +``` + +## Worker + +```lua +return models.infer('after 1 reply ' .. item) +``` diff --git a/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.log b/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.log new file mode 100644 index 000000000..0932d24ad --- /dev/null +++ b/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.log @@ -0,0 +1,117 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"default-limit-of-eight"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":1,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0 4 event - {"args":null,"index":2,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.1","var":{}} +0 5 event - {"args":null,"index":3,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Worker","task":"0.2","var":{}} +0 6 event - {"args":null,"index":4,"item":null,"kind":"task_started","provenance":{"seq":6,"task":"0"},"section":"Main","target":"Worker","task":"0.3","var":{}} +0 7 event - {"args":null,"index":5,"item":null,"kind":"task_started","provenance":{"seq":7,"task":"0"},"section":"Main","target":"Worker","task":"0.4","var":{}} +0 8 event - {"args":null,"index":6,"item":null,"kind":"task_started","provenance":{"seq":8,"task":"0"},"section":"Main","target":"Worker","task":"0.5","var":{}} +0 9 event - {"args":null,"index":7,"item":null,"kind":"task_started","provenance":{"seq":9,"task":"0"},"section":"Main","target":"Worker","task":"0.6","var":{}} +0 10 event - {"args":null,"index":8,"item":null,"kind":"task_started","provenance":{"seq":10,"task":"0"},"section":"Main","target":"Worker","task":"0.7","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Worker"} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Worker"} +0.3 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.3"},"section":"Worker"} +0.4 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.4"},"section":"Worker"} +0.5 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.5"},"section":"Worker"} +0.6 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.6"},"section":"Worker"} +0.7 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.7"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Worker"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Worker"} +0.3 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.3"},"section":"Worker"} +0.4 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.4"},"section":"Worker"} +0.5 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.5"},"section":"Worker"} +0.6 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.6"},"section":"Worker"} +0.7 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.7"},"section":"Worker"} +0.0 2 effect 0 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 1 reply 1","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.1 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 2 reply 2","role":"user"}],"round":1,"temperature":null,"thinking":null,"tools":[]}} +0.2 2 effect 2 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 3 reply 3","role":"user"}],"round":2,"temperature":null,"thinking":null,"tools":[]}} +0.3 2 effect 3 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 4 reply 4","role":"user"}],"round":3,"temperature":null,"thinking":null,"tools":[]}} +0.4 2 effect 4 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply 5","role":"user"}],"round":4,"temperature":null,"thinking":null,"tools":[]}} +0.5 2 effect 5 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 6 reply 6","role":"user"}],"round":5,"temperature":null,"thinking":null,"tools":[]}} +0.6 2 effect 6 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 7 reply 7","role":"user"}],"round":6,"temperature":null,"thinking":null,"tools":[]}} +0.7 2 effect 7 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 8 reply 8","role":"user"}],"round":7,"temperature":null,"thinking":null,"tools":[]}} +0.0 2 answer 0 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"1","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"1","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 11 event - {"args":null,"index":9,"item":null,"kind":"task_started","provenance":{"seq":11,"task":"0"},"section":"Main","target":"Worker","task":"0.8","var":{}} +0.8 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.8"},"section":"Worker"} +0.8 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.8"},"section":"Worker"} +0.8 2 effect 8 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 9 reply 9","role":"user"}],"round":8,"temperature":null,"thinking":null,"tools":[]}} +0.1 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"2","tool_calls":[]}}} +0.1 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.1"},"section":"Worker"} +0.1 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.1"},"round":1,"section":"Worker","text":"2","turn":1} +0.1 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.1"},"section":"Worker"} +0.1 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.1"},"section":"Worker"} +0.1 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.1"},"section":"Worker"} +0.1 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.1"},"section":"Worker"} +0.1 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.1"},"section":"Worker","task":"0.1"} +0.2 2 answer 2 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"3","tool_calls":[]}}} +0.2 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.2"},"section":"Worker"} +0.2 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.2"},"round":2,"section":"Worker","text":"3","turn":1} +0.2 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.2"},"section":"Worker"} +0.2 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.2"},"section":"Worker"} +0.2 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.2"},"section":"Worker"} +0.2 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.2"},"section":"Worker"} +0.2 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.2"},"section":"Worker","task":"0.2"} +0.3 2 answer 3 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"4","tool_calls":[]}}} +0.3 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.3"},"section":"Worker"} +0.3 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.3"},"round":3,"section":"Worker","text":"4","turn":1} +0.3 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.3"},"section":"Worker"} +0.3 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.3"},"section":"Worker"} +0.3 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.3"},"section":"Worker"} +0.3 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.3"},"section":"Worker"} +0.3 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.3"},"section":"Worker","task":"0.3"} +0.4 2 answer 4 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"5","tool_calls":[]}}} +0.4 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.4"},"section":"Worker"} +0.4 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.4"},"round":4,"section":"Worker","text":"5","turn":1} +0.4 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.4"},"section":"Worker"} +0.4 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.4"},"section":"Worker"} +0.4 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.4"},"section":"Worker"} +0.4 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.4"},"section":"Worker"} +0.4 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.4"},"section":"Worker","task":"0.4"} +0.5 2 answer 5 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"6","tool_calls":[]}}} +0.5 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.5"},"section":"Worker"} +0.5 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.5"},"round":5,"section":"Worker","text":"6","turn":1} +0.5 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.5"},"section":"Worker"} +0.5 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.5"},"section":"Worker"} +0.5 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.5"},"section":"Worker"} +0.5 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.5"},"section":"Worker"} +0.5 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.5"},"section":"Worker","task":"0.5"} +0.6 2 answer 6 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"7","tool_calls":[]}}} +0.6 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.6"},"section":"Worker"} +0.6 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.6"},"round":6,"section":"Worker","text":"7","turn":1} +0.6 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.6"},"section":"Worker"} +0.6 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.6"},"section":"Worker"} +0.6 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.6"},"section":"Worker"} +0.6 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.6"},"section":"Worker"} +0.6 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.6"},"section":"Worker","task":"0.6"} +0.7 2 answer 7 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"8","tool_calls":[]}}} +0.7 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.7"},"section":"Worker"} +0.7 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.7"},"round":7,"section":"Worker","text":"8","turn":1} +0.7 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.7"},"section":"Worker"} +0.7 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.7"},"section":"Worker"} +0.7 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.7"},"section":"Worker"} +0.7 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.7"},"section":"Worker"} +0.7 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.7"},"section":"Worker","task":"0.7"} +0.8 2 answer 8 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"9","tool_calls":[]}}} +0.8 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.8"},"section":"Worker"} +0.8 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.8"},"round":8,"section":"Worker","text":"9","turn":1} +0.8 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.8"},"section":"Worker"} +0.8 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.8"},"section":"Worker"} +0.8 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.8"},"section":"Worker"} +0.8 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.8"},"section":"Worker"} +0.8 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.8"},"section":"Worker","task":"0.8"} +0 12 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":12,"task":"0"},"section":"Main"} +0 13 event - {"kind":"lua_teardown_started","provenance":{"seq":13,"task":"0"},"section":"Main"} +0 14 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":14,"task":"0"},"section":"Main"} +0 15 event - {"kind":"section_finished","provenance":{"seq":15,"task":"0"},"section":"Main"} +0 16 event - {"kind":"run_succeeded","provenance":{"seq":16,"task":"0"},"section":"default-limit-of-eight"} +result ok "9 joined" diff --git a/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.md b/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.md new file mode 100644 index 000000000..7bf814a32 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/06-default-limit-of-eight.md @@ -0,0 +1,22 @@ +--- +name: default-limit-of-eight +description: Spawns nine tasks under the default limit +runvil: 0 +--- + +## Main + +```lua +local set = {} +for index = 1, 9 do + set[index] = tasks.spawn('../Worker', { index = index }) +end +local results = tasks.join(set) +return #results .. ' joined' +``` + +## Worker + +```lua +return models.infer('after ' .. sys.index .. ' reply ' .. sys.index) +``` diff --git a/crates/runvil/tests/suite/mock_model.rs b/crates/runvil/tests/suite/mock_model.rs index 6bc710395..28b7fd358 100644 --- a/crates/runvil/tests/suite/mock_model.rs +++ b/crates/runvil/tests/suite/mock_model.rs @@ -32,10 +32,7 @@ use serde_json::Value; /// The model a golden run selects. pub(crate) const MODEL: &str = "mock-model"; -/// A second model, for a prompt that switches models. -const SECOND_MODEL: &str = "mock-model-2"; - -/// The context window both models list: 131,072 tokens. +/// The context window the model lists: 131,072 tokens. const WINDOW: NonZeroU32 = NonZeroU32::MIN.saturating_add(131_071); /// What separates an instruction's steps. @@ -50,26 +47,18 @@ enum Step { Call { tool: String, args: Value }, } -/// Lists two models and answers each round by its request's instruction. +/// Lists one model and answers each round by its request's instruction. pub(crate) struct MockModel { catalog: ModelCatalog, } impl MockModel { - /// A mock model listing [`MODEL`] and a second model. + /// A mock model listing [`MODEL`]. pub(crate) fn new() -> Result { - let mut models = Vec::new(); - for name in [MODEL, SECOND_MODEL] { - let id = ModelId::gateway(name)?; - models.push(ModelInfo::new( - id, - "a mock model", - WINDOW, - ThinkingMode::Never, - )); - } + let id = ModelId::gateway(MODEL)?; + let model = ModelInfo::new(id, "the mock model", WINDOW, ThinkingMode::Never); Ok(MockModel { - catalog: ModelCatalog::new(models)?, + catalog: ModelCatalog::new([model])?, }) } } diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs index 977896873..d9a251b75 100644 --- a/crates/runvil/tests/suite/tasks.rs +++ b/crates/runvil/tests/suite/tasks.rs @@ -9,3 +9,28 @@ use crate::golden::check; async fn a_spawned_task_is_joined() { check("01-spawn-and-join").await; } + +#[tokio::test(start_paused = true)] +async fn a_task_survives_its_owners_move_to_the_next_section() { + check("02-survives-fall-through").await; +} + +#[tokio::test(start_paused = true)] +async fn a_task_stays_queued_until_its_owner_waits() { + check("03-queued-until-owner-waits").await; +} + +#[tokio::test(start_paused = true)] +async fn a_task_sees_its_inputs_as_they_were_at_the_spawn() { + check("04-inputs-copied-at-spawn").await; +} + +#[tokio::test(start_paused = true)] +async fn a_limit_of_one_starts_the_second_task_after_the_first_ends() { + check("05-limit-of-one").await; +} + +#[tokio::test(start_paused = true)] +async fn the_default_limit_starts_eight_of_nine_tasks() { + check("06-default-limit-of-eight").await; +} From dd38c1e9b2e0e8b7fc09c4c05f6eb4cdce991192 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:31:26 -0800 Subject: [PATCH 4/9] Test waiting on tasks and their results Golden tests 7 to 12: `join` returns two tool loops' results in the order listed, `join_any` returns the first task to end, a wait returns a task's text and a failed task's error as data, a second wait on a delivered task is refused as `task_consumed`, an owner reads a task's store write after joining it, and a timed join ends while a later join collects the task. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- .../golden/tasks/07-join-two-tool-loops.log | 57 +++++++++++++++++++ .../golden/tasks/07-join-two-tool-loops.md | 25 ++++++++ .../golden/tasks/08-join-any-first-to-end.log | 30 ++++++++++ .../golden/tasks/08-join-any-first-to-end.md | 28 +++++++++ .../golden/tasks/09-result-and-error.log | 26 +++++++++ .../tests/golden/tasks/09-result-and-error.md | 28 +++++++++ .../tests/golden/tasks/10-wait-twice.log | 18 ++++++ .../tests/golden/tasks/10-wait-twice.md | 21 +++++++ .../golden/tasks/11-store-after-join.log | 23 ++++++++ .../tests/golden/tasks/11-store-after-join.md | 20 +++++++ .../tests/golden/tasks/12-timed-join.log | 24 ++++++++ .../tests/golden/tasks/12-timed-join.md | 21 +++++++ crates/runvil/tests/suite/tasks.rs | 30 ++++++++++ 13 files changed, 351 insertions(+) create mode 100644 crates/runvil/tests/golden/tasks/07-join-two-tool-loops.log create mode 100644 crates/runvil/tests/golden/tasks/07-join-two-tool-loops.md create mode 100644 crates/runvil/tests/golden/tasks/08-join-any-first-to-end.log create mode 100644 crates/runvil/tests/golden/tasks/08-join-any-first-to-end.md create mode 100644 crates/runvil/tests/golden/tasks/09-result-and-error.log create mode 100644 crates/runvil/tests/golden/tasks/09-result-and-error.md create mode 100644 crates/runvil/tests/golden/tasks/10-wait-twice.log create mode 100644 crates/runvil/tests/golden/tasks/10-wait-twice.md create mode 100644 crates/runvil/tests/golden/tasks/11-store-after-join.log create mode 100644 crates/runvil/tests/golden/tasks/11-store-after-join.md create mode 100644 crates/runvil/tests/golden/tasks/12-timed-join.log create mode 100644 crates/runvil/tests/golden/tasks/12-timed-join.md diff --git a/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.log b/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.log new file mode 100644 index 000000000..bf77c921b --- /dev/null +++ b/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.log @@ -0,0 +1,57 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"join-two-tool-loops"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":"first","kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":"second","kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Worker"} +0.0 2 effect 0 {"Render":{"alias":"default","entry":"0.0.0","offered":[{"dynamic_set":[],"pack":"echo","static_set":["echo/say"]}]}} +0.1 2 effect 1 {"Render":{"alias":"default","entry":"0.1.0","offered":[{"dynamic_set":[],"pack":"echo","static_set":["echo/say"]}]}} +0.0 2 answer 0 {"Render":[{"dynamic_set":[],"pack":"echo","static_set":[{"changes_files":false,"describes_tools":false,"description":"Says back the text it is given.","id":"echo/say","model_only_call":false,"parameters_schema":{"properties":{"text":{"type":"string"}},"required":["text"],"type":"object"},"structured_output":false,"survives_stop":false}]}]} +0.1 2 answer 1 {"Render":[{"dynamic_set":[],"pack":"echo","static_set":[{"changes_files":false,"describes_tools":false,"description":"Says back the text it is given.","id":"echo/say","model_only_call":false,"parameters_schema":{"properties":{"text":{"type":"string"}},"required":["text"],"type":"object"},"structured_output":false,"survives_stop":false}]}]} +0.0 3 event - {"kind":"tool_scope_validation_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"tool_scope_validation_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.1 3 event - {"kind":"tool_scope_validation_started","provenance":{"seq":3,"task":"0.1"},"section":"Worker"} +0.1 4 event - {"kind":"tool_scope_validation_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Worker"} +0.0 5 effect 2 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"call echo_say {\"text\":\"first\"} then reply first done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":["echo_say"]}} +0.1 5 effect 3 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"call echo_say {\"text\":\"second\"} then reply second done","role":"user"}],"round":1,"temperature":null,"thinking":null,"tools":["echo_say"]}} +0.0 5 answer 2 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":null,"tool_calls":["echo_say"]}}} +0.1 5 answer 3 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":null,"tool_calls":["echo_say"]}}} +0.0 6 event - {"kind":"model_turn_completed","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"calls":[{"arguments":{"text":"first"},"id":"call-0","name":"echo_say","tool":"echo/say","vendor":"acme"}],"kind":"assistant_tool_calls","model":"mock-model","provenance":{"seq":7,"task":"0.0"},"round":0,"section":"Worker","turn":1} +0.1 6 event - {"kind":"model_turn_completed","provenance":{"seq":6,"task":"0.1"},"section":"Worker"} +0.1 7 event - {"calls":[{"arguments":{"text":"second"},"id":"call-1","name":"echo_say","tool":"echo/say","vendor":"acme"}],"kind":"assistant_tool_calls","model":"mock-model","provenance":{"seq":7,"task":"0.1"},"round":1,"section":"Worker","turn":1} +0.0 8 effect 4 {"ToolCall":{"alias":"echo_say","args":{"text":"first"},"origin":{"caller":"model","entry":"0.0.0","model":"mock-model","section":"Worker"},"tool":"echo/say"}} +0.1 8 effect 5 {"ToolCall":{"alias":"echo_say","args":{"text":"second"},"origin":{"caller":"model","entry":"0.1.0","model":"mock-model","section":"Worker"},"tool":"echo/say"}} +0.0 8 answer 4 {"ToolCall":{"Ok":{"text":"first","trust":"trusted"}}} +0.1 8 answer 5 {"ToolCall":{"Ok":{"text":"second","trust":"trusted"}}} +0.0 9 event - {"kind":"tool_call_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker"} +0.0 10 event - {"alias":"echo_say","content":"first","kind":"tool_result","provenance":{"seq":10,"task":"0.0"},"section":"Worker","tool":"echo/say","tool_call_id":"call-0","trusted":true,"turn":1,"vendor":"acme"} +0.1 9 event - {"kind":"tool_call_succeeded","provenance":{"seq":9,"task":"0.1"},"section":"Worker"} +0.1 10 event - {"alias":"echo_say","content":"second","kind":"tool_result","provenance":{"seq":10,"task":"0.1"},"section":"Worker","tool":"echo/say","tool_call_id":"call-1","trusted":true,"turn":1,"vendor":"acme"} +0.0 11 effect 6 {"Chat":{"after":0,"alias":"default","keep":1,"max_tokens":null,"messages":[{"content":"","role":"assistant","tool_calls":[{"function":{"arguments":"{\"text\":\"first\"}","name":"echo_say"},"id":"call-0","type":"function"}]},{"content":"first","role":"tool","tool_call_id":"call-0"}],"round":2,"temperature":null,"thinking":null,"tools":["echo_say"]}} +0.1 11 effect 7 {"Chat":{"after":1,"alias":"default","keep":1,"max_tokens":null,"messages":[{"content":"","role":"assistant","tool_calls":[{"function":{"arguments":"{\"text\":\"second\"}","name":"echo_say"},"id":"call-1","type":"function"}]},{"content":"second","role":"tool","tool_call_id":"call-1"}],"round":3,"temperature":null,"thinking":null,"tools":["echo_say"]}} +0.0 11 answer 6 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"first done","tool_calls":[]}}} +0.1 11 answer 7 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"second done","tool_calls":[]}}} +0.0 12 event - {"kind":"model_turn_completed","provenance":{"seq":12,"task":"0.0"},"section":"Worker"} +0.0 13 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"chat","provenance":{"seq":13,"task":"0.0"},"round":2,"section":"Worker","text":"first done","turn":2} +0.1 12 event - {"kind":"model_turn_completed","provenance":{"seq":12,"task":"0.1"},"section":"Worker"} +0.1 13 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"chat","provenance":{"seq":13,"task":"0.1"},"round":3,"section":"Worker","text":"second done","turn":2} +0.0 14 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":14,"task":"0.0"},"section":"Worker"} +0.0 15 event - {"kind":"lua_teardown_started","provenance":{"seq":15,"task":"0.0"},"section":"Worker"} +0.0 16 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":16,"task":"0.0"},"section":"Worker"} +0.0 17 event - {"kind":"section_finished","provenance":{"seq":17,"task":"0.0"},"section":"Worker"} +0.0 18 event - {"kind":"task_succeeded","provenance":{"seq":18,"task":"0.0"},"section":"Worker","task":"0.0"} +0.1 14 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":14,"task":"0.1"},"section":"Worker"} +0.1 15 event - {"kind":"lua_teardown_started","provenance":{"seq":15,"task":"0.1"},"section":"Worker"} +0.1 16 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":16,"task":"0.1"},"section":"Worker"} +0.1 17 event - {"kind":"section_finished","provenance":{"seq":17,"task":"0.1"},"section":"Worker"} +0.1 18 event - {"kind":"task_succeeded","provenance":{"seq":18,"task":"0.1"},"section":"Worker","task":"0.1"} +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"run_succeeded","provenance":{"seq":9,"task":"0"},"section":"join-two-tool-loops"} +result ok "second done | first done" diff --git a/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.md b/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.md new file mode 100644 index 000000000..ab251bb47 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/07-join-two-tool-loops.md @@ -0,0 +1,25 @@ +--- +name: join-two-tool-loops +description: Joins two tasks that each run a tool loop +runvil: 0 +uses: + - acme/echo +--- + +## Main + +```lua +local first = tasks.spawn('../Worker', { item = 'first' }) +local second = tasks.spawn('../Worker', { item = 'second' }) +local results = tasks.join({ second, first }) +return results[1].result .. ' | ' .. results[2].result +``` + +## Worker + +```lua +tools.offer('echo/say') +messages:user('call echo_say {"text":"' .. item .. '"} then reply ' .. item .. ' done') +models.loop() +return messages[#messages].content +``` diff --git a/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.log b/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.log new file mode 100644 index 000000000..7700c9197 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.log @@ -0,0 +1,30 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"join-any-first-to-end"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Slow","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Quick","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Slow"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Quick"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Slow"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Quick"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Quick"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Quick"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Quick"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Quick"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Quick","task":"0.1"} +0 5 event - {"kind":"lua","message":"0.1 true quick","provenance":{"seq":5,"task":"0"},"section":"Main"} +0.0 2 effect 0 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply slow","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.0 2 answer 0 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"slow","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Slow"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Slow","text":"slow","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Slow"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Slow"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Slow"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Slow"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Slow","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"join-any-first-to-end"} +result ok "quick then slow" diff --git a/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.md b/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.md new file mode 100644 index 000000000..63689fa46 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/08-join-any-first-to-end.md @@ -0,0 +1,28 @@ +--- +name: join-any-first-to-end +description: Waits for the first of two tasks to end +runvil: 0 +--- + +## Main + +```lua +local slow = tasks.spawn('../Slow') +local quick = tasks.spawn('../Quick') +local task, ok, result = tasks.join_any({ slow, quick }) +log(task.task .. ' ' .. tostring(ok) .. ' ' .. result) +local rest = tasks.join({ slow }) +return result .. ' then ' .. rest[1].result +``` + +## Slow + +```lua +return models.infer('after 5 reply slow') +``` + +## Quick + +```lua +return 'quick' +``` diff --git a/crates/runvil/tests/golden/tasks/09-result-and-error.log b/crates/runvil/tests/golden/tasks/09-result-and-error.log new file mode 100644 index 000000000..3ce32e412 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/09-result-and-error.log @@ -0,0 +1,26 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"result-and-error"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Good","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Bad","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Good"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Bad"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Good"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Good"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Good"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Good"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Good"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Good","task":"0.0"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Bad"} +0.1 2 event - {"kind":"lua_chunk_failed","provenance":{"seq":2,"task":"0.1"},"section":"Bad"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Bad"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Bad"} +0.1 5 event - {"kind":"task_failed","provenance":{"seq":5,"task":"0.1"},"section":"Bad","task":"0.1"} +0 5 event - {"kind":"lua","message":"good: true text","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua","message":"bad: false lua 09-result-and-error.md:27: the task broke. The Lua code failed at runtime. Fix the code at this location.","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"run_succeeded","provenance":{"seq":11,"task":"0"},"section":"result-and-error"} +result ok "the owner ran on" diff --git a/crates/runvil/tests/golden/tasks/09-result-and-error.md b/crates/runvil/tests/golden/tasks/09-result-and-error.md new file mode 100644 index 000000000..a7d4e7d5c --- /dev/null +++ b/crates/runvil/tests/golden/tasks/09-result-and-error.md @@ -0,0 +1,28 @@ +--- +name: result-and-error +description: Joins a task that returns text and a task that raises +runvil: 0 +--- + +## Main + +```lua +local good = tasks.spawn('../Good') +local bad = tasks.spawn('../Bad') +local results = tasks.join({ good, bad }) +log('good: ' .. tostring(results[1].ok) .. ' ' .. results[1].result) +log('bad: ' .. tostring(results[2].ok) .. ' ' .. results[2].result.kind .. ' ' .. results[2].result.message) +return 'the owner ran on' +``` + +## Good + +```lua +return 'text' +``` + +## Bad + +```lua +error('the task broke') +``` diff --git a/crates/runvil/tests/golden/tasks/10-wait-twice.log b/crates/runvil/tests/golden/tasks/10-wait-twice.log new file mode 100644 index 000000000..fee7bfe58 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/10-wait-twice.log @@ -0,0 +1,18 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"wait-twice"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0 4 event - {"kind":"lua","message":"second wait: false task_consumed","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"run_succeeded","provenance":{"seq":9,"task":"0"},"section":"wait-twice"} +result ok "done" diff --git a/crates/runvil/tests/golden/tasks/10-wait-twice.md b/crates/runvil/tests/golden/tasks/10-wait-twice.md new file mode 100644 index 000000000..634c7a595 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/10-wait-twice.md @@ -0,0 +1,21 @@ +--- +name: wait-twice +description: Waits twice on one task +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local first = tasks.join({ worker }) +local ok, err = pcall(tasks.join, { worker }) +log('second wait: ' .. tostring(ok) .. ' ' .. err.kind) +return first[1].result +``` + +## Worker + +```lua +return 'done' +``` diff --git a/crates/runvil/tests/golden/tasks/11-store-after-join.log b/crates/runvil/tests/golden/tasks/11-store-after-join.log new file mode 100644 index 000000000..e43906f3a --- /dev/null +++ b/crates/runvil/tests/golden/tasks/11-store-after-join.log @@ -0,0 +1,23 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"store-after-join"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 effect 0 {"Vfs":{"op":{"Write":{"contents":"written by the task","path":"notes.md"}},"store":"0"}} +0.0 2 answer 0 {"Vfs":{"Ok":"Unit"}} +0.0 3 event - {"kind":"vfs_write_succeeded","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"task_succeeded","provenance":{"seq":8,"task":"0.0"},"section":"Worker","task":"0.0"} +0 4 effect 1 {"Vfs":{"op":{"Read":{"end":null,"path":"notes.md","start":null}},"store":"0"}} +0 4 answer 1 {"Vfs":{"Ok":{"Text":"written by the task"}}} +0 5 event - {"kind":"vfs_read_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"store-after-join"} +result ok "written by the task" diff --git a/crates/runvil/tests/golden/tasks/11-store-after-join.md b/crates/runvil/tests/golden/tasks/11-store-after-join.md new file mode 100644 index 000000000..1be9a7837 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/11-store-after-join.md @@ -0,0 +1,20 @@ +--- +name: store-after-join +description: Reads a task's store write after joining it +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +tasks.join({ worker }) +return store.read('notes.md') +``` + +## Worker + +```lua +store.write('notes.md', 'written by the task') +return 'wrote' +``` diff --git a/crates/runvil/tests/golden/tasks/12-timed-join.log b/crates/runvil/tests/golden/tasks/12-timed-join.log new file mode 100644 index 000000000..c28aeb9ca --- /dev/null +++ b/crates/runvil/tests/golden/tasks/12-timed-join.log @@ -0,0 +1,24 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"timed-join"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua","message":"timed out: true, entries: 0","provenance":{"seq":5,"task":"0"},"section":"Main"} +0.0 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"done","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"done","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"timed-join"} +result ok "done" diff --git a/crates/runvil/tests/golden/tasks/12-timed-join.md b/crates/runvil/tests/golden/tasks/12-timed-join.md new file mode 100644 index 000000000..562ea4897 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/12-timed-join.md @@ -0,0 +1,21 @@ +--- +name: timed-join +description: Time-limits a join on a task that waits on the model +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local results, timed_out = tasks.join({ worker }, { timeout = 1 }) +log('timed out: ' .. tostring(timed_out) .. ', entries: ' .. #results) +local later = tasks.join({ worker }) +return later[1].result +``` + +## Worker + +```lua +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs index d9a251b75..a65100ce3 100644 --- a/crates/runvil/tests/suite/tasks.rs +++ b/crates/runvil/tests/suite/tasks.rs @@ -34,3 +34,33 @@ async fn a_limit_of_one_starts_the_second_task_after_the_first_ends() { async fn the_default_limit_starts_eight_of_nine_tasks() { check("06-default-limit-of-eight").await; } + +#[tokio::test(start_paused = true)] +async fn join_returns_two_tool_loops_results_in_the_order_listed() { + check("07-join-two-tool-loops").await; +} + +#[tokio::test(start_paused = true)] +async fn join_any_returns_the_first_task_to_end() { + check("08-join-any-first-to-end").await; +} + +#[tokio::test(start_paused = true)] +async fn a_wait_returns_a_tasks_text_and_a_failed_tasks_error_as_data() { + check("09-result-and-error").await; +} + +#[tokio::test(start_paused = true)] +async fn a_second_wait_on_a_delivered_task_is_refused_as_consumed() { + check("10-wait-twice").await; +} + +#[tokio::test(start_paused = true)] +async fn an_owner_reads_a_tasks_store_write_after_joining_it() { + check("11-store-after-join").await; +} + +#[tokio::test(start_paused = true)] +async fn a_timed_join_ends_and_a_later_join_collects_the_task() { + check("12-timed-join").await; +} From 4250659ee8577e85aa2af5f0d432a3b72727b889 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:32:06 -0800 Subject: [PATCH 5/9] Test finding out what is running Golden tests 13 to 17: `tasks.pending` lists the tasks still live, `tasks.ready` reads false before a task ends and true after, it reads true while a reply waits and false once a wait takes it, `tasks.status` reads `queued`, `chat`, then `done` around a model call, and it shows the latest note a task set. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- .../runvil/tests/golden/tasks/13-pending.log | 42 +++++++++++++++++++ .../runvil/tests/golden/tasks/13-pending.md | 33 +++++++++++++++ .../tests/golden/tasks/14-ready-after-end.log | 31 ++++++++++++++ .../tests/golden/tasks/14-ready-after-end.md | 29 +++++++++++++ .../golden/tasks/15-ready-with-reply.log | 26 ++++++++++++ .../tests/golden/tasks/15-ready-with-reply.md | 31 ++++++++++++++ .../tasks/16-status-through-a-model-call.log | 26 ++++++++++++ .../tasks/16-status-through-a-model-call.md | 24 +++++++++++ crates/runvil/tests/golden/tasks/17-note.log | 24 +++++++++++ crates/runvil/tests/golden/tasks/17-note.md | 23 ++++++++++ crates/runvil/tests/suite/tasks.rs | 25 +++++++++++ 11 files changed, 314 insertions(+) create mode 100644 crates/runvil/tests/golden/tasks/13-pending.log create mode 100644 crates/runvil/tests/golden/tasks/13-pending.md create mode 100644 crates/runvil/tests/golden/tasks/14-ready-after-end.log create mode 100644 crates/runvil/tests/golden/tasks/14-ready-after-end.md create mode 100644 crates/runvil/tests/golden/tasks/15-ready-with-reply.log create mode 100644 crates/runvil/tests/golden/tasks/15-ready-with-reply.md create mode 100644 crates/runvil/tests/golden/tasks/16-status-through-a-model-call.log create mode 100644 crates/runvil/tests/golden/tasks/16-status-through-a-model-call.md create mode 100644 crates/runvil/tests/golden/tasks/17-note.log create mode 100644 crates/runvil/tests/golden/tasks/17-note.md diff --git a/crates/runvil/tests/golden/tasks/13-pending.log b/crates/runvil/tests/golden/tasks/13-pending.log new file mode 100644 index 000000000..c6e7e0e8d --- /dev/null +++ b/crates/runvil/tests/golden/tasks/13-pending.log @@ -0,0 +1,42 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"pending"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Slow","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Quick","task":"0.1","var":{}} +0 5 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Slow","task":"0.2","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Slow"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Quick"} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Slow"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Slow"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Quick"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Quick"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Quick"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Quick"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Quick"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Quick","task":"0.1"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Slow"} +0 6 event - {"kind":"lua","message":"pending: 0.0 0.2","provenance":{"seq":6,"task":"0"},"section":"Main"} +0.0 2 effect 0 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply slow","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.2 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply slow","role":"user"}],"round":1,"temperature":null,"thinking":null,"tools":[]}} +0.0 2 answer 0 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"slow","tool_calls":[]}}} +0.2 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"slow","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Slow"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Slow","text":"slow","turn":1} +0.2 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.2"},"section":"Slow"} +0.2 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.2"},"round":1,"section":"Slow","text":"slow","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Slow"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Slow"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Slow"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Slow"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Slow","task":"0.0"} +0.2 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.2"},"section":"Slow"} +0.2 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.2"},"section":"Slow"} +0.2 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.2"},"section":"Slow"} +0.2 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.2"},"section":"Slow"} +0.2 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.2"},"section":"Slow","task":"0.2"} +0 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"run_succeeded","provenance":{"seq":11,"task":"0"},"section":"pending"} +result ok "0 pending at the end" diff --git a/crates/runvil/tests/golden/tasks/13-pending.md b/crates/runvil/tests/golden/tasks/13-pending.md new file mode 100644 index 000000000..ea3ef6ef1 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/13-pending.md @@ -0,0 +1,33 @@ +--- +name: pending +description: Lists the tasks still live after one is joined +runvil: 0 +--- + +## Main + +```lua +local first = tasks.spawn('../Slow') +local second = tasks.spawn('../Quick') +local third = tasks.spawn('../Slow') +tasks.join({ second }) +local live = {} +for index, task in ipairs(tasks.pending()) do + live[index] = task.task +end +log('pending: ' .. table.concat(live, ' ')) +tasks.join({ first, third }) +return #tasks.pending() .. ' pending at the end' +``` + +## Slow + +```lua +return models.infer('after 5 reply slow') +``` + +## Quick + +```lua +return 'quick' +``` diff --git a/crates/runvil/tests/golden/tasks/14-ready-after-end.log b/crates/runvil/tests/golden/tasks/14-ready-after-end.log new file mode 100644 index 000000000..b46316082 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/14-ready-after-end.log @@ -0,0 +1,31 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"ready-after-end"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"kind":"lua","message":"before: false","provenance":{"seq":3,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Quick","task":"0.0","var":{}} +0 5 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Slow","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Quick"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Slow"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Quick"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Quick"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Quick"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Quick"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Quick"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Quick","task":"0.0"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Slow"} +0.1 2 effect 0 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply slow","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.1 2 answer 0 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"slow","tool_calls":[]}}} +0.1 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.1"},"section":"Slow"} +0.1 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.1"},"round":0,"section":"Slow","text":"slow","turn":1} +0.1 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.1"},"section":"Slow"} +0.1 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.1"},"section":"Slow"} +0.1 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.1"},"section":"Slow"} +0.1 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.1"},"section":"Slow"} +0.1 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.1"},"section":"Slow","task":"0.1"} +0 6 event - {"kind":"lua","message":"after: true","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"run_succeeded","provenance":{"seq":11,"task":"0"},"section":"ready-after-end"} +result ok "quick" diff --git a/crates/runvil/tests/golden/tasks/14-ready-after-end.md b/crates/runvil/tests/golden/tasks/14-ready-after-end.md new file mode 100644 index 000000000..bdf225820 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/14-ready-after-end.md @@ -0,0 +1,29 @@ +--- +name: ready-after-end +description: Reads tasks.ready before and after a task ends +runvil: 0 +--- + +## Main + +```lua +local quick = tasks.spawn('../Quick') +local slow = tasks.spawn('../Slow') +log('before: ' .. tostring(tasks.ready(quick))) +tasks.join({ slow }) +log('after: ' .. tostring(tasks.ready(quick))) +local results = tasks.join({ quick }) +return results[1].result +``` + +## Quick + +```lua +return 'quick' +``` + +## Slow + +```lua +return models.infer('after 5 reply slow') +``` diff --git a/crates/runvil/tests/golden/tasks/15-ready-with-reply.log b/crates/runvil/tests/golden/tasks/15-ready-with-reply.log new file mode 100644 index 000000000..df8f4bcd1 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/15-ready-with-reply.log @@ -0,0 +1,26 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"ready-with-reply"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Other","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Other"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"inbox_waiting","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Other"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Other"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Other"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Other"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Other"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Other","task":"0.1"} +0 5 event - {"kind":"lua","message":"reply waiting: true","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua","message":"reply taken: false","provenance":{"seq":6,"task":"0"},"section":"Main"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"task_canceled","provenance":{"seq":5,"task":"0.0"},"section":"Worker","task":"0.0"} +0 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"run_succeeded","provenance":{"seq":11,"task":"0"},"section":"ready-with-reply"} +result ok "a draft" diff --git a/crates/runvil/tests/golden/tasks/15-ready-with-reply.md b/crates/runvil/tests/golden/tasks/15-ready-with-reply.md new file mode 100644 index 000000000..5c66f302d --- /dev/null +++ b/crates/runvil/tests/golden/tasks/15-ready-with-reply.md @@ -0,0 +1,31 @@ +--- +name: ready-with-reply +description: Reads tasks.ready on a task that replied and stays open +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local other = tasks.spawn('../Other') +tasks.join({ other }) +log('reply waiting: ' .. tostring(tasks.ready(worker))) +local _, _, reply = tasks.join_any({ worker }) +log('reply taken: ' .. tostring(tasks.ready(worker))) +tasks.cancel(worker) +return reply +``` + +## Worker + +```lua +tasks.reply('a draft') +return 'never reached' +``` + +## Other + +```lua +return 'other' +``` diff --git a/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.log b/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.log new file mode 100644 index 000000000..3eea6b212 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.log @@ -0,0 +1,26 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"status-through-a-model-call"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"kind":"lua","message":"before: queued","provenance":{"seq":3,"task":"0"},"section":"Main"} +0 5 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0 4 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 4 answer 0 "Timer" +0 6 event - {"kind":"lua","message":"during: chat","provenance":{"seq":6,"task":"0"},"section":"Main"} +0.0 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"done","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"done","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 7 event - {"kind":"lua","message":"after: done, ok true","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_teardown_started","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"section_finished","provenance":{"seq":11,"task":"0"},"section":"Main"} +0 12 event - {"kind":"run_succeeded","provenance":{"seq":12,"task":"0"},"section":"status-through-a-model-call"} +result ok "read three times" diff --git a/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.md b/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.md new file mode 100644 index 000000000..bebce47db --- /dev/null +++ b/crates/runvil/tests/golden/tasks/16-status-through-a-model-call.md @@ -0,0 +1,24 @@ +--- +name: status-through-a-model-call +description: Reads a task's status before, during, and after its model call +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +log('before: ' .. tasks.status(worker).blocked) +tasks.join({ worker }, { timeout = 1 }) +log('during: ' .. tasks.status(worker).blocked) +tasks.join({ worker }) +local status = tasks.status(worker) +log('after: ' .. status.state .. ', ok ' .. tostring(status.ok)) +return 'read three times' +``` + +## Worker + +```lua +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/golden/tasks/17-note.log b/crates/runvil/tests/golden/tasks/17-note.log new file mode 100644 index 000000000..914c99b6b --- /dev/null +++ b/crates/runvil/tests/golden/tasks/17-note.log @@ -0,0 +1,24 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"note"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua","message":"note: second note","provenance":{"seq":5,"task":"0"},"section":"Main"} +0.0 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"done","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Worker","text":"done","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Worker","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"note"} +result ok "read the note" diff --git a/crates/runvil/tests/golden/tasks/17-note.md b/crates/runvil/tests/golden/tasks/17-note.md new file mode 100644 index 000000000..0ec8fe1e3 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/17-note.md @@ -0,0 +1,23 @@ +--- +name: note +description: Reads the latest progress note a task set +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +tasks.join({ worker }, { timeout = 1 }) +log('note: ' .. tasks.status(worker).note) +tasks.join({ worker }) +return 'read the note' +``` + +## Worker + +```lua +tasks.note('first note') +tasks.note('second note') +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs index a65100ce3..beda26ddb 100644 --- a/crates/runvil/tests/suite/tasks.rs +++ b/crates/runvil/tests/suite/tasks.rs @@ -64,3 +64,28 @@ async fn an_owner_reads_a_tasks_store_write_after_joining_it() { async fn a_timed_join_ends_and_a_later_join_collects_the_task() { check("12-timed-join").await; } + +#[tokio::test(start_paused = true)] +async fn pending_lists_the_tasks_still_live() { + check("13-pending").await; +} + +#[tokio::test(start_paused = true)] +async fn ready_reads_false_before_a_task_ends_and_true_after() { + check("14-ready-after-end").await; +} + +#[tokio::test(start_paused = true)] +async fn ready_reads_true_while_a_reply_waits_and_false_once_it_is_taken() { + check("15-ready-with-reply").await; +} + +#[tokio::test(start_paused = true)] +async fn status_reads_queued_then_chat_then_done_around_a_model_call() { + check("16-status-through-a-model-call").await; +} + +#[tokio::test(start_paused = true)] +async fn status_shows_the_latest_note_a_task_set() { + check("17-note").await; +} From c9d7ba313eabf2b6d04e28da454d43b1c00ce7b8 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:33:08 -0800 Subject: [PATCH 6/9] Test canceling, steering, and ownership of tasks Golden tests 18 to 23: a cancel stops a queued task, a task waiting on the model, and a task that owns a task; an owner steers a worker between its replies; `tasks.receive` returns messages oldest first and waits for the next; a task that acts on its sibling, on itself, or on a made-up id is refused as `task_not_owned`; and a task reads its own status. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- .../runvil/tests/golden/tasks/18-cancel.log | 36 ++++++++++++++++++ crates/runvil/tests/golden/tasks/18-cancel.md | 36 ++++++++++++++++++ .../golden/tasks/19-steer-with-replies.log | 25 ++++++++++++ .../golden/tasks/19-steer-with-replies.md | 29 ++++++++++++++ .../golden/tasks/20-receive-oldest-first.log | 24 ++++++++++++ .../golden/tasks/20-receive-oldest-first.md | 27 +++++++++++++ .../golden/tasks/21-sibling-not-owned.log | 29 ++++++++++++++ .../golden/tasks/21-sibling-not-owned.md | 38 +++++++++++++++++++ .../tasks/22-self-and-unknown-not-owned.log | 19 ++++++++++ .../tasks/22-self-and-unknown-not-owned.md | 23 +++++++++++ .../tests/golden/tasks/23-own-status.log | 17 +++++++++ .../tests/golden/tasks/23-own-status.md | 20 ++++++++++ crates/runvil/tests/suite/tasks.rs | 30 +++++++++++++++ 13 files changed, 353 insertions(+) create mode 100644 crates/runvil/tests/golden/tasks/18-cancel.log create mode 100644 crates/runvil/tests/golden/tasks/18-cancel.md create mode 100644 crates/runvil/tests/golden/tasks/19-steer-with-replies.log create mode 100644 crates/runvil/tests/golden/tasks/19-steer-with-replies.md create mode 100644 crates/runvil/tests/golden/tasks/20-receive-oldest-first.log create mode 100644 crates/runvil/tests/golden/tasks/20-receive-oldest-first.md create mode 100644 crates/runvil/tests/golden/tasks/21-sibling-not-owned.log create mode 100644 crates/runvil/tests/golden/tasks/21-sibling-not-owned.md create mode 100644 crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.log create mode 100644 crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.md create mode 100644 crates/runvil/tests/golden/tasks/23-own-status.log create mode 100644 crates/runvil/tests/golden/tasks/23-own-status.md diff --git a/crates/runvil/tests/golden/tasks/18-cancel.log b/crates/runvil/tests/golden/tasks/18-cancel.log new file mode 100644 index 000000000..604efcff8 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/18-cancel.log @@ -0,0 +1,36 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"cancel"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Waits","task":"0.1","var":{}} +0 5 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Owns","task":"0.2","var":{}} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Waits"} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Owns"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Waits"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Owns"} +0.2 2 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":2,"task":"0.2"},"section":"Owns","target":"Waits","task":"0.2.0","var":{}} +0.2.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2.0"},"section":"Waits"} +0.2.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2.0"},"section":"Waits"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.1 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0.2.0 2 effect 2 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":1,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Waits"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Waits"} +0.1 5 event - {"kind":"task_canceled","provenance":{"seq":5,"task":"0.1"},"section":"Waits","task":"0.1"} +0.2.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.2.0"},"section":"Waits"} +0.2.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.2.0"},"section":"Waits"} +0.2.0 5 event - {"kind":"task_abandoned","provenance":{"seq":5,"task":"0.2.0"},"reason":"owner_aborted","section":"Waits","task":"0.2.0"} +0.2 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.2"},"section":"Owns"} +0.2 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.2"},"section":"Owns"} +0.2 5 event - {"kind":"task_canceled","provenance":{"seq":5,"task":"0.2"},"section":"Owns","task":"0.2"} +0 6 event - {"kind":"lua","message":"0.0: false canceled","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua","message":"0.1: false canceled","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua","message":"0.2: false canceled","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"lua_teardown_started","provenance":{"seq":10,"task":"0"},"section":"Main"} +0 11 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":11,"task":"0"},"section":"Main"} +0 12 event - {"kind":"section_finished","provenance":{"seq":12,"task":"0"},"section":"Main"} +0 13 event - {"kind":"run_succeeded","provenance":{"seq":13,"task":"0"},"section":"cancel"} +0.1 2 answer 1 "Dropped" +0.2.0 2 answer 2 "Dropped" +result ok "canceled three" diff --git a/crates/runvil/tests/golden/tasks/18-cancel.md b/crates/runvil/tests/golden/tasks/18-cancel.md new file mode 100644 index 000000000..f6b5835f9 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/18-cancel.md @@ -0,0 +1,36 @@ +--- +name: cancel +description: Cancels a queued task, a task waiting on the model, and a task that owns a task +runvil: 0 +--- + +## Main + +```lua +local queued = tasks.spawn('../Waits') +tasks.cancel(queued) +local waiting = tasks.spawn('../Waits') +local owner = tasks.spawn('../Owns') +tasks.join({ waiting, owner }, { timeout = 1 }) +tasks.cancel(waiting) +tasks.cancel(owner) +for _, task in ipairs({ queued, waiting, owner }) do + local _, ok, err = tasks.join_any({ task }) + log(task.task .. ': ' .. tostring(ok) .. ' ' .. err.kind) +end +return 'canceled three' +``` + +## Waits + +```lua +return models.infer('after 5 reply done') +``` + +## Owns + +```lua +local inner = tasks.spawn('../Waits') +local results = tasks.join({ inner }) +return results[1].result +``` diff --git a/crates/runvil/tests/golden/tasks/19-steer-with-replies.log b/crates/runvil/tests/golden/tasks/19-steer-with-replies.log new file mode 100644 index 000000000..9a7382ccc --- /dev/null +++ b/crates/runvil/tests/golden/tasks/19-steer-with-replies.log @@ -0,0 +1,25 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"steer-with-replies"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"inbox_waiting","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0 4 event - {"kind":"lua","message":"first reply: a long draft","provenance":{"seq":4,"task":"0"},"section":"Main"} +0.0 3 event - {"kind":"inbox_message","provenance":{"seq":3,"task":"0.0"},"section":"Worker","text":"shorten it"} +0.0 4 event - {"kind":"lua","message":"worker got: shorten it","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"inbox_waiting","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0 5 event - {"kind":"lua","message":"second reply: a short draft","provenance":{"seq":5,"task":"0"},"section":"Main"} +0.0 6 event - {"kind":"inbox_message","provenance":{"seq":6,"task":"0.0"},"section":"Worker","text":"done"} +0.0 7 event - {"kind":"lua","message":"worker got: done","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"lua_teardown_started","provenance":{"seq":9,"task":"0.0"},"section":"Worker"} +0.0 10 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":10,"task":"0.0"},"section":"Worker"} +0.0 11 event - {"kind":"section_finished","provenance":{"seq":11,"task":"0.0"},"section":"Worker"} +0.0 12 event - {"kind":"task_succeeded","provenance":{"seq":12,"task":"0.0"},"section":"Worker","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"steer-with-replies"} +result ok "finished after done" diff --git a/crates/runvil/tests/golden/tasks/19-steer-with-replies.md b/crates/runvil/tests/golden/tasks/19-steer-with-replies.md new file mode 100644 index 000000000..9dcf91d2e --- /dev/null +++ b/crates/runvil/tests/golden/tasks/19-steer-with-replies.md @@ -0,0 +1,29 @@ +--- +name: steer-with-replies +description: Steers a worker with two messages between its replies +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local _, _, draft = tasks.join_any({ worker }) +log('first reply: ' .. draft) +tasks.send(worker, 'shorten it') +local _, _, shorter = tasks.join_any({ worker }) +log('second reply: ' .. shorter) +tasks.send(worker, 'done') +local results = tasks.join({ worker }) +return results[1].result +``` + +## Worker + +```lua +local message = tasks.reply('a long draft') +log('worker got: ' .. message) +message = tasks.reply('a short draft') +log('worker got: ' .. message) +return 'finished after ' .. message +``` diff --git a/crates/runvil/tests/golden/tasks/20-receive-oldest-first.log b/crates/runvil/tests/golden/tasks/20-receive-oldest-first.log new file mode 100644 index 000000000..951bcadcf --- /dev/null +++ b/crates/runvil/tests/golden/tasks/20-receive-oldest-first.log @@ -0,0 +1,24 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"receive-oldest-first"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"inbox_message","provenance":{"seq":0,"task":"0.0"},"section":"Worker","text":"one"} +0.0 1 event - {"kind":"inbox_message","provenance":{"seq":1,"task":"0.0"},"section":"Worker","text":"two"} +0.0 2 event - {"kind":"section_started","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_chunk_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"inbox_waiting","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua","message":"the worker waits on: inbox","provenance":{"seq":5,"task":"0"},"section":"Main"} +0.0 5 event - {"kind":"inbox_message","provenance":{"seq":5,"task":"0.0"},"section":"Worker","text":"three"} +0.0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker"} +0.0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0.0"},"section":"Worker"} +0.0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0.0"},"section":"Worker"} +0.0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0.0"},"section":"Worker"} +0.0 10 event - {"kind":"task_succeeded","provenance":{"seq":10,"task":"0.0"},"section":"Worker","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"receive-oldest-first"} +result ok "one two three" diff --git a/crates/runvil/tests/golden/tasks/20-receive-oldest-first.md b/crates/runvil/tests/golden/tasks/20-receive-oldest-first.md new file mode 100644 index 000000000..b489a1575 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/20-receive-oldest-first.md @@ -0,0 +1,27 @@ +--- +name: receive-oldest-first +description: Sends a task two messages before it runs and a third while it waits +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +tasks.send(worker, 'one') +tasks.send(worker, 'two') +tasks.join({ worker }, { timeout = 1 }) +log('the worker waits on: ' .. tasks.status(worker).blocked) +tasks.send(worker, 'three') +local results = tasks.join({ worker }) +return results[1].result +``` + +## Worker + +```lua +local first = tasks.receive() +local second = tasks.receive() +local third = tasks.receive() +return first .. ' ' .. second .. ' ' .. third +``` diff --git a/crates/runvil/tests/golden/tasks/21-sibling-not-owned.log b/crates/runvil/tests/golden/tasks/21-sibling-not-owned.log new file mode 100644 index 000000000..4ec694921 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/21-sibling-not-owned.log @@ -0,0 +1,29 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"sibling-not-owned"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Sibling","task":"0.0","var":{}} +0 4 event - {"args":null,"index":null,"item":"0.0","kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Prober","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Sibling"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Prober"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Sibling"} +0.0 2 event - {"kind":"inbox_waiting","provenance":{"seq":2,"task":"0.0"},"section":"Sibling"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Prober"} +0.1 2 event - {"kind":"lua","message":"join: false task_not_owned","provenance":{"seq":2,"task":"0.1"},"section":"Prober"} +0.1 3 event - {"kind":"lua","message":"ready: false task_not_owned","provenance":{"seq":3,"task":"0.1"},"section":"Prober"} +0.1 4 event - {"kind":"lua","message":"status: false task_not_owned","provenance":{"seq":4,"task":"0.1"},"section":"Prober"} +0.1 5 event - {"kind":"lua","message":"send: false task_not_owned","provenance":{"seq":5,"task":"0.1"},"section":"Prober"} +0.1 6 event - {"kind":"lua","message":"cancel: false task_not_owned","provenance":{"seq":6,"task":"0.1"},"section":"Prober"} +0.1 7 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":7,"task":"0.1"},"section":"Prober"} +0.1 8 event - {"kind":"lua_teardown_started","provenance":{"seq":8,"task":"0.1"},"section":"Prober"} +0.1 9 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":9,"task":"0.1"},"section":"Prober"} +0.1 10 event - {"kind":"section_finished","provenance":{"seq":10,"task":"0.1"},"section":"Prober"} +0.1 11 event - {"kind":"task_succeeded","provenance":{"seq":11,"task":"0.1"},"section":"Prober","task":"0.1"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Sibling"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Sibling"} +0.0 5 event - {"kind":"task_canceled","provenance":{"seq":5,"task":"0.0"},"section":"Sibling","task":"0.0"} +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"run_succeeded","provenance":{"seq":9,"task":"0"},"section":"sibling-not-owned"} +result ok "tried five calls on 0.0" diff --git a/crates/runvil/tests/golden/tasks/21-sibling-not-owned.md b/crates/runvil/tests/golden/tasks/21-sibling-not-owned.md new file mode 100644 index 000000000..ac109553a --- /dev/null +++ b/crates/runvil/tests/golden/tasks/21-sibling-not-owned.md @@ -0,0 +1,38 @@ +--- +name: sibling-not-owned +description: A task tries to act on its sibling +runvil: 0 +--- + +## Main + +```lua +local sibling = tasks.spawn('../Sibling') +local prober = tasks.spawn('../Prober', { item = sibling.task }) +local results = tasks.join({ prober }) +tasks.cancel(sibling) +return results[1].result +``` + +## Sibling + +```lua +return tasks.receive() +``` + +## Prober + +```lua +local tries = { + { 'join', function() return tasks.join({ item }) end }, + { 'ready', function() return tasks.ready(item) end }, + { 'status', function() return tasks.status(item) end }, + { 'send', function() return tasks.send(item, 'hello') end }, + { 'cancel', function() return tasks.cancel(item) end }, +} +for _, try in ipairs(tries) do + local ok, err = pcall(try[2]) + log(try[1] .. ': ' .. tostring(ok) .. ' ' .. err.kind) +end +return 'tried five calls on ' .. item +``` diff --git a/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.log b/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.log new file mode 100644 index 000000000..c3e20cc76 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.log @@ -0,0 +1,19 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"self-and-unknown-not-owned"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Prober","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Prober"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Prober"} +0.0 2 event - {"kind":"lua","message":"0.0: false task_not_owned","provenance":{"seq":2,"task":"0.0"},"section":"Prober"} +0.0 3 event - {"kind":"lua","message":"9.9: false task_not_owned","provenance":{"seq":3,"task":"0.0"},"section":"Prober"} +0.0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Prober"} +0.0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0.0"},"section":"Prober"} +0.0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Prober"} +0.0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0.0"},"section":"Prober"} +0.0 8 event - {"kind":"task_succeeded","provenance":{"seq":8,"task":"0.0"},"section":"Prober","task":"0.0"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"self-and-unknown-not-owned"} +result ok "tried two ids" diff --git a/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.md b/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.md new file mode 100644 index 000000000..69e1e8223 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.md @@ -0,0 +1,23 @@ +--- +name: self-and-unknown-not-owned +description: A task tries to cancel itself and a made-up id +runvil: 0 +--- + +## Main + +```lua +local prober = tasks.spawn('../Prober') +local results = tasks.join({ prober }) +return results[1].result +``` + +## Prober + +```lua +for _, id in ipairs({ sys.taskid, '9.9' }) do + local ok, err = pcall(tasks.cancel, id) + log(id .. ': ' .. tostring(ok) .. ' ' .. err.kind) +end +return 'tried two ids' +``` diff --git a/crates/runvil/tests/golden/tasks/23-own-status.log b/crates/runvil/tests/golden/tasks/23-own-status.log new file mode 100644 index 000000000..180d36135 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/23-own-status.log @@ -0,0 +1,17 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"own-status"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Prober","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Prober"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Prober"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Prober"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Prober"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Prober"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Prober"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Prober","task":"0.0"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"own-status"} +result ok "Prober is running" diff --git a/crates/runvil/tests/golden/tasks/23-own-status.md b/crates/runvil/tests/golden/tasks/23-own-status.md new file mode 100644 index 000000000..8217996d2 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/23-own-status.md @@ -0,0 +1,20 @@ +--- +name: own-status +description: A task reads its own status +runvil: 0 +--- + +## Main + +```lua +local prober = tasks.spawn('../Prober') +local results = tasks.join({ prober }) +return results[1].result +``` + +## Prober + +```lua +local status = tasks.status(sys.taskid) +return status.target .. ' is ' .. status.state +``` diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs index beda26ddb..507453055 100644 --- a/crates/runvil/tests/suite/tasks.rs +++ b/crates/runvil/tests/suite/tasks.rs @@ -89,3 +89,33 @@ async fn status_reads_queued_then_chat_then_done_around_a_model_call() { async fn status_shows_the_latest_note_a_task_set() { check("17-note").await; } + +#[tokio::test(start_paused = true)] +async fn cancel_stops_a_queued_task_a_waiting_task_and_a_task_that_owns_a_task() { + check("18-cancel").await; +} + +#[tokio::test(start_paused = true)] +async fn an_owner_steers_a_worker_between_its_replies() { + check("19-steer-with-replies").await; +} + +#[tokio::test(start_paused = true)] +async fn receive_returns_messages_oldest_first_and_waits_for_the_next() { + check("20-receive-oldest-first").await; +} + +#[tokio::test(start_paused = true)] +async fn a_task_that_acts_on_its_sibling_is_refused_as_not_the_owner() { + check("21-sibling-not-owned").await; +} + +#[tokio::test(start_paused = true)] +async fn a_task_that_cancels_itself_or_a_made_up_id_is_refused_as_not_the_owner() { + check("22-self-and-unknown-not-owned").await; +} + +#[tokio::test(start_paused = true)] +async fn a_task_reads_its_own_status() { + check("23-own-status").await; +} From 98ea037d7f500e06247160ab5acfc538fded0826 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:34:05 -0800 Subject: [PATCH 7/9] Test the endings, ids, and fanout of tasks Golden tests 24 to 30: a chain that returns with a live task fails with `tasks_live` and the task and its own task are abandoned, a chain that fails keeps its own error, a task waiting after a reply does not count as live, a task's task is numbered under it, a `call`, an `execute`, and a timed wait each take a child number, two runs of one prompt give the same log, and a fanout reports the task events of the same spawns made by hand. A guard test requires every prompt in the folder to have a stored log and a test that names it. The prompt that test 28 executes is in `helpers/`. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- .../golden/tasks/24-return-with-live-task.log | 25 ++++ .../golden/tasks/24-return-with-live-task.md | 27 +++++ .../golden/tasks/25-error-with-live-task.log | 18 +++ .../golden/tasks/25-error-with-live-task.md | 19 +++ .../tasks/26-return-with-waiting-task.log | 16 +++ .../tasks/26-return-with-waiting-task.md | 20 ++++ .../tests/golden/tasks/27-nested-ids.log | 25 ++++ .../tests/golden/tasks/27-nested-ids.md | 27 +++++ .../tasks/28-ids-skip-other-children.log | 39 +++++++ .../tasks/28-ids-skip-other-children.md | 30 +++++ .../tests/golden/tasks/29-same-log-twice.log | 31 +++++ .../tests/golden/tasks/29-same-log-twice.md | 27 +++++ .../runvil/tests/golden/tasks/30a-fanout.log | 33 ++++++ .../runvil/tests/golden/tasks/30a-fanout.md | 22 ++++ .../tests/golden/tasks/30b-spawns-by-hand.log | 33 ++++++ .../tests/golden/tasks/30b-spawns-by-hand.md | 25 ++++ .../tests/golden/tasks/helpers/greets.md | 11 ++ crates/runvil/tests/suite/golden.rs | 20 +++- crates/runvil/tests/suite/tasks.rs | 108 +++++++++++++++++- 19 files changed, 551 insertions(+), 5 deletions(-) create mode 100644 crates/runvil/tests/golden/tasks/24-return-with-live-task.log create mode 100644 crates/runvil/tests/golden/tasks/24-return-with-live-task.md create mode 100644 crates/runvil/tests/golden/tasks/25-error-with-live-task.log create mode 100644 crates/runvil/tests/golden/tasks/25-error-with-live-task.md create mode 100644 crates/runvil/tests/golden/tasks/26-return-with-waiting-task.log create mode 100644 crates/runvil/tests/golden/tasks/26-return-with-waiting-task.md create mode 100644 crates/runvil/tests/golden/tasks/27-nested-ids.log create mode 100644 crates/runvil/tests/golden/tasks/27-nested-ids.md create mode 100644 crates/runvil/tests/golden/tasks/28-ids-skip-other-children.log create mode 100644 crates/runvil/tests/golden/tasks/28-ids-skip-other-children.md create mode 100644 crates/runvil/tests/golden/tasks/29-same-log-twice.log create mode 100644 crates/runvil/tests/golden/tasks/29-same-log-twice.md create mode 100644 crates/runvil/tests/golden/tasks/30a-fanout.log create mode 100644 crates/runvil/tests/golden/tasks/30a-fanout.md create mode 100644 crates/runvil/tests/golden/tasks/30b-spawns-by-hand.log create mode 100644 crates/runvil/tests/golden/tasks/30b-spawns-by-hand.md create mode 100644 crates/runvil/tests/golden/tasks/helpers/greets.md diff --git a/crates/runvil/tests/golden/tasks/24-return-with-live-task.log b/crates/runvil/tests/golden/tasks/24-return-with-live-task.log new file mode 100644 index 000000000..fe093973d --- /dev/null +++ b/crates/runvil/tests/golden/tasks/24-return-with-live-task.log @@ -0,0 +1,25 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"return-with-live-task"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Owns","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Owns"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Owns"} +0.0 2 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":2,"task":"0.0"},"section":"Owns","target":"Waits","task":"0.0.0","var":{}} +0.0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0.0"},"section":"Waits"} +0.0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0.0"},"section":"Waits"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Main"} +0.0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0.0"},"section":"Waits"} +0.0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0.0"},"section":"Waits"} +0.0.0 5 event - {"kind":"task_abandoned","provenance":{"seq":5,"task":"0.0.0"},"reason":"owner_aborted","section":"Waits","task":"0.0.0"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Owns"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Owns"} +0.0 5 event - {"kind":"task_abandoned","provenance":{"seq":5,"task":"0.0"},"reason":"owner_returned","section":"Owns","task":"0.0"} +0 9 event - {"kind":"run_failed","provenance":{"seq":9,"task":"0"},"section":"return-with-live-task"} +0.0.0 2 answer 1 "Dropped" +result error tasks_live "chain ended with tasks still live: 0.0; wait on or cancel every task a chain spawns before it ends" diff --git a/crates/runvil/tests/golden/tasks/24-return-with-live-task.md b/crates/runvil/tests/golden/tasks/24-return-with-live-task.md new file mode 100644 index 000000000..6de5c4868 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/24-return-with-live-task.md @@ -0,0 +1,27 @@ +--- +name: return-with-live-task +description: Returns while a task that owns a task still runs +runvil: 0 +--- + +## Main + +```lua +local owner = tasks.spawn('../Owns') +tasks.join({ owner }, { timeout = 1 }) +return 'returned early' +``` + +## Owns + +```lua +local inner = tasks.spawn('../Waits') +local results = tasks.join({ inner }) +return results[1].result +``` + +## Waits + +```lua +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/golden/tasks/25-error-with-live-task.log b/crates/runvil/tests/golden/tasks/25-error-with-live-task.log new file mode 100644 index 000000000..88768fa20 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/25-error-with-live-task.log @@ -0,0 +1,18 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"error-with-live-task"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Waits","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Waits"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Waits"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply done","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0 5 event - {"kind":"lua_chunk_failed","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Main"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Waits"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Waits"} +0.0 5 event - {"kind":"task_abandoned","provenance":{"seq":5,"task":"0.0"},"reason":"owner_failed","section":"Waits","task":"0.0"} +0 8 event - {"kind":"run_failed","provenance":{"seq":8,"task":"0"},"section":"error-with-live-task"} +0.0 2 answer 1 "Dropped" +result error task_not_owned "task `9.9` is not a task this chain owns" diff --git a/crates/runvil/tests/golden/tasks/25-error-with-live-task.md b/crates/runvil/tests/golden/tasks/25-error-with-live-task.md new file mode 100644 index 000000000..8aa00deeb --- /dev/null +++ b/crates/runvil/tests/golden/tasks/25-error-with-live-task.md @@ -0,0 +1,19 @@ +--- +name: error-with-live-task +description: Fails with its own error while a task still runs +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Waits') +tasks.join({ worker }, { timeout = 1 }) +tasks.cancel('9.9') +``` + +## Waits + +```lua +return models.infer('after 5 reply done') +``` diff --git a/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.log b/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.log new file mode 100644 index 000000000..37ea66fd6 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.log @@ -0,0 +1,16 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"return-with-waiting-task"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"inbox_waiting","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"task_abandoned","provenance":{"seq":5,"task":"0.0"},"reason":"owner_returned","section":"Worker","task":"0.0"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"return-with-waiting-task"} +result ok "returned with a draft" diff --git a/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.md b/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.md new file mode 100644 index 000000000..4957eafff --- /dev/null +++ b/crates/runvil/tests/golden/tasks/26-return-with-waiting-task.md @@ -0,0 +1,20 @@ +--- +name: return-with-waiting-task +description: Returns while a task waits after a reply +runvil: 0 +--- + +## Main + +```lua +local worker = tasks.spawn('../Worker') +local _, _, reply = tasks.join_any({ worker }) +return 'returned with ' .. reply +``` + +## Worker + +```lua +tasks.reply('a draft') +return 'never reached' +``` diff --git a/crates/runvil/tests/golden/tasks/27-nested-ids.log b/crates/runvil/tests/golden/tasks/27-nested-ids.log new file mode 100644 index 000000000..18b37cd28 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/27-nested-ids.log @@ -0,0 +1,25 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"nested-ids"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Outer","task":"0.0","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Outer"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Outer"} +0.0 2 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":2,"task":"0.0"},"section":"Outer","target":"Inner","task":"0.0.0","var":{}} +0.0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0.0"},"section":"Inner"} +0.0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0.0"},"section":"Inner"} +0.0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0.0"},"section":"Inner"} +0.0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0.0"},"section":"Inner"} +0.0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0.0"},"section":"Inner"} +0.0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0.0"},"section":"Inner"} +0.0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0.0"},"section":"Inner","task":"0.0.0"} +0.0 3 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":3,"task":"0.0"},"section":"Outer"} +0.0 4 event - {"kind":"lua_teardown_started","provenance":{"seq":4,"task":"0.0"},"section":"Outer"} +0.0 5 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Outer"} +0.0 6 event - {"kind":"section_finished","provenance":{"seq":6,"task":"0.0"},"section":"Outer"} +0.0 7 event - {"kind":"task_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Outer","task":"0.0"} +0 4 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":4,"task":"0"},"section":"Main"} +0 5 event - {"kind":"lua_teardown_started","provenance":{"seq":5,"task":"0"},"section":"Main"} +0 6 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"section_finished","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"run_succeeded","provenance":{"seq":8,"task":"0"},"section":"nested-ids"} +result ok "0.0 owns 0.0.0" diff --git a/crates/runvil/tests/golden/tasks/27-nested-ids.md b/crates/runvil/tests/golden/tasks/27-nested-ids.md new file mode 100644 index 000000000..e10bf6801 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/27-nested-ids.md @@ -0,0 +1,27 @@ +--- +name: nested-ids +description: A task spawns a task of its own +runvil: 0 +--- + +## Main + +```lua +local outer = tasks.spawn('../Outer') +local results = tasks.join({ outer }) +return results[1].result +``` + +## Outer + +```lua +local inner = tasks.spawn('../Inner') +local results = tasks.join({ inner }) +return sys.taskid .. ' owns ' .. results[1].result +``` + +## Inner + +```lua +return sys.taskid +``` diff --git a/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.log b/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.log new file mode 100644 index 000000000..0cd477037 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.log @@ -0,0 +1,39 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"ids-skip-other-children"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"kind":"section_started","provenance":{"seq":3,"task":"0"},"section":"Called"} +0 4 event - {"kind":"lua_chunk_started","provenance":{"seq":4,"task":"0"},"section":"Called"} +0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0"},"section":"Called"} +0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0"},"section":"Called"} +0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0"},"section":"Called"} +0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0"},"section":"Called"} +0 9 event - {"kind":"section_started","provenance":{"seq":9,"task":"0"},"section":"Only"} +0 10 event - {"kind":"lua_chunk_started","provenance":{"seq":10,"task":"0"},"section":"Only"} +0 11 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":11,"task":"0"},"section":"Only"} +0 12 event - {"kind":"lua_teardown_started","provenance":{"seq":12,"task":"0"},"section":"Only"} +0 13 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":13,"task":"0"},"section":"Only"} +0 14 event - {"kind":"section_finished","provenance":{"seq":14,"task":"0"},"section":"Only"} +0 16 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":16,"task":"0"},"section":"Main","target":"Worker","task":"0.2","var":{}} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Worker"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Worker"} +0.2 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.2"},"section":"Worker"} +0.2 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.2"},"section":"Worker"} +0.2 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.2"},"section":"Worker"} +0.2 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.2"},"section":"Worker"} +0.2 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.2"},"section":"Worker","task":"0.2"} +0 17 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":17,"task":"0"},"section":"Main","target":"Worker","task":"0.4","var":{}} +0.4 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.4"},"section":"Worker"} +0.4 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.4"},"section":"Worker"} +0.4 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.4"},"section":"Worker"} +0.4 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.4"},"section":"Worker"} +0.4 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.4"},"section":"Worker"} +0.4 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.4"},"section":"Worker"} +0.4 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.4"},"section":"Worker","task":"0.4"} +0 18 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":18,"task":"0"},"section":"Main"} +0 19 event - {"kind":"lua_teardown_started","provenance":{"seq":19,"task":"0"},"section":"Main"} +0 20 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":20,"task":"0"},"section":"Main"} +0 21 event - {"kind":"section_finished","provenance":{"seq":21,"task":"0"},"section":"Main"} +0 22 event - {"kind":"run_succeeded","provenance":{"seq":22,"task":"0"},"section":"ids-skip-other-children"} +0 15 effect 0 {"Timer":{"seconds":1.0}} +0 15 answer 0 "Dropped" +result ok "called, hello Ada, 0.2, 0.4" diff --git a/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.md b/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.md new file mode 100644 index 000000000..d8d009a1e --- /dev/null +++ b/crates/runvil/tests/golden/tasks/28-ids-skip-other-children.md @@ -0,0 +1,30 @@ +--- +name: ids-skip-other-children +description: Spawns tasks after a call, an execute, and a timed wait +runvil: 0 +prompts: [greets.md] +--- + +## Main + +```lua +local called = call('../Called') +local executed = execute('greets.md', 'Ada') +local first = tasks.spawn('../Worker') +tasks.join({ first }, { timeout = 1 }) +local second = tasks.spawn('../Worker') +tasks.join({ second }) +return called .. ', ' .. executed .. ', ' .. first.task .. ', ' .. second.task +``` + +## Called + +```lua +return 'called' +``` + +## Worker + +```lua +return 'done' +``` diff --git a/crates/runvil/tests/golden/tasks/29-same-log-twice.log b/crates/runvil/tests/golden/tasks/29-same-log-twice.log new file mode 100644 index 000000000..90e08d291 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/29-same-log-twice.log @@ -0,0 +1,31 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"same-log-twice"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 4 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Slow","task":"0.0","var":{}} +0 5 event - {"args":null,"index":null,"item":null,"kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Quick","task":"0.1","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Slow"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Quick"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Slow"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Quick"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Quick"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Quick"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Quick"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Quick"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Quick","task":"0.1"} +0 3 effect 0 {"Timer":{"seconds":1.0}} +0.0 2 effect 1 {"Chat":{"after":null,"alias":"default","keep":0,"max_tokens":null,"messages":[{"content":"after 5 reply slow","role":"user"}],"round":0,"temperature":null,"thinking":null,"tools":[]}} +0 3 answer 0 "Timer" +0.0 2 answer 1 {"Chat":{"Ok":{"finish_reason":null,"model":"mock-model","reply":"slow","tool_calls":[]}}} +0.0 3 event - {"kind":"model_turn_completed","provenance":{"seq":3,"task":"0.0"},"section":"Slow"} +0.0 4 event - {"finish_reason":null,"kind":"assistant_reply","metrics":null,"model":"mock-model","origin":"infer","provenance":{"seq":4,"task":"0.0"},"round":0,"section":"Slow","text":"slow","turn":1} +0.0 5 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":5,"task":"0.0"},"section":"Slow"} +0.0 6 event - {"kind":"lua_teardown_started","provenance":{"seq":6,"task":"0.0"},"section":"Slow"} +0.0 7 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":7,"task":"0.0"},"section":"Slow"} +0.0 8 event - {"kind":"section_finished","provenance":{"seq":8,"task":"0.0"},"section":"Slow"} +0.0 9 event - {"kind":"task_succeeded","provenance":{"seq":9,"task":"0.0"},"section":"Slow","task":"0.0"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"same-log-twice"} +result ok "true quick slow" diff --git a/crates/runvil/tests/golden/tasks/29-same-log-twice.md b/crates/runvil/tests/golden/tasks/29-same-log-twice.md new file mode 100644 index 000000000..1d62cc6bd --- /dev/null +++ b/crates/runvil/tests/golden/tasks/29-same-log-twice.md @@ -0,0 +1,27 @@ +--- +name: same-log-twice +description: Two tasks, a timed join, and a delayed model reply +runvil: 0 +--- + +## Main + +```lua +local slow = tasks.spawn('../Slow') +local quick = tasks.spawn('../Quick') +local first, timed_out = tasks.join({ slow, quick }, { timeout = 1 }) +local second = tasks.join({ slow }) +return tostring(timed_out) .. ' ' .. first[2].result .. ' ' .. second[1].result +``` + +## Slow + +```lua +return models.infer('after 5 reply slow') +``` + +## Quick + +```lua +return 'quick' +``` diff --git a/crates/runvil/tests/golden/tasks/30a-fanout.log b/crates/runvil/tests/golden/tasks/30a-fanout.log new file mode 100644 index 000000000..5edef0e3d --- /dev/null +++ b/crates/runvil/tests/golden/tasks/30a-fanout.log @@ -0,0 +1,33 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"fanout"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":1,"item":"red","kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0 4 event - {"args":null,"index":2,"item":"green","kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.1","var":{}} +0 5 event - {"args":null,"index":3,"item":"blue","kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Worker","task":"0.2","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Worker"} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Worker"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Worker"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Worker"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Worker"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Worker"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Worker","task":"0.1"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Worker"} +0.2 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.2"},"section":"Worker"} +0.2 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.2"},"section":"Worker"} +0.2 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.2"},"section":"Worker"} +0.2 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.2"},"section":"Worker"} +0.2 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.2"},"section":"Worker","task":"0.2"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"fanout"} +result ok "red 1, green 2, blue 3" diff --git a/crates/runvil/tests/golden/tasks/30a-fanout.md b/crates/runvil/tests/golden/tasks/30a-fanout.md new file mode 100644 index 000000000..b94e625ce --- /dev/null +++ b/crates/runvil/tests/golden/tasks/30a-fanout.md @@ -0,0 +1,22 @@ +--- +name: fanout +description: Fans out over three members +runvil: 0 +--- + +## Main + +```lua +local results = tasks.fanout('../Worker', { 'red', 'green', 'blue' }) +local texts = {} +for index, result in ipairs(results) do + texts[index] = result.text +end +return table.concat(texts, ', ') +``` + +## Worker + +```lua +return item .. ' ' .. sys.index +``` diff --git a/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.log b/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.log new file mode 100644 index 000000000..c8695042f --- /dev/null +++ b/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.log @@ -0,0 +1,33 @@ +0 0 event - {"kind":"run_started","provenance":{"seq":0,"task":"0"},"section":"spawns-by-hand"} +0 1 event - {"kind":"section_started","provenance":{"seq":1,"task":"0"},"section":"Main"} +0 2 event - {"kind":"lua_chunk_started","provenance":{"seq":2,"task":"0"},"section":"Main"} +0 3 event - {"args":null,"index":1,"item":"red","kind":"task_started","provenance":{"seq":3,"task":"0"},"section":"Main","target":"Worker","task":"0.0","var":{}} +0 4 event - {"args":null,"index":2,"item":"green","kind":"task_started","provenance":{"seq":4,"task":"0"},"section":"Main","target":"Worker","task":"0.1","var":{}} +0 5 event - {"args":null,"index":3,"item":"blue","kind":"task_started","provenance":{"seq":5,"task":"0"},"section":"Main","target":"Worker","task":"0.2","var":{}} +0.0 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.0"},"section":"Worker"} +0.1 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.1"},"section":"Worker"} +0.2 0 event - {"kind":"section_started","provenance":{"seq":0,"task":"0.2"},"section":"Worker"} +0.0 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.0"},"section":"Worker"} +0.0 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.0"},"section":"Worker"} +0.0 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.0"},"section":"Worker"} +0.0 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.0"},"section":"Worker"} +0.0 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.0"},"section":"Worker"} +0.0 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.0"},"section":"Worker","task":"0.0"} +0.1 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.1"},"section":"Worker"} +0.1 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.1"},"section":"Worker"} +0.1 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.1"},"section":"Worker"} +0.1 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.1"},"section":"Worker"} +0.1 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.1"},"section":"Worker"} +0.1 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.1"},"section":"Worker","task":"0.1"} +0.2 1 event - {"kind":"lua_chunk_started","provenance":{"seq":1,"task":"0.2"},"section":"Worker"} +0.2 2 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":2,"task":"0.2"},"section":"Worker"} +0.2 3 event - {"kind":"lua_teardown_started","provenance":{"seq":3,"task":"0.2"},"section":"Worker"} +0.2 4 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":4,"task":"0.2"},"section":"Worker"} +0.2 5 event - {"kind":"section_finished","provenance":{"seq":5,"task":"0.2"},"section":"Worker"} +0.2 6 event - {"kind":"task_succeeded","provenance":{"seq":6,"task":"0.2"},"section":"Worker","task":"0.2"} +0 6 event - {"kind":"lua_chunk_succeeded","provenance":{"seq":6,"task":"0"},"section":"Main"} +0 7 event - {"kind":"lua_teardown_started","provenance":{"seq":7,"task":"0"},"section":"Main"} +0 8 event - {"kind":"lua_teardown_succeeded","provenance":{"seq":8,"task":"0"},"section":"Main"} +0 9 event - {"kind":"section_finished","provenance":{"seq":9,"task":"0"},"section":"Main"} +0 10 event - {"kind":"run_succeeded","provenance":{"seq":10,"task":"0"},"section":"spawns-by-hand"} +result ok "red 1, green 2, blue 3" diff --git a/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.md b/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.md new file mode 100644 index 000000000..97957869d --- /dev/null +++ b/crates/runvil/tests/golden/tasks/30b-spawns-by-hand.md @@ -0,0 +1,25 @@ +--- +name: spawns-by-hand +description: Spawns and joins three tasks by hand +runvil: 0 +--- + +## Main + +```lua +local set = {} +for index, color in ipairs({ 'red', 'green', 'blue' }) do + set[index] = tasks.spawn('../Worker', { item = color, index = index }) +end +local texts = {} +for index, result in ipairs(tasks.join(set)) do + texts[index] = result.result +end +return table.concat(texts, ', ') +``` + +## Worker + +```lua +return item .. ' ' .. sys.index +``` diff --git a/crates/runvil/tests/golden/tasks/helpers/greets.md b/crates/runvil/tests/golden/tasks/helpers/greets.md new file mode 100644 index 000000000..76b4e3f16 --- /dev/null +++ b/crates/runvil/tests/golden/tasks/helpers/greets.md @@ -0,0 +1,11 @@ +--- +name: greets +description: Greets the text it is given +runvil: 0 +--- + +## Only + +```lua +return 'hello ' .. args.text +``` diff --git a/crates/runvil/tests/suite/golden.rs b/crates/runvil/tests/suite/golden.rs index e95492129..7c254fb9a 100644 --- a/crates/runvil/tests/suite/golden.rs +++ b/crates/runvil/tests/suite/golden.rs @@ -10,7 +10,8 @@ //! //! Every golden run has the same Host and the same setup, so a test //! differs from another only in its prompt: the Host finds prompts by file -//! name in the tests' folder, holds the built-in +//! name in the tests' folder and in its `helpers/` folder, which holds the +//! prompts a test executes, holds the built-in //! Packs and [`ECHO`], sleeps on tokio's clock, which each test pauses, and //! stamps every run's start as the Unix epoch. The model is the //! [`MockModel`]. The Harness draws a new seed for every run, so a golden @@ -96,6 +97,12 @@ impl Timer for PausedClock { } } +/// Whether [`BLESS`] is set, so this process stores logs and compares +/// none. +pub(crate) fn blessing() -> bool { + std::env::var_os(BLESS).is_some() +} + /// The folder that holds the golden prompts and their logs. pub(crate) fn folder() -> PathBuf { PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/golden/tasks") @@ -108,8 +115,13 @@ fn host() -> Arc { panic!("the registry takes the echo Pack: {error}"); } let mut prompts = PromptSet::new(); - if let Err(error) = prompts.insert_dir(folder()) { - panic!("the golden folder is listed: {error}"); + for directory in [folder(), folder().join("helpers")] { + if let Err(error) = prompts.insert_dir(&directory) { + panic!( + "the golden folder {} is listed: {error}", + directory.display() + ); + } } Arc::new(Host { packs: Packs::new(registry), @@ -192,7 +204,7 @@ fn difference(expected: &str, actual: &str) -> Option { pub(crate) async fn check(stem: &str) -> String { let actual = run(stem).await; let path = folder().join(format!("{stem}.log")); - if std::env::var_os(BLESS).is_some() { + if blessing() { if let Err(error) = std::fs::write(&path, &actual) { panic!("cannot store {}: {error}", path.display()); } diff --git a/crates/runvil/tests/suite/tasks.rs b/crates/runvil/tests/suite/tasks.rs index 507453055..054723372 100644 --- a/crates/runvil/tests/suite/tasks.rs +++ b/crates/runvil/tests/suite/tasks.rs @@ -3,7 +3,11 @@ //! beside it. The plan `plan/2026-11-18-1-tasks-golden-tests.plan.md` //! lists what each prompt does and what its log must show. -use crate::golden::check; +use std::collections::BTreeSet; + +use serde_json::Value; + +use crate::golden::{blessing, check, folder}; #[tokio::test(start_paused = true)] async fn a_spawned_task_is_joined() { @@ -119,3 +123,105 @@ async fn a_task_that_cancels_itself_or_a_made_up_id_is_refused_as_not_the_owner( async fn a_task_reads_its_own_status() { check("23-own-status").await; } + +#[tokio::test(start_paused = true)] +async fn a_chain_that_returns_with_a_live_task_fails_and_the_task_is_abandoned() { + check("24-return-with-live-task").await; +} + +#[tokio::test(start_paused = true)] +async fn a_chain_that_fails_with_a_live_task_keeps_its_own_error() { + check("25-error-with-live-task").await; +} + +#[tokio::test(start_paused = true)] +async fn a_chain_that_returns_with_a_task_waiting_after_a_reply_ends_normally() { + check("26-return-with-waiting-task").await; +} + +#[tokio::test(start_paused = true)] +async fn a_tasks_task_is_numbered_under_it() { + check("27-nested-ids").await; +} + +#[tokio::test(start_paused = true)] +async fn a_call_an_execute_and_a_timed_wait_each_take_a_child_number() { + check("28-ids-skip-other-children").await; +} + +#[tokio::test(start_paused = true)] +async fn two_runs_of_one_prompt_give_the_same_log() { + let first = check("29-same-log-twice").await; + let second = check("29-same-log-twice").await; + assert_eq!(first, second, "the second run's log differs from the first"); +} + +/// The task events of `log`, in order, each without its provenance: the +/// starts and endings of tasks, apart from where their chains stood. +fn task_events(log: &str) -> Vec { + log.lines() + .filter_map(|line| line.splitn(5, ' ').nth(4)) + .filter_map(|payload| serde_json::from_str::(payload).ok()) + .filter(|payload| { + payload["kind"] + .as_str() + .is_some_and(|kind| kind.starts_with("task_")) + }) + .map(|mut payload| { + if let Some(fields) = payload.as_object_mut() { + fields.remove("provenance"); + } + payload + }) + .collect() +} + +#[tokio::test(start_paused = true)] +async fn a_fanout_reports_the_task_events_of_the_same_spawns_made_by_hand() { + let fanout = task_events(&check("30a-fanout").await); + let by_hand = task_events(&check("30b-spawns-by-hand").await); + assert_eq!( + fanout.len(), + 6, + "three starts and three endings: {fanout:?}" + ); + assert_eq!(fanout, by_hand); +} + +/// The stems of the files in the golden folder with the extension +/// `extension`. +fn stems(extension: &str) -> BTreeSet { + let entries = match std::fs::read_dir(folder()) { + Ok(entries) => entries, + Err(error) => panic!("the golden folder is listed: {error}"), + }; + entries + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.extension().is_some_and(|found| found == extension)) + .filter_map(|path| Some(path.file_stem()?.to_str()?.to_owned())) + .collect() +} + +#[test] +fn every_golden_prompt_has_a_stored_log_and_a_test_that_names_it() { + let prompts = stems("md"); + // While the tests beside this one are storing their logs, the folder + // holds only the logs stored so far. + let logs = if blessing() { + prompts.clone() + } else { + stems("log") + }; + assert_eq!( + prompts, logs, + "each prompt has one stored log, and each stored log one prompt" + ); + let tests = include_str!("tasks.rs"); + for stem in &prompts { + assert!( + tests.contains(&format!("check(\"{stem}\")")), + "no test in tasks.rs runs {stem}.md" + ); + } +} From 8167c4d0d065c137b890a87ea78c4299bd089801 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:42:24 -0800 Subject: [PATCH 8/9] Record what the golden tests of tasks found Every stored log matched its row of the plan's table. Two things differ from the first draft: the mock model lists one model, and test 25's owner fails through a refused call, because Lua's `error()` puts line numbers of an internal Lua file into the run's error. The plan also records the verification results and the overlap with the existing tests. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- plan/2026-11-18-1-tasks-golden-tests.plan.md | 30 +++++++++++++------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/plan/2026-11-18-1-tasks-golden-tests.plan.md b/plan/2026-11-18-1-tasks-golden-tests.plan.md index d6275c374..8a8756d8a 100644 --- a/plan/2026-11-18-1-tasks-golden-tests.plan.md +++ b/plan/2026-11-18-1-tasks-golden-tests.plan.md @@ -4,25 +4,25 @@ overview: Add 30 tests of Lua's `tasks` table to the `runvil` crate's suite. Eac todos: - id: plan-copy content: "Copy this plan to plan/2026-11-18-1-tasks-golden-tests.plan.md on branch tasks-golden-tests and commit it alone" - status: pending + status: completed - id: runner content: "Add the runner, the mock model, and test 1, and show that a changed log line fails the test" - status: pending + status: completed - id: cases content: "Add tests 2 to 30, one commit per area, checking each generated log against its row in the table" - status: pending + status: completed - id: overlap content: "List which existing tests already cover each of the 30, for the PR description" - status: pending + status: completed - id: gist content: "Reword Test #29 in the gist to drop \"the same seed\"" - status: pending + status: completed - id: verify content: "Run every verification command and vibelight" - status: pending + status: completed - id: pr content: "Push the branch to the fork and open one PR against cppalliance/runvil-harness" - status: pending + status: in_progress --- # Golden tests for tasks @@ -72,7 +72,7 @@ A developer writes a prompt file, runs the suite once in bless mode to write its - `call ` asks for one tool call. Steps joined by ` then ` are taken one per round, so `call echo {} then reply done` is a two-round tool loop. - Any other text fails the round with an error that quotes the text. - States and validation: - - A guard test fails when a prompt file has no log, a log has no prompt, or a prompt file is not named in the suite. + - A guard test fails when a prompt file has no log, a log has no prompt, or a prompt file is not named in the suite. While a run is storing logs, the guard checks only that each prompt is named. - A missing log outside bless mode fails the test and names the bless command. - Errors and recovery: - A mismatch reports the file, the first differing line number, the expected line, and the actual line. @@ -142,7 +142,7 @@ A developer writes a prompt file, runs the suite once in bless mode to write its | 22 | `22-self-and-unknown-not-owned` | A task tries to cancel itself and the made-up id `9.9`. | Each raises `task_not_owned`. | 22, 23 | | 23 | `23-own-status` | A task reads `tasks.status(sys.taskid)`. | Its own status, with state `running`. | 22 | | 24 | `24-return-with-live-task` | Returns while a task that owns a task runs. | The run fails with `tasks_live` and both end `abandoned`. | 24, 25 | -| 25 | `25-error-with-live-task` | Raises an error while a task runs. | The run fails with that error, and the task ends `abandoned`. | 24 | +| 25 | `25-error-with-live-task` | Makes a call that is refused, uncaught, while a task runs. | The run fails with that call's error, `task_not_owned`, and the task ends `abandoned`. | 24 | | 26 | `26-return-with-waiting-task` | Returns while a task waits after a reply. | The run ends normally and the task ends `abandoned`. | 24 | | 27 | `27-nested-ids` | A task spawns a task. | Ids `0.0` and `0.0.0`, each with one start and one terminal event. | 26, 27 | | 28 | `28-ids-skip-other-children` | Makes a `call` and an `execute`, spawns a task, waits on it with a time limit, spawns another. | The tasks are `0.2` and `0.4`. | 26 | @@ -162,8 +162,8 @@ The change is test code and test data in the `runvil` crate, plus one dev-depend - The clock is tokio's, paused. A sleep completes in the order of its deadline with no real waiting, which fixes the order in which timers and delayed model replies answer. - Modules and interfaces: - `golden`: builds the Host and the `RunSetup`, runs a prompt by file stem, renders the record, and compares or blesses. Its Host finds prompts by file name in the test folder, sleeps on the paused clock, and stamps every run's start as the Unix epoch. - - `mock_model`: an `InferenceBroker` that lists two models, `mock-model` and `mock-model-2`, and answers by the instruction grammar in the Functional Specification. - - `tasks`: one line per test naming its file stem, the guard test, and the two tests that compare logs with each other (29 and 30). + - `mock_model`: an `InferenceBroker` that lists one model, `mock-model`, and answers by the instruction grammar in the Functional Specification. + - `tasks`: one short test per prompt naming its file stem, the guard test, and the two tests that compare logs with each other (29 and 30). - One fixture Pack with a single `echo` tool, for the tool loops of test 7, registered as `crates/runvil/tests/suite/vfs.rs` registers its `acme/desk` Pack. - File and public API changes: - New: `crates/runvil/tests/suite/golden.rs`, `mock_model.rs`, `tasks.rs`, and three `mod` lines in `main.rs`. @@ -217,6 +217,9 @@ The 30 tests are the deliverable, so the plan for testing is about trusting the - The clock is paused, not real. Tests of timeouts and of "while a task waits on the model" need a fixed order of answers, and real sleeps would make the order depend on the machine. - The tests join the existing `suite` target. A new test target or crate would add a link step and nothing else. - The existing tests stay untouched, and a new test may cover what an existing one covers. The new suite's purpose is a readable prompt and outcome for each behavior. The PR description lists the overlap. + - The mock model lists one model. Test 4 switches models with `models.use('strong')`, and the switch shows in the label that `models.current()` reports, so a second model would add nothing. + - In test 25 the owner fails through a refused call, `tasks.cancel('9.9')`, left uncaught. Lua's `error()` puts a stack trace in the run's error, and the trace names line numbers of `crates/runvil-internal/lua/src/__impl_coro.lua`, so the stored log would change with any edit of that file. + - A prompt reports what it observed with `log()`, which the record holds as an event, and returns a short summary. Each observation is then one line of the stored log. - Rejected alternatives: - A runner on the Engine with its scripted model. It could fix the seed, but it would not go through the Harness or a Host's broker. Revisit if a test needs a fixed seed. - A log of events only. It is about a third shorter, and it hides timers and model requests. Revisit if the logs prove too noisy to review. @@ -228,6 +231,11 @@ The 30 tests are the deliverable, so the plan for testing is about trusting the - A change to any event's fields changes every stored log. Bless mode rewrites them, and the commit's diff shows what changed. - Two sleeps with the same deadline could complete in either order. Prompts give concurrent model replies different delays. - Justin's command-line Host and Kirill's replay need the same answer to what a stored log holds. This plan's format is a proposal to them, not a decision for them. + - What the build found: every stored log matched its row of the table, with the one change to test 25 above. No row of the list of behaviors had to change. + - Verification on 2026-11-18: the workspace has 4,083 passing tests, 34 of them new (31 in `tasks`, 2 in `mock_model`, 1 in `golden`), and the lint, format, docs, facade surface, structural, and wording commands pass. `cargo dylint` did not run, because it is not installed on this machine. + - `vibelight` outlines a crate's `src`, so its run over the workspace says nothing about a test tree. Run on the three new files, it gave three `test-knob` leads, for the words `mock` and `test` in names inside test code. Nothing was changed for them. + - The 31 stored logs hold 931 lines in total. + - A search of the existing tests found tests that assert each of the 30 behaviors, in full for 21 and in part for 9. All of them step the Engine, and none runs through the Harness. The PR description lists the closest existing tests for each. ### Deferred and Out of Scope From f795c91a0bd9a218f5954d22f755d65eacbe6f09 Mon Sep 17 00:00:00 2001 From: Andrew Rogers <87336074+rogers-andrew@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:43:25 -0800 Subject: [PATCH 9/9] Mark the plan's last step done The pull request is cppalliance/runvil-harness#25. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 --- plan/2026-11-18-1-tasks-golden-tests.plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan/2026-11-18-1-tasks-golden-tests.plan.md b/plan/2026-11-18-1-tasks-golden-tests.plan.md index 8a8756d8a..ddf574ebb 100644 --- a/plan/2026-11-18-1-tasks-golden-tests.plan.md +++ b/plan/2026-11-18-1-tasks-golden-tests.plan.md @@ -22,7 +22,7 @@ todos: status: completed - id: pr content: "Push the branch to the fork and open one PR against cppalliance/runvil-harness" - status: in_progress + status: completed --- # Golden tests for tasks