Repository navigation
Add 30 golden tests of tasks - #25
Closed
rogers-andrew wants to merge 9 commits into
Closed
rogers-andrew wants to merge 9 commits into
rogers-andrew wants to merge 9 commits into
Conversation
Thirty tests of Lua's `tasks` table, each a small prompt beside the log that running it through the Harness must produce. The plan lists the 29 behaviors under test, the 30 tests, the runner, and the format of a stored log. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A golden test is a prompt file beside the log its run must produce. The runner runs the prompt through a Harness on a Host with a mock model and a paused clock, renders every record the recorder received and the run's result as text, and compares the text with the stored log. `RUNVIL_BLESS=1` stores the run's log instead. - The mock model answers each round from an instruction in the request's newest user message: `reply`, `after <seconds> reply`, or `call <tool> <json>`, with steps joined by ` then `. - The Host holds the built-in Packs and `acme/echo`, whose one tool says back its `text`. - tokio's `test-util` feature joins the crate's dev-dependencies for the paused clock. - Test 1 spawns one task and joins it. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 2 to 6: a task survives its owner's move to the next section, stays queued while its owner's Lua runs, sees its inputs as they were at the spawn, waits for a slot under a limit of one, and waits for a slot under the default limit of eight. The mock model lists one model, since a model switch shows in the label alone. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 7 to 12: `join` returns two tool loops' results in the order listed, `join_any` returns the first task to end, a wait returns a task's text and a failed task's error as data, a second wait on a delivered task is refused as `task_consumed`, an owner reads a task's store write after joining it, and a timed join ends while a later join collects the task. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 13 to 17: `tasks.pending` lists the tasks still live, `tasks.ready` reads false before a task ends and true after, it reads true while a reply waits and false once a wait takes it, `tasks.status` reads `queued`, `chat`, then `done` around a model call, and it shows the latest note a task set. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 18 to 23: a cancel stops a queued task, a task waiting on the model, and a task that owns a task; an owner steers a worker between its replies; `tasks.receive` returns messages oldest first and waits for the next; a task that acts on its sibling, on itself, or on a made-up id is refused as `task_not_owned`; and a task reads its own status. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 24 to 30: a chain that returns with a live task fails with `tasks_live` and the task and its own task are abandoned, a chain that fails keeps its own error, a task waiting after a reply does not count as live, a task's task is numbered under it, a `call`, an `execute`, and a timed wait each take a child number, two runs of one prompt give the same log, and a fanout reports the task events of the same spawns made by hand. A guard test requires every prompt in the folder to have a stored log and a test that names it. The prompt that test 28 executes is in `helpers/`. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every stored log matched its row of the plan's table. Two things differ from the first draft: the mock model lists one model, and test 25's owner fails through a refused call, because Lua's `error()` puts line numbers of an internal Lua file into the run's error. The plan also records the verification results and the overlap with the existing tests. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
The pull request is cppalliance#25. Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the added Harness-based task golden tests.
No open issues 🎉
What changed in this PR
Adds readable golden tests for Lua’s tasks API through the public Harness, complementing existing Engine-level tests without changing production code.
Changes:
- Adds a golden runner with record comparison, bless mode, a scripted mock model, and a paused clock.
- Adds 30 task scenarios covering scheduling, joins, messaging, ownership, cancellation, identifiers, repeatability, and fanout.
- Adds fixture-registration checks and documents expected behaviors and verification.
| File | Description |
|---|---|
plan/2026-11-18-1-tasks-golden-tests.plan.md |
Documents scope, expected behaviors, and verification. |
crates/runvil/tests/suite/tasks.rs |
Registers scenarios, fixture guards, and cross-run comparisons. |
crates/runvil/tests/suite/mock_model.rs |
Implements prompt-driven model replies and tool calls. |
crates/runvil/tests/suite/main.rs |
Registers the new test modules. |
crates/runvil/tests/suite/golden.rs |
Adds Harness execution, log rendering, comparison, and blessing. |
crates/runvil/tests/golden/tasks/helpers/greets.md |
Supplies the prompt used by the execute scenario. |
crates/runvil/tests/golden/tasks/30b-spawns-by-hand.md |
Spawns tasks manually for comparison with fanout. |
crates/runvil/tests/golden/tasks/30b-spawns-by-hand.log |
Stores expected manual-spawn events and results. |
crates/runvil/tests/golden/tasks/30a-fanout.md |
Runs the equivalent tasks through fanout. |
crates/runvil/tests/golden/tasks/30a-fanout.log |
Stores expected fanout events and results. |
crates/runvil/tests/golden/tasks/29-same-log-twice.md |
Exercises repeatability with tasks and a timed join. |
crates/runvil/tests/golden/tasks/29-same-log-twice.log |
Stores the baseline for repeated-run comparison. |
crates/runvil/tests/golden/tasks/28-ids-skip-other-children.md |
Checks child numbering across calls, execution, and waits. |
crates/runvil/tests/golden/tasks/28-ids-skip-other-children.log |
Records expected task identifier gaps. |
crates/runvil/tests/golden/tasks/27-nested-ids.md |
Exercises nested task identifiers. |
crates/runvil/tests/golden/tasks/27-nested-ids.log |
Records nested identifiers and lifecycle events. |
crates/runvil/tests/golden/tasks/26-return-with-waiting-task.md |
Returns while a task waits after replying. |
crates/runvil/tests/golden/tasks/26-return-with-waiting-task.log |
Records successful return and task abandonment. |
crates/runvil/tests/golden/tasks/25-error-with-live-task.md |
Fails an owner while its task remains live. |
crates/runvil/tests/golden/tasks/25-error-with-live-task.log |
Records preserved owner error and task abandonment. |
crates/runvil/tests/golden/tasks/24-return-with-live-task.md |
Returns with live nested tasks. |
crates/runvil/tests/golden/tasks/24-return-with-live-task.log |
Records tasks_live and recursive abandonment. |
crates/runvil/tests/golden/tasks/23-own-status.md |
Reads a task’s own status. |
crates/runvil/tests/golden/tasks/23-own-status.log |
Records successful self-status access. |
crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.md |
Attempts cancellation of self and an unknown identifier. |
crates/runvil/tests/golden/tasks/22-self-and-unknown-not-owned.log |
Records both ownership errors. |
crates/runvil/tests/golden/tasks/21-sibling-not-owned.md |
Attempts task operations on a sibling. |
crates/runvil/tests/golden/tasks/21-sibling-not-owned.log |
Records rejected sibling operations. |
crates/runvil/tests/golden/tasks/20-receive-oldest-first.md |
Exercises inbox ordering and waiting for another message. |
crates/runvil/tests/golden/tasks/20-receive-oldest-first.log |
Records ordered delivery and inbox waiting. |
crates/runvil/tests/golden/tasks/19-steer-with-replies.md |
Steers a worker between successive replies. |
crates/runvil/tests/golden/tasks/19-steer-with-replies.log |
Records replies, messages, and worker completion. |
crates/runvil/tests/golden/tasks/18-cancel.md |
Cancels queued, model-waiting, and parent tasks. |
crates/runvil/tests/golden/tasks/18-cancel.log |
Records cancellation results and descendant abandonment. |
crates/runvil/tests/golden/tasks/17-note.md |
Sets successive task progress notes. |
crates/runvil/tests/golden/tasks/17-note.log |
Records observation of the latest note. |
crates/runvil/tests/golden/tasks/16-status-through-a-model-call.md |
Reads task status around a model call. |
crates/runvil/tests/golden/tasks/16-status-through-a-model-call.log |
Records queued, chat-blocked, and completed status. |
crates/runvil/tests/golden/tasks/15-ready-with-reply.md |
Checks readiness before and after consuming a reply. |
crates/runvil/tests/golden/tasks/15-ready-with-reply.log |
Records readiness changing after reply delivery. |
crates/runvil/tests/golden/tasks/14-ready-after-end.md |
Checks readiness before and after task completion. |
crates/runvil/tests/golden/tasks/14-ready-after-end.log |
Records readiness changing after completion. |
crates/runvil/tests/golden/tasks/13-pending.md |
Lists live tasks after joining one task. |
crates/runvil/tests/golden/tasks/13-pending.log |
Records pending identifiers and final empty state. |
crates/runvil/tests/golden/tasks/12-timed-join.md |
Times out a join, then collects the task later. |
crates/runvil/tests/golden/tasks/12-timed-join.log |
Records timeout followed by successful collection. |
crates/runvil/tests/golden/tasks/11-store-after-join.md |
Reads a task’s store write after joining it. |
crates/runvil/tests/golden/tasks/11-store-after-join.log |
Records file operations and the observed contents. |
crates/runvil/tests/golden/tasks/10-wait-twice.md |
Attempts to consume a task result twice. |
crates/runvil/tests/golden/tasks/10-wait-twice.log |
Records the second wait’s task_consumed error. |
crates/runvil/tests/golden/tasks/09-result-and-error.md |
Joins successful and failing tasks. |
crates/runvil/tests/golden/tasks/09-result-and-error.log |
Records results and errors returned as data. |
crates/runvil/tests/golden/tasks/08-join-any-first-to-end.md |
Waits for the first task to finish. |
crates/runvil/tests/golden/tasks/08-join-any-first-to-end.log |
Records early delivery and later remaining completion. |
crates/runvil/tests/golden/tasks/07-join-two-tool-loops.md |
Joins concurrent tool loops in reversed input order. |
crates/runvil/tests/golden/tasks/07-join-two-tool-loops.log |
Records tool calls and ordered join results. |
crates/runvil/tests/golden/tasks/06-default-limit-of-eight.md |
Spawns nine tasks under the default concurrency limit. |
crates/runvil/tests/golden/tasks/06-default-limit-of-eight.log |
Records eight admissions before the ninth starts. |
crates/runvil/tests/golden/tasks/05-limit-of-one.md |
Runs two tasks with concurrency limited to one. |
crates/runvil/tests/golden/tasks/05-limit-of-one.log |
Records sequential task admission and completion. |
crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.md |
Observes spawn inputs after owner state changes. |
crates/runvil/tests/golden/tasks/04-inputs-copied-at-spawn.log |
Records copied inputs, variables, and model labels. |
crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.md |
Polls task status before the owner waits. |
crates/runvil/tests/golden/tasks/03-queued-until-owner-waits.log |
Records queued observations before task admission. |
crates/runvil/tests/golden/tasks/02-survives-fall-through.md |
Joins a task after moving to the next section. |
crates/runvil/tests/golden/tasks/02-survives-fall-through.log |
Records task survival across the section transition. |
crates/runvil/tests/golden/tasks/01-spawn-and-join.md |
Exercises basic task spawning and joining. |
crates/runvil/tests/golden/tasks/01-spawn-and-join.log |
Records the basic task lifecycle and result. |
crates/runvil/Cargo.toml |
Enables Tokio’s test-only paused-clock support. |
Review effort: Balanced
rogers-andrew
marked this pull request as draft
November 18, 2026 21:47
Contributor
Author
|
Closing. A search of the existing tests found that they already assert these behaviors (21 of the 30 in full, 9 in part), so this PR adds duplicate coverage. The branch is kept. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thirty tests of Lua's
taskstable, run through the Harness. Each test is two files incrates/runvil/tests/golden/tasks/: a small prompt that does one thing with tasks, and the log that running it must produce. No production code changes.Plan:
plan/2026-11-18-1-tasks-golden-tests.plan.md. It lists the 29 behaviors under test and, for each of the 30 tests, what the prompt does and what its log must show.How a test works
crates/runvil/tests/suite/golden.rs) runs<stem>.mdon a Host throughHarness::run, renders every record the recorder received plus the run's result as text, and compares the text with<stem>.log.RUNVIL_BLESS=1stores the run's log instead of comparing.mock_model.rs): anInferenceBrokerthat reads its instruction from the newest user message, so the prompt itself says what the model answers. The forms arereply <text>,after <seconds> reply <text>, andcall <tool> <json>, with steps joined bythen.To read one, open
12-timed-join.mdbeside12-timed-join.log: the prompt time-limits ajoinon a task that waits five seconds on the model, and the log shows the timer answer, thelog()linetimed out: true, entries: 0, the model's answer, and the result.For review
error(). Theerror()message carries a stack trace with line numbers of__impl_coro.lua, which would change the stored log on unrelated edits.test-util, for the paused clock.Overlap with the existing tests
A search found existing tests that assert each of the 30 behaviors: in full for 21, in part for 9. The closest ones, listed below, step the Engine. Other existing tests already run tasks through the Harness (
harness-stop-timing.rs,harness-inbox.rs,harness-files-shared.rs,effect_loop-cap.rsincrates/harness-internal/runner/tests/it/), though they assert stops, the inbox, files, and caps, not these behaviors. The existing tests are untouched.The nine that existing tests cover only in part, and what the new test adds:
02-survives-fall-through05-limit-of-onetasks.concurrency(1)(asserted under a run ceiling of 1)06-default-limit-of-eight17-note18-cancel21-sibling-not-ownedready,status, and cancel from Lua (only its send is asserted)24-return-with-live-task28-ids-skip-other-childrenexecuteand after a timed wait30a-fanout,30b-spawns-by-handClosest existing test for each of the 30
a_finished_child_moves_its_entry_to_done_and_reports_task_succeededa_task_spawned_in_the_preamble_is_joined_in_the_next_sectionspawn_returns_to_its_caller_before_the_child_runstask_started_includes_the_spawn_seeds_and_the_child_sees_thema_queued_task_reads_blocked_queued_and_its_start_event_fires_at_admissionfanout_over_a_large_collection_admits_arms_under_the_ceilingjoin_reports_a_failed_member_without_raisingjoin_any_returns_the_first_finished_member_and_the_rest_keep_runningjoin_reports_a_failed_member_without_raisingwaiting_on_a_delivered_task_raises_task_consumedjoin_any_joins_the_task_it_returns_and_only_that_taskjoin_returns_timed_out_with_the_unfinished_members_absentpending_lists_the_callers_live_tasks_in_spawn_orderjoin_any_returns_the_first_finished_member_and_the_rest_keep_runningready_is_true_while_a_reply_waits_undelivered_and_false_once_a_wait_takes_itstatus_reports_a_parked_task_and_then_a_finished_onestatus_reports_a_parked_task_and_then_a_finished_onecancel_ends_a_parked_task_idempotently_and_reports_task_canceled_onceafter_a_send_a_second_wait_returns_the_tasks_next_replymessages_delivered_before_the_call_return_at_once_oldest_firsta_non_owner_sending_by_id_string_raises_task_not_owneda_non_owner_is_refused_with_task_not_ownedvisible_also_admits_a_caller_running_inside_the_taska_chain_ending_with_live_author_tasks_fails_as_tasks_live_naming_the_idsa_chain_failing_with_a_live_task_keeps_its_own_error_and_abandons_the_taskan_owner_that_ends_with_an_idle_task_keeps_its_own_result_and_abandons_the_taska_nested_fanout_nests_its_arm_ids_under_the_outer_armtwo_runs_of_the_same_prompt_produce_identical_idstwo_runs_under_the_same_params_and_answers_are_identicalfanout_yields_a_spawn_per_member_starting_with_the_firstVerification
cargo nextest run --locked --workspace --all-features: 4,083 passed, 34 of them new (31 intasks, 2 inmock_model, 1 ingolden).cargo clippy --workspace --all-targets --all-featureswithCARGO_BUILD_WARNINGS=deny,cargo fmt --all --check, bothcargo doccommands withRUSTDOCFLAGS="-D warnings",cargo +nightly-2026-09-05 xtask api --check,cargo test -p build-xtask, andnode --test tools/docs-claims.mjs: all pass.cargo dylintdid not run, because it is not installed on the machine that built this.vibelightoutlines a crate'ssrc, so it reports nothing on a test tree. Run on the three new Rust files, it gave threetest-knobleads for the wordsmockandtestin names inside test code.The full checks ran at
4afed45fe. The branch was then rebased ontoee199e3ae, which changes two decision pages, and therunvilcrate's 44 tests and the wording check were run again.🤖 Generated with Claude Code