Skip to content

Add 30 golden tests of tasks - #25

Closed
rogers-andrew wants to merge 9 commits into
cppalliance:masterfrom
rogers-andrew:tasks-golden-tests
Closed

rogers-andrew wants to merge 9 commits into
cppalliance:masterfrom
rogers-andrew:tasks-golden-tests

Conversation

@rogers-andrew

@rogers-andrew rogers-andrew commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Thirty tests of Lua's tasks table, run through the Harness. Each test is two files in crates/runvil/tests/golden/tasks/: a small prompt that does one thing with tasks, and the log that running it must produce. No production code changes.

Plan: plan/2026-11-18-1-tasks-golden-tests.plan.md. It lists the 29 behaviors under test and, for each of the 30 tests, what the prompt does and what its log must show.

How a test works

  • The runner (crates/runvil/tests/suite/golden.rs) runs <stem>.md on a Host through Harness::run, renders every record the recorder received plus the run's result as text, and compares the text with <stem>.log.
  • A failure names the first line that differs, with the expected and the actual line. RUNVIL_BLESS=1 stores the run's log instead of comparing.
  • The model is a mock (mock_model.rs): an InferenceBroker that reads its instruction from the newest user message, so the prompt itself says what the model answers. The forms are reply <text>, after <seconds> reply <text>, and call <tool> <json>, with steps joined by then.
  • The clock is tokio's, paused, so a timeout and a delayed model reply answer in a fixed order with no real waiting. The whole suite runs in under a second.
  • A guard test fails when a prompt has no stored log or no test that names it.

To read one, open 12-timed-join.md beside 12-timed-join.log: the prompt time-limits a join on a task that waits five seconds on the model, and the log shows the timer answer, the log() line timed out: true, entries: 0, the model's answer, and the result.

For review

  • A stored log holds the whole record: every effect, answer, and event, one per line. This is the lesson plan's open question that the command-line Host and replay share, so the format here is a proposal.
  • The seed is not fixed. The Harness draws a new seed for every run, so a golden prompt uses no randomness, and test 29 runs one prompt twice and requires the same log.
  • Test 25 fails its owner through a refused call, not Lua's error(). The error() message carries a stack trace with line numbers of __impl_coro.lua, which would change the stored log on unrelated edits.
  • One dev-dependency feature: tokio's test-util, for the paused clock.
  • All 30 tests are in this one PR by the author's choice. The commits go area by area.

Overlap with the existing tests

A search found existing tests that assert each of the 30 behaviors: in full for 21, in part for 9. The closest ones, listed below, step the Engine. Other existing tests already run tasks through the Harness (harness-stop-timing.rs, harness-inbox.rs, harness-files-shared.rs, effect_loop-cap.rs in crates/harness-internal/runner/tests/it/), though they assert stops, the inbox, files, and caps, not these behaviors. The existing tests are untouched.

The nine that existing tests cover only in part, and what the new test adds:

# New test What existing tests leave out
2 02-survives-fall-through A join after a fall-through from one section to the next (asserted only from the preamble)
5 05-limit-of-one Start-after-end under tasks.concurrency(1) (asserted under a run ceiling of 1)
6 06-default-limit-of-eight The ninth task's start (the default of 8 and the peak of live arms are asserted)
17 17-note A later note replacing an earlier one
18 18-cancel The abandoned ending of a task owned by a canceled task
21 21-sibling-not-owned A sibling's wait, ready, status, and cancel from Lua (only its send is asserted)
24 24-return-with-live-task A leaked task that owns a task, both abandoned
28 28-ids-skip-other-children A later task's id after an execute and after a timed wait
30 30a-fanout, 30b-spawns-by-hand A fanout's task events compared with the same spawns made by hand
Closest existing test for each of the 30
# Existing coverage Closest existing test
1 Yes a_finished_child_moves_its_entry_to_done_and_reports_task_succeeded
2 Partly a_task_spawned_in_the_preamble_is_joined_in_the_next_section
3 Yes spawn_returns_to_its_caller_before_the_child_runs
4 Yes task_started_includes_the_spawn_seeds_and_the_child_sees_them
5 Partly a_queued_task_reads_blocked_queued_and_its_start_event_fires_at_admission
6 Partly fanout_over_a_large_collection_admits_arms_under_the_ceiling
7 Yes join_reports_a_failed_member_without_raising
8 Yes join_any_returns_the_first_finished_member_and_the_rest_keep_running
9 Yes join_reports_a_failed_member_without_raising
10 Yes waiting_on_a_delivered_task_raises_task_consumed
11 Yes join_any_joins_the_task_it_returns_and_only_that_task
12 Yes join_returns_timed_out_with_the_unfinished_members_absent
13 Yes pending_lists_the_callers_live_tasks_in_spawn_order
14 Yes join_any_returns_the_first_finished_member_and_the_rest_keep_running
15 Yes ready_is_true_while_a_reply_waits_undelivered_and_false_once_a_wait_takes_it
16 Yes status_reports_a_parked_task_and_then_a_finished_one
17 Partly status_reports_a_parked_task_and_then_a_finished_one
18 Partly cancel_ends_a_parked_task_idempotently_and_reports_task_canceled_once
19 Yes after_a_send_a_second_wait_returns_the_tasks_next_reply
20 Yes messages_delivered_before_the_call_return_at_once_oldest_first
21 Partly a_non_owner_sending_by_id_string_raises_task_not_owned
22 Yes a_non_owner_is_refused_with_task_not_owned
23 Yes visible_also_admits_a_caller_running_inside_the_task
24 Partly a_chain_ending_with_live_author_tasks_fails_as_tasks_live_naming_the_ids
25 Yes a_chain_failing_with_a_live_task_keeps_its_own_error_and_abandons_the_task
26 Yes an_owner_that_ends_with_an_idle_task_keeps_its_own_result_and_abandons_the_task
27 Yes a_nested_fanout_nests_its_arm_ids_under_the_outer_arm
28 Partly two_runs_of_the_same_prompt_produce_identical_ids
29 Yes two_runs_under_the_same_params_and_answers_are_identical
30 Partly fanout_yields_a_spawn_per_member_starting_with_the_first

Verification

  • cargo nextest run --locked --workspace --all-features: 4,083 passed, 34 of them new (31 in tasks, 2 in mock_model, 1 in golden).
  • cargo clippy --workspace --all-targets --all-features with CARGO_BUILD_WARNINGS=deny, cargo fmt --all --check, both cargo doc commands with RUSTDOCFLAGS="-D warnings", cargo +nightly-2026-09-05 xtask api --check, cargo test -p build-xtask, and node --test tools/docs-claims.mjs: all pass.
  • cargo dylint did not run, because it is not installed on the machine that built this.
  • Editing one line of a stored log by hand failed that test with the line number, the expected line, and the actual line. The edit was reverted.
  • vibelight outlines a crate's src, so it reports nothing on a test tree. Run on the three new Rust files, it gave three test-knob leads for the words mock and test in names inside test code.

The full checks ran at 4afed45fe. The branch was then rebased onto ee199e3ae, which changes two decision pages, and the runvil crate's 44 tests and the wording check were run again.

🤖 Generated with Claude Code

rogers-andrew and others added 8 commits November 18, 2026 13:42
Thirty tests of Lua's `tasks` table, each a small prompt beside the log that running it through the Harness must produce. The plan lists the 29 behaviors under test, the 30 tests, the runner, and the format of a stored log.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A golden test is a prompt file beside the log its run must produce. The runner runs the prompt through a Harness on a Host with a mock model and a paused clock, renders every record the recorder received and the run's result as text, and compares the text with the stored log. `RUNVIL_BLESS=1` stores the run's log instead.

- The mock model answers each round from an instruction in the request's newest user message: `reply`, `after <seconds> reply`, or `call <tool> <json>`, with steps joined by ` then `.
- The Host holds the built-in Packs and `acme/echo`, whose one tool says back its `text`.
- tokio's `test-util` feature joins the crate's dev-dependencies for the paused clock.
- Test 1 spawns one task and joins it.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 2 to 6: a task survives its owner's move to the next section, stays queued while its owner's Lua runs, sees its inputs as they were at the spawn, waits for a slot under a limit of one, and waits for a slot under the default limit of eight. The mock model lists one model, since a model switch shows in the label alone.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 7 to 12: `join` returns two tool loops' results in the order listed, `join_any` returns the first task to end, a wait returns a task's text and a failed task's error as data, a second wait on a delivered task is refused as `task_consumed`, an owner reads a task's store write after joining it, and a timed join ends while a later join collects the task.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 13 to 17: `tasks.pending` lists the tasks still live, `tasks.ready` reads false before a task ends and true after, it reads true while a reply waits and false once a wait takes it, `tasks.status` reads `queued`, `chat`, then `done` around a model call, and it shows the latest note a task set.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 18 to 23: a cancel stops a queued task, a task waiting on the model, and a task that owns a task; an owner steers a worker between its replies; `tasks.receive` returns messages oldest first and waits for the next; a task that acts on its sibling, on itself, or on a made-up id is refused as `task_not_owned`; and a task reads its own status.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden tests 24 to 30: a chain that returns with a live task fails with `tasks_live` and the task and its own task are abandoned, a chain that fails keeps its own error, a task waiting after a reply does not count as live, a task's task is numbered under it, a `call`, an `execute`, and a timed wait each take a child number, two runs of one prompt give the same log, and a fanout reports the task events of the same spawns made by hand.

A guard test requires every prompt in the folder to have a stored log and a test that names it. The prompt that test 28 executes is in `helpers/`.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every stored log matched its row of the plan's table. Two things differ from the first draft: the mock model lists one model, and test 25's owner fails through a refused call, because Lua's `error()` puts line numbers of an internal Lua file into the run's error. The plan also records the verification results and the overlap with the existing tests.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 21:43

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a09f64f6-f038-49c1-a496-be4dbc236533
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

The pull request is cppalliance#25.

Plan: /plan/2026-11-18-1-tasks-golden-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the added Harness-based task golden tests.

No open issues 🎉

What changed in this PR

Adds readable golden tests for Lua’s tasks API through the public Harness, complementing existing Engine-level tests without changing production code.

Changes:

  • Adds a golden runner with record comparison, bless mode, a scripted mock model, and a paused clock.
  • Adds 30 task scenarios covering scheduling, joins, messaging, ownership, cancellation, identifiers, repeatability, and fanout.
  • Adds fixture-registration checks and documents expected behaviors and verification.
File Description
plan/​2026-11-18-1-tasks-golden-tests.plan.md Documents scope, expected behaviors, and verification.
crates/​runvil/​tests/​suite/​tasks.rs Registers scenarios, fixture guards, and cross-run comparisons.
crates/​runvil/​tests/​suite/​mock_model.rs Implements prompt-driven model replies and tool calls.
crates/​runvil/​tests/​suite/​main.rs Registers the new test modules.
crates/​runvil/​tests/​suite/​golden.rs Adds Harness execution, log rendering, comparison, and blessing.
crates/​runvil/​tests/​golden/​tasks/​helpers/​greets.md Supplies the prompt used by the execute scenario.
crates/​runvil/​tests/​golden/​tasks/​30b-spawns-by-hand.md Spawns tasks manually for comparison with fanout.
crates/​runvil/​tests/​golden/​tasks/​30b-spawns-by-hand.log Stores expected manual-spawn events and results.
crates/​runvil/​tests/​golden/​tasks/​30a-fanout.md Runs the equivalent tasks through fanout.
crates/​runvil/​tests/​golden/​tasks/​30a-fanout.log Stores expected fanout events and results.
crates/​runvil/​tests/​golden/​tasks/​29-same-log-twice.md Exercises repeatability with tasks and a timed join.
crates/​runvil/​tests/​golden/​tasks/​29-same-log-twice.log Stores the baseline for repeated-run comparison.
crates/​runvil/​tests/​golden/​tasks/​28-ids-skip-other-children.md Checks child numbering across calls, execution, and waits.
crates/​runvil/​tests/​golden/​tasks/​28-ids-skip-other-children.log Records expected task identifier gaps.
crates/​runvil/​tests/​golden/​tasks/​27-nested-ids.md Exercises nested task identifiers.
crates/​runvil/​tests/​golden/​tasks/​27-nested-ids.log Records nested identifiers and lifecycle events.
crates/​runvil/​tests/​golden/​tasks/​26-return-with-waiting-task.md Returns while a task waits after replying.
crates/​runvil/​tests/​golden/​tasks/​26-return-with-waiting-task.log Records successful return and task abandonment.
crates/​runvil/​tests/​golden/​tasks/​25-error-with-live-task.md Fails an owner while its task remains live.
crates/​runvil/​tests/​golden/​tasks/​25-error-with-live-task.log Records preserved owner error and task abandonment.
crates/​runvil/​tests/​golden/​tasks/​24-return-with-live-task.md Returns with live nested tasks.
crates/​runvil/​tests/​golden/​tasks/​24-return-with-live-task.log Records tasks_live and recursive abandonment.
crates/​runvil/​tests/​golden/​tasks/​23-own-status.md Reads a task’s own status.
crates/​runvil/​tests/​golden/​tasks/​23-own-status.log Records successful self-status access.
crates/​runvil/​tests/​golden/​tasks/​22-self-and-unknown-not-owned.md Attempts cancellation of self and an unknown identifier.
crates/​runvil/​tests/​golden/​tasks/​22-self-and-unknown-not-owned.log Records both ownership errors.
crates/​runvil/​tests/​golden/​tasks/​21-sibling-not-owned.md Attempts task operations on a sibling.
crates/​runvil/​tests/​golden/​tasks/​21-sibling-not-owned.log Records rejected sibling operations.
crates/​runvil/​tests/​golden/​tasks/​20-receive-oldest-first.md Exercises inbox ordering and waiting for another message.
crates/​runvil/​tests/​golden/​tasks/​20-receive-oldest-first.log Records ordered delivery and inbox waiting.
crates/​runvil/​tests/​golden/​tasks/​19-steer-with-replies.md Steers a worker between successive replies.
crates/​runvil/​tests/​golden/​tasks/​19-steer-with-replies.log Records replies, messages, and worker completion.
crates/​runvil/​tests/​golden/​tasks/​18-cancel.md Cancels queued, model-waiting, and parent tasks.
crates/​runvil/​tests/​golden/​tasks/​18-cancel.log Records cancellation results and descendant abandonment.
crates/​runvil/​tests/​golden/​tasks/​17-note.md Sets successive task progress notes.
crates/​runvil/​tests/​golden/​tasks/​17-note.log Records observation of the latest note.
crates/​runvil/​tests/​golden/​tasks/​16-status-through-a-model-call.md Reads task status around a model call.
crates/​runvil/​tests/​golden/​tasks/​16-status-through-a-model-call.log Records queued, chat-blocked, and completed status.
crates/​runvil/​tests/​golden/​tasks/​15-ready-with-reply.md Checks readiness before and after consuming a reply.
crates/​runvil/​tests/​golden/​tasks/​15-ready-with-reply.log Records readiness changing after reply delivery.
crates/​runvil/​tests/​golden/​tasks/​14-ready-after-end.md Checks readiness before and after task completion.
crates/​runvil/​tests/​golden/​tasks/​14-ready-after-end.log Records readiness changing after completion.
crates/​runvil/​tests/​golden/​tasks/​13-pending.md Lists live tasks after joining one task.
crates/​runvil/​tests/​golden/​tasks/​13-pending.log Records pending identifiers and final empty state.
crates/​runvil/​tests/​golden/​tasks/​12-timed-join.md Times out a join, then collects the task later.
crates/​runvil/​tests/​golden/​tasks/​12-timed-join.log Records timeout followed by successful collection.
crates/​runvil/​tests/​golden/​tasks/​11-store-after-join.md Reads a task’s store write after joining it.
crates/​runvil/​tests/​golden/​tasks/​11-store-after-join.log Records file operations and the observed contents.
crates/​runvil/​tests/​golden/​tasks/​10-wait-twice.md Attempts to consume a task result twice.
crates/​runvil/​tests/​golden/​tasks/​10-wait-twice.log Records the second wait’s task_consumed error.
crates/​runvil/​tests/​golden/​tasks/​09-result-and-error.md Joins successful and failing tasks.
crates/​runvil/​tests/​golden/​tasks/​09-result-and-error.log Records results and errors returned as data.
crates/​runvil/​tests/​golden/​tasks/​08-join-any-first-to-end.md Waits for the first task to finish.
crates/​runvil/​tests/​golden/​tasks/​08-join-any-first-to-end.log Records early delivery and later remaining completion.
crates/​runvil/​tests/​golden/​tasks/​07-join-two-tool-loops.md Joins concurrent tool loops in reversed input order.
crates/​runvil/​tests/​golden/​tasks/​07-join-two-tool-loops.log Records tool calls and ordered join results.
crates/​runvil/​tests/​golden/​tasks/​06-default-limit-of-eight.md Spawns nine tasks under the default concurrency limit.
crates/​runvil/​tests/​golden/​tasks/​06-default-limit-of-eight.log Records eight admissions before the ninth starts.
crates/​runvil/​tests/​golden/​tasks/​05-limit-of-one.md Runs two tasks with concurrency limited to one.
crates/​runvil/​tests/​golden/​tasks/​05-limit-of-one.log Records sequential task admission and completion.
crates/​runvil/​tests/​golden/​tasks/​04-inputs-copied-at-spawn.md Observes spawn inputs after owner state changes.
crates/​runvil/​tests/​golden/​tasks/​04-inputs-copied-at-spawn.log Records copied inputs, variables, and model labels.
crates/​runvil/​tests/​golden/​tasks/​03-queued-until-owner-waits.md Polls task status before the owner waits.
crates/​runvil/​tests/​golden/​tasks/​03-queued-until-owner-waits.log Records queued observations before task admission.
crates/​runvil/​tests/​golden/​tasks/​02-survives-fall-through.md Joins a task after moving to the next section.
crates/​runvil/​tests/​golden/​tasks/​02-survives-fall-through.log Records task survival across the section transition.
crates/​runvil/​tests/​golden/​tasks/​01-spawn-and-join.md Exercises basic task spawning and joining.
crates/​runvil/​tests/​golden/​tasks/​01-spawn-and-join.log Records the basic task lifecycle and result.
crates/​runvil/​Cargo.toml Enables Tokio’s test-only paused-clock support.

Review effort: Balanced

@rogers-andrew
rogers-andrew marked this pull request as draft November 18, 2026 21:47
@rogers-andrew

Copy link
Copy Markdown
Contributor Author

Closing. A search of the existing tests found that they already assert these behaviors (21 of the 30 in full, 9 in part), so this PR adds duplicate coverage. The branch is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants