Skip to content

test(workflow): cross-session e2e — two CLI sessions, two clones, one remote - #84

Merged
kanfil merged 8 commits into
mainfrom
factory/mission/adlc-cli-phase2
Nov 8, 2026
Merged

kanfil merged 8 commits into
mainfrom
factory/mission/adlc-cli-phase2

Conversation

@kanfil

@kanfil kanfil commented Nov 7, 2026

Copy link
Copy Markdown
Member

Summary

Phase 2 local verification of the run-transport (Phase 0, #82) and trace-bundle (Phase 1, #83) work: a repeatable e2e drill that drives two separate adlc-cli sessions in two separate clones of one bare remote, CLI-only (no in-process engine API).

  • Session A (pod-a): workflow run → paused at gate (exit 3, lane contract) → workflow park to refs/factory-runs/<id>
  • Session B (pod-b): guarded workflow restore → workflow resume → completed (exit 0)

Verified invariants

# Invariant Source
V1 Pause exits 3 and the resume hint carries the run id lane contract
V2 Ref tree excludes lease.json/.push-ts, carries trace/ ADR-477, ADR-490
V3 Restore refuses mismatched --issue-ref and existing run dir ADR-391-am3
V4 Every restored event is a valid ADR-362 envelope with ADR-364 canonical type ADR-362/364
V5 Seeded secret never crosses the session boundary (0 hits in restored dir) ADR-478
V6 Default retention prunes successful events, keeps manifest; no lease conflict after cross-session resume ADR-490

Verification

  • node tests/e2e-cross-session.mjs — 25/25 assertions PASS standalone.
  • Wired into npm test — full run: 928/928 unit + both e2e drills green, exit 0.

Stacked on #83 (which stacks on #82) — only the top commit is new.

Notes

No production code changed; this PR adds regression protection for the cross-session path the factory depends on.

…ct at park

- syncRunRef force-pushes refs/factory-runs/<id> after each transition,
  throttled by ADLC_WORKFLOW_PUSH_INTERVAL (default 60s, 0 = every one),
  fail-soft so an offline pod never fails the run (ADR-475)
- stateStart/statePause/stateFail and Engine.finishSession push
  unconditionally (exit paths); step starts push throttled
- parkRun redacts every UTF-8 run file in place before hashing, so a
  leaked secret never reaches the remote (ADR-478)
- run refs exclude machine-local lease.json and .push-ts (ADR-477)
- extract the text redactor to workflow/redact.mjs; shell.mjs re-exports
- tests: workflow-run-sync.test.mjs covers AC-1..AC-5

@kanfil kanfil left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

factory-review — PR #84 (phase-2 focus range, single-review mode)

Reviewed revision: head 762e39742ee1b39cd7a3d6f604d5e3378d076b1d · base main (1ac800a4) · merge-base 51e8c1db · focus diff 65b6e2f..762e397 (1 commit, test-only: tests/e2e-cross-session.mjs + package.json test-script wiring). Lower-stack commits are reviewed under #82/#83 and not re-reviewed here.

Design docs: no PRD/AD exists in the repo itself; the spec-compliance pass falls back to the governing ADRs (ADR-475, ADR-477, ADR-478, ADR-490, ADR-362, ADR-364) from the workspace design-record memory, read-only.

Test evidence (run at exact head 762e397)

  • node tests/e2e-cross-session.mjs — 25/25 assertions PASS, exit 0
  • npm test — 928/928 unit tests pass, 0 fail (36 suites) + both e2e drills green, exit 0

Pass summaries (REVIEW.md 4-pass policy)

  • bugs (Important): clean. All 25 assertions were traced to observable behavior that changes on regression — none are vacuous. The V5 secret scan is backed by real redaction: redact.mjs masks values of secret-shaped env names (CROSS_SESSION_TOKEN, ≥8 chars) at capture (steps/shell.mjs:51), in the trace (trace.mjs redactDeep), and at park (refs.mjs redactRunDir); restoreRun copies ref blobs verbatim, so the restored-dir scan transitively covers the pushed ref. The ANSI strip helper correctly includes the ESC byte (\u001b).
  • security (Important): clean. Synthetic secret only; the drill never logs env or raw CLI output; the temp scratch dir is removed in finally.
  • spec-compliance (Important): clean. V2 matches ADR-477's EXCLUDED_RUN_FILES (lease.json/.push-ts) and ADR-490's trace/ layout (manifest.json/events.jsonl/artifacts.json); V4's envelope check matches ADR-362's fields (schema_version: 1, agent_id, ts, type, payload), ADR-364's exact 6-type canonical set, and ADR-490's payload.adlc namespace; V5 matches ADR-478's capture-time redaction contract; V6 matches ADR-490's fail-closed default (successful run → events pruned, manifest survives, retention_class: manifest-only).
  • quality (Nit): 4 findings below.

Findings (Nit — non-blocking)

  1. Nit tests/e2e-cross-session.mjs:196 — assertion label "manifest has no lease conflict fields" overstates what !existsSync(lease.json) proves (lease cleanup on completion, not an actual cross-session lease-conflict scenario). Consider renaming the label to match the assertion.
  2. Nit tests/e2e-cross-session.mjs:44,50 — git()/cli() helpers use spawnSync without timeout; a hung CLI or git call hangs npm test instead of failing it (CI job timeout would eventually catch it, a local run would just hang).
  3. Nit tests/e2e-cross-session.mjs:185-197 (V6 coverage scope) — after pod B completes, nothing asserts the completed run is re-parked/pushed back to the remote (the B→remote leg of the cross-session loop), and ADR-477's local-fidelity side (pod A retains lease.json locally) is unasserted. Coverage-scope only; the assertions that do exist are correct.
  4. Nit tests/e2e-cross-session.mjs:177 — the V5 scan's catch {} // binary file silently skips unreadable files; readFileSync(..., "utf-8") rarely throws, so the branch is effectively dead and could mask an unexpected scan failure if it ever fired.

Merge disposition

merge-ready — zero Important findings; the 4 Nits are optional follow-ups, none block merge. (Single-review mode: verdict posted as a Review comment only; approval/merge is left to the human gate.)

@kanfil kanfil added the validation Passed automated factory-review validation; awaiting human approval label Nov 7, 2026
factory-review-fix added 6 commits November 7, 2026 21:29
…symlinks

Fix three Important review findings on PR #82 (iter 1):

- I-1: syncRunRef now runs redactRunDir before every push, not just at
  park — the remote sees the run dir on every state transition, so a
  secret-valued input must be masked before the first push lands on
  origin (ADR-478, ADR-497). Redaction failure is fail-soft (warn).
- I-2: an unreadable .push-ts throttle stamp (EACCES, TOCTOU removal)
  is treated as absent — syncRunRef proceeds to push instead of
  throwing past state.save() and marking a healthy run FAILED
  (ADR-497 fail-soft).
- I-3: both run-dir walkers (hashDir, redactRunDir) use lstat and skip
  symlinks — redaction can no longer write through a symlink outside
  the run dir, symlinked directories are not traversed into the pushed
  tree, and dangling symlinks no longer fail the push.

Also adds a 30s timeout to the git push spawnSync (review nit N-1).

New regression tests I-1..I-3 in tests/workflow-run-sync.test.mjs pin
each contract; full suite 922/922.
Windows CI failure (iter 1): chmod 000 does not make a file unreadable
on Windows — POSIX mode bits don't map to Windows ACLs — so the stamp
read succeeded, the throttle saw a recent push, and the advance was
skipped, failing the I-2 assertion on all three windows jobs.

Replace the chmod simulation with a directory at the .push-ts path:
readFileSync on a directory throws EISDIR on Linux, macOS, and Windows
alike, exercising the exact guarded read path in syncRunRef on every
platform. Production code unchanged. Full suite 922/922.
…run dir raw

Fix Important finding I-4 (converge iter 3): the iter-1 in-place
redaction in syncRunRef silently masked token-shaped secret inputs in
the LIVE local run dir on the first transition — a resume() that does
not re-pass inputs then fed '***' into remaining steps, and a failed
run could not be resumed with its original inputs (RunState.load reads
inputs.json as the only input source). This regressed pre-fix behavior
(local dir stayed raw; only parkRun redacted) and contradicted ADR-497
('no run file edited in place except redaction at park') and ADR-478
('local debugging may keep raw output; the wire must not').

syncRunRef now builds the pushed tree from a throwaway staging copy:
copy the run dir to a temp dir (machine-local files excluded per
ADR-477, symlinks skipped per ADR-478), redact the copy, hash+push the
copy, discard it. The live run dir stays byte-identical across
transitions. Fail-closed: if redaction of the staged copy fails, the
push is skipped entirely — the wire never receives unredacted content —
while the transition itself stays fail-soft (no throw).

New regression tests I-4a/I-4b pin: local inputs.json stays raw across
start/pause syncs while the remote stays masked (incl. resume-level
state.inputs check), and a redaction failure (read-only inputs.json)
skips the push with no throw and an unchanged remote. Full suite
924/924.
…ifacts

Implements ADR-490 bundle v1 with engine lifecycle events as the producer
(ADR-498): ADR-488 session journals remain a later, additive producer.

- trace/manifest.json: schema v1 identity — run/workflow/issue refs, repo
  SHAs and remote, agent/session IDs, model+route, container digest,
  skill-lock and MCP-config hashes, outcome, retention class, workload
  class; always retained
- trace/events.jsonl: RunState.appendLog mirrors every lifecycle entry as
  an ADR-362 envelope (schema_version 1, agent_id, ts, type, payload) with
  ADR-364 canonical types; original entry under payload.adlc; state
  helpers and engine run/resume/finishSession append transition events
- capture-time redaction: every string entering the trace passes the
  shared redactor (redactDeep keeps JSON valid)
- fail-closed retention: failed/aborted keep full events; completed runs
  prune to manifest-only unless ADLC_TRACE_SUCCESS_SAMPLE_RATE selects
  them (deterministic sha256(run_id) sampling); finalize runs before the
  ref sync so pruned traces never reach the remote
- trace/artifacts.json: content-addressed entries (sha256, media type,
  size, storage URI, retention, sensitivity, producing event) with strict
  validation
- all trace writes fail-soft: telemetry never fails a run
- tests: workflow-trace.test.mjs covers AC-T1..AC-T6 (928/928 + E2E green)
…step head_sha

Review pr-83 iter-1 Important findings (ADR-490 spec compliance):

I-1 — mid-run syncs pushed trace/events.jsonl while retention was still
pending, so a run that later completed had already uploaded the very
blobs finalize prunes; the "pruned trace never reaches the remote"
guarantee was tip-only. syncRunRef now trims trace/events.jsonl from the
staged copy while state.json reports status "running"; paused runs
(cross-session restore/resume continuity, park/AC-T6) and terminal
statuses ship the tree per retention class. The live run dir keeps
recording events.

I-2 — the manifest recorded head_sha once at init and never refreshed
it, so a completed run's final revision was missing from the bundle
(ADR-490 replay identity). finalizeTrace now refreshes
repository.head_sha; run_start_sha keeps the run-start revision.

Tests: I-1 asserts the mid-run remote tree lacks trace/events.jsonl while
running and ships it paused/terminal per retention; I-2 asserts finalize
records the post-step HEAD after steps moved the repo.
… remote

Phase 2 local verification of Phases 0+1, as a repeatable drill instead
of a one-off manual run:

- Session A (clone A): workflow run → paused at gate (exit 3, lane
  contract) → park to refs/factory-runs/<id>
- Session B (clone B): guarded restore → resume → completed (exit 0)

Verified invariants:
- V1 pause exit code + resume hint carry the run id
- V2 ref tree excludes machine-local lease.json/.push-ts (ADR-477) and
  carries trace/{manifest,events,artifacts} (ADR-490)
- V3 restore refuses mismatched --issue-ref and an existing run dir
- V4 every restored event line is a valid ADR-362 envelope with an
  ADR-364 canonical type
- V5 a seeded secret never crosses the session boundary (0 hits in the
  restored run dir; step output carries the mask)
- V6 default retention prunes successful events and keeps the manifest
  (fail-closed), no lease conflict after cross-session resume

Drill joins npm test (25 assertions, 928/928 + 2 E2E green).
@kanfil
kanfil force-pushed the factory/mission/adlc-cli-phase2 branch from 762e397 to 5ee84dd Compare November 7, 2026 20:48
@kanfil

kanfil commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Rebase-only fix iteration (factory-review self-heal, run=pr-84).

  • Rebased factory/mission/adlc-cli-phase2 onto healed phase-1 head 4ed847cf (mid-run trace/events.jsonl exclusion + manifest post-step SHA). Rebase was clean — git skipped 65b6e2f as already applied upstream and replayed the single phase-2 test commit. New head: 5ee84dd.
  • No code changes: diff vs phase-1 head is strictly test-only (tests/e2e-cross-session.mjs + package.json test-script wiring).
  • Re-validated green locally:
    • tests/e2e-cross-session.mjs: exit 0, all assertions green (V1 pause/resume contract, V2 remote ref tree, V3 guarded restore, V4/V5 restored bundle validation, V6 resume → completed).
    • Full npm test: exit 0 — 935/935 unit tests pass, node-gate e2e (with --skip-agent) and cross-session e2e both green.
  • V2 interaction note: parked runs still carry trace/events.jsonl as asserted — the phase-1 mid-run exclusion only applies to running-state pushes, and the drill parks before asserting the ref tree, so the new exclusion and the parked-includes assertion do not conflict.
  • Pushed via --force-with-lease pinned to old head 762e397 (remote verified unmoved before push).

The drill's work step is `echo leaked=$CROSS_SESSION_TOKEN`. On POSIX
the shell expands the variable, the secret enters the step output, and
capture-redaction must mask it with `***`. On Windows the CLI's shell
step runs through cmd.exe (Node's exec default, no shell override in
src/workflow/steps/shell.mjs), which does not expand POSIX variables —
the output literally reads `leaked=$CROSS_SESSION_TOKEN`, so there is
nothing to redact and the `***` assertion can never hold.

On win32 assert the unexpanded variable instead; the redaction path
stays covered by the POSIX lanes and the unit tests. The raw-secret
absence scan already runs on all platforms. Nothing else is skipped:
the remaining 24 assertions exercise real transport machinery on
Windows.

Pre-existing on Windows (old head 762e397 CI run 42400572978 failed
the same assertion); not introduced by the rebase.
@kanfil

kanfil commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Fix iteration 2 (factory-review self-heal, run=pr-84) — Windows portability of the mask assertion.

Disclosure — pre-existing failure, not introduced by the rebase: the Windows CI failure at head 5ee84dd (and at the original pre-rebase head 762e397, CI run 42400572978) is the same single assertion: work-step output carries the mask. It was never caught locally because local verification ran on Linux only.

Root cause (verified in source): the drill's work step is echo leaked=$CROSS_SESSION_TOKEN. The CLI's shell step runs Node's child_process.exec with no shell override (src/workflow/steps/shell.mjs:47), which defaults to cmd.exe on Windows — cmd does not expand POSIX variables, so the step output literally reads leaked=$CROSS_SESSION_TOKEN. The secret value never enters the stream, so capture-redaction has nothing to mask and the *** assertion cannot hold on Windows. All other invariants (pause/park/restore/resume/retention/transport) pass on Windows.

Fix (test-only): made the assertion platform-aware in tests/e2e-cross-session.mjs:

  • POSIX (linux/darwin): unchanged — asserts the step output carries the redaction mask ***.
  • win32: asserts the output carries the literal unexpanded $CROSS_SESSION_TOKEN (proving no expansion happened). The raw-secret absence scan over the entire restored run dir already runs on all platforms and is unchanged. A comment in the test explains why the assertion differs on Windows; the redaction path itself stays covered by the POSIX lanes and the unit tests.
  • Nothing else is skipped — the remaining assertions still exercise real transport machinery on Windows.

Local verification (Linux — POSIX branch):

  • node tests/e2e-cross-session.mjs: exit 0, all assertions green.
  • Full npm test: exit 0 — 935/935 unit tests pass, node-gate and cross-session e2e both green.

New head: b857f8d (new conventional commit on top of 5ee84dd, no amend/rebase). Pushed with --force-with-lease pinned to 5ee84dd.

@kanfil

kanfil commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

factory-review converge — PR #84, iteration 2

Verified revision: b857f8d02f6b3bbb9b7146407a89a70b4ce9e786 (branch factory/mission/adlc-cli-phase2, base main, merge-base 51e8c1db).

Rebase integrity

Diff 4ed847cf..b857f8d0 is strictly test-only: tests/e2e-cross-session.mjs (new, 222 lines) + package.json test-script wiring (one line). Diff vs the originally-reviewed head 762e397 is exactly one hunk — the platform-aware mask assertion; nothing else drifted through the rebase.

Drill integrity

All 25 assertions intact and pinning V1–V6: pause exit 3 + resume hint with run id; ref tree excludes lease.json/.push-ts (ADR-477) and carries trace/{manifest,events,artifacts} (ADR-490); guarded restore (mismatched issue-ref refused, existing run dir refused); ADR-362 envelope validity + ADR-364 canonical type; seeded secret absent from the restored run dir (all-platform scan); fail-closed retention (events pruned, manifest survives, manifest-only, no lease fields). The win32 mask branch is meaningful, not vacuous: it asserts the literal unexpanded $CROSS_SESSION_TOKEN appears in state.json — proving the work-step output was captured (so the secret-absence scan is non-vacuous) and that no expansion occurred — while the raw-secret scan keeps enforcing ADR-478's actual requirement (secret absence) on every platform.

Independent verification (fresh detached worktree at b857f8d)

  • node tests/e2e-cross-session.mjs — 25/25 PASS, exit 0
  • npm test — 935/935 unit tests (36 suites), 0 fail + node-gate and cross-session e2e drills green, exit 0

Interaction with the healed stack — confirmed

The phase-1 mid-run exclusion and the drill's park-includes-events assertion compose because they live on disjoint code paths: MIDRUN_EXCLUDED_FILES trimming applies only inside syncRunRef's staged-copy branch, gated on state.json status "running" (src/workflow/refs.mjs:224). parkRun requires status "paused" (refs.mjs:275) and pushes the live run dir through pushRunRef, whose hashDir excludes only EXCLUDED_RUN_FILES (lease.json/.push-ts). So a paused run ships trace/events.jsonl (what the drill asserts), while pod B's resume-time step-boundary syncs (status running) keep the mid-run stream off the wire, and the terminal push ships per retention class. The phase-0 staging-copy redaction path is still meaningfully exercised: the pause transition fires syncRunRef (src/workflow/state.mjs:167), which stages a copy and redacts it (redactRunDir(staged.dir)); the drill's CROSS_SESSION_TOKEN (secret-shaped name, 26-char value) is masked there, and the restored-dir scan + POSIX *** assertion prove the wire never saw the raw secret.

CI evidence

All 10 checks pass at b857f8d0 (Actions run 42413521241), including the three windows-latest lanes (Node 18/20/22) that failed at 5ee84dd before the iter-2 fix.

Finding-set comparison

Iter-1 (at 762e397): 0 Important, 4 nits. Since then: rebase (no semantic change — verified by direct diff) + iter-2 test-only Windows portability fix. REVIEW.md 4-pass re-review over 4ed847cf..b857f8d0 found no new Important issues; none of the 4 original nits worsened (unchanged in substance; line numbers shifted +13 below line 181).

Verdict: DONE — merge-ready at b857f8d0. Human code-owner approval still required.

@kanfil kanfil left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 PASS — Ready for human review

Reviewed revision: b857f8d02f6b3bbb9b7146407a89a70b4ce9e786 · branch factory/mission/adlc-cli-phase2 · base main · merge-base 51e8c1db · focus diff 4ed847cf..b857f8d0 (test-only: tests/e2e-cross-session.mjs + package.json test-script wiring). Lower-stack commits reviewed under #82/#83 — not re-reviewed here.

Independent drill run (fresh detached checkout at b857f8d)

  • node tests/e2e-cross-session.mjs — 25/25 assertions PASS, exit 0
  • npm test — 935/935 unit tests (36 suites), 0 fail + both e2e drills green, exit 0

CI evidence

All 10 checks pass at this head (Actions run 42413521241), including windows-latest Node 18/20/22 — the lanes that failed before the iter-2 fix.

Rebase + fix verification

  • Diff 4ed847cf..b857f8d0 strictly test-only; diff vs originally-reviewed 762e397 is exactly the platform-aware mask hunk.
  • All 25 assertions still pin the V1–V6 invariants (ADR-475/477/478/490/362/364).
  • win32 branch is non-vacuous: asserts the literal unexpanded $CROSS_SESSION_TOKEN in state.json (work-step output captured, no expansion by cmd.exe — root cause verified in src/workflow/steps/shell.mjs: execAsync has no shell override) plus the all-platform raw-secret absence scan.
  • Park vs mid-run-exclusion interaction confirmed at source level: syncRunRef trims trace/events.jsonl only when status is "running"; parkRun (paused-only) pushes the live run dir, which ships the event stream the drill's restore/resume needs.

Remaining nits (advisory, optional follow-ups — none block merge)

  1. cli()/git() helpers (tests/e2e-cross-session.mjs:44,50) have no timeout — a hung CLI call in npm test hangs the suite rather than failing it (a CI job timeout eventually catches it).
  2. V6 coverage scope (tests/e2e-cross-session.mjs:~198-213): the B→remote re-park leg (completed run pushed back; pod A's local lease.json retention, ADR-477 local-fidelity side) is unasserted.
  3. catch {} // binary file in the secret scan (~line 190) is an effectively dead branch — readFileSync(..., "utf-8") rarely throws.
  4. The POSIX mask assertion (includes("***")) is satisfied by any *** in state.json; binding it to leaked=*** would tie the mask provably to the work-step output.

Zero Important findings. This is a convergence verdict from the factory-review self-heal loop — the human code-owner's approval is still required before merge; nothing here approves or merges the PR.

@kanfil
kanfil marked this pull request as ready for review November 8, 2026 21:57
@kanfil
kanfil merged commit 887da16 into main Nov 8, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

validation Passed automated factory-review validation; awaiting human approval

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant