Repository navigation
test(workflow): cross-session e2e — two CLI sessions, two clones, one remote - #84
Conversation
…ct at park - syncRunRef force-pushes refs/factory-runs/<id> after each transition, throttled by ADLC_WORKFLOW_PUSH_INTERVAL (default 60s, 0 = every one), fail-soft so an offline pod never fails the run (ADR-475) - stateStart/statePause/stateFail and Engine.finishSession push unconditionally (exit paths); step starts push throttled - parkRun redacts every UTF-8 run file in place before hashing, so a leaked secret never reaches the remote (ADR-478) - run refs exclude machine-local lease.json and .push-ts (ADR-477) - extract the text redactor to workflow/redact.mjs; shell.mjs re-exports - tests: workflow-run-sync.test.mjs covers AC-1..AC-5
kanfil
left a comment
There was a problem hiding this comment.
factory-review — PR #84 (phase-2 focus range, single-review mode)
Reviewed revision: head 762e39742ee1b39cd7a3d6f604d5e3378d076b1d · base main (1ac800a4) · merge-base 51e8c1db · focus diff 65b6e2f..762e397 (1 commit, test-only: tests/e2e-cross-session.mjs + package.json test-script wiring). Lower-stack commits are reviewed under #82/#83 and not re-reviewed here.
Design docs: no PRD/AD exists in the repo itself; the spec-compliance pass falls back to the governing ADRs (ADR-475, ADR-477, ADR-478, ADR-490, ADR-362, ADR-364) from the workspace design-record memory, read-only.
Test evidence (run at exact head 762e397)
node tests/e2e-cross-session.mjs— 25/25 assertions PASS, exit 0npm test— 928/928 unit tests pass, 0 fail (36 suites) + both e2e drills green, exit 0
Pass summaries (REVIEW.md 4-pass policy)
- bugs (Important): clean. All 25 assertions were traced to observable behavior that changes on regression — none are vacuous. The V5 secret scan is backed by real redaction:
redact.mjsmasks values of secret-shaped env names (CROSS_SESSION_TOKEN, ≥8 chars) at capture (steps/shell.mjs:51), in the trace (trace.mjsredactDeep), and at park (refs.mjsredactRunDir);restoreRuncopies ref blobs verbatim, so the restored-dir scan transitively covers the pushed ref. The ANSIstriphelper correctly includes the ESC byte (\u001b). - security (Important): clean. Synthetic secret only; the drill never logs env or raw CLI output; the temp scratch dir is removed in
finally. - spec-compliance (Important): clean. V2 matches ADR-477's
EXCLUDED_RUN_FILES(lease.json/.push-ts) and ADR-490's trace/ layout (manifest.json/events.jsonl/artifacts.json); V4's envelope check matches ADR-362's fields (schema_version: 1,agent_id,ts,type,payload), ADR-364's exact 6-type canonical set, and ADR-490'spayload.adlcnamespace; V5 matches ADR-478's capture-time redaction contract; V6 matches ADR-490's fail-closed default (successful run → events pruned, manifest survives,retention_class: manifest-only). - quality (Nit): 4 findings below.
Findings (Nit — non-blocking)
- Nit
tests/e2e-cross-session.mjs:196— assertion label "manifest has no lease conflict fields" overstates what!existsSync(lease.json)proves (lease cleanup on completion, not an actual cross-session lease-conflict scenario). Consider renaming the label to match the assertion. - Nit
tests/e2e-cross-session.mjs:44,50—git()/cli()helpers usespawnSyncwithouttimeout; a hung CLI or git call hangsnpm testinstead of failing it (CI job timeout would eventually catch it, a local run would just hang). - Nit
tests/e2e-cross-session.mjs:185-197(V6 coverage scope) — after pod B completes, nothing asserts the completed run is re-parked/pushed back to the remote (the B→remote leg of the cross-session loop), and ADR-477's local-fidelity side (pod A retainslease.jsonlocally) is unasserted. Coverage-scope only; the assertions that do exist are correct. - Nit
tests/e2e-cross-session.mjs:177— the V5 scan'scatch {} // binary filesilently skips unreadable files;readFileSync(..., "utf-8")rarely throws, so the branch is effectively dead and could mask an unexpected scan failure if it ever fired.
Merge disposition
merge-ready — zero Important findings; the 4 Nits are optional follow-ups, none block merge. (Single-review mode: verdict posted as a Review comment only; approval/merge is left to the human gate.)
…symlinks Fix three Important review findings on PR #82 (iter 1): - I-1: syncRunRef now runs redactRunDir before every push, not just at park — the remote sees the run dir on every state transition, so a secret-valued input must be masked before the first push lands on origin (ADR-478, ADR-497). Redaction failure is fail-soft (warn). - I-2: an unreadable .push-ts throttle stamp (EACCES, TOCTOU removal) is treated as absent — syncRunRef proceeds to push instead of throwing past state.save() and marking a healthy run FAILED (ADR-497 fail-soft). - I-3: both run-dir walkers (hashDir, redactRunDir) use lstat and skip symlinks — redaction can no longer write through a symlink outside the run dir, symlinked directories are not traversed into the pushed tree, and dangling symlinks no longer fail the push. Also adds a 30s timeout to the git push spawnSync (review nit N-1). New regression tests I-1..I-3 in tests/workflow-run-sync.test.mjs pin each contract; full suite 922/922.
Windows CI failure (iter 1): chmod 000 does not make a file unreadable on Windows — POSIX mode bits don't map to Windows ACLs — so the stamp read succeeded, the throttle saw a recent push, and the advance was skipped, failing the I-2 assertion on all three windows jobs. Replace the chmod simulation with a directory at the .push-ts path: readFileSync on a directory throws EISDIR on Linux, macOS, and Windows alike, exercising the exact guarded read path in syncRunRef on every platform. Production code unchanged. Full suite 922/922.
…run dir raw
Fix Important finding I-4 (converge iter 3): the iter-1 in-place
redaction in syncRunRef silently masked token-shaped secret inputs in
the LIVE local run dir on the first transition — a resume() that does
not re-pass inputs then fed '***' into remaining steps, and a failed
run could not be resumed with its original inputs (RunState.load reads
inputs.json as the only input source). This regressed pre-fix behavior
(local dir stayed raw; only parkRun redacted) and contradicted ADR-497
('no run file edited in place except redaction at park') and ADR-478
('local debugging may keep raw output; the wire must not').
syncRunRef now builds the pushed tree from a throwaway staging copy:
copy the run dir to a temp dir (machine-local files excluded per
ADR-477, symlinks skipped per ADR-478), redact the copy, hash+push the
copy, discard it. The live run dir stays byte-identical across
transitions. Fail-closed: if redaction of the staged copy fails, the
push is skipped entirely — the wire never receives unredacted content —
while the transition itself stays fail-soft (no throw).
New regression tests I-4a/I-4b pin: local inputs.json stays raw across
start/pause syncs while the remote stays masked (incl. resume-level
state.inputs check), and a redaction failure (read-only inputs.json)
skips the push with no throw and an unchanged remote. Full suite
924/924.
…ifacts Implements ADR-490 bundle v1 with engine lifecycle events as the producer (ADR-498): ADR-488 session journals remain a later, additive producer. - trace/manifest.json: schema v1 identity — run/workflow/issue refs, repo SHAs and remote, agent/session IDs, model+route, container digest, skill-lock and MCP-config hashes, outcome, retention class, workload class; always retained - trace/events.jsonl: RunState.appendLog mirrors every lifecycle entry as an ADR-362 envelope (schema_version 1, agent_id, ts, type, payload) with ADR-364 canonical types; original entry under payload.adlc; state helpers and engine run/resume/finishSession append transition events - capture-time redaction: every string entering the trace passes the shared redactor (redactDeep keeps JSON valid) - fail-closed retention: failed/aborted keep full events; completed runs prune to manifest-only unless ADLC_TRACE_SUCCESS_SAMPLE_RATE selects them (deterministic sha256(run_id) sampling); finalize runs before the ref sync so pruned traces never reach the remote - trace/artifacts.json: content-addressed entries (sha256, media type, size, storage URI, retention, sensitivity, producing event) with strict validation - all trace writes fail-soft: telemetry never fails a run - tests: workflow-trace.test.mjs covers AC-T1..AC-T6 (928/928 + E2E green)
…step head_sha Review pr-83 iter-1 Important findings (ADR-490 spec compliance): I-1 — mid-run syncs pushed trace/events.jsonl while retention was still pending, so a run that later completed had already uploaded the very blobs finalize prunes; the "pruned trace never reaches the remote" guarantee was tip-only. syncRunRef now trims trace/events.jsonl from the staged copy while state.json reports status "running"; paused runs (cross-session restore/resume continuity, park/AC-T6) and terminal statuses ship the tree per retention class. The live run dir keeps recording events. I-2 — the manifest recorded head_sha once at init and never refreshed it, so a completed run's final revision was missing from the bundle (ADR-490 replay identity). finalizeTrace now refreshes repository.head_sha; run_start_sha keeps the run-start revision. Tests: I-1 asserts the mid-run remote tree lacks trace/events.jsonl while running and ships it paused/terminal per retention; I-2 asserts finalize records the post-step HEAD after steps moved the repo.
… remote
Phase 2 local verification of Phases 0+1, as a repeatable drill instead
of a one-off manual run:
- Session A (clone A): workflow run → paused at gate (exit 3, lane
contract) → park to refs/factory-runs/<id>
- Session B (clone B): guarded restore → resume → completed (exit 0)
Verified invariants:
- V1 pause exit code + resume hint carry the run id
- V2 ref tree excludes machine-local lease.json/.push-ts (ADR-477) and
carries trace/{manifest,events,artifacts} (ADR-490)
- V3 restore refuses mismatched --issue-ref and an existing run dir
- V4 every restored event line is a valid ADR-362 envelope with an
ADR-364 canonical type
- V5 a seeded secret never crosses the session boundary (0 hits in the
restored run dir; step output carries the mask)
- V6 default retention prunes successful events and keeps the manifest
(fail-closed), no lease conflict after cross-session resume
Drill joins npm test (25 assertions, 928/928 + 2 E2E green).
762e397 to
5ee84dd
Compare
|
Rebase-only fix iteration (factory-review self-heal, run=pr-84).
|
The drill's work step is `echo leaked=$CROSS_SESSION_TOKEN`. On POSIX the shell expands the variable, the secret enters the step output, and capture-redaction must mask it with `***`. On Windows the CLI's shell step runs through cmd.exe (Node's exec default, no shell override in src/workflow/steps/shell.mjs), which does not expand POSIX variables — the output literally reads `leaked=$CROSS_SESSION_TOKEN`, so there is nothing to redact and the `***` assertion can never hold. On win32 assert the unexpanded variable instead; the redaction path stays covered by the POSIX lanes and the unit tests. The raw-secret absence scan already runs on all platforms. Nothing else is skipped: the remaining 24 assertions exercise real transport machinery on Windows. Pre-existing on Windows (old head 762e397 CI run 42400572978 failed the same assertion); not introduced by the rebase.
|
Fix iteration 2 (factory-review self-heal, run=pr-84) — Windows portability of the mask assertion. Disclosure — pre-existing failure, not introduced by the rebase: the Windows CI failure at head Root cause (verified in source): the drill's work step is Fix (test-only): made the assertion platform-aware in
Local verification (Linux — POSIX branch):
New head: |
factory-review converge — PR #84, iteration 2Verified revision: Rebase integrityDiff Drill integrityAll 25 assertions intact and pinning V1–V6: pause exit 3 + resume hint with run id; ref tree excludes lease.json/.push-ts (ADR-477) and carries trace/{manifest,events,artifacts} (ADR-490); guarded restore (mismatched issue-ref refused, existing run dir refused); ADR-362 envelope validity + ADR-364 canonical type; seeded secret absent from the restored run dir (all-platform scan); fail-closed retention (events pruned, manifest survives, manifest-only, no lease fields). The win32 mask branch is meaningful, not vacuous: it asserts the literal unexpanded Independent verification (fresh detached worktree at b857f8d)
Interaction with the healed stack — confirmedThe phase-1 mid-run exclusion and the drill's park-includes-events assertion compose because they live on disjoint code paths: CI evidenceAll 10 checks pass at Finding-set comparisonIter-1 (at Verdict: DONE — merge-ready at |
kanfil
left a comment
There was a problem hiding this comment.
🟢 PASS — Ready for human review
Reviewed revision: b857f8d02f6b3bbb9b7146407a89a70b4ce9e786 · branch factory/mission/adlc-cli-phase2 · base main · merge-base 51e8c1db · focus diff 4ed847cf..b857f8d0 (test-only: tests/e2e-cross-session.mjs + package.json test-script wiring). Lower-stack commits reviewed under #82/#83 — not re-reviewed here.
Independent drill run (fresh detached checkout at b857f8d)
node tests/e2e-cross-session.mjs— 25/25 assertions PASS, exit 0npm test— 935/935 unit tests (36 suites), 0 fail + both e2e drills green, exit 0
CI evidence
All 10 checks pass at this head (Actions run 42413521241), including windows-latest Node 18/20/22 — the lanes that failed before the iter-2 fix.
Rebase + fix verification
- Diff
4ed847cf..b857f8d0strictly test-only; diff vs originally-reviewed762e397is exactly the platform-aware mask hunk. - All 25 assertions still pin the V1–V6 invariants (ADR-475/477/478/490/362/364).
- win32 branch is non-vacuous: asserts the literal unexpanded
$CROSS_SESSION_TOKENin state.json (work-step output captured, no expansion by cmd.exe — root cause verified insrc/workflow/steps/shell.mjs:execAsynchas no shell override) plus the all-platform raw-secret absence scan. - Park vs mid-run-exclusion interaction confirmed at source level:
syncRunReftrimstrace/events.jsonlonly when status is"running";parkRun(paused-only) pushes the live run dir, which ships the event stream the drill's restore/resume needs.
Remaining nits (advisory, optional follow-ups — none block merge)
cli()/git()helpers (tests/e2e-cross-session.mjs:44,50) have no timeout — a hung CLI call innpm testhangs the suite rather than failing it (a CI job timeout eventually catches it).- V6 coverage scope (tests/e2e-cross-session.mjs:~198-213): the B→remote re-park leg (completed run pushed back; pod A's local lease.json retention, ADR-477 local-fidelity side) is unasserted.
catch {} // binary filein the secret scan (~line 190) is an effectively dead branch —readFileSync(..., "utf-8")rarely throws.- The POSIX mask assertion (
includes("***")) is satisfied by any***in state.json; binding it toleaked=***would tie the mask provably to the work-step output.
Zero Important findings. This is a convergence verdict from the factory-review self-heal loop — the human code-owner's approval is still required before merge; nothing here approves or merges the PR.
Summary
Phase 2 local verification of the run-transport (Phase 0, #82) and trace-bundle (Phase 1, #83) work: a repeatable e2e drill that drives two separate adlc-cli sessions in two separate clones of one bare remote, CLI-only (no in-process engine API).
workflow run→ paused at gate (exit 3, lane contract) →workflow parktorefs/factory-runs/<id>workflow restore→workflow resume→ completed (exit 0)Verified invariants
lease.json/.push-ts, carriestrace/--issue-refand existing run dirVerification
node tests/e2e-cross-session.mjs— 25/25 assertions PASS standalone.npm test— full run: 928/928 unit + both e2e drills green, exit 0.Stacked on #83 (which stacks on #82) — only the top commit is new.
Notes
No production code changed; this PR adds regression protection for the cross-session path the factory depends on.