-
Notifications
You must be signed in to change notification settings - Fork 64
ci(relayflows): gate feature and fix PRs with isolated red-green proofs #1612
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
bf923b6
ci(relayflows): add PR-specific red-green proof gate
fe507da
fix(ci): harden RelayFlow PR proof gate
803aa78
fix(ci): resolve PR proof CodeQL findings
882ad0b
fix(ci): bound observation reads by handle
67290f4
fix(pr-proof): fail closed before sandbox execution
7cdbec3
fix(pr-proof): close cancellation and output gaps
62d445b
test(pr-proof): treat killed zombies as terminated
7a66abe
fix(pr-proof): harden rerun ownership and progress parsing
08703fe
test(pr-proof): skip POSIX process assertion without ps
9b424f8
fix(pr-proof): defer progress parse failures
54caeba
fix(ci): use one Cloud API key for PR proof
7afe0fe
refactor(cloud): model API key auth explicitly
3017a30
fix(cloud): preserve explicit API key endpoint
f5283e4
fix(cloud): scope API key auth to workflows
d478000
refactor(cloud): isolate workflow API key client
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
54 changes: 54 additions & 0 deletions
54
.agentworkforce/trajectories/completed/2026-08/traj_yr3f5u3r6zdz/summary.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Trajectory: Add PR-specific Cloud RelayFlow red-green proof infrastructure | ||
|
|
||
| > **Status:** ✅ Completed | ||
| > **Confidence:** 92% | ||
| > **Started:** August 25, 2026 at 10:45 AM | ||
| > **Completed:** August 25, 2026 at 11:14 AM | ||
|
|
||
| --- | ||
|
|
||
| ## Summary | ||
|
|
||
| Implemented trusted PR classification and single-case dispatch, a fail-closed Cloud RelayFlow that proves base then head in distinct per-step sandboxes, provenance/evidence validation, GitHub Action wiring, tests, template guidance, and credential rollout documentation. | ||
|
|
||
| **Approach:** Kept pull_request_target as a data-only trusted dispatcher, ran PR code only in Cloud, represented expected-red as a structured successful observation, disabled repair retries, validated exact base/head SHAs and sandbox IDs, and tested both the contracts and the complete repository. | ||
|
|
||
| --- | ||
|
|
||
| ## Key Decisions | ||
|
|
||
| ### Require exactly one PR proof case and a structured observation result | ||
| - **Chose:** Require exactly one PR proof case and a structured observation result | ||
| - **Reasoning:** One feature or fix should trigger only its own case. The case runner must exit successfully after observing behavior and write bug/absent/fixed plus an exact signature; test crashes, missing tests, skips, and build failures remain infrastructure failures and cannot count as expected red. | ||
|
|
||
| ### Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes | ||
| - **Chose:** Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes | ||
| - **Reasoning:** The GitHub runner must never checkout or execute untrusted head code while holding Cloud credentials. It will fetch and validate one declarative case manifest, stage exact base/head SHAs, and submit the trusted base-branch RelayFlow. | ||
|
|
||
| ### Disable automatic repair and retries for PR proof workflows | ||
| - **Chose:** Disable automatic repair and retries for PR proof workflows | ||
| - **Rejected:** Default retry policy, Continue on failure | ||
| - **Reasoning:** A red/green gate must preserve the first observation and fail closed. RelayFlow defaults to agent-assisted repair retries when agents exist, so the proof workflow explicitly uses fail-fast with zero step retries. | ||
|
|
||
| ### Stage the validated generated input in the disposable CI git index | ||
| - **Chose:** Stage the validated generated input in the disposable CI git index | ||
| - **Rejected:** Change global code-sync semantics, Commit a placeholder input file | ||
| - **Reasoning:** Cloud code sync uploads git-known files. The per-PR input is generated after checkout, so the action must git-add it without committing or pushing or the Cloud workflow cannot see the exact case and SHAs. | ||
|
|
||
| ### Require explicit proof classification on every PR | ||
| - **Chose:** Require explicit proof classification on every PR | ||
| - **Rejected:** Infer only from conventional titles, Run Cloud for every PR | ||
| - **Reasoning:** Silently treating missing metadata as non-functional would let non-conventional feature and fix titles bypass the gate. Every PR must declare feature, bugfix, or non-functional; only the first two launch Cloud. | ||
|
|
||
| --- | ||
|
|
||
| ## Chapters | ||
|
|
||
| ### 1. Work | ||
| *Agent: default* | ||
|
|
||
| - Require exactly one PR proof case and a structured observation result: Require exactly one PR proof case and a structured observation result | ||
| - Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes: Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes | ||
| - Disable automatic repair and retries for PR proof workflows: Disable automatic repair and retries for PR proof workflows | ||
| - Stage the validated generated input in the disposable CI git index: Stage the validated generated input in the disposable CI git index | ||
| - Require explicit proof classification on every PR: Require explicit proof classification on every PR |
128 changes: 128 additions & 0 deletions
128
.agentworkforce/trajectories/completed/2026-08/traj_yr3f5u3r6zdz/trajectory.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| { | ||
| "id": "traj_yr3f5u3r6zdz", | ||
| "version": 1, | ||
| "task": { | ||
| "title": "Add PR-specific Cloud RelayFlow red-green proof infrastructure" | ||
| }, | ||
| "status": "completed", | ||
| "startedAt": "2026-08-25T08:45:32.835Z", | ||
| "completedAt": "2026-08-25T09:14:31.949Z", | ||
| "agents": [ | ||
| { | ||
| "name": "default", | ||
| "role": "lead", | ||
| "joinedAt": "2026-08-25T08:50:30.315Z" | ||
| } | ||
| ], | ||
| "chapters": [ | ||
| { | ||
| "id": "chap_cco5mxx3i8f5", | ||
| "title": "Work", | ||
| "agentName": "default", | ||
| "startedAt": "2026-08-25T08:50:30.315Z", | ||
| "endedAt": "2026-08-25T09:14:31.949Z", | ||
| "events": [ | ||
| { | ||
| "ts": 1787647830316, | ||
| "type": "decision", | ||
| "content": "Require exactly one PR proof case and a structured observation result: Require exactly one PR proof case and a structured observation result", | ||
| "raw": { | ||
| "question": "Require exactly one PR proof case and a structured observation result", | ||
| "chosen": "Require exactly one PR proof case and a structured observation result", | ||
| "alternatives": [], | ||
| "reasoning": "One feature or fix should trigger only its own case. The case runner must exit successfully after observing behavior and write bug/absent/fixed plus an exact signature; test crashes, missing tests, skips, and build failures remain infrastructure failures and cannot count as expected red." | ||
| }, | ||
| "significance": "high" | ||
| }, | ||
| { | ||
| "ts": 1787647830374, | ||
| "type": "decision", | ||
| "content": "Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes: Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes", | ||
| "raw": { | ||
| "question": "Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes", | ||
| "chosen": "Use pull_request_target only as a trusted dispatcher; execute PR case code only inside Cloud sandboxes", | ||
| "alternatives": [], | ||
| "reasoning": "The GitHub runner must never checkout or execute untrusted head code while holding Cloud credentials. It will fetch and validate one declarative case manifest, stage exact base/head SHAs, and submit the trusted base-branch RelayFlow." | ||
| }, | ||
| "significance": "high" | ||
| }, | ||
| { | ||
| "ts": 1787649023999, | ||
| "type": "decision", | ||
| "content": "Disable automatic repair and retries for PR proof workflows: Disable automatic repair and retries for PR proof workflows", | ||
| "raw": { | ||
| "question": "Disable automatic repair and retries for PR proof workflows", | ||
| "chosen": "Disable automatic repair and retries for PR proof workflows", | ||
| "alternatives": [ | ||
| { | ||
| "option": "Default retry policy", | ||
| "reason": "" | ||
| }, | ||
| { | ||
| "option": "Continue on failure", | ||
| "reason": "" | ||
| } | ||
| ], | ||
| "reasoning": "A red/green gate must preserve the first observation and fail closed. RelayFlow defaults to agent-assisted repair retries when agents exist, so the proof workflow explicitly uses fail-fast with zero step retries." | ||
| }, | ||
| "significance": "high" | ||
| }, | ||
| { | ||
| "ts": 1787649024705, | ||
| "type": "decision", | ||
| "content": "Stage the validated generated input in the disposable CI git index: Stage the validated generated input in the disposable CI git index", | ||
| "raw": { | ||
| "question": "Stage the validated generated input in the disposable CI git index", | ||
| "chosen": "Stage the validated generated input in the disposable CI git index", | ||
| "alternatives": [ | ||
| { | ||
| "option": "Change global code-sync semantics", | ||
| "reason": "" | ||
| }, | ||
| { | ||
| "option": "Commit a placeholder input file", | ||
| "reason": "" | ||
| } | ||
| ], | ||
| "reasoning": "Cloud code sync uploads git-known files. The per-PR input is generated after checkout, so the action must git-add it without committing or pushing or the Cloud workflow cannot see the exact case and SHAs." | ||
| }, | ||
| "significance": "high" | ||
| }, | ||
| { | ||
| "ts": 1787649025196, | ||
| "type": "decision", | ||
| "content": "Require explicit proof classification on every PR: Require explicit proof classification on every PR", | ||
| "raw": { | ||
| "question": "Require explicit proof classification on every PR", | ||
| "chosen": "Require explicit proof classification on every PR", | ||
| "alternatives": [ | ||
| { | ||
| "option": "Infer only from conventional titles", | ||
| "reason": "" | ||
| }, | ||
| { | ||
| "option": "Run Cloud for every PR", | ||
| "reason": "" | ||
| } | ||
| ], | ||
| "reasoning": "Silently treating missing metadata as non-functional would let non-conventional feature and fix titles bypass the gate. Every PR must declare feature, bugfix, or non-functional; only the first two launch Cloud." | ||
| }, | ||
| "significance": "high" | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "retrospective": { | ||
| "summary": "Implemented trusted PR classification and single-case dispatch, a fail-closed Cloud RelayFlow that proves base then head in distinct per-step sandboxes, provenance/evidence validation, GitHub Action wiring, tests, template guidance, and credential rollout documentation.", | ||
| "approach": "Kept pull_request_target as a data-only trusted dispatcher, ran PR code only in Cloud, represented expected-red as a structured successful observation, disabled repair retries, validated exact base/head SHAs and sandbox IDs, and tested both the contracts and the complete repository.", | ||
| "confidence": 0.92 | ||
| }, | ||
| "commits": [], | ||
| "filesChanged": [], | ||
| "projectId": "AgentWorkforce/relay", | ||
| "tags": [], | ||
| "_trace": { | ||
| "startRef": "191e5f14a343431ec42282c1103c9f014854ac0b", | ||
| "endRef": "191e5f14a343431ec42282c1103c9f014854ac0b" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| name: RelayFlow PR Proof | ||
|
|
||
| # Security boundary: pull_request_target gives this trusted base-branch | ||
| # dispatcher access to Cloud credentials. It MUST NOT checkout, import, or run | ||
| # PR-head code on the GitHub runner. The exact head is cloned only inside the | ||
| # isolated Cloud proof sandboxes by scripts/pr-proof/run-arm.mjs. | ||
| on: | ||
| pull_request_target: | ||
| types: [opened, synchronize, reopened, edited, ready_for_review] | ||
|
khaliqgant marked this conversation as resolved.
|
||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| description: Pull request number to prove | ||
| required: true | ||
| type: number | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
| statuses: write | ||
|
|
||
| concurrency: | ||
| group: relayflow-pr-proof-${{ github.event.pull_request.number || inputs.pr_number }} | ||
| cancel-in-progress: true | ||
|
khaliqgant marked this conversation as resolved.
khaliqgant marked this conversation as resolved.
|
||
|
|
||
| jobs: | ||
| proof: | ||
| name: RelayFlow PR proof dispatcher | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 70 | ||
| steps: | ||
| - name: Checkout trusted base | ||
| uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 | ||
| with: | ||
| ref: ${{ github.event.pull_request.base.sha || github.sha }} | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 | ||
| with: | ||
| node-version: 22.14.0 | ||
|
|
||
| # pull_request_target runs are attached to the base SHA. Publish the | ||
| # stable required context explicitly on the exact validated PR head. | ||
| - name: Start required status on PR head | ||
| id: status | ||
| env: | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| run: >- | ||
| node scripts/pr-proof/report-status.mjs start | ||
| --event "$GITHUB_EVENT_PATH" | ||
| --github-output "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Classify PR and validate declared case | ||
| id: prepare | ||
| env: | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| run: >- | ||
| node scripts/pr-proof/prepare.mjs | ||
| --event "$GITHUB_EVENT_PATH" | ||
| --output .relayflow/pr-proof-input.json | ||
| --expected-head-sha "${{ steps.status.outputs.head_sha }}" | ||
| --github-output "$GITHUB_OUTPUT" | ||
| --summary "$GITHUB_STEP_SUMMARY" | ||
|
|
||
| - name: Confirm Cloud proof credential | ||
| if: steps.prepare.outputs.required == 'true' | ||
| env: | ||
| CLOUD_API_URL: ${{ secrets.CLOUD_API_URL }} | ||
| CLOUD_API_KEY: ${{ secrets.RELAYFLOW_PR_PROOF_CLOUD_API_KEY }} | ||
| run: | | ||
| test -n "$CLOUD_API_URL" | ||
| test -n "$CLOUD_API_KEY" | ||
|
|
||
| # Cloud code sync intentionally uploads only paths known to git. Add the | ||
| # generated, validated input to this disposable runner's index so it is | ||
| # present in the Cloud workspace; this does not create a commit or push. | ||
| - name: Include proof input in Cloud code sync | ||
| if: steps.prepare.outputs.required == 'true' | ||
| run: git add -f -- .relayflow/pr-proof-input.json | ||
|
|
||
| - name: Install the released Agent Relay CLI | ||
| if: steps.prepare.outputs.required == 'true' | ||
| run: npm install --global "agent-relay@$(node -p "require('./package.json').version")" | ||
|
|
||
| - name: Run isolated Cloud red-green proof | ||
| if: steps.prepare.outputs.required == 'true' | ||
| id: cloud | ||
| env: | ||
| CLOUD_API_URL: ${{ secrets.CLOUD_API_URL }} | ||
| CLOUD_API_KEY: ${{ secrets.RELAYFLOW_PR_PROOF_CLOUD_API_KEY }} | ||
| PR_PROOF_CLOUD_TIMEOUT_MS: '3600000' | ||
| run: node scripts/pr-proof/run-cloud.mjs workflows/pr-proof.ts | ||
|
|
||
| - name: Upload Cloud proof logs | ||
| if: always() && steps.prepare.outputs.required == 'true' | ||
|
khaliqgant marked this conversation as resolved.
|
||
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 | ||
| with: | ||
| name: relayflow-pr-proof-${{ github.event.pull_request.number || inputs.pr_number }} | ||
| path: .workflow-artifacts/pr-proof/cloud.log | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Finish required status on PR head | ||
| # report-status verifies that this run still owns the latest pending | ||
| # context before publishing. That makes cancellation terminal without | ||
| # allowing a cancelled predecessor to overwrite a replacement run. | ||
| if: always() && steps.status.outputs.head_sha != '' | ||
| env: | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| run: >- | ||
| node scripts/pr-proof/report-status.mjs finish | ||
| --sha "${{ steps.status.outputs.head_sha }}" | ||
| --job-status "${{ job.status }}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.