From aaaf5fb48a185546e0cc17b94438b8322988b306 Mon Sep 17 00:00:00 2001 From: Subin George Date: Wed, 1 Jul 2026 21:40:19 +0530 Subject: [PATCH 01/12] RFC-0056: CRCR Support for Nightly & Periodic CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Proposes extending CRCR to support scheduled (nightly/periodic) CI dispatches for downstream repositories. Presents two options — EventBridge cron vs. upstream GitHub Actions workflow — with detailed tradeoffs, implementation effort, and open questions for WG discussion. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 379 +++++++++++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 RFC-0056-CRCR-Nightly-Periodic-CI.md diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md new file mode 100644 index 00000000..10787504 --- /dev/null +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -0,0 +1,379 @@ +# CRCR Support for Nightly & Periodic CI + +**Authors:** +* @groenenboomj +* @jewelkm89 +* @subinz1 + +**Status:** Draft — for CRCR Working Group discussion + +**Date:** June 2026 + +## Summary + +Extend the Cross-Repository CI Relay (CRCR) to support nightly and periodic CI schedules for downstream repositories. Currently, CRCR only dispatches on `pull_request` and `push` events from `pytorch/pytorch`. This RFC proposes adding scheduled dispatch capabilities so downstream backends can report nightly/periodic CI results to the PyTorch HUD. + +## Motivation + +CRCR currently dispatches downstream CI on `pull_request` and `push` events from `pytorch/pytorch`. The webhook Lambda receives these GitHub webhook events, generates a `delivery_id`, sends `repository_dispatch` to all allowlisted downstream repos, and sets `DISPATCHED` in Redis. Downstream repos run their CI, then report results back via the callback Lambda, which validates the state machine (`DISPATCHED → IN_PROGRESS → COMPLETED`) and forwards metrics to HUD. + +Nightly and periodic runs have no upstream trigger. They are cron-scheduled jobs (e.g., nightly builds against `main` HEAD, weekly compatibility tests against release branches). This creates two blockers: + +1. **No dispatch.** Without an upstream webhook event, there is no `repository_dispatch` to downstream repos. Downstream nightly jobs would have to self-trigger via their own `schedule: cron`. + +2. **No callback path.** The state machine rejects callbacks without a prior `DISPATCHED` record (HTTP 400: "no prior dispatch"). Even if a downstream repo runs a nightly job and tries to report results, the callback is rejected. + +**Impact:** Downstream backends cannot report nightly/periodic CI results to HUD. This is a gap for L3/L4 backends that need to show nightly compatibility on `hud.pytorch.org/crcr`. + +## Current Architecture + +### Supported Events + +The webhook Lambda accepts two GitHub event types (see `_SUPPORTED_EVENTS` in `webhook/lambda_function.py`): + +| Event | Source | When | +|-------|--------|------| +| `pull_request` | GitHub webhook | PR opened, reopened, synchronize, closed | +| `push` | GitHub webhook | Push to any branch in `pytorch/pytorch` | + +Both follow the same dispatch path: + +``` +GitHub webhook (pull_request or push) + ↓ +Webhook Lambda: + 1. Verify GitHub signature (X-Hub-Signature-256) + 2. Check event type ∈ {pull_request, push} + 3. Check repo == upstream_repo (pytorch/pytorch) + 4. Generate delivery_id from X-GitHub-Delivery header + 5. For each allowlisted downstream repo: + - Mint GitHub App installation token + - Send repository_dispatch(event_type, client_payload) + - Set DISPATCHED state in Redis + ↓ +Downstream repo receives repository_dispatch + ↓ +Downstream workflow runs CI, calls callback action: + - in_progress → Callback Lambda validates state, sets IN_PROGRESS in Redis + - completed → Callback Lambda validates state, sets COMPLETED in Redis + ↓ +Callback Lambda forwards trusted + untrusted payloads to HUD + ↓ +HUD → DynamoDB → ClickHouse → hud.pytorch.org/crcr +``` + +### Existing Precedent: EventBridge in the Callback Lambda + +The callback Lambda already handles EventBridge-triggered events for zombie cleanup. When `event.source == "crcr.sweeper"`, it runs the cleanup handler instead of processing a callback. This is the exact pattern Option 2 would follow — an EventBridge cron rule invoking a Lambda handler. + +```python +# callback/lambda_function.py (line 18-26, existing) +if event.get("source") == "crcr.sweeper": + config = get_config() + result = cleanup_handler.handle(config) + ... +``` + +### Downstream Workflow Structure + +Downstream repos listen for `repository_dispatch` with the event types they want: + +```yaml +# Existing downstream pattern +on: + repository_dispatch: + types: [pull_request, push] # L1 receiver already handles both +``` + +The `client_payload` always contains `event_type`, `delivery_id`, and the upstream webhook payload. Downstream workflows branch on `event_type` to extract PR number, SHA, ref, etc. + +## Proposed Implementation + +Two options are presented for WG discussion. + +### Option 1: Upstream Cron Workflow in pytorch/pytorch → Webhook Lambda + +Add a GitHub Actions workflow in `pytorch/pytorch` that runs on a `schedule: cron` trigger. This workflow constructs a synthetic webhook-like payload and POSTs it to the webhook Lambda's existing `/github/webhook` endpoint. The Lambda processes it like any other webhook — validates, dispatches to downstream repos, sets `DISPATCHED` in Redis. + +``` +pytorch/pytorch scheduled workflow (e.g., daily 00:00 UTC) + ↓ +1. Fetch main HEAD SHA (git ls-remote or checkout) +2. Construct synthetic payload mimicking a push/nightly event +3. POST to webhook Lambda endpoint with authentication + ↓ +Webhook Lambda: + - Verify auth (new: OIDC or shared secret — NOT GitHub webhook signature) + - Parse synthetic payload + - event_type = "nightly" + - Generate delivery_id + - Call _dispatch_to_allowlist() + ↓ +(Same as existing dispatch from here) +``` + +**Upstream workflow:** + +```yaml +# pytorch/pytorch/.github/workflows/crcr-nightly-dispatch.yml +name: CRCR Nightly Dispatch +on: + schedule: + - cron: '0 0 * * *' # midnight UTC + workflow_dispatch: {} # manual trigger + +jobs: + dispatch: + runs-on: ubuntu-latest + permissions: + id-token: write + steps: + - name: Get main HEAD SHA + id: sha + run: | + SHA=$(git ls-remote https://github.com/pytorch/pytorch HEAD | cut -f1) + echo "sha=$SHA" >> "$GITHUB_OUTPUT" + + - name: Trigger CRCR nightly dispatch + run: | + TOKEN=$(curl -s \ + -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=pytorch-cross-repo-ci-relay" \ + | jq -r .value) + + curl -X POST "${{ secrets.RELAY_WEBHOOK_URL }}" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $TOKEN" \ + -H "X-GitHub-Event: nightly" \ + -d '{ + "action": "nightly", + "repository": {"full_name": "pytorch/pytorch"}, + "ref": "refs/heads/main", + "after": "${{ steps.sha.outputs.sha }}" + }' +``` + +**Downstream workflow change** (same for both options): + +```yaml +on: + repository_dispatch: + types: [pull_request, push, nightly] +``` + +#### Option 1 — Pros + +| # | Advantage | Detail | +|---|-----------|--------| +| 1 | No new AWS infrastructure | No EventBridge rule, no Terraform. Uses existing Lambda endpoint. | +| 2 | Visible in pytorch/pytorch | The nightly trigger appears in the upstream repo's Actions tab. PyTorch maintainers can see run history, logs, and failures. | +| 3 | Manual trigger | `workflow_dispatch` means anyone with repo write access can manually re-trigger a nightly dispatch from the GitHub UI. | +| 4 | Familiar | It's a GitHub Actions workflow. No AWS console, no Terraform, no new infrastructure concepts for contributors. | +| 5 | Schedule flexibility | Multiple cron entries or separate workflows can define different schedules without touching the relay. | +| 6 | PR-able changes | Changing the schedule is a PR to `pytorch/pytorch`, reviewed by maintainers with standard CI. | +| 7 | Decoupled from relay deployment | The schedule lives in a workflow file, not in infrastructure. Relay re-deployments don't affect the cron schedule. | + +#### Option 1 — Cons + +| # | Disadvantage | Detail | +|---|-------------|--------| +| 1 | Authentication complexity | The webhook Lambda currently validates requests using GitHub webhook signatures (`X-Hub-Signature-256`). A workflow-triggered POST doesn't have this signature. Requires adding a new auth path (OIDC or shared secret). | +| 2 | Not a real GitHub webhook | The synthetic payload mimics a webhook but isn't one. The Lambda's `_verify_signature` would reject it. Needs a new code path to parse and validate this non-standard input. | +| 3 | Requires changes in pytorch/pytorch | Adding a workflow to the upstream repo requires buy-in from PyTorch maintainers. | +| 4 | GitHub cron unreliability | GitHub's `schedule:` trigger is best-effort. During high-load periods, cron runs can be delayed by minutes to hours or skipped entirely. | +| 5 | Secret management | If using a shared secret (not OIDC), the secret must be stored in `pytorch/pytorch` repo secrets and rotated in sync with the Lambda config. | +| 6 | Dependency on pytorch/pytorch | Downstream nightly scheduling depends on an upstream workflow running. If Actions runners are down or quotas exhausted, all downstream nightlies stop. | +| 7 | Event handler branching | A `nightly` event type has neither a `pull_request` nor a standard `push` payload shape, requiring new parsing logic and a synthetic payload contract. | +| 8 | `_SUPPORTED_EVENTS` gating | The Lambda's `_SUPPORTED_EVENTS = {"pull_request", "push"}` rejects unknown event types before signature verification. Adding `nightly` means changing this gate and adding the corresponding auth path. | + +#### Option 1 — Implementation Effort + +| Component | Work | Effort | +|-----------|------|--------| +| pytorch/pytorch workflow | New `.github/workflows/crcr-nightly-dispatch.yml` (~40 LOC) | ~0.5 day | +| Webhook Lambda auth | New auth path: OIDC verification (port `jwt_helper` from callback Lambda) or shared-secret validation | ~1.5 days | +| Webhook Lambda handler | Expand `_SUPPORTED_EVENTS`, add nightly event parsing path, generate `delivery_id` | ~1 day | +| Downstream workflow | Add `nightly` to `repository_dispatch.types` | ~0.5 day | +| HUD view | Filter/view for `event_type != pull_request` on `/crcr/nightly` | ~1 day | +| Testing | End-to-end: workflow → Lambda → downstream → callback → HUD | ~1 day | +| Org approval | PR to pytorch/pytorch — maintainer review cycle | ~1-5 days | +| **Total** | | **~5.5-9.5 days** | + +--- + +### Option 2: EventBridge Cron → Webhook Lambda + +Add an AWS EventBridge rule (cron schedule) that invokes the webhook Lambda directly. The Lambda detects the EventBridge source (no HTTP request, no GitHub signature), fetches the current `main` HEAD SHA, generates a synthetic `delivery_id`, and dispatches to downstream repos using the existing `_dispatch_to_allowlist` machinery. The downstream flow is identical to a PR or push dispatch. + +``` +EventBridge cron (e.g., daily 00:00 UTC) + ↓ +Webhook Lambda (new handler path: event.source == "crcr.scheduler") + ↓ +1. Fetch pytorch/pytorch main HEAD SHA via GitHub API +2. Build synthetic client_payload: + - event_type: "nightly" (or "periodic") + - delivery_id: "nightly-{date}-{uuid}" + - payload: {repository: {full_name: "pytorch/pytorch"}, head_sha: "..."} +3. Call _dispatch_to_allowlist() — reuses existing dispatch logic: + - For each allowlisted repo: repository_dispatch + DISPATCHED in Redis + ↓ +Downstream repo receives repository_dispatch with event_type: nightly + ↓ +(Same as today: build → test → callback in_progress → callback completed) + ↓ +Callback Lambda → HUD → DynamoDB → ClickHouse +``` + +#### Option 2 — Pros + +| # | Advantage | Detail | +|---|-----------|--------| +| 1 | Full pipeline reuse | State machine, callbacks, HUD ingestion, timing metrics — all work unchanged. Zero modifications to the callback Lambda, HUD API, or ClickHouse schema. | +| 2 | Minimal downstream changes | Downstream repos add one event type string. No new workflows, no new actions, no registration steps. | +| 3 | No auth changes | EventBridge invokes the Lambda directly (not over HTTP). No signature validation, no OIDC. The Lambda receives a structured AWS event, which is inherently trusted. | +| 4 | Proven pattern | The zombie cleanup handler already uses EventBridge → Lambda (`source: crcr.sweeper`). Same architecture. | +| 5 | Central control | The relay controls who gets nightly dispatches (allowlist-gated) and when they fire. | +| 6 | State machine integrity | Every nightly run has a real `DISPATCHED` record. No special-casing or bypasses. | +| 7 | Schedule reliability | EventBridge has an SLA of 99.99%. Far more reliable than GitHub's best-effort cron. | +| 8 | Consistent SHA | All downstream repos build against the same HEAD SHA fetched once at dispatch time. | +| 9 | No cross-repo dependency | Everything lives in `pytorch/test-infra`. No PR to `pytorch/pytorch`, no maintainer approval outside the CRCR team. | + +#### Option 2 — Cons + +| # | Disadvantage | Detail | +|---|-------------|--------| +| 1 | New infrastructure | Requires a new EventBridge rule and a new handler path in the webhook Lambda. Terraform + Lambda code to deploy and maintain. | +| 2 | Inflexible scheduling | All downstream repos get the same cron schedule. A backend that wants every-6-hours or weekly can't customize without additional EventBridge rules. | +| 3 | Single point of failure | If the EventBridge rule or Lambda errors, no downstream repos get their nightly dispatch. Requires CloudWatch monitoring/alerting. | +| 4 | No self-service trigger | Downstream repos can't trigger an ad-hoc nightly run. They must wait for the next scheduled dispatch or ask a relay admin. | +| 5 | SHA staleness | The HEAD SHA is fetched once. If `main` advances between dispatch and the downstream build starting, the tested SHA is already stale. | +| 6 | Allowlist complexity | Needs a way to opt repos in/out of nightly dispatches. Adds a new dimension to the allowlist config. | + +#### Option 2 — Implementation Effort + +| Component | Work | Effort | +|-----------|------|--------| +| EventBridge rule | Terraform: `aws_cloudwatch_event_rule` + `aws_cloudwatch_event_target` pointing to webhook Lambda | ~0.5 day | +| Webhook Lambda handler | New path for `source: crcr.scheduler`: fetch HEAD SHA, build synthetic payload, call `_dispatch_to_allowlist()`. ~100-150 LOC | ~1 day | +| Allowlist | Add `nightly: true/false` per-repo flag or a separate `nightly:` section | ~0.5 day | +| Downstream workflow | Add `nightly` to `repository_dispatch.types`, branch `run-name` on `event_type` | ~0.5 day | +| HUD view | Filter/view for `event_type != pull_request` on `/crcr/nightly` | ~1 day | +| Testing | End-to-end test with `TorchedHat/pytorch-redhat-ci` | ~0.5 day | +| **Total** | | **~4 days** | + +## Side-by-Side Comparison + +| Criteria | Option 2: EventBridge → Lambda | Option 1: pytorch/pytorch Cron → Lambda | +|----------|-------------------------------|----------------------------------------| +| New AWS infrastructure | Yes (EventBridge rule) | No | +| Changes to pytorch/pytorch | No | Yes (new workflow file) | +| Webhook Lambda changes | New handler path (~150 LOC) | New auth path + event parser (~250 LOC) | +| Callback Lambda changes | None | None | +| Auth changes | None (internal invoke) | Significant (OIDC or shared secret) | +| Downstream repo changes | Add `nightly` to dispatch types | Add `nightly` to dispatch types | +| State machine integrity | Full | Full (with new auth path) | +| Timing metrics | Full | Full | +| Central schedule control | Yes (Terraform) | No (workflow in upstream repo) | +| Manual re-trigger | Lambda console/CLI | Built-in (`workflow_dispatch`) | +| Schedule reliability | High (EventBridge 99.99% SLA) | Medium (GitHub cron best-effort) | +| Per-repo schedule flexibility | Low (single cron) | Medium (multiple workflows) | +| Operational visibility | CloudWatch logs/metrics | GitHub Actions run history | +| Org approval needed | No (test-infra only) | Yes (pytorch/pytorch PR) | +| Security surface change | None | New auth path in webhook Lambda | +| **Implementation effort** | **~4 days** | **~5.5-9.5 days** | + +## Metrics + +- **Nightly dispatch success rate**: Percentage of scheduled dispatches that successfully reach all allowlisted downstream repos. +- **Nightly callback completion rate**: Percentage of dispatched nightly runs that report back `COMPLETED` via callback. +- **HUD coverage**: Number of downstream backends with nightly results visible on `hud.pytorch.org/crcr`. +- **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD. + +## Drawbacks + +- Adds complexity to the dispatch pipeline (new event type, new handler path). +- Nightly failures have no upstream PR to annotate — requires a separate notification mechanism. +- Concurrent nightly and PR-triggered runs may compete for downstream CI resources. +- Increases the surface area of the relay's responsibility. + +## Alternatives + +**Do nothing.** Downstream repos run nightly CI independently and report results in their own dashboards. PyTorch maintainers have no visibility into nightly downstream health. This is the current state. + +**Downstream self-dispatch.** Each downstream repo configures its own `schedule: cron` and bypasses the state machine. This breaks the callback path (no `DISPATCHED` record) and requires per-repo configuration with no central control. + +## Prior Art + +- **CRCR Zombie Cleanup (EventBridge → Lambda)**: The existing `crcr.sweeper` pattern in the callback Lambda is the direct precedent for Option 2. +- **GitHub Actions scheduled workflows**: Widely used for nightly builds across the PyTorch ecosystem (e.g., `pytorch/pytorch` nightly builds, `pytorch/vision` nightly tests). +- **RFC-0050**: The original CRCR RFC that established the dispatch → callback → HUD pipeline for `pull_request` events. +- **RFC-0054**: HUD integration RFC that defined the ClickHouse schema and dashboard views for CRCR results. + +## Feasibility Assessment + +**Option 1 has real advantages for visibility and self-service.** `workflow_dispatch` is genuinely useful — being able to click "Run workflow" in the GitHub UI to trigger an ad-hoc nightly dispatch without AWS console access is valuable for debugging. The schedule living in a workflow file (not Terraform) means it's visible, grep-able, and PR-able by anyone with repo access. These advantages could justify the additional effort if the WG prioritizes visibility and self-service. + +**Option 2 is lower effort and lower risk.** Auth is the key differentiator — Option 2 requires zero auth changes (EventBridge invokes the Lambda directly as an internal AWS call, inherently trusted), while Option 1 requires adding a second authentication mechanism to the webhook Lambda. Option 2 follows the exact same EventBridge → Lambda pattern already used for zombie cleanup. Everything stays in `pytorch/test-infra` with no cross-repo coordination. + +## Unresolved Questions + +### For WG Discussion + +1. **Which option does the WG prefer?** Option 2 (lower effort, central control, no auth changes) or Option 1 (upstream visibility, self-service triggers, more flexible)? + +2. **Schedule ownership:** Central cron (EventBridge or upstream workflow) or per-repo opt-in schedules? + +3. **Scope:** Should nightly dispatches go to all allowlisted repos, or only repos that opt in via a config flag? + +4. **SHA policy:** Always `main` HEAD, or allow per-repo target branches? + +5. **Failure SLA:** Is a HUD view sufficient, or do we need active notifications (Slack, issues)? + +6. **Manual triggers:** How important is the ability to manually re-trigger a nightly from the GitHub UI without AWS access? + +7. **Periodic vs. nightly:** Do we need both `nightly` and `periodic` event types from day one, or start with `nightly` only and add `periodic` later? + +### Shared Considerations (Both Options) + +**HUD Changes.** The HUD currently groups results by `pr_number`. For nightly runs there is no PR. Use `pr_number = 0` as a sentinel for non-PR runs. The `event_type` field already flows through the pipeline — it just needs to carry `nightly` or `periodic` instead of `pull_request`. Add a filter/view on `hud.pytorch.org/crcr` for non-PR results, or a dedicated `/crcr/nightly` page. + +**Allowlist Scoping.** Not all L1+ repos should receive nightly dispatches. Consider: + +```yaml +L2: + - org/repo: + oncalls: user1, user2 + nightly: true # opt-in to nightly dispatches +``` + +Or gate on allowlist level (e.g., only L3+ get nightlies by default). + +**SHA Selection.** For nightly runs, which SHA to test against? +- `main` HEAD at dispatch time (most common, default) +- Latest release tag (for release compatibility testing) +- A specific branch (e.g., `release/2.x`) + +**Failure Notifications.** PR failures are visible on the PR. Nightly failures have no PR to annotate. +- Dedicated Slack channel for CRCR nightly failures +- Auto-create GitHub issues on consecutive failures +- HUD dashboard alert on `/crcr/nightly` +- Email to repo oncalls from the allowlist + +**Deduplication.** If a nightly dispatch happens while a PR-triggered run is in progress for the same downstream repo, the state machine handles them independently (different `delivery_id`). However, concurrent builds may compete for downstream CI resources. Consider whether the downstream workflow should use `concurrency:` groups to avoid parallel nightly + PR builds. + +## Resolution + +TBD — pending WG discussion. + +### Level of Support + +TBD + +### Next Steps + +TBD + +#### Tracking Issue + +TBD From e36486b8aff023307ab626e9326d4a53e72d7624 Mon Sep 17 00:00:00 2001 From: Subin George Date: Mon, 13 Jul 2026 21:22:37 +0530 Subject: [PATCH 02/12] Use commit SHA as delivery_id instead of nightly-{date}-{uuid} MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per @atalman's review feedback — the main HEAD SHA is already fetched, so use it as the dispatch/correlation ID. This makes the ID meaningful and lets HUD correlate nightly runs directly to the tested commit. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 10787504..21798b21 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -213,7 +213,8 @@ Webhook Lambda (new handler path: event.source == "crcr.scheduler") 1. Fetch pytorch/pytorch main HEAD SHA via GitHub API 2. Build synthetic client_payload: - event_type: "nightly" (or "periodic") - - delivery_id: "nightly-{date}-{uuid}" + - delivery_id: "" (the main HEAD commit SHA — meaningful, correlatable, + and lets HUD map runs directly to github.com/pytorch/pytorch/commit/) - payload: {repository: {full_name: "pytorch/pytorch"}, head_sha: "..."} 3. Call _dispatch_to_allowlist() — reuses existing dispatch logic: - For each allowlisted repo: repository_dispatch + DISPATCHED in Redis From 6fad51583a09f392b0467cd6475f28c7ebb75fea Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 14 Jul 2026 18:14:07 +0530 Subject: [PATCH 03/12] Add Option 3 (Authenticated Self-Report) to RFC Adds Option C proposed by @atalman: downstream repos self-schedule via their own crons and report results back through OIDC-validated callbacks. Includes pros/cons analysis, implementation effort estimate, updated comparison table, and open questions around state machine divergence, trust model, and missing-run detection. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 118 ++++++++++++++++++++++----- 1 file changed, 98 insertions(+), 20 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 21798b21..235fe55c 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -263,26 +263,90 @@ Callback Lambda → HUD → DynamoDB → ClickHouse | Testing | End-to-end test with `TorchedHat/pytorch-redhat-ci` | ~0.5 day | | **Total** | | **~4 days** | +### Option 3: Authenticated Self-Report (Option C) + +*Proposed by @atalman in [PR #98 comment](https://github.com/pytorch/rfcs/pull/98#issuecomment-4962790260).* + +Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay stops being a trigger and becomes a **validating ingest endpoint**. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped and replaced with authorization + SHA-validity at callback time. + +``` +Downstream repo's own cron schedule + ↓ +Fetches pytorch/pytorch HEAD SHA (or nightly/viable-strict ref) + ↓ +Runs CI against that SHA + ↓ +Calls back to the relay with: + - OIDC token (proves repo identity) + - dispatch_id = pytorch/pytorch commit SHA + - event_type = "nightly" or "periodic" + ↓ +Relay validates: + 1. OIDC token → repo is on allowlist + 2. GET /repos/pytorch/pytorch/commits/{sha} → SHA is real + ↓ +Upsert record keyed by (repo, SHA) → HUD +``` + +#### Option 3 — Pros + +| # | Advantage | Detail | +|---|-----------|--------| +| 1 | No trigger from pytorch/pytorch | Nothing in upstream emits an event. No new workflows, branches, or tags. | +| 2 | No new AWS infrastructure | No EventBridge, no Terraform. | +| 3 | Self-service schedule | Each downstream repo owns its cron. Changes are PRs to the downstream repo. | +| 4 | Manual re-trigger | `workflow_dispatch` on the downstream cron workflow re-runs the nightly. | +| 5 | Coordination-free correlation | dispatch_id is the SHA, derivable by any actor independently. | + +#### Option 3 — Cons + +| # | Disadvantage | Detail | +|---|-------------|--------| +| 1 | State machine bypass | Drops the `DISPATCHED` precondition. "Upsert" is a fundamentally different model from the existing state machine. Requires a new callback Lambda code path. | +| 2 | Trust model weakening | The relay can verify the SHA is real, but not that CI actually ran against it. A downstream could self-report results for a SHA it never tested. | +| 3 | No SHA alignment | Each downstream independently fetches HEAD. If `main` advances between repos' crons, they test different commits. Cross-backend comparison on HUD is fragmented. | +| 4 | Missing-run detection is silent | No `DISPATCHED` record means no zombie sweeper coverage. If a downstream's cron silently breaks, nobody on the relay side knows. | +| 5 | SHA overwrite on re-runs | Upsert keyed by `(repo, SHA)` overwrites previous results. No audit trail of multiple runs against the same SHA. | +| 6 | No timing metrics | Without `dispatched_at`, queue time metrics are lost. | +| 7 | SHA validation adds dependency | `GET /repos/pytorch/pytorch/commits/{sha}` requires GitHub API availability and rate limits. Caching specifics are undefined. | +| 8 | Callback payload contract undefined | Current callbacks carry `delivery_id`, PR metadata. A nightly self-report has different fields. The new payload schema is not specified. | +| 9 | Significant Lambda changes | New callback Lambda code path (~200 LOC), SHA validation + caching, upsert logic. The complexity shifts from AWS resources to Lambda code. | + +#### Option 3 — Implementation Effort + +| Component | Work | Effort | +|-----------|------|--------| +| Callback Lambda upsert path | New code path: skip state machine, validate OIDC + SHA, upsert | ~2 days | +| SHA validation + caching | GitHub API integration + cache layer | ~1 day | +| Downstream cron workflow | New workflow in each downstream: fetch SHA, run CI, call callback | ~1 day per repo | +| HUD view | Filter/view for non-PR results grouped by SHA | ~1 day | +| Testing | End-to-end test with TorchedHat/pytorch-redhat-ci | ~1 day | +| **Total** | | **~5-6 days** | + +--- + ## Side-by-Side Comparison -| Criteria | Option 2: EventBridge → Lambda | Option 1: pytorch/pytorch Cron → Lambda | -|----------|-------------------------------|----------------------------------------| -| New AWS infrastructure | Yes (EventBridge rule) | No | -| Changes to pytorch/pytorch | No | Yes (new workflow file) | -| Webhook Lambda changes | New handler path (~150 LOC) | New auth path + event parser (~250 LOC) | -| Callback Lambda changes | None | None | -| Auth changes | None (internal invoke) | Significant (OIDC or shared secret) | -| Downstream repo changes | Add `nightly` to dispatch types | Add `nightly` to dispatch types | -| State machine integrity | Full | Full (with new auth path) | -| Timing metrics | Full | Full | -| Central schedule control | Yes (Terraform) | No (workflow in upstream repo) | -| Manual re-trigger | Lambda console/CLI | Built-in (`workflow_dispatch`) | -| Schedule reliability | High (EventBridge 99.99% SLA) | Medium (GitHub cron best-effort) | -| Per-repo schedule flexibility | Low (single cron) | Medium (multiple workflows) | -| Operational visibility | CloudWatch logs/metrics | GitHub Actions run history | -| Org approval needed | No (test-infra only) | Yes (pytorch/pytorch PR) | -| Security surface change | None | New auth path in webhook Lambda | -| **Implementation effort** | **~4 days** | **~5.5-9.5 days** | +| Criteria | Option 2: EventBridge → Lambda | Option 1: pytorch/pytorch Cron → Lambda | Option 3: Authenticated Self-Report | +|----------|-------------------------------|----------------------------------------|--------------------------------------| +| New AWS infrastructure | Yes (EventBridge rule) | No | No | +| Changes to pytorch/pytorch | No | Yes (new workflow file) | No | +| Webhook Lambda changes | New handler path (~150 LOC) | New auth path + event parser (~250 LOC) | None | +| Callback Lambda changes | None | None | ~200 LOC (upsert + SHA validation) | +| Auth changes | None (internal invoke) | Significant (OIDC or shared secret) | Modified (OIDC as sole trust anchor) | +| State machine | Preserved | Preserved (with new auth path) | Bypassed (upsert replaces state machine) | +| Downstream repo changes | Add `nightly` to dispatch types | Add `nightly` to dispatch types | New cron workflow + callback | +| SHA alignment across backends | Guaranteed (single dispatch) | Guaranteed (single dispatch) | Not guaranteed | +| Missing-run detection | Zombie sweeper works | Zombie sweeper works | Not supported | +| Timing metrics | Full | Full | Partial (no dispatched_at) | +| Trust model | Relay-controlled dispatch | Relay-controlled dispatch | Self-reported (SHA verified, execution not verified) | +| Central schedule control | Yes (Terraform) | No (workflow in upstream repo) | No (each downstream independently) | +| Manual re-trigger | Lambda console/CLI | Built-in (`workflow_dispatch`) | Downstream `workflow_dispatch` | +| Schedule reliability | High (EventBridge 99.99% SLA) | Medium (GitHub cron best-effort) | Medium (GitHub cron per downstream) | +| Operational visibility | CloudWatch logs/metrics | GitHub Actions run history | Downstream repo Actions tab | +| Org approval needed | No (test-infra only) | Yes (pytorch/pytorch PR) | No | +| Security surface change | None | New auth path in webhook Lambda | New callback Lambda entry point | +| **Implementation effort** | **~4 days** | **~5.5-9.5 days** | **~5-6 days** | ## Metrics @@ -317,11 +381,13 @@ Callback Lambda → HUD → DynamoDB → ClickHouse **Option 2 is lower effort and lower risk.** Auth is the key differentiator — Option 2 requires zero auth changes (EventBridge invokes the Lambda directly as an internal AWS call, inherently trusted), while Option 1 requires adding a second authentication mechanism to the webhook Lambda. Option 2 follows the exact same EventBridge → Lambda pattern already used for zombie cleanup. Everything stays in `pytorch/test-infra` with no cross-repo coordination. +**Option 3 gives downstream repos full autonomy but trades state machine guarantees.** By shifting the trigger to downstream crons and validating via OIDC + SHA at callback time, it avoids touching upstream infra entirely. However, this is a fundamentally different trust model — the relay trusts that downstream repos actually ran CI against the claimed SHA, but has no dispatch record to verify this. Missing-run detection, SHA alignment across backends, and queue-time metrics are all sacrificed. The Lambda changes are non-trivial (new upsert path, SHA validation, caching), and the callback payload contract needs careful design to avoid breaking the existing state machine path. + ## Unresolved Questions ### For WG Discussion -1. **Which option does the WG prefer?** Option 2 (lower effort, central control, no auth changes) or Option 1 (upstream visibility, self-service triggers, more flexible)? +1. **Which option does the WG prefer?** Option 2 (lower effort, central control, no auth changes), Option 1 (upstream visibility, self-service triggers, more flexible), or Option 3 (downstream autonomy, no upstream changes, but weaker guarantees)? 2. **Schedule ownership:** Central cron (EventBridge or upstream workflow) or per-repo opt-in schedules? @@ -335,7 +401,19 @@ Callback Lambda → HUD → DynamoDB → ClickHouse 7. **Periodic vs. nightly:** Do we need both `nightly` and `periodic` event types from day one, or start with `nightly` only and add `periodic` later? -### Shared Considerations (Both Options) +### Option 3-specific Questions + +8. **State machine divergence:** Is the WG comfortable maintaining two distinct callback models — state machine (`DISPATCHED → IN_PROGRESS → COMPLETED`) for PR/push and upsert for nightly/periodic — in the same Lambda? + +9. **Trust model:** Without a `DISPATCHED` record, how do we verify that a downstream repo actually ran CI against the SHA it claims? Is OIDC + SHA-existence sufficient, or do we need execution attestation? + +10. **SHA alignment:** If downstream repos fetch `main` HEAD independently, they will test different SHAs when `main` advances between crons. Is fragmented cross-backend comparison on HUD acceptable? + +11. **Missing-run detection:** Without `DISPATCHED` records, the zombie sweeper cannot detect silent cron failures in downstream repos. What replaces this? Should we add a "last seen" heartbeat per repo? + +12. **SHA overwrite on re-runs:** Upsert keyed by `(repo, SHA)` silently overwrites previous results. Should we maintain an audit trail of multiple runs against the same SHA? + +### Shared Considerations (All Options) **HUD Changes.** The HUD currently groups results by `pr_number`. For nightly runs there is no PR. Use `pr_number = 0` as a sentinel for non-PR runs. The `event_type` field already flows through the pipeline — it just needs to carry `nightly` or `periodic` instead of `pull_request`. Add a filter/view on `hud.pytorch.org/crcr` for non-PR results, or a dedicated `/crcr/nightly` page. From 9fdce1c05acd202dfa1579edae6c9943a08a37a4 Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 14 Jul 2026 18:16:46 +0530 Subject: [PATCH 04/12] Rewrite RFC to focus on Authenticated Self-Report design Remove Option 1 (upstream cron workflow) and Option 2 (EventBridge) sections. Present only the authenticated self-report approach proposed by @atalman, where downstream repos self-schedule nightly/periodic CI and report results via OIDC-validated callbacks with SHA-keyed upserts. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 306 ++++----------------------- 1 file changed, 37 insertions(+), 269 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 235fe55c..7cb30ac5 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -11,7 +11,7 @@ ## Summary -Extend the Cross-Repository CI Relay (CRCR) to support nightly and periodic CI schedules for downstream repositories. Currently, CRCR only dispatches on `pull_request` and `push` events from `pytorch/pytorch`. This RFC proposes adding scheduled dispatch capabilities so downstream backends can report nightly/periodic CI results to the PyTorch HUD. +Extend the Cross-Repository CI Relay (CRCR) to support nightly and periodic CI schedules for downstream repositories. Currently, CRCR only dispatches on `pull_request` and `push` events from `pytorch/pytorch`. This RFC proposes adding an authenticated self-report model so downstream backends can independently schedule nightly/periodic CI and report results to the PyTorch HUD. ## Motivation @@ -62,18 +62,6 @@ Callback Lambda forwards trusted + untrusted payloads to HUD HUD → DynamoDB → ClickHouse → hud.pytorch.org/crcr ``` -### Existing Precedent: EventBridge in the Callback Lambda - -The callback Lambda already handles EventBridge-triggered events for zombie cleanup. When `event.source == "crcr.sweeper"`, it runs the cleanup handler instead of processing a callback. This is the exact pattern Option 2 would follow — an EventBridge cron rule invoking a Lambda handler. - -```python -# callback/lambda_function.py (line 18-26, existing) -if event.get("source") == "crcr.sweeper": - config = get_config() - result = cleanup_handler.handle(config) - ... -``` - ### Downstream Workflow Structure Downstream repos listen for `repository_dispatch` with the event types they want: @@ -87,187 +75,11 @@ on: The `client_payload` always contains `event_type`, `delivery_id`, and the upstream webhook payload. Downstream workflows branch on `event_type` to extract PR number, SHA, ref, etc. -## Proposed Implementation - -Two options are presented for WG discussion. - -### Option 1: Upstream Cron Workflow in pytorch/pytorch → Webhook Lambda - -Add a GitHub Actions workflow in `pytorch/pytorch` that runs on a `schedule: cron` trigger. This workflow constructs a synthetic webhook-like payload and POSTs it to the webhook Lambda's existing `/github/webhook` endpoint. The Lambda processes it like any other webhook — validates, dispatches to downstream repos, sets `DISPATCHED` in Redis. - -``` -pytorch/pytorch scheduled workflow (e.g., daily 00:00 UTC) - ↓ -1. Fetch main HEAD SHA (git ls-remote or checkout) -2. Construct synthetic payload mimicking a push/nightly event -3. POST to webhook Lambda endpoint with authentication - ↓ -Webhook Lambda: - - Verify auth (new: OIDC or shared secret — NOT GitHub webhook signature) - - Parse synthetic payload - - event_type = "nightly" - - Generate delivery_id - - Call _dispatch_to_allowlist() - ↓ -(Same as existing dispatch from here) -``` - -**Upstream workflow:** - -```yaml -# pytorch/pytorch/.github/workflows/crcr-nightly-dispatch.yml -name: CRCR Nightly Dispatch -on: - schedule: - - cron: '0 0 * * *' # midnight UTC - workflow_dispatch: {} # manual trigger - -jobs: - dispatch: - runs-on: ubuntu-latest - permissions: - id-token: write - steps: - - name: Get main HEAD SHA - id: sha - run: | - SHA=$(git ls-remote https://github.com/pytorch/pytorch HEAD | cut -f1) - echo "sha=$SHA" >> "$GITHUB_OUTPUT" - - - name: Trigger CRCR nightly dispatch - run: | - TOKEN=$(curl -s \ - -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ - "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=pytorch-cross-repo-ci-relay" \ - | jq -r .value) - - curl -X POST "${{ secrets.RELAY_WEBHOOK_URL }}" \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $TOKEN" \ - -H "X-GitHub-Event: nightly" \ - -d '{ - "action": "nightly", - "repository": {"full_name": "pytorch/pytorch"}, - "ref": "refs/heads/main", - "after": "${{ steps.sha.outputs.sha }}" - }' -``` - -**Downstream workflow change** (same for both options): - -```yaml -on: - repository_dispatch: - types: [pull_request, push, nightly] -``` - -#### Option 1 — Pros - -| # | Advantage | Detail | -|---|-----------|--------| -| 1 | No new AWS infrastructure | No EventBridge rule, no Terraform. Uses existing Lambda endpoint. | -| 2 | Visible in pytorch/pytorch | The nightly trigger appears in the upstream repo's Actions tab. PyTorch maintainers can see run history, logs, and failures. | -| 3 | Manual trigger | `workflow_dispatch` means anyone with repo write access can manually re-trigger a nightly dispatch from the GitHub UI. | -| 4 | Familiar | It's a GitHub Actions workflow. No AWS console, no Terraform, no new infrastructure concepts for contributors. | -| 5 | Schedule flexibility | Multiple cron entries or separate workflows can define different schedules without touching the relay. | -| 6 | PR-able changes | Changing the schedule is a PR to `pytorch/pytorch`, reviewed by maintainers with standard CI. | -| 7 | Decoupled from relay deployment | The schedule lives in a workflow file, not in infrastructure. Relay re-deployments don't affect the cron schedule. | - -#### Option 1 — Cons - -| # | Disadvantage | Detail | -|---|-------------|--------| -| 1 | Authentication complexity | The webhook Lambda currently validates requests using GitHub webhook signatures (`X-Hub-Signature-256`). A workflow-triggered POST doesn't have this signature. Requires adding a new auth path (OIDC or shared secret). | -| 2 | Not a real GitHub webhook | The synthetic payload mimics a webhook but isn't one. The Lambda's `_verify_signature` would reject it. Needs a new code path to parse and validate this non-standard input. | -| 3 | Requires changes in pytorch/pytorch | Adding a workflow to the upstream repo requires buy-in from PyTorch maintainers. | -| 4 | GitHub cron unreliability | GitHub's `schedule:` trigger is best-effort. During high-load periods, cron runs can be delayed by minutes to hours or skipped entirely. | -| 5 | Secret management | If using a shared secret (not OIDC), the secret must be stored in `pytorch/pytorch` repo secrets and rotated in sync with the Lambda config. | -| 6 | Dependency on pytorch/pytorch | Downstream nightly scheduling depends on an upstream workflow running. If Actions runners are down or quotas exhausted, all downstream nightlies stop. | -| 7 | Event handler branching | A `nightly` event type has neither a `pull_request` nor a standard `push` payload shape, requiring new parsing logic and a synthetic payload contract. | -| 8 | `_SUPPORTED_EVENTS` gating | The Lambda's `_SUPPORTED_EVENTS = {"pull_request", "push"}` rejects unknown event types before signature verification. Adding `nightly` means changing this gate and adding the corresponding auth path. | - -#### Option 1 — Implementation Effort - -| Component | Work | Effort | -|-----------|------|--------| -| pytorch/pytorch workflow | New `.github/workflows/crcr-nightly-dispatch.yml` (~40 LOC) | ~0.5 day | -| Webhook Lambda auth | New auth path: OIDC verification (port `jwt_helper` from callback Lambda) or shared-secret validation | ~1.5 days | -| Webhook Lambda handler | Expand `_SUPPORTED_EVENTS`, add nightly event parsing path, generate `delivery_id` | ~1 day | -| Downstream workflow | Add `nightly` to `repository_dispatch.types` | ~0.5 day | -| HUD view | Filter/view for `event_type != pull_request` on `/crcr/nightly` | ~1 day | -| Testing | End-to-end: workflow → Lambda → downstream → callback → HUD | ~1 day | -| Org approval | PR to pytorch/pytorch — maintainer review cycle | ~1-5 days | -| **Total** | | **~5.5-9.5 days** | - ---- - -### Option 2: EventBridge Cron → Webhook Lambda - -Add an AWS EventBridge rule (cron schedule) that invokes the webhook Lambda directly. The Lambda detects the EventBridge source (no HTTP request, no GitHub signature), fetches the current `main` HEAD SHA, generates a synthetic `delivery_id`, and dispatches to downstream repos using the existing `_dispatch_to_allowlist` machinery. The downstream flow is identical to a PR or push dispatch. - -``` -EventBridge cron (e.g., daily 00:00 UTC) - ↓ -Webhook Lambda (new handler path: event.source == "crcr.scheduler") - ↓ -1. Fetch pytorch/pytorch main HEAD SHA via GitHub API -2. Build synthetic client_payload: - - event_type: "nightly" (or "periodic") - - delivery_id: "" (the main HEAD commit SHA — meaningful, correlatable, - and lets HUD map runs directly to github.com/pytorch/pytorch/commit/) - - payload: {repository: {full_name: "pytorch/pytorch"}, head_sha: "..."} -3. Call _dispatch_to_allowlist() — reuses existing dispatch logic: - - For each allowlisted repo: repository_dispatch + DISPATCHED in Redis - ↓ -Downstream repo receives repository_dispatch with event_type: nightly - ↓ -(Same as today: build → test → callback in_progress → callback completed) - ↓ -Callback Lambda → HUD → DynamoDB → ClickHouse -``` - -#### Option 2 — Pros - -| # | Advantage | Detail | -|---|-----------|--------| -| 1 | Full pipeline reuse | State machine, callbacks, HUD ingestion, timing metrics — all work unchanged. Zero modifications to the callback Lambda, HUD API, or ClickHouse schema. | -| 2 | Minimal downstream changes | Downstream repos add one event type string. No new workflows, no new actions, no registration steps. | -| 3 | No auth changes | EventBridge invokes the Lambda directly (not over HTTP). No signature validation, no OIDC. The Lambda receives a structured AWS event, which is inherently trusted. | -| 4 | Proven pattern | The zombie cleanup handler already uses EventBridge → Lambda (`source: crcr.sweeper`). Same architecture. | -| 5 | Central control | The relay controls who gets nightly dispatches (allowlist-gated) and when they fire. | -| 6 | State machine integrity | Every nightly run has a real `DISPATCHED` record. No special-casing or bypasses. | -| 7 | Schedule reliability | EventBridge has an SLA of 99.99%. Far more reliable than GitHub's best-effort cron. | -| 8 | Consistent SHA | All downstream repos build against the same HEAD SHA fetched once at dispatch time. | -| 9 | No cross-repo dependency | Everything lives in `pytorch/test-infra`. No PR to `pytorch/pytorch`, no maintainer approval outside the CRCR team. | - -#### Option 2 — Cons - -| # | Disadvantage | Detail | -|---|-------------|--------| -| 1 | New infrastructure | Requires a new EventBridge rule and a new handler path in the webhook Lambda. Terraform + Lambda code to deploy and maintain. | -| 2 | Inflexible scheduling | All downstream repos get the same cron schedule. A backend that wants every-6-hours or weekly can't customize without additional EventBridge rules. | -| 3 | Single point of failure | If the EventBridge rule or Lambda errors, no downstream repos get their nightly dispatch. Requires CloudWatch monitoring/alerting. | -| 4 | No self-service trigger | Downstream repos can't trigger an ad-hoc nightly run. They must wait for the next scheduled dispatch or ask a relay admin. | -| 5 | SHA staleness | The HEAD SHA is fetched once. If `main` advances between dispatch and the downstream build starting, the tested SHA is already stale. | -| 6 | Allowlist complexity | Needs a way to opt repos in/out of nightly dispatches. Adds a new dimension to the allowlist config. | - -#### Option 2 — Implementation Effort - -| Component | Work | Effort | -|-----------|------|--------| -| EventBridge rule | Terraform: `aws_cloudwatch_event_rule` + `aws_cloudwatch_event_target` pointing to webhook Lambda | ~0.5 day | -| Webhook Lambda handler | New path for `source: crcr.scheduler`: fetch HEAD SHA, build synthetic payload, call `_dispatch_to_allowlist()`. ~100-150 LOC | ~1 day | -| Allowlist | Add `nightly: true/false` per-repo flag or a separate `nightly:` section | ~0.5 day | -| Downstream workflow | Add `nightly` to `repository_dispatch.types`, branch `run-name` on `event_type` | ~0.5 day | -| HUD view | Filter/view for `event_type != pull_request` on `/crcr/nightly` | ~1 day | -| Testing | End-to-end test with `TorchedHat/pytorch-redhat-ci` | ~0.5 day | -| **Total** | | **~4 days** | - -### Option 3: Authenticated Self-Report (Option C) +## Proposed Design: Authenticated Self-Report *Proposed by @atalman in [PR #98 comment](https://github.com/pytorch/rfcs/pull/98#issuecomment-4962790260).* -Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay stops being a trigger and becomes a **validating ingest endpoint**. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped and replaced with authorization + SHA-validity at callback time. +Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay stops being a trigger and becomes a **validating ingest endpoint**. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped for nightly/periodic events and replaced with authorization + SHA-validity at callback time. ``` Downstream repo's own cron schedule @@ -288,7 +100,7 @@ Relay validates: Upsert record keyed by (repo, SHA) → HUD ``` -#### Option 3 — Pros +### Advantages | # | Advantage | Detail | |---|-----------|--------| @@ -296,23 +108,23 @@ Upsert record keyed by (repo, SHA) → HUD | 2 | No new AWS infrastructure | No EventBridge, no Terraform. | | 3 | Self-service schedule | Each downstream repo owns its cron. Changes are PRs to the downstream repo. | | 4 | Manual re-trigger | `workflow_dispatch` on the downstream cron workflow re-runs the nightly. | -| 5 | Coordination-free correlation | dispatch_id is the SHA, derivable by any actor independently. | +| 5 | Coordination-free correlation | `dispatch_id` is the SHA, derivable by any actor independently. | -#### Option 3 — Cons +### Open Questions and Considerations -| # | Disadvantage | Detail | -|---|-------------|--------| -| 1 | State machine bypass | Drops the `DISPATCHED` precondition. "Upsert" is a fundamentally different model from the existing state machine. Requires a new callback Lambda code path. | -| 2 | Trust model weakening | The relay can verify the SHA is real, but not that CI actually ran against it. A downstream could self-report results for a SHA it never tested. | -| 3 | No SHA alignment | Each downstream independently fetches HEAD. If `main` advances between repos' crons, they test different commits. Cross-backend comparison on HUD is fragmented. | -| 4 | Missing-run detection is silent | No `DISPATCHED` record means no zombie sweeper coverage. If a downstream's cron silently breaks, nobody on the relay side knows. | -| 5 | SHA overwrite on re-runs | Upsert keyed by `(repo, SHA)` overwrites previous results. No audit trail of multiple runs against the same SHA. | +| # | Concern | Detail | +|---|---------|--------| +| 1 | State machine bypass | Drops the `DISPATCHED` precondition. "Upsert" is a fundamentally different model from the existing state machine. Requires a new callback Lambda code path that coexists with the existing PR/push state machine. | +| 2 | Trust model change | The relay can verify the SHA is real, but not that CI actually ran against it. A downstream could self-report results for a SHA it never tested. Is OIDC + SHA-existence sufficient, or do we need execution attestation? | +| 3 | No SHA alignment | Each downstream independently fetches HEAD. If `main` advances between repos' crons, they test different commits. Cross-backend comparison on HUD may be fragmented. | +| 4 | Missing-run detection is silent | No `DISPATCHED` record means no zombie sweeper coverage. If a downstream's cron silently breaks, nobody on the relay side knows. Should we add a "last seen" heartbeat per repo? | +| 5 | SHA overwrite on re-runs | Upsert keyed by `(repo, SHA)` overwrites previous results. No audit trail of multiple runs against the same SHA. Should we maintain run history? | | 6 | No timing metrics | Without `dispatched_at`, queue time metrics are lost. | -| 7 | SHA validation adds dependency | `GET /repos/pytorch/pytorch/commits/{sha}` requires GitHub API availability and rate limits. Caching specifics are undefined. | -| 8 | Callback payload contract undefined | Current callbacks carry `delivery_id`, PR metadata. A nightly self-report has different fields. The new payload schema is not specified. | -| 9 | Significant Lambda changes | New callback Lambda code path (~200 LOC), SHA validation + caching, upsert logic. The complexity shifts from AWS resources to Lambda code. | +| 7 | SHA validation adds dependency | `GET /repos/pytorch/pytorch/commits/{sha}` requires GitHub API availability and rate limits. Caching strategy needs to be defined. | +| 8 | Callback payload contract undefined | Current callbacks carry `delivery_id`, PR metadata. A nightly self-report has different fields. The new payload schema needs to be specified. | +| 9 | Callback Lambda complexity | New callback Lambda code path (~200 LOC), SHA validation + caching, upsert logic. The complexity shifts from AWS resources to Lambda code. | -#### Option 3 — Implementation Effort +### Implementation Effort | Component | Work | Effort | |-----------|------|--------| @@ -320,116 +132,72 @@ Upsert record keyed by (repo, SHA) → HUD | SHA validation + caching | GitHub API integration + cache layer | ~1 day | | Downstream cron workflow | New workflow in each downstream: fetch SHA, run CI, call callback | ~1 day per repo | | HUD view | Filter/view for non-PR results grouped by SHA | ~1 day | -| Testing | End-to-end test with TorchedHat/pytorch-redhat-ci | ~1 day | +| Testing | End-to-end test with `TorchedHat/pytorch-redhat-ci` | ~1 day | | **Total** | | **~5-6 days** | ---- - -## Side-by-Side Comparison - -| Criteria | Option 2: EventBridge → Lambda | Option 1: pytorch/pytorch Cron → Lambda | Option 3: Authenticated Self-Report | -|----------|-------------------------------|----------------------------------------|--------------------------------------| -| New AWS infrastructure | Yes (EventBridge rule) | No | No | -| Changes to pytorch/pytorch | No | Yes (new workflow file) | No | -| Webhook Lambda changes | New handler path (~150 LOC) | New auth path + event parser (~250 LOC) | None | -| Callback Lambda changes | None | None | ~200 LOC (upsert + SHA validation) | -| Auth changes | None (internal invoke) | Significant (OIDC or shared secret) | Modified (OIDC as sole trust anchor) | -| State machine | Preserved | Preserved (with new auth path) | Bypassed (upsert replaces state machine) | -| Downstream repo changes | Add `nightly` to dispatch types | Add `nightly` to dispatch types | New cron workflow + callback | -| SHA alignment across backends | Guaranteed (single dispatch) | Guaranteed (single dispatch) | Not guaranteed | -| Missing-run detection | Zombie sweeper works | Zombie sweeper works | Not supported | -| Timing metrics | Full | Full | Partial (no dispatched_at) | -| Trust model | Relay-controlled dispatch | Relay-controlled dispatch | Self-reported (SHA verified, execution not verified) | -| Central schedule control | Yes (Terraform) | No (workflow in upstream repo) | No (each downstream independently) | -| Manual re-trigger | Lambda console/CLI | Built-in (`workflow_dispatch`) | Downstream `workflow_dispatch` | -| Schedule reliability | High (EventBridge 99.99% SLA) | Medium (GitHub cron best-effort) | Medium (GitHub cron per downstream) | -| Operational visibility | CloudWatch logs/metrics | GitHub Actions run history | Downstream repo Actions tab | -| Org approval needed | No (test-infra only) | Yes (pytorch/pytorch PR) | No | -| Security surface change | None | New auth path in webhook Lambda | New callback Lambda entry point | -| **Implementation effort** | **~4 days** | **~5.5-9.5 days** | **~5-6 days** | - ## Metrics -- **Nightly dispatch success rate**: Percentage of scheduled dispatches that successfully reach all allowlisted downstream repos. -- **Nightly callback completion rate**: Percentage of dispatched nightly runs that report back `COMPLETED` via callback. +- **Nightly callback completion rate**: Percentage of expected nightly runs (per downstream cron schedule) that successfully report results via callback. - **HUD coverage**: Number of downstream backends with nightly results visible on `hud.pytorch.org/crcr`. - **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD. ## Drawbacks -- Adds complexity to the dispatch pipeline (new event type, new handler path). +- Introduces a second callback model (upsert) alongside the existing state machine, increasing Lambda complexity. - Nightly failures have no upstream PR to annotate — requires a separate notification mechanism. - Concurrent nightly and PR-triggered runs may compete for downstream CI resources. -- Increases the surface area of the relay's responsibility. +- No central visibility into whether downstream nightlies are running or silently broken. +- SHA fragmentation across backends makes cross-repo nightly comparison harder. ## Alternatives **Do nothing.** Downstream repos run nightly CI independently and report results in their own dashboards. PyTorch maintainers have no visibility into nightly downstream health. This is the current state. -**Downstream self-dispatch.** Each downstream repo configures its own `schedule: cron` and bypasses the state machine. This breaks the callback path (no `DISPATCHED` record) and requires per-repo configuration with no central control. +**Central dispatch (EventBridge or upstream workflow).** The relay centrally triggers nightly dispatches to downstream repos, preserving the full state machine. This guarantees SHA alignment, zombie detection, and timing metrics but requires either new AWS infrastructure or changes to `pytorch/pytorch`. ## Prior Art -- **CRCR Zombie Cleanup (EventBridge → Lambda)**: The existing `crcr.sweeper` pattern in the callback Lambda is the direct precedent for Option 2. - **GitHub Actions scheduled workflows**: Widely used for nightly builds across the PyTorch ecosystem (e.g., `pytorch/pytorch` nightly builds, `pytorch/vision` nightly tests). - **RFC-0050**: The original CRCR RFC that established the dispatch → callback → HUD pipeline for `pull_request` events. - **RFC-0054**: HUD integration RFC that defined the ClickHouse schema and dashboard views for CRCR results. -## Feasibility Assessment - -**Option 1 has real advantages for visibility and self-service.** `workflow_dispatch` is genuinely useful — being able to click "Run workflow" in the GitHub UI to trigger an ad-hoc nightly dispatch without AWS console access is valuable for debugging. The schedule living in a workflow file (not Terraform) means it's visible, grep-able, and PR-able by anyone with repo access. These advantages could justify the additional effort if the WG prioritizes visibility and self-service. - -**Option 2 is lower effort and lower risk.** Auth is the key differentiator — Option 2 requires zero auth changes (EventBridge invokes the Lambda directly as an internal AWS call, inherently trusted), while Option 1 requires adding a second authentication mechanism to the webhook Lambda. Option 2 follows the exact same EventBridge → Lambda pattern already used for zombie cleanup. Everything stays in `pytorch/test-infra` with no cross-repo coordination. - -**Option 3 gives downstream repos full autonomy but trades state machine guarantees.** By shifting the trigger to downstream crons and validating via OIDC + SHA at callback time, it avoids touching upstream infra entirely. However, this is a fundamentally different trust model — the relay trusts that downstream repos actually ran CI against the claimed SHA, but has no dispatch record to verify this. Missing-run detection, SHA alignment across backends, and queue-time metrics are all sacrificed. The Lambda changes are non-trivial (new upsert path, SHA validation, caching), and the callback payload contract needs careful design to avoid breaking the existing state machine path. - ## Unresolved Questions ### For WG Discussion -1. **Which option does the WG prefer?** Option 2 (lower effort, central control, no auth changes), Option 1 (upstream visibility, self-service triggers, more flexible), or Option 3 (downstream autonomy, no upstream changes, but weaker guarantees)? - -2. **Schedule ownership:** Central cron (EventBridge or upstream workflow) or per-repo opt-in schedules? - -3. **Scope:** Should nightly dispatches go to all allowlisted repos, or only repos that opt in via a config flag? - -4. **SHA policy:** Always `main` HEAD, or allow per-repo target branches? - -5. **Failure SLA:** Is a HUD view sufficient, or do we need active notifications (Slack, issues)? - -6. **Manual triggers:** How important is the ability to manually re-trigger a nightly from the GitHub UI without AWS access? +1. **State machine divergence:** Is the WG comfortable maintaining two distinct callback models — state machine (`DISPATCHED → IN_PROGRESS → COMPLETED`) for PR/push and upsert for nightly/periodic — in the same Lambda? -7. **Periodic vs. nightly:** Do we need both `nightly` and `periodic` event types from day one, or start with `nightly` only and add `periodic` later? +2. **Trust model:** Without a `DISPATCHED` record, how do we verify that a downstream repo actually ran CI against the SHA it claims? Is OIDC + SHA-existence sufficient? -### Option 3-specific Questions +3. **SHA alignment:** If downstream repos fetch `main` HEAD independently, they will test different SHAs when `main` advances between crons. Is fragmented cross-backend comparison on HUD acceptable? -8. **State machine divergence:** Is the WG comfortable maintaining two distinct callback models — state machine (`DISPATCHED → IN_PROGRESS → COMPLETED`) for PR/push and upsert for nightly/periodic — in the same Lambda? +4. **Missing-run detection:** Without `DISPATCHED` records, the zombie sweeper cannot detect silent cron failures in downstream repos. What replaces this? -9. **Trust model:** Without a `DISPATCHED` record, how do we verify that a downstream repo actually ran CI against the SHA it claims? Is OIDC + SHA-existence sufficient, or do we need execution attestation? +5. **Scope:** Should all allowlisted repos be eligible for nightly self-report, or only repos at a certain level (e.g., L3+)? -10. **SHA alignment:** If downstream repos fetch `main` HEAD independently, they will test different SHAs when `main` advances between crons. Is fragmented cross-backend comparison on HUD acceptable? +6. **SHA policy:** Always `main` HEAD, or allow per-repo target branches (nightly branch, viable-strict, release branches)? -11. **Missing-run detection:** Without `DISPATCHED` records, the zombie sweeper cannot detect silent cron failures in downstream repos. What replaces this? Should we add a "last seen" heartbeat per repo? +7. **Failure SLA:** Is a HUD view sufficient, or do we need active notifications (Slack, issues)? -12. **SHA overwrite on re-runs:** Upsert keyed by `(repo, SHA)` silently overwrites previous results. Should we maintain an audit trail of multiple runs against the same SHA? +8. **Periodic vs. nightly:** Do we need both `nightly` and `periodic` event types from day one, or start with `nightly` only and add `periodic` later? -### Shared Considerations (All Options) +### Shared Considerations **HUD Changes.** The HUD currently groups results by `pr_number`. For nightly runs there is no PR. Use `pr_number = 0` as a sentinel for non-PR runs. The `event_type` field already flows through the pipeline — it just needs to carry `nightly` or `periodic` instead of `pull_request`. Add a filter/view on `hud.pytorch.org/crcr` for non-PR results, or a dedicated `/crcr/nightly` page. -**Allowlist Scoping.** Not all L1+ repos should receive nightly dispatches. Consider: +**Allowlist Scoping.** Not all L1+ repos should report nightly results. Consider: ```yaml L2: - org/repo: oncalls: user1, user2 - nightly: true # opt-in to nightly dispatches + nightly: true # opt-in to nightly self-report ``` -Or gate on allowlist level (e.g., only L3+ get nightlies by default). +Or gate on allowlist level (e.g., only L3+ can self-report nightlies by default). **SHA Selection.** For nightly runs, which SHA to test against? -- `main` HEAD at dispatch time (most common, default) +- `main` HEAD at cron time (most common, default) - Latest release tag (for release compatibility testing) - A specific branch (e.g., `release/2.x`) @@ -439,7 +207,7 @@ Or gate on allowlist level (e.g., only L3+ get nightlies by default). - HUD dashboard alert on `/crcr/nightly` - Email to repo oncalls from the allowlist -**Deduplication.** If a nightly dispatch happens while a PR-triggered run is in progress for the same downstream repo, the state machine handles them independently (different `delivery_id`). However, concurrent builds may compete for downstream CI resources. Consider whether the downstream workflow should use `concurrency:` groups to avoid parallel nightly + PR builds. +**Deduplication.** If a nightly self-report arrives while a PR-triggered run is in progress for the same downstream repo, they are handled independently (different `delivery_id` / event type). However, concurrent builds may compete for downstream CI resources. Consider whether the downstream workflow should use `concurrency:` groups to avoid parallel nightly + PR builds. ## Resolution From 837fb79bb7a5a85e70d29bf8af81e6f2dd8b19b7 Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 14 Jul 2026 18:20:49 +0530 Subject: [PATCH 05/12] Simplify RFC: remove open questions, add file changes table Keep only the design structure and implementation details. Replace open questions/drawbacks/alternatives with a concrete file changes table showing exactly what needs to be modified. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 82 +++------------------------- 1 file changed, 8 insertions(+), 74 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 7cb30ac5..8f8760be 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -110,20 +110,6 @@ Upsert record keyed by (repo, SHA) → HUD | 4 | Manual re-trigger | `workflow_dispatch` on the downstream cron workflow re-runs the nightly. | | 5 | Coordination-free correlation | `dispatch_id` is the SHA, derivable by any actor independently. | -### Open Questions and Considerations - -| # | Concern | Detail | -|---|---------|--------| -| 1 | State machine bypass | Drops the `DISPATCHED` precondition. "Upsert" is a fundamentally different model from the existing state machine. Requires a new callback Lambda code path that coexists with the existing PR/push state machine. | -| 2 | Trust model change | The relay can verify the SHA is real, but not that CI actually ran against it. A downstream could self-report results for a SHA it never tested. Is OIDC + SHA-existence sufficient, or do we need execution attestation? | -| 3 | No SHA alignment | Each downstream independently fetches HEAD. If `main` advances between repos' crons, they test different commits. Cross-backend comparison on HUD may be fragmented. | -| 4 | Missing-run detection is silent | No `DISPATCHED` record means no zombie sweeper coverage. If a downstream's cron silently breaks, nobody on the relay side knows. Should we add a "last seen" heartbeat per repo? | -| 5 | SHA overwrite on re-runs | Upsert keyed by `(repo, SHA)` overwrites previous results. No audit trail of multiple runs against the same SHA. Should we maintain run history? | -| 6 | No timing metrics | Without `dispatched_at`, queue time metrics are lost. | -| 7 | SHA validation adds dependency | `GET /repos/pytorch/pytorch/commits/{sha}` requires GitHub API availability and rate limits. Caching strategy needs to be defined. | -| 8 | Callback payload contract undefined | Current callbacks carry `delivery_id`, PR metadata. A nightly self-report has different fields. The new payload schema needs to be specified. | -| 9 | Callback Lambda complexity | New callback Lambda code path (~200 LOC), SHA validation + caching, upsert logic. The complexity shifts from AWS resources to Lambda code. | - ### Implementation Effort | Component | Work | Effort | @@ -141,19 +127,15 @@ Upsert record keyed by (repo, SHA) → HUD - **HUD coverage**: Number of downstream backends with nightly results visible on `hud.pytorch.org/crcr`. - **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD. -## Drawbacks - -- Introduces a second callback model (upsert) alongside the existing state machine, increasing Lambda complexity. -- Nightly failures have no upstream PR to annotate — requires a separate notification mechanism. -- Concurrent nightly and PR-triggered runs may compete for downstream CI resources. -- No central visibility into whether downstream nightlies are running or silently broken. -- SHA fragmentation across backends makes cross-repo nightly comparison harder. - -## Alternatives +### File Changes -**Do nothing.** Downstream repos run nightly CI independently and report results in their own dashboards. PyTorch maintainers have no visibility into nightly downstream health. This is the current state. - -**Central dispatch (EventBridge or upstream workflow).** The relay centrally triggers nightly dispatches to downstream repos, preserving the full state machine. This guarantees SHA alignment, zombie detection, and timing metrics but requires either new AWS infrastructure or changes to `pytorch/pytorch`. +| File | Change | +|------|--------| +| `callback/lambda_function.py` | New code path: detect `event_type ∈ {nightly, periodic}`, skip state machine, validate OIDC + SHA, upsert record to DynamoDB | +| `callback/sha_validator.py` | New module: `GET /repos/pytorch/pytorch/commits/{sha}` with TTL cache to avoid repeated GitHub API calls | +| `allowlist.yml` | Add `nightly: true/false` per-repo flag to control which repos can self-report nightly results | +| Downstream workflow (per repo) | New `schedule: cron` workflow that fetches `pytorch/pytorch` HEAD SHA, runs CI, and calls the callback with `event_type: nightly` and `dispatch_id: ` | +| HUD (`torchci/`) | Filter/view for `event_type != pull_request` on `/crcr/nightly` or dedicated nightly page | ## Prior Art @@ -161,54 +143,6 @@ Upsert record keyed by (repo, SHA) → HUD - **RFC-0050**: The original CRCR RFC that established the dispatch → callback → HUD pipeline for `pull_request` events. - **RFC-0054**: HUD integration RFC that defined the ClickHouse schema and dashboard views for CRCR results. -## Unresolved Questions - -### For WG Discussion - -1. **State machine divergence:** Is the WG comfortable maintaining two distinct callback models — state machine (`DISPATCHED → IN_PROGRESS → COMPLETED`) for PR/push and upsert for nightly/periodic — in the same Lambda? - -2. **Trust model:** Without a `DISPATCHED` record, how do we verify that a downstream repo actually ran CI against the SHA it claims? Is OIDC + SHA-existence sufficient? - -3. **SHA alignment:** If downstream repos fetch `main` HEAD independently, they will test different SHAs when `main` advances between crons. Is fragmented cross-backend comparison on HUD acceptable? - -4. **Missing-run detection:** Without `DISPATCHED` records, the zombie sweeper cannot detect silent cron failures in downstream repos. What replaces this? - -5. **Scope:** Should all allowlisted repos be eligible for nightly self-report, or only repos at a certain level (e.g., L3+)? - -6. **SHA policy:** Always `main` HEAD, or allow per-repo target branches (nightly branch, viable-strict, release branches)? - -7. **Failure SLA:** Is a HUD view sufficient, or do we need active notifications (Slack, issues)? - -8. **Periodic vs. nightly:** Do we need both `nightly` and `periodic` event types from day one, or start with `nightly` only and add `periodic` later? - -### Shared Considerations - -**HUD Changes.** The HUD currently groups results by `pr_number`. For nightly runs there is no PR. Use `pr_number = 0` as a sentinel for non-PR runs. The `event_type` field already flows through the pipeline — it just needs to carry `nightly` or `periodic` instead of `pull_request`. Add a filter/view on `hud.pytorch.org/crcr` for non-PR results, or a dedicated `/crcr/nightly` page. - -**Allowlist Scoping.** Not all L1+ repos should report nightly results. Consider: - -```yaml -L2: - - org/repo: - oncalls: user1, user2 - nightly: true # opt-in to nightly self-report -``` - -Or gate on allowlist level (e.g., only L3+ can self-report nightlies by default). - -**SHA Selection.** For nightly runs, which SHA to test against? -- `main` HEAD at cron time (most common, default) -- Latest release tag (for release compatibility testing) -- A specific branch (e.g., `release/2.x`) - -**Failure Notifications.** PR failures are visible on the PR. Nightly failures have no PR to annotate. -- Dedicated Slack channel for CRCR nightly failures -- Auto-create GitHub issues on consecutive failures -- HUD dashboard alert on `/crcr/nightly` -- Email to repo oncalls from the allowlist - -**Deduplication.** If a nightly self-report arrives while a PR-triggered run is in progress for the same downstream repo, they are handled independently (different `delivery_id` / event type). However, concurrent builds may compete for downstream CI resources. Consider whether the downstream workflow should use `concurrency:` groups to avoid parallel nightly + PR builds. - ## Resolution TBD — pending WG discussion. From be89ac12d1876668e6a66fddbc61136b9e32ab5d Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 14 Jul 2026 18:25:33 +0530 Subject: [PATCH 06/12] Add Previously Considered Options section to RFC Briefly document the two alternatives (EventBridge cron and upstream workflow) that were evaluated before settling on the authenticated self-report design, with reasons they were set aside. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 8f8760be..c59051eb 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -137,6 +137,16 @@ Upsert record keyed by (repo, SHA) → HUD | Downstream workflow (per repo) | New `schedule: cron` workflow that fetches `pytorch/pytorch` HEAD SHA, runs CI, and calls the callback with `event_type: nightly` and `dispatch_id: ` | | HUD (`torchci/`) | Filter/view for `event_type != pull_request` on `/crcr/nightly` or dedicated nightly page | +## Previously Considered Options + +Two alternative approaches were evaluated before arriving at the authenticated self-report design: + +**Option A: EventBridge Cron → Webhook Lambda.** An AWS EventBridge rule on a cron schedule invokes the webhook Lambda directly. The Lambda fetches `pytorch/pytorch` main HEAD SHA, builds a synthetic `client_payload`, and dispatches to downstream repos via the existing `_dispatch_to_allowlist()` path. This preserves the full state machine and guarantees SHA alignment across all backends. However, it introduces new AWS infrastructure (EventBridge rule, Terraform config, CloudWatch alarms) and centralizes schedule control — downstream repos cannot customize their own cron timing without additional EventBridge rules. + +**Option B: Upstream Cron Workflow in pytorch/pytorch → Webhook Lambda.** A `schedule: cron` workflow in `pytorch/pytorch` constructs a synthetic payload and POSTs it to the webhook Lambda endpoint with OIDC authentication. This gives upstream visibility (schedule appears in the Actions tab) and built-in manual re-trigger via `workflow_dispatch`. However, it requires adding a second authentication path (OIDC or shared secret) to the webhook Lambda, changes to `pytorch/pytorch` requiring maintainer approval, and depends on GitHub cron reliability. + +Both options were set aside in favor of the self-report model because they require either new AWS infrastructure or upstream repo changes, while the proposed design keeps all changes within the callback Lambda and downstream repos. + ## Prior Art - **GitHub Actions scheduled workflows**: Widely used for nightly builds across the PyTorch ecosystem (e.g., `pytorch/pytorch` nightly builds, `pytorch/vision` nightly tests). From 37189a155691d530b35539551b0db0149311e830 Mon Sep 17 00:00:00 2001 From: Subin George Date: Wed, 15 Jul 2026 15:27:25 +0530 Subject: [PATCH 07/12] Specify nightly branch as SHA source, add example workflow Nightly runs fetch HEAD from pytorch/pytorch nightly branch (updated daily by trigger_nightly_core.yml). Periodic runs use main or viable/strict. dispatch_id is the commit SHA per @atalman's suggestion. Includes SHA source table, example downstream workflow, and updated flow diagram. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 62 +++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 6 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index c59051eb..6bf992c7 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -79,18 +79,30 @@ The `client_payload` always contains `event_type`, `delivery_id`, and the upstre *Proposed by @atalman in [PR #98 comment](https://github.com/pytorch/rfcs/pull/98#issuecomment-4962790260).* -Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay stops being a trigger and becomes a **validating ingest endpoint**. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped for nightly/periodic events and replaced with authorization + SHA-validity at callback time. +Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay becomes a **validating ingest endpoint** for nightly/periodic events. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped for these event types and replaced with authorization + SHA-validity at callback time. + +### SHA Sources + +| Event type | Branch | SHA source | Rationale | +|------------|--------|------------|-----------| +| `nightly` | [`pytorch/pytorch/tree/nightly`](https://github.com/pytorch/pytorch/tree/nightly) | Top-of-tree commit on the `nightly` branch | The `nightly` branch is updated daily by [`trigger_nightly_core.yml`](https://github.com/pytorch/test-infra/blob/main/.github/workflows/trigger_nightly_core.yml). It represents the latest nightly-validated state of PyTorch. | +| `periodic` | `main` or `viable/strict` | Top-of-tree commit on the target branch | Periodic tests run against the latest `main` HEAD or the latest viable/strict commit. | + +### Flow ``` -Downstream repo's own cron schedule +Downstream repo's cron schedule (e.g., daily 02:00 UTC) ↓ -Fetches pytorch/pytorch HEAD SHA (or nightly/viable-strict ref) +Fetch top-of-tree SHA: + - Nightly: git ls-remote pytorch/pytorch refs/heads/nightly + - Periodic: git ls-remote pytorch/pytorch refs/heads/main ↓ Runs CI against that SHA ↓ Calls back to the relay with: - OIDC token (proves repo identity) - - dispatch_id = pytorch/pytorch commit SHA + - dispatch_id = the commit SHA (idempotent, correlatable, + maps directly to github.com/pytorch/pytorch/commit/) - event_type = "nightly" or "periodic" ↓ Relay validates: @@ -100,6 +112,43 @@ Relay validates: Upsert record keyed by (repo, SHA) → HUD ``` +### Example: Downstream Nightly Workflow + +```yaml +# downstream-repo/.github/workflows/crcr-nightly.yml +name: CRCR Nightly CI +on: + schedule: + - cron: '0 2 * * *' # daily at 02:00 UTC + workflow_dispatch: {} # manual re-trigger + +jobs: + nightly: + runs-on: ubuntu-latest + permissions: + id-token: write + steps: + - name: Get nightly branch HEAD SHA + id: sha + run: | + SHA=$(git ls-remote https://github.com/pytorch/pytorch refs/heads/nightly | cut -f1) + echo "sha=$SHA" >> "$GITHUB_OUTPUT" + echo "Testing against nightly SHA: $SHA" + + - name: Build and test against nightly + run: | + # Clone pytorch at the nightly SHA, build, run tests + ... + + - name: Report results to CRCR + uses: ./.github/actions/cross-repo-ci-relay + with: + dispatch_id: ${{ steps.sha.outputs.sha }} + event_type: nightly + status: completed + conclusion: ${{ job.status }} +``` + ### Advantages | # | Advantage | Detail | @@ -108,7 +157,8 @@ Upsert record keyed by (repo, SHA) → HUD | 2 | No new AWS infrastructure | No EventBridge, no Terraform. | | 3 | Self-service schedule | Each downstream repo owns its cron. Changes are PRs to the downstream repo. | | 4 | Manual re-trigger | `workflow_dispatch` on the downstream cron workflow re-runs the nightly. | -| 5 | Coordination-free correlation | `dispatch_id` is the SHA, derivable by any actor independently. | +| 5 | Coordination-free correlation | `dispatch_id` is the nightly branch HEAD SHA — meaningful, idempotent, and lets HUD map runs directly to `github.com/pytorch/pytorch/commit/`. | +| 6 | Leverages existing nightly branch | The `nightly` branch already exists and is updated daily by `trigger_nightly_core.yml`. No new infrastructure needed to determine the SHA. | ### Implementation Effort @@ -134,7 +184,7 @@ Upsert record keyed by (repo, SHA) → HUD | `callback/lambda_function.py` | New code path: detect `event_type ∈ {nightly, periodic}`, skip state machine, validate OIDC + SHA, upsert record to DynamoDB | | `callback/sha_validator.py` | New module: `GET /repos/pytorch/pytorch/commits/{sha}` with TTL cache to avoid repeated GitHub API calls | | `allowlist.yml` | Add `nightly: true/false` per-repo flag to control which repos can self-report nightly results | -| Downstream workflow (per repo) | New `schedule: cron` workflow that fetches `pytorch/pytorch` HEAD SHA, runs CI, and calls the callback with `event_type: nightly` and `dispatch_id: ` | +| Downstream workflow (per repo) | New `schedule: cron` workflow: fetch top-of-tree SHA from `nightly` branch (or `main` for periodic), run CI, call callback with `event_type: nightly` and `dispatch_id: ` | | HUD (`torchci/`) | Filter/view for `event_type != pull_request` on `/crcr/nightly` or dedicated nightly page | ## Previously Considered Options From 813d1643da5711ad2cdd350f93754aeac5b10e2b Mon Sep 17 00:00:00 2001 From: Subin George Date: Wed, 15 Jul 2026 15:37:30 +0530 Subject: [PATCH 08/12] Clarify single-callback model for nightly/periodic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No in_progress state — downstream workflows report the final result in one callback. No Redis state tracking, no zombie sweeper needed. Adds comparison table showing PR/push vs nightly/periodic callback models. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 31 +++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 6bf992c7..63c938f0 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -79,7 +79,17 @@ The `client_payload` always contains `event_type`, `delivery_id`, and the upstre *Proposed by @atalman in [PR #98 comment](https://github.com/pytorch/rfcs/pull/98#issuecomment-4962790260).* -Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay becomes a **validating ingest endpoint** for nightly/periodic events. The `DISPATCHED → IN_PROGRESS → COMPLETED` state machine precondition is dropped for these event types and replaced with authorization + SHA-validity at callback time. +Instead of a central scheduler dispatching *to* downstream repos, each downstream repo drives its own schedule and reports results back. The relay becomes a **validating ingest endpoint** for nightly/periodic events. The full state machine is replaced with a **single-callback model**: + +| | PR / push (existing) | Nightly / periodic (new) | +|---|---|---| +| **Trigger** | Upstream webhook → relay dispatches to downstream | Downstream cron (self-triggered) | +| **State machine** | `DISPATCHED → IN_PROGRESS → COMPLETED` (two callbacks, Redis state tracking) | No state machine — single callback with final result | +| **Callbacks** | Two: `in_progress` then `completed` | One: `completed` only | +| **Redis** | Required (state tracking + zombie sweeper) | Not used | +| **Validation** | GitHub webhook signature (`X-Hub-Signature-256`) | OIDC token + SHA existence | + +This significantly simplifies the relay path for nightly/periodic: no Redis writes, no state transitions, no zombie sweeper coverage. The downstream workflow runs to completion and reports the final result in a single callback. ### SHA Sources @@ -97,19 +107,23 @@ Fetch top-of-tree SHA: - Nightly: git ls-remote pytorch/pytorch refs/heads/nightly - Periodic: git ls-remote pytorch/pytorch refs/heads/main ↓ -Runs CI against that SHA +Runs CI against that SHA (build, test, etc.) ↓ -Calls back to the relay with: +Single callback to the relay (no in_progress step): - OIDC token (proves repo identity) - dispatch_id = the commit SHA (idempotent, correlatable, maps directly to github.com/pytorch/pytorch/commit/) - event_type = "nightly" or "periodic" + - status = "completed" + - conclusion = "success" | "failure" | "timed_out" ↓ Relay validates: - 1. OIDC token → repo is on allowlist - 2. GET /repos/pytorch/pytorch/commits/{sha} → SHA is real + 1. OIDC token → repo is on allowlist with nightly enabled + 2. GET /repos/pytorch/pytorch/commits/{sha} → SHA exists + ↓ +Direct upsert to DynamoDB (no Redis, no state machine) ↓ -Upsert record keyed by (repo, SHA) → HUD +DynamoDB → ClickHouse replicator → HUD ``` ### Example: Downstream Nightly Workflow @@ -140,7 +154,10 @@ jobs: # Clone pytorch at the nightly SHA, build, run tests ... + # Single callback — no in_progress step needed for nightly. + # Reports the final result directly to the relay. - name: Report results to CRCR + if: always() uses: ./.github/actions/cross-repo-ci-relay with: dispatch_id: ${{ steps.sha.outputs.sha }} @@ -181,7 +198,7 @@ jobs: | File | Change | |------|--------| -| `callback/lambda_function.py` | New code path: detect `event_type ∈ {nightly, periodic}`, skip state machine, validate OIDC + SHA, upsert record to DynamoDB | +| `callback/lambda_function.py` | New code path: detect `event_type ∈ {nightly, periodic}`, skip state machine entirely (no Redis), validate OIDC + SHA, single upsert to DynamoDB | | `callback/sha_validator.py` | New module: `GET /repos/pytorch/pytorch/commits/{sha}` with TTL cache to avoid repeated GitHub API calls | | `allowlist.yml` | Add `nightly: true/false` per-repo flag to control which repos can self-report nightly results | | Downstream workflow (per repo) | New `schedule: cron` workflow: fetch top-of-tree SHA from `nightly` branch (or `main` for periodic), run CI, call callback with `event_type: nightly` and `dispatch_id: ` | From 70d55793bcc19d013296a5c60d839d67ea283f1b Mon Sep 17 00:00:00 2001 From: Subin George Date: Thu, 16 Jul 2026 13:08:46 +0530 Subject: [PATCH 09/12] Add Replay & Recovery section to nightly/periodic RFC Document the manual replay procedure via workflow_dispatch. Callbacks are idempotent (SHA-based delivery_id + upsert), so re-running is safe. Automated detection deferred to WG. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 63c938f0..5b1a7fc2 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -194,6 +194,18 @@ jobs: - **HUD coverage**: Number of downstream backends with nightly results visible on `hud.pytorch.org/crcr`. - **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD. +## Replay & Recovery + +Nightly/periodic pipelines are **idempotent by design**: the `delivery_id` is the upstream commit SHA, and the callback upserts into DynamoDB, so re-running the same workflow for the same SHA is safe and produces no duplicates. + +**Manual replay procedure** (Option 1 — adopted for initial launch): + +1. Navigate to the downstream repo's Actions tab (e.g., `TorchedHat/pytorch-redhat-ci` → Actions → "CRCR Nightly"). +2. Click "Run workflow" (`workflow_dispatch` trigger is already enabled). +3. The workflow fetches the current `nightly` branch HEAD SHA, runs CI, and reports results via the callback action — identical to a cron-triggered run. + +No centralized replay endpoint is needed at this stage. Automated missed-nightly detection (self-healing re-triggers) may be considered in a future iteration based on WG feedback. + ### File Changes | File | Change | From a9f7a93d59d978ba438a5ea457d2fb79ecfac050 Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 4 Aug 2026 19:53:02 +0530 Subject: [PATCH 10/12] Add Multi-CI Provider Authentication section to RFC-0056 Document the multi-issuer OIDC design for supporting Buildkite and other CI providers alongside GitHub Actions. Uses vLLM as a concrete example. Links to tracking issue pytorch/test-infra#8326. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 70 ++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 5b1a7fc2..c4047d89 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -206,6 +206,76 @@ Nightly/periodic pipelines are **idempotent by design**: the `delivery_id` is th No centralized replay endpoint is needed at this stage. Automated missed-nightly detection (self-healing re-triggers) may be considered in a future iteration based on WG feedback. +## Multi-CI Provider Authentication + +The self-report model currently relies on GitHub Actions OIDC tokens for caller identity. To support downstream backends that run CI on other platforms (e.g., Buildkite, GitLab CI), the relay's `jwt_helper` needs to become multi-issuer. + +### Problem + +`jwt_helper.verify_oidc_token()` is hardcoded to validate tokens from a single issuer (`https://token.actions.githubusercontent.com`). Backends running on Buildkite (e.g., vLLM at `vllm-project/vllm`) cannot authenticate to the callback Lambda because their OIDC tokens come from a different issuer (`https://agent.buildkite.com`). + +### Design: Issuer-Based Dispatch + +Add a registry of trusted issuers, each with its own JWKS endpoint and claim-extraction logic. The `verify_oidc_token` function becomes: + +1. Strip `Bearer ` prefix (existing) +2. Decode the JWT header **unverified** to read the `iss` claim +3. Look up the issuer in the registry → reject unknown issuers with 401 +4. Fetch the signing key from the issuer-specific JWKS endpoint +5. Verify the signature, audience (`pytorch-cross-repo-ci-relay`), and issuer +6. Extract `verified_repo` using the issuer-specific claim mapper + +```python +_ISSUERS = { + "https://token.actions.githubusercontent.com": { + "jwks": "https://token.actions.githubusercontent.com/.well-known/jwks", + "repo_claim": lambda claims: claims["repository"], + }, + "https://agent.buildkite.com": { + "jwks": "https://agent.buildkite.com/.well-known/jwks", + "repo_claim": lambda claims: _buildkite_to_repo( + claims["organization_slug"], claims["pipeline_slug"] + ), + }, +} +``` + +### Buildkite Identity Mapping + +Buildkite OIDC tokens have no `repository` claim. They provide `organization_slug` and `pipeline_slug`. A static mapping resolves these to a GitHub `owner/repo`: + +```python +_BUILDKITE_REPO_MAP = { + ("vllm", "vllm-ci"): "vllm-project/vllm", + # Add more as backends onboard +} +``` + +The mapping is maintained in `jwt_helper.py` so that identity resolution is complete before the callback handler runs. Everything downstream of `verified_repo` (callback handler, allowlist, Redis, HUD) is unchanged. + +### CI Provider Reference + +| CI Engine | Issuer (`iss`) | JWKS Endpoint | Repo Claim | +|-----------|---------------|---------------|------------| +| GitHub Actions | `https://token.actions.githubusercontent.com` | `.../.well-known/jwks` | `claims["repository"]` | +| Buildkite | `https://agent.buildkite.com` | `.../.well-known/jwks` | `(organization_slug, pipeline_slug)` → static map | +| GitLab CI | `https://gitlab.com` | `.../-/oauth/discovery/keys` | Future — `claims["project_path"]` | +| CircleCI | `https://oidc.circleci.com/org/` | `.../.well-known/jwks.json` | Future — org-specific mapping | + +Only GitHub Actions and Buildkite are in scope for initial implementation. GitLab and CircleCI can be added later by extending the `_ISSUERS` registry. + +### Implementation Scope + +| Component | Change | LOC | +|-----------|--------|-----| +| `utils/jwt_helper.py` | Multi-issuer dispatch + Buildkite mapping | ~45 | +| `callback/lambda_function.py` | None — interface unchanged | 0 | +| `callback/callback_handler.py` | None | 0 | +| Tests | 3 new test cases (Buildkite verify, unknown pipeline, unknown issuer) | ~15 | +| **Total** | | **~60** | + +Tracking issue: [pytorch/test-infra#8326](https://github.com/pytorch/test-infra/issues/8326) + ### File Changes | File | Change | From 5fd44c7d4f2fc8a80c01ac4971a7cc7096589d00 Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 25 Aug 2026 20:55:08 +0530 Subject: [PATCH 11/12] Address review feedback: metrics, SHA caveat, multi-CI provider MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Metrics: Replace unmeasurable "completion rate" with observable metrics (callbacks/day, time since last callback). Add explicit note that relay cannot know downstream schedules. 2. SHA validation: Add security boundary statement — SHA existence proves the commit is real, not that CI ran against it. Trust boundary is OIDC identity + allowlist. 3. Multi-CI Provider Authentication: Rewrite to reflect shipped code: - UUIDs (organization_id/pipeline_id) instead of slugs - External ci_providers.yml config instead of hardcoded map - required_claims for branch pinning (fork PR protection) - Correct pipeline naming - Reference actual PRs (#8453, #8468) 4. Update status from Draft to Implemented. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 83 +++++++++++++++------------- 1 file changed, 45 insertions(+), 38 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index c4047d89..4823367c 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -5,7 +5,7 @@ * @jewelkm89 * @subinz1 -**Status:** Draft — for CRCR Working Group discussion +**Status:** Implemented — all phases shipped and live on HUD **Date:** June 2026 @@ -126,6 +126,8 @@ Direct upsert to DynamoDB (no Redis, no state machine) DynamoDB → ClickHouse replicator → HUD ``` +**Security boundary:** SHA validation proves the commit exists on `pytorch/pytorch`, but does not prove that the downstream repo actually ran CI against it. This is inherent to self-reporting and is an accepted trust trade-off — the OIDC token establishes *who* is reporting, and the allowlist controls *which* repos are trusted to self-report truthfully. + ### Example: Downstream Nightly Workflow ```yaml @@ -190,9 +192,12 @@ jobs: ## Metrics -- **Nightly callback completion rate**: Percentage of expected nightly runs (per downstream cron schedule) that successfully report results via callback. +- **Callbacks received per backend per day**: Count of nightly/periodic callback payloads ingested per downstream repo per 24h window. Observable from DynamoDB/ClickHouse without knowledge of downstream schedules. +- **Time since last callback**: Per-backend staleness indicator — if the relay hasn't received a nightly callback from a registered backend in >36 hours, the health card on HUD marks it as degraded. - **HUD coverage**: Number of downstream backends with nightly results visible on `hud.pytorch.org/crcr`. -- **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD. +- **Time-to-detection**: How quickly a nightly regression in a downstream backend is surfaced on HUD (measured from cron trigger to HUD row appearing). + +> **Note:** The relay has no knowledge of a downstream repo's cron schedule, so "expected runs" is undefined under the self-report model. Staleness detection (time since last callback) serves as the practical proxy for missed runs. ## Replay & Recovery @@ -208,22 +213,20 @@ No centralized replay endpoint is needed at this stage. Automated missed-nightly ## Multi-CI Provider Authentication -The self-report model currently relies on GitHub Actions OIDC tokens for caller identity. To support downstream backends that run CI on other platforms (e.g., Buildkite, GitLab CI), the relay's `jwt_helper` needs to become multi-issuer. - -### Problem +The self-report model currently relies on GitHub Actions OIDC tokens for caller identity. To support downstream backends that run CI on other platforms (e.g., Buildkite, GitLab CI), the relay's `jwt_helper` has been extended to support multiple issuers. -`jwt_helper.verify_oidc_token()` is hardcoded to validate tokens from a single issuer (`https://token.actions.githubusercontent.com`). Backends running on Buildkite (e.g., vLLM at `vllm-project/vllm`) cannot authenticate to the callback Lambda because their OIDC tokens come from a different issuer (`https://agent.buildkite.com`). +### Shipped Design: Issuer-Based Dispatch with External Config -### Design: Issuer-Based Dispatch +> **Status:** Implemented in [pytorch/test-infra#8453](https://github.com/pytorch/test-infra/pull/8453) (multi-issuer OIDC) and [pytorch/test-infra#8468](https://github.com/pytorch/test-infra/pull/8468) (externalized config). -Add a registry of trusted issuers, each with its own JWKS endpoint and claim-extraction logic. The `verify_oidc_token` function becomes: +The `verify_oidc_token` function: -1. Strip `Bearer ` prefix (existing) -2. Decode the JWT header **unverified** to read the `iss` claim -3. Look up the issuer in the registry → reject unknown issuers with 401 -4. Fetch the signing key from the issuer-specific JWKS endpoint -5. Verify the signature, audience (`pytorch-cross-repo-ci-relay`), and issuer -6. Extract `verified_repo` using the issuer-specific claim mapper +1. Strips `Bearer ` prefix +2. Decodes the JWT header **unverified** to read the `iss` claim +3. Looks up the issuer in `_ISSUERS` → rejects unknown issuers with 401 +4. Fetches the signing key from the issuer-specific JWKS endpoint +5. Verifies the signature, audience (`pytorch-cross-repo-ci-relay`), and issuer +6. Extracts `verified_repo` using the issuer-specific claim mapper ```python _ISSUERS = { @@ -233,46 +236,48 @@ _ISSUERS = { }, "https://agent.buildkite.com": { "jwks": "https://agent.buildkite.com/.well-known/jwks", - "repo_claim": lambda claims: _buildkite_to_repo( - claims["organization_slug"], claims["pipeline_slug"] - ), + "repo_claim": lambda claims: _buildkite_to_repo(claims), }, } ``` ### Buildkite Identity Mapping -Buildkite OIDC tokens have no `repository` claim. They provide `organization_slug` and `pipeline_slug`. A static mapping resolves these to a GitHub `owner/repo`: +Buildkite OIDC tokens have no `repository` claim. They provide `organization_id` and `pipeline_id` (immutable UUIDs). The mapping from these UUIDs to a GitHub `owner/repo` is maintained in an external config file (`config/ci_providers.yml`), loaded at runtime: -```python -_BUILDKITE_REPO_MAP = { - ("vllm", "vllm-ci"): "vllm-project/vllm", - # Add more as backends onboard -} +```yaml +# config/ci_providers.yml +buildkite: + - organization_id: "a1b2c3d4-..." # vllm org UUID + pipeline_id: "e5f6g7h8-..." # ci pipeline UUID + github_repo: "vllm-project/vllm" + required_claims: + build_branch: ["main"] # only trust tokens from main branch builds ``` -The mapping is maintained in `jwt_helper.py` so that identity resolution is complete before the callback handler runs. Everything downstream of `verified_repo` (callback handler, allowlist, Redis, HUD) is unchanged. +**Key design decisions:** +- Keyed on immutable **UUIDs** (`organization_id` / `pipeline_id`), not slugs. Slugs are renameable — a released slug can be claimed by another org, which would be a privilege escalation. +- `required_claims` constrains which builds can authenticate. For vLLM, this pins to `build_branch: [main]` because the pipeline builds fork PRs — any job in a fork PR build can mint a token, so without branch pinning any fork could impersonate the vLLM backend. +- Config is loaded at Lambda startup, cached in-process. ### CI Provider Reference | CI Engine | Issuer (`iss`) | JWKS Endpoint | Repo Claim | |-----------|---------------|---------------|------------| | GitHub Actions | `https://token.actions.githubusercontent.com` | `.../.well-known/jwks` | `claims["repository"]` | -| Buildkite | `https://agent.buildkite.com` | `.../.well-known/jwks` | `(organization_slug, pipeline_slug)` → static map | +| Buildkite | `https://agent.buildkite.com` | `.../.well-known/jwks` | UUID lookup from `ci_providers.yml` | | GitLab CI | `https://gitlab.com` | `.../-/oauth/discovery/keys` | Future — `claims["project_path"]` | | CircleCI | `https://oidc.circleci.com/org/` | `.../.well-known/jwks.json` | Future — org-specific mapping | -Only GitHub Actions and Buildkite are in scope for initial implementation. GitLab and CircleCI can be added later by extending the `_ISSUERS` registry. +Only GitHub Actions and Buildkite are implemented. GitLab and CircleCI can be added by extending the `_ISSUERS` registry and `ci_providers.yml`. -### Implementation Scope +### Implementation (shipped) -| Component | Change | LOC | +| Component | Change | PR | |-----------|--------|-----| -| `utils/jwt_helper.py` | Multi-issuer dispatch + Buildkite mapping | ~45 | -| `callback/lambda_function.py` | None — interface unchanged | 0 | -| `callback/callback_handler.py` | None | 0 | -| Tests | 3 new test cases (Buildkite verify, unknown pipeline, unknown issuer) | ~15 | -| **Total** | | **~60** | +| `utils/jwt_helper.py` | Multi-issuer dispatch + Buildkite UUID mapping | [#8453](https://github.com/pytorch/test-infra/pull/8453) | +| `config/ci_providers.yml` | Externalized provider config with `required_claims` | [#8468](https://github.com/pytorch/test-infra/pull/8468) | +| Tests | Buildkite verify, unknown pipeline, unknown issuer, branch pinning | included in #8453 | Tracking issue: [pytorch/test-infra#8326](https://github.com/pytorch/test-infra/issues/8326) @@ -304,16 +309,18 @@ Both options were set aside in favor of the self-report model because they requi ## Resolution -TBD — pending WG discussion. +Implemented — all phases shipped and operational. ### Level of Support -TBD +Accepted — adopted by CRCR Working Group. Nightly CI is live for `pytorch/crcr-test` and `TorchedHat/pytorch-redhat-ci`. Buildkite OIDC onboarded for `vllm-project/vllm`. ### Next Steps -TBD +- Onboard additional downstream backends requesting nightly reporting +- Consider automated staleness alerting (>36h without callback → degraded health) +- Extend `ci_providers.yml` for GitLab CI providers when demand arises #### Tracking Issue -TBD +[pytorch/test-infra#8326](https://github.com/pytorch/test-infra/issues/8326) From 9b650273dd6411f6490afc0335be098b66755b29 Mon Sep 17 00:00:00 2001 From: Subin George Date: Tue, 25 Aug 2026 20:58:58 +0530 Subject: [PATCH 12/12] Make Previously Considered Options definitive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The self-report model is shipped — update wording from "evaluated before arriving at" to explicitly state the decision: Option A and B rejected, self-report adopted and implemented. --- RFC-0056-CRCR-Nightly-Periodic-CI.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RFC-0056-CRCR-Nightly-Periodic-CI.md b/RFC-0056-CRCR-Nightly-Periodic-CI.md index 4823367c..03893879 100644 --- a/RFC-0056-CRCR-Nightly-Periodic-CI.md +++ b/RFC-0056-CRCR-Nightly-Periodic-CI.md @@ -293,13 +293,13 @@ Tracking issue: [pytorch/test-infra#8326](https://github.com/pytorch/test-infra/ ## Previously Considered Options -Two alternative approaches were evaluated before arriving at the authenticated self-report design: +Two alternative approaches were evaluated. The authenticated self-report model (described above) was selected and shipped. **Option A: EventBridge Cron → Webhook Lambda.** An AWS EventBridge rule on a cron schedule invokes the webhook Lambda directly. The Lambda fetches `pytorch/pytorch` main HEAD SHA, builds a synthetic `client_payload`, and dispatches to downstream repos via the existing `_dispatch_to_allowlist()` path. This preserves the full state machine and guarantees SHA alignment across all backends. However, it introduces new AWS infrastructure (EventBridge rule, Terraform config, CloudWatch alarms) and centralizes schedule control — downstream repos cannot customize their own cron timing without additional EventBridge rules. **Option B: Upstream Cron Workflow in pytorch/pytorch → Webhook Lambda.** A `schedule: cron` workflow in `pytorch/pytorch` constructs a synthetic payload and POSTs it to the webhook Lambda endpoint with OIDC authentication. This gives upstream visibility (schedule appears in the Actions tab) and built-in manual re-trigger via `workflow_dispatch`. However, it requires adding a second authentication path (OIDC or shared secret) to the webhook Lambda, changes to `pytorch/pytorch` requiring maintainer approval, and depends on GitHub cron reliability. -Both options were set aside in favor of the self-report model because they require either new AWS infrastructure or upstream repo changes, while the proposed design keeps all changes within the callback Lambda and downstream repos. +**Decision:** Both options were rejected. The self-report model was adopted because it requires no new AWS infrastructure, no upstream repo changes, and gives each downstream repo full control over its own schedule. This is now implemented and live. ## Prior Art