Skip to content

Add opt-in upload-failure issue reporting to the action, and global watch for stale nightly - #172

Closed
Carreau wants to merge 3 commits into
scientific-python:mainfrom
Carreau:nightly-failure-reporting
Closed

Add opt-in upload-failure issue reporting to the action, and global watch for stale nightly#172
Carreau wants to merge 3 commits into
scientific-python:mainfrom
Carreau:nightly-failure-reporting

Conversation

@Carreau

@Carreau Carreau commented Jul 24, 2026

Copy link
Copy Markdown
Member

AI GENERATED, NOT REVIEWED IN DEPTH, MOSTLY FOR DISCUSSION in #171

(This is based on an old commit as I was not on my computer, but I'll ask Claude to redo on main, but the principle is the same)

--
Adds two capabilities to the nightly upload workflow, plus lint CI:

  1. Opt-in upload-failure reporting in the action itself: when an upload fails, open a tracking issue on the calling repository and close it automatically on the next success.
  2. A channel-freshness monitor on this repo: a scheduled job that files issues when packages stop receiving nightly uploads, escalating from this coordination repo to the projects' own repos.

Both are implemented as small Python scripts run from GitHub Actions. No runtime dependency is added for consumers of the action.

1. Failure reporting (report_failures)

Extra composite steps wrap the existing pixi upload step:

jobs:
  upload:
    permissions:
      issues: write        # required by report_failures
      contents: read       # only if the job also checks out to build wheels
    steps:
      - uses: scientific-python/upload-nightly-action@<sha>
        with:
          artifacts_path: dist
          anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
          report_failures: true
  • On failure: opens (or reuses) a nightly-upload-failure issue on the calling repo, then fails the job.
  • On the next success: comments and closes it. Idempotent (found by label, not duplicated).
  • Uses the built-in github.token by default (no new secret). Logic in scripts/report_failure.py, run via the action's pixi Python. It uses requests, which is already in the pixi lock (a dependency of anaconda-client), so consumers gain no new dependency.
  • New inputs: report_failures, github_token (defaults to ${{ github.token }}), issue_repository.

2. Channel-freshness monitor (.github/workflows/monitor-nightly.yml)

A daily job queries the anaconda.org API for the scientific-python-nightly-wheels channel and files issues on this repo:

  • > 30 days without an upload: open a stale-nightly issue per package.
  • > 60 days: additionally open a nightly-purge-candidate issue.
  • Both auto-close when a fresh upload lands.

It complements remove-wheels.yml (which prunes old versions); this only opens/closes issues and never deletes anything. It honors packages-ignore-from-cleanup.txt (so exempt packages such as openblas-libs are not flagged), guards on github.repository_owner == 'scientific-python', and uses a least-privilege built-in token.

Notifying the producing projects directly (opt-in)

If a Personal Access Token is provided as secrets.NIGHTLY_UPLOAD_ISSUE_PAT (the default GITHUB_TOKEN cannot write issues on other repos) and the wheel appears in the hand-maintained packages-source-repos.yaml mapping:

  • > 15 days: open a nightly-upload-stalled issue on the wheel's own source repo, escalated with a comment at the 30- and 60-day marks, auto-closed on recovery.
  • If the PAT is missing/expired/unauthorized, a nightly-pat-invalid issue is opened on this repo instead (using the built-in token) so maintainers rotate the secret; auto-closed once the token works again.

Fully opt-in: with no PAT or an empty mapping, this behaviour is skipped entirely.

packages-source-repos.yaml uses a packages: map where each entry is either wheel: owner/repo or wheel: {repo: owner/repo}, leaving room for per-repo config (labels, assignees, per-repo thresholds) later without a format change. It ships with an inert placeholder entry.

Logic lives in .github/scripts/monitor_nightly.py (PyGithub + requests + PyYAML, pinned in .github/scripts/requirements.txt).

3. Lint CI

.github/workflows/lint.yml runs ruff check and ruff format --check on scripts/ and .github/scripts/ on push/PR.

Configuration required (maintainers)

  • No config needed for the 30/60-day coordination-repo behaviour.
  • To enable producer-repo notifications: add the NIGHTLY_UPLOAD_ISSUE_PAT secret (a PAT with issues: write on the target repos) and populate packages-source-repos.yaml.
  • Consumers who want failure reporting set report_failures: true and grant permissions: issues: write.

Notes

  • Runs on the current pixi-based action (v0.6.4); the failure-reporting steps wrap the existing upload step without changing it.
  • No untrusted input flows into run:/script: bodies. Package names reach only API calls; issue_repository/upload.outcome are passed via env: and read from os.environ/process.env. Actions are SHA-pinned (checkout v7.0.0, setup-python v6.3.0).
  • Only GithubException from the PAT check is treated as "PAT broken"; transient/network errors are logged without filing a false alarm.

Testing

Local validation only (no live GitHub run yet):

  • ruff check and ruff format --check pass; py_compile clean; all YAML parses.
  • Monitor smoke-tested against the live anaconda.org API with stubbed GitHub: reads the ignore list, iterates all 35 packages, writes the freshness summary.
  • Verified with fakes: producer open / 30-day escalate / 60-day escalate / recovery-close; invalid-PAT opens the self-repo issue and valid-PAT closes it; failure-report open-on-failure and close-on-success.

Note: the "Testing" section is local-only — the workflows haven't run end-to-end on GitHub yet (real issue creation, scheduled trigger, PAT auth). Trim that section if you don't want to imply live verification.

@Carreau
Carreau force-pushed the nightly-failure-reporting branch 3 times, most recently from 8b4bbc7 to 2aa8917 Compare July 25, 2026 07:03
claude and others added 3 commits July 25, 2026 09:13
With report_failures: true, a failed upload opens a tracking issue on the
calling repository (label nightly-upload-failure) and the next successful
upload closes it automatically. Idempotent: existing open issues are found
by label rather than duplicated.

Reporting is best-effort: the step is continue-on-error and the script treats
a read-only token (HTTP 401/403, e.g. on pull requests and forks) as a clean
skip, so it never fails a consumer's build. A genuine upload failure is still
surfaced by a dedicated step.

Logic lives in scripts/report_failure.py, run via the action's pixi Python. It
uses requests, already in the pixi lock (a dependency of anaconda-client), so
no dependency is added for consumers. Uses the built-in github.token by
default; consumers only need to grant permissions: issues: write. New inputs:
report_failures, github_token, issue_repository.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A daily job queries the anaconda.org API for the scientific-python-
nightly-wheels channel and files issues on this repo:

- > 30 days without an upload opens a stale-nightly issue per package
- > 60 days additionally opens a nightly-purge-candidate issue
  (both auto-closed when a fresh upload lands)

It also, optionally, notifies each wheel's own source repository:

- if a PAT is provided (secrets.NIGHTLY_UPLOAD_ISSUE_PAT) and the wheel is
  listed in packages-source-repos.yaml, a nightly-upload-stalled issue is
  opened on the producing repo at 15 days, escalated by comment at 30/60
  days, and auto-closed on recovery
- if that PAT is missing/expired/unauthorized, a nightly-pat-invalid issue
  is opened on this repo instead (auto-closed once the token works)

Complements remove-wheels.yml (prunes old versions); this only opens/closes
issues. Honors packages-ignore-from-cleanup.txt, guards on the scientific-
python owner, and uses least-privilege for the built-in token. Logic lives in
.github/scripts/monitor_nightly.py (PyGithub + requests + PyYAML, pinned in
.github/scripts/requirements.txt).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run ruff check and ruff format --check on scripts/ and .github/scripts/ on
push/PR via uvx (uv), with the ruff version pinned inline and the rule
selection pinned in ruff.toml so a new ruff release cannot change the lint
outcome unexpectedly. The lint job needs no secrets, so it is safe on forks
and pull requests. Ignore Python caches.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Carreau
Carreau force-pushed the nightly-failure-reporting branch from 2aa8917 to 29a6c80 Compare July 25, 2026 07:14
@Carreau Carreau closed this Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants