[VC-59470] Run the e2e suites against master every night

[wallrj-cyberark]

The end-to-end suites now run against master every night at 02:00 UTC. This is the last substantive piece of the ticket that #833 and #837 began.

Why now?

Nothing runs the suites on master. #833 added a Run workflow button and #837 made the pull request labels work, but both need a person. A green master therefore still says nothing about whether the e2e tests pass.

The three manual runs since 2026-09-16 show exactly what we have been missing:

Dispatch	ark-test-e2e	ngts-test-e2e	test-e2e
09-16	success	success	failure
09-17	success	success	failure
09-18	success	success	failure

Please expect the first nights to be red

test-e2e has failed on every run for three days. It is not this repository: the cause is CIS dropping async Kafka messages, which is being worked on separately and is still open. We decided to schedule all three suites anyway rather than hold test-e2e back, because a suite that is genuinely broken should look broken.

Do not read the first red nightly as a regression from this pull request.

The same table is the first real evidence that the flake behind the TEMPORARY comment on ark-test-e2e has gone — three for three on master. That comment is now stale, but removing the label gate is a separate decision and not part of this change.

What changes

• A schedule trigger on e2e.yaml, 0 2 * * *. Midnight is already govulncheck, and 02:00 UTC puts the result in before the working day.
• github.event_name == 'workflow_dispatch' becomes github.event_name != 'pull_request' in the three job gates. Outside a pull request there are no labels to read, so everything should run; this covers the schedule and the button together. Pull request behaviour is untouched.
• A github.repository check on each gate, copying govulncheck.yaml, so the nightly does not run in forks.
• A notify job, nightly only.

About the Slack notification

SLACK_WEBHOOK does not exist on this repository yet. Until it is added the job writes the failure to the run summary and raises a warning annotation, so a missing secret cannot turn a passing nightly red. Add the secret and it starts posting with no further change.

It deliberately only fires on the schedule. A suite that fails on a labelled pull request is already in front of whoever added the label, and a Slack message for that would be noise. It posts once per run rather than once per suite, so a bad night is one notification, and names which suites failed.

Gate behaviour across every event, including forks and the notify job

Event	Suites that run	Slack
nightly schedule, all pass	all three	no
nightly schedule, test-e2e fails	all three	yes
workflow_dispatch	all three	no
PR opened, no labels	nothing	no
PR synchronize, has test-e2e	test-e2e	no
label test-ngts added	ngts-test-e2e	no
label kind/cleanup added	nothing	no
a suite fails on a labelled PR	that suite	no
nightly in a fork	nothing	no

How this was tested

The schedule itself cannot be proved before merge, since schedule only fires from the default branch. What I did check:

• Both workflows parse, and no if: expression contains a stray newline from YAML folding.
• The gate logic was walked through every row of the table above, including the fork case and the notify condition.
• The notify script was run locally in both states. With SLACK_WEBHOOK empty it writes the summary line, emits the warning and exits 0. With it set it builds the payload and posts; jq is used to build the JSON so the colons, angle brackets and em dash in the message cannot break it.
• The cleanup step's keep-e2e-cluster gate reads github.event.pull_request.labels, which is empty on a schedule, so the negation holds and the nightly still deletes its GKE cluster.

After merge I will watch the first nightly and report what it does.

[with Claude]

[wallrj-cyberark]

Self-review of this branch turned up four things worth fixing, now applied in eca2a5b. Two more I have left alone and explained.

Fixed

1. contains(needs.*.result, 'failure') missed cancelled. needs.<job>.result is one of success, failure, cancelled or skipped. A nightly that is stopped rather than failed would have left the notify job skipped, so nobody would hear about a nightly that never finished — the exact blind spot this pull request exists to close. The condition now matches cancelled too, and uses !cancelled() rather than always() so that cancelling the whole run does not make it announce its own cancellation.

2. The notify job had no timeout, and curl had no --max-time. A Slack endpoint that accepts the connection and then stalls would have held a runner for the default six hours. Now timeout-minutes: 5 on the job and --max-time 30 on the curl. The repository already treats this as real: test-e2e carries a backstop comment for the same reason, and hack/e2e/test.sh uses --max-time in its polling loop.

3. The run link in the job summary was not clickable. I was writing the Slack mrkdwn string verbatim into $GITHUB_STEP_SUMMARY, which is Markdown. <url|View run> is not Markdown and renders as literal text. That matters today rather than cosmetically: while SLACK_WEBHOOK is unset, the summary is the only place the failure is recorded. The summary now uses [View run](url) and the Slack payload keeps mrkdwn, both built from one shared results string.

4. The CONTRIBUTING.md wording was backwards. "A failure there is not necessarily caused by your change" read as a claim about failures on master, where your change is not present. It now says: if a suite fails on your pull request, check the most recent nightly first, because master may already be broken.

Not fixed, deliberately

5. test-e2e leaks two Venafi service accounts per run. hack/e2e/test.sh creates venafi-kubernetes-agent-e2e-registry-${RANDOM} and venafi-kubernetes-agent-e2e-agent-${RANDOM} in the shared tenant, and nothing deletes them. Both guards are ineffective: the registry one checks for a Kubernetes Secret in a cluster that is recreated every run, and the agent one matches on the cluster's OIDC issuer URL, which is unique per cluster. The GKE cluster is cleaned up; the service accounts are tenant-scoped and outlive it.

This is pre-existing, but this pull request makes it worse — two per night, indefinitely, rather than two per labelled run. ${RANDOM} is 15-bit, so as the leftovers accumulate a duplicate name becomes likely, and the create runs under --fail-with-body with set -o errexit, so a clash would fail the nightly and look like a test regression.

Out of scope for a scheduling change, so it is tracked separately with a suggested cleanup step or reaper.

6. The github.repository guard also covers workflow_dispatch. Pressing Run workflow in a fork now gives an all-green run in which every job skipped, which could be mistaken for a pass. The secrets do not exist in a fork so such a run would have failed anyway, and silently skipping is arguably better than a confusing failure. Flagging it so it is a deliberate choice rather than a side effect of copying govulncheck.yaml. Happy to narrow the guard to schedule only if you would rather.

Checked and cleared

• && binds tighter than || in GitHub expressions, so repo && (notPR || labelIs<X> || (notLabeled && containsLabel)) groups as intended. Walked against all four pull request activity types, workflow_dispatch and schedule.
• On a schedule event the object filter over a null pull_request yields an empty array, so the keep-e2e-cluster negation holds and the nightly still deletes its GKE cluster. No leak there.
• The notify job needs no github.repository guard of its own: in a fork all three suites are skipped, so the failure test is false anyway.
• jq is preinstalled on ubuntu-latest. The em dash, colons and angle brackets survive the jq --arg round trip, and I ran the script in both the webhook-set and webhook-unset states.