ci: split local merge-gate E2E from staging nightly/release

[piotr-iohk]

Goal

Split Bitkit E2E CI so:

1. Local docker/regtest is the PR merge gate (self-contained app/UI signal).
2. Staging (Blocktank/pubky) runs nightly, on release-* PRs, and via workflow_dispatch.
3. The required GitHub check e2e-status keeps working (same job name, still reported on every non-draft PR).

This is an implementation ticket for an agent. Work spans this repo plus bitkit-ios and bitkit-android. Do not merge skip_local on the existing workflow.

Why

Infra must not red every PR

On 10 Sep 2026, Blocktank staging (api.stag0.blocktank.to) stopped marking transfer-to-spending orders paid (state: created, paidSat: 0) after an ln2-watcher crash and a ~2k unpaid-invoice backlog. Same class of stag0 failure as Feb 23 and Aug 8 (RabbitMQ / watcher). That reded e2e-status on all bitkit-android and bitkit-ios PRs even when the app was fine.

Staging shards that complete a real LSP path (@multi_address_2, transfer-to-spending) belong off the merge gate so stag0 stays usable for the real LSP path. Migration workflows are already nightly — leave them.

Local is not actually self-contained today

BACKEND=local only switches Electrum/mining to docker. Both apps still hardcode Blocktank to stag0 on regtest:

• iOS Bitkit/Constants/Env.swift → blocktankBaseUrl default https://api.stag0.blocktank.to
• Android app/src/main/java/to/bitkit/env/Env.kt → same

Local @transfer_1 / @transfer_max (and HW transfer) create Blocktank orders, then pay a deposit address on the docker chain. Those orders never go paid. Hardware wallet already documents this:

// test/specs/hardware-wallet.e2e.ts (@hardware_wallet_3)
if (getBackend() === 'regtest') {
  // We only check balance on staging regtest backend, local does not have Blocktank.
  await expectSpendingBalance(0, { condition: 'gt', timeout: 60_000 });
}

That unpaid-order leak is a plausible source of the 2k backlog, not just “three staging transfers per PR.” Moving staging off PRs without skipping these local cases only solves half the incident.

Related, not blocking: #34 (Blocktank in local docker) would make local truly self-contained later. Do not wait on it.

Better signal for agentic PR volume

A lot of recent Bitkit work is agent-driven: many small PRs, parallel branches, retries on red CI. That only works if the merge gate is a product signal, not a shared-infra weather report.

• Staging-on-PR couples every agent PR to stag0 + GitHub-hosted Macs. One watcher crash makes every agent treat infra as an app bug, re-push, and pile more unpaid orders.
• Local-on-PR (docker Electrum + local LND) is the right gate for “did this change break the app?”
• Staging nightly / release / dispatch is the right gate for “does the real LSP/pubky path still work?”

Keep e2e-status as that local signal so agents and humans can trust a green merge box.

Current shape

Workflow	Repo	Trigger	What it runs
e2e-tests.yml	bitkit-ios	PR + dispatch	Local (self-hosted Mac Mini, 1 shard) and staging (macos-latest: @multi_address_2, @pubky)
e2e.yml	bitkit-android	PR + dispatch	Local (ubuntu-latest × 7) and staging (@multi_address_2, @pubky, @hardware_wallet)
e2e_migration.yml	both	cron 02:00/03:00 UTC + dispatch	Unchanged — leave it
e2e-status job	both app workflows	every non-draft PR	ANDs local and staging; this name is the required ruleset check

Default-branch rulesets:

• iOS: required e2e-status, Run Integration Tests, Run Tests
• Android: required e2e-status, build

GitHub matches the check name e2e-status. Renaming it, not emitting it on some PRs, or leaving staging in needs: after those jobs stop running, will block merges.

Do not merge synonymdev/bitkit-ios#744 (skip_local dispatch input). After this split, gh workflow run e2e-staging is the staging poke and never queues the Mini. Close #744 when the new workflow exists.

Target shape — separate workflows

Keep existing e2e-tests.yml / e2e.yml as the local merge gate.

Add e2e-staging.yml on each app repo (nightly + dispatch + release PRs).

Leave e2e_migration.yml alone. Stagger staging nightly after migration (suggest ~04:00 UTC) so stag0 is not hit by restore wallets and LSP transfers in the same hour.

Trigger	Local docker	Staging matrix	Status check
PR → master (feature)	Yes	No	e2e-status = local only
PR head or base release-*	Yes	Yes	e2e-status + e2e-staging (see skip-success below)
Nightly cron ~04:00 UTC on default branch	No	Yes	Slack on failure; not a PR check
workflow_dispatch on local workflow (e2e.yml / e2e-tests.yml)	Yes	No	Informational (does not satisfy the PR e2e-status check)
workflow_dispatch on staging workflow (e2e-staging.yml)	No	Yes	Informational + Slack #bitkit-staging-nightly
Migration crons	Unchanged	Unchanged	Unchanged

Staging matrix (move as-is, then align HW):

• @multi_address_2
• @pubky (paykit/profile — pubky infra, not Blocktank, but still not a merge gate)
• Android @hardware_wallet full path (channel assert). iOS HW currently runs on the Mini and skips spending-balance; after the split, put HW transfer completion on staging for both platforms; keep connect/receive/send-onchain on local where Trezor emulator works.

Hard constraint: do not break e2e-status

In the existing local workflow, keep a job whose name: is exactly e2e-status.

It must still run on every non-draft PR, including the path-filter skip path that currently succeeds with “no code changes.”

Change the job, not the name:

e2e-status:
  if: always() && github.event.pull_request.draft == false
  name: e2e-status
  needs: [detect-changes, e2e-tests-local]   # drop e2e-tests-staging
  # verify only needs.e2e-tests-local.result == success when code == true

If staging stays in needs: after those jobs are removed or skipped on feature PRs, needs.*.result is skipped and the aggregator fails — that is the usual way this check breaks.

Do not add e2e-staging as a required ruleset context unless that workflow also runs on every PR and skip-succeeds on feature PRs. A missing required check blocks merge the same way a red one does.

Recommended skip-success pattern for a new e2e-staging job (required only if we decide to enforce staging on release PRs via GitHub):

• Feature PRs: job runs, exits 0, prints “skipped — not a release PR.”
• release-* head or base: run the real staging matrix; fail the job if shards fail.

v1: implement that for release-* only. A needs-staging-e2e label is v2 (pull_request types must include labeled). Dispatch is enough for LSP/pubky feature PRs in v1.

Do not implement “green nightly within N hours of cut” as a GitHub required check. Nightly is master SHA, not the release SHA.

Implementation order

GitHub constraint: workflow_dispatch only exists for workflow files already on the default branch. A brand-new e2e-staging.yml cannot be “Run workflow”’d until it is merged. It can run on the PR that introduces it if that file has on: pull_request (same-repo PRs; the workflow from the PR head is used).

Do not ship skip-success on the first staging PR or that PR will no-op and you will not have tested the matrix.

Stacked PRs, per app repo (bitkit-ios, bitkit-android in parallel):

PR A — add e2e-staging.yml only (merge first)

• New file: schedule + workflow_dispatch (e2e_branch only) + pull_request.
• Always run the real staging matrix on this PR (no release-* skip yet). Existing e2e.yml / e2e-tests.yml stays unchanged, so the merge gate is untouched. This PR will run staging twice (old job + new workflow); that is expected and is the test.
• Staging grep includes @transfer_1|@transfer_max plus current @multi_address_2, @pubky, and Android @hardware_wallet as specified above.
• Slack to #bitkit-staging-nightly via secrets.SLACK_WEBHOOK_URL_STAGING (already set).
• After merge: gh workflow run e2e-staging.yml --ref <branch> works for pokes.

PR B — stacked on A (or on master after A merges) — switch the merge gate

• Strip build-staging / e2e-tests-staging from the existing local workflow.
• e2e-status needs local only (same job name:).
• On e2e-staging.yml, add skip-success for feature PRs; real run on release-* head/base, schedule, and dispatch. Feature PRs must still emit the e2e-staging check (green skip) so a later required context cannot go missing. Do not add it to the ruleset in v1 unless product wants it required on release PRs only via that skip-success job.
• Do not add suite= / skip_local / combined full mode.

This repo — skip local LSP orders, with or just after A

• Spec-level skip when BACKEND=local for @transfer_1 / @transfer_max. Safe on staging (BACKEND=regtest still runs them).
• Do not merge this skip before PR A is live, or those tests disappear from CI until staging exists.
• Keep @transfer_2 on local grep; stop grepping @transfer as a blob.

Then

• Close ios#744 without merge (dispatch e2e-staging.yml instead).
• Confirm one scheduled or dispatched run posted to #bitkit-staging-nightly.

Later (out of v1)

• needs-staging-e2e label
• iOS local sharding (one Mini: sharding does not shorten a single PR)
• GitHub-hosted iOS sim + remote Linux docker (separate track)
• Include Blocktank in Regtest docker-compose #34 local Blocktank in docker

Acceptance criteria

• Feature PRs: required e2e-status still appears and goes green when local passes (or path-filter skip). Staging infra down does not red that check.
• e2e-status job name: unchanged. Ruleset does not need a rename.
• Staging lives in a new workflow file, not a skip_local flag on the old one.
• BACKEND=local no longer creates unpaid Blocktank orders. @transfer_1 / @transfer_max run on staging (e2e-staging.yml shard); @transfer_2 stays local-only.
• Nightly staging runs on master; posts to #bitkit-staging-nightly via secrets.SLACK_WEBHOOK_URL_STAGING (already set on ios + android).
• workflow_dispatch on the new workflow can poke staging without queuing iOS local Mini.
• release-* PRs still get a staging signal (skip-success job or explicit required check — implement option A).
• Migration crons unchanged.
• ios#744 closed without merge.
• AGENTS.md / README in this repo updated for which tags are local vs staging.

Context

• Slack (10 Sep, stag0 down, Denys volume discussion): https://synonymworkspace.slack.com/archives/C03UWHGKUNP/p1789031337183729
• ln2 follow-up: https://github.com/synonymdev/blocktank-lsp-ln2/issues/277
• Do not merge: feat(ci): add skip_local input to skip local E2E jobs on workflow_dispatch bitkit-ios#744
• Local Blocktank (follow-up, not this ticket): Include Blocktank in Regtest docker-compose #34

Agent notes

• Implement in this order: PR A (e2e-staging.yml on ios + android, self-test via pull_request) → merge A → spec skip in this repo → PR B (strip staging from local workflow, skip-success on feature PRs) → close ios#744.
• Touch app repos in PRs of their own; do not try to change ios/android files from this repo.
• Do not edit GitHub rulesets unless e2e-status would stop reporting; the intended change is job internals, not the required context name.
• Do not add “Made with Cursor” or similar attribution in PRs/issues.
• English, conventional commits (ci:, test:). No changelog fragments unless a workflow change is user-facing (it is not).