feat(bitbucket): run a hub on Bitbucket Pipelines

.github/workflows/ci.yml

@@ -283,6 +283,11 @@ jobs:
             exit 1
           fi
 
+      # Skips while the published template has no bitbucket-pipelines.yml.
+      - name: go test -run TestTemplatePipelines
+        if: steps.template.outputs.public == 'true'
+        run: TOUCHMARK_E2E_TEMPLATE="$PWD/.ci/template" go test -count=1 -v -run '^TestTemplatePipelines$' ./internal/e2e/bitbucket/
+
   release-snapshot:
     # The release without publishing anything (docs/project/release.md):
     # the scripts of the release and of the Action by their tests;

docs/agents.md

@@ -47,8 +47,8 @@ no lock file, no header, no marker in their files. It never checks a target out
 and never runs a script, hook, filter or LFS from the hub or a target. It delivers
 nothing to a repository without the opt-in file, unless `targets.yml` subscribes it
 (`opt_in: assumed`). There is no pull mode (targets do not
-fetch from the hub), no remote pack sources, no Azure DevOps driver, and no hub on
-Bitbucket Pipelines: Bitbucket Cloud is a provider a hub elsewhere delivers to.
+fetch from the hub), no remote pack sources, no Azure DevOps driver, and no `setup` for
+Bitbucket: a hub on Bitbucket Cloud runs in Bitbucket Pipelines, set up by hand.
 
 ## Mental model
 

docs/concepts/security.md

@@ -34,7 +34,7 @@ reviewed merges reach. `security.write_isolation` in `hub.yml` says how:
 
 | Mode | The write key | touchmark checks |
 |---|---|---|
-| `platform` (default) | in the hub's CI: GitHub's environment `touchmark-distribute` limited to the default branch; GitLab's protected, masked, hidden variable scoped to that environment | a probe in the jobs any branch can start; on GitHub also the environment's branch policy, through the hub channel |
+| `platform` (default) | in the hub's CI: GitHub's environment `touchmark-distribute` limited to the default branch; GitLab's protected, masked, hidden variable scoped to that environment; on Bitbucket Premium a secured variable of the deployment `touchmark-distribute`, which only the default branch may deploy to | a probe in the jobs any branch can start; on GitHub also the environment's branch policy, through the hub channel; on Bitbucket the hub's statement in `security.reason`, since the API does not show the deployment permissions |
 | `external` | released by an external secret store through the CI's OIDC token, bound to the default branch (GitHub `sub` = `repo:ORG/HUB:ref:refs/heads/main` or `…:environment:touchmark-distribute`; GitLab `project_path`, `ref`, `ref_type`, `ref_protected`) | the job's own context |
 | `none` | anywhere; the risk is accepted | nothing; `security.reason` is required, and every report and `doctor` show it as a warning |
 
@@ -52,10 +52,20 @@ their own workflow instead of running yours. So the probe runs where an attacker
 - **Gitea and Forgejo Actions:** every branch sees every secret, so `platform` cannot
   hold and `distribute` refuses it there. Run the hub's CI on GitHub or GitLab, or use
   `external`, or `none` with every branch protected.
+- **Bitbucket Pipelines:** `touchmark probe` runs in every pipeline, in a step without
+  the deployment, and fails if it sees a write key: a repository or workspace variable
+  reaches every branch. A deployment variable reaches any step that names the
+  deployment, in any branch's `bitbucket-pipelines.yml`, unless the environment's
+  deployment permissions (Premium) admit the default branch alone; Bitbucket's API does
+  not show them, so under `platform` `distribute` and `doctor` refuse to run unless
+  `security.reason` states them. Without Premium use `none` with a reason, or
+  `external`. See [A hub on Bitbucket Cloud](../getting-started/bitbucket.md).
 
 **The guards** of `distribute` (and of `doctor` in CI), each exit 2 with nothing written:
 the run is not on the default branch, or is a pull request event; on GitLab the ref is
-not protected or the job is not in the environment `touchmark-distribute`; the probe saw
+not protected or the job is not in the environment `touchmark-distribute`; on Bitbucket
+the step has no `deployment: touchmark-distribute`, the default branch could not be read,
+or `platform` comes without the statement in `security.reason`; the probe saw
 a write key or did not run; an operation flag was given in CI; git is older than 2.45;
 the write key belongs to another account than `hub.yml`'s `writer`. `plan` in CI exits 2
 when it sees any write key. A run on a commit that is no longer the tip of the default

docs/getting-started/bitbucket.md

@@ -0,0 +1,182 @@
+# A hub on Bitbucket Cloud
+
+A hub hosted on Bitbucket Cloud, whose CI is Bitbucket Pipelines: the template's
+`bitbucket-pipelines.yml` checks every pull request to the hub and delivers from the
+default branch. Bitbucket Data Center is not supported.
+
+!!! warning "Not verified on a live Bitbucket yet"
+    The Bitbucket support is built from Atlassian's documentation and tested against a
+    stand-in of its API, not on a live workspace. The points a first live run must
+    confirm are listed under [Assumed, not yet seen live](#assumed-not-yet-seen-live).
+
+`touchmark setup` has nothing for Bitbucket yet: the steps below are by hand.
+
+## Where the write key can live
+
+Bitbucket Pipelines gives a step the variables of a deployment environment only when the
+step names it (`deployment: touchmark-distribute`). But any branch's copy of
+`bitbucket-pipelines.yml` can name it too, so the write key stays on the default branch
+only when the environment's **deployment permissions** stop the steps of other branches
+([they pause](https://support.atlassian.com/bitbucket-cloud/docs/set-up-and-monitor-deployments/)).
+Those permissions are a **Premium** feature, and Bitbucket's API does not show them, so
+touchmark cannot check them. Choose by plan:
+
+| Plan | `hub.yml` | What keeps the write key |
+|---|---|---|
+| Premium | `write_isolation: platform` and a `reason` that states the restriction | the deployment `touchmark-distribute` admits the default branch only, and only admins or selected users deploy to it; you state it, touchmark cannot read it |
+| Free, Standard | `write_isolation: none` and a `reason` | nothing but who may push: every branch of the hub can read the write key; protect the branches and keep write access to the hub tight |
+| any | `write_isolation: external` | an external secret store that releases the key to the default branch only |
+
+```yaml
+# hub.yml of a hub on Bitbucket Premium
+version: 1
+id: acme-eng
+writer: "{3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d}"   # the writer account's UUID
+security:
+  write_isolation: platform
+  reason: "Bitbucket Premium: only admins may deploy to touchmark-distribute, from main only"
+```
+
+Under `platform` without a `reason`, which is how the template ships, `distribute` and
+`doctor` refuse to run on Bitbucket Pipelines (exit 2): a hub on Free or Standard is
+never taken for isolated. On every plan the `probe` step fails a pipeline that sees a
+write key in a repository or workspace variable.
+
+## Steps
+
+1. **Create the hub.** In your workspace choose *Create → Repository → Import
+   repository*, and give the URL of
+   [the template](https://github.com/bedrock-python/engineering-assets-template),
+   `https://github.com/bedrock-python/engineering-assets-template.git`. Keep it private
+   unless the hub may be public. Turn Pipelines on under *Repository settings →
+   Pipelines → Settings*.
+2. **Create two bot accounts**, Atlassian accounts that only touchmark uses, each with an
+   API token (Atlassian account settings → *Security → API tokens*, with scopes):
+    - the *reader*, with read access to your target repositories and the scopes
+      `read:user`, `read:workspace`, `read:repository`, `read:pullrequest` (all
+      `…:bitbucket`);
+    - the *writer*, with write access to the targets only (not admin, and no access to
+      the hub), and the reader's scopes plus `write:repository` and `write:pullrequest`.
+   A token's scopes do not narrow the repositories it reaches, the account's permissions
+   do. See [Bitbucket Cloud](../guide/providers.md#bitbucket-cloud) for the details.
+3. **Create the hub's access token.** Under *Repository settings → Security → Access
+   tokens* of the hub, create one with *Repositories: Read* and *Pull requests: Write*.
+   Pipelines gives a step no API token of its own: touchmark reads the hub's default
+   branch, its tip and visibility with this token, and keeps `plan`'s comment in the hub
+   pull request with it. It reaches the hub alone, and every pipeline of the hub sees
+   it, so give it nothing more; `Pull requests: Write` also lets it merge, so restrict
+   who may merge into the default branch (step 5).
+4. **Store the keys** under *Repository settings → Pipelines → Repository variables*
+   and *→ Deployments*, all **Secured**:
+    - repository variables: `TOUCHMARK_READ_TOKEN` (the reader's API token) and
+      `TOUCHMARK_PIPELINES_TOKEN` (the hub's access token);
+    - create the deployment environment `touchmark-distribute` and store the writer's
+      API token as its variable `TOUCHMARK_WRITE_TOKEN`, and nowhere else: not as a
+      repository or workspace variable;
+    - on Premium, open the environment's settings: allow deployments from the default
+      branch only, **and** let only admins deploy (*Only allow admins to deploy to this
+      environment*) or only named users or groups (*Allow selected users or groups to
+      deploy to this environment*). The branch alone is not enough: anyone with write
+      access to the hub can run the custom pipelines on the default branch and pass
+      them variables ([below](#what-the-pipelines-do)).
+   The hub delivers to Bitbucket Cloud with these one-provider names; with `providers`
+   in `hub.yml` they carry the provider's id (`TOUCHMARK_BB_WRITE_TOKEN`).
+5. **Protect the hub.** Under *Repository settings → Branch restrictions*, let no one
+   push to the default branch directly and only your maintainers merge into it; keep the
+   bot accounts and the hub's access token out of those lists. Bitbucket reads none of
+   the template's CODEOWNERS files: replace `@acme/hub-maintainers` in them with your
+   team, or delete them with the other platforms' CI files (`touchmark check` warns
+   about the placeholder).
+6. **Schedule the runs.** Under *Pipelines → Schedules*, create two schedules on the
+   default branch: a daily one for the custom pipeline `distribute`, and a weekly one
+   for `doctor`.
+7. **Describe your hub.** In `hub.yml` set `id`, `writer` to the writer account's UUID
+   in quotes (`GET https://api.bitbucket.org/2.0/user` with its token shows it), and
+   `security` as above. List your repositories in `targets.yml` (`org: <workspace>` with
+   `match`, or `repo: <workspace>/<repository>`; Bitbucket has no topics).
+8. **Open a pull request** and check the `probe`, `check` and `plan` steps. `plan`
+   keeps its report in one comment of the pull request, and the report files are the
+   step's artifacts.
+9. **Merge it.** The push pipeline of the default branch runs `probe`, then
+   `distribute`, which opens a pull request in every repository that has
+   [opted in](opt-in.md). *Run pipeline* runs the custom pipelines `distribute` and
+   `doctor` by hand.
+
+## What the pipelines do
+
+| Pipeline | Steps |
+|---|---|
+| pull request to the hub | `probe`, `check`, `plan --strict --comment` |
+| push to `main` or `master`, the custom pipeline `distribute` | `probe`, then `distribute` with `deployment: touchmark-distribute` |
+| the custom pipeline `doctor` | `probe`, then `doctor` with `deployment: touchmark-distribute` |
+
+- **The default branch.** Pipelines names no default branch, so touchmark reads it with
+  `TOUCHMARK_PIPELINES_TOKEN` (a public hub answers without it) and refuses
+  `distribute` and `doctor` on any other branch, or when it cannot read it. The push
+  pipeline starts on `main` and `master`: add your default branch to its pattern if it
+  has another name.
+- **The deployment.** `distribute` and `doctor` run only in a step with
+  `deployment: touchmark-distribute` (`BITBUCKET_DEPLOYMENT_ENVIRONMENT`). Bitbucket
+  runs one deployment to an environment at a time, so two runs never overlap.
+- **Time.** The `distribute` step may run 6 hours (`max-time: 360`); touchmark starts no
+  target after 5 hours 30 minutes (`--deadline 5h30m`), and the next run picks up what is
+  left.
+- **Reports.** Bitbucket has no step summary or annotations: the report is in the step's
+  log, and `touchmark-report.{md,json,jsonl}` and `touchmark-doctor.{md,json}` are the
+  steps' artifacts (kept 14 days).
+- **The plan comment** is written with the hub's access token. Bitbucket shows HTML as
+  text, so the comment has no folded sections, and its marker is a Markdown link
+  reference definition, `[touchmark-plan]: # "touchmark plan: <hub id>"`, which renders
+  as nothing. An access token cannot tell its own account, so touchmark edits the newest
+  comment with the marker that Bitbucket lets it edit, and writes a new one otherwise.
+- **The fingerprint** of the hub is `bitbucket.org/<repository UUID>`, lowercase and
+  without braces: `bitbucket.org/3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d`. `--hub-fp` and
+  `previous_fingerprints` also take the UUID as `BITBUCKET_REPO_UUID` gives it, in
+  braces.
+- **Custom pipelines** accept variables when they run: anyone with write access to the
+  hub can run them, through the website or the API, and a pipeline variable overrides a
+  deployment variable. The template's scripts reference no variable, so the website asks
+  for none, but the API takes any. touchmark's git ignores the variables that would
+  change its config or the programs it runs (`GIT_CONFIG_*`, `GIT_SSH_COMMAND`,
+  `LD_PRELOAD`, …; see the [threat model](../project/threat-model.md#tokens-and-git)),
+  but whoever runs the step chooses its whole environment. So restrict the deployment
+  `touchmark-distribute` to admins or named users, not only to the default branch: on
+  Premium, deployment permissions pause a run by anyone else.
+
+## Check it
+
+`touchmark doctor --hub-token`, run from a clone of the hub, reads where the keys live
+with the token in `TOUCHMARK_HUB_TOKEN`: a one-off access token of the hub with
+*Repositories: Read* and *Pipelines: Read*, or a maintainer's API token with
+`read:repository:bitbucket` and `read:pipeline:bitbucket` (not the pipelines' own
+token, which needs no Pipelines scope). It finds that a write key in a repository or workspace variable fails, one in the
+deployment `touchmark-distribute` is unknown (the API does not show who may deploy; it
+is `ok` when the environment admits only admins), and a variable that is not secured
+warns. Workspace variables need a workspace administrator's token; without one they are
+reported as not read.
+
+## Assumed, not yet seen live
+
+- Pipelines runs the step scripts with the image's shell, not its `touchmark`
+  entrypoint, and the image's user (uid 65532) may write the report files into the
+  clone. If not, set `run-as-user` on the steps.
+- The `plan` step's `git fetch` writes into the clone's `.git`, and Bitbucket clones as
+  root ([BCLOUD-23217](https://jira.atlassian.com/browse/BCLOUD-23217)): if Pipelines
+  runs the step as the image's user (uid 65532), the fetch may fail with a permission
+  error. Then set `run-as-user: 0` on the `plan` step.
+- `git fetch origin` works in a pull request's step with Pipelines' own credentials.
+- `BITBUCKET_DEPLOYMENT_ENVIRONMENT` holds the environment's name (`touchmark-distribute`).
+- An access token with *Pull requests: Write* may comment on the hub's pull requests,
+  and a `PUT` of another account's comment answers 403.
+- Bitbucket renders a link reference definition in a comment as nothing.
+- `restrictions.admin_only` of a deployment environment, which the API returns but does
+  not document.
+- Custom pipeline variables via the API: a run started through the API with variables
+  the YAML does not declare gets them in its environment, overriding deployment
+  variables, and a deployment restricted to admins or selected users pauses it when
+  someone else starts it.
+- A scheduled run, and the push pipeline after a merge, pass a deployment restricted to
+  admins or selected users; Atlassian says deployment permissions hold scheduled and
+  automated deployments too, until an allowed user resumes them.
+
+Next: [opt a repository in](opt-in.md).