diff --git a/docs/agents.md b/docs/agents.md index 7ef9a4b..bdff59d 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -9,7 +9,7 @@ | Program | `touchmark`, one static Go binary; the same binary in the image `ghcr.io/bedrock-python/touchmark` and the GitHub Action `bedrock-python/touchmark` | | Requires | git 2.31+ for `check`, `status`, `apply`, `manifest`; git 2.45+ for `plan` and `distribute`, which read the targets (the image brings its own) | | Install | archives on [GitHub releases](https://github.com/bedrock-python/touchmark/releases) · `go install github.com/bedrock-python/touchmark/cmd/touchmark@latest` (Go 1.26+) · `docker run ghcr.io/bedrock-python/touchmark:` | -| Platforms | GitHub (github.com, GHE.com, GHES 3.19+), GitLab 17.0+, Gitea 1.26+, Forgejo 15+ | +| Platforms | GitHub (github.com, GHE.com, GHES 3.19+), GitLab 17.0+, Gitea 1.26+, Forgejo 15+; as providers a hub delivers to: Bitbucket Cloud, Azure DevOps Services | | Hub template | | | Source | | @@ -47,8 +47,9 @@ no lock file, no header, no marker in their files. It never checks a target out and never runs a script, hook, filter or LFS from the hub or a target. It delivers nothing to a repository without the opt-in file, unless `targets.yml` subscribes it (`opt_in: assumed`). There is no pull mode (targets do not -fetch from the hub), no remote pack sources, no Azure DevOps driver, and no `setup` for -Bitbucket: a hub on Bitbucket Cloud runs in Bitbucket Pipelines, set up by hand. +fetch from the hub), no remote pack sources, no `setup` for Bitbucket (a hub on Bitbucket Cloud +runs in Bitbucket Pipelines, set up by hand), and no hub on Azure Pipelines: Azure DevOps is a +provider a hub elsewhere delivers to. ## Mental model @@ -251,7 +252,7 @@ ten `::error` and ten `::warning` annotations. | `version` | `1` | format version | | `id` | — | slug, 3–40 characters; names the branch `touchmark/`; `change-me` fails `check` | | `writer`, `sign` | — , `auto` | single-provider shorthand: the writer account, commit signing (`auto`, `always`) | -| `providers[]` | from CI | `id`, `type` (`github`, `gitlab`, `gitea`, `forgejo`), `url` (required for gitea and forgejo), `api_url`, `ca_file`, `writer`, `known_authors`, `automation_accounts`, `sign`, `limits`; replaces the shorthand | +| `providers[]` | from CI | `id`, `type` (`github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, `azure-devops`), `url` (required for gitea, forgejo and azure-devops: `https://dev.azure.com/`), `api_url`, `ca_file`, `writer`, `known_authors`, `automation_accounts`, `sign`, `limits`; replaces the shorthand | | `branch`, `branch_aliases` | `touchmark/`, none | the sync branch, and former names whose pull requests stay the hub's | | `previous_fingerprints` | none | the hub's fingerprints before it moved | | `opt_in_file` | `.engineering-assets.yml` | the opt-in file's path in targets | @@ -275,7 +276,9 @@ the one provider whose `url` it lies under (`github` defaults to `https://github `gitlab` to `https://gitlab.com`); none fails `check`, and so do several unless the entry's `provider` or `defaults.provider` picks one of them, credentials, a query, a fragment, and any scheme but `https` (`http` for loopback only). On GitHub, Gitea and -Forgejo a repository URL, and an `exclude` URL without `**`, names `owner/name`. +Forgejo a repository URL, and an `exclude` URL without `**`, names `owner/name`; on Azure +DevOps it is `https://dev.azure.com///_git/`, naming +`/`, and an `org` URL is the organization itself. `.engineering-assets.yml` (in each target; `touchmark schema opt-in`): `version`, `enabled` (default `true`; `false` opts out whatever the hub says), `packs` (added to @@ -284,7 +287,7 @@ covers everything under it). At most 64 KiB. `.touchmark/operations.yml` (in the hub; `touchmark schema operations`): `recreate[]` (`target`, `head`: acts while the branch head is that commit), -`forget_declines[]` (`target`, `pr`: acts once; on Bitbucket Cloud while present), `allow_mass_close` (`max`, `until`), +`forget_declines[]` (`target`, `pr`: acts once; on Bitbucket Cloud and Azure DevOps while present), `allow_mass_close` (`max`, `until`), `adopt_unmarked` (`until`). Dates are `YYYY-MM-DD`, UTC, inclusive. Environment, one set per provider; `` is the provider id upper-cased with `-` as @@ -307,7 +310,8 @@ from its environment once read: every managed file would look `local`. touchmark refuses a shallow clone and a partial clone (`--filter`). In CI: `fetch-depth: 0` (GitHub, Gitea), `GIT_DEPTH: "0"` (GitLab). 2. **Reader and writer are two different accounts** on every platform: two GitHub Apps, - two GitLab service accounts (or group access tokens), two Gitea or Forgejo bot users. + two GitLab service accounts (or group access tokens), two Gitea or Forgejo bot users, + two Azure DevOps users with personal access tokens (Code read; Code read & write). The read key reaches any branch of the hub; the write key must not. 3. **The write key is visible to the default branch only.** GitHub: secrets in the environment `touchmark-distribute` with deployment branches *Selected* = the default diff --git a/docs/concepts/memory.md b/docs/concepts/memory.md index 2e302dd..40bda51 100644 --- a/docs/concepts/memory.md +++ b/docs/concepts/memory.md @@ -109,6 +109,17 @@ request was open: declines it; bots' closes wait for the cooldown, and the third in a row counts as a decline. +## On Azure DevOps + +touchmark never writes to an abandoned pull request on Azure DevOps either: whether its +properties, which hold the marker there, can still be changed is not known, and the +memory of Bitbucket Cloud needs no such write. Everything [above](#on-bitbucket-cloud) +applies, with three differences: the marker lives in the property `touchmark.marker` +rather than the description, so the edit that records a new opt-in state changes only +that property; descriptions do carry the tick box *Rebuild this branch* while a branch is +paused; and touchmark never reactivates an abandoned pull request, though Azure DevOps +would let it. + ## Limits - Only the last 50 closed pull requests on the sync branches count. diff --git a/docs/guide/operations.md b/docs/guide/operations.md index 65e1cd4..a0ea0a8 100644 --- a/docs/guide/operations.md +++ b/docs/guide/operations.md @@ -37,7 +37,7 @@ cleanup to stay safe: | Entry | Acts | Then | |---|---|---| | `recreate` | while the sync branch's head is exactly `head` | a new push to the branch makes it inert | -| `forget_declines` | once: touchmark marks pull request `pr` revoked; on Bitbucket Cloud, while the entry is present (see below) | inert | +| `forget_declines` | once: touchmark marks pull request `pr` revoked; on Bitbucket Cloud and Azure DevOps, while the entry is present (see below) | inert | | `allow_mass_close` | up to `max` closes in a run, until the end of `until` (UTC) | inert after the date | | `adopt_unmarked` | until the end of `until` (UTC) | inert after the date | @@ -72,6 +72,10 @@ closed. Removed earlier, the decline is in force again; an open pull request sta memory never closes one, and its description then names the decline under *Previously declined*. +On Azure DevOps a `forget_declines` entry acts while it is present too: touchmark never +writes to an abandoned pull request there, so it cannot mark one revoked. Descriptions +there do carry *Rebuild this branch*. + ## On your machine A maintainer who holds the write key can run the same operations as flags of diff --git a/docs/guide/providers.md b/docs/guide/providers.md index c70a988..155c35e 100644 --- a/docs/guide/providers.md +++ b/docs/guide/providers.md @@ -201,6 +201,105 @@ their scopes: `doctor` reports both as unknown, so keep a reminder to rotate the reads 15 times a minute by default, with two targets at once, and writes at most once a second; `limits` overrides it. +## Azure DevOps + +A provider of type `azure-devops` is one organization of Azure DevOps Services, and its +`url` names it: `https://dev.azure.com/` (one provider per organization; +the old `https://.visualstudio.com` form and Azure DevOps Server are not +supported). `plan`, `distribute` and `doctor` work on it from a hub on GitHub or GitLab, +or wherever you run touchmark; a hub whose own CI is Azure Pipelines is not supported +yet, and `setup` has nothing for it. + +```yaml +# hub.yml +providers: + - id: ado + type: azure-devops + url: https://dev.azure.com/acme + writer: 3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d # the writer user's identity id +``` + +**Accounts.** Use two users that only touchmark uses, each with a personal access token +(PAT) for this organization only: service principals of Microsoft Entra ID are not +supported yet. Global PATs (for all organizations) stop working on 2026-12-01. + +| | Reader | Writer | +|---|---|---| +| Variable | `TOUCHMARK_ADO_READ_TOKEN` | `TOUCHMARK_ADO_WRITE_TOKEN` | +| Scopes | Code (read) | Code (read & write) | +| Repositories | read access to the targets | Contribute, Create branch and Contribute to pull requests on the targets only; no administration, and no access to the hub | + +A PAT cannot read its own expiry or scopes: `doctor` reports both as unknown, so keep a +reminder to renew the tokens before they expire. + +- **Accounts are identity ids.** Display names are neither unique nor stable, so + `writer`, `known_authors` and `automation_accounts` name users by their identity id, a + GUID: `GET https://dev.azure.com//_apis/connectionData` with the user's + token shows it as `authenticatedUser.id`. +- **Targets.** A repository is `/`; web URLs such as + `https://dev.azure.com/acme/Billing/_git/api` work. `org` names the whole organization + (`org: acme`, or its URL `https://dev.azure.com/acme`); select a project's repositories + with `match` (`Billing/*`). Azure DevOps has no topics: `check` refuses `topics` on its + entries. Disabled repositories are skipped. Project and repository names with spaces + can be reached through `org` only: `targets.yml` paths take letters, digits, `.`, `-` + and `_`. +- **The marker lives in a pull request property** (`touchmark.marker`), not in the + description, which Azure DevOps limits to 4 000 characters: the description is short, + its lists are cut to fit, and the marker still records every change. Editing the + description in the web UI leaves the marker as it is, and a marker pasted into the + description is ignored. The property is not protected, though: anyone who may + contribute to pull requests in the project, project Readers included by default, may + be able to write it through the API (see the + [threat model](../project/threat-model.md)). touchmark writes the property right after + it opens a pull request, trying a few times; should it still fail, it abandons the new + pull request and opens another, so that none stays without its marker. +- **People's text in the description** is theirs: a close or a refresh of the marker + writes the property alone and sends no description, so a description people filled up + to the limit never stops touchmark from closing its pull request. +- **Labels** are added by name, as `pr.labels` says. **Drafts** are Azure DevOps' own + draft flag (`pr.draft`). +- **Closing.** touchmark closes its own pull request by abandoning it, after it stored the + closing marker. It never reactivates an abandoned pull request, nor changes it: a pull + request touchmark closed stays closed, and the next proposal is a new one. +- **Declines** are remembered without writing to the abandoned pull request, as on + Bitbucket Cloud: while a pull request is open its marker records the state of the + opt-in file, and a person's decline holds while that file is parsed the same. touchmark + leaves no comment on it; the report says `declined`. To have declined content proposed + again, edit `packs` or `ignore`, or add a `forget_declines` entry, which acts while it + is present (see [Memory of declined pull requests](../concepts/memory.md#on-bitbucket-cloud)). +- **Comments** are threads touchmark closes at once, so that they never wait for a + resolution. +- **Branch policies** refuse direct pushes to the branches they protect, and the writer + reads none upfront: `doctor` shows `rules` as unknown; a push a policy refuses + (TF402455) is `blocked:rules:policy`, one the writer lacks a permission for (TF401027) + `blocked:permission:push`. The writer's permissions are read with the Has Permissions + API when the token may use it; when it may not, `doctor` shows `access` as unknown and + the first push or pull request meets a missing permission. +- **Commit author.** touchmark authors the writer's commits with its sign-in address when + Azure DevOps shows one, else with `@touchmark.invalid`. +- **Pace.** Azure DevOps meters a user's load over five minutes and announces it in its + `X-RateLimit-*` headers, which pause the provider before it delays requests, and in + `Retry-After` on requests it delayed, which pauses the provider for that long; touchmark + reads 120 times a minute with four targets at once and writes at most 30 times a minute + by default; `limits` overrides it. +- **The reader needs a token.** Azure DevOps sends anonymous reads of files (the Trees + API) and of pull request properties to its sign-in page even in public projects, so a + plan without the reader's token fails on Azure DevOps targets. +- **Cost.** Listing a target's pull requests reads the marker of every open one and of the + 60 newest abandoned and completed ones (touchmark remembers the 50 newest closed pull + requests); older closed pull requests count as without a marker. + +What only a live organization can confirm, and the driver assumes from the REST +reference: the size limit of pull request properties and whether they survive an +abandon; whether labels in a create request are applied (touchmark adds missing ones +after it); the error of a duplicate pull request (TF401179); whether `closedBy` is set +on abandoned pull requests; the order of pull request listings and where they cut +descriptions; the answer to a refused PAT (401, a sign-in redirect or a 203 page); the +exact texts of push refusals; the `mode` strings of the Trees API; how disabled and +renamed repositories answer; who can write pull request properties (whether project +Readers, through "Contribute to pull requests", can set or change `touchmark.marker`); +and whether `Retry-After` comes on answers that went through. + ## Moving - **A target moves to another provider on the same host** (a new App, a new group): add diff --git a/docs/project/threat-model.md b/docs/project/threat-model.md index d11d484..9bf5f1e 100644 --- a/docs/project/threat-model.md +++ b/docs/project/threat-model.md @@ -71,6 +71,7 @@ How to read the tests column: |---|---|---|---| | Opt-in file: at most 64 KiB, a bounded number of YAML anchors and aliases, a strict schema | `config` (`decodeStrict`, `ParseOptIn`), the drivers' `ReadFile` with a size limit | `TestDecodeLimits`, `TestParseHubAnchors`, `TestParseOptIn`, `TestOptInIgnoreLimit`, `FuzzParse`, `TestPlanAssumeOptInTooLarge` | — | | Marker: at most 16 KiB, at most 64 KiB inflated, strict JSON | `marker` | `TestParseGzipBomb`, `TestParseRejectsHeader`, `TestParseRejectsData`, `TestParseRejectsPayload`, `TestParseMaxChanges`, `TestFindLargeBody`, `FuzzParse`, `FuzzFind` | — | +| Azure DevOps: the marker lives in a pull request property (`touchmark.marker`), apart from the description; the author, the head, the head's repository (a fork) and the closer come from the API, never from the property; a new pull request whose property cannot be stored is abandoned, or marked through an edit when the abandon fails | `azuredevops` (`markerOf`, `toPR`, `storeNewMarker`), `decide.Identity`, `distribute` (`markCreated`) | `TestPRs`, `TestCreatePRMarkerAbandons`, `TestRunAzureDevOpsMarkerLost` *(git 2.45)* | The property is not protected. Anyone who may contribute to pull requests in the project may be able to write it through the API: project Readers have "Contribute to pull requests" by default, and the REST reference names only the scope `vso.code_write` (to confirm live, [providers.md](../guide/providers.md#azure-devops)). Such a person can block a target (`blocked:marker-invalid`, by breaking the marker of touchmark's open pull request, until an `adopt` entry takes it over) or, before the team abandons a pull request, write `closed.by=touchmark` into its marker so that the decline does not count and the content is proposed again. They cannot make a pull request of their own touchmark's: its author, head and fork come from the API. | | API answers: at most 32 MiB | `httpx` (`DefaultMaxBody`) | `TestMaxBody`, `TestOnlyHTTPXSpeaksHTTP` | — | | Objects of a target: a commit at most 1 MiB, a blob at most 8 MiB; more is `failed:git` | `gitx` (`MaxCommitObject`, `MaxReadBlob`), `distribute.failCommit` | `TestObjectLimits` | — | | Only the `recreate` and `repropose` checkboxes act; every other text of a target is data | `prbody.Ticked`, `prbody.Inert` | `TestTicked`, `TestUntick`, `FuzzTicked`, `TestInert`, `TestMemoryReproposeBypassesOnce`, `TestRunKeptBodyIsInert` *(git 2.45)* | — | diff --git a/docs/reference/compatibility.md b/docs/reference/compatibility.md index efebe96..1d1a551 100644 --- a/docs/reference/compatibility.md +++ b/docs/reference/compatibility.md @@ -32,7 +32,8 @@ changelog before you merge it. and reject fields they do not know. - **What touchmark leaves in repositories:** the marker in pull request bodies (`touchmark:v1`, an HTML comment, or a Markdown reference definition on Bitbucket - Cloud), the commit trailers (`Touchmark-Hub`, `Touchmark-Stream`, + Cloud; on Azure DevOps the comment lives in the pull request property + `touchmark.marker`), the commit trailers (`Touchmark-Hub`, `Touchmark-Stream`, `Touchmark-Content`, `Touchmark-Hub-Commit`), the control lines of a pull request body, the marker of the plan comment, and the sync branch's name. Every release reads what an earlier one wrote. The marker's payload is frozen: a change to it is a `touchmark:v2` diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index 7089ae7..8e6661a 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -41,7 +41,9 @@ history. `touchmark manifest` prints it. ``. It records the fingerprint and what the pull request carries. On Bitbucket Cloud, which shows HTML comments as text, the same payload is a Markdown link reference definition: -`[touchmark]: # "touchmark:v1 hub=… fp=… stream=sync key=sha256:… data=…"`. +`[touchmark]: # "touchmark:v1 hub=… fp=… stream=sync key=sha256:… data=…"`. On Azure +DevOps, whose descriptions hold 4 000 characters, the comment lives in the pull request +property `touchmark.marker` instead of the description. **opt-in file** — `.engineering-assets.yml` at a target's root. Its presence is consent; it can add packs and `ignore` paths, and `enabled: false` in it opts the target out. diff --git a/docs/reference/hub.md b/docs/reference/hub.md index 93184d3..0e1846c 100644 --- a/docs/reference/hub.md +++ b/docs/reference/hub.md @@ -106,11 +106,11 @@ In a hub pull request, `providers` and `ca_file` are read from the default branc | `security.private_targets_in_public_hub` | one of `skip`, `deliver` | What a public hub does with non-public targets: skip them, or deliver and let their names reach public CI logs. Default: skip. | | `providers` | list of objects | Platforms this hub delivers to. Targets refer to them by id. | | `providers[].id` | string, required | Name targets.yml uses for this provider, and the <ID> of its TOUCHMARK_<ID>_\* variables. | -| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, required | github: github.com, GHE.com or GitHub Enterprise Server; gitlab; gitea; forgejo; bitbucket: Bitbucket Cloud (plan, distribute and doctor; setup does not set up a hub on it yet). | -| `providers[].url` | string | Web URL of the instance. Default for github: https://github.com; for gitlab: https://gitlab.com; for bitbucket: https://bitbucket.org, the only url it takes without api_url. Required for gitea and forgejo. A target targets.yml names by web URL belongs to the provider whose url it lies under. | +| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, `azure-devops`, required | github: github.com, GHE.com or GitHub Enterprise Server; gitlab; gitea; forgejo; bitbucket: Bitbucket Cloud; azure-devops: one organization of Azure DevOps Services (bitbucket and azure-devops: plan, distribute and doctor; setup does not set up a hub on them yet). | +| `providers[].url` | string | Web URL of the instance. Default for github: https://github.com; for gitlab: https://gitlab.com; for bitbucket: https://bitbucket.org, the only url it takes without api_url. Required for gitea and forgejo, and for azure-devops: https://dev.azure.com/<organization>, one provider per organization. A target targets.yml names by web URL belongs to the provider whose url it lies under. | | `providers[].api_url` | string | API URL, when it cannot be derived from url. | | `providers[].ca_file` | string | Hub file with extra CA certificates for this instance. | -| `providers[].writer` | string, up to 255 characters | The account that writes to targets, as the platform names it: on GitHub the App's bot, <slug>[bot]; on GitLab, Gitea and Forgejo the username of the service account or bot user; on Bitbucket the bot account's UUID in braces, {…}. | +| `providers[].writer` | string, up to 255 characters | The account that writes to targets, as the platform names it: on GitHub the App's bot, <slug>[bot]; on GitLab, Gitea and Forgejo the username of the service account or bot user; on Bitbucket the bot account's UUID in braces, {…}; on Azure DevOps the user's id, a GUID. | | `providers[].known_authors` | list of strings, each up to 255 characters | Former writer accounts whose pull requests still count as this hub's own. | | `providers[].automation_accounts` | list of strings, each up to 255 characters | Accounts whose closing of a pull request is automatic, not a team's decision. | | `providers[].sign` | one of `auto`, `always` | auto: sign where it costs nothing and where target rules require it; always: sign every commit. Default: auto. | @@ -129,7 +129,7 @@ In a hub pull request, `providers` and `ca_file` are read from the default branc | `packs..formerly` | list of strings, each up to 64 characters | Former names of this pack. Their history counts as this pack's own. | | `writer` | string, up to 255 characters | Single-provider shorthand: the writer account, as the platform names it (on GitHub the App's bot, <slug>[bot]). Without platform, the provider's type and URL come from the CI environment, or locally from the hub's origin remote. Not allowed with providers. | | `sign` | one of `auto`, `always` | Single-provider shorthand: commit signing. Not allowed with providers. | -| `platform` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket` | Single-provider shorthand: the provider type, which is also its id. Not allowed with providers. | +| `platform` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, `azure-devops` | Single-provider shorthand: the provider type, which is also its id. Not allowed with providers. | | `base_url` | string | Single-provider shorthand: the provider URL, for self-managed instances. Needs platform; not allowed with providers. | diff --git a/docs/reference/operations.md b/docs/reference/operations.md index c6524ba..7a5b84d 100644 --- a/docs/reference/operations.md +++ b/docs/reference/operations.md @@ -22,8 +22,8 @@ writes the path alone. Every entry limits itself, so a forgotten one does nothing: a branch head that moved, a decline already revoked, a date that passed. The exception is a `forget_declines` entry -for a Bitbucket Cloud target: a declined pull request cannot be edited there, so it is -never marked revoked, and the entry acts while it is present (see +for a Bitbucket Cloud or Azure DevOps target: touchmark never edits a declined pull +request there, so it is never marked revoked, and the entry acts while it is present (see [Memory of declined pull requests](../concepts/memory.md#on-bitbucket-cloud)). `check` warns about entries past their date. `plan` on a hub pull request shows each entry's effect: `applies`, `none`, `expired` or @@ -40,7 +40,7 @@ warns about entries past their date. | `recreate` | list of objects | Rebuild a paused sync branch from scratch. Commits others added to it are dropped. An entry acts only while the branch head is the given commit. | | `recreate[].target` | string, up to 512 characters, required | One target repository: <provider>:<path> or <path>, as touchmark prints it. | | `recreate[].head` | string, required | Full commit id of the sync branch head to replace: 40 or 64 lowercase hex digits. | -| `forget_declines` | list of objects | Propose again the content of a pull request the target closed without merging. touchmark marks that pull request revoked, so the entry acts once; on Bitbucket Cloud, where a declined pull request cannot be edited, the entry acts while it is present. | +| `forget_declines` | list of objects | Propose again the content of a pull request the target closed without merging. touchmark marks that pull request revoked, so the entry acts once; on Bitbucket Cloud and Azure DevOps, where touchmark never edits a declined pull request, the entry acts while it is present. | | `forget_declines[].target` | string, up to 512 characters, required | One target repository: <provider>:<path> or <path>, as touchmark prints it. | | `forget_declines[].pr` | integer ≥ 1, required | Number of the declined pull request. | | `allow_mass_close` | object | Lift the guard against closing many pull requests in one run, up to max closes, until the date. | diff --git a/docs/reference/output.md b/docs/reference/output.md index cd1a2c5..36d3b38 100644 --- a/docs/reference/output.md +++ b/docs/reference/output.md @@ -87,7 +87,7 @@ and the branch head before and after. | `assume_opt_in` | boolean | plan --assume-opt-in: for this report only, every target without an opt-in file, or whose opt-in file is not a regular file or is too large, counts as opted in; one whose opt-in file says enabled: false stays opted out. | | `providers` | list of objects, required | The hub's providers, in hub.yml order. | | `providers[].id` | string, required | — | -| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, required | — | +| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, `azure-devops`, required | — | | `providers[].host` | string, not empty, required | The provider's host, with a port when it has one. | | `providers[].reader` | string | Login of the read account. | | `providers[].writer` | string | Login of the write account. | @@ -199,7 +199,7 @@ hub, for each provider's write identity, and for each target. See | `hub_checks[].detail` | string | What the check found; absent for a target the report does not name. | | `providers` | list of objects, required | The hub's providers, in hub.yml order. | | `providers[].id` | string, required | — | -| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, required | — | +| `providers[].type` | one of `github`, `gitlab`, `gitea`, `forgejo`, `bitbucket`, `azure-devops`, required | — | | `providers[].host` | string, not empty, required | The provider's host, with a port when it has one. | | `providers[].writer` | string | Login of the writer hub.yml names. | | `providers[].self` | string | Login the write credential acts as. | diff --git a/docs/reference/targets.md b/docs/reference/targets.md index 50216ef..ab25421 100644 --- a/docs/reference/targets.md +++ b/docs/reference/targets.md @@ -190,8 +190,8 @@ either way. | `targets[].provider` | string | Id of a provider in hub.yml. | | `targets[].packs` | list of strings, each up to 64 characters | — | | `targets[].opt_in` | one of `required`, `assumed` | required (the default): a repository the entry selects gets nothing until it adds the opt-in file. assumed: it counts as opted in without one, with the packs targets.yml gives it, until its opt-in file says enabled: false. One entry with assumed is enough to subscribe a repository. | -| `targets[].org` | string, up to 512 characters | Every repository in a GitHub organisation, a GitLab, Gitea or Forgejo namespace or a Bitbucket workspace (a synonym of group), named by its path or its web URL. An entry has exactly one of `repo`, `org` and `group`. | -| `targets[].topics` | list of strings | Only repositories with all of these topics. Bitbucket has no topics: check refuses them on its entries; use match. Only with `org` or `group`. | +| `targets[].org` | string, up to 512 characters | Every repository in a GitHub organisation, a GitLab, Gitea or Forgejo namespace, a Bitbucket workspace or an Azure DevOps organization (the provider's url; a project is selected with match) (a synonym of group), named by its path or its web URL. An entry has exactly one of `repo`, `org` and `group`. | +| `targets[].topics` | list of strings | Only repositories with all of these topics. Bitbucket and Azure DevOps have no topics: check refuses them on their entries; use match. Only with `org` or `group`. | | `targets[].subgroups` | boolean | Include repositories in nested groups. Default: true. Only with `org` or `group`. | | `targets[].forks` | boolean | Include forks. Default: false. Only with `org` or `group`. | | `targets[].match` | list of strings, each up to 512 characters | Only repositories whose full path, without the provider, matches at least one of these patterns: \* within one segment, \*\* any number of segments, ? one character, compared ignoring case, such as acme/svc-\* or platform/\*\*/api. Quote a pattern that starts with \*, which YAML reads as an alias. Only with `org` or `group`. | diff --git a/internal/cli/distribute_deps.go b/internal/cli/distribute_deps.go index 5f30a3e..a83c1de 100644 --- a/internal/cli/distribute_deps.go +++ b/internal/cli/distribute_deps.go @@ -32,7 +32,7 @@ import ( type distributeDriver func(config.ResolvedProvider, auth.Credential, *httpx.Client) (platform.Writer, error) // The write drivers of distribute by provider type: Gitea and Forgejo, -// GitLab, GitHub and Bitbucket Cloud (drivers.go). Tests install the fake platform here; tests that do must +// GitLab, GitHub, Bitbucket Cloud and Azure DevOps (drivers.go). Tests install the fake platform here; tests that do must // not run in parallel. var distributeDrivers = map[string]distributeDriver{} diff --git a/internal/cli/drivers.go b/internal/cli/drivers.go index a2bc36e..3242a4d 100644 --- a/internal/cli/drivers.go +++ b/internal/cli/drivers.go @@ -1,6 +1,7 @@ package cli import ( + "github.com/bedrock-python/touchmark/internal/platform/azuredevops" "github.com/bedrock-python/touchmark/internal/platform/bitbucket" "github.com/bedrock-python/touchmark/internal/platform/gitea" "github.com/bedrock-python/touchmark/internal/platform/github" @@ -9,8 +10,9 @@ import ( // init registers the platform drivers this build has: Gitea and Forgejo, one // driver for both, GitLab, GitHub (github.com, GHE.com and GitHub -// Enterprise Server) and Bitbucket Cloud. Tests that install their own -// drivers replace the maps whole and restore them afterwards. +// Enterprise Server), Bitbucket Cloud and Azure DevOps Services. Tests that +// install their own drivers replace the maps whole and restore them +// afterwards. func init() { for _, typ := range []string{"gitea", "forgejo"} { planDrivers[typ] = gitea.NewReader @@ -22,4 +24,6 @@ func init() { distributeDrivers["github"] = github.NewWriter planDrivers["bitbucket"] = bitbucket.NewReader distributeDrivers["bitbucket"] = bitbucket.NewWriter + planDrivers["azure-devops"] = azuredevops.NewReader + distributeDrivers["azure-devops"] = azuredevops.NewWriter } diff --git a/internal/cli/drivers_test.go b/internal/cli/drivers_test.go index 7c54133..4816de7 100644 --- a/internal/cli/drivers_test.go +++ b/internal/cli/drivers_test.go @@ -4,9 +4,9 @@ import "testing" // TestDriversRegistered: this build plans and distributes on every // platform it supports: Gitea and Forgejo, one driver for both, GitLab, -// GitHub and Bitbucket Cloud. +// GitHub, Bitbucket Cloud and Azure DevOps Services. func TestDriversRegistered(t *testing.T) { - for _, typ := range []string{"gitea", "forgejo", "gitlab", "github", "bitbucket"} { + for _, typ := range []string{"gitea", "forgejo", "gitlab", "github", "bitbucket", "azure-devops"} { if planDrivers[typ] == nil || distributeDrivers[typ] == nil { t.Errorf("no %s driver: plan %v, distribute %v", typ, planDrivers[typ] != nil, distributeDrivers[typ] != nil) } diff --git a/internal/cli/plan.go b/internal/cli/plan.go index 2c32d3b..acb1b3e 100644 --- a/internal/cli/plan.go +++ b/internal/cli/plan.go @@ -33,8 +33,8 @@ import ( type planDriver func(config.ResolvedProvider, auth.Credential, *httpx.Client) (platform.Reader, error) // The drivers of plan by provider type, and a snapshot source that replaces -// plan's own. This build has the Gitea and Forgejo, GitLab, GitHub and -// Bitbucket Cloud drivers (drivers.go). plan's own source is distribute's: +// plan's own. This build has the Gitea and Forgejo, GitLab, GitHub, +// Bitbucket Cloud and Azure DevOps drivers (drivers.go). plan's own source is distribute's: // a private git repository per target (sharedDeps), so that plan and // distribute --dry-run decide alike. Tests install the fake platform here; a planSnapshots that // is only a snapshot.Source (the fake in memory mode) keeps the in-memory @@ -50,9 +50,10 @@ var ( var gitVersion = func(ctx context.Context) ([3]int, error) { return gitx.New("").Version(ctx) } // basicUsers are the users a token goes with in HTTP Basic headers (GitHub, -// GitLab, Gitea and Bitbucket git over HTTPS, GitLab's job token): the +// GitLab, Gitea and Bitbucket git over HTTPS, GitLab's job token, Azure +// DevOps' REST API with an empty user and its git with "touchmark"): the // base64 forms of "user:token" are masked too. -var basicUsers = []string{"x-access-token", "oauth2", hubch.GitLabUser, "x-bitbucket-api-token-auth"} +var basicUsers = []string{"x-access-token", "oauth2", hubch.GitLabUser, "x-bitbucket-api-token-auth", "", "touchmark"} // Suffixes of the credential variables after a provider's prefix. var ( diff --git a/internal/cli/setup.go b/internal/cli/setup.go index 9465478..3f6c444 100644 --- a/internal/cli/setup.go +++ b/internal/cli/setup.go @@ -101,6 +101,9 @@ func runSetup(ctx context.Context, e *env, o *options, s *setupOptions) (err err return configErrorf("setup does not set up bitbucket yet: set up a hub on Bitbucket Pipelines by hand (the bot accounts and their API "+ "tokens, the hub's access token, the repository and deployment variables), as %s shows; a hub on GitHub or GitLab "+ "delivers to Bitbucket Cloud as one of its providers", docsurl.GettingStarted("bitbucket")) + case "azure-devops": + return configErrorf("setup does not set up azure-devops yet: a hub on Azure Pipelines comes in a later release; " + + "a hub on GitHub or GitLab delivers to Azure DevOps as one of its providers") case "github", "gitlab": default: return usageErrorf("unknown platform %q: setup sets up github or gitlab", platform) diff --git a/internal/cli/setup_test.go b/internal/cli/setup_test.go index da1a77c..95ba042 100644 --- a/internal/cli/setup_test.go +++ b/internal/cli/setup_test.go @@ -45,7 +45,8 @@ func TestSetupRefusals(t *testing.T) { want string }{ {"no-platform", nil, []string{"setup"}, "missing the platform"}, - {"unknown", nil, []string{"setup", "azure-devops", "--hub", h.dir}, `unknown platform "azure-devops"`}, + {"unknown", nil, []string{"setup", "sourcehut", "--hub", h.dir}, `unknown platform "sourcehut"`}, + {"azure-devops", nil, []string{"setup", "azure-devops", "--hub", h.dir}, "setup does not set up azure-devops yet"}, {"bitbucket", nil, []string{"setup", "bitbucket", "--hub", h.dir}, "setup does not set up bitbucket yet"}, {"gitea", nil, []string{"setup", "gitea", "--hub", h.dir}, "every branch the secrets"}, {"forgejo", nil, []string{"setup", "forgejo"}, "setup does not set up forgejo"}, diff --git a/internal/config/assume_test.go b/internal/config/assume_test.go index 79a36ad..b0f6ded 100644 --- a/internal/config/assume_test.go +++ b/internal/config/assume_test.go @@ -128,3 +128,33 @@ func TestOptInEnabled(t *testing.T) { t.Errorf("enabled changes the hash: %s, %s", a, b) } } + +// TestAssumeAzureDevOps: an org entry of an Azure DevOps provider names the +// organization, which no repository path (project/repository) starts +// with, so any target of the provider may lie in it, as its match allows. +func TestAssumeAzureDevOps(t *testing.T) { + hub := &Hub{ID: "acme-eng", Providers: []Provider{ + {ID: "ado", Type: "azure-devops", URL: "https://dev.azure.com/acme"}, + {ID: "gh", Type: "github"}, + }} + targets := &Targets{ + Defaults: Defaults{Provider: "ado"}, + Targets: []Entry{ + {Org: "acme", OptIn: OptInAssumed, Match: []string{"Billing/*"}}, + }, + } + for _, tc := range []struct { + target string + want Assumption + }{ + {"Billing/api", Assumption{Unresolved: []string{"org: acme"}}}, + {"ado:billing/web", Assumption{Unresolved: []string{"org: acme"}}}, + {"Web/site", Assumption{}}, + {"gh:Billing/api", Assumption{}}, + } { + got, err := Assume(hub, targets, ref(t, tc.target)) + if err != nil || !reflect.DeepEqual(got, tc.want) { + t.Errorf("Assume(%s) = %+v, %v; want %+v", tc.target, got, err, tc.want) + } + } +} diff --git a/internal/config/check.go b/internal/config/check.go index 6503d15..a5f3477 100644 --- a/internal/config/check.go +++ b/internal/config/check.go @@ -192,7 +192,11 @@ func (c *checker) exclude() { } segs := strings.Split(pat.Path, "/") if typ := c.flatType(res.provider(pat.Provider)); typ != "" && len(segs) > 2 && !slices.Contains(segs, "**") { - c.warnf(TargetsFile, "exclude[%d]: %s has %d path segments, and a %s repository path is owner/name, so it excludes nothing", i, ex, len(segs), typ) + shape := "a " + typ + " repository path is owner/name" + if typ == "azure-devops" { + shape = "an Azure DevOps repository path is project/repository" + } + c.warnf(TargetsFile, "exclude[%d]: %s has %d path segments, and %s, so it excludes nothing", i, ex, len(segs), shape) } for j := range c.targets.Targets { e := &c.targets.Targets[j] @@ -219,7 +223,10 @@ func (c *checker) exclude() { // match warns about a pattern of an org or group entry's match that no // repository of its namespace can match: it selects nothing. A namespace // on a platform without nested namespaces (flatType) holds no subgroups, -// whatever subgroups says. +// whatever subgroups says. On Azure DevOps the namespace is the +// organization, which no repository path starts with (they are +// project/repository): a pattern there selects nothing only when it is +// deeper than that. func (c *checker) match() { res := resolver{hub: c.hub, targets: c.targets} for i := range c.targets.Targets { @@ -231,7 +238,18 @@ func (c *checker) match() { if err != nil { continue } - typ := c.flatType(res.provider(firstNonEmpty(e.Provider, ns.Provider))) + id := res.provider(firstNonEmpty(e.Provider, ns.Provider)) + typ := c.flatType(id) + if c.providerType(id) == "azure-devops" { + for j, pat := range e.Match { + segs := strings.Split(pat, "/") + if checkMatchPattern(pat) == nil && len(segs) != 2 && !slices.Contains(segs, "**") { + c.warnf(TargetsFile, "targets[%d].match[%d]: %s has %d path segments, and an Azure DevOps repository path is project/repository, "+ + "so it selects nothing", i, j, pat, len(segs)) + } + } + continue + } for j, pat := range e.Match { if checkMatchPattern(pat) == nil && !globUnder(pat, ns.Path, includeSubgroups(e) && typ == "") { where := "under " + ns.Path @@ -248,12 +266,13 @@ func (c *checker) match() { } // flatType returns the type of the provider with id when its platform has no -// nested namespaces (github, gitea, forgejo, bitbucket): a repository path -// there is owner/name. It returns "" for any other type, and for a provider -// hub.yml does not list (the implicit one, whose type the CI tells). +// nested namespaces (github, gitea, forgejo, bitbucket, azure-devops): a +// repository path there is owner/name (project/repository on Azure DevOps). +// It returns "" for any other type, and for a provider hub.yml does not +// list (the implicit one, whose type the CI tells). func (c *checker) flatType(id string) string { switch typ := c.providerType(id); typ { - case "github", "gitea", "forgejo", "bitbucket": + case "github", "gitea", "forgejo", "bitbucket", "azure-devops": return typ } return "" @@ -270,9 +289,9 @@ func (c *checker) providerType(id string) string { return "" } -// topics refuses topics on an entry whose provider is Bitbucket Cloud, -// whose repositories have none: the entry would select nothing, and the -// driver refuses the selector. +// topics refuses topics on an entry whose provider is Bitbucket Cloud or +// Azure DevOps, whose repositories have none: the entry would select +// nothing, and the driver refuses the selector. func (c *checker) topics() { res := resolver{hub: c.hub, targets: c.targets} for i := range c.targets.Targets { @@ -285,9 +304,13 @@ func (c *checker) topics() { continue } id := res.provider(firstNonEmpty(e.Provider, ns.Provider)) - if c.providerType(id) == "bitbucket" { + switch c.providerType(id) { + case "bitbucket": c.errorf(TargetsFile, "targets[%d].topics: provider %s is Bitbucket, whose repositories have no topics; "+ "select them with match: (paths like %s/svc-*) or list them with repo:", i, id, ns.Path) + case "azure-devops": + c.errorf(TargetsFile, "targets[%d].topics: provider %s is Azure DevOps, whose repositories have no topics; "+ + "select them with match: (paths like /svc-*) or list them with repo: (/)", i, id) } } } diff --git a/internal/config/check_test.go b/internal/config/check_test.go index 4ba9f75..4c9b7f2 100644 --- a/internal/config/check_test.go +++ b/internal/config/check_test.go @@ -189,6 +189,26 @@ func TestCheck(t *testing.T) { }, warnings: []string{"targets.yml: exclude[0]: acme/legacy/old has 3 path segments, and a bitbucket repository path is owner/name, so it excludes nothing"}, }, + { + name: "topics and match on azure-devops", + hub: &Hub{ID: "acme-eng", Providers: []Provider{ + {ID: "ado", Type: "azure-devops", URL: "https://dev.azure.com/acme"}, + }}, + targets: &Targets{ + Targets: []Entry{ + {Org: "acme", Topics: []string{"python"}}, + {Org: "acme", Match: []string{"Billing/*", "Web/**", "Billing/api/x"}}, + }, + Exclude: []string{"Legacy/old/x"}, + }, + errors: []string{ + "targets.yml: targets[0].topics: provider ado is Azure DevOps, whose repositories have no topics; select them with match: (paths like /svc-*)", + }, + warnings: []string{ + "targets.yml: exclude[0]: Legacy/old/x has 3 path segments, and an Azure DevOps repository path is project/repository, so it excludes nothing", + "targets.yml: targets[1].match[2]: Billing/api/x has 3 path segments, and an Azure DevOps repository path is project/repository, so it selects nothing", + }, + }, { name: "listed and excluded", hub: selectHub(), diff --git a/internal/config/config.go b/internal/config/config.go index a848a64..5d306d8 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -97,7 +97,7 @@ type Security struct { type Provider struct { ID string `yaml:"id"` - Type string `yaml:"type"` // github | gitlab | gitea | forgejo | bitbucket + Type string `yaml:"type"` // github | gitlab | gitea | forgejo | bitbucket | azure-devops URL string `yaml:"url"` APIURL string `yaml:"api_url"` CAFile string `yaml:"ca_file"` diff --git a/internal/config/hub.go b/internal/config/hub.go index f42bd65..65a4b9b 100644 --- a/internal/config/hub.go +++ b/internal/config/hub.go @@ -26,7 +26,7 @@ const ( // Enumerations of hub.yml. var ( - providerTypes = []string{"github", "gitlab", "gitea", "forgejo", "bitbucket"} + providerTypes = []string{"github", "gitlab", "gitea", "forgejo", "bitbucket", "azure-devops"} linkHubModes = []string{"auto", "always", "never"} isolationModes = []string{"platform", "external", "none"} privateTargets = []string{"skip", "deliver"} @@ -253,10 +253,15 @@ func checkProvider(pr *Provider, f providerFields, p *problems) { } } else if pr.Type == "gitea" || pr.Type == "forgejo" { p.errorf(f.name("url"), "required for %s", pr.Type) + } else if pr.Type == "azure-devops" { + p.errorf(f.name("url"), "required for azure-devops: https://dev.azure.com/") } if pr.Type == "bitbucket" && f.has("url") && checkURL(pr.URL) == nil && !f.has("api_url") && !isBitbucketCloud(pr.URL) { p.errorf(f.name("url"), "%s", bitbucketURLError) } + if pr.Type == "azure-devops" && f.has("url") && checkURL(pr.URL) == nil && !azureDevOpsURLOK(pr.URL, f.has("api_url")) { + p.errorf(f.name("url"), "%s", azureDevOpsURLError) + } if f.has("api_url") { if err := checkURL(pr.APIURL); err != nil { p.errorf(f.name("api_url"), "%v", err) diff --git a/internal/config/providers.go b/internal/config/providers.go index d60f07d..9da404e 100644 --- a/internal/config/providers.go +++ b/internal/config/providers.go @@ -7,6 +7,7 @@ import ( "fmt" "net/url" "os" + "regexp" "slices" "strings" @@ -37,6 +38,51 @@ func isBitbucketCloud(raw string) bool { return err == nil && strings.EqualFold(u.Scheme, "https") && strings.EqualFold(u.Host, "bitbucket.org") && u.Path == "" } +// Azure DevOps Services: the host of every organization's web URL and REST +// base, https://dev.azure.com/. A provider of type +// azure-devops is one organization; its url names it. +const azureDevOpsHost = "dev.azure.com" + +// azureDevOpsURLError is the complaint about an azure-devops provider whose +// url is not an organization of Azure DevOps Services. +const azureDevOpsURLError = "a provider of type azure-devops is one organization of Azure DevOps Services: " + + "url is https://dev.azure.com/ (one provider per organization; the old " + + "https://.visualstudio.com form and Azure DevOps Server are not supported). " + + "Elsewhere (a test server) set api_url too, and keep the organization as the url's only path segment" + +// azureOrgRe matches an organization name of Azure DevOps Services: +// letters, digits and hyphens, starting and ending with a letter or digit. +var azureOrgRe = regexp.MustCompile(`^[A-Za-z0-9]([A-Za-z0-9-]{0,48}[A-Za-z0-9])?$`) + +// AzureDevOpsOrg returns the organization an azure-devops provider's url +// names, its only path segment ("acme" of https://dev.azure.com/acme); ok is +// false when the url has another shape. +func AzureDevOpsOrg(raw string) (org string, ok bool) { + u, err := url.Parse(strings.TrimRight(raw, "/")) + if err != nil || u.Host == "" { + return "", false + } + org = strings.TrimPrefix(u.Path, "/") + if !azureOrgRe.MatchString(org) { + return "", false + } + return org, true +} + +// azureDevOpsURLOK reports whether raw, a checked URL, can be an +// azure-devops provider's url: one organization segment, and, without an +// api_url, on https://dev.azure.com. +func azureDevOpsURLOK(raw string, hasAPIURL bool) bool { + if _, ok := AzureDevOpsOrg(raw); !ok { + return false + } + if hasAPIURL { + return true + } + u, err := url.Parse(raw) + return err == nil && strings.EqualFold(u.Scheme, "https") && strings.EqualFold(u.Host, azureDevOpsHost) +} + // ResolvedProvider is a provider from hub.yml (or the implicit one) with // everything a driver needs besides credentials. type ResolvedProvider struct { @@ -46,7 +92,8 @@ type ResolvedProvider struct { // APIURL is the REST base: https://api.github.com, https://api. // for *.ghe.com, /api/v3 for GitHub Enterprise Server, // /api/v4 for GitLab, /api/v1 for Gitea and Forgejo, - // https://api.bitbucket.org/2.0 for Bitbucket Cloud; + // https://api.bitbucket.org/2.0 for Bitbucket Cloud, the url itself + // (https://dev.azure.com/) for Azure DevOps; // Provider.APIURL when set. On GitHub Actions for the hub's own host, // GITHUB_API_URL wins over the derived URL, not over Provider.APIURL: // an explicit api_url means the same in CI and locally. @@ -94,7 +141,9 @@ type ResolvedProvider struct { // Every provider is checked again, with ParseHub's defaults, so a hub built // in code resolves like a parsed one: a valid id, no duplicate ids, a known // type, the public URL of github, gitlab and bitbucket when URL is empty, -// an api_url for a bitbucket provider anywhere but Bitbucket Cloud, https URLs +// an api_url for a bitbucket provider anywhere but Bitbucket Cloud, the url +// of an organization for azure-devops (an api_url anywhere but +// dev.azure.com), https URLs // (http only for localhost), sign auto. A nil hub is a legacy hub. Error // messages never quote environment values. A nil getenv reads the process // environment. @@ -289,6 +338,9 @@ func resolveProvider(p Provider, env ciEnv) (ResolvedProvider, error) { if p.Type == "bitbucket" && p.APIURL == "" && !isBitbucketCloud(p.URL) { return ResolvedProvider{}, fmt.Errorf("url: %s", bitbucketURLError) } + if p.Type == "azure-devops" && !azureDevOpsURLOK(p.URL, p.APIURL != "") { + return ResolvedProvider{}, fmt.Errorf("url: %s", azureDevOpsURLError) + } setDefault(&p.Sign, defaultSign) u, err := url.Parse(p.URL) if err != nil { @@ -337,6 +389,10 @@ func derivedAPIURL(typ, webURL, hostname, host string) string { case "bitbucket": // resolveProvider asks for api_url anywhere else. return bitbucketAPIURL + case "azure-devops": + // The REST API of an organization is under its web URL, + // https://dev.azure.com//_apis. + return webURL } return webURL + "/api/v1" } diff --git a/internal/config/providers_test.go b/internal/config/providers_test.go index 36c0a99..e0f5568 100644 --- a/internal/config/providers_test.go +++ b/internal/config/providers_test.go @@ -40,6 +40,8 @@ func TestResolveProviders(t *testing.T) { {ID: "bb", Type: "bitbucket"}, {ID: "bb-url", Type: "bitbucket", URL: "https://Bitbucket.org/"}, {ID: "bb-test", Type: "bitbucket", URL: "http://localhost:8080", APIURL: "http://localhost:8080/2.0"}, + {ID: "ado", Type: "azure-devops", URL: "https://dev.azure.com/acme/"}, + {ID: "ado-test", Type: "azure-devops", URL: "http://localhost:8081/acme", APIURL: "http://localhost:8081/acme"}, }} want := map[string]resolvedURLs{ "gh": {"https://github.com", "github.com", "https://api.github.com", "https://api.github.com/graphql", "TOUCHMARK_GH_"}, @@ -56,6 +58,8 @@ func TestResolveProviders(t *testing.T) { "bb": {"https://bitbucket.org", "bitbucket.org", "https://api.bitbucket.org/2.0", "", "TOUCHMARK_BB_"}, "bb-url": {"https://Bitbucket.org", "bitbucket.org", "https://api.bitbucket.org/2.0", "", "TOUCHMARK_BB_URL_"}, "bb-test": {"http://localhost:8080", "localhost:8080", "http://localhost:8080/2.0", "", "TOUCHMARK_BB_TEST_"}, + "ado": {"https://dev.azure.com/acme", "dev.azure.com", "https://dev.azure.com/acme", "", "TOUCHMARK_ADO_"}, + "ado-test": {"http://localhost:8081/acme", "localhost:8081", "http://localhost:8081/acme", "", "TOUCHMARK_ADO_TEST_"}, } // The CI environment does not matter for declared providers, except // GITHUB_API_URL on the hub's own host (below). @@ -304,8 +308,23 @@ func TestResolveProvidersErrors(t *testing.T) { }, { name: "bad type", + hub: &Hub{Providers: []Provider{{ID: "srht", Type: "sourcehut"}}}, + want: `hub.yml: providers[0]: type: "sourcehut" must be one of github, gitlab, gitea, forgejo, bitbucket, azure-devops`, + }, + { + name: "azure-devops without url", hub: &Hub{Providers: []Provider{{ID: "ado", Type: "azure-devops"}}}, - want: `hub.yml: providers[0]: type: "azure-devops" must be one of github, gitlab, gitea, forgejo, bitbucket`, + want: "hub.yml: providers[0]: url: required for azure-devops", + }, + { + name: "azure-devops at a project", + hub: &Hub{Providers: []Provider{{ID: "ado", Type: "azure-devops", URL: "https://dev.azure.com/acme/Billing"}}}, + want: "hub.yml: providers[0]: url: a provider of type azure-devops is one organization of Azure DevOps Services", + }, + { + name: "azure-devops elsewhere without api_url", + hub: &Hub{Providers: []Provider{{ID: "ado", Type: "azure-devops", URL: "https://ado.example.com/acme"}}}, + want: "hub.yml: providers[0]: url: a provider of type azure-devops is one organization", }, { name: "bitbucket elsewhere without api_url", diff --git a/internal/config/select.go b/internal/config/select.go index 43a477b..89f6b1d 100644 --- a/internal/config/select.go +++ b/internal/config/select.go @@ -60,22 +60,40 @@ func (r resolver) matchPattern(pat Ref, target Ref) bool { // mayContain reports whether the org or group entry e, whose namespace is // ns, could select target: the target lies in the namespace (directly, or // in a nested one when subgroups are included) on a compatible provider, -// and its path passes the entry's match patterns. +// and its path passes the entry's match patterns. On Azure DevOps the +// namespace is the provider's organization, which repository paths +// (project/repository) do not start with: any target of the provider may +// lie in it. func (r resolver) mayContain(e *Entry, ns Ref, target Ref) bool { - if !sameProvider(r.provider(firstNonEmpty(e.Provider, ns.Provider)), r.provider(target.Provider)) { + provider := r.provider(firstNonEmpty(e.Provider, ns.Provider)) + if !sameProvider(provider, r.provider(target.Provider)) { return false } + if len(e.Match) > 0 && !matchAnyPath(e.Match, target.Path) { + return false + } + if r.providerType(provider) == "azure-devops" { + return true + } prefix := strings.ToLower(ns.Path) + "/" path := strings.ToLower(target.Path) if !strings.HasPrefix(path, prefix) { return false } - if len(e.Match) > 0 && !matchAnyPath(e.Match, target.Path) { - return false - } return includeSubgroups(e) || !strings.Contains(path[len(prefix):], "/") } +// providerType returns the type of the provider with id in hub.yml, "" for +// one it does not list. +func (r resolver) providerType(id string) string { + for _, p := range r.hub.Providers { + if p.ID == id { + return p.Type + } + } + return "" +} + func firstNonEmpty(a, b string) string { if a != "" { return a diff --git a/internal/config/testdata/hub/invalid/provider-azure-devops-no-url.yml b/internal/config/testdata/hub/invalid/provider-azure-devops-no-url.yml new file mode 100644 index 0000000..556884d --- /dev/null +++ b/internal/config/testdata/hub/invalid/provider-azure-devops-no-url.yml @@ -0,0 +1,6 @@ +# error: providers[0].url: required for azure-devops: https://dev.azure.com/ +version: 1 +id: acme-eng +providers: + - id: ado + type: azure-devops diff --git a/internal/config/testdata/hub/invalid/provider-azure-devops-project.yml b/internal/config/testdata/hub/invalid/provider-azure-devops-project.yml new file mode 100644 index 0000000..9ffa0fc --- /dev/null +++ b/internal/config/testdata/hub/invalid/provider-azure-devops-project.yml @@ -0,0 +1,7 @@ +# error: providers[0].url: a provider of type azure-devops is one organization of Azure DevOps Services +version: 1 +id: acme-eng +providers: + - id: ado + type: azure-devops + url: https://dev.azure.com/acme/Billing diff --git a/internal/config/testdata/hub/invalid/provider-azure-devops-visualstudio.yml b/internal/config/testdata/hub/invalid/provider-azure-devops-visualstudio.yml new file mode 100644 index 0000000..5a84835 --- /dev/null +++ b/internal/config/testdata/hub/invalid/provider-azure-devops-visualstudio.yml @@ -0,0 +1,7 @@ +# error: providers[0].url: a provider of type azure-devops is one organization of Azure DevOps Services +version: 1 +id: acme-eng +providers: + - id: ado + type: azure-devops + url: https://acme.visualstudio.com diff --git a/internal/config/testdata/hub/invalid/provider-bad-type.yml b/internal/config/testdata/hub/invalid/provider-bad-type.yml index b3d995b..15d7d6d 100644 --- a/internal/config/testdata/hub/invalid/provider-bad-type.yml +++ b/internal/config/testdata/hub/invalid/provider-bad-type.yml @@ -1,6 +1,6 @@ -# error: providers[0].type: "azure-devops" must be one of +# error: providers[0].type: "sourcehut" must be one of version: 1 id: acme-eng providers: - - id: ado - type: azure-devops + - id: srht + type: sourcehut diff --git a/internal/config/testdata/hub/invalid/shorthand-bad-platform.yml b/internal/config/testdata/hub/invalid/shorthand-bad-platform.yml index ea220d9..203c824 100644 --- a/internal/config/testdata/hub/invalid/shorthand-bad-platform.yml +++ b/internal/config/testdata/hub/invalid/shorthand-bad-platform.yml @@ -1,4 +1,4 @@ -# error: platform: "azure-devops" must be one of +# error: platform: "sourcehut" must be one of version: 1 id: acme-eng -platform: azure-devops +platform: sourcehut diff --git a/internal/config/testdata/hub/valid/azure-devops.yml b/internal/config/testdata/hub/valid/azure-devops.yml new file mode 100644 index 0000000..ecec213 --- /dev/null +++ b/internal/config/testdata/hub/valid/azure-devops.yml @@ -0,0 +1,15 @@ +version: 1 +id: acme-eng +providers: + - id: ado + type: azure-devops + url: https://dev.azure.com/acme + writer: 3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d + known_authors: [0b7e5a2c-9d4f-4e1b-8a3c-6f5d2e1c0b9a] + - id: ado-other + type: azure-devops + url: https://dev.azure.com/Fabrikam-Fiber/ + - id: ado-test + type: azure-devops + url: http://localhost:8080/acme + api_url: http://localhost:8080/acme diff --git a/internal/config/testdata/hub/valid/shorthand-azure-devops.yml b/internal/config/testdata/hub/valid/shorthand-azure-devops.yml new file mode 100644 index 0000000..033b74f --- /dev/null +++ b/internal/config/testdata/hub/valid/shorthand-azure-devops.yml @@ -0,0 +1,5 @@ +version: 1 +id: acme-eng +platform: azure-devops +base_url: https://dev.azure.com/acme +writer: 3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d diff --git a/internal/config/urls.go b/internal/config/urls.go index c810f57..65ea3ac 100644 --- a/internal/config/urls.go +++ b/internal/config/urls.go @@ -106,6 +106,12 @@ func siteOf(raw string) (site, bool) { return site{origin: scheme + "://" + host, segs: segs}, true } +// same reports whether s and base are the same place: the same origin and +// path segments, compared ignoring case. +func (s site) same(base site) bool { + return s.origin == base.origin && slices.EqualFunc(s.segs, base.segs, strings.EqualFold) +} + // under reports whether s lies strictly beneath base: the same origin, and // base's path segments, compared ignoring case, followed by at least one // more. @@ -153,7 +159,12 @@ func (t *Targets) HasURLs() bool { // namespaces, a repository URL must name owner/name (workspace/repository on // Bitbucket, as in https://bitbucket.org/acme/billing) and an organisation // URL one namespace; on GitLab a URL with a /-/ segment points inside a project and -// is refused. +// is refused. On Azure DevOps, whose provider url is an organization +// (https://dev.azure.com/acme), a repository URL is +// https://dev.azure.com/acme//_git/ and names +// /, and the organization URL itself is the namespace +// of an org entry (the whole organization: a project is selected with +// match). // // The result is a copy: targets is left as it is, and returned as it is // when it holds no URL. Errors name the field of each URL that did not @@ -233,7 +244,9 @@ func (r urlResolver) resolve(s string, k urlKind, want, fallback string) (string } var match []int for i, ps := range r.sites { - if ps.origin != "" && u.under(ps) { + // An Azure DevOps organization is the provider's url itself. + whole := k == urlNamespace && r.providers[i].Type == "azure-devops" && u.same(ps) + if ps.origin != "" && (u.under(ps) || whole) { match = append(match, i) } } @@ -261,7 +274,14 @@ func (r urlResolver) resolve(s string, k urlKind, want, fallback string) (string s, r.list(match)) } p := r.providers[match[0]] - segs := slices.Clone(u.segs[len(r.sites[match[0]].segs):]) + base := r.sites[match[0]].segs + segs := slices.Clone(u.segs[len(base):]) + if p.Type == "azure-devops" { + var err error + if segs, err = azureSegments(segs, base, k); err != nil { + return "", fmt.Errorf("%s: %w", s, err) + } + } if last := len(segs) - 1; k != urlNamespace && len(segs[last]) > len(".git") && strings.HasSuffix(strings.ToLower(segs[last]), ".git") { // A repository URL may end in .git; dropping it from a pattern // would widen it (acme/*.git to acme/*). @@ -305,6 +325,30 @@ func (r urlResolver) list(indexes []int) string { return strings.Join(parts, ", ") } +// azureSegments returns the path an Azure DevOps URL of kind k names, from +// segs, its path segments after the provider's url base (the +// organization): the organization itself for an org entry, which takes the +// organization URL only; / for a repository URL, +// /_git/; and for an exclude entry the same, or a +// pattern of the project's repositories (Legacy/*, Legacy/**). +func azureSegments(segs, base []string, k urlKind) ([]string, error) { + const repoForm = "an Azure DevOps repository URL is https://dev.azure.com///_git/" + switch { + case k == urlNamespace && len(segs) == 0 && len(base) > 0: + return []string{base[len(base)-1]}, nil + case k == urlNamespace: + return nil, errors.New("an Azure DevOps org entry names the whole organization, the provider's url https://dev.azure.com/; " + + "select a project's repositories with match: (/*)") + case len(segs) == 3 && segs[1] == "_git": + return []string{segs[0], segs[2]}, nil + case k == urlExclude && !slices.Contains(segs, "_git") && (len(segs) == 2 && isGlob(segs[1]) || slices.Contains(segs, "**")): + return segs, nil + case k == urlExclude: + return nil, errors.New(repoForm + ", or a pattern of a project's repositories like https://dev.azure.com///*") + } + return nil, errors.New(repoForm) +} + // checkURLShape checks the path segments a URL of kind k names on a // platform of type typ: GitHub, Gitea, Forgejo and Bitbucket have no nested // namespaces, so an exclude URL of more than owner/name, without a "**" diff --git a/internal/config/urls_test.go b/internal/config/urls_test.go index db0fab9..620e8d3 100644 --- a/internal/config/urls_test.go +++ b/internal/config/urls_test.go @@ -376,3 +376,70 @@ providers: } } } + +// TestResolveURLsAzureDevOps: an Azure DevOps provider is one organization; +// its repository URLs are /_git/, and the organization +// URL itself is the namespace of an org entry. +func TestResolveURLsAzureDevOps(t *testing.T) { + provs := resolvedProviders(t, `version: 1 +id: acme-eng +providers: + - id: ado + type: azure-devops + url: https://dev.azure.com/acme + - id: other + type: azure-devops + url: https://dev.azure.com/fabrikam +`) + for _, tc := range []struct { + entry Entry + want string // the resolved value, or a substring of the error after "!" + }{ + {Entry{Repo: "https://dev.azure.com/acme/Billing/_git/api"}, "ado:Billing/api"}, + {Entry{Repo: "https://dev.azure.com/ACME/Billing/_git/api/"}, "ado:Billing/api"}, + {Entry{Repo: "https://dev.azure.com/fabrikam/Web/_git/site"}, "other:Web/site"}, + {Entry{Org: "https://dev.azure.com/acme"}, "ado:acme"}, + {Entry{Org: "https://dev.azure.com/acme/"}, "ado:acme"}, + {Entry{Org: "https://dev.azure.com/acme/Billing"}, "!an Azure DevOps org entry names the whole organization"}, + {Entry{Repo: "https://dev.azure.com/acme/Billing/api"}, "!an Azure DevOps repository URL is https://dev.azure.com///_git/"}, + {Entry{Repo: "https://dev.azure.com/acme/Billing/_git/api/pullrequest/3"}, "!an Azure DevOps repository URL is"}, + {Entry{Repo: "https://dev.azure.com/contoso/Billing/_git/api"}, "!is not under the url of any provider"}, + } { + got, err := ResolveURLs(&Targets{Targets: []Entry{tc.entry}}, provs) + if want, ok := strings.CutPrefix(tc.want, "!"); ok { + if err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("%+v: error %v, want one with %q", tc.entry, err, want) + } + continue + } + if err != nil { + t.Errorf("%+v: %v", tc.entry, err) + continue + } + if v := selectorValue(&got.Targets[0]); v != tc.want { + t.Errorf("%+v: %q, want %q", tc.entry, v, tc.want) + } + } + for _, tc := range []struct{ ex, want string }{ + {"https://dev.azure.com/acme/Legacy/_git/old", "ado:Legacy/old"}, + {"https://dev.azure.com/acme/Legacy/_git/old-*", "ado:Legacy/old-*"}, + {"https://dev.azure.com/acme/Legacy/*", "ado:Legacy/*"}, + {"https://dev.azure.com/acme/Legacy/**", "ado:Legacy/**"}, + {"https://dev.azure.com/acme/Legacy/old", "!or a pattern of a project's repositories"}, + } { + got, err := ResolveURLs(&Targets{Exclude: []string{tc.ex}}, provs) + if want, ok := strings.CutPrefix(tc.want, "!"); ok { + if err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("exclude %s: error %v, want one with %q", tc.ex, err, want) + } + continue + } + if err != nil { + t.Errorf("exclude %s: %v", tc.ex, err) + continue + } + if got.Exclude[0] != tc.want { + t.Errorf("exclude %s: %q, want %q", tc.ex, got.Exclude[0], tc.want) + } + } +} diff --git a/internal/distribute/commit.go b/internal/distribute/commit.go index 0cacce9..2031098 100644 --- a/internal/distribute/commit.go +++ b/internal/distribute/commit.go @@ -424,13 +424,15 @@ func (r *run) commitPerson(p *provider) gitx.Person { // Without it, the author is "@touchmark.invalid", the account's UUID // without braces under a domain that can never exist (RFC 6761, .invalid): // the commit then names the bot by its UUID, links to no account, and -// can never be taken for someone else's. +// can never be taken for someone else's. Azure DevOps has none either: its +// driver gives the user's sign-in address when connectionData shows one, +// else the author is "@touchmark.invalid". func noReplyEmail(typ, host, id, login string) string { if h, _, err := net.SplitHostPort(host); err == nil { host = h } switch typ { - case "bitbucket": + case "bitbucket", "azure-devops": return strings.Trim(cmp.Or(id, login), "{}") + "@touchmark.invalid" case "gitlab": if id != "" { diff --git a/internal/distribute/keptbody_test.go b/internal/distribute/keptbody_test.go new file mode 100644 index 0000000..10003aa --- /dev/null +++ b/internal/distribute/keptbody_test.go @@ -0,0 +1,56 @@ +package distribute + +import ( + "errors" + "testing" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" + "github.com/bedrock-python/touchmark/internal/platform/fake" + "github.com/bedrock-python/touchmark/internal/prbody" +) + +// TestKeptBody: a body touchmark writes back is, on every platform that +// keeps the marker in the body, byte for byte what it always was: people's +// text made inert, the new marker last. Where the marker lives apart (Azure +// DevOps), a description touchmark leaves as read stays as read, people's +// text and all, and only its marker line is checked as touchmark's text; a +// description touchmark changes (a control ticked off) is made inert as +// anywhere. +func TestKeptBody(t *testing.T) { + t.Parallel() + const ( + oldLine = "" + newLine = "" + ) + read := "Sync.\n\n- [x] Rebuild the branch\n/label ~smuggled\ncc @bob\n\n" + oldLine + unticked := prbody.Untick(read, prbody.ControlRecreate) + if unticked == read { + t.Fatal("fixture: Untick changed nothing") + } + for _, f := range []fake.Flavor{fake.GitHub, fake.GitLab, fake.Gitea, fake.Forgejo, fake.Bitbucket, fake.AzureDevOps} { + x := &targetExec{t: &target{prov: &provider{caps: fake.CapsFor(f)}}} + for _, body := range []string{read, unticked} { + got, kept := x.keptBody(read, body, newLine) + inert := prbody.ReplaceMarker(prbody.Inert(marker.Strip(body)), newLine) + if f != fake.AzureDevOps || body != read { + if got != inert || kept { + t.Errorf("%s: keptBody = %q (kept %v), want %q", f, got, kept, inert) + } + continue + } + desc, line := marker.Detach(got) + if !kept || desc != marker.Strip(read) || line != newLine { + t.Errorf("%s: keptBody = %q (kept %v): the description must stay as read", f, got, kept) + } + act := writeAct{kind: actEdit, pr: 1, edit: platform.PREdit{Body: &got}, kept: kept} + if err := unsafeText([]writeAct{act}); err != nil { + t.Errorf("%s: a kept description is checked as touchmark's text: %v", f, err) + } + act.kept = false + if err := unsafeText([]writeAct{act}); !errors.Is(err, prbody.ErrUnsafe) { + t.Errorf("%s: people's text sent as is passes the check: %v", f, err) + } + } + } +} diff --git a/internal/distribute/run_azuredevops_test.go b/internal/distribute/run_azuredevops_test.go new file mode 100644 index 0000000..c20190c --- /dev/null +++ b/internal/distribute/run_azuredevops_test.go @@ -0,0 +1,355 @@ +package distribute + +import ( + "net/http" + "slices" + "strings" + "testing" + "time" + + "github.com/bedrock-python/touchmark/internal/decide" + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" + "github.com/bedrock-python/touchmark/internal/platform/fake" + "github.com/bedrock-python/touchmark/internal/report" +) + +// The delivery scenarios on Azure DevOps' flavor of the fake: the marker +// lives in a pull request property, apart from a description of at most +// 4 000 bytes (Caps.MarkerInProperties); labels go by name; abandoned pull +// requests are never written to again (Caps.ClosedImmutable). Every run +// goes through Run's full phase C in git mode. + +// azWorld is a git world of the Azure DevOps flavor. +func azWorld(t *testing.T) *gitWorld { + t.Helper() + g := newGitWorld(t, fake.WithFlavor(fake.AzureDevOps)) + if c := g.p.Caps(); !c.ClosedImmutable || c.NoLabels || !c.BodyControls() || c.Marker != platform.MarkerInProperties || c.MaxBody != 4000 { + t.Fatalf("fixture: caps %+v", c) + } + return g +} + +// azCheckBody checks what touchmark writes on Azure DevOps: a description +// of at most 4 000 bytes without a marker line, the marker in the comment +// frame kept apart (the body's last line as the driver reads it back), the +// forget_declines hint in the footnote of an open pull request, the hub's +// labels; it returns the marker. +func (g *gitWorld) azCheckBody(pr platform.PR) marker.Marker { + g.t.Helper() + desc, line := marker.Detach(pr.Body) + if len(desc) > 4000 { + g.t.Errorf("#%d: a description of %d bytes", pr.Number, len(desc)) + } + if !strings.HasPrefix(line, " Rebuild\n\n" + line, "- [ ] Rebuild", line}, + } { + t.Run(tc.name, func(t *testing.T) { + desc, got := Detach(tc.body) + if desc != tc.desc || got != tc.line { + t.Fatalf("Detach = %q, %.40q; want %q, %.40q", desc, got, tc.desc, tc.line) + } + back := Attach(desc, got) + if Strip(back) != Strip(tc.body) { + t.Errorf("Strip(Attach) = %q, want %q", Strip(back), Strip(tc.body)) + } + m1, s1 := Find(back, []string{ourFP}) + m2, s2 := Find(tc.body, []string{ourFP}) + if s1 != s2 || m1.Key != m2.Key { + t.Errorf("Find(Attach) = %v %s, Find(body) = %v %s", s1, m1.Key, s2, m2.Key) + } + }) + } + // A marker pasted into the description never counts: only the stored + // line does. + if got := Attach("text\n\n"+other, line); got != "text\n\n"+line { + t.Errorf("Attach with a marker in the description = %q", got) + } + if got := Attach("text\n\n"+other, ""); got != "text" { + t.Errorf("Attach without a stored line = %q", got) + } + if got := Attach("", ""); got != "" { + t.Errorf("Attach of nothing = %q", got) + } +} diff --git a/internal/marker/fuzz_test.go b/internal/marker/fuzz_test.go index 9789dfe..5c93365 100644 --- a/internal/marker/fuzz_test.go +++ b/internal/marker/fuzz_test.go @@ -185,3 +185,28 @@ func FuzzFind(f *testing.F) { } }) } + +// FuzzDetach: a body split for a platform that keeps the marker apart and +// joined again keeps its human part, and the line kept apart is a marker +// line or nothing. +func FuzzDetach(f *testing.F) { + for _, line := range seedLines(f) { + f.Add("text\r\n\r\n" + line + " \n") + } + f.Add("") + f.Fuzz(func(t *testing.T, body string) { + desc, line := Detach(body) + if line != "" && !IsLine(line) { + t.Fatalf("Detach kept %.80q apart, no marker line", line) + } + if Strip(desc) != desc { + t.Fatal("the description keeps a marker line or trailing blanks") + } + if got := Strip(Attach(desc, line)); got != Strip(body) { + t.Fatalf("Strip(Attach(Detach)) = %.80q, want %.80q", got, Strip(body)) + } + if d2, l2 := Detach(Attach(desc, line)); d2 != desc || l2 != line { + t.Fatal("Detach(Attach(Detach(body))) differs from Detach(body)") + } + }) +} diff --git a/internal/platform/azuredevops/azuredevops.go b/internal/platform/azuredevops/azuredevops.go new file mode 100644 index 0000000..1550128 --- /dev/null +++ b/internal/platform/azuredevops/azuredevops.go @@ -0,0 +1,130 @@ +// Package azuredevops is the platform driver for Azure DevOps Services +// (dev.azure.com, REST API 7.1). One provider is one organization: +// https://dev.azure.com/. Azure DevOps Server (on premises) +// and the old https://.visualstudio.com URLs are not served. +// +// NewReader serves plan; NewWriter serves distribute and doctor: it opens, +// edits, abandons and comments on pull requests, labels them, keeps the +// marker in a pull request property, pushes over git, and reports the +// writer's access for doctor (platform.Checker). +// +// Authentication is a personal access token (PAT) of a user: the reader's +// with the scope Code (read), the writer's with Code (read & write), of two +// different users. The REST API gets HTTP Basic with an empty user name and +// the token as the password, git over HTTPS Basic with any non-empty user +// name (https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate, +// "Use a PAT in your code"; +// https://learn.microsoft.com/en-us/azure/devops/repos/git/auth-overview, +// "Git interactions require a username, which can be anything except an +// empty string"). Every API request sends X-TFS-FedAuthRedirect: Suppress, +// so that a refused credential answers 401 instead of a redirect to the +// sign-in page; a redirect to sign-in, or the 203 with an HTML page that +// Azure DevOps answers some refused credentials with, is ClassAuth all the +// same. Service principals of Microsoft Entra ID (Bearer tokens) come later. +// +// What the driver relies on, from the REST reference +// (https://learn.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-7.1), +// the concept pages it links, and anonymous answers of public projects +// (2026-10-09): +// - accounts are identities known by their id, a GUID, which never +// changes; displayName is neither unique nor stable and uniqueName is +// empty for service identities, so a login in hub.yml is a GUID, and +// Account.ID and Account.Login are that GUID. GET +// {org}/_apis/connectionData names the credential's identity +// (authenticatedUser; anonymous gets the all-a GUID of public access), +// GET https://vssps.dev.azure.com/{org}/_apis/identities?identityIds= +// another; +// - an organization holds projects, a project holds repositories; there +// are no topics and no nested groups. A repository is +// /; its id is a GUID, and repository-scoped calls +// go through {org}/_apis/git/repositories/{id} without the project. An +// empty repository has no defaultBranch; a disabled one says isDisabled +// (and shows no default branch either); +// - files: the Items API describes a path (objectId, gitObjectType, +// isSymLink, commitId), the Trees API a folder's entries with their git +// mode and size, the Blobs API a blob's bytes ($format=octetstream). A +// 404 tells what is missing by its typeKey: GitItemNotFoundException +// (the path), GitUnresolvableToCommitException (the branch or commit), +// GitRepositoryNotFoundException (the repository); +// - pull requests: status active, completed (merged) or abandoned; +// isDraft; closedBy only when one pull request is read alone (listings +// leave it out and cut descriptions to 400 characters); a description +// holds at most 4 000 characters (Pull Requests - Update); labels are +// added by name (Pull Request Labels - Create); an abandoned pull +// request could be reactivated, which touchmark never does; properties +// (Pull Request Properties) carry the marker under touchmark.marker; +// - limits are in TSTUs over a sliding five minutes, announced in +// X-RateLimit-Remaining and X-RateLimit-Reset (which internal/throttle +// reads from every answer) and in Retry-After, which the driver reads +// from refusals (a rate limit) and from answers that went through +// delayed (a pause of the throttle, without a strike); a blocked +// request is 429 with TF400733 (https://learn.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits); +// - branch policies refuse direct pushes to a branch they protect +// (TF402455) and a missing permission refuses a push with TF401027; the +// writer reads neither upfront. +// +// What only a live organization can confirm is marked "assumed" where the +// driver relies on it, and listed in docs/guide/providers.md. +// +// The driver never retries or paces (internal/throttle does), with one +// exception: the marker property of a pull request CreatePR has just +// opened, which is tried again a few times and, should it still fail, gets +// the pull request abandoned (an own pull request without its marker would +// block its target). It knows the marker only as an opaque line it stores +// apart (marker.Detach, marker.Attach: platform.MarkerInProperties), and +// masks secrets in its errors. +package azuredevops + +import ( + "errors" + + "github.com/bedrock-python/touchmark/internal/auth" + "github.com/bedrock-python/touchmark/internal/config" + "github.com/bedrock-python/touchmark/internal/httpx" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// NewReader returns the read driver for provider p (type "azure-devops") +// with credential c (an auth.Token, a PAT with Code (read)) over client, +// which sends the token only to the organization's API hosts. A zero +// credential reads anonymously, and gets little: public projects only, and +// even there Azure DevOps sends anonymous reads of the Trees API (ReadFile) +// and of pull request properties (PRs, the marker) to its sign-in page, +// which the driver reports as ClassAuth: an anonymous reader serves +// listings of public repositories, not a plan. +func NewReader(p config.ResolvedProvider, c auth.Credential, client *httpx.Client) (platform.Reader, error) { + cl, err := newClient(p, c, client) + if err != nil { + return nil, err + } + return &reader{c: cl}, nil +} + +// NewWriter returns the write driver for provider p with credential c, a +// PAT with Code (read & write) of the user that writes (a write identity is +// never anonymous). Target checks the user's permissions on the repository +// and returns a TargetWriter bound to it; Azure DevOps cannot mint narrower +// tokens, so Close only retires the TargetWriter: later calls through it +// fail with ClassAuth. +func NewWriter(p config.ResolvedProvider, c auth.Credential, client *httpx.Client) (platform.Writer, error) { + if c.Kind == 0 { + return nil, errors.New("azure-devops: the write driver needs a token") + } + cl, err := newClient(p, c, client) + if err != nil { + return nil, err + } + return &writer{reader: reader{c: cl}}, nil +} + +// reader is platform.Reader over one identity. +type reader struct{ c *client } + +// writer is platform.Writer: a reader that can mint TargetWriters. +type writer struct{ reader } + +var ( + _ platform.Reader = (*reader)(nil) + _ platform.Writer = (*writer)(nil) + _ platform.Checker = (*writer)(nil) +) diff --git a/internal/platform/azuredevops/check.go b/internal/platform/azuredevops/check.go new file mode 100644 index 0000000..c571e8a --- /dev/null +++ b/internal/platform/azuredevops/check.go @@ -0,0 +1,99 @@ +package azuredevops + +import ( + "context" + "fmt" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// Check reports the writer's checks for doctor (platform.Checker). +// +// With no repository, the identity's: "token-expiry" and "scopes" are +// unknown, since reading a PAT's expiry and scopes takes the PAT lifecycle +// API with a Microsoft Entra token, not the PAT itself; "2fa" is unknown +// too, being the directory's business; the identity is read first +// (connectionData), so a refused token fails the call. +// +// With repositories, per repository: "access" from Has Permissions on the +// Git Repositories namespace (contribute, create branches and contribute to +// pull requests: ok; one missing: fail; the API not open to the token: +// unknown), fail when the writer does not see the repository; "rules" +// unknown: branch policies and push policies refuse a push at run time +// (TF402455), and the writer reads none upfront. +func (w *writer) Check(ctx context.Context, repos []platform.Repo, _ []string) ([]platform.Finding, error) { + if len(repos) == 0 { + return w.identityChecks(ctx) + } + var out []platform.Finding + for _, r := range repos { + fs, err := w.repoChecks(ctx, r) + if err != nil { + return nil, err + } + out = append(out, fs...) + } + return out, nil +} + +// identityChecks are the checks of the writer's token and identity. +func (w *writer) identityChecks(ctx context.Context) ([]platform.Finding, error) { + if _, err := w.c.selfAccount(ctx); err != nil { + return nil, err + } + return []platform.Finding{ + {Check: "token-expiry", Status: platform.FindingUnknown, + Detail: "a personal access token cannot read its own expiry; the user's Personal access tokens page in Azure DevOps shows it (global PATs stop working on 2026-12-01)"}, + {Check: "scopes", Status: platform.FindingUnknown, + Detail: "a personal access token cannot read its own scopes; the writer's token needs Code (read & write) for this organization only, and nothing more"}, + {Check: "2fa", Status: platform.FindingUnknown, + Detail: "Azure DevOps leaves multi-factor authentication to Microsoft Entra ID, which the token cannot read"}, + }, nil +} + +// repoChecks are the checks of one repository. +func (w *writer) repoChecks(ctx context.Context, r platform.Repo) ([]platform.Finding, error) { + const op = "check" + find := func(check string, status platform.FindingStatus, format string, args ...any) platform.Finding { + return platform.Finding{Repo: r.Path, Check: check, Status: status, Detail: fmt.Sprintf(format, args...)} + } + rules := find("rules", platform.FindingUnknown, "branch policies and push policies are met at push time (TF402455); the writer reads none upfront") + repo, err := w.c.targetRepo(ctx, op, r) + switch { + case err == nil: + case stops(err): + return nil, err + case platform.ClassOf(err) == platform.ClassNotFound: + return []platform.Finding{find("access", platform.FindingFail, "the writer does not see this repository")}, nil + default: + return []platform.Finding{find("access", platform.FindingUnknown, "%v", err), rules}, nil + } + var missing []string + for _, p := range []struct { + bits int + name string + }{ + {permContribute, "Contribute"}, + {permCreateBranch, "Create branch"}, + {permPullRequestContib, "Contribute to pull requests"}, + } { + ok, known, err := w.c.hasPermission(ctx, op, repo, p.bits) + switch { + case err != nil && stops(err): + return nil, err + case err != nil: + return []platform.Finding{find("access", platform.FindingUnknown, "%v", err), rules}, nil + case !known: + return []platform.Finding{find("access", platform.FindingUnknown, + "the token may not ask Azure DevOps for its permissions (Permissions - Has Permissions); a push or a pull request meets a missing one"), rules}, nil + case !ok: + missing = append(missing, p.name) + } + } + access := find("access", platform.FindingOK, "the writer may push, create branches and contribute to pull requests") + if len(missing) > 0 { + access = find("access", platform.FindingFail, "the writer lacks %v on the repository; give its user these Git repository permissions "+ + "(directly, or through a group such as the project's Contributors)", missing) + } + return []platform.Finding{access, rules}, nil +} diff --git a/internal/platform/azuredevops/check_test.go b/internal/platform/azuredevops/check_test.go new file mode 100644 index 0000000..227e99f --- /dev/null +++ b/internal/platform/azuredevops/check_test.go @@ -0,0 +1,54 @@ +package azuredevops + +import ( + "context" + "fmt" + "net/http" + "testing" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +func TestCheck(t *testing.T) { + s := newAPIServer(t) + targetWorld(t, s, map[int]bool{permContribute: true, permCreateBranch: true}) + w := newTestWriter(t, s, testToken(t)) + ids, err := w.Check(context.Background(), nil, nil) + if err != nil { + t.Fatal(err) + } + if len(ids) != 3 { + t.Fatalf("identity checks %+v", ids) + } + for _, f := range ids { + if f.Status != platform.FindingUnknown || f.Repo != "" { + t.Errorf("identity check %+v", f) + } + } + repo := testRepo() + repo.Host = w.c.host + fs, err := w.Check(context.Background(), []platform.Repo{repo}, nil) + if err != nil { + t.Fatal(err) + } + if len(fs) != 2 || fs[0].Check != "access" || fs[0].Status != platform.FindingFail || fs[1].Check != "rules" || fs[1].Status != platform.FindingUnknown { + t.Errorf("repository checks %+v", fs) + } + + // All allowed: ok; the permissions API closed to the token: unknown. + targetWorld(t, s, allPerms) + if fs, err := w.Check(context.Background(), []platform.Repo{repo}, nil); err != nil || fs[0].Status != platform.FindingOK { + t.Errorf("all allowed: %+v, %v", fs, err) + } + for _, bits := range []int{permContribute, permCreateBranch, permPullRequestContib} { + s.json(apisPath("permissions", gitNamespace, fmt.Sprint(bits)), http.StatusForbidden, errorBody("X", "no")) + } + if fs, err := w.Check(context.Background(), []platform.Repo{repo}, nil); err != nil || fs[0].Status != platform.FindingUnknown { + t.Errorf("no permissions API: %+v, %v", fs, err) + } + // A repository the writer does not see fails. + s.json(repoPath(), http.StatusNotFound, errorBody(keyRepoNotFound, "TF401019")) + if fs, err := w.Check(context.Background(), []platform.Repo{repo}, nil); err != nil || len(fs) != 1 || fs[0].Status != platform.FindingFail { + t.Errorf("an unseen repository: %+v, %v", fs, err) + } +} diff --git a/internal/platform/azuredevops/client.go b/internal/platform/azuredevops/client.go new file mode 100644 index 0000000..fbfa9f8 --- /dev/null +++ b/internal/platform/azuredevops/client.go @@ -0,0 +1,367 @@ +package azuredevops + +import ( + "context" + "encoding/base64" + "errors" + "fmt" + "net/http" + "net/url" + "strconv" + "strings" + "sync" + "time" + "unicode" + + "github.com/bedrock-python/touchmark/internal/auth" + "github.com/bedrock-python/touchmark/internal/config" + "github.com/bedrock-python/touchmark/internal/httpx" + "github.com/bedrock-python/touchmark/internal/platform" + "github.com/bedrock-python/touchmark/internal/redact" + "github.com/bedrock-python/touchmark/internal/throttle" +) + +// apiVersion is the REST API version every request asks for. +const apiVersion = "7.1" + +// gitUser is the user name of git's Basic credentials: Azure Repos takes +// any non-empty one with a PAT. The REST API gets an empty one (its Basic +// form is ":"). Both forms are registered for masking. +const gitUser = "touchmark" + +// cloudHost is the host of Azure DevOps Services; its identities live under +// vssps.dev.azure.com. +const cloudHost = "dev.azure.com" + +// snippetLimit bounds the body kept from a failed raw request, in bytes, as +// httpx keeps for JSON requests. +const snippetLimit = 1 << 10 + +// client is the HTTP side of one driver: one organization under one +// identity. It is safe for concurrent use. +type client struct { + http *httpx.Client + // org is the organization; api the REST base without a trailing slash + // (https://dev.azure.com/acme), whose /_apis/… are the organization's + // APIs; vssps the base of the identity APIs + // (https://vssps.dev.azure.com/acme; api itself elsewhere, a test + // server). + org, api, vssps string + apiURL *url.URL + // web is the provider's url without a trailing slash; git remotes and + // web links are under it. host is the provider host (Repo.Host). + web, host string + token string // "" when anonymous + auth *httpx.Auth + masks *redact.Registry + + mu sync.Mutex + self *platform.Account // cached by selfAccount +} + +// newClient checks the provider and the credential and builds the client. +func newClient(p config.ResolvedProvider, cred auth.Credential, hc *httpx.Client) (*client, error) { + if p.Type != "azure-devops" { + return nil, fmt.Errorf("azure-devops: provider %s has type %q, not azure-devops", p.ID, p.Type) + } + if hc == nil { + return nil, errors.New("azure-devops: no HTTP client") + } + org, ok := config.AzureDevOpsOrg(p.URL) + if !ok { + return nil, fmt.Errorf("azure-devops: provider %s: url %q does not name one organization", p.ID, p.URL) + } + api, apiURL, err := baseURL("api_url", p.APIURL) + if err != nil { + return nil, fmt.Errorf("azure-devops: provider %s: %w", p.ID, err) + } + web, webURL, err := baseURL("url", p.URL) + if err != nil { + return nil, fmt.Errorf("azure-devops: provider %s: %w", p.ID, err) + } + host := strings.ToLower(p.Host) + if host == "" { + host = strings.ToLower(webURL.Host) + } + c := &client{ + http: hc, + org: org, + api: api, + apiURL: apiURL, + vssps: api, + web: web, + host: host, + masks: redact.New(), + } + hosts := []string{strings.ToLower(apiURL.Host)} + if strings.EqualFold(apiURL.Host, cloudHost) { + c.vssps = "https://vssps." + cloudHost + "/" + url.PathEscape(org) + hosts = append(hosts, "vssps."+cloudHost) + } + switch cred.Kind { + case 0: + if cred.Token != "" || cred.AppID != "" || len(cred.AppKey) > 0 { + return nil, errors.New("azure-devops: a credential without a kind") + } + case auth.Token: + if cred.Token == "" { + return nil, errors.New("azure-devops: an empty token") + } + c.token = cred.Token + c.masks.Add(cred.Token, "", gitUser) + c.auth = &httpx.Auth{Hosts: hosts, Header: c.apiHeader} + case auth.App: + return nil, fmt.Errorf("azure-devops: provider %s: a GitHub App cannot sign in to Azure DevOps; use a personal access token of a user", p.ID) + default: + return nil, fmt.Errorf("azure-devops: provider %s: unknown credential kind %d", p.ID, cred.Kind) + } + return c, nil +} + +// baseURL checks an http(s) base URL without credentials, query or +// fragment, and returns it without a trailing slash. +func baseURL(what, raw string) (string, *url.URL, error) { + s := strings.TrimRight(raw, "/") + u, err := url.Parse(s) + switch { + case s == "": + return "", nil, fmt.Errorf("%s is empty", what) + case err != nil: + return "", nil, fmt.Errorf("%s does not parse", what) + case u.Scheme != "https" && u.Scheme != "http": + return "", nil, fmt.Errorf("%s is not an http or https URL", what) + case u.Host == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "": + return "", nil, fmt.Errorf("%s must be a URL with a host and without credentials, query or fragment", what) + } + return s, u, nil +} + +// apiHeader is the Authorization header of API requests: Basic with an +// empty user name and the PAT. +func (c *client) apiHeader(context.Context) (string, error) { + return "Basic " + base64.StdEncoding.EncodeToString([]byte(":"+c.token)), nil +} + +// gitHeader is the Authorization header of git over HTTPS: Basic with a +// non-empty user name and the PAT. +func (c *client) gitHeader(ctx context.Context) (string, error) { + if err := ctx.Err(); err != nil { + return "", err + } + return "Basic " + base64.StdEncoding.EncodeToString([]byte(gitUser+":"+c.token)), nil +} + +// mask replaces the credential's forms in s. +func (c *client) mask(s string) string { return c.masks.Replace(s) } + +// endpoint joins base and path segments, each escaped. +func endpoint(base string, segments ...string) string { + var b strings.Builder + b.WriteString(base) + for _, s := range segments { + b.WriteByte('/') + b.WriteString(url.PathEscape(s)) + } + return b.String() +} + +// apis returns the organization's API URL of segments: {api}/_apis/…. +func (c *client) apis(segments ...string) string { + return endpoint(c.api, append([]string{"_apis"}, segments...)...) +} + +// repoAPI returns the URL of a repository's API, by its id, without the +// project: {api}/_apis/git/repositories/{id}/…. +func (c *client) repoAPI(id string, segments ...string) string { + return c.apis(append([]string{"git", "repositories", id}, segments...)...) +} + +// headers are sent with every API request: JSON, and no redirect to the +// sign-in page for a refused credential (401 instead). +func headers(accept string) http.Header { + h := http.Header{} + h.Set("Accept", accept) + h.Set("X-TFS-FedAuthRedirect", "Suppress") + return h +} + +// call sends one API request of op with api-version and query, a JSON +// body in (nil for none; sent as contentType when it is not ""), and +// decodes a 2xx JSON response into out (nil skips decoding). Every error is +// a masked *platform.Error, but a canceled context's. +func (c *client) call(ctx context.Context, op, method, u string, query url.Values, contentType string, in, out any) (*httpx.Response, error) { + q := url.Values{} + for k, v := range query { + q[k] = v + } + q.Set("api-version", apiVersion) + u += "?" + q.Encode() + h := headers("application/json") + if contentType != "" { + h.Set("Content-Type", contentType) + } + resp, err := c.http.JSONWith(ctx, method, u, c.auth, h, in, out) + if resp != nil && signIn(resp.Status, resp.Header) { + return resp, c.signInError(op, resp.Status) + } + if err != nil { + return resp, c.apiError(op, err) + } + slowDown(ctx, resp.Status, resp.Header) + return resp, nil +} + +// slowDown pauses the provider's throttle when a successful answer +// carries Retry-After: Azure DevOps delays the requests of an identity +// over its threshold and asks it, in Retry-After, to wait before the next +// one so as not to be delayed or blocked +// (https://learn.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits). +// The wait goes to the Meter of ctx as an exhausted budget +// (throttle.Observe: a pause without a strike), bounded by maxRetryAfter; +// nothing without a Meter. A refusal with Retry-After is a rate limit +// instead (statusError). +func slowDown(ctx context.Context, status int, h http.Header) { + if status < 200 || status > 299 || strings.TrimSpace(h.Get("Retry-After")) == "" { + return + } + d := retryAfter(http.Header{"Retry-After": h.Values("Retry-After")}, now()) + if d <= 0 { + return + } + secs := int64((d + time.Second - 1) / time.Second) + throttle.Observe(ctx, http.Header{"X-Ratelimit-Remaining": {"0"}, "X-Ratelimit-Reset": {strconv.FormatInt(secs, 10)}}) +} + +// get is call with GET. +func (c *client) get(ctx context.Context, op, u string, query url.Values, out any) (*httpx.Response, error) { + return c.call(ctx, op, http.MethodGet, u, query, "", nil, out) +} + +// signIn reports whether a response is Azure DevOps sending a refused +// credential to its sign-in page: a redirect to _signin (or to another +// host), or a 203 with HTML, which some refusals get even with +// X-TFS-FedAuthRedirect (assumed from the observed redirect of an +// anonymous request for a private resource, 2026-10-09). +func signIn(status int, h http.Header) bool { + switch status { + case http.StatusNonAuthoritativeInfo: + return strings.Contains(strings.ToLower(h.Get("Content-Type")), "html") + case http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther, http.StatusTemporaryRedirect: + loc := strings.ToLower(h.Get("Location")) + return strings.Contains(loc, "_signin") || strings.Contains(loc, "/signin") || strings.Contains(loc, "login.microsoftonline") + } + return false +} + +// signInError is the ClassAuth error of a redirect to the sign-in page. +func (c *client) signInError(op string, status int) error { + why := "Azure DevOps sent the request to its sign-in page: the credential is missing, expired or refused" + if c.token == "" { + why = "Azure DevOps sent the anonymous request to its sign-in page: it answers some reads only with a token, " + + "even in public projects (the Trees API, pull request properties); give the reader a personal access token with Code (read)" + } + return &platform.Error{Op: op, Class: platform.ClassAuth, Status: status, Err: errors.New(why)} +} + +// raw reads the bytes of u with api-version and query, a GET outside JSON +// (a blob's content). A non-2xx answer is classified as call classifies it. +func (c *client) raw(ctx context.Context, op, u string, query url.Values) ([]byte, error) { + q := url.Values{} + for k, v := range query { + q[k] = v + } + q.Set("api-version", apiVersion) + full := u + "?" + q.Encode() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, full, nil) + if err != nil { + return nil, &platform.Error{Op: op, Class: platform.ClassInvalid, Err: c.masked(err)} + } + for k, v := range headers("application/octet-stream") { + req.Header[k] = v + } + resp, err := c.http.Do(req, c.auth) + if err != nil { + return nil, c.apiError(op, err) + } + if signIn(resp.Status, resp.Header) { + return nil, c.signInError(op, resp.Status) + } + if resp.Status >= 200 && resp.Status <= 299 { + slowDown(ctx, resp.Status, resp.Header) + return resp.Body, nil + } + return nil, c.statusError(op, &httpx.StatusError{ + Method: http.MethodGet, URL: u, Status: resp.Status, + Header: resp.Header, Snippet: snippet(resp.Body), + }) +} + +// snippet returns at most snippetLimit bytes of body on one line. +func snippet(body []byte) string { + if len(body) > snippetLimit { + body = body[:snippetLimit] + } + return strings.Join(strings.FieldsFunc(strings.ToValidUTF8(string(body), "?"), func(r rune) bool { + return unicode.IsSpace(r) || unicode.IsControl(r) + }), " ") +} + +// list is a collection answer: {"count": n, "value": [...]}. +type list[T any] struct { + Count int `json:"count"` + Value []T `json:"value"` +} + +// pageSize is the $top of the paged listings (pull requests). +const pageSize = 100 + +// listPages reads a collection of T at u with query, $top pageSize and +// growing $skip, and calls each for every item in order, until a page is +// short. complete is false when the listing was capped at maxPages. An +// error of a page after the first comes with the page in err (pageError). +func listPages[T any](ctx context.Context, c *client, op, u string, query url.Values, maxPages int, each func(T) error) (complete bool, err error) { + for n := 0; n < maxPages; n++ { + q := url.Values{} + for k, v := range query { + q[k] = v + } + q.Set("$top", fmt.Sprint(pageSize)) + q.Set("$skip", fmt.Sprint(n*pageSize)) + var p list[T] + if _, err := c.get(ctx, op, u, q, &p); err != nil { + if n > 0 { + return false, &pageError{page: n + 1, err: err} + } + return false, err + } + for _, it := range p.Value { + if err := each(it); err != nil { + return false, err + } + } + if len(p.Value) < pageSize { + return true, nil + } + } + return false, nil +} + +// pageError is the failure of a page after the first of a listing. +type pageError struct { + page int + err error +} + +func (e *pageError) Error() string { return fmt.Sprintf("page %d: %v", e.page, e.err) } +func (e *pageError) Unwrap() error { return e.err } + +// splitRepoPath splits "project/repository". Azure DevOps has no nested +// projects: a path of another shape names no repository (ok false). +func splitRepoPath(p string) (project, name string, ok bool) { + project, name, found := strings.Cut(p, "/") + if !found || project == "" || name == "" || strings.Contains(name, "/") || + strings.ContainsAny(p, "\x00?#") || project == "." || project == ".." || name == "." || name == ".." { + return "", "", false + } + return project, name, true +} diff --git a/internal/platform/azuredevops/errors.go b/internal/platform/azuredevops/errors.go new file mode 100644 index 0000000..a58619c --- /dev/null +++ b/internal/platform/azuredevops/errors.go @@ -0,0 +1,349 @@ +package azuredevops + +import ( + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "math" + "net/http" + "strconv" + "strings" + "time" + "unicode" + + "github.com/bedrock-python/touchmark/internal/httpx" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// maxRetryAfter bounds a wait read from response headers: a broken or +// hostile header must not park the provider for days. +const maxRetryAfter = time.Hour + +// now is the clock of Retry-After dates and reset times (tests replace it). +var now = time.Now + +// typeKeys of the 404s the driver tells apart (the typeKey of Azure +// DevOps' error body; the REST reference of Items - Get and the answers of +// public repositories, 2026-10-09). +const ( + keyItemNotFound = "GitItemNotFoundException" + keyUnresolvable = "GitUnresolvableToCommitException" + keyRepoNotFound = "GitRepositoryNotFoundException" + keyPRExists = "GitPullRequestExistsException" + keyProjectNotFound = "ProjectDoesNotExistWithNameException" +) + +// Error codes in messages: TF400733 is a request blocked by the rate limit +// (https://learn.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits), +// TF400813 a credential the organization does not authorize, TF401179 an +// active pull request from the same source and target that exists already +// (assumed: the message Azure Repos is known to send; to confirm live). +const ( + tfRateLimited = "TF400733" + tfNotAuthed = "TF400813" + tfPRExists = "TF401179" +) + +// maskedError carries a masked message over the original error, which +// errors.Is and errors.As still see. +type maskedError struct { + msg string + err error +} + +func (e *maskedError) Error() string { return e.msg } +func (e *maskedError) Unwrap() error { return e.err } + +// masked returns err with its message masked. +func (c *client) masked(err error) error { + return &maskedError{msg: c.mask(err.Error()), err: err} +} + +// apiError classifies err, the failure of an API request of op, into a +// platform.Error: a non-2xx status by its code, a timeout as transient, a +// request httpx refused as invalid, a certificate the client does not +// trust, a response over the size bound or of an unexpected shape as +// unknown, any other transport failure as transient. A canceled context +// stays unclassified. Messages never hold the credential nor headers. +func (c *client) apiError(op string, err error) error { + var se *httpx.StatusError + if errors.As(err, &se) { + return c.statusError(op, se) + } + class := platform.ClassTransient + switch { + case errors.Is(err, context.Canceled): + return fmt.Errorf("%s: %w", op, c.masked(err)) + case errors.Is(err, context.DeadlineExceeded): + case errors.Is(err, httpx.ErrRefused): + class = platform.ClassInvalid + case errors.Is(err, httpx.ErrTooLarge), certificateError(err), decodeError(err): + class = platform.ClassUnknown + } + return &platform.Error{Op: op, Class: class, Err: c.masked(err)} +} + +// certificateError reports whether err is a TLS certificate failure. +func certificateError(err error) bool { + var ( + verify *tls.CertificateVerificationError + unknown x509.UnknownAuthorityError + invalid x509.CertificateInvalidError + hostError x509.HostnameError + ) + return errors.As(err, &verify) || errors.As(err, &unknown) || errors.As(err, &invalid) || errors.As(err, &hostError) +} + +// decodeError reports whether err is a JSON body that does not decode. +func decodeError(err error) bool { + var ( + syntax *json.SyntaxError + typ *json.UnmarshalTypeError + ) + return errors.As(err, &syntax) || errors.As(err, &typ) || errors.Is(err, errShape) +} + +// errShape is wrapped by errors about responses that decode but lack what +// the driver depends on. +var errShape = errors.New("unexpected response") + +// shapeError is an error of op about a response that lacks what the driver +// depends on: the API changed shape. +func shapeError(op, format string, args ...any) error { + return &platform.Error{Op: op, Class: platform.ClassUnknown, + Err: fmt.Errorf("%w: "+format, append([]any{errShape}, args...)...)} +} + +// apiMessage is Azure DevOps' error body: {"$id": "1", "innerException": +// null, "message": "TF401174: The item … could not be found …", +// "typeName": "…GitItemNotFoundException, …", "typeKey": +// "GitItemNotFoundException", "errorCode": 0, "eventId": 3000}. +type apiMessage struct { + Message string `json:"message"` + TypeKey string `json:"typeKey"` +} + +// parseMessage decodes an error body; a body of another shape gives the +// zero apiMessage. +func parseMessage(snippet string) apiMessage { + var m apiMessage + if json.Unmarshal([]byte(snippet), &m) != nil { + return apiMessage{} + } + return m +} + +// apiFailure is the error under a classified non-2xx answer: its masked +// description, and the typeKey of the body ("" when none). +type apiFailure struct { + msg string + typeKey string + err error // ErrNotFound for a 404, nil otherwise +} + +func (e *apiFailure) Error() string { return e.msg } +func (e *apiFailure) Unwrap() error { return e.err } + +// typeKeyOf returns the typeKey of the API error in err, "" when there is +// none. +func typeKeyOf(err error) string { + var f *apiFailure + if errors.As(err, &f) { + return f.typeKey + } + return "" +} + +// statusError classifies a non-2xx response of op: 401 auth, 403 +// permission, 404 not found (with its typeKey, which callers read), 409 +// conflict, 429 or TF400733 rate limited, any status with Retry-After rate +// limited too, 5xx and 408 transient, other 4xx invalid. A 401 or a +// TF400813 is auth whatever else it says. +func (c *client) statusError(op string, se *httpx.StatusError) error { + m := parseMessage(se.Snippet) + detail := oneLine(m.Message) + if detail == "" { + detail = se.Snippet + } + if m.TypeKey != "" { + detail += " (" + m.TypeKey + ")" + } + f := &apiFailure{msg: c.mask(describe(se, detail)), typeKey: m.TypeKey} + e := &platform.Error{Op: op, Class: statusClass(se.Status), Status: se.Status, Err: f} + switch { + case se.Status == http.StatusNotFound: + f.err = platform.ErrNotFound + f.msg += ": " + platform.ErrNotFound.Error() + case se.Status == http.StatusTooManyRequests || strings.Contains(m.Message, tfRateLimited) || se.Header.Get("Retry-After") != "": + // Retry-After on any refusal means the platform wants a pause + // (https://learn.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits). + e.Class, e.RetryAfter = platform.ClassRateLimited, retryAfter(se.Header, now()) + case strings.Contains(m.Message, tfNotAuthed): + e.Class = platform.ClassAuth + case se.Status == http.StatusBadRequest && (m.TypeKey == keyPRExists || strings.Contains(m.Message, tfPRExists)): + e.Class = platform.ClassConflict + } + return e +} + +// oneLine joins s's words with single spaces. +func oneLine(s string) string { + return strings.Join(strings.FieldsFunc(s, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }), " ") +} + +// statusClass maps an HTTP status to the class the core reacts to. +func statusClass(status int) platform.Class { + switch { + case status == http.StatusUnauthorized: + return platform.ClassAuth + case status == http.StatusForbidden: + return platform.ClassPermission + case status == http.StatusNotFound, status == http.StatusGone: + return platform.ClassNotFound + case status == http.StatusConflict: + return platform.ClassConflict + case status == http.StatusTooManyRequests: + return platform.ClassRateLimited + case status == http.StatusRequestTimeout: + return platform.ClassTransient + case status == http.StatusNotImplemented: + return platform.ClassUnsupported + case status >= 500: + return platform.ClassTransient + case status >= 400: + return platform.ClassInvalid + } + // An unexpected redirect or informational status. + return platform.ClassUnknown +} + +// describe returns "METHOD URL: status text: detail". +func describe(se *httpx.StatusError, detail string) string { + var b strings.Builder + fmt.Fprintf(&b, "%s %s: %d", se.Method, se.URL, se.Status) + if text := http.StatusText(se.Status); text != "" { + b.WriteString(" " + text) + } + if detail != "" { + b.WriteString(": " + detail) + } + return b.String() +} + +// retryAfter reads how long to wait from a response's headers, in order: +// Retry-After (seconds or an HTTP date), then X-RateLimit-Reset, a Unix +// time (seconds to wait when it is small). It is 0 when none is usable, +// and at most maxRetryAfter. +func retryAfter(h http.Header, at time.Time) time.Duration { + d, ok := time.Duration(0), false + if v := strings.TrimSpace(h.Get("Retry-After")); v != "" { + if secs, err := strconv.ParseInt(v, 10, 64); err == nil { + d, ok = seconds(secs), true + } else if t, err := http.ParseTime(v); err == nil { + d, ok = t.Sub(at), true + } + } + if !ok { + if n, err := strconv.ParseInt(strings.TrimSpace(h.Get("X-RateLimit-Reset")), 10, 64); err == nil { + if n > 1_000_000_000 { + d, ok = time.Unix(n, 0).Sub(at), true + } else { + d, ok = seconds(n), true + } + } + } + switch { + case !ok || d < 0: + return 0 + case d > maxRetryAfter: + return maxRetryAfter + } + return d +} + +// seconds converts a count of seconds, saturating instead of overflowing. +func seconds(n int64) time.Duration { + if n < 0 { + return -1 + } + if n > math.MaxInt64/int64(time.Second) { + return maxRetryAfter + } + return time.Duration(n) * time.Second +} + +// closedError is the error of a TargetWriter used after Close. +func closedError(op string) error { + return &platform.Error{Op: op, Class: platform.ClassAuth, Err: errors.New("the target writer is closed")} +} + +// invalid is a ClassInvalid error of op: the request was wrong. +func invalid(op, format string, args ...any) error { + return &platform.Error{Op: op, Class: platform.ClassInvalid, Err: fmt.Errorf(format, args...)} +} + +// notFound is a ClassNotFound error of op that wraps ErrNotFound. +func notFound(op, format string, args ...any) error { + return &platform.Error{Op: op, Class: platform.ClassNotFound, Err: fmt.Errorf(format+": %w", append(args, platform.ErrNotFound)...)} +} + +// conflict is a ClassConflict error of op. +func conflict(op, format string, args ...any) error { + return &platform.Error{Op: op, Class: platform.ClassConflict, Err: fmt.Errorf(format, args...)} +} + +// unsupported is a ClassUnsupported error of op. +func unsupported(op, format string, args ...any) error { + return &platform.Error{Op: op, Class: platform.ClassUnsupported, Err: fmt.Errorf(format, args...)} +} + +// unknown is a ClassUnknown error of op that says what happened (format +// and args) and, when err is not nil, what err says, with err's status but +// without the ErrNotFound err may wrap: a 404 that must not count as a +// missing file. +func unknown(op string, err error, format string, args ...any) error { + msg := fmt.Sprintf(format, args...) + if err != nil { + msg += ": " + detail(err) + } + // The typeKey stays readable (typeKeyOf); the ErrNotFound does not. + return &platform.Error{Op: op, Class: platform.ClassUnknown, Status: statusOf(err), Err: &apiFailure{msg: msg, typeKey: typeKeyOf(err)}} +} + +// detail returns what err says without its class: the message of the +// *platform.Error in it, without the ErrNotFound suffix. +func detail(err error) string { + var pe *platform.Error + if errors.As(err, &pe) && pe.Err != nil { + return strings.TrimSuffix(pe.Err.Error(), ": "+platform.ErrNotFound.Error()) + } + return err.Error() +} + +// statusOf returns the HTTP status of a classified error, 0 for none. +func statusOf(err error) int { + var pe *platform.Error + if errors.As(err, &pe) { + return pe.Status + } + return 0 +} + +// stops reports whether err ends a call at once, whatever it already +// found: the context ended, the platform limits the rate or refuses the +// credential. +func stops(err error) bool { + if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + return true + } + c := platform.ClassOf(err) + return c == platform.ClassRateLimited || c == platform.ClassAuth +} + +// fatal reports whether err must fail a listing instead of making it +// incomplete: it stops the call, or the platform failed for now (the caller +// retries the whole listing). +func fatal(err error) bool { return stops(err) || platform.ClassOf(err) == platform.ClassTransient } diff --git a/internal/platform/azuredevops/errors_test.go b/internal/platform/azuredevops/errors_test.go new file mode 100644 index 0000000..2130f56 --- /dev/null +++ b/internal/platform/azuredevops/errors_test.go @@ -0,0 +1,94 @@ +package azuredevops + +import ( + "context" + "errors" + "net/http" + "strings" + "testing" + "time" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// TestErrors: answers are classified by status, typeKey, TF code and +// Retry-After, and never show the token. +func TestErrors(t *testing.T) { + for _, tc := range []struct { + name string + status int + header map[string]string + body any + want platform.Class + typeKey string + after time.Duration + }{ + {name: "401", status: 401, body: errorBody("UnauthorizedRequestException", "TF400813: The user is not authorized to access this resource."), want: platform.ClassAuth}, + {name: "403", status: 403, body: errorBody("UnauthorizedRequestException", "TF401019: no permission"), want: platform.ClassPermission}, + {name: "404 repository", status: 404, body: errorBody(keyRepoNotFound, "TF401019: The Git repository does not exist"), want: platform.ClassNotFound, typeKey: keyRepoNotFound}, + {name: "429", status: 429, header: map[string]string{"Retry-After": "30"}, body: errorBody("RequestBlockedException", tfRateLimited+": The request has been canceled"), want: platform.ClassRateLimited, after: 30 * time.Second}, + {name: "Retry-After on 400", status: 400, header: map[string]string{"Retry-After": "5"}, body: errorBody("X", "slow down"), want: platform.ClassRateLimited, after: 5 * time.Second}, + {name: "TF400733 on 503", status: 503, body: errorBody("X", tfRateLimited+": blocked"), want: platform.ClassRateLimited}, + {name: "503", status: 503, body: "busy", want: platform.ClassTransient}, + {name: "400", status: 400, body: errorBody("InvalidArgumentValueException", "bad"), want: platform.ClassInvalid}, + {name: "409", status: 409, body: errorBody(keyPRExists, tfPRExists+": An active pull request for the source and target branch already exists."), want: platform.ClassConflict, typeKey: keyPRExists}, + {name: "400 duplicate", status: 400, body: errorBody(keyPRExists, tfPRExists+": exists"), want: platform.ClassConflict, typeKey: keyPRExists}, + {name: "sign-in redirect", status: 302, header: map[string]string{"Location": "https://spsprodcus4.vssps.visualstudio.com/_signin?realm=dev.azure.com"}, body: "moved", want: platform.ClassAuth}, + } { + t.Run(tc.name, func(t *testing.T) { + s := newAPIServer(t) + s.handle(http.MethodGet, apisPath("connectionData"), func(w http.ResponseWriter, _ *http.Request) { + for k, v := range tc.header { + w.Header().Set(k, v) + } + writeJSON(w, tc.status, tc.body) + }) + token := testToken(t) + _, err := newTestReader(t, s, token).Self(context.Background()) + wantClass(t, tc.name, err, tc.want) + if got := typeKeyOf(err); tc.typeKey != "" && got != tc.typeKey { + t.Errorf("typeKey %q, want %q", got, tc.typeKey) + } + var pe *platform.Error + if errors.As(err, &pe) && pe.RetryAfter != tc.after { + t.Errorf("RetryAfter %v, want %v", pe.RetryAfter, tc.after) + } + if strings.Contains(err.Error(), token) { + t.Errorf("the error shows the token: %v", err) + } + if tc.status == 404 && !errors.Is(err, platform.ErrNotFound) { + t.Errorf("a 404 without ErrNotFound: %v", err) + } + }) + } +} + +// TestSignInPage: a 203 with an HTML page is a refused credential. +func TestSignInPage(t *testing.T) { + s := newAPIServer(t) + s.handle(http.MethodGet, apisPath("connectionData"), func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(http.StatusNonAuthoritativeInfo) + _, _ = w.Write([]byte("Azure DevOps Services | Sign In")) + }) + _, err := newTestReader(t, s, testToken(t)).Self(context.Background()) + wantClass(t, "203", err, platform.ClassAuth) +} + +func TestRetryAfter(t *testing.T) { + at := time.Unix(1_800_000_000, 0) + for _, tc := range []struct { + h http.Header + want time.Duration + }{ + {http.Header{"Retry-After": {"12"}}, 12 * time.Second}, + {http.Header{"X-Ratelimit-Reset": {"1800000060"}}, time.Minute}, + {http.Header{"Retry-After": {"999999"}}, maxRetryAfter}, + {http.Header{"Retry-After": {"soon"}}, 0}, + {http.Header{}, 0}, + } { + if got := retryAfter(tc.h, at); got != tc.want { + t.Errorf("retryAfter(%v) = %v, want %v", tc.h, got, tc.want) + } + } +} diff --git a/internal/platform/azuredevops/faults_test.go b/internal/platform/azuredevops/faults_test.go new file mode 100644 index 0000000..363b1be --- /dev/null +++ b/internal/platform/azuredevops/faults_test.go @@ -0,0 +1,378 @@ +package azuredevops + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "testing" + "time" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" + "github.com/bedrock-python/touchmark/internal/throttle" +) + +// noSleep makes storeNewMarker's pauses instant for the test and records +// them. +func noSleep(t *testing.T) *[]time.Duration { + t.Helper() + var waits []time.Duration + old := sleep + sleep = func(ctx context.Context, d time.Duration) error { + waits = append(waits, d) + return ctx.Err() + } + t.Cleanup(func() { sleep = old }) + return &waits +} + +// failProperties makes the first n PATCHes of #11's properties fail with +// status, header h and an error body; the others go to next. +func failProperties(s *apiServer, n, status int, h http.Header, next http.HandlerFunc) { + calls := 0 + s.handle(http.MethodPatch, repoPath("pullRequests", "11", "properties"), func(w http.ResponseWriter, r *http.Request) { + calls++ + if n < 0 || calls <= n { + for k, v := range h { + w.Header()[k] = v + } + writeJSON(w, status, errorBody("VssServiceException", "TF000000: failed")) + return + } + next(w, r) + }) +} + +// abandonRoute answers the PATCH of #11, recording its bodies; fail makes +// it answer 500. +func abandonRoute(t *testing.T, s *apiServer, pr *map[string]any, fail bool) *[]updatePR { + t.Helper() + var got []updatePR + s.handle(http.MethodPatch, repoPath("pullrequests", "11"), func(w http.ResponseWriter, r *http.Request) { + var in updatePR + if err := json.NewDecoder(r.Body).Decode(&in); err != nil { + t.Fatal(err) + } + got = append(got, in) + if fail { + writeJSON(w, http.StatusInternalServerError, errorBody("VssServiceException", "down")) + return + } + if in.Status != nil { + (*pr)["status"] = *in.Status + (*pr)["closedDate"] = "2026-03-01T00:00:00Z" + } + writeJSON(w, http.StatusOK, *pr) + }) + return &got +} + +// propertyHandler is the properties PATCH of createWorld, taken back from +// the routes before a test replaces it. +func propertyHandler(s *apiServer) http.HandlerFunc { + s.mu.Lock() + defer s.mu.Unlock() + return s.routes[http.MethodPatch+" "+repoPath("pullRequests", "11", "properties")] +} + +// TestCreatePRMarkerRetried: a property write that fails for now is tried +// again, after the driver's pauses; the pull request stays active. +func TestCreatePRMarkerRetried(t *testing.T) { + waits := noSleep(t) + s := newAPIServer(t) + _, stored := createWorld(t, s) + failProperties(s, 2, http.StatusServiceUnavailable, nil, propertyHandler(s)) + tw := newTarget(t, s) + body := "text\n\n" + testLine + pr, err := tw.CreatePR(context.Background(), platform.NewPR{Head: syncBranch, Base: "main", Title: "t", Body: body}) + if err != nil { + t.Fatal(err) + } + if n := len(s.requests(http.MethodPatch, repoPath("pullRequests", "11", "properties"))); n != 3 { + t.Errorf("%d property writes, want 3", n) + } + if len(*stored) != 1 || pr.Body != body || pr.State != platform.Open { + t.Errorf("stored %+v, CreatePR = %+v", *stored, pr) + } + if fmt.Sprint(*waits) != "[1s 2s]" { + t.Errorf("waits %v", *waits) + } + if n := len(s.requests(http.MethodPatch, repoPath("pullrequests", "11"))); n != 0 { + t.Errorf("the pull request was patched %d times", n) + } +} + +// TestCreatePRMarkerAbandons: a property that cannot be stored gets the +// new pull request abandoned, and the call fails with the property's class; +// a rate limit asking for a long wait, or a refusal, is not tried again; +// a failed abandon says that the pull request stays active. +func TestCreatePRMarkerAbandons(t *testing.T) { + body := "text\n\n" + testLine + for _, tc := range []struct { + name string + status int + h http.Header + attempts int + class platform.Class + fail bool + }{ + {"transient", http.StatusServiceUnavailable, nil, 4, platform.ClassTransient, false}, + {"long rate limit", http.StatusTooManyRequests, http.Header{"Retry-After": {"120"}}, 1, platform.ClassRateLimited, false}, + {"short rate limit", http.StatusTooManyRequests, http.Header{"Retry-After": {"5"}}, 4, platform.ClassRateLimited, false}, + {"refused", http.StatusForbidden, nil, 1, platform.ClassPermission, false}, + {"abandon fails", http.StatusServiceUnavailable, nil, 4, platform.ClassTransient, true}, + } { + t.Run(tc.name, func(t *testing.T) { + waits := noSleep(t) + s := newAPIServer(t) + pr, _ := createWorld(t, s) + failProperties(s, -1, tc.status, tc.h, nil) + patches := abandonRoute(t, s, pr, tc.fail) + tw := newTarget(t, s) + _, err := tw.CreatePR(context.Background(), platform.NewPR{Head: syncBranch, Base: "main", Title: "t", Body: body}) + wantClass(t, "CreatePR", err, tc.class) + if n := len(s.requests(http.MethodPatch, repoPath("pullRequests", "11", "properties"))); n != tc.attempts { + t.Errorf("%d property writes, want %d", n, tc.attempts) + } + if len(*patches) != 1 || (*patches)[0].Status == nil || *(*patches)[0].Status != statusAbandoned || + (*patches)[0].Description != nil || (*patches)[0].Title != nil { + t.Errorf("PATCHes %+v, want one abandon", *patches) + } + if tc.name == "short rate limit" && fmt.Sprint(*waits) != "[5s 5s 5s]" { + t.Errorf("waits %v", *waits) + } + switch { + case tc.fail && !strings.Contains(err.Error(), "stays active without its marker"): + t.Errorf("error %v", err) + case !tc.fail && !strings.Contains(err.Error(), "#11 was abandoned"): + t.Errorf("error %v", err) + } + // No label is added to an abandoned pull request. + if n := len(s.requests(http.MethodPost, repoPath("pullRequests", "11", "labels"))); n != 0 { + t.Errorf("%d labels added", n) + } + }) + } +} + +// TestEditPROrder: an edit of an active pull request that is not a close +// sends its PATCH first, then the labels, then the property. A property +// that fails then leaves the marker of the previous body, so that the next +// run sees the edit still to make. +func TestEditPROrder(t *testing.T) { + s := newAPIServer(t) + patches := editWorld(t, s, statusActive) + tw := newTarget(t, s) + newLine := strings.Replace(testLine, "key=sha256:6b1f", "key=sha256:0000", 1) + body := "new text\n\n" + newLine + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body, AddLabels: []string{"engineering-assets"}}); err != nil { + t.Fatal(err) + } + var order []string + for _, c := range s.writes() { + order = append(order, c.Method+" "+c.Path[strings.LastIndex(c.Path, "/")+1:]) + } + if strings.Join(order, ", ") != "PATCH 11, POST labels, PATCH properties" { + t.Errorf("writes %v", order) + } + if len(*patches) != 1 || (*patches)[0].Status != nil || *(*patches)[0].Description != "new text" { + t.Errorf("PATCH %+v", *patches) + } + + // The property fails after the description went through. + failProperties(s, -1, http.StatusInternalServerError, nil, nil) + next := "newer text\n\n" + testLine + _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &next}) + wantClass(t, "a failed property", err, platform.ClassTransient) + r := newTestReader(t, s, testToken(t)) + prs, err := r.PRs(context.Background(), testRepo(), []string{syncBranch}, []platform.Account{{ID: botID}}) + if err != nil || len(prs) != 1 { + t.Fatalf("PRs = %v, %v", prs, err) + } + if desc, line := marker.Detach(prs[0].Body); desc != "newer text" || line != newLine { + t.Errorf("after the failure: description %q, marker %q: the marker must stay the previous one", desc, line) + } +} + +// TestEditPRKeepsDescription: a body whose description is the current one, +// but for line endings, trailing blanks and a marker line a person pasted, +// writes the property alone. +func TestEditPRKeepsDescription(t *testing.T) { + s := newAPIServer(t) + patches, pr := editWorldPR(t, s, statusActive) + (*pr)["description"] = "old text\r\nsecond line \r\n" + testLine + "\r\n\r\n" + tw := newTarget(t, s) + newLine := strings.Replace(testLine, "key=sha256:6b1f", "key=sha256:0000", 1) + body := "old text\nsecond line\n\n" + newLine + got, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body}) + if err != nil { + t.Fatal(err) + } + if len(*patches) != 0 { + t.Errorf("PATCH %+v for an unchanged description", *patches) + } + if _, line := marker.Detach(got.Body); line != newLine { + t.Errorf("marker %q", line) + } +} + +// TestEditPRCloseLongDescription: a close whose description would not fit +// stores its marker and abandons the pull request, the description as it +// was; closing it again is nothing. +func TestEditPRCloseLongDescription(t *testing.T) { + s := newAPIServer(t) + patches := editWorld(t, s, statusActive) + tw := newTarget(t, s) + closedLine := strings.Replace(testLine, "key=sha256:6b1f", "key=sha256:c105", 1) + body := strings.Repeat("x", 3990) + " cc @\u2060bob, @\u2060eve\n\n" + closedLine + closed := platform.Closed + pr, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body, State: &closed}) + if err != nil { + t.Fatal(err) + } + if len(*patches) != 1 || (*patches)[0].Description != nil || (*patches)[0].Status == nil { + t.Errorf("PATCH %+v, want an abandon without the description", *patches) + } + if desc, line := marker.Detach(pr.Body); pr.State != platform.Closed || desc != "old text" || line != closedLine { + t.Errorf("EditPR = %s, %q, %q", pr.State, desc, line) + } + s.reset() + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body, State: &closed}); err != nil { + t.Errorf("closing again: %v", err) + } + if w := s.writes(); len(w) != 0 { + t.Errorf("writes when closing again: %v", w) + } + // An active pull request's edit that does not fit still fails. + s2 := newAPIServer(t) + editWorld(t, s2, statusActive) + _, err = newTarget(t, s2).EditPR(context.Background(), 11, platform.PREdit{Body: &body}) + wantClass(t, "a long description", err, platform.ClassInvalid) + if w := s2.writes(); len(w) != 0 { + t.Errorf("writes for a refused edit: %v", w) + } +} + +// TestReadFileDisabled: a disabled repository shows no default branch, +// as an empty one does, but its files are unknown, never missing. +func TestReadFileDisabled(t *testing.T) { + s := newAPIServer(t) + fileWorld(t, s) + disabled := repoJSON(repoID, "Billing", "api", "") + disabled["isDisabled"] = true + s.json(repoPath(), http.StatusOK, disabled) + r := newTestReader(t, s, testToken(t)) + repo := testRepo() + repo.DefaultBranch = "" + _, err := r.ReadFile(context.Background(), repo, "", ".engineering-assets.yml", 1<<20) + if errors.Is(err, platform.ErrNotFound) { + t.Errorf("a disabled repository's file is missing: %v", err) + } + wantClass(t, "a disabled repository", err, platform.ClassUnknown) + repo = testRepo() + repo.Disabled = true + s.reset() + _, err = r.ReadFile(context.Background(), repo, "", ".engineering-assets.yml", 1<<20) + wantClass(t, "a repository known to be disabled", err, platform.ClassUnknown) + if n := len(s.requests("", "")); n != 0 { + t.Errorf("%d requests for a repository known to be disabled", n) + } +} + +// TestAnonymousSignIn: an anonymous read sent to the sign-in page says +// that Azure DevOps wants a token for it. +func TestAnonymousSignIn(t *testing.T) { + s := newAPIServer(t) + s.handle(http.MethodGet, repoPath("pullRequests", "7", "properties"), func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Location", "https://spsprodweu5.vssps.visualstudio.com/_signin?realm=dev.azure.com") + w.WriteHeader(http.StatusFound) + }) + _, err := newTestReader(t, s, "").c.markerOf(context.Background(), "read", repoID, 7) + wantClass(t, "anonymous properties", err, platform.ClassAuth) + if err == nil || !strings.Contains(err.Error(), "only with a token") || strings.Contains(err.Error(), "not public") { + t.Errorf("error %v", err) + } +} + +// TestRetryAfterOnSuccess: an answer that went through with Retry-After +// pauses the provider's throttle, without a strike; one without it does +// not. +func TestRetryAfterOnSuccess(t *testing.T) { + s := newAPIServer(t) + calls := 0 + s.handle(http.MethodGet, apisPath("connectionData"), func(w http.ResponseWriter, _ *http.Request) { + calls++ + if calls == 1 { + w.Header().Set("Retry-After", "7") + } + writeJSON(w, http.StatusOK, connJSON(botID, "Microsoft.IdentityModel.Claims.ClaimsIdentity;x\\bot@acme.example", "bot@acme.example")) + }) + at := time.Unix(1_800_000_000, 0) + g := throttle.New(throttle.Options{Name: "ado", Clock: throttle.Clock{ + Now: func() time.Time { return at }, + Sleep: func(context.Context, time.Duration) error { return nil }, + }}) + ctx := throttle.With(context.Background(), g) + r := newTestReader(t, s, testToken(t)) + if _, err := r.c.selfAccount(ctx); err != nil { + t.Fatal(err) + } + st := g.Stats() + if st.Pauses != 1 || st.Paused != 7*time.Second || st.Limits != 0 { + t.Errorf("stats %+v, want one pause of 7s and no rate limit", st) + } + r2 := newTestReader(t, s, testToken(t)) + if _, err := r2.c.selfAccount(ctx); err != nil { + t.Fatal(err) + } + if st := g.Stats(); st.Pauses != 1 { + t.Errorf("stats %+v after an answer without Retry-After", st) + } +} + +// TestPRsClosedMarkerBound: the markers of pull requests that are not +// active are read for the newest maxClosedReads of each status only. +func TestPRsClosedMarkerBound(t *testing.T) { + s := newAPIServer(t) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + var all []any + const n = maxClosedReads + 5 + for i := 1; i <= n; i++ { + p := prJSON(int64(i), statusAbandoned, syncBranch, "main", bot, "short") + all = append(all, p) + full := prJSON(int64(i), statusAbandoned, syncBranch, "main", bot, "short") + full["closedBy"] = bot + s.json(repoPath("pullrequests", fmt.Sprint(i)), http.StatusOK, full) + s.json(repoPath("pullRequests", fmt.Sprint(i), "properties"), http.StatusOK, propsJSON(testLine)) + } + all = append(all, prJSON(n+1, statusActive, syncBranch, "main", bot, "open")) + s.json(repoPath("pullRequests", fmt.Sprint(n+1), "properties"), http.StatusOK, propsJSON(testLine)) + s.handle(http.MethodGet, repoPath("pullrequests"), func(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, collection(all...)) + }) + s.json(repoPath("refs"), http.StatusOK, collection(map[string]any{"name": "refs/heads/" + syncBranch, "objectId": headSHA})) + prs, err := newTestReader(t, s, testToken(t)).PRs(context.Background(), testRepo(), []string{syncBranch}, []platform.Account{{ID: botID}}) + if err != nil { + t.Fatal(err) + } + reads, withMarker := 0, 0 + for _, c := range s.requests(http.MethodGet, "") { + if strings.HasSuffix(c.Path, "/properties") { + reads++ + } + } + for _, pr := range prs { + if _, line := marker.Detach(pr.Body); line != "" { + withMarker++ + if pr.State == platform.Closed && pr.Number <= n-maxClosedReads { + t.Errorf("#%d, older than the bound, has its marker", pr.Number) + } + } + } + if reads != maxClosedReads+1 || withMarker != maxClosedReads+1 || len(prs) != n+1 { + t.Errorf("%d property reads, %d markers of %d pull requests; want %d", reads, withMarker, len(prs), maxClosedReads+1) + } +} diff --git a/internal/platform/azuredevops/files.go b/internal/platform/azuredevops/files.go new file mode 100644 index 0000000..ccaffce --- /dev/null +++ b/internal/platform/azuredevops/files.go @@ -0,0 +1,265 @@ +package azuredevops + +import ( + "context" + "crypto/sha1" + "encoding/hex" + "errors" + "fmt" + "net/url" + "path" + "strings" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// Git modes of tree entries, as the Trees API writes them ("100644", +// "40000"; the sample answers of Trees - Get). +const ( + modeFile = "100644" + modeExecutable = "100755" + modeSymlink = "120000" + modeSubmodule = "160000" +) + +// apiItem is an item of the Items API (GitItem): a path at a commit. +type apiItem struct { + ObjectID string `json:"objectId"` + GitObjectType string `json:"gitObjectType"` + CommitID string `json:"commitId"` + Path string `json:"path"` + IsFolder bool `json:"isFolder"` + IsSymLink bool `json:"isSymLink"` +} + +// apiTree is a tree of the Trees API (GitTreeRef) with its entries. +type apiTree struct { + ObjectID string `json:"objectId"` + TreeEntries []apiTreeEntry `json:"treeEntries"` +} + +// apiTreeEntry is one entry of a tree (GitTreeEntryRef). +type apiTreeEntry struct { + ObjectID string `json:"objectId"` + RelativePath string `json:"relativePath"` + Mode string `json:"mode"` + GitObjectType string `json:"gitObjectType"` + Size *int64 `json:"size"` +} + +// ReadFile returns a regular file of r at ref ("" is the default branch +// head, a full commit id that commit, anything else a branch), in three +// reads that describe the same commit: +// +// 1. the Items API describes the file's folder at ref (its tree id and the +// commit ref resolves to): GET …/items?path=&versionDescriptor…; +// 2. the Trees API lists that folder: the file's entry gives its git mode, +// blob id and size (the Items API has no mode, and no size); +// 3. the Blobs API gives the blob's bytes ($format=octetstream), whose +// git id must be the entry's. +// +// Symlinks, submodules and directories are ErrNotRegular, a file over max +// bytes ErrTooLarge. +// +// Only a path missing at an existing commit is ErrNotFound: the folder's +// 404 with typeKey GitItemNotFoundException, or a folder without the +// entry. The core takes a missing opt-in file for an opt-out and closes the +// target's pull requests: a missing branch or commit +// (GitUnresolvableToCommitException), a missing repository, or any other +// 404 are ClassUnknown. An empty repository has no file: ErrNotFound. A +// disabled repository (r.Disabled, or isDisabled when the repository is +// read for its default branch) is ClassUnknown: it shows no default branch, +// as an empty one does, and its files are unknown, not missing. A default +// branch that r names and the repository no longer has is read once more +// as the repository names it now. +func (d *reader) ReadFile(ctx context.Context, r platform.Repo, ref, filePath string, max int64) (platform.File, error) { + const op = "read file" + if !guidRe.MatchString(r.ID) { + return platform.File{}, invalid(op, "repository %s has no id", r.Path) + } + if err := checkTreePath(filePath); err != nil { + return platform.File{}, invalid(op, "%v", err) + } + if max < 0 { + return platform.File{}, invalid(op, "negative size limit %d", max) + } + if r.Disabled { + return platform.File{}, unknown(op, nil, "%s: the repository is disabled", r.Path) + } + dir, name := path.Split(filePath) + dir = "/" + strings.TrimSuffix(dir, "/") + folder, err := d.c.folder(ctx, op, r, ref, dir) + if err != nil { + return platform.File{}, err + } + switch { + case folder.IsSymLink: + return platform.File{}, fmt.Errorf("%s: %s: %s leads through a symlink at %s: %w", op, r.Path, filePath, dir, platform.ErrNotRegular) + case !strings.EqualFold(folder.GitObjectType, "tree"): + return platform.File{}, notFound(op, "%s: %s at %s: %s is no directory", r.Path, filePath, folder.CommitID, dir) + case !isHexOID(folder.ObjectID): + return platform.File{}, shapeError(op, "%s: the folder %s has no tree id", r.Path, dir) + } + var tree apiTree + if _, err := d.c.get(ctx, op, d.c.repoAPI(r.ID, "trees", strings.ToLower(folder.ObjectID)), nil, &tree); err != nil { + if platform.ClassOf(err) == platform.ClassNotFound { + // Never ErrNotFound: the Items API has just shown the tree. + return platform.File{}, unknown(op, err, "%s: the tree of %s at %s is not found", r.Path, dir, folder.CommitID) + } + return platform.File{}, err + } + var e *apiTreeEntry + for i := range tree.TreeEntries { + if tree.TreeEntries[i].RelativePath == name { + e = &tree.TreeEntries[i] + break + } + } + if e == nil { + return platform.File{}, notFound(op, "%s: %s at %s", r.Path, filePath, folder.CommitID) + } + what := "" + switch { + case e.Mode == modeSymlink: + what = "a symlink" + case e.Mode == modeSubmodule || strings.EqualFold(e.GitObjectType, "commit"): + what = "a submodule" + case strings.EqualFold(e.GitObjectType, "tree"): + what = "a directory" + case e.Mode != modeFile && e.Mode != modeExecutable || !strings.EqualFold(e.GitObjectType, "blob"): + what = fmt.Sprintf("an entry of mode %q and type %q", e.Mode, e.GitObjectType) + } + if what != "" { + return platform.File{}, fmt.Errorf("%s: %s: %s is %s: %w", op, r.Path, filePath, what, platform.ErrNotRegular) + } + switch { + case !isHexOID(e.ObjectID): + return platform.File{}, shapeError(op, "%s: the entry of %s has no blob id", r.Path, filePath) + case e.Size == nil || *e.Size < 0: + return platform.File{}, shapeError(op, "%s: the entry of %s has no size", r.Path, filePath) + case *e.Size > max: + return platform.File{}, fmt.Errorf("%s: %s: %s has %d bytes, more than %d: %w", op, r.Path, filePath, *e.Size, max, platform.ErrTooLarge) + } + oid := strings.ToLower(e.ObjectID) + content, err := d.c.raw(ctx, op, d.c.repoAPI(r.ID, "blobs", oid), url.Values{"$format": {"octetstream"}}) + switch { + case err != nil && platform.ClassOf(err) == platform.ClassNotFound: + // Never ErrNotFound: the tree of the same commit has just shown it. + return platform.File{}, unknown(op, err, "%s: %s at %s: the tree shows it, its blob is not found", r.Path, filePath, folder.CommitID) + case err != nil: + return platform.File{}, err + case int64(len(content)) != *e.Size || blobID(content) != oid: + return platform.File{}, shapeError(op, "%s: the blob of %s is not %s (%d bytes)", r.Path, filePath, oid, *e.Size) + } + mode := modeFile + if e.Mode == modeExecutable { + mode = modeExecutable + } + return platform.File{Path: filePath, Mode: mode, OID: oid, Content: content}, nil +} + +// folder reads the item of dir (an absolute folder path, "/" for the root) +// of r at ref. A missing path is ErrNotFound; a missing branch, commit or +// repository, or any other 404, is ClassUnknown. ref "" is r's default +// branch: an empty repository is ErrNotFound, a disabled one (which shows +// no default branch either) ClassUnknown, and a default branch the +// repository no longer has is read once more as the repository names it. +func (c *client) folder(ctx context.Context, op string, r platform.Repo, ref, dir string) (*apiItem, error) { + if ref != "" { + return c.item(ctx, op, r, ref, dir) + } + branch := r.DefaultBranch + for range 2 { + if branch != "" { + it, err := c.item(ctx, op, r, branch, dir) + if err == nil || typeKeyOf(err) != keyUnresolvable { + return it, err + } + } + // No default branch, or not where r says: the repository tells. + repo, err := c.getRepo(ctx, op, r.ID) + switch { + case err != nil && platform.ClassOf(err) == platform.ClassNotFound: + return nil, unknown(op, err, "%s: the repository is not found", r.Path) + case err != nil: + return nil, err + case repo.IsDisabled: + // A disabled repository shows no default branch either: never + // ErrNotFound, which would read as a missing opt-in file. + return nil, unknown(op, nil, "%s: the repository is disabled", r.Path) + case repo.DefaultBranch == "": + return nil, notFound(op, "%s: the repository is empty", r.Path) + case strings.TrimPrefix(repo.DefaultBranch, "refs/heads/") == branch: + return nil, unknown(op, nil, "%s: the default branch %s does not resolve", r.Path, branch) + } + branch = strings.TrimPrefix(repo.DefaultBranch, "refs/heads/") + } + return nil, unknown(op, nil, "%s: the default branch moved while it was read", r.Path) +} + +// item reads the item of dir at version (a branch name or a full commit +// id): GET {org}/_apis/git/repositories/{id}/items?path=…& +// versionDescriptor.version=…&versionDescriptor.versionType=branch|commit. +func (c *client) item(ctx context.Context, op string, r platform.Repo, version, dir string) (*apiItem, error) { + kind := "branch" + if isHexOID(version) { + kind = "commit" + } + q := url.Values{ + "path": {dir}, + "versionDescriptor.version": {version}, + "versionDescriptor.versionType": {kind}, + } + var it apiItem + _, err := c.get(ctx, op, c.repoAPI(r.ID, "items"), q, &it) + switch { + case err == nil: + return &it, nil + case typeKeyOf(err) == keyItemNotFound: + return nil, notFound(op, "%s: %s at %s %s", r.Path, dir, kind, version) + case platform.ClassOf(err) == platform.ClassNotFound: + // The branch, the commit or the repository: never a missing file. + return nil, unknown(op, err, "%s: %s %s", r.Path, kind, version) + } + return nil, err +} + +// checkTreePath accepts a repository-relative path: no leading slash, no +// empty, "." or ".." segment, no NUL. +func checkTreePath(p string) error { + if p == "" { + return errors.New("empty path") + } + if strings.IndexByte(p, 0) >= 0 { + return fmt.Errorf("path %q contains NUL", p) + } + for seg := range strings.SplitSeq(p, "/") { + if seg == "" || seg == "." || seg == ".." { + return fmt.Errorf("path %q has an empty, \".\" or \"..\" segment", p) + } + } + return nil +} + +// blobID is git's SHA-1 id of a blob with content: Azure Repos hosts SHA-1 +// repositories only. +func blobID(content []byte) string { + h := sha1.New() + fmt.Fprintf(h, "blob %d\x00", len(content)) + h.Write(content) + return hex.EncodeToString(h.Sum(nil)) +} + +// isHexOID reports whether s is a full SHA-1 object id. +func isHexOID(s string) bool { + if len(s) != 40 { + return false + } + for i := 0; i < len(s); i++ { + c := s[i] + if (c < '0' || c > '9') && (c < 'a' || c > 'f') && (c < 'A' || c > 'F') { + return false + } + } + return true +} diff --git a/internal/platform/azuredevops/files_test.go b/internal/platform/azuredevops/files_test.go new file mode 100644 index 0000000..0b99610 --- /dev/null +++ b/internal/platform/azuredevops/files_test.go @@ -0,0 +1,167 @@ +package azuredevops + +import ( + "context" + "errors" + "net/http" + "testing" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// Ids of the trees and the commit of the file tests. +const ( + headCommit = "af56d96fdbd7c26e9fc94336b6f50dcc6ceff484" + rootTree = "6f5bf80f955183fe9b0c8640961303620245c8a4" + ciTree = "d1d5c2d49045d52bba6419652d6ecb2cd560dc29" +) + +// fileWorld declares a repository at headCommit on branch main: a root +// with .engineering-assets.yml, run.sh (executable), link (a symlink), +// sub (a submodule), big.txt and the folder ci with build.yml; any other +// folder is missing. The items route answers for the folders "/" and +// "/ci" at main or headCommit, a missing branch with +// GitUnresolvableToCommitException. +func fileWorld(t *testing.T, s *apiServer) map[string]string { + t.Helper() + files := map[string]string{ + ".engineering-assets.yml": "packs: [agents]\n", + "run.sh": "#!/bin/sh\necho hi\n", + "build.yml": "on: push\n", + "big.txt": "0123456789", + } + entry := func(name, mode, typ, oid string, size int) map[string]any { + return map[string]any{"objectId": oid, "relativePath": name, "mode": mode, "gitObjectType": typ, "size": size, + "url": "https://dev.azure.com/" + org + "/_apis/git/repositories/" + repoID + "/blobs/" + oid} + } + blob := func(name string) map[string]any { + mode := modeFile + if name == "run.sh" { + mode = modeExecutable + } + return entry(name, mode, "blob", gitBlobID(files[name]), len(files[name])) + } + s.json(repoPath("trees", rootTree), http.StatusOK, map[string]any{"objectId": rootTree, "size": 5, "treeEntries": []any{ + blob(".engineering-assets.yml"), blob("run.sh"), blob("big.txt"), + entry("link", modeSymlink, "blob", gitBlobID("target"), 6), + entry("sub", modeSubmodule, "commit", "3aae318f1661c50c34effbbf6882119ed161f2d6", 0), + entry("ci", "40000", "tree", ciTree, 1), + }}) + s.json(repoPath("trees", ciTree), http.StatusOK, map[string]any{"objectId": ciTree, "treeEntries": []any{blob("build.yml")}}) + for name, content := range files { + content := content + s.handle(http.MethodGet, repoPath("blobs", gitBlobID(content)), func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("$format") != "octetstream" { + t.Errorf("a blob without $format=octetstream") + } + w.Header().Set("Content-Type", "application/octet-stream; api-version=7.1") + _, _ = w.Write([]byte(content)) + }) + _ = name + } + s.handle(http.MethodGet, repoPath("items"), func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + version, kind := q.Get("versionDescriptor.version"), q.Get("versionDescriptor.versionType") + known := version == "main" && kind == "branch" || version == headCommit && kind == "commit" + if !known { + writeJSON(w, http.StatusNotFound, errorBody(keyUnresolvable, "TF401175: The version descriptor could not be resolved to a version in the repository")) + return + } + item := map[string]any{"gitObjectType": "tree", "commitId": headCommit, "path": q.Get("path"), "isFolder": true} + switch q.Get("path") { + case "/": + item["objectId"] = rootTree + case "/ci": + item["objectId"] = ciTree + case "/run.sh": + item = map[string]any{"objectId": gitBlobID(files["run.sh"]), "gitObjectType": "blob", "commitId": headCommit, "path": "/run.sh"} + case "/link": + item = map[string]any{"objectId": gitBlobID("target"), "gitObjectType": "blob", "commitId": headCommit, "path": "/link", "isSymLink": true} + default: + writeJSON(w, http.StatusNotFound, errorBody(keyItemNotFound, "TF401174: The item '"+q.Get("path")+"' could not be found in the repository")) + return + } + writeJSON(w, http.StatusOK, item) + }) + return files +} + +func TestReadFile(t *testing.T) { + s := newAPIServer(t) + files := fileWorld(t, s) + r := newTestReader(t, s, testToken(t)) + ctx := context.Background() + repo := testRepo() + + f, err := r.ReadFile(ctx, repo, "", ".engineering-assets.yml", 1<<20) + if err != nil || string(f.Content) != files[".engineering-assets.yml"] || f.Mode != "100644" || + f.OID != gitBlobID(files[".engineering-assets.yml"]) || f.Path != ".engineering-assets.yml" { + t.Errorf("ReadFile = %+v, %v", f, err) + } + if f, err := r.ReadFile(ctx, repo, headCommit, "run.sh", 1<<20); err != nil || f.Mode != "100755" { + t.Errorf("an executable at a commit: %+v, %v", f, err) + } + if f, err := r.ReadFile(ctx, repo, "main", "ci/build.yml", 1<<20); err != nil || string(f.Content) != files["build.yml"] { + t.Errorf("a nested file: %+v, %v", f, err) + } + for _, p := range []string{"link", "sub", "ci", "link/x"} { + if _, err := r.ReadFile(ctx, repo, "", p, 1<<20); !errors.Is(err, platform.ErrNotRegular) { + t.Errorf("%s: %v, want ErrNotRegular", p, err) + } + } + for _, p := range []string{"missing.yml", "docs/missing.yml", "run.sh/x"} { + if _, err := r.ReadFile(ctx, repo, "", p, 1<<20); !errors.Is(err, platform.ErrNotFound) { + t.Errorf("%s: %v, want ErrNotFound", p, err) + } + } + if _, err := r.ReadFile(ctx, repo, "", "big.txt", 5); !errors.Is(err, platform.ErrTooLarge) { + t.Errorf("a large file: %v", err) + } + // A missing branch is never a missing file. + _, err = r.ReadFile(ctx, repo, "nope", ".engineering-assets.yml", 1<<20) + wantClass(t, "a missing branch", err, platform.ClassUnknown) + if errors.Is(err, platform.ErrNotFound) { + t.Error("a missing branch is ErrNotFound") + } + for _, bad := range []string{"", "/x", "a//b", "../x"} { + _, err := r.ReadFile(ctx, repo, "", bad, 1) + wantClass(t, "path "+bad, err, platform.ClassInvalid) + } +} + +// TestReadFileDefaultBranch: the default branch r names is gone: the +// repository tells the current one; an empty repository has no file; a +// missing repository is never a missing file. +func TestReadFileDefaultBranch(t *testing.T) { + s := newAPIServer(t) + fileWorld(t, s) + r := newTestReader(t, s, testToken(t)) + ctx := context.Background() + s.json(repoPath(), http.StatusOK, repoJSON(repoID, "Billing", "api", "main")) + old := testRepo() + old.DefaultBranch = "master" + if _, err := r.ReadFile(ctx, old, "", ".engineering-assets.yml", 1<<20); err != nil { + t.Errorf("a renamed default branch: %v", err) + } + s.json(repoPath(), http.StatusOK, repoJSON(repoID, "Billing", "api", "")) + empty := testRepo() + empty.DefaultBranch = "" + if _, err := r.ReadFile(ctx, empty, "", ".engineering-assets.yml", 1<<20); !errors.Is(err, platform.ErrNotFound) { + t.Errorf("an empty repository: %v", err) + } + s.json(repoPath(), http.StatusNotFound, errorBody(keyRepoNotFound, "TF401019: gone")) + _, err := r.ReadFile(ctx, empty, "", ".engineering-assets.yml", 1<<20) + wantClass(t, "a missing repository", err, platform.ClassUnknown) +} + +// TestReadFileBlobMismatch: bytes that are not the entry's blob are no +// file. +func TestReadFileBlobMismatch(t *testing.T) { + s := newAPIServer(t) + files := fileWorld(t, s) + s.handle(http.MethodGet, repoPath("blobs", gitBlobID(files["build.yml"])), func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("on: pull\n")) + }) + _, err := newTestReader(t, s, testToken(t)).ReadFile(context.Background(), testRepo(), "", "ci/build.yml", 1<<20) + wantClass(t, "mismatch", err, platform.ClassUnknown) +} diff --git a/internal/platform/azuredevops/helpers_test.go b/internal/platform/azuredevops/helpers_test.go new file mode 100644 index 0000000..16e0046 --- /dev/null +++ b/internal/platform/azuredevops/helpers_test.go @@ -0,0 +1,314 @@ +package azuredevops + +import ( + "bytes" + "crypto/rand" + "crypto/sha1" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "sync" + "testing" + + "github.com/bedrock-python/touchmark/internal/auth" + "github.com/bedrock-python/touchmark/internal/config" + "github.com/bedrock-python/touchmark/internal/httpx" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// The fixtures below follow the REST reference of Azure DevOps Services 7.1 +// (https://learn.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-7.1) +// and anonymous answers of public projects (dnceng-public/public, +// azure-sdk/public, 2026-10-09): every field the driver reads, and a few it +// ignores, as the API sends them. + +// org is the organization of the tests: the server answers under /acme. +const org = "acme" + +// Ids of the tests. +const ( + projectID = "cbb18261-c48f-4abb-8651-8cdcb5474649" + repoID = "2459d599-fdb2-4d28-9810-daeec061cf90" + otherRepo = "382d78c1-5459-4030-925f-949b4a10dfff" + forkRepo = "5febef5a-833d-4e14-b9c0-14cb638f91e6" + botID = "56b0f042-b05e-86d9-b0f7-c5ddb1e76385" + userID = "08c37c0d-dc6c-6635-9662-1fa8dad90878" +) + +// apiServer is an httptest server that answers the routes a test declares +// and records every request. An undeclared route fails the test, and so +// does any write unless the test is the writer's (allowWrites). Routes are +// escaped paths, from /acme on. Every request must carry api-version=7.1 +// and X-TFS-FedAuthRedirect: Suppress. +type apiServer struct { + t *testing.T + srv *httptest.Server + mu sync.Mutex + routes map[string]http.HandlerFunc + calls []apiCall + // allowWrites lets the writer's tests write (their declared routes). + allowWrites bool +} + +// apiCall is one request the server received. +type apiCall struct { + Method, Path string // Path escaped + Query url.Values + Auth string // Authorization + ContentType string + Body []byte +} + +func newAPIServer(t *testing.T) *apiServer { + t.Helper() + s := &apiServer{t: t, routes: map[string]http.HandlerFunc{}} + s.srv = httptest.NewServer(http.HandlerFunc(s.serve)) + t.Cleanup(s.srv.Close) + t.Cleanup(func() { + if s.allowWrites { + return + } + for _, c := range s.writes() { + t.Errorf("the reader wrote: %s %s", c.Method, c.Path) + } + }) + return s +} + +// url is the provider's url (and api_url): the server's /acme. +func (s *apiServer) url() string { return s.srv.URL + "/" + org } + +func (s *apiServer) serve(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + r.Body = io.NopCloser(bytes.NewReader(body)) + p := r.URL.EscapedPath() + s.mu.Lock() + s.calls = append(s.calls, apiCall{Method: r.Method, Path: p, Query: r.URL.Query(), Auth: r.Header.Get("Authorization"), + ContentType: r.Header.Get("Content-Type"), Body: body}) + h := s.routes[r.Method+" "+p] + s.mu.Unlock() + if r.URL.Query().Get("api-version") != "7.1" { + s.t.Errorf("%s %s without api-version=7.1", r.Method, p) + } + if r.Header.Get("X-TFS-FedAuthRedirect") != "Suppress" { + s.t.Errorf("%s %s without X-TFS-FedAuthRedirect: Suppress", r.Method, p) + } + if h == nil { + s.t.Errorf("unexpected request %s %s?%s", r.Method, p, r.URL.RawQuery) + writeJSON(w, http.StatusNotFound, errorBody("NotFoundException", "not found")) + return + } + h(w, r) +} + +// handle declares a route: method and an escaped path from /acme on +// ("/acme/_apis/git/repositories/…"). +func (s *apiServer) handle(method, path string, h http.HandlerFunc) { + s.mu.Lock() + defer s.mu.Unlock() + s.routes[method+" "+path] = h +} + +// json declares a GET route that answers status with v as JSON. +func (s *apiServer) json(path string, status int, v any) { + s.handle(http.MethodGet, path, func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, status, v) }) +} + +// requests returns the recorded calls of method to path ("" matches all). +func (s *apiServer) requests(method, path string) []apiCall { + s.mu.Lock() + defer s.mu.Unlock() + var out []apiCall + for _, c := range s.calls { + if (method == "" || c.Method == method) && (path == "" || c.Path == path) { + out = append(out, c) + } + } + return out +} + +// writes returns the recorded requests that are not GETs. +func (s *apiServer) writes() []apiCall { + var out []apiCall + for _, c := range s.requests("", "") { + if c.Method != http.MethodGet { + out = append(out, c) + } + } + return out +} + +// reset forgets the recorded calls. +func (s *apiServer) reset() { + s.mu.Lock() + s.calls = nil + s.mu.Unlock() +} + +func writeJSON(w http.ResponseWriter, status int, v any) { + w.Header().Set("Content-Type", "application/json; charset=utf-8; api-version=7.1") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(v) +} + +// errorBody is Azure DevOps' error body. +func errorBody(typeKey, msg string) map[string]any { + return map[string]any{"$id": "1", "innerException": nil, "message": msg, + "typeName": "Microsoft.TeamFoundation.Git.Server." + typeKey + ", Microsoft.TeamFoundation.SourceControl.WebServer", + "typeKey": typeKey, "errorCode": 0, "eventId": 3000} +} + +// collection is a collection answer. +func collection(items ...any) map[string]any { + if items == nil { + items = []any{} + } + return map[string]any{"count": len(items), "value": items} +} + +// testToken returns a token made at run time, as long as a real PAT. +func testToken(t *testing.T) string { + t.Helper() + b := make([]byte, 26) + if _, err := rand.Read(b); err != nil { + t.Fatal(err) + } + return hex.EncodeToString(b)[:52] +} + +// provider returns the resolved Azure DevOps provider at the server. +func (s *apiServer) provider() config.ResolvedProvider { + u, _ := url.Parse(s.srv.URL) + return config.ResolvedProvider{ + Provider: config.Provider{ID: "ado", Type: "azure-devops", URL: s.url(), APIURL: s.url()}, + Host: strings.ToLower(u.Host), + APIURL: s.url(), + EnvPrefix: "TOUCHMARK_ADO_", + } +} + +// httpClient is the HTTP client of the tests. +func httpClient() *httpx.Client { return httpx.New(httpx.Options{}) } + +// newTestReader returns a reader with token ("" reads anonymously). +func newTestReader(t *testing.T, s *apiServer, token string) *reader { + t.Helper() + cred := auth.Credential{} + if token != "" { + cred = auth.Credential{Kind: auth.Token, Token: token} + } + r, err := NewReader(s.provider(), cred, httpClient()) + if err != nil { + t.Fatal(err) + } + return r.(*reader) +} + +// newTestWriter returns a writer with token; its routes may write. +func newTestWriter(t *testing.T, s *apiServer, token string) *writer { + t.Helper() + s.allowWrites = true + w, err := NewWriter(s.provider(), auth.Credential{Kind: auth.Token, Token: token}, httpClient()) + if err != nil { + t.Fatal(err) + } + return w.(*writer) +} + +// Paths of the routes. +func apisPath(segs ...string) string { return "/" + org + "/_apis/" + strings.Join(segs, "/") } +func repoPath(segs ...string) string { + return apisPath(append([]string{"git", "repositories", repoID}, segs...)...) +} + +// repoJSON is a repository as the API answers it; defaultBranch "" leaves +// it out (an empty repository). +func repoJSON(id, project, name, defaultBranch string) map[string]any { + r := map[string]any{ + "id": id, + "name": name, + "url": "https://dev.azure.com/" + org + "/" + projectID + "/_apis/git/repositories/" + id, + "project": map[string]any{"id": projectID, "name": project, "state": "wellFormed", "visibility": "private", + "revision": 20, "lastUpdateTime": "2023-05-16T15:27:53.67Z"}, + "size": 12337, + "remoteUrl": "https://" + org + "@dev.azure.com/" + org + "/" + project + "/_git/" + name, + "sshUrl": "git@ssh.dev.azure.com:v3/" + org + "/" + project + "/" + name, + "webUrl": "https://dev.azure.com/" + org + "/" + project + "/_git/" + name, + "isDisabled": false, + "isInMaintenance": false, + } + if defaultBranch != "" { + r["defaultBranch"] = "refs/heads/" + defaultBranch + } + return r +} + +// identityRef is an identity as pull requests name it; subject is its +// descriptor ("svc.…", "aad.…"). +func identityRef(id, name, subject string) map[string]any { + return map[string]any{"displayName": name, "id": id, "uniqueName": nil, "descriptor": subject, + "url": "https://spsprodcus4.vssps.visualstudio.com/A10a5dd58/_apis/Identities/" + id} +} + +// testRepo is the platform.Repo of repoJSON(repoID, "Billing", "api", "main"). +func testRepo() platform.Repo { + return platform.Repo{Host: "x", ID: repoID, Path: "Billing/api", DefaultBranch: "main", ObjectFormat: "sha1"} +} + +// prJSON is a pull request as the API answers it. +func prJSON(id int64, status, head, base string, author map[string]any, desc string) map[string]any { + p := map[string]any{ + "repository": repoJSON(repoID, "Billing", "api", "main"), + "pullRequestId": id, + "codeReviewId": id, + "status": status, + "createdBy": author, + "creationDate": "2026-02-18T04:35:01.9643061Z", + "title": "chore: sync engineering assets", + "description": desc, + "sourceRefName": "refs/heads/" + head, + "targetRefName": "refs/heads/" + base, + "mergeStatus": "succeeded", + "isDraft": false, + "lastMergeSourceCommit": map[string]any{"commitId": "3aae318f1661c50c34effbbf6882119ed161f2d6"}, + "reviewers": []any{}, + "url": "https://dev.azure.com/" + org + "/" + projectID + "/_apis/git/repositories/" + repoID + "/pullRequests/" + fmt.Sprint(id), + } + if status != statusActive { + p["closedDate"] = "2026-02-18T05:35:05.1682215Z" + } + return p +} + +// propsJSON is a pull request's properties with the marker line ("" for +// none). +func propsJSON(line string) map[string]any { + v := map[string]any{ + "Microsoft.Git.PullRequest.SourceRefName": map[string]any{"$type": "System.String", "$value": "refs/heads/touchmark/acme-eng"}, + } + if line != "" { + v[markerProperty] = map[string]any{"$type": "System.String", "$value": line} + } + return map[string]any{"count": len(v), "value": v} +} + +// gitBlobID is git's id of a blob with content. +func gitBlobID(content string) string { + h := sha1.New() + fmt.Fprintf(h, "blob %d\x00", len(content)) + h.Write([]byte(content)) + return hex.EncodeToString(h.Sum(nil)) +} + +// classOf asserts the class of err. +func wantClass(t *testing.T, what string, err error, want platform.Class) { + t.Helper() + if got := platform.ClassOf(err); got != want { + t.Errorf("%s: class %v (%v), want %v", what, got, err, want) + } +} diff --git a/internal/platform/azuredevops/probe.go b/internal/platform/azuredevops/probe.go new file mode 100644 index 0000000..cfcf534 --- /dev/null +++ b/internal/platform/azuredevops/probe.go @@ -0,0 +1,222 @@ +package azuredevops + +import ( + "context" + "errors" + "net/url" + "regexp" + "strings" + "time" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// Capabilities of Azure DevOps Services. +const ( + // maxBody bounds a description: 4 000 characters (Pull Requests - + // Update: "Description (up to 4000 characters)"). The core counts + // bytes, never fewer than characters, and the marker lives apart + // (MarkerInProperties), so 4 000 bytes of description always fit. + maxBody = 4000 + // markerProperty is the pull request property that holds the marker + // line. + markerProperty = "touchmark.marker" +) + +// Checks the writer cannot make upfront: branch policies (a required +// reviewer, a build) refuse a direct push to the branch they protect +// (TF402455), and so do the repository's push policies; the writer may +// read neither before it pushes. +const ( + branchPolicies = "branch-policies" + pushPolicies = "push-policies" +) + +// limits are the default pacing. Azure DevOps meters TSTUs, not requests: +// 200 within any sliding five minutes before it delays a user, which the +// X-RateLimit-* headers announce before it happens (internal/throttle +// pauses on them). Four targets and two git fetches at once, 120 reads and +// 30 writes a minute and half a second between two writes keep an +// ordinary run far from it; providers[].limits in hub.yml can raise them. +var limits = platform.Limits{Reads: 4, GitReads: 2, ReadsPerMinute: 120, WritesPerMinute: 30, MinInterval: 500 * time.Millisecond} + +// guidRe matches an identity id. +var guidRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) + +// publicAccess is the identity connectionData names for an anonymous +// request (seen anonymously, 2026-10-09). +const publicAccess = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" + +// apiProperty is one value of a property bag: {"$type": …, "$value": …}. +type apiProperty struct { + Value any `json:"$value"` +} + +// text returns the property's value as a string, "" when it is no string. +func (p apiProperty) text() string { + s, _ := p.Value.(string) + return s +} + +// apiIdentity is an identity of connectionData and of the Identities API. +type apiIdentity struct { + ID string `json:"id"` + Descriptor string `json:"descriptor"` + SubjectDescriptor string `json:"subjectDescriptor"` + ProviderDisplayName string `json:"providerDisplayName"` + IsContainer bool `json:"isContainer"` + Properties map[string]apiProperty `json:"properties"` +} + +// apiIdentityRef is an identity as pull requests name it (IdentityRef). +type apiIdentityRef struct { + ID string `json:"id"` + DisplayName string `json:"displayName"` + UniqueName string `json:"uniqueName"` + Descriptor string `json:"descriptor"` + IsContainer bool `json:"isContainer"` +} + +// connectionData is GET {org}/_apis/connectionData: the identity the +// credential acts as. It is not in the REST reference; Azure DevOps' own +// clients read it, and its shape is the one seen anonymously (2026-10-09). +type connectionData struct { + AuthenticatedUser *apiIdentity `json:"authenticatedUser"` +} + +// Probe reports the capabilities of Azure DevOps Services, which has one +// version: it asks the API nothing. +func (d *reader) Probe(context.Context) (platform.Caps, error) { + return platform.Caps{ + Flavor: "azure-devops", + MaxBody: maxBody, + Draft: platform.DraftNative, + // closedBy names who abandoned a pull request (read alone). + CloserKnown: true, + // An abandoned pull request is never changed again by touchmark: + // whether its properties can still be written is not known, and + // memory without writes needs none (platform.Caps.ClosedImmutable). + ClosedImmutable: true, + // Descriptions hold 4 000 characters: the marker lives in a + // property. + Marker: platform.MarkerInProperties, + RuntimeOnly: []string{branchPolicies, pushPolicies}, + Limits: limits, + }, nil +} + +// Self is the identity the credential acts as: connectionData's +// authenticatedUser. Its email is the identity's Account property when it +// is an address (a user's sign-in name), for the commits of a writer. +func (d *reader) Self(ctx context.Context) (platform.Account, error) { + return d.c.selfAccount(ctx) +} + +// selfAccount returns the credential's identity, looked up once. +func (c *client) selfAccount(ctx context.Context) (platform.Account, error) { + const op = "get account" + c.mu.Lock() + if c.self != nil { + a := *c.self + c.mu.Unlock() + return a, nil + } + c.mu.Unlock() + if c.token == "" { + return platform.Account{}, &platform.Error{Op: op, Class: platform.ClassAuth, Err: errors.New("no credential: the reader is anonymous")} + } + var cd connectionData + if _, err := c.get(ctx, op, c.apis("connectionData"), nil, &cd); err != nil { + return platform.Account{}, err + } + u := cd.AuthenticatedUser + switch { + case u == nil || !guidRe.MatchString(u.ID): + return platform.Account{}, shapeError(op, "connectionData names no authenticated identity") + case strings.EqualFold(u.ID, publicAccess) || strings.HasPrefix(u.Descriptor, "System:PublicAccess"): + return platform.Account{}, &platform.Error{Op: op, Class: platform.ClassAuth, + Err: errors.New("the token is not accepted: Azure DevOps took the request for an anonymous one")} + } + a := identityAccount(u) + if email := u.Properties["Account"].text(); validEmail(email) { + a.Email = email + } + c.mu.Lock() + c.self = &a + c.mu.Unlock() + return a, nil +} + +// validEmail reports whether s can be a commit author's address: one "@" +// between non-empty parts, and nothing git would refuse or misread. +func validEmail(s string) bool { + local, domain, ok := strings.Cut(s, "@") + return ok && local != "" && domain != "" && !strings.Contains(domain, "@") && + !strings.ContainsFunc(s, func(r rune) bool { return r <= ' ' || r == '<' || r == '>' || r == 0x7f }) +} + +// Lookup resolves a login, an identity id (a GUID), to its account: GET +// https://vssps.dev.azure.com/{org}/_apis/identities?identityIds={id}. +// Azure DevOps names identities by display names that are neither unique +// nor stable, so anything else is ClassInvalid; an id the organization does +// not know is ErrNotFound. +func (d *reader) Lookup(ctx context.Context, login string) (platform.Account, error) { + const op = "look up account" + if !guidRe.MatchString(login) { + return platform.Account{}, invalid(op, "%q: Azure DevOps logins are identity ids (GUIDs), such as 3f2a8d4e-1b6c-4f0a-9e7d-5c2b1a0f9e8d, "+ + "as connectionData shows them (a display name or an email names no identity reliably)", login) + } + var found list[*apiIdentity] + q := url.Values{"identityIds": {login}, "queryMembership": {"None"}} + if _, err := d.c.get(ctx, op, endpoint(d.c.vssps, "_apis", "identities"), q, &found); err != nil { + return platform.Account{}, err + } + for _, id := range found.Value { + if id != nil && strings.EqualFold(id.ID, login) { + return identityAccount(id), nil + } + } + return platform.Account{}, notFound(op, "identity %s", login) +} + +// identityAccount converts an identity: its id, lowercased, is both its id +// and its login. +func identityAccount(u *apiIdentity) platform.Account { + return platform.Account{ + ID: strings.ToLower(u.ID), + Login: strings.ToLower(u.ID), + Kind: kindOf(u.SubjectDescriptor, u.Descriptor, u.IsContainer), + } +} + +// refAccount converts an identity reference of a pull request; the zero +// Account when it names none. +func refAccount(r *apiIdentityRef) platform.Account { + if r == nil || !guidRe.MatchString(r.ID) { + return platform.Account{} + } + id := strings.ToLower(r.ID) + return platform.Account{ID: id, Login: id, Kind: kindOf(r.Descriptor, "", r.IsContainer)} +} + +// kindOf tells people from automation by an identity's descriptors: a +// subject descriptor "svc." is a service identity (a project's build +// service, an app), a bot; "aadsp." a service principal of Microsoft Entra +// ID; "aad." and "msa." users; the identity descriptor's type tells the same +// for the Identities API (Microsoft.TeamFoundation.ServiceIdentity, +// Microsoft.VisualStudio.Services.Claims.AadServicePrincipal). A group, or +// anything else, is unknown. +func kindOf(subject, descriptor string, container bool) platform.AccountKind { + switch { + case container: + return platform.KindUnknown + case strings.HasPrefix(subject, "svc."), strings.HasPrefix(descriptor, "Microsoft.TeamFoundation.ServiceIdentity;"): + return platform.KindBot + case strings.HasPrefix(subject, "aadsp."), strings.HasPrefix(descriptor, "Microsoft.VisualStudio.Services.Claims.AadServicePrincipal;"): + return platform.KindServiceAccount + case strings.HasPrefix(subject, "aad."), strings.HasPrefix(subject, "msa."), + strings.HasPrefix(descriptor, "Microsoft.IdentityModel.Claims.ClaimsIdentity;"): + return platform.KindUser + } + return platform.KindUnknown +} diff --git a/internal/platform/azuredevops/probe_test.go b/internal/platform/azuredevops/probe_test.go new file mode 100644 index 0000000..608ad6b --- /dev/null +++ b/internal/platform/azuredevops/probe_test.go @@ -0,0 +1,155 @@ +package azuredevops + +import ( + "context" + "encoding/base64" + "errors" + "net/http" + "slices" + "strings" + "testing" + + "github.com/bedrock-python/touchmark/internal/auth" + "github.com/bedrock-python/touchmark/internal/config" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// connJSON is connectionData as the API answers it for identity id. +func connJSON(id, descriptor, account string) map[string]any { + u := map[string]any{"id": id, "descriptor": descriptor, "providerDisplayName": "touchmark bot", "isActive": true, + "properties": map[string]any{"Account": map[string]any{"$type": "System.String", "$value": account}}, "resourceVersion": 2} + return map[string]any{"authenticatedUser": u, "authorizedUser": u, "instanceId": "6fcc92e5-73a7-4f88-8d13-d9045b45fb27", + "deploymentType": "hosted", "locationServiceData": map[string]any{"serviceOwner": "00025394-6065-48ca-87d9-7f5672854ef7"}} +} + +func TestProbe(t *testing.T) { + s := newAPIServer(t) + c, err := newTestReader(t, s, "").Probe(context.Background()) + if err != nil { + t.Fatal(err) + } + switch { + case c.Flavor != "azure-devops", c.MaxBody != 4000, c.Draft != platform.DraftNative, + !c.CloserKnown, !c.ClosedImmutable, c.NoLabels, c.LabelsByID, c.WorkflowPerm, c.QuickActions, + c.Marker != platform.MarkerInProperties, !c.BodyControls(): + t.Errorf("Probe = %+v", c) + } + if !slices.Equal(c.RuntimeOnly, []string{"branch-policies", "push-policies"}) || c.Limits.Reads == 0 || c.Limits.ReadsPerMinute == 0 { + t.Errorf("RuntimeOnly %v, Limits %+v", c.RuntimeOnly, c.Limits) + } + if len(s.requests("", "")) != 0 { + t.Error("Probe asked the API") + } +} + +func TestSelf(t *testing.T) { + s := newAPIServer(t) + token := testToken(t) + s.json(apisPath("connectionData"), http.StatusOK, connJSON(strings.ToUpper(botID), "Microsoft.IdentityModel.Claims.ClaimsIdentity;x\\bot@acme.example", "bot@acme.example")) + r := newTestReader(t, s, token) + a, err := r.Self(context.Background()) + if err != nil { + t.Fatal(err) + } + if a.ID != botID || a.Login != botID || a.Email != "bot@acme.example" || a.Kind != platform.KindUser { + t.Errorf("Self = %+v", a) + } + if _, err := r.Self(context.Background()); err != nil || len(s.requests("", "")) != 1 { + t.Errorf("Self again: %v, %d requests", err, len(s.requests("", ""))) + } + // The REST API gets Basic with an empty user name. + want := "Basic " + base64.StdEncoding.EncodeToString([]byte(":"+token)) + if got := s.requests("", "")[0].Auth; got != want { + t.Errorf("Authorization %q, want Basic :", got) + } + + // A token Azure DevOps takes for anonymous access is refused. + s2 := newAPIServer(t) + s2.json(apisPath("connectionData"), http.StatusOK, connJSON(publicAccess, "System:PublicAccess;"+publicAccess, "Anonymous")) + _, err = newTestReader(t, s2, token).Self(context.Background()) + wantClass(t, "public access", err, platform.ClassAuth) + + // An anonymous reader has no identity. + _, err = newTestReader(t, newAPIServer(t), "").Self(context.Background()) + wantClass(t, "anonymous", err, platform.ClassAuth) +} + +func TestLookup(t *testing.T) { + s := newAPIServer(t) + s.handle(http.MethodGet, apisPath("identities"), func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("identityIds") != userID { + writeJSON(w, http.StatusOK, collection()) + return + } + writeJSON(w, http.StatusOK, collection(map[string]any{ + "id": userID, "descriptor": "Microsoft.IdentityModel.Claims.ClaimsIdentity;x\\dev@acme.example", + "subjectDescriptor": "aad.MDhjMzdjMGQtZGM2Yy03NjM1LTk2NjItMWZhOGRhZDkwODc4", "providerDisplayName": "Dev", + "isActive": true, "properties": map[string]any{"Account": map[string]any{"$type": "System.String", "$value": "dev@acme.example"}}, + })) + }) + r := newTestReader(t, s, testToken(t)) + a, err := r.Lookup(context.Background(), userID) + if err != nil || a.ID != userID || a.Login != userID || a.Kind != platform.KindUser { + t.Errorf("Lookup = %+v, %v", a, err) + } + _, err = r.Lookup(context.Background(), botID) + if !errors.Is(err, platform.ErrNotFound) { + t.Errorf("an unknown id: %v", err) + } + for _, login := range []string{"dev@acme.example", "Dev", "{" + userID + "}"} { + _, err := r.Lookup(context.Background(), login) + wantClass(t, login, err, platform.ClassInvalid) + if err != nil && !strings.Contains(err.Error(), "identity ids (GUIDs)") { + t.Errorf("%s: %v", login, err) + } + } +} + +// TestKinds: descriptors tell people from automation. +func TestKinds(t *testing.T) { + for _, tc := range []struct { + subject, descriptor string + want platform.AccountKind + }{ + {"svc.NmZjYzkyZTU", "", platform.KindBot}, + {"aadsp.MDA0", "", platform.KindServiceAccount}, + {"aad.MDA0", "", platform.KindUser}, + {"msa.MDA0", "", platform.KindUser}, + {"", "Microsoft.TeamFoundation.ServiceIdentity;x", platform.KindBot}, + {"vssgp.Uy0x", "", platform.KindUnknown}, + } { + if got := kindOf(tc.subject, tc.descriptor, false); got != tc.want { + t.Errorf("kindOf(%q, %q) = %v, want %v", tc.subject, tc.descriptor, got, tc.want) + } + } + if kindOf("aad.x", "", true) != platform.KindUnknown { + t.Error("a group is a user") + } +} + +func TestNewClient(t *testing.T) { + p := config.ResolvedProvider{Provider: config.Provider{ID: "ado", Type: "azure-devops", URL: "https://dev.azure.com/acme"}, + Host: "dev.azure.com", APIURL: "https://dev.azure.com/acme"} + c, err := newClient(p, auth.Credential{Kind: auth.Token, Token: "t0k3n-0123456789abcdef"}, httpClient()) + if err != nil { + t.Fatal(err) + } + if c.org != "acme" || c.vssps != "https://vssps.dev.azure.com/acme" || !slices.Contains(c.auth.Hosts, "vssps.dev.azure.com") { + t.Errorf("client: org %q, vssps %q, hosts %v", c.org, c.vssps, c.auth.Hosts) + } + for _, bad := range []struct { + p config.ResolvedProvider + cred auth.Credential + }{ + {config.ResolvedProvider{Provider: config.Provider{ID: "x", Type: "github", URL: "https://github.com"}, APIURL: "https://api.github.com"}, auth.Credential{}}, + {config.ResolvedProvider{Provider: config.Provider{ID: "x", Type: "azure-devops", URL: "https://dev.azure.com"}, APIURL: "https://dev.azure.com"}, auth.Credential{}}, + {p, auth.Credential{Kind: auth.App, AppID: "1"}}, + } { + if _, err := newClient(bad.p, bad.cred, httpClient()); err == nil { + t.Errorf("newClient(%+v) accepted", bad.p.Provider) + } + } + if _, err := NewWriter(p, auth.Credential{}, httpClient()); err == nil { + t.Error("an anonymous writer") + } +} diff --git a/internal/platform/azuredevops/pulls.go b/internal/platform/azuredevops/pulls.go new file mode 100644 index 0000000..4ed3a59 --- /dev/null +++ b/internal/platform/azuredevops/pulls.go @@ -0,0 +1,539 @@ +package azuredevops + +import ( + "cmp" + "context" + "errors" + "fmt" + "net/url" + "slices" + "strconv" + "strings" + "time" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// Bounds of listings and reads. +const ( + // maxPRPages bounds a listing of one head's pull requests in one + // repository. PRs has no way to say it saw only part: past the bound it + // fails. + maxPRPages = 100 + // maxSweepPages bounds the active pull requests of one author in one + // project. + maxSweepPages = 100 + // maxClosedReads bounds, per call and per status, the pull requests + // that are not active whose marker property is read (one GET each), + // newest first, and the abandoned ones whose closer is read (one GET + // each, the same ones): memory looks at the 50 newest closed pull + // requests of touchmark's (decide.MemoryConfig.Window, which hub.yml + // does not change), and the core reads no completed one. An older one + // reports neither a marker nor a closer: a closed pull request without + // a marker is nothing to the core. + maxClosedReads = 60 + // listedDescription is how long a description a listing may show in + // full: listings cut descriptions to 400 characters (assumed from the + // observed listing; the REST reference says nothing), so a longer one, + // counted in bytes, which are never fewer, is read alone. + listedDescription = 390 +) + +// Pull request statuses (PullRequestStatus). +const ( + statusActive = "active" + statusAbandoned = "abandoned" + statusCompleted = "completed" +) + +// apiPR is a pull request as the API reports it (GitPullRequest), in +// listings and alone. +type apiPR struct { + ID int64 `json:"pullRequestId"` + Status string `json:"status"` + IsDraft bool `json:"isDraft"` + Title string `json:"title"` + Description *string `json:"description"` + SourceRefName string `json:"sourceRefName"` + TargetRefName string `json:"targetRefName"` + CreatedBy *apiIdentityRef `json:"createdBy"` + ClosedBy *apiIdentityRef `json:"closedBy"` + CreationDate *time.Time `json:"creationDate"` + ClosedDate *time.Time `json:"closedDate"` + Repository *apiRepo `json:"repository"` + ForkSource *struct { + Repository *apiRepo `json:"repository"` + } `json:"forkSource"` + LastMergeSourceCommit *struct { + CommitID string `json:"commitId"` + } `json:"lastMergeSourceCommit"` + Labels []apiLabel `json:"labels"` +} + +// apiLabel is a pull request's label (WebApiTagDefinition). +type apiLabel struct { + ID string `json:"id"` + Name string `json:"name"` + Active *bool `json:"active"` +} + +// apiRef is a ref of the Refs API (GitRef). +type apiRef struct { + Name string `json:"name"` + ObjectID string `json:"objectId"` +} + +// check reports what the driver depends on that a pull request lacks. +func (p *apiPR) check(op string) error { + switch { + case p.ID <= 0: + return shapeError(op, "a pull request without an id") + case p.Status != statusActive && p.Status != statusAbandoned && p.Status != statusCompleted: + return shapeError(op, "pull request %d has status %q", p.ID, p.Status) + case p.Repository == nil || !guidRe.MatchString(p.Repository.ID): + return shapeError(op, "pull request %d lacks its repository", p.ID) + case !strings.HasPrefix(p.TargetRefName, "refs/heads/"): + return shapeError(op, "pull request %d has target %q", p.ID, p.TargetRefName) + case p.CreationDate == nil: + return shapeError(op, "pull request %d has no creationDate", p.ID) + } + return nil +} + +// head returns the source branch of p, "" when it is no branch. +func (p *apiPR) head() string { + h, ok := strings.CutPrefix(p.SourceRefName, "refs/heads/") + if !ok { + return "" + } + return h +} + +// authorID returns the id of p's author, lowercased, "" for none. +func (p *apiPR) authorID() string { return refAccount(p.CreatedBy).ID } + +// headRepo returns the id of the repository p's head lives in: its fork's, +// or its own. +func (p *apiPR) headRepo() string { + if p.ForkSource != nil { + if p.ForkSource.Repository == nil || !guidRe.MatchString(p.ForkSource.Repository.ID) { + return "" + } + return strings.ToLower(p.ForkSource.Repository.ID) + } + return strings.ToLower(p.Repository.ID) +} + +// cut reports whether the description p shows may be cut by a listing. +func (p *apiPR) cut() bool { return p.Description != nil && len(*p.Description) >= listedDescription } + +// toPR converts an API pull request, its stored marker line appended to +// its description (marker.Attach); BaseExists and HeadSHA are left to the +// caller. Abandoned pull requests are closed, completed ones merged. +func (c *client) toPR(p *apiPR, markerLine string) platform.PR { + desc := "" + if p.Description != nil { + desc = *p.Description + } + pr := platform.PR{ + Number: p.ID, + State: platform.Open, + Draft: p.IsDraft, + Head: p.head(), + Base: strings.TrimPrefix(p.TargetRefName, "refs/heads/"), + RepoID: strings.ToLower(p.Repository.ID), + HeadRepoID: p.headRepo(), + Title: p.Title, + Body: marker.Attach(desc, markerLine), + Labels: []string{}, + Author: refAccount(p.CreatedBy), + CreatedAt: p.CreationDate.UTC(), + } + if pr.Head == "" { + pr.HeadRepoID = "" + } + web := p.Repository.WebURL + if web == "" && p.Repository.Project != nil && p.Repository.Project.Name != "" && p.Repository.Name != "" { + web = c.web + "/" + url.PathEscape(p.Repository.Project.Name) + "/_git/" + url.PathEscape(p.Repository.Name) + } + if web != "" { + pr.URL = web + "/pullrequest/" + strconv.FormatInt(p.ID, 10) + } + for _, l := range p.Labels { + if l.Name != "" && (l.Active == nil || *l.Active) { + pr.Labels = append(pr.Labels, l.Name) + } + } + switch p.Status { + case statusCompleted: + pr.State = platform.Merged + case statusAbandoned: + pr.State = platform.Closed + } + if pr.State != platform.Open { + if by := refAccount(p.ClosedBy); by.ID != "" { + pr.ClosedBy = &by + } + if p.ClosedDate != nil { + pr.ClosedAt = p.ClosedDate.UTC() + } + } + return pr +} + +// PRs returns, newest first, the pull requests of r from heads by authors +// in every status and the active ones by anyone: per head, GET +// {org}/_apis/git/repositories/{id}/pullrequests?searchCriteria. +// sourceRefName=refs/heads/&searchCriteria.status=all, pages of 100, +// filtered again here. Those from forks (forkSource) come with their own +// HeadRepoID. The listing is complete or the call fails. +// +// Each pull request kept costs reads of its own: its properties, for the +// marker (Pull Request Properties - List; one request each), for every +// active one and for the maxClosedReads newest abandoned and completed +// ones each; the pull request alone when the listing may have cut its +// description, and for an abandoned one, to learn its closer (listings +// leave closedBy out; the same maxClosedReads newest). An older pull +// request that is not active comes without its marker. BaseExists reads +// the Refs API for a base other than r's default branch (once per base); +// HeadSHA, for an active pull request from r itself, is its source +// branch's head (Refs API: lastMergeSourceCommit lags behind pushes until +// the merge is computed again), "" when the branch is gone. +func (d *reader) PRs(ctx context.Context, r platform.Repo, heads []string, authors []platform.Account) ([]platform.PR, error) { + const op = "list pull requests" + if !guidRe.MatchString(r.ID) { + return nil, invalid(op, "repository %s has no id", r.Path) + } + wanted := setOf(heads) + if len(wanted) == 0 { + return []platform.PR{}, nil + } + ids := accountIDs(authors) + var found []apiPR + seen := map[int64]bool{} + for _, h := range sortedKeys(wanted) { + q := url.Values{ + "searchCriteria.sourceRefName": {"refs/heads/" + h}, + "searchCriteria.status": {"all"}, + } + complete, err := listPages(ctx, d.c, op, d.c.repoAPI(r.ID, "pullrequests"), q, maxPRPages, func(p apiPR) error { + if err := p.check(op); err != nil { + return err + } + if !seen[p.ID] && wanted[p.head()] && strings.EqualFold(p.Repository.ID, r.ID) && (p.Status == statusActive || ids[p.authorID()]) { + seen[p.ID] = true + found = append(found, p) + } + return nil + }) + if err != nil { + return nil, err + } + if !complete { + return nil, &platform.Error{Op: op, Class: platform.ClassUnknown, + Err: fmt.Errorf("%s has more pull requests from %s than %d pages; touchmark does not act on a partial list", r.Path, h, maxPRPages)} + } + } + slices.SortFunc(found, func(a, b apiPR) int { return cmp.Compare(b.ID, a.ID) }) + out := make([]platform.PR, 0, len(found)) + bases := map[string]bool{} + closed := map[string]int{} + for i := range found { + p := &found[i] + alone := p.cut() + props := true + if p.Status != statusActive { + props = closed[p.Status] < maxClosedReads + closed[p.Status]++ + if props && p.Status == statusAbandoned { + alone = true + } + } + pr, err := d.c.complete(ctx, op, r, p, alone, props, bases) + if err != nil { + return nil, err + } + out = append(out, pr) + } + return out, nil +} + +// complete reads what a listed pull request p of r lacks (the pull request +// alone when alone is set, its marker property when props is set, its +// base, its head) and converts it. known holds the bases found so far. +func (c *client) complete(ctx context.Context, op string, r platform.Repo, p *apiPR, alone, props bool, known map[string]bool) (platform.PR, error) { + if alone { + full, err := c.getPR(ctx, op, r.ID, p.ID) + if err != nil { + return platform.PR{}, err + } + *p = *full + } + line := "" + if props { + var err error + if line, err = c.markerOf(ctx, op, r.ID, p.ID); err != nil { + return platform.PR{}, err + } + } + var err error + pr := c.toPR(p, line) + if pr.BaseExists, err = c.baseExists(ctx, op, r, pr.Base, known); err != nil { + return platform.PR{}, err + } + if pr.State == platform.Open && pr.Head != "" && pr.HeadRepoID == pr.RepoID { + if pr.HeadSHA, err = c.branchHead(ctx, op, r.ID, pr.Head); err != nil { + return platform.PR{}, err + } + } + return pr, nil +} + +// getPR reads pull request number of the repository with id alone: GET +// {org}/_apis/git/repositories/{id}/pullrequests/{number}. +func (c *client) getPR(ctx context.Context, op, repoID string, number int64) (*apiPR, error) { + var p apiPR + if _, err := c.get(ctx, op, c.repoAPI(repoID, "pullrequests", strconv.FormatInt(number, 10)), nil, &p); err != nil { + return nil, err + } + if err := p.check(op); err != nil { + return nil, err + } + if p.ID != number { + return nil, shapeError(op, "GET pull request %d answers %d", number, p.ID) + } + return &p, nil +} + +// markerOf returns the marker line stored in pull request number's +// properties (touchmark.marker), "" when it has none: GET +// {org}/_apis/git/repositories/{id}/pullRequests/{number}/properties. +func (c *client) markerOf(ctx context.Context, op, repoID string, number int64) (string, error) { + var props struct { + Value map[string]apiProperty `json:"value"` + } + if _, err := c.get(ctx, op, c.repoAPI(repoID, "pullRequests", strconv.FormatInt(number, 10), "properties"), nil, &props); err != nil { + return "", err + } + line := props.Value[markerProperty].text() + if strings.ContainsAny(line, "\r\n") || !marker.IsLine(line) { + // Not a marker line, or one with more in it: no marker the core + // may read (it never wrote such a value). + return "", nil + } + return line, nil +} + +// baseExists reports whether branch, a pull request's base in r, exists: +// r's default branch does; another is looked up once per call (known holds +// the answers). A pull request whose base was deleted keeps its name. +func (c *client) baseExists(ctx context.Context, op string, r platform.Repo, branch string, known map[string]bool) (bool, error) { + if branch == r.DefaultBranch { + return true, nil + } + if v, ok := known[branch]; ok { + return v, nil + } + head, err := c.branchHead(ctx, op, r.ID, branch) + if err != nil { + return false, err + } + known[branch] = head != "" + return known[branch], nil +} + +// branchHead returns the head commit of branch in the repository with id, +// "" when the branch does not exist: GET …/refs?filter=heads/, +// whose filter is a prefix, so the exact name is looked for in the answer. +func (c *client) branchHead(ctx context.Context, op, repoID, branch string) (string, error) { + var refs list[apiRef] + q := url.Values{"filter": {"heads/" + branch}} + if _, err := c.get(ctx, op, c.repoAPI(repoID, "refs"), q, &refs); err != nil { + return "", err + } + for _, ref := range refs.Value { + if ref.Name == "refs/heads/"+branch { + if !isHexOID(ref.ObjectID) { + return "", shapeError(op, "branch %s has no head", branch) + } + return strings.ToLower(ref.ObjectID), nil + } + } + return "", nil +} + +// OpenPRsBy lists the active pull requests by authors from heads in every +// repository this identity sees, for the stale sweep: the organization's +// repositories are listed once (for their projects and for the report), +// then, per project and author, GET {org}/{project}/_apis/git/pullrequests? +// searchCriteria.creatorId=&searchCriteria.status=active, filtered by +// head here. Each pull request found reads its properties (the marker) and, +// when its description may be cut, itself. HeadSHA stays "": the sweep +// closes pull requests and reads no heads. +// +// A failure with a rate limit, a refused credential, a transient failure or +// the end of ctx fails the call; any other failure of a project's listing +// makes the result incomplete, as does a listing capped at its bound. A +// pull request gone since the listing is left out; one that cannot be read +// otherwise makes the result incomplete. +func (d *reader) OpenPRsBy(ctx context.Context, authors []platform.Account, heads []string) (platform.Swept, error) { + const op = "list open pull requests" + ids := accountIDs(authors) + wanted := setOf(heads) + out := platform.Swept{PRs: []platform.RepoPR{}, Complete: true} + if len(ids) == 0 || len(wanted) == 0 { + return out, nil + } + if d.c.token == "" { + return platform.Swept{}, &platform.Error{Op: op, Class: platform.ClassAuth, Err: errAnonymousSweep} + } + all, err := d.c.listRepos(ctx, op) + if err != nil { + return platform.Swept{}, err + } + repos := map[string]*apiRepo{} + var projects []string + seenProject := map[string]bool{} + for i := range all { + r := &all[i] + repos[strings.ToLower(r.ID)] = r + if p := strings.ToLower(r.Project.ID); !seenProject[p] { + seenProject[p] = true + projects = append(projects, p) + } + } + slices.Sort(projects) + type hit struct{ pr apiPR } + hits := map[string]hit{} + for _, project := range projects { + for _, id := range sortedKeys(ids) { + q := url.Values{"searchCriteria.creatorId": {id}, "searchCriteria.status": {statusActive}} + complete, err := listPages(ctx, d.c, op, endpoint(d.c.api, project, "_apis", "git", "pullrequests"), q, maxSweepPages, func(p apiPR) error { + if err := p.check(op); err != nil { + return err + } + if p.Status == statusActive && wanted[p.head()] && ids[p.authorID()] { + hits[strings.ToLower(p.Repository.ID)+"#"+strconv.FormatInt(p.ID, 10)] = hit{pr: p} + } + return nil + }) + switch { + case err == nil: + out.Complete = out.Complete && complete + case fatal(err): + return platform.Swept{}, err + default: + out.Complete = false + } + } + } + keys := make([]string, 0, len(hits)) + for k := range hits { + keys = append(keys, k) + } + slices.Sort(keys) + bases := map[string]map[string]bool{} + for _, k := range keys { + p := hits[k].pr + id := strings.ToLower(p.Repository.ID) + repo := repos[id] + if repo == nil { + // Created after the listing of repositories, or not visible as + // a repository: read it. + got, err := d.c.getRepo(ctx, op, id) + switch { + case err == nil: + repo = got + case platform.ClassOf(err) == platform.ClassNotFound: + continue + case stops(err): + return platform.Swept{}, err + default: + out.Complete = false + continue + } + repos[id] = repo + } + r := d.c.toRepo(repo) + if bases[id] == nil { + bases[id] = map[string]bool{} + } + pr, err := d.c.sweptPR(ctx, op, r, &p, bases[id]) + switch { + case err == nil: + case platform.ClassOf(err) == platform.ClassNotFound: + continue // gone since the listing + case stops(err): + return platform.Swept{}, err + default: + out.Complete = false + continue + } + out.PRs = append(out.PRs, platform.RepoPR{Repo: r, PR: pr}) + } + slices.SortFunc(out.PRs, func(a, b platform.RepoPR) int { + if c := strings.Compare(strings.ToLower(a.Repo.Path), strings.ToLower(b.Repo.Path)); c != 0 { + return c + } + if c := strings.Compare(a.Repo.ID, b.Repo.ID); c != 0 { + return c + } + return cmp.Compare(b.PR.Number, a.PR.Number) + }) + return out, nil +} + +// sweptPR completes a pull request the sweep found, without its head. +func (c *client) sweptPR(ctx context.Context, op string, r platform.Repo, p *apiPR, known map[string]bool) (platform.PR, error) { + if p.cut() { + full, err := c.getPR(ctx, op, r.ID, p.ID) + if err != nil { + return platform.PR{}, err + } + *p = *full + } + line, err := c.markerOf(ctx, op, r.ID, p.ID) + if err != nil { + return platform.PR{}, err + } + pr := c.toPR(p, line) + pr.BaseExists, err = c.baseExists(ctx, op, r, pr.Base, known) + return pr, err +} + +// errAnonymousSweep is why an anonymous reader cannot sweep: it has no +// pull requests of its own. +var errAnonymousSweep = errors.New("no credential: the reader is anonymous, and only an identity has pull requests to sweep") + +// accountIDs returns the set of the accounts' ids, lowercased. +func accountIDs(accounts []platform.Account) map[string]bool { + ids := map[string]bool{} + for _, a := range accounts { + if guidRe.MatchString(a.ID) { + ids[strings.ToLower(a.ID)] = true + } + } + return ids +} + +// setOf returns the set of the non-empty strings of list. +func setOf(list []string) map[string]bool { + out := map[string]bool{} + for _, s := range list { + if s != "" { + out[s] = true + } + } + return out +} + +// sortedKeys returns the keys of m in order. +func sortedKeys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + slices.Sort(out) + return out +} diff --git a/internal/platform/azuredevops/pulls_test.go b/internal/platform/azuredevops/pulls_test.go new file mode 100644 index 0000000..9d9c046 --- /dev/null +++ b/internal/platform/azuredevops/pulls_test.go @@ -0,0 +1,174 @@ +package azuredevops + +import ( + "context" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// testLine is a marker line, as the core stores it. +const testLine = "" + +// syncBranch is touchmark's branch of the tests. +const syncBranch = "touchmark/acme-eng" + +const headSHA = "9d0e1f2a3b4c5d6e7f8091a2b3c4d5e6f7081920" + +// prWorld declares the pull requests of the sync branch: #7 active by the +// bot (a long description, cut in the listing), #5 abandoned by a person, +// #4 completed, #3 active by someone else, #2 abandoned by someone else +// (left out: not by our authors), #6 active from a fork; and their +// properties, single reads, refs. +func prWorld(t *testing.T, s *apiServer) { + t.Helper() + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + user := identityRef(userID, "Dev", "aad.ZGV2") + long := strings.Repeat("Long text. ", 50) + p7 := prJSON(7, statusActive, syncBranch, "main", bot, long) + p7["labels"] = []any{map[string]any{"id": "l1", "name": "engineering-assets", "active": true}} + p7cut := prJSON(7, statusActive, syncBranch, "main", bot, long[:400]) + p5 := prJSON(5, statusAbandoned, syncBranch, "main", bot, "short") + p5full := prJSON(5, statusAbandoned, syncBranch, "main", bot, "short") + p5full["closedBy"] = user + p4 := prJSON(4, statusCompleted, syncBranch, "release", bot, "merged") + p3 := prJSON(3, statusActive, syncBranch, "main", user, "theirs") + p2 := prJSON(2, statusAbandoned, syncBranch, "main", user, "theirs") + p6 := prJSON(6, statusActive, syncBranch, "main", user, "from a fork") + p6["forkSource"] = map[string]any{"name": "refs/heads/" + syncBranch, "repository": repoJSON(forkRepo, "Billing", "api-fork", "main")} + s.handle(http.MethodGet, repoPath("pullrequests"), func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + if q.Get("searchCriteria.sourceRefName") != "refs/heads/"+syncBranch || q.Get("searchCriteria.status") != "all" { + writeJSON(w, http.StatusOK, collection()) + return + } + writeJSON(w, http.StatusOK, collection(p2, p3, p4, p5, p6, p7cut)) + }) + s.json(repoPath("pullrequests", "7"), http.StatusOK, p7) + s.json(repoPath("pullrequests", "5"), http.StatusOK, p5full) + for _, n := range []int{2, 3, 4, 5, 6, 7} { + line := "" + if n == 7 || n == 5 || n == 4 { + line = testLine + } + s.json(repoPath("pullRequests", fmt.Sprint(n), "properties"), http.StatusOK, propsJSON(line)) + } + s.handle(http.MethodGet, repoPath("refs"), func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Query().Get("filter") { + case "heads/" + syncBranch: + writeJSON(w, http.StatusOK, collection( + map[string]any{"name": "refs/heads/" + syncBranch + "-old", "objectId": "1111111111111111111111111111111111111111"}, + map[string]any{"name": "refs/heads/" + syncBranch, "objectId": headSHA})) + default: + writeJSON(w, http.StatusOK, collection()) + } + }) +} + +func TestPRs(t *testing.T) { + s := newAPIServer(t) + prWorld(t, s) + r := newTestReader(t, s, testToken(t)) + prs, err := r.PRs(context.Background(), testRepo(), []string{syncBranch, ""}, []platform.Account{{ID: botID}}) + if err != nil { + t.Fatal(err) + } + var nums []int64 + for _, pr := range prs { + nums = append(nums, pr.Number) + } + if fmt.Sprint(nums) != "[7 6 5 4 3]" { + t.Fatalf("PRs = %v, want [7 6 5 4 3]", nums) + } + p7, p6, p5, p4, p3 := prs[0], prs[1], prs[2], prs[3], prs[4] + if p7.State != platform.Open || p7.Head != syncBranch || p7.Base != "main" || !p7.BaseExists || p7.HeadSHA != headSHA || + p7.Author.ID != botID || p7.RepoID != repoID || p7.HeadRepoID != repoID || + p7.URL != "https://dev.azure.com/acme/Billing/_git/api/pullrequest/7" || len(p7.Labels) != 1 { + t.Errorf("#7 = %+v", p7) + } + // The description read alone, then the stored marker. + if !strings.HasPrefix(p7.Body, strings.Repeat("Long text. ", 49)) || !strings.HasSuffix(p7.Body, "\n\n"+testLine) { + t.Errorf("#7 body %q", p7.Body) + } + if m, status := marker.Find(p7.Body, nil); status != marker.Foreign || m.Key != "" { + t.Errorf("the marker of #7 is not in its body: %v", status) + } + if p6.HeadRepoID != forkRepo || p6.HeadSHA != "" { + t.Errorf("#6 from a fork: %+v", p6) + } + if p5.State != platform.Closed || p5.ClosedBy == nil || p5.ClosedBy.ID != userID || p5.ClosedAt.IsZero() || p5.Body != "short\n\n"+testLine { + t.Errorf("#5 = %+v", p5) + } + if p4.State != platform.Merged || p4.BaseExists || p4.ClosedBy != nil { + t.Errorf("#4 = %+v", p4) + } + if p3.Author.ID != userID || p3.Body != "theirs" { + t.Errorf("#3 = %+v", p3) + } + // One listing, #7 and #5 read alone, five property reads. + if n := len(s.requests(http.MethodGet, repoPath("pullrequests"))); n != 1 { + t.Errorf("%d listings", n) + } + if len(s.requests(http.MethodGet, repoPath("pullrequests", "7"))) != 1 || len(s.requests(http.MethodGet, repoPath("pullrequests", "5"))) != 1 || + len(s.requests(http.MethodGet, repoPath("pullrequests", "3"))) != 0 { + t.Error("single reads: want #7 (a cut description) and #5 (abandoned) only") + } + if none, err := r.PRs(context.Background(), testRepo(), nil, nil); err != nil || len(none) != 0 { + t.Errorf("no heads: %v, %v", none, err) + } +} + +// TestPRsMarkerProperty: only a marker line counts as the stored marker; +// a marker pasted into the description never does. +func TestPRsMarkerProperty(t *testing.T) { + s := newAPIServer(t) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + s.json(repoPath("pullrequests"), http.StatusOK, collection( + prJSON(9, statusActive, syncBranch, "main", bot, "text\n\n"+testLine), + prJSON(8, statusActive, syncBranch, "main", bot, "text"))) + s.json(repoPath("pullRequests", "9", "properties"), http.StatusOK, propsJSON("")) + s.json(repoPath("pullRequests", "8", "properties"), http.StatusOK, propsJSON("not a marker")) + s.json(repoPath("refs"), http.StatusOK, collection()) + prs, err := newTestReader(t, s, testToken(t)).PRs(context.Background(), testRepo(), []string{syncBranch}, []platform.Account{{ID: botID}}) + if err != nil { + t.Fatal(err) + } + for _, pr := range prs { + if pr.Body != "text" { + t.Errorf("#%d body %q, want the description alone", pr.Number, pr.Body) + } + } +} + +func TestOpenPRsBy(t *testing.T) { + s := newAPIServer(t) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + other := repoJSON(otherRepo, "Web", "site", "main") + other["project"].(map[string]any)["id"] = "aaaaaaaa-0000-0000-0000-000000000002" + s.json(apisPath("git", "repositories"), http.StatusOK, collection(repoJSON(repoID, "Billing", "api", "main"), other)) + s.handle(http.MethodGet, "/"+org+"/"+projectID+"/_apis/git/pullrequests", func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + if q.Get("searchCriteria.creatorId") != botID || q.Get("searchCriteria.status") != "active" { + t.Errorf("sweep query %v", q) + } + writeJSON(w, http.StatusOK, collection( + prJSON(7, statusActive, syncBranch, "main", bot, "ours"), + prJSON(8, statusActive, "feature", "main", bot, "another branch"))) + }) + s.json("/"+org+"/aaaaaaaa-0000-0000-0000-000000000002/_apis/git/pullrequests", http.StatusForbidden, errorBody("X", "TF401019")) + s.json(repoPath("pullRequests", "7", "properties"), http.StatusOK, propsJSON(testLine)) + swept, err := newTestReader(t, s, testToken(t)).OpenPRsBy(context.Background(), []platform.Account{{ID: botID}}, []string{syncBranch}) + if err != nil { + t.Fatal(err) + } + if swept.Complete || len(swept.PRs) != 1 || swept.PRs[0].PR.Number != 7 || swept.PRs[0].Repo.Path != "Billing/api" || + !strings.HasSuffix(swept.PRs[0].PR.Body, testLine) || !swept.PRs[0].PR.BaseExists { + t.Errorf("OpenPRsBy = %+v", swept) + } + _, err = newTestReader(t, s, "").OpenPRsBy(context.Background(), []platform.Account{{ID: botID}}, []string{syncBranch}) + wantClass(t, "anonymous sweep", err, platform.ClassAuth) +} diff --git a/internal/platform/azuredevops/repos.go b/internal/platform/azuredevops/repos.go new file mode 100644 index 0000000..4a2f69c --- /dev/null +++ b/internal/platform/azuredevops/repos.go @@ -0,0 +1,205 @@ +package azuredevops + +import ( + "context" + "errors" + "net/url" + "slices" + "strings" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +// apiRepo is a repository as the API reports it (GitRepository). +type apiRepo struct { + ID string `json:"id"` + Name string `json:"name"` + Project *apiProject `json:"project"` + DefaultBranch string `json:"defaultBranch"` + WebURL string `json:"webUrl"` + IsDisabled bool `json:"isDisabled"` + IsFork bool `json:"isFork"` +} + +// apiProject is a repository's project (TeamProjectReference). +type apiProject struct { + ID string `json:"id"` + Name string `json:"name"` + Visibility string `json:"visibility"` +} + +// check reports what the driver depends on that a repository lacks. +func (r *apiRepo) check(op string) error { + switch { + case !guidRe.MatchString(r.ID) || r.Name == "": + return shapeError(op, "a repository without id or name") + case r.Project == nil || !guidRe.MatchString(r.Project.ID) || r.Project.Name == "": + return shapeError(op, "repository %s has no project", r.ID) + case strings.Contains(r.Project.Name, "/") || strings.Contains(r.Name, "/"): + return shapeError(op, "repository %s/%s: a name with a slash", r.Project.Name, r.Name) + } + return nil +} + +// path is the repository's /. +func (r *apiRepo) path() string { return r.Project.Name + "/" + r.Name } + +// toRepo converts an API repository. Its visibility is its project's +// (public or private). A repository without a default branch is empty +// (nothing was pushed yet); a disabled one is reported as such (it shows +// no default branch either, seen anonymously), and the core skips it. +// Azure Repos has no archiving and always takes pull requests. +func (c *client) toRepo(r *apiRepo) platform.Repo { + visibility := "private" + if strings.EqualFold(r.Project.Visibility, "public") { + visibility = "public" + } + web := r.WebURL + if web == "" { + web = c.web + "/" + url.PathEscape(r.Project.Name) + "/_git/" + url.PathEscape(r.Name) + } + return platform.Repo{ + Host: c.host, + ID: strings.ToLower(r.ID), + Path: r.path(), + DefaultBranch: strings.TrimPrefix(r.DefaultBranch, "refs/heads/"), + WebURL: web, + Visibility: visibility, + ObjectFormat: "sha1", + Disabled: r.IsDisabled, + Fork: r.IsFork, + Empty: r.DefaultBranch == "", + } +} + +// Repo returns one repository by its / path: GET +// {org}/{project}/_apis/git/repositories/{name}. The API takes names +// ignoring case; the result carries the canonical path. A missing project +// or repository is ErrNotFound. +func (d *reader) Repo(ctx context.Context, path string) (platform.Repo, error) { + const op = "get repository" + project, name, ok := splitRepoPath(path) + if !ok { + return platform.Repo{}, notFound(op, "repository %q: Azure DevOps paths are project/repository", path) + } + var r apiRepo + if _, err := d.c.get(ctx, op, endpoint(d.c.api, project, "_apis", "git", "repositories", name), nil, &r); err != nil { + return platform.Repo{}, err + } + if err := r.check(op); err != nil { + return platform.Repo{}, err + } + return d.c.toRepo(&r), nil +} + +// getRepo reads a repository by its id: GET +// {org}/_apis/git/repositories/{id}. +func (c *client) getRepo(ctx context.Context, op, id string) (*apiRepo, error) { + var r apiRepo + if _, err := c.get(ctx, op, c.repoAPI(id), nil, &r); err != nil { + return nil, err + } + if err := r.check(op); err != nil { + return nil, err + } + if !strings.EqualFold(r.ID, id) { + return nil, shapeError(op, "repository %s answers as %s", id, r.ID) + } + return &r, nil +} + +// listRepos reads every repository of the organization the identity sees, +// across its projects: GET {org}/_apis/git/repositories, which the API +// answers in one collection. +func (c *client) listRepos(ctx context.Context, op string) ([]apiRepo, error) { + var all list[apiRepo] + if _, err := c.get(ctx, op, c.apis("git", "repositories"), nil, &all); err != nil { + return nil, err + } + for i := range all.Value { + if err := all.Value[i].check(op); err != nil { + return nil, err + } + } + return all.Value, nil +} + +// Resolve lists the repositories sel selects, sorted by path ignoring case. +// A namespace is the organization itself (its name, as the provider's url +// gives it, ignoring case): an org entry selects every repository of every +// project the identity sees, and match narrows it to projects +// (Billing/*). Azure DevOps has no topics, so a selector with topics is +// ClassInvalid (check refuses such an entry before), and no nested +// namespaces, so Subgroups changes nothing. Forks only with sel.Forks. +// Disabled repositories are included, for the core to skip and report. +func (d *reader) Resolve(ctx context.Context, sel platform.Selector) (platform.Resolved, error) { + const op = "resolve" + if sel.Repo != "" { + r, err := d.Repo(ctx, sel.Repo) + if err != nil { + return platform.Resolved{}, err + } + return platform.Resolved{Repos: []platform.Repo{r}, Complete: true}, nil + } + switch { + case sel.Namespace == "": + return platform.Resolved{}, invalid(op, "the selector names no repository and no namespace") + case len(sel.Topics) > 0: + return platform.Resolved{}, &platform.Error{Op: op, Class: platform.ClassInvalid, Err: errTopics} + case !strings.EqualFold(sel.Namespace, d.c.org): + return platform.Resolved{}, notFound(op, "namespace %q: an Azure DevOps provider is the organization %s, and an org entry names it "+ + "(select a project's repositories with match: /*)", sel.Namespace, d.c.org) + } + found, err := d.c.listRepos(ctx, op) + if err != nil { + return platform.Resolved{}, err + } + out := platform.Resolved{Repos: []platform.Repo{}, Complete: true} + seen := map[string]bool{} + for i := range found { + r := &found[i] + id := strings.ToLower(r.ID) + if seen[id] || r.IsFork && !sel.Forks { + continue + } + seen[id] = true + out.Repos = append(out.Repos, d.c.toRepo(r)) + } + sortRepos(out.Repos) + return out, nil +} + +// errTopics is why a selector with topics is refused. +var errTopics = errors.New("repositories on Azure DevOps have no topics; select them with match (/svc-*) or list them by repo") + +// sortRepos orders repositories by path ignoring case, then path, then id. +func sortRepos(repos []platform.Repo) { + slices.SortFunc(repos, func(a, b platform.Repo) int { + if c := strings.Compare(strings.ToLower(a.Path), strings.ToLower(b.Path)); c != 0 { + return c + } + if c := strings.Compare(a.Path, b.Path); c != 0 { + return c + } + return strings.Compare(a.ID, b.ID) + }) +} + +// Remote is the repository's git URL under the provider's url, +// https://dev.azure.com/{org}/{project}/_git/{repository}, never a URL the +// API names, with the credential's Basic header (none when anonymous). +func (d *reader) Remote(_ context.Context, r platform.Repo) (platform.Remote, error) { + const op = "get remote" + if r.Host != "" && !strings.EqualFold(r.Host, d.c.host) { + return platform.Remote{}, invalid(op, "repository %s is on %s, not %s", r.Path, r.Host, d.c.host) + } + project, name, ok := splitRepoPath(r.Path) + if !ok { + return platform.Remote{}, invalid(op, "%q is not a project/repository path", r.Path) + } + rem := platform.Remote{URL: d.c.web + "/" + url.PathEscape(project) + "/_git/" + url.PathEscape(name)} + if d.c.token != "" { + rem.Header = d.c.gitHeader + } + return rem, nil +} diff --git a/internal/platform/azuredevops/repos_test.go b/internal/platform/azuredevops/repos_test.go new file mode 100644 index 0000000..fa9dc55 --- /dev/null +++ b/internal/platform/azuredevops/repos_test.go @@ -0,0 +1,104 @@ +package azuredevops + +import ( + "context" + "encoding/base64" + "errors" + "net/http" + "testing" + + "github.com/bedrock-python/touchmark/internal/platform" +) + +func TestRepo(t *testing.T) { + s := newAPIServer(t) + s.json("/"+org+"/Billing/_apis/git/repositories/API", http.StatusOK, repoJSON(repoID, "Billing", "api", "main")) + s.json("/"+org+"/Billing/_apis/git/repositories/gone", http.StatusNotFound, errorBody(keyRepoNotFound, "TF401019: The Git repository with name or identifier gone does not exist")) + s.json("/"+org+"/My%20Project/_apis/git/repositories/web", http.StatusOK, repoJSON(otherRepo, "My Project", "web", "")) + r := newTestReader(t, s, testToken(t)) + got, err := r.Repo(context.Background(), "Billing/API") + if err != nil { + t.Fatal(err) + } + want := platform.Repo{Host: r.c.host, ID: repoID, Path: "Billing/api", DefaultBranch: "main", + WebURL: "https://dev.azure.com/acme/Billing/_git/api", Visibility: "private", ObjectFormat: "sha1"} + if got.Host != want.Host || got.ID != want.ID || got.Path != want.Path || got.DefaultBranch != want.DefaultBranch || + got.WebURL != want.WebURL || got.Visibility != want.Visibility || got.Empty || got.Disabled || got.Fork { + t.Errorf("Repo = %+v", got) + } + empty, err := r.Repo(context.Background(), "My Project/web") + if err != nil || !empty.Empty || empty.DefaultBranch != "" || empty.Path != "My Project/web" { + t.Errorf("an empty repository: %+v, %v", empty, err) + } + if _, err := r.Repo(context.Background(), "Billing/gone"); !errors.Is(err, platform.ErrNotFound) { + t.Errorf("a missing repository: %v", err) + } + for _, bad := range []string{"api", "a/b/c", "Billing/"} { + if _, err := r.Repo(context.Background(), bad); !errors.Is(err, platform.ErrNotFound) { + t.Errorf("Repo(%q) = %v, want ErrNotFound", bad, err) + } + } +} + +func TestResolve(t *testing.T) { + s := newAPIServer(t) + fork := repoJSON(forkRepo, "Billing", "api-fork", "main") + fork["isFork"] = true + disabled := repoJSON(otherRepo, "Archive", "old", "") + disabled["isDisabled"] = true + public := repoJSON("11111111-2222-3333-4444-555555555555", "web", "Site", "main") + public["project"].(map[string]any)["visibility"] = "public" + s.json(apisPath("git", "repositories"), http.StatusOK, collection(repoJSON(repoID, "Billing", "api", "main"), fork, disabled, public)) + r := newTestReader(t, s, testToken(t)) + res, err := r.Resolve(context.Background(), platform.Selector{Namespace: "ACME"}) + if err != nil { + t.Fatal(err) + } + var paths []string + for _, x := range res.Repos { + paths = append(paths, x.Path) + } + if !res.Complete || len(paths) != 3 || paths[0] != "Archive/old" || paths[1] != "Billing/api" || paths[2] != "web/Site" { + t.Fatalf("Resolve = %v (complete %v)", paths, res.Complete) + } + if !res.Repos[0].Disabled || res.Repos[2].Visibility != "public" { + t.Errorf("disabled %v, visibility %q", res.Repos[0].Disabled, res.Repos[2].Visibility) + } + withForks, err := r.Resolve(context.Background(), platform.Selector{Namespace: org, Forks: true}) + if err != nil || len(withForks.Repos) != 4 { + t.Errorf("with forks: %d, %v", len(withForks.Repos), err) + } + _, err = r.Resolve(context.Background(), platform.Selector{Namespace: org, Topics: []string{"python"}}) + wantClass(t, "topics", err, platform.ClassInvalid) + _, err = r.Resolve(context.Background(), platform.Selector{Namespace: "Billing"}) + if !errors.Is(err, platform.ErrNotFound) { + t.Errorf("a project as namespace: %v", err) + } +} + +func TestRemote(t *testing.T) { + s := newAPIServer(t) + token := testToken(t) + r := newTestReader(t, s, token) + repo := testRepo() + repo.Host = r.c.host + repo.Path = "My Project/api" + rem, err := r.Remote(context.Background(), repo) + if err != nil { + t.Fatal(err) + } + if rem.URL != s.url()+"/My%20Project/_git/api" { + t.Errorf("URL %q", rem.URL) + } + h, err := rem.Header(context.Background()) + if err != nil || h != "Basic "+base64.StdEncoding.EncodeToString([]byte("touchmark:"+token)) { + t.Errorf("git header %q, %v", h, err) + } + anon, err := newTestReader(t, s, "").Remote(context.Background(), repo) + if err != nil || anon.Header != nil { + t.Errorf("anonymous remote: %+v, %v", anon, err) + } + repo.Host = "elsewhere.example" + _, err = r.Remote(context.Background(), repo) + wantClass(t, "another host", err, platform.ClassInvalid) +} diff --git a/internal/platform/azuredevops/target.go b/internal/platform/azuredevops/target.go new file mode 100644 index 0000000..27751b3 --- /dev/null +++ b/internal/platform/azuredevops/target.go @@ -0,0 +1,720 @@ +package azuredevops + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/url" + "slices" + "strconv" + "strings" + "sync" + "time" + "unicode/utf16" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// The security namespace of Git repositories and its permission bits +// (https://learn.microsoft.com/en-us/azure/devops/organizations/security/namespace-reference, +// "Git Repositories": ID 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87, token +// repoV2//; the bits are the namespace's +// actions, as GET _apis/securitynamespaces lists them: GenericContribute 4, +// CreateBranch 16, PullRequestContribute 16384). +const ( + gitNamespace = "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87" + permContribute = 4 + permCreateBranch = 16 + permPullRequestContib = 16384 +) + +// maxDescription is the most characters a description holds (Pull +// Requests - Update), counted as .NET counts them, in UTF-16 code units +// (assumed). +const maxDescription = 4000 + +// Target checks that the writer may write to r and returns a TargetWriter +// bound to it, before the target's writes. The repository is read as the +// writer, by its id (a rename does not matter): one the writer cannot see, +// or one that answers with another id, is ClassNotFound. +// +// The permissions come from Permissions - Has Permissions on the Git +// Repositories namespace, evaluated for the caller: Contents needs +// GenericContribute and CreateBranch (the sync branch is new the first +// time; whoever creates a branch may force-push it), PRs +// PullRequestContribute. One that is missing is ClassPermission with Rule +// "contents" or "pull-requests". Whether a PAT with the Code scopes alone +// may call that API is not documented: when it refuses (401 or 403 once the +// credential itself was accepted), the check is skipped and a push or a +// pull request meets a missing permission instead (TF401027 → +// blocked:permission:push; a 403 → ClassPermission). Azure DevOps has no +// permission of its own for CI files, so Workflows needs nothing more. +func (w *writer) Target(ctx context.Context, r platform.Repo, need platform.Perms) (platform.TargetWriter, error) { + const op = "target" + if _, err := w.c.selfAccount(ctx); err != nil { + return nil, err + } + if _, err := w.Remote(ctx, r); err != nil { + return nil, err + } + repo, err := w.c.targetRepo(ctx, op, r) + if err != nil { + return nil, err + } + checks := []struct { + want bool + bits int + rule string + what string + }{ + {need.Contents, permContribute, "contents", "contribute (push)"}, + {need.Contents, permCreateBranch, "contents", "create branches"}, + {need.PRs, permPullRequestContib, "pull-requests", "contribute to pull requests"}, + } + for _, ch := range checks { + if !ch.want { + continue + } + ok, known, err := w.c.hasPermission(ctx, op, repo, ch.bits) + if err != nil { + return nil, err + } + if !known { + // The API is not open to this token: the writes find out. + break + } + if !ok { + return nil, &platform.Error{Op: op, Class: platform.ClassPermission, Status: http.StatusForbidden, Rule: ch.rule, + Err: fmt.Errorf("the writer may not %s in %s", ch.what, repo.path())} + } + } + pr := w.c.toRepo(repo) + rem, err := w.Remote(ctx, pr) + if err != nil { + return nil, err + } + return &target{c: w.c, repo: pr, remote: rem.URL}, nil +} + +// hasPermission evaluates bits of the Git Repositories namespace on repo +// for the caller: GET {org}/_apis/permissions/{namespace}/{bits}?tokens= +// repoV2//. known is false when the API refuses the +// token (401 or 403): the credential was accepted by connectionData just +// before, so the refusal is the token's scope. A rate limit, a transient +// failure or the end of ctx is err. +func (c *client) hasPermission(ctx context.Context, op string, repo *apiRepo, bits int) (ok, known bool, err error) { + var answer list[bool] + token := "repoV2/" + strings.ToLower(repo.Project.ID) + "/" + strings.ToLower(repo.ID) + q := url.Values{"tokens": {token}, "alwaysAllowAdministrators": {"true"}} + _, err = c.get(ctx, op, c.apis("permissions", gitNamespace, strconv.Itoa(bits)), q, &answer) + switch { + case err == nil: + case platform.ClassOf(err) == platform.ClassAuth && statusOf(err) == http.StatusUnauthorized, + platform.ClassOf(err) == platform.ClassPermission: + return false, false, nil + case stops(err) || platform.ClassOf(err) == platform.ClassTransient: + return false, false, err + default: + return false, false, nil + } + if len(answer.Value) != 1 { + return false, false, shapeError(op, "Has Permissions answered %d values for one token", len(answer.Value)) + } + return answer.Value[0], true, nil +} + +// targetRepo reads r as the identity, by its id. A repository that is gone +// or answers with another id is ClassNotFound. +func (c *client) targetRepo(ctx context.Context, op string, r platform.Repo) (*apiRepo, error) { + if !guidRe.MatchString(r.ID) { + return nil, invalid(op, "repository %s has no id", r.Path) + } + var repo apiRepo + if _, err := c.get(ctx, op, c.repoAPI(r.ID), nil, &repo); err != nil { + return nil, err + } + if err := repo.check(op); err != nil { + return nil, err + } + if !strings.EqualFold(repo.ID, r.ID) { + return nil, notFound(op, "%s is another repository (%s) now, not %s", r.Path, repo.ID, r.ID) + } + return &repo, nil +} + +// target is a platform.TargetWriter. The writer's token is not narrowed: +// Close retires the TargetWriter, and later calls through it fail with +// ClassAuth, as after a revoked per-target token elsewhere. +type target struct { + c *client + repo platform.Repo + remote string + + mu sync.Mutex + closed bool +} + +// live returns closedError(op) after Close. +func (t *target) live(op string) error { + t.mu.Lock() + defer t.mu.Unlock() + if t.closed { + return closedError(op) + } + return nil +} + +// Remote is the repository's URL with the writer's Basic header, which +// fails with ClassAuth after Close. +func (t *target) Remote() platform.Remote { + return platform.Remote{URL: t.remote, Header: func(ctx context.Context) (string, error) { + if err := t.live("git credentials"); err != nil { + return "", err + } + return t.c.gitHeader(ctx) + }} +} + +// Close retires the TargetWriter; closing twice is fine. +func (t *target) Close() error { + t.mu.Lock() + t.closed = true + t.mu.Unlock() + return nil +} + +// apiLabelName names a label to add (WebApiCreateTagRequestData). +type apiLabelName struct { + Name string `json:"name"` +} + +// createPR is the body of POST …/pullrequests. +type createPR struct { + SourceRefName string `json:"sourceRefName"` + TargetRefName string `json:"targetRefName"` + Title string `json:"title"` + Description string `json:"description"` + IsDraft bool `json:"isDraft"` + Labels []apiLabelName `json:"labels,omitempty"` +} + +// updatePR is the body of PATCH …/pullrequests/{id}: only what changes +// (Pull Requests - Update takes status, title, description and, where +// retargeting is on, targetRefName). +type updatePR struct { + Title *string `json:"title,omitempty"` + Description *string `json:"description,omitempty"` + TargetRefName *string `json:"targetRefName,omitempty"` + Status *string `json:"status,omitempty"` +} + +// patchOp is one operation of a JSON patch (application/json-patch+json). +type patchOp struct { + Op string `json:"op"` + Path string `json:"path"` + Value *string `json:"value,omitempty"` +} + +// checkDescription refuses a description Azure DevOps would refuse: more +// than maxDescription characters (UTF-16 code units, assumed). +func checkDescription(op, desc string) error { + if n := len(utf16.Encode([]rune(desc))); n > maxDescription { + return invalid(op, "a description of %d characters, more than Azure DevOps' %d", n, maxDescription) + } + return nil +} + +// CreatePR opens a pull request from np.Head in the repository itself to +// np.Base: a draft when np.Draft (Azure DevOps' own flag), with np.Labels +// by name. np.Body is split (marker.Detach): the description is the body +// without its marker line, and the marker line goes into the pull request's +// property touchmark.marker right after the pull request is made (Pull +// Request Properties - Update). Labels the answer lacks are added one by +// one (whether labels in the create request are applied is to be confirmed +// live). +// +// An active pull request from the same head in the repository itself, to +// whatever base, is returned with ErrExists before anything is written; +// when the POST is refused as a duplicate (409, or 400 with +// GitPullRequestExistsException or TF401179) and such a pull request is +// active then, it is returned with ErrExists too. +// +// The property is written with replace, which is idempotent: a transient +// failure or a rate limit is tried again, up to len(markerDelays) more +// times (storeNewMarker). Should it still fail, the pull request just made +// is abandoned and the call fails with the property's error: an own pull +// request abandoned by the writer without a marker blocks nothing and +// counts as no decline, and the core opens a new one. Only when the +// abandon fails too does the pull request stay active without its marker; +// the core's reconcile then writes the marker through EditPR. A failure of +// a label fails the call with the pull request open and its marker +// stored. +func (t *target) CreatePR(ctx context.Context, np platform.NewPR) (platform.PR, error) { + const op = "create pull request" + if err := t.live(op); err != nil { + return platform.PR{}, err + } + switch { + case np.Head == "" || np.Base == "": + return platform.PR{}, invalid(op, "head and base are required") + case np.Head == np.Base: + return platform.PR{}, invalid(op, "head and base are both %q", np.Head) + case strings.TrimSpace(np.Title) == "": + return platform.PR{}, invalid(op, "the title is empty") + } + desc, line := marker.Detach(np.Body) + if err := checkDescription(op, desc); err != nil { + return platform.PR{}, err + } + open, err := t.activeFrom(ctx, op, np.Head, np.Base) + if err != nil { + return platform.PR{}, err + } + if open != nil { + return t.exists(ctx, op, open, np.Head, 0) + } + body := createPR{ + SourceRefName: "refs/heads/" + np.Head, + TargetRefName: "refs/heads/" + np.Base, + Title: np.Title, + Description: desc, + IsDraft: np.Draft, + } + for _, l := range uniqueLabels(np.Labels) { + body.Labels = append(body.Labels, apiLabelName{Name: l}) + } + var p apiPR + _, err = t.c.call(ctx, op, http.MethodPost, t.c.repoAPI(t.repo.ID, "pullrequests"), nil, "", body, &p) + if err != nil { + if platform.ClassOf(err) == platform.ClassConflict || platform.ClassOf(err) == platform.ClassInvalid && typeKeyOf(err) == keyPRExists { + return t.existing(ctx, op, np, err) + } + return platform.PR{}, err + } + if err := p.check(op); err != nil { + return platform.PR{}, err + } + if line != "" { + if err := t.storeNewMarker(ctx, op, p.ID, line); err != nil { + return platform.PR{}, err + } + } + if err := t.addLabels(ctx, op, &p, np.Labels); err != nil { + return platform.PR{}, err + } + return t.finish(ctx, op, &p, line, true) +} + +// markerDelays are the pauses before the attempts of storeNewMarker after +// the first; a rate limit's RetryAfter wins when it is longer, and one +// longer than maxMarkerWait ends the attempts. abandonTime bounds the +// abandon that follows failed attempts, which runs even when ctx has ended. +var markerDelays = []time.Duration{time.Second, 2 * time.Second, 4 * time.Second} + +const ( + maxMarkerWait = 30 * time.Second + abandonTime = 30 * time.Second +) + +// sleep waits d, or until ctx ends (its error). Tests replace it. +var sleep = func(ctx context.Context, d time.Duration) error { + tm := time.NewTimer(d) + defer tm.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-tm.C: + return nil + } +} + +// storeNewMarker writes line into the property of pull request number, +// which CreatePR has just opened. It is the one write the driver tries +// again itself: replace is idempotent, and a pull request left without its +// marker would block its target (marker-invalid). A transient failure or a +// rate limit is tried again after markerDelays; any other failure, the +// last one, or the end of ctx stops. Then the pull request is abandoned +// and the error says so, with the class of the property's failure (the +// core reconciles a transient one and opens a new pull request). +func (t *target) storeNewMarker(ctx context.Context, op string, number int64, line string) error { + var err error + for i := 0; ; i++ { + if err = t.setMarker(ctx, op, number, line); err == nil { + return nil + } + class := platform.ClassOf(err) + if i == len(markerDelays) || class != platform.ClassTransient && class != platform.ClassRateLimited || ctx.Err() != nil { + break + } + d := markerDelays[i] + if class == platform.ClassRateLimited { + var pe *platform.Error + if errors.As(err, &pe) && pe.RetryAfter > maxMarkerWait { + break + } + if pe != nil { + d = max(d, pe.RetryAfter) + } + } + if sleep(ctx, d) != nil { + break + } + } + actx, cancel := context.WithTimeout(context.WithoutCancel(ctx), abandonTime) + defer cancel() + s := statusAbandoned + var got apiPR + _, aerr := t.c.call(actx, op, http.MethodPatch, t.c.repoAPI(t.repo.ID, "pullrequests", strconv.FormatInt(number, 10)), nil, "", updatePR{Status: &s}, &got) + if aerr == nil && got.Status == statusActive { + aerr = errors.New("the pull request is still active") + } + if aerr != nil { + return fmt.Errorf("%w; #%d stays active without its marker: abandoning it failed: %w", err, number, aerr) + } + return fmt.Errorf("#%d was abandoned, since its marker could not be stored: %w", number, err) +} + +// existing returns the active pull request from np.Head in the repository +// itself with ErrExists, after refused, a refusal of CreatePR's POST as a +// duplicate; refused alone when none is active. +func (t *target) existing(ctx context.Context, op string, np platform.NewPR, refused error) (platform.PR, error) { + match, err := t.activeFrom(ctx, op, np.Head, np.Base) + if err != nil { + return platform.PR{}, errors.Join(refused, err) + } + if match == nil { + return platform.PR{}, refused + } + return t.exists(ctx, op, match, np.Head, statusOf(refused)) +} + +// activeFrom returns the active pull request from head in the repository +// itself, the one to base first; nil when none is active. One from a +// fork's branch of that name is not from head. +func (t *target) activeFrom(ctx context.Context, op, head, base string) (*apiPR, error) { + var match *apiPR + q := url.Values{"searchCriteria.sourceRefName": {"refs/heads/" + head}, "searchCriteria.status": {statusActive}} + complete, err := listPages(ctx, t.c, op, t.c.repoAPI(t.repo.ID, "pullrequests"), q, maxPRPages, func(p apiPR) error { + if err := p.check(op); err != nil { + return err + } + if p.Status != statusActive || p.head() != head || p.ForkSource != nil || !strings.EqualFold(p.Repository.ID, t.repo.ID) { + return nil + } + if match == nil || p.TargetRefName == "refs/heads/"+base && match.TargetRefName != "refs/heads/"+base { + match = &p + } + return nil + }) + switch { + case err != nil: + return nil, err + case !complete && match == nil: + return nil, &platform.Error{Op: op, Class: platform.ClassUnknown, + Err: fmt.Errorf("%s has more active pull requests from %s than %d pages", t.repo.Path, head, maxPRPages)} + } + return match, nil +} + +// exists returns p, the active pull request from head, with the ErrExists +// of CreatePR; status is the HTTP status that told it (0 when a listing +// did). +func (t *target) exists(ctx context.Context, op string, p *apiPR, head string, status int) (platform.PR, error) { + line, err := t.c.markerOf(ctx, op, t.repo.ID, p.ID) + if err != nil { + return platform.PR{}, err + } + pr, err := t.finish(ctx, op, p, line, p.cut()) + if err != nil { + return platform.PR{}, err + } + return pr, &platform.Error{Op: op, Class: platform.ClassConflict, Status: status, + Err: fmt.Errorf("#%d is active from %s: %w", p.ID, head, platform.ErrExists)} +} + +// finish converts a pull request the API answered with, as PRs reports it: +// read alone first when alone is set (a listing's cut description), with +// line, its marker, BaseExists and, for an active one from the repository +// itself, its head. +func (t *target) finish(ctx context.Context, op string, p *apiPR, line string, alone bool) (platform.PR, error) { + if alone { + full, err := t.c.getPR(ctx, op, t.repo.ID, p.ID) + if err != nil { + return platform.PR{}, err + } + *p = *full + } + pr := t.c.toPR(p, line) + var err error + if pr.BaseExists, err = t.c.baseExists(ctx, op, t.repo, pr.Base, map[string]bool{}); err != nil { + return platform.PR{}, err + } + if pr.State == platform.Open && pr.Head != "" && pr.HeadRepoID == pr.RepoID { + if pr.HeadSHA, err = t.c.branchHead(ctx, op, t.repo.ID, pr.Head); err != nil { + return platform.PR{}, err + } + } + return pr, nil +} + +// setMarker writes line into pull request number's property +// touchmark.marker, or removes the property when line is "" (Pull Request +// Properties - Update: replace adds a missing property, remove of a +// missing one does nothing). +func (t *target) setMarker(ctx context.Context, op string, number int64, line string) error { + patch := []patchOp{{Op: "replace", Path: "/" + markerProperty, Value: &line}} + if line == "" { + patch = []patchOp{{Op: "remove", Path: "/" + markerProperty}} + } + _, err := t.c.call(ctx, op, http.MethodPatch, t.c.repoAPI(t.repo.ID, "pullRequests", strconv.FormatInt(number, 10), "properties"), + nil, "application/json-patch+json", patch, nil) + return err +} + +// addLabels adds to p, by name, each of names it lacks (Pull Request +// Labels - Create), and records them on p. +func (t *target) addLabels(ctx context.Context, op string, p *apiPR, names []string) error { + have := map[string]bool{} + for _, l := range p.Labels { + if l.Active == nil || *l.Active { + have[strings.ToLower(l.Name)] = true + } + } + for _, name := range uniqueLabels(names) { + if have[strings.ToLower(name)] { + continue + } + var got apiLabel + if _, err := t.c.call(ctx, op, http.MethodPost, t.c.repoAPI(t.repo.ID, "pullRequests", strconv.FormatInt(p.ID, 10), "labels"), + nil, "", apiLabelName{Name: name}, &got); err != nil { + return err + } + have[strings.ToLower(name)] = true + p.Labels = append(p.Labels, apiLabel{ID: got.ID, Name: name}) + } + return nil +} + +// uniqueLabels returns the non-empty names, each once (ignoring case), in +// order. +func uniqueLabels(names []string) []string { + var out []string + seen := map[string]bool{} + for _, n := range names { + if k := strings.ToLower(strings.TrimSpace(n)); k != "" && !seen[k] { + seen[k] = true + out = append(out, strings.TrimSpace(n)) + } + } + return out +} + +// EditPR changes a pull request. It reads the pull request and its stored +// marker first. The description changes only when e.Body without its +// marker line differs from the current description without any marker +// line a person pasted into it (marker.Strip), line endings and trailing +// whitespace aside (sameDescription): an edit that keeps people's +// description, as a close or a marker refresh does where the marker lives +// apart, sends no description at all. +// +// - Active, State Closed: the marker line of e.Body goes into the +// property first (when it differs from the stored one), then the +// labels to add, then one PATCH with what changes of title, +// description, base and status abandoned: the closing marker is stored +// before the pull request is abandoned, after which touchmark never +// writes to it again. A description that would not fit Azure DevOps' +// limit stays as it is: the close does not wait on a description +// people lengthened. +// - Active otherwise: one PATCH with what changes of title, description +// and base first, then the labels, then the property. A failure in +// between leaves the marker of the previous body, which the next run +// sees as an edit still to make, never a marker that claims a +// description the pull request does not have. +// - Abandoned: touchmark never reactivates one, nor changes it +// (Caps.ClosedImmutable): State Open is ClassUnsupported, and so is any +// change of title, body or base; State Closed alone changes nothing, +// and neither does State Closed with the marker already stored, whatever +// the description (a close that left it as it was). +// - Completed: any change is ClassConflict, as a pull request merged +// since the core read it. +// +// State Merged is ClassUnsupported: touchmark completes no pull request. +// The draft flag never changes; labels are only added. +func (t *target) EditPR(ctx context.Context, number int64, e platform.PREdit) (platform.PR, error) { + const op = "edit pull request" + if err := t.live(op); err != nil { + return platform.PR{}, err + } + switch { + case number <= 0: + return platform.PR{}, notFound(op, "pull request #%d", number) + case e.Title != nil && strings.TrimSpace(*e.Title) == "": + return platform.PR{}, invalid(op, "the title is empty") + case e.State != nil && *e.State == platform.Merged: + return platform.PR{}, unsupported(op, "touchmark completes no pull request") + case e.State != nil && *e.State != platform.Open && *e.State != platform.Closed: + return platform.PR{}, invalid(op, "state %q: only open and closed can be set", *e.State) + case e.Base != nil && *e.Base == "": + return platform.PR{}, invalid(op, "the base is empty") + } + cur, err := t.c.getPR(ctx, op, t.repo.ID, number) + if err != nil { + return platform.PR{}, err + } + stored, err := t.c.markerOf(ctx, op, t.repo.ID, number) + if err != nil { + return platform.PR{}, err + } + curDesc := "" + if cur.Description != nil { + curDesc = *cur.Description + } + var desc, line string + if e.Body != nil { + desc, line = marker.Detach(*e.Body) + } + base := strings.TrimPrefix(cur.TargetRefName, "refs/heads/") + closing := e.State != nil && *e.State == platform.Closed + newTitle := e.Title != nil && *e.Title != cur.Title + newDesc := e.Body != nil && !sameDescription(desc, curDesc) + newMarker := e.Body != nil && line != stored + newBase := e.Base != nil && *e.Base != base + switch cur.Status { + case statusCompleted: + if newTitle || newDesc || newMarker || newBase || e.State != nil { + return platform.PR{}, conflict(op, "#%d is completed: it can no longer be changed", number) + } + return t.finish(ctx, op, cur, stored, false) + case statusAbandoned: + switch { + case e.State != nil && *e.State == platform.Open: + return platform.PR{}, unsupported(op, "#%d is abandoned: touchmark never reactivates a pull request", number) + case closing && !newTitle && !newMarker && !newBase: + // Closed again with its marker: the description may be one the + // close left as it was. + case newTitle || newDesc || newMarker || newBase: + return platform.PR{}, unsupported(op, "#%d is abandoned: touchmark changes active pull requests only", number) + } + return t.finish(ctx, op, cur, stored, false) + } + if newDesc { + if err := checkDescription(op, desc); err != nil { + if !closing { + return platform.PR{}, err + } + newDesc = false + } + } + var patch updatePR + if newTitle { + patch.Title = e.Title + } + if newDesc { + patch.Description = &desc + } + if newBase { + ref := "refs/heads/" + *e.Base + patch.TargetRefName = &ref + } + if closing { + s := statusAbandoned + patch.Status = &s + if newMarker { + if err := t.setMarker(ctx, op, number, line); err != nil { + return platform.PR{}, err + } + stored = line + } + if err := t.addLabels(ctx, op, cur, e.AddLabels); err != nil { + return platform.PR{}, err + } + if cur, err = t.patchPR(ctx, op, cur, patch); err != nil { + return platform.PR{}, err + } + return t.finish(ctx, op, cur, stored, cur.cut()) + } + if cur, err = t.patchPR(ctx, op, cur, patch); err != nil { + return platform.PR{}, err + } + if err := t.addLabels(ctx, op, cur, e.AddLabels); err != nil { + return platform.PR{}, err + } + if newMarker { + if err := t.setMarker(ctx, op, number, line); err != nil { + return platform.PR{}, err + } + stored = line + } + return t.finish(ctx, op, cur, stored, cur.cut()) +} + +// patchPR sends patch to pull request cur (Pull Requests - Update) and +// returns the pull request it answers with, with cur's labels when the +// answer lists none; cur itself when patch changes nothing. A patch that +// abandons the pull request and answers an active one is ClassConflict. +func (t *target) patchPR(ctx context.Context, op string, cur *apiPR, patch updatePR) (*apiPR, error) { + if patch == (updatePR{}) { + return cur, nil + } + var got apiPR + if _, err := t.c.call(ctx, op, http.MethodPatch, t.c.repoAPI(t.repo.ID, "pullrequests", strconv.FormatInt(cur.ID, 10)), nil, "", patch, &got); err != nil { + return nil, err + } + if err := got.check(op); err != nil { + return nil, err + } + if patch.Status != nil && got.Status == statusActive { + return nil, conflict(op, "Azure DevOps did not abandon #%d", cur.ID) + } + if len(got.Labels) == 0 { + got.Labels = cur.Labels + } + return &got, nil +} + +// sameDescription reports whether desc, a description touchmark would +// write, is the current description cur, which may hold marker lines a +// person pasted (marker.Strip drops them, as toPR does), whatever line +// endings and trailing whitespace either has. +func sameDescription(desc, cur string) bool { + norm := func(s string) string { + return strings.TrimRight(strings.ReplaceAll(s, "\r\n", "\n"), " \t\r\n") + } + return norm(marker.Strip(desc)) == norm(marker.Strip(cur)) +} + +// Comment adds a comment to pull request number: a new thread with one +// text comment, closed so that it never waits for a resolution (Pull +// Request Threads - Create; a closed thread does not block the policy that +// asks for resolved comments). Azure DevOps takes comments on abandoned and +// completed pull requests too. +func (t *target) Comment(ctx context.Context, number int64, body string) error { + const op = "comment" + if err := t.live(op); err != nil { + return err + } + switch { + case number <= 0: + return notFound(op, "pull request #%d", number) + case strings.TrimSpace(body) == "": + return invalid(op, "the comment is empty") + } + thread := map[string]any{ + "comments": []map[string]any{{"parentCommentId": 0, "content": body, "commentType": "text"}}, + "status": "closed", + } + _, err := t.c.call(ctx, op, http.MethodPost, t.c.repoAPI(t.repo.ID, "pullRequests", strconv.FormatInt(number, 10), "threads"), nil, "", thread, nil) + return err +} + +// EnsureLabels returns the names as they are: Azure DevOps creates a label +// when a pull request first gets it by name (Pull Request Labels - Create), +// so there is nothing to create beforehand. +func (t *target) EnsureLabels(_ context.Context, names []string) ([]string, error) { + if err := t.live("ensure labels"); err != nil { + return nil, err + } + return slices.Clone(uniqueLabels(names)), nil +} diff --git a/internal/platform/azuredevops/target_test.go b/internal/platform/azuredevops/target_test.go new file mode 100644 index 0000000..f41c6ae --- /dev/null +++ b/internal/platform/azuredevops/target_test.go @@ -0,0 +1,391 @@ +package azuredevops + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" +) + +// targetWorld declares what Target reads: the writer's identity, the +// repository by its id, and Has Permissions answering allow for every bit +// (perms decides per bit; a missing bit is denied). +func targetWorld(t *testing.T, s *apiServer, perms map[int]bool) { + t.Helper() + s.json(apisPath("connectionData"), http.StatusOK, connJSON(botID, "Microsoft.IdentityModel.Claims.ClaimsIdentity;x\\bot@acme.example", "bot@acme.example")) + s.json(repoPath(), http.StatusOK, repoJSON(repoID, "Billing", "api", "main")) + for _, bits := range []int{permContribute, permCreateBranch, permPullRequestContib} { + bits := bits + s.handle(http.MethodGet, apisPath("permissions", gitNamespace, fmt.Sprint(bits)), func(w http.ResponseWriter, r *http.Request) { + if got := r.URL.Query().Get("tokens"); got != "repoV2/"+projectID+"/"+repoID { + t.Errorf("security token %q", got) + } + allow, ok := perms[bits] + writeJSON(w, http.StatusOK, collection(ok && allow)) + }) + } +} + +// allPerms allows everything the writer needs. +var allPerms = map[int]bool{permContribute: true, permCreateBranch: true, permPullRequestContib: true} + +func newTarget(t *testing.T, s *apiServer) *target { + t.Helper() + w := newTestWriter(t, s, testToken(t)) + repo := testRepo() + repo.Host = w.c.host + tw, err := w.Target(context.Background(), repo, platform.Perms{Contents: true, PRs: true}) + if err != nil { + t.Fatal(err) + } + return tw.(*target) +} + +func TestTargetPermissions(t *testing.T) { + for _, tc := range []struct { + name string + perms map[int]bool + need platform.Perms + rule string + }{ + {"all", allPerms, platform.Perms{Contents: true, PRs: true}, ""}, + {"no push", map[int]bool{permCreateBranch: true, permPullRequestContib: true}, platform.Perms{Contents: true, PRs: true}, "contents"}, + {"no branch", map[int]bool{permContribute: true, permPullRequestContib: true}, platform.Perms{Contents: true}, "contents"}, + {"no pull requests", map[int]bool{permContribute: true, permCreateBranch: true}, platform.Perms{Contents: true, PRs: true}, "pull-requests"}, + {"pull requests only", map[int]bool{permPullRequestContib: true}, platform.Perms{PRs: true}, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + s := newAPIServer(t) + targetWorld(t, s, tc.perms) + w := newTestWriter(t, s, testToken(t)) + repo := testRepo() + repo.Host = w.c.host + _, err := w.Target(context.Background(), repo, tc.need) + var pe *platform.Error + switch { + case tc.rule == "" && err != nil: + t.Errorf("Target: %v", err) + case tc.rule != "" && (!errors.As(err, &pe) || pe.Class != platform.ClassPermission || pe.Rule != tc.rule): + t.Errorf("Target = %v, want ClassPermission rule %s", err, tc.rule) + } + }) + } +} + +// TestTargetWithoutPermissionsAPI: a token the permissions API refuses +// gets its TargetWriter; a repository that is another one now is gone. +func TestTargetWithoutPermissionsAPI(t *testing.T) { + s := newAPIServer(t) + targetWorld(t, s, allPerms) + for _, bits := range []int{permContribute, permCreateBranch, permPullRequestContib} { + s.json(apisPath("permissions", gitNamespace, fmt.Sprint(bits)), http.StatusUnauthorized, + errorBody("UnauthorizedRequestException", "TF400813: The user is not authorized to access this resource.")) + } + w := newTestWriter(t, s, testToken(t)) + repo := testRepo() + repo.Host = w.c.host + if _, err := w.Target(context.Background(), repo, platform.Perms{Contents: true, PRs: true}); err != nil { + t.Errorf("Target without the permissions API: %v", err) + } + if n := len(s.requests(http.MethodGet, apisPath("permissions", gitNamespace, fmt.Sprint(permContribute)))); n != 1 { + t.Errorf("%d permission requests, want 1 (the rest skipped)", n) + } + s.json(repoPath(), http.StatusOK, repoJSON(otherRepo, "Billing", "api", "main")) + _, err := w.Target(context.Background(), repo, platform.Perms{Contents: true}) + if !errors.Is(err, platform.ErrNotFound) { + t.Errorf("another repository now: %v", err) + } +} + +// createWorld declares the routes of CreatePR on top of targetWorld: no +// active pull request until the POST, which answers #11; its properties +// and labels. +func createWorld(t *testing.T, s *apiServer) (created *map[string]any, props *[]patchOp) { + t.Helper() + targetWorld(t, s, allPerms) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + var pr map[string]any + var stored []patchOp + line := "" + s.handle(http.MethodGet, repoPath("pullrequests"), func(w http.ResponseWriter, _ *http.Request) { + if pr == nil { + writeJSON(w, http.StatusOK, collection()) + return + } + writeJSON(w, http.StatusOK, collection(pr)) + }) + s.handle(http.MethodPost, repoPath("pullrequests"), func(w http.ResponseWriter, r *http.Request) { + var in createPR + if err := json.NewDecoder(r.Body).Decode(&in); err != nil { + t.Fatal(err) + } + pr = prJSON(11, statusActive, strings.TrimPrefix(in.SourceRefName, "refs/heads/"), strings.TrimPrefix(in.TargetRefName, "refs/heads/"), bot, in.Description) + pr["isDraft"] = in.IsDraft + pr["title"] = in.Title + if len(in.Labels) > 0 { + // Only the first label is applied by the create request. + pr["labels"] = []any{map[string]any{"id": "l0", "name": in.Labels[0].Name, "active": true}} + } + writeJSON(w, http.StatusCreated, pr) + }) + s.handle(http.MethodGet, repoPath("pullrequests", "11"), func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, pr) }) + s.handle(http.MethodPatch, repoPath("pullRequests", "11", "properties"), func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Content-Type") != "application/json-patch+json" { + t.Errorf("properties sent as %q", r.Header.Get("Content-Type")) + } + var ops []patchOp + if err := json.NewDecoder(r.Body).Decode(&ops); err != nil { + t.Fatal(err) + } + stored = append(stored, ops...) + for _, op := range ops { + if op.Op == "remove" { + line = "" + } else if op.Value != nil { + line = *op.Value + } + } + writeJSON(w, http.StatusOK, propsJSON(line)) + }) + s.handle(http.MethodGet, repoPath("pullRequests", "11", "properties"), func(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, propsJSON(line)) + }) + s.handle(http.MethodPost, repoPath("pullRequests", "11", "labels"), func(w http.ResponseWriter, r *http.Request) { + var in apiLabelName + _ = json.NewDecoder(r.Body).Decode(&in) + labels, _ := pr["labels"].([]any) + pr["labels"] = append(labels, map[string]any{"id": "l" + in.Name, "name": in.Name, "active": true}) + writeJSON(w, http.StatusOK, map[string]any{"id": "l" + in.Name, "name": in.Name, "active": true}) + }) + s.json(repoPath("refs"), http.StatusOK, collection(map[string]any{"name": "refs/heads/" + syncBranch, "objectId": headSHA})) + return &pr, &stored +} + +func TestCreatePR(t *testing.T) { + s := newAPIServer(t) + _, stored := createWorld(t, s) + tw := newTarget(t, s) + body := "## Sync\n\ntext\n\n" + testLine + pr, err := tw.CreatePR(context.Background(), platform.NewPR{Head: syncBranch, Base: "main", Title: "chore: sync", + Body: body, Labels: []string{"engineering-assets", "deps"}, Draft: true}) + if err != nil { + t.Fatal(err) + } + posts := s.requests(http.MethodPost, repoPath("pullrequests")) + var in createPR + if len(posts) != 1 || json.Unmarshal(posts[0].Body, &in) != nil { + t.Fatalf("POSTs %d", len(posts)) + } + if in.Description != "## Sync\n\ntext" || !in.IsDraft || in.SourceRefName != "refs/heads/"+syncBranch || len(in.Labels) != 2 { + t.Errorf("POST %+v: the description must not hold the marker", in) + } + if len(*stored) != 1 || (*stored)[0].Path != "/touchmark.marker" || *(*stored)[0].Value != testLine { + t.Errorf("properties %+v", *stored) + } + if pr.Number != 11 || pr.Body != body || !pr.Draft || pr.HeadSHA != headSHA || len(pr.Labels) != 2 { + t.Errorf("CreatePR = %+v", pr) + } + if n := len(s.requests(http.MethodPost, repoPath("pullRequests", "11", "labels"))); n != 1 { + t.Errorf("%d labels added after the create, want 1", n) + } + + // The same head again: the active pull request, with ErrExists. + again, err := tw.CreatePR(context.Background(), platform.NewPR{Head: syncBranch, Base: "main", Title: "chore: sync", Body: body}) + if !errors.Is(err, platform.ErrExists) || again.Number != 11 || again.Body != body { + t.Errorf("a second CreatePR = #%d, %v", again.Number, err) + } + + // A description over 4 000 characters is refused before anything. + _, err = tw.CreatePR(context.Background(), platform.NewPR{Head: "other", Base: "main", Title: "t", Body: strings.Repeat("ж", 4001) + "\n\n" + testLine}) + wantClass(t, "a long description", err, platform.ClassInvalid) +} + +// TestCreatePRDuplicate: a POST refused as a duplicate returns the active +// pull request that appeared meanwhile. +func TestCreatePRDuplicate(t *testing.T) { + s := newAPIServer(t) + targetWorld(t, s, allPerms) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + calls := 0 + s.handle(http.MethodGet, repoPath("pullrequests"), func(w http.ResponseWriter, _ *http.Request) { + calls++ + if calls == 1 { + writeJSON(w, http.StatusOK, collection()) + return + } + writeJSON(w, http.StatusOK, collection(prJSON(12, statusActive, syncBranch, "main", bot, "theirs"))) + }) + s.json(repoPath("pullRequests", "12", "properties"), http.StatusOK, propsJSON(testLine)) + s.json(repoPath("refs"), http.StatusOK, collection()) + s.handle(http.MethodPost, repoPath("pullrequests"), func(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusConflict, errorBody(keyPRExists, tfPRExists+": An active pull request for the source and target branch already exists.")) + }) + pr, err := newTarget(t, s).CreatePR(context.Background(), platform.NewPR{Head: syncBranch, Base: "main", Title: "t", Body: "x\n\n" + testLine}) + if !errors.Is(err, platform.ErrExists) || pr.Number != 12 { + t.Errorf("CreatePR = #%d, %v", pr.Number, err) + } +} + +// editWorld declares an active #11 with a stored marker, and the PATCH of +// the pull request. +func editWorld(t *testing.T, s *apiServer, status string) (patches *[]updatePR) { + t.Helper() + patches, _ = editWorldPR(t, s, status) + return patches +} + +// editWorldPR is editWorld that also returns #11 as the routes answer it. +func editWorldPR(t *testing.T, s *apiServer, status string) (patches *[]updatePR, pr *map[string]any) { + t.Helper() + pr, _ = createWorld(t, s) + bot := identityRef(botID, "touchmark bot", "aad.Ym90") + *pr = prJSON(11, status, syncBranch, "main", bot, "old text") + var got []updatePR + s.handle(http.MethodPatch, repoPath("pullrequests", "11"), func(w http.ResponseWriter, r *http.Request) { + var in updatePR + if err := json.NewDecoder(r.Body).Decode(&in); err != nil { + t.Fatal(err) + } + got = append(got, in) + if in.Description != nil { + (*pr)["description"] = *in.Description + } + if in.Title != nil { + (*pr)["title"] = *in.Title + } + if in.Status != nil { + (*pr)["status"] = *in.Status + (*pr)["closedDate"] = "2026-03-01T00:00:00Z" + } + writeJSON(w, http.StatusOK, *pr) + }) + return &got, pr +} + +func TestEditPR(t *testing.T) { + s := newAPIServer(t) + patches := editWorld(t, s, statusActive) + tw := newTarget(t, s) + newLine := strings.Replace(testLine, "key=sha256:6b1f", "key=sha256:0000", 1) + body := "new text\n\n" + newLine + closed := platform.Closed + pr, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body, State: &closed, AddLabels: []string{"engineering-assets"}}) + if err != nil { + t.Fatal(err) + } + // The property first, then the labels, then one PATCH. + var order []string + for _, c := range s.writes() { + order = append(order, c.Method+" "+c.Path[strings.LastIndex(c.Path, "/")+1:]) + } + if strings.Join(order, ", ") != "PATCH properties, POST labels, PATCH 11" { + t.Errorf("writes %v", order) + } + if len(*patches) != 1 || (*patches)[0].Description == nil || *(*patches)[0].Description != "new text" || + (*patches)[0].Status == nil || *(*patches)[0].Status != "abandoned" || (*patches)[0].Title != nil { + t.Errorf("PATCH %+v", *patches) + } + if pr.State != platform.Closed || pr.Body != body { + t.Errorf("EditPR = %+v", pr) + } + if _, line := marker.Detach(pr.Body); line != newLine { + t.Errorf("the stored marker %q", line) + } + + // Abandoned now: never reactivated, never changed; closing again is + // nothing. + s.reset() + open := platform.Open + _, err = tw.EditPR(context.Background(), 11, platform.PREdit{State: &open}) + wantClass(t, "reopen", err, platform.ClassUnsupported) + other := "other\n\n" + newLine + _, err = tw.EditPR(context.Background(), 11, platform.PREdit{Body: &other}) + wantClass(t, "edit an abandoned one", err, platform.ClassUnsupported) + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{State: &closed, Body: &body}); err != nil { + t.Errorf("closing an abandoned one with its own body: %v", err) + } + if w := s.writes(); len(w) != 0 { + t.Errorf("writes to an abandoned pull request: %v", w) + } +} + +// TestEditPRUnchanged: an edit that changes nothing writes nothing, and a +// title alone is one PATCH without description. +func TestEditPRUnchanged(t *testing.T) { + s := newAPIServer(t) + patches := editWorld(t, s, statusActive) + tw := newTarget(t, s) + same := "old text\n\n" + testLine + // The marker alone goes to the property, without a PATCH of the pull + // request. + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &same}); err != nil || len(*patches) != 0 { + t.Fatalf("setup: %v, %+v", err, *patches) + } + s.reset() + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &same}); err != nil { + t.Fatal(err) + } + if w := s.writes(); len(w) != 0 { + t.Errorf("writes for no change: %+v", w) + } + title := "chore: new title" + if _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Title: &title}); err != nil { + t.Fatal(err) + } + if len(*patches) != 1 || (*patches)[0].Title == nil || (*patches)[0].Description != nil || (*patches)[0].Status != nil { + t.Errorf("PATCH %+v", *patches) + } +} + +func TestEditPRCompleted(t *testing.T) { + s := newAPIServer(t) + editWorld(t, s, statusCompleted) + tw := newTarget(t, s) + body := "x\n\n" + testLine + _, err := tw.EditPR(context.Background(), 11, platform.PREdit{Body: &body}) + wantClass(t, "a completed pull request", err, platform.ClassConflict) + merged := platform.Merged + _, err = tw.EditPR(context.Background(), 11, platform.PREdit{State: &merged}) + wantClass(t, "merging", err, platform.ClassUnsupported) +} + +func TestCommentAndLabels(t *testing.T) { + s := newAPIServer(t) + targetWorld(t, s, allPerms) + s.handle(http.MethodPost, repoPath("pullRequests", "11", "threads"), func(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, map[string]any{"id": 1}) + }) + tw := newTarget(t, s) + if err := tw.Comment(context.Background(), 11, "hello"); err != nil { + t.Fatal(err) + } + var thread struct { + Comments []struct { + Content string `json:"content"` + CommentType string `json:"commentType"` + } `json:"comments"` + Status string `json:"status"` + } + posts := s.requests(http.MethodPost, repoPath("pullRequests", "11", "threads")) + if len(posts) != 1 || json.Unmarshal(posts[0].Body, &thread) != nil || len(thread.Comments) != 1 || + thread.Comments[0].Content != "hello" || thread.Comments[0].CommentType != "text" || thread.Status != "closed" { + t.Errorf("thread %s", posts[0].Body) + } + names, err := tw.EnsureLabels(context.Background(), []string{"a", "A", "b", ""}) + if err != nil || strings.Join(names, ",") != "a,b" { + t.Errorf("EnsureLabels = %v, %v", names, err) + } + if err := tw.Close(); err != nil { + t.Fatal(err) + } + wantClass(t, "after Close", tw.Comment(context.Background(), 11, "x"), platform.ClassAuth) + if _, err := tw.Remote().Header(context.Background()); platform.ClassOf(err) != platform.ClassAuth { + t.Errorf("git credentials after Close: %v", err) + } +} diff --git a/internal/platform/fake/fake.go b/internal/platform/fake/fake.go index 5578c66..00fffd8 100644 --- a/internal/platform/fake/fake.go +++ b/internal/platform/fake/fake.go @@ -16,7 +16,11 @@ // quick actions are refused (git mode runs them and records a violation); // closers are not reported on Gitea and Forgejo; on Bitbucket Cloud labels // are refused and a pull request closed without merging can never be -// edited or reopened (Caps.ClosedImmutable). Branch rulesets, the +// edited or reopened (Caps.ClosedImmutable), and so on Azure DevOps, whose +// marker lives in a pull request property (Caps.MarkerInProperties: the +// description alone is held to Caps.MaxBody, a marker line in it never +// counts, and a person editing the description leaves the stored marker +// as it is). Branch rulesets, the // Preflighter of GitHub's driver (WithPreflight) and its API commits // (WithAPICommits, platform.Committer) are modeled in rules.go. // @@ -51,6 +55,7 @@ import ( "time" "github.com/bedrock-python/touchmark/internal/gitx" + "github.com/bedrock-python/touchmark/internal/marker" "github.com/bedrock-python/touchmark/internal/platform" "github.com/bedrock-python/touchmark/internal/snapshot" ) @@ -66,6 +71,8 @@ const ( Forgejo Flavor = "forgejo" // Bitbucket is Bitbucket Cloud. Bitbucket Flavor = "bitbucket" + // AzureDevOps is Azure DevOps Services. + AzureDevOps Flavor = "azure-devops" ) // Tree entry modes. @@ -84,8 +91,11 @@ var Epoch = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) // Markdown reference definition on Bitbucket, MarkerInRefDef) and has no // API commit (WithAPICommits adds one). Bitbucket's are those its driver's // Probe reports: native drafts, no labels, a known closer, closed pull -// requests immutable, no permission of its own for CI files. A flavor other -// than the five known ones gets GitHub's capabilities under its own name. +// requests immutable, no permission of its own for CI files. Azure +// DevOps' are its driver's too: a 4 000-byte description with the marker +// in a property (MarkerInProperties), native drafts, labels by name, a +// known closer, closed pull requests immutable. A flavor other than the +// six known ones gets GitHub's capabilities under its own name. func CapsFor(f Flavor) platform.Caps { c := platform.Caps{ Flavor: string(f), @@ -116,6 +126,12 @@ func CapsFor(f Flavor) platform.Caps { c.Marker = platform.MarkerInRefDef c.RuntimeOnly = []string{"branch-restrictions"} c.Limits = platform.Limits{Reads: 2, GitReads: 2, ReadsPerMinute: 15, MinInterval: time.Second} + case AzureDevOps: + c.MaxBody = 4000 + c.ClosedImmutable = true + c.Marker = platform.MarkerInProperties + c.RuntimeOnly = []string{"branch-policies", "push-policies"} + c.Limits = platform.Limits{Reads: 4, GitReads: 2, ReadsPerMinute: 120, WritesPerMinute: 30, MinInterval: 500 * time.Millisecond} default: c.WorkflowPerm = true c.Limits = platform.Limits{Reads: 8, GitReads: 4, WritesPerMinute: 60, WritesPerHour: 450, MinInterval: time.Second} @@ -174,6 +190,10 @@ type Platform struct { calls []call err error // the first setup error + // markerFaults are the failures of the marker property queued by + // FailMarkerNext. + markerFaults []markerFault + // logRequests turns on the log of API requests (WithRequestLog). logRequests bool requests []Request @@ -230,6 +250,13 @@ type fault struct { applied bool } +// markerFault is a failed write of the marker property (FailMarkerNext): +// the error, and whether CreatePR leaves the pull request active. +type markerFault struct { + err error + leaveOpen bool +} + type call struct { method string args []string @@ -699,6 +726,7 @@ func (p *Platform) AddPR(repoID string, pr platform.PR) int64 { if pr.HeadRepoID == "" { pr.HeadRepoID = repoID } + pr.Body = p.stored(pr.Body) if err := p.normalizePR(s, &pr); err != nil { p.setupf("AddPR(%s): %v", repoID, err) return 0 @@ -749,7 +777,13 @@ func (p *Platform) SetPRState(repoID string, number int64, state platform.PRStat // move its head. Number and RepoID cannot change; the result is checked // and completed like AddPR's, except that BaseExists and HeadRepoID are // kept as fn leaves them: an empty HeadRepoID is a PR whose fork was -// deleted. +// deleted. Where the marker lives in a property (Caps.MarkerInProperties) +// the body fn leaves is the description, without any marker line in it, +// and the stored marker stays, as when a person edits the description in +// the web UI, which shows no properties. That is no protection of the +// marker: whoever may contribute to pull requests may be able to write +// their properties through the API (project Readers included, by default; +// see docs/project/threat-model.md), which this fake does not model. func (p *Platform) UpdatePR(repoID string, number int64, fn func(*platform.PR)) { p.mu.Lock() defer p.mu.Unlock() @@ -759,6 +793,10 @@ func (p *Platform) UpdatePR(repoID string, number int64, fn func(*platform.PR)) } pr := p.storedPR(ps.pr) fn(&pr) + if p.apart() { + _, line := marker.Detach(ps.pr.Body) + pr.Body = marker.Attach(pr.Body, line) + } pr.Number, pr.RepoID = number, repoID if err := p.normalizePR(p.repos[repoID], &pr); err != nil { p.setupf("UpdatePR(%s, #%d): %v", repoID, number, err) @@ -969,6 +1007,41 @@ func (p *Platform) queue(method string, f fault) { p.queued[method] = append(p.queued[method], f) } +// FailMarkerNext makes the next write of the marker property fail with +// err, where the marker lives in a property (Caps.MarkerInProperties), as +// Azure DevOps' driver meets it once its own attempts are spent; the rest +// of the call takes effect, in the driver's order. CreatePR makes the pull +// request without its marker and the writer abandons it at once (it stays +// active with leaveOpen, as when the abandon fails too). EditPR of an +// active pull request that is not a close changes its title, description, +// base and labels and keeps the previous marker; a close changes nothing +// (its marker is stored before anything else). Either fails with err. +// Faults queue; a call that writes no new marker line takes none. On a +// flavor that keeps the marker in the body it is a setup error. +func (p *Platform) FailMarkerNext(err error, leaveOpen bool) { + p.mu.Lock() + defer p.mu.Unlock() + if !p.apart() { + p.setupf("FailMarkerNext: %s keeps the marker in the body", p.caps.Flavor) + return + } + if err == nil { + return + } + p.markerFaults = append(p.markerFaults, markerFault{err: err, leaveOpen: leaveOpen}) +} + +// takeMarkerFault returns the next queued failure of the marker property, +// if any. Called with p.mu held. +func (p *Platform) takeMarkerFault() (markerFault, bool) { + if len(p.markerFaults) == 0 { + return markerFault{}, false + } + f := p.markerFaults[0] + p.markerFaults = p.markerFaults[1:] + return f, true +} + // Fail makes every call of method (or "*") fail with err, without effect, // until Fail(method, nil). Queued faults fire first. func (p *Platform) Fail(method string, err error) { @@ -1154,6 +1227,21 @@ func (p *Platform) storedPR(pr platform.PR) platform.PR { return pr } +// apart reports whether the flavor keeps the marker apart from the +// description (Caps.MarkerInProperties). +func (p *Platform) apart() bool { return p.caps.Marker == platform.MarkerInProperties } + +// stored is body as the platform keeps it: where the marker lives in a +// property, the description without marker lines followed by the last +// marker line of body (marker.Attach of marker.Detach), as the driver reads +// it back; body itself elsewhere. +func (p *Platform) stored(body string) string { + if !p.apart() { + return body + } + return marker.Attach(marker.Detach(body)) +} + // prView is a pull request as the API reports it: without the closer of a // PR closed unmerged when the flavor does not report closers. func (p *Platform) prView(pr platform.PR) platform.PR { @@ -1174,6 +1262,9 @@ func (p *Platform) prURL(s *repoState, number int64) string { return base + "/pulls/" + n case Bitbucket: return base + "/pull-requests/" + n + case AzureDevOps: + project, name, _ := strings.Cut(s.repo.Path, "/") + return "https://" + p.host + "/" + project + "/_git/" + name + "/pullrequest/" + n } return base + "/pull/" + n } diff --git a/internal/platform/fake/fake_test.go b/internal/platform/fake/fake_test.go index cd6078a..f25e51b 100644 --- a/internal/platform/fake/fake_test.go +++ b/internal/platform/fake/fake_test.go @@ -46,7 +46,7 @@ func TestCapsFor(t *testing.T) { t.Errorf("Probe with an unknown account = %+v, %v; want an auth error", got, err) } } - if c := fake.CapsFor("azure-devops"); c.Flavor != "azure-devops" || !c.WorkflowPerm || c.Draft != platform.DraftNative { + if c := fake.CapsFor("sourcehut"); c.Flavor != "sourcehut" || !c.WorkflowPerm || c.Draft != platform.DraftNative { t.Errorf("CapsFor(unknown) = %+v, want GitHub's under its own name", c) } } diff --git a/internal/platform/fake/gitserve.go b/internal/platform/fake/gitserve.go index aaa514c..4b12a11 100644 --- a/internal/platform/fake/gitserve.go +++ b/internal/platform/fake/gitserve.go @@ -83,12 +83,12 @@ func (p *Platform) gitHeader(id string) func(context.Context) (string, error) { } // appTokens reports whether the flavor is GitHub-like: identities are Apps -// whose tokens carry permissions (every flavor but GitLab, Gitea, Forgejo -// and Bitbucket, whose identities are accounts with tokens of their own). -// Called with mu held. +// whose tokens carry permissions (every flavor but GitLab, Gitea, Forgejo, +// Bitbucket and Azure DevOps, whose identities are accounts with tokens of +// their own). Called with mu held. func (p *Platform) appTokens() bool { switch Flavor(p.caps.Flavor) { - case GitLab, Gitea, Forgejo, Bitbucket: + case GitLab, Gitea, Forgejo, Bitbucket, AzureDevOps: return false } return true diff --git a/internal/platform/fake/meter.go b/internal/platform/fake/meter.go index b45661c..ff9551b 100644 --- a/internal/platform/fake/meter.go +++ b/internal/platform/fake/meter.go @@ -162,11 +162,15 @@ func (t *target) missingLabels(names []string) int { // before the pull request is known to exist (first) and after (then): the // pull request; on GitHub then each missing label and the call that puts // the labels on it; on Gitea and Forgejo first each missing label (the -// pull request takes their ids); on GitLab nothing more (labels by name). +// pull request takes their ids); on GitLab nothing more (labels by name); +// on Azure DevOps then the property that holds the marker (labels by name +// in the request). func (t *target) createWrites(np platform.NewPR) (first, then int) { switch Flavor(t.p.caps.Flavor) { case GitLab: return 1, 0 + case AzureDevOps: + return 1, 1 case Gitea, Forgejo: return 1 + t.missingLabels(np.Labels), 0 } @@ -179,11 +183,19 @@ func (t *target) createWrites(np platform.NewPR) (first, then int) { // editWrites is how many writes the flavor's driver sends for e, before // the edit is known to apply (first) and after (then): the edit, on Gitea // and Forgejo after a state or base change in a request of its own, on -// Bitbucket followed by the decline of a close; then labels added in a call -// of their own but on GitLab, after each missing one is created. +// Bitbucket followed by the decline of a close, on Azure DevOps after the +// property of a new body's marker and one call per label added; then +// labels added in a call of their own but on GitLab and Azure DevOps, after +// each missing one is created. func (t *target) editWrites(e platform.PREdit) (first, then int) { first = 1 flavor := Flavor(t.p.caps.Flavor) + if flavor == AzureDevOps { + if e.Body != nil { + first++ + } + return first + len(dedupe(e.AddLabels)), 0 + } if (flavor == Gitea || flavor == Forgejo) && (e.State != nil || e.Base != nil) && (e.Title != nil || e.Body != nil) { first++ } diff --git a/internal/platform/fake/writer.go b/internal/platform/fake/writer.go index adb7420..63aefaa 100644 --- a/internal/platform/fake/writer.go +++ b/internal/platform/fake/writer.go @@ -11,6 +11,7 @@ import ( "strings" "time" + "github.com/bedrock-python/touchmark/internal/marker" "github.com/bedrock-python/touchmark/internal/platform" "github.com/bedrock-python/touchmark/internal/throttle" ) @@ -200,7 +201,7 @@ func (t *target) createPR(ctx context.Context, np platform.NewPR) (platform.PR, case strings.TrimSpace(np.Title) == "": return platform.PR{}, invalid(op, "the title is empty") } - if err := p.checkText(op, np.Body); err != nil { + if err := p.checkBody(op, np.Body); err != nil { return platform.PR{}, err } if err := t.p.checkLabels(op, np.Labels); err != nil { @@ -242,7 +243,7 @@ func (t *target) createPR(ctx context.Context, np platform.NewPR) (platform.PR, HeadRepoID: s.repo.ID, BaseExists: true, Title: p.draftTitle(np.Title, np.Draft), - Body: np.Body, + Body: p.stored(np.Body), Labels: labels, Author: t.as, CreatedAt: p.now(), @@ -256,6 +257,18 @@ func (t *target) createPR(ctx context.Context, np platform.NewPR) (platform.PR, p.runQuickActions(s, ps, np.Body, t.as, "the description") } } + if desc, line := marker.Detach(np.Body); p.apart() && line != "" { + if f, ok := p.takeMarkerFault(); ok { + // The property failed: the pull request has no marker, and + // the writer abandons it unless that fails too. + ps.pr.Body = desc + if !f.leaveOpen { + closer := t.as + ps.pr.State, ps.pr.ClosedBy, ps.pr.ClosedAt = platform.Closed, &closer, p.now() + } + return platform.PR{}, f.err + } + } return p.prView(ps.pr), nil }) } @@ -300,10 +313,31 @@ func (t *target) editPR(ctx context.Context, number int64, e platform.PREdit) (p if e.Title != nil && strings.TrimSpace(*e.Title) == "" { return platform.PR{}, invalid(op, "the title is empty") } - if e.Body != nil { - if err := p.checkText(op, *e.Body); err != nil { + // Where the marker lives apart, the description changes only when it + // differs from the current one (sameDescription), and a close + // leaves one that would not fit as it is, as Azure DevOps' driver + // does. + closing := e.State != nil && *e.State == platform.Closed + var desc, line, curDesc, stored string + keepDesc := false + if p.apart() && e.Body != nil { + curDesc, stored = marker.Detach(pr.Body) + desc, line = marker.Detach(*e.Body) + keepDesc = sameDescription(desc, curDesc) + } + switch { + case e.Body == nil, keepDesc: + case !p.apart(): + if err := p.checkBody(op, *e.Body); err != nil { return platform.PR{}, err } + default: + if err := p.checkText(op, desc); err != nil { + if !closing { + return platform.PR{}, err + } + keepDesc = true + } } if e.State != nil { switch { @@ -317,13 +351,21 @@ func (t *target) editPR(ctx context.Context, number int64, e platform.PREdit) (p return platform.PR{}, invalid(op, "base %q is empty or the head", *e.Base) } if p.caps.ClosedImmutable && pr.State == platform.Closed { - changes := e.Title != nil && p.draftTitle(*e.Title, pr.Draft) != pr.Title || - e.Body != nil && *e.Body != pr.Body || e.Base != nil && *e.Base != pr.Base + newTitle := e.Title != nil && p.draftTitle(*e.Title, pr.Draft) != pr.Title + newBase := e.Base != nil && *e.Base != pr.Base + changes := newTitle || e.Body != nil && p.stored(*e.Body) != pr.Body || newBase + if p.apart() && e.Body != nil { + changes = newTitle || newBase || line != stored || !keepDesc && !closing + } switch { case e.State != nil && *e.State == platform.Open: return platform.PR{}, unsupported(op, "#%d: a declined pull request cannot be reopened", number) case changes: return platform.PR{}, unsupported(op, "#%d is declined: only open pull requests can be changed", number) + case p.apart(): + // Closed again: a close that left the description as it + // was changes nothing. + return p.prView(pr), nil } } if err := t.p.checkLabels(op, e.AddLabels); err != nil { @@ -338,11 +380,27 @@ func (t *target) editPR(ctx context.Context, number int64, e platform.PREdit) (p return platform.PR{}, invalid(op, "#%d cannot be reopened: its head branch %q is gone", number, pr.Head) } } + var markerErr error + if p.apart() && e.Body != nil && line != stored && pr.State == platform.Open { + if f, ok := p.takeMarkerFault(); ok { + if closing { + // The closing marker goes first: nothing changed. + return platform.PR{}, f.err + } + markerErr, line = f.err, stored + } + } if e.Title != nil { pr.Title = p.draftTitle(*e.Title, pr.Draft) } - if e.Body != nil { - pr.Body = *e.Body + switch { + case e.Body == nil: + case p.apart() && keepDesc: + pr.Body = marker.Attach(curDesc, line) + case p.apart(): + pr.Body = marker.Attach(desc, line) + default: + pr.Body = p.stored(*e.Body) } if e.Base != nil { pr.Base, pr.BaseExists = *e.Base, true @@ -374,10 +432,24 @@ func (t *target) editPR(ctx context.Context, number int64, e platform.PREdit) (p p.runQuickActions(s, ps, *e.Body, t.as, "the description") } } + if markerErr != nil { + return platform.PR{}, markerErr + } return p.prView(ps.pr), nil }) } +// sameDescription reports whether desc, a description the writer would +// write where the marker lives apart, is the current description cur, +// marker lines, line endings and trailing whitespace aside, as Azure +// DevOps' driver compares them. +func sameDescription(desc, cur string) bool { + norm := func(s string) string { + return strings.TrimRight(strings.ReplaceAll(marker.Strip(s), "\r\n", "\n"), " \t\r\n") + } + return norm(desc) == norm(cur) +} + // Comment adds a comment by the writer. The body must not be blank, must // fit Caps.MaxBody and, with Caps.QuickActions, have no line that starts // with "/"; in git mode GitLab runs such lines as quick actions instead. @@ -395,8 +467,12 @@ func (t *target) Comment(ctx context.Context, number int64, body string) error { if strings.TrimSpace(body) == "" { return struct{}{}, invalid(op, "the comment is empty") } - if err := p.checkText(op, body); err != nil { - return struct{}{}, err + // Where descriptions are short (the marker apart), comments are not: + // Azure DevOps bounds its comments far above its descriptions. + if !p.apart() { + if err := p.checkText(op, body); err != nil { + return struct{}{}, err + } } ps.comments = append(ps.comments, Comment{Author: t.as, Body: body, CreatedAt: p.now()}) if p.git != nil && p.caps.QuickActions { @@ -462,6 +538,18 @@ func hasPrefixFold(s, prefix string) bool { return len(s) >= len(prefix) && strings.EqualFold(s[:len(prefix)], prefix) } +// checkBody checks a pull request's body as checkText does, but where the +// marker lives in a property (Caps.MarkerInProperties): there the +// description alone, without its marker line (marker.Detach), is held to +// Caps.MaxBody. +func (p *Platform) checkBody(op, body string) error { + if !p.apart() { + return p.checkText(op, body) + } + desc, _ := marker.Detach(body) + return p.checkText(op, desc) +} + // checkText checks a body or comment against the flavor: at most // Caps.MaxBody bytes and, with Caps.QuickActions, no line whose first // non-blank character is "/" (in memory mode: git mode runs them). diff --git a/internal/platform/fake/writer_test.go b/internal/platform/fake/writer_test.go index 3ae60f3..58187a4 100644 --- a/internal/platform/fake/writer_test.go +++ b/internal/platform/fake/writer_test.go @@ -7,6 +7,7 @@ import ( "testing" "time" + "github.com/bedrock-python/touchmark/internal/marker" "github.com/bedrock-python/touchmark/internal/platform" "github.com/bedrock-python/touchmark/internal/platform/fake" ) @@ -376,3 +377,69 @@ func TestBitbucketFlavor(t *testing.T) { e.p.SetPRState(r.ID, pr.Number, platform.Open, nil, time.Time{}) wantSetupErr(t, e.p, "cannot be reopened") } + +// TestAzureDevOpsFlavor: the Azure DevOps flavor reports the capabilities +// of its driver, keeps the marker apart from the description (the +// description alone fits Caps.MaxBody, a marker pasted into it never +// counts, a person editing it keeps the stored marker), labels by name, +// and treats an abandoned pull request as final, as on Bitbucket. +func TestAzureDevOpsFlavor(t *testing.T) { + t.Parallel() + ctx := t.Context() + c := fake.CapsFor(fake.AzureDevOps) + if c.Flavor != "azure-devops" || c.NoLabels || !c.ClosedImmutable || c.Marker != platform.MarkerInProperties || !c.BodyControls() || + c.Draft != platform.DraftNative || !c.CloserKnown || c.WorkflowPerm || c.LabelsByID || c.QuickActions || c.MaxBody != 4000 { + t.Errorf("CapsFor(AzureDevOps) = %+v", c) + } + e := newEnv(t, fake.WithFlavor(fake.AzureDevOps)) + r := e.repo("Billing/api", "README.md", "x") + tw := e.target(r) + line := testMarkerLine(t) + long := strings.Repeat("x", 3990) + pr, err := tw.CreatePR(ctx, platform.NewPR{Head: "touchmark/hub", Base: "main", Title: "sync", + Body: long + "\n\n" + line, Labels: []string{"engineering-assets"}, Draft: true}) + if err != nil || pr.URL != "https://github.com/Billing/_git/api/pullrequest/1" || !pr.Draft || pr.Body != long+"\n\n"+line || + len(pr.Labels) != 1 { + t.Fatalf("CreatePR = %+v, %v", pr, err) + } + _, err = tw.CreatePR(ctx, platform.NewPR{Head: "other", Base: "main", Title: "sync", Body: long + "0123456789x"}) + wantClass(t, "a description over 4 000 bytes", err, platform.ClassInvalid) + + // A marker pasted into the description does not count: the stored one + // does, and a person's edit keeps it. + e.p.UpdatePR(r.ID, pr.Number, func(p *platform.PR) { + p.Body = "We rewrote it.\n\n" + strings.Replace(line, "stream=sync", "stream=x", 1) + }) + e.ok() + if got := e.p.PR(r.ID, pr.Number); got.Body != "We rewrote it.\n\n"+line { + t.Errorf("after a person's edit: %q", got.Body) + } + + closed := platform.Closed + body := "closed by touchmark\n\n" + line + got, err := tw.EditPR(ctx, pr.Number, platform.PREdit{Body: &body, State: &closed}) + if err != nil || got.State != platform.Closed || got.Body != body || got.ClosedBy == nil || got.ClosedBy.ID != e.writer.ID { + t.Fatalf("close = %+v, %v", got, err) + } + open := platform.Open + _, err = tw.EditPR(ctx, pr.Number, platform.PREdit{State: &open}) + wantClass(t, "reactivate", err, platform.ClassUnsupported) + _, err = tw.EditPR(ctx, pr.Number, platform.PREdit{Body: ptr("again")}) + wantClass(t, "edit an abandoned one", err, platform.ClassUnsupported) + if _, err := tw.EditPR(ctx, pr.Number, platform.PREdit{State: &closed, Body: &body}); err != nil { + t.Errorf("closing again with the same body: %v", err) + } +} + +// testMarkerLine is a real marker line of a test hub. +func testMarkerLine(t *testing.T) string { + t.Helper() + line, err := marker.Encode(marker.Marker{Key: "sha256:6b1f0c3a9e2d4b586b1f0c3a9e2d4b586b1f0c3a9e2d4b586b1f0c3a9e2d4b58", Data: marker.Data{ + V: marker.Version, Stream: "sync", Hub: "acme-eng", FP: "github.com/712345678", + DecidedAt: "3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3", Engine: "0.3.0", + }}) + if err != nil { + t.Fatal(err) + } + return line +} diff --git a/internal/platform/platform.go b/internal/platform/platform.go index 6eb0074..2745fba 100644 --- a/internal/platform/platform.go +++ b/internal/platform/platform.go @@ -1,5 +1,5 @@ // Package platform defines what touchmark needs from a code hosting platform -// (GitHub, GitLab, Gitea, Forgejo, Bitbucket Cloud; later Azure DevOps): the +// (GitHub, GitLab, Gitea, Forgejo, Bitbucket Cloud, Azure DevOps): the // Reader and Writer interfaces drivers implement, their capabilities, and one // error model. // @@ -132,9 +132,13 @@ type PR struct { // BaseExists is false when the target branch was deleted. BaseExists bool Title string - Body string // raw body, marker included - Labels []string - Author Account + // Body is the raw body, marker included: its last line on every + // platform. Where the platform keeps the marker outside the description + // (Caps.Marker MarkerInProperties) the driver appends it, as + // marker.Attach does, so the core reads the same Body everywhere. + Body string + Labels []string + Author Account // ClosedBy is who closed or merged the PR; nil if open or not reported // (see Caps.CloserKnown). ClosedBy *Account @@ -142,7 +146,9 @@ type PR struct { ClosedAt time.Time // zero while open } -// NewPR is a pull request to create. +// NewPR is a pull request to create. Its Body, and PREdit's, end with the +// marker line; a driver of a MarkerInProperties platform stores that line +// apart (marker.Detach) and writes the rest as the description. type NewPR struct { Head, Base, Title, Body string // Labels are created by the driver when missing. @@ -255,8 +261,21 @@ const ( type MarkerStore uint8 const ( - MarkerInBody MarkerStore = iota // last line of the description - MarkerInProperties // PR properties (Azure DevOps, later) + MarkerInBody MarkerStore = iota // last line of the description + // MarkerInProperties keeps the marker line out of the description, in + // a property of the pull request (Azure DevOps: the properties API, + // key touchmark.marker), for a platform whose descriptions are too + // short to hold one (4 000 characters there). The core still writes + // and reads one Body with the marker as its last line, in the comment + // frame: a driver of such a platform splits what the core writes with + // marker.Detach (the description, without marker lines; the line, to + // the property) and joins what it reads with marker.Attach (the + // description without marker lines, then the property's line), so the + // description a person edits can never forge or hide the marker. The + // description alone must fit MaxBody: prbody reserves no room for the + // marker there. Other platforms see no change: their Body is the + // description itself. + MarkerInProperties // MarkerInRefDef is the last line of the description too, with the // payload of MarkerInBody wrapped in a Markdown link reference // definition, `[touchmark]: # "touchmark:v1 …"`, which renderers do not @@ -280,11 +299,14 @@ type Limits struct { // Caps describes a platform instance, found once per run by Reader.Probe. type Caps struct { - // Flavor is "github", "ghe.com", "ghes", "gitlab", "gitea", "forgejo" - // or "bitbucket" (Bitbucket Cloud). + // Flavor is "github", "ghe.com", "ghes", "gitlab", "gitea", "forgejo", + // "bitbucket" (Bitbucket Cloud) or "azure-devops" (Azure DevOps + // Services). Flavor string Version string - // MaxBody is the body budget in bytes, marker included. + // MaxBody is the body budget in bytes, marker included, but where the + // marker lives apart (MarkerInProperties): there it bounds the + // description alone. MaxBody int Draft DraftStyle DraftPrefix string diff --git a/internal/prbody/helpers_test.go b/internal/prbody/helpers_test.go index e8796f3..fe278e9 100644 --- a/internal/prbody/helpers_test.go +++ b/internal/prbody/helpers_test.go @@ -27,7 +27,7 @@ const ( ) // flavors are the platforms of the golden bodies. -var flavors = []string{"github", "gitlab", "gitea", "bitbucket"} +var flavors = []string{"github", "gitlab", "gitea", "bitbucket", "azure-devops"} // capsOf returns the capabilities that matter to bodies, as the drivers // report them (and fake.CapsFor does). @@ -42,6 +42,10 @@ func capsOf(flavor string) platform.Caps { c.Marker = platform.MarkerInRefDef c.NoLabels = true c.ClosedImmutable = true + case "azure-devops": + c.MaxBody = 4000 + c.Marker = platform.MarkerInProperties + c.ClosedImmutable = true } return c } diff --git a/internal/prbody/prbody.go b/internal/prbody/prbody.go index 628aadc..aa9e228 100644 --- a/internal/prbody/prbody.go +++ b/internal/prbody/prbody.go @@ -168,6 +168,12 @@ type Input struct { // records the hash of the human part (marker.Data.Body), that is how a // body is built: human, then its hash into the marker, then human + // "\n\n" + marker, which fits the budget whatever the marker's length. + // + // Where the platform keeps the marker apart (Caps.Marker + // platform.MarkerInProperties, Azure DevOps) the body still ends with + // it, for the driver to store it apart (marker.Detach), but + // Caps.MaxBody bounds the human part alone: no room is reserved for the + // marker, and the marker line does not count. Marker string } @@ -368,15 +374,16 @@ func (r *renderer) fit() (string, error) { } } limit := r.in.Caps.MaxBody - if limit > 0 && r.in.Marker == "" { + apart := r.in.Caps.Marker == platform.MarkerInProperties + if limit > 0 && r.in.Marker == "" && !apart { if limit <= markerRoom { return "", fmt.Errorf("%w: %d bytes leave no room for a marker of up to %d", ErrTooLarge, limit, marker.MaxLine) } limit -= markerRoom } - for k := 0; limit > 0 && r.size(n) > limit; { + for k := 0; limit > 0 && r.budgetSize(n) > limit; { if k == numLists { - return "", fmt.Errorf("%w: %d bytes with every list cut, more than %d", ErrTooLarge, r.size(n), limit) + return "", fmt.Errorf("%w: %d bytes with every list cut, more than %d", ErrTooLarge, r.budgetSize(n), limit) } if n[k] == 0 { k++ @@ -431,6 +438,20 @@ func (r *renderer) pieces(n [numLists]int) [13]piece { } } +// budgetSize is what Caps.MaxBody bounds of r.body(n): all of it, but +// where the platform keeps the marker apart (platform.MarkerInProperties), +// whose marker line and the blank line before it do not count. +func (r *renderer) budgetSize(n [numLists]int) int { + size := r.size(n) + if r.in.Caps.Marker != platform.MarkerInProperties || r.in.Marker == "" { + return size + } + if size == len(r.in.Marker) { + return 0 + } + return size - len(r.in.Marker) - len("\n\n") +} + // size is len(r.body(n)), without building the body. func (r *renderer) size(n [numLists]int) int { total, parts := 0, 0 diff --git a/internal/prbody/render_test.go b/internal/prbody/render_test.go index 97534fb..c441c4e 100644 --- a/internal/prbody/render_test.go +++ b/internal/prbody/render_test.go @@ -9,6 +9,7 @@ import ( "testing" "github.com/bedrock-python/touchmark/internal/marker" + "github.com/bedrock-python/touchmark/internal/platform" ) // intro is the pr.intro_file of the tests. @@ -159,7 +160,17 @@ func checkBody(t *testing.T, in Input, body string) { } } } - if in.Caps.MaxBody > 0 && len(full) > in.Caps.MaxBody { + if in.Caps.Marker == platform.MarkerInProperties { + // The marker lives apart: the description alone is bounded, and it + // is what the driver stores (marker.Detach). + desc, line := marker.Detach(body) + if in.Caps.MaxBody > 0 && len(desc) > in.Caps.MaxBody { + t.Errorf("description of %d bytes, more than %d", len(desc), in.Caps.MaxBody) + } + if line != in.Marker || marker.Attach(desc, line) != body { + t.Error("the body does not split into its description and its marker line") + } + } else if in.Caps.MaxBody > 0 && len(full) > in.Caps.MaxBody { t.Errorf("body of %d bytes with its marker, more than %d", len(full), in.Caps.MaxBody) } if in.Marker != "" { diff --git a/internal/prbody/testdata/assumed/azure-devops.md b/internal/prbody/testdata/assumed/azure-devops.md new file mode 100644 index 0000000..6c504fc --- /dev/null +++ b/internal/prbody/testdata/assumed/azure-devops.md @@ -0,0 +1,19 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +The hub subscribed this repository, which has no `.engineering-assets.yml`. To add packs or keep files out of the sync, add `.engineering-assets.yml` with `packs` or `ignore`; to stop these pull requests, add it with `enabled: false`. + +### Changes + +| Change | File | Pack | +|---|---|---| +| create | `AGENTS.md` | `agents` | +| create | `prompts/review.md` | `agents` | + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/declined/azure-devops.md b/internal/prbody/testdata/declined/azure-devops.md new file mode 100644 index 0000000..8c18b39 --- /dev/null +++ b/internal/prbody/testdata/declined/azure-devops.md @@ -0,0 +1,21 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### Changes + +| Change | File | Pack | +|---|---|---| +| update | `AGENTS.md` | `agents` | +| create | `prompts/review.md` | `agents` | + +### Previously declined + +Some of these changes were in #7 and #9, which were closed without merging. This pull request brings the whole current set anyway, because the files of a pack go together. To keep a file as it is here, add it to `ignore` in `.engineering-assets.yml`. + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/huge/azure-devops.md b/internal/prbody/testdata/huge/azure-devops.md new file mode 100644 index 0000000..49454da --- /dev/null +++ b/internal/prbody/testdata/huge/azure-devops.md @@ -0,0 +1,87 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### ⚠ Sensitive paths + +These files can run code in CI or change how tools and agents work in this repository. Review them first. + +- `.github/workflows/docs.yml`: update, pack `ci` +- `.pre-commit-config.yaml`: create, pack `base` +- `bin/setup`: create, executable, pack `base` + +### Changes + +1003 changes, too many to list here. + +
+150 files here differ from the hub and stay as they are + +touchmark does not update files changed in this repository. To take the hub's version of one, run `touchmark apply --adopt `. To stop seeing it here, add it to `ignore` in `.engineering-assets.yml`. + +- `docs/handbook/chapter-00/local-000.md` +- `docs/handbook/chapter-00/local-012.md` +- `docs/handbook/chapter-00/local-024.md` +- `docs/handbook/chapter-00/local-036.md` +- `docs/handbook/chapter-00/local-048.md` +- `docs/handbook/chapter-00/local-060.md` +- `docs/handbook/chapter-00/local-072.md` +- `docs/handbook/chapter-00/local-084.md` +- `docs/handbook/chapter-00/local-096.md` +- `docs/handbook/chapter-00/local-108.md` +- `docs/handbook/chapter-00/local-120.md` +- `docs/handbook/chapter-00/local-132.md` +- `docs/handbook/chapter-00/local-144.md` +- `docs/handbook/chapter-01/local-001.md` +- `docs/handbook/chapter-01/local-013.md` +- `docs/handbook/chapter-01/local-025.md` +- `docs/handbook/chapter-01/local-037.md` +- `docs/handbook/chapter-01/local-049.md` +- `docs/handbook/chapter-01/local-061.md` +- `docs/handbook/chapter-01/local-073.md` +- `docs/handbook/chapter-01/local-085.md` +- `docs/handbook/chapter-01/local-097.md` +- `docs/handbook/chapter-01/local-109.md` +- `docs/handbook/chapter-01/local-121.md` +- `docs/handbook/chapter-01/local-133.md` +- `docs/handbook/chapter-01/local-145.md` +- `docs/handbook/chapter-02/local-002.md` +- `docs/handbook/chapter-02/local-014.md` +- `docs/handbook/chapter-02/local-026.md` +- `docs/handbook/chapter-02/local-038.md` +- `docs/handbook/chapter-02/local-050.md` +- `docs/handbook/chapter-02/local-062.md` +- `docs/handbook/chapter-02/local-074.md` +- `docs/handbook/chapter-02/local-086.md` +- `docs/handbook/chapter-02/local-098.md` +- `docs/handbook/chapter-02/local-110.md` +- `docs/handbook/chapter-02/local-122.md` +- `docs/handbook/chapter-02/local-134.md` +- `docs/handbook/chapter-02/local-146.md` +- `docs/handbook/chapter-03/local-003.md` +- `docs/handbook/chapter-03/local-015.md` +- `docs/handbook/chapter-03/local-027.md` +- `docs/handbook/chapter-03/local-039.md` +- `docs/handbook/chapter-03/local-051.md` +- `docs/handbook/chapter-03/local-063.md` +- `docs/handbook/chapter-03/local-075.md` +- `docs/handbook/chapter-03/local-087.md` +- `docs/handbook/chapter-03/local-099.md` +- `docs/handbook/chapter-03/local-111.md` +- `docs/handbook/chapter-03/local-123.md` +- `docs/handbook/chapter-03/local-135.md` +- `docs/handbook/chapter-03/local-147.md` +- `docs/handbook/chapter-04/local-004.md` +- `docs/handbook/chapter-04/local-016.md` +- `docs/handbook/chapter-04/local-028.md` + +…and 95 more. + +
+ +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/local/azure-devops.md b/internal/prbody/testdata/local/azure-devops.md new file mode 100644 index 0000000..12c6563 --- /dev/null +++ b/internal/prbody/testdata/local/azure-devops.md @@ -0,0 +1,27 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` at commit `3f2c1ab9d8e7`. + +### Changes + +| Change | File | Pack | +|---|---|---| +| update | `AGENTS.md` | `agents` | + +
+3 files here differ from the hub and stay as they are + +touchmark does not update files changed in this repository. To take the hub's version of one, run `touchmark apply --adopt `. To stop seeing it here, add it to `ignore` in `.engineering-assets.yml`. + +- `.github/CODEOWNERS` +- `CONTRIBUTING.md` +- `docs/guide.md` + +
+ +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/new/azure-devops.md b/internal/prbody/testdata/new/azure-devops.md new file mode 100644 index 0000000..0681dc4 --- /dev/null +++ b/internal/prbody/testdata/new/azure-devops.md @@ -0,0 +1,30 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### ⚠ Sensitive paths + +These files can run code in CI or change how tools and agents work in this repository. Review them first. + +- `.claude/settings.json`: update, pack `claude` +- `scripts/check.sh`: create, executable, pack `base` +- `tools/lint.sh`: chmod +x, pack `base` + +### Changes + +| Change | File | Pack | +|---|---|---| +| ⚠ update | `.claude/settings.json` | `claude` | +| ⚠ chmod +x | `tools/lint.sh` | `base` | +| ⚠ create, executable | `scripts/check.sh` | `base` | +| delete | `prompts/old.md` | `agents` | +| update | `AGENTS.md` | `agents` | +| chmod -x | `docs/guide.md` | `base` | +| create | `prompts/review.md` | `agents` | + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/nothing-more/azure-devops.md b/internal/prbody/testdata/nothing-more/azure-devops.md new file mode 100644 index 0000000..be92e0b --- /dev/null +++ b/internal/prbody/testdata/nothing-more/azure-devops.md @@ -0,0 +1,20 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### Changes + +| Change | File | Pack | +|---|---|---| +| create | `prompts/review.md` | `agents` | + +### Nothing more to sync + +The hub proposes nothing more for this repository: the default branch has the hub's versions of these files, or they are ignored or changed here. This branch has commits touchmark did not make, so touchmark leaves this pull request open. Close it, or keep your commits and merge it. + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/paused/azure-devops.md b/internal/prbody/testdata/paused/azure-devops.md new file mode 100644 index 0000000..7477351 --- /dev/null +++ b/internal/prbody/testdata/paused/azure-devops.md @@ -0,0 +1,34 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### ⚠ Sensitive paths + +These files can run code in CI or change how tools and agents work in this repository. Review them first. + +- `.github/workflows/lint.yml`: create, pack `ci` (after a rebuild) + +### Changes + +| Change | File | Pack | +|---|---|---| +| update | `AGENTS.md` | `agents` | +| create | `prompts/review.md` | `agents` | + +### touchmark paused + +This branch has commits touchmark did not make, so touchmark stopped updating it to keep them. The changes above are what the branch holds now. A rebuild from the default branch would bring: + +- ⚠ create `.github/workflows/lint.yml` (pack `ci`) +- update `AGENTS.md` (pack `agents`) + +To rebuild the branch and drop the other commits, tick **Rebuild this branch** below. + +- [ ] Rebuild this branch (drops commits added by others) + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/prbody/testdata/sensitive-only/azure-devops.md b/internal/prbody/testdata/sensitive-only/azure-devops.md new file mode 100644 index 0000000..0eeb437 --- /dev/null +++ b/internal/prbody/testdata/sensitive-only/azure-devops.md @@ -0,0 +1,32 @@ +Acme keeps its shared engineering files (agent instructions, editor and CI settings) in one hub. +Review this like any other change; questions go to the platform team's channel. + +touchmark syncs packs `agents`, `claude` and `base` from the hub `acme-eng` ([`github.com/acme/engineering-assets`](https://redirect.github.com/acme/engineering-assets)) at commit [`3f2c1ab9d8e7`](https://redirect.github.com/acme/engineering-assets/commit/3f2c1ab9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3). + +### ⚠ Sensitive paths + +These files can run code in CI or change how tools and agents work in this repository. Review them first. + +- `.gitea/workflows/ci.yml`: create, pack `ci` +- `.github/CODEOWNERS`: delete, pack `base` +- `.github/workflows/ci.yml`: update, pack `ci` +- `deploy/prod.yaml`: update, pack `ops` +- `scripts/release.sh`: chmod +x, pack `base` + +This pull request adds `.gitea/workflows` or `.forgejo/workflows`, which turns off `.github/workflows` in this repository on Gitea and Forgejo. + +### Changes + +| Change | File | Pack | +|---|---|---| +| ⚠ delete | `.github/CODEOWNERS` | `base` | +| ⚠ update | `.github/workflows/ci.yml` | `ci` | +| ⚠ update | `deploy/prod.yaml` | `ops` | +| ⚠ chmod +x | `scripts/release.sh` | `base` | +| ⚠ create | `.gitea/workflows/ci.yml` | `ci` | + +--- + +If this pull request is closed without merging, touchmark remembers it and does not propose the same changes again; a new one comes when the hub changes these files. Editing `packs` or `ignore` in `.engineering-assets.yml` resets that memory, and `ignore` opts a file out for good. To have the same changes proposed again sooner, the hub's maintainers can add a `forget_declines` entry with the number of this pull request to `.touchmark/operations.yml`. touchmark rewrites this description, so please comment instead of editing it. + + \ No newline at end of file diff --git a/internal/report/doctor_test.go b/internal/report/doctor_test.go index 13a84dc..3810274 100644 --- a/internal/report/doctor_test.go +++ b/internal/report/doctor_test.go @@ -99,7 +99,7 @@ func TestDoctorSchema(t *testing.T) { bad := map[string]func(*Doctor){ "status": func(d *Doctor) { d.HubChecks[0].Status = "red" }, "check": func(d *Doctor) { d.HubChecks[0].Name = "Write Isolation" }, - "provider": func(d *Doctor) { d.Providers[0].Type = "azure-devops" }, + "provider": func(d *Doctor) { d.Providers[0].Type = "sourcehut" }, "summary": func(d *Doctor) { delete(d.Summary, StatusUnknown) }, "schema": func(d *Doctor) { d.Schema = "doctor/v2" }, } diff --git a/schemas/doctor.schema.json b/schemas/doctor.schema.json index d375c15..710fa43 100644 --- a/schemas/doctor.schema.json +++ b/schemas/doctor.schema.json @@ -113,7 +113,7 @@ "required": ["id", "type", "host", "resolve_complete", "checks"], "properties": { "id": { "$ref": "#/$defs/providerId" }, - "type": { "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket"] }, + "type": { "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket", "azure-devops"] }, "host": { "description": "The provider's host, with a port when it has one.", "type": "string", diff --git a/schemas/hub.schema.json b/schemas/hub.schema.json index 54d2779..782ec4c 100644 --- a/schemas/hub.schema.json +++ b/schemas/hub.schema.json @@ -216,6 +216,16 @@ "properties": { "platform": { "const": "bitbucket" } } }, "then": { "properties": { "base_url": { "pattern": "^https://[Bb][Ii][Tt][Bb][Uu][Cc][Kk][Ee][Tt]\\.[Oo][Rr][Gg]/*$" } } } + }, + { + "if": { + "required": ["platform"], + "properties": { "platform": { "const": "azure-devops" } } + }, + "then": { + "required": ["base_url"], + "properties": { "base_url": { "pattern": "^https://[Dd][Ee][Vv]\\.[Aa][Zz][Uu][Rr][Ee]\\.[Cc][Oo][Mm]/[A-Za-z0-9]([A-Za-z0-9-]{0,48}[A-Za-z0-9])?/*$" } } + } } ], "$defs": { @@ -246,8 +256,8 @@ "pattern": "^[a-z][a-z0-9-]{0,31}$" }, "providerType": { - "description": "github: github.com, GHE.com or GitHub Enterprise Server; gitlab; gitea; forgejo; bitbucket: Bitbucket Cloud (plan, distribute and doctor; setup does not set up a hub on it yet).", - "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket"] + "description": "github: github.com, GHE.com or GitHub Enterprise Server; gitlab; gitea; forgejo; bitbucket: Bitbucket Cloud; azure-devops: one organization of Azure DevOps Services (bitbucket and azure-devops: plan, distribute and doctor; setup does not set up a hub on them yet).", + "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket", "azure-devops"] }, "sign": { "description": "auto: sign where it costs nothing and where target rules require it; always: sign every commit. Default: auto.", @@ -287,7 +297,7 @@ }, "type": { "$ref": "#/$defs/providerType" }, "url": { - "description": "Web URL of the instance. Default for github: https://github.com; for gitlab: https://gitlab.com; for bitbucket: https://bitbucket.org, the only url it takes without api_url. Required for gitea and forgejo. A target targets.yml names by web URL belongs to the provider whose url it lies under.", + "description": "Web URL of the instance. Default for github: https://github.com; for gitlab: https://gitlab.com; for bitbucket: https://bitbucket.org, the only url it takes without api_url. Required for gitea and forgejo, and for azure-devops: https://dev.azure.com/, one provider per organization. A target targets.yml names by web URL belongs to the provider whose url it lies under.", "$ref": "#/$defs/url" }, "api_url": { @@ -299,7 +309,7 @@ "$ref": "#/$defs/relativePath" }, "writer": { - "description": "The account that writes to targets, as the platform names it: on GitHub the App's bot, [bot]; on GitLab, Gitea and Forgejo the username of the service account or bot user; on Bitbucket the bot account's UUID in braces, {…}.", + "description": "The account that writes to targets, as the platform names it: on GitHub the App's bot, [bot]; on GitLab, Gitea and Forgejo the username of the service account or bot user; on Bitbucket the bot account's UUID in braces, {…}; on Azure DevOps the user's id, a GUID.", "$ref": "#/$defs/account" }, "known_authors": { @@ -335,10 +345,20 @@ "allOf": [ { "if": { - "properties": { "type": { "enum": ["gitea", "forgejo"] } } + "properties": { "type": { "enum": ["gitea", "forgejo", "azure-devops"] } } }, "then": { "required": ["url"] } }, + { + "if": { + "required": ["type", "url"], + "properties": { + "type": { "const": "azure-devops" }, + "url": { "not": { "pattern": "^https://[Dd][Ee][Vv]\\.[Aa][Zz][Uu][Rr][Ee]\\.[Cc][Oo][Mm]/[A-Za-z0-9]([A-Za-z0-9-]{0,48}[A-Za-z0-9])?/*$" } } + } + }, + "then": { "required": ["api_url"] } + }, { "if": { "required": ["type", "url"], diff --git a/schemas/operations.schema.json b/schemas/operations.schema.json index 2a712bd..ba88dd4 100644 --- a/schemas/operations.schema.json +++ b/schemas/operations.schema.json @@ -28,7 +28,7 @@ } }, "forget_declines": { - "description": "Propose again the content of a pull request the target closed without merging. touchmark marks that pull request revoked, so the entry acts once; on Bitbucket Cloud, where a declined pull request cannot be edited, the entry acts while it is present.", + "description": "Propose again the content of a pull request the target closed without merging. touchmark marks that pull request revoked, so the entry acts once; on Bitbucket Cloud and Azure DevOps, where touchmark never edits a declined pull request, the entry acts while it is present.", "type": ["array", "null"], "items": { "type": "object", diff --git a/schemas/report.schema.json b/schemas/report.schema.json index d48721f..7b11508 100644 --- a/schemas/report.schema.json +++ b/schemas/report.schema.json @@ -142,7 +142,7 @@ "required": ["id", "type", "host", "write_check", "resolve_complete"], "properties": { "id": { "$ref": "#/$defs/providerId" }, - "type": { "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket"] }, + "type": { "enum": ["github", "gitlab", "gitea", "forgejo", "bitbucket", "azure-devops"] }, "host": { "description": "The provider's host, with a port when it has one.", "type": "string", diff --git a/schemas/targets.schema.json b/schemas/targets.schema.json index bac2d24..4088d8a 100644 --- a/schemas/targets.schema.json +++ b/schemas/targets.schema.json @@ -89,7 +89,7 @@ "required": ["org"], "properties": { "org": { - "description": "Every repository in a GitHub organisation, a GitLab, Gitea or Forgejo namespace or a Bitbucket workspace (a synonym of group), named by its path or its web URL.", + "description": "Every repository in a GitHub organisation, a GitLab, Gitea or Forgejo namespace, a Bitbucket workspace or an Azure DevOps organization (the provider's url; a project is selected with match) (a synonym of group), named by its path or its web URL.", "$ref": "#/$defs/namespace" }, "provider": { "$ref": "#/$defs/providerId" }, @@ -134,7 +134,7 @@ } }, "topics": { - "description": "Only repositories with all of these topics. Bitbucket has no topics: check refuses them on its entries; use match.", + "description": "Only repositories with all of these topics. Bitbucket and Azure DevOps have no topics: check refuses them on their entries; use match.", "type": ["array", "null"], "items": { "type": "string", "pattern": "[^\t\n\u000b\f\r \u00a0\u1680\u2000-\u200a\u2028\u2029\u202f\u205f\u3000\ufeff]" } },