Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions .github/actions/publish-to-s3/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Publish assets to S3
description: >-
Authenticate to AWS with OIDC, upload built assets to S3, and invalidate the
matching CloudFront paths so the change is visible. No long-lived credentials
are involved.

The calling job must declare `permissions: id-token: write`, which a composite
action cannot do on its own.

inputs:
role-to-assume:
description: >-
ARN of the role to assume via OIDC. The role's trust policy must name the
calling repository's subject claim -- it is repo-specific, so a role that
works for one repository will not work for another without an entry.
required: true
aws-region:
description: AWS region.
required: false
default: us-west-2
bucket:
description: S3 bucket name.
required: true
prefix:
description: >-
Key prefix within the bucket. Leave empty to publish to the bucket root,
which is the shape a project gets when it has a bucket to itself. With no
prefix the invalidation covers the whole distribution.
required: false
default: ""
sources:
description: >-
Directories to upload, one per line. A directory named `dist` uploads to
the prefix root; anything else uploads to a subdirectory of that name.
required: false
default: |
dist
css
distribution-id:
description: CloudFront distribution to invalidate. Omit to skip invalidation.
required: false
default: ""
acl:
description: >-
Object ACL, or "none" to set none. AWS recommends ACLs disabled and
access granted by bucket policy, which is the default here and what new
buckets get. A bucket still in legacy ObjectWriter mode with no policy
needs "public-read", or its objects upload fine and are unreadable --
set public-base-url as well and the action will catch that.
required: false
default: none
public-base-url:
description: >-
Public URL the prefix is served from, with a trailing slash. When set,
the job summary links every published file, and after a real publish one
of them is fetched to confirm it is readable. S3 website endpoints have
no directory listing, so naming the files is the only way to link them.
required: false
default: ""
dry-run:
description: Show what would be uploaded and invalidated, and change nothing.
required: false
default: "false"

runs:
using: composite
steps:
- uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ inputs.role-to-assume }}
aws-region: ${{ inputs.aws-region }}

- name: Publish
shell: bash
env:
BUCKET: ${{ inputs.bucket }}
PREFIX: ${{ inputs.prefix }}
SOURCES: ${{ inputs.sources }}
DISTRIBUTION: ${{ inputs.distribution-id }}
ACL: ${{ inputs.acl }}
DRY_RUN: ${{ inputs.dry-run }}
BASE_URL: ${{ inputs.public-base-url }}
run: |
ARGS=(--bucket "$BUCKET" --prefix "$PREFIX" --acl "$ACL")
ARGS+=(--manifest "$RUNNER_TEMP/published.txt")
[ -n "$BASE_URL" ] && ARGS+=(--verify-url "$BASE_URL")
[ -n "$DISTRIBUTION" ] && ARGS+=(--distribution "$DISTRIBUTION")
[ "$DRY_RUN" = "true" ] && ARGS+=(--dry-run)
while IFS= read -r line; do
[ -n "$line" ] && ARGS+=(--source "$line")
done <<< "$SOURCES"
"${{ github.action_path }}/../../../bin/publish-to-s3.sh" "${ARGS[@]}"

- name: Summarize
shell: bash
env:
BUCKET: ${{ inputs.bucket }}
PREFIX: ${{ inputs.prefix }}
BASE_URL: ${{ inputs.public-base-url }}
DRY_RUN: ${{ inputs.dry-run }}
run: |
MANIFEST="$RUNNER_TEMP/published.txt"
COUNT="$(grep -c . "$MANIFEST" 2>/dev/null || echo 0)"
{
echo "### S3 publish${DRY_RUN:+ (dry run)}"
echo
echo "| | |"
echo "|---|---|"
echo "| bucket | \`$BUCKET\` |"
echo "| prefix | \`$PREFIX\` |"
echo "| files | $COUNT |"
echo "| dry run | $DRY_RUN |"
echo
if [ -n "$BASE_URL" ] && [ "$COUNT" -gt 0 ]; then
echo "#### Published files"
echo
sort "$MANIFEST" | head -n 40 | while IFS= read -r key; do
[ -n "$key" ] && echo "- [\`$key\`](${BASE_URL%/}/$key)"
done
if [ "$COUNT" -gt 40 ]; then
echo "- …and $((COUNT - 40)) more"
fi
fi
} >> "$GITHUB_STEP_SUMMARY"
43 changes: 43 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,49 @@ jobs:
|| { echo "::error::unhelpful error: $(cat /tmp/err)"; exit 1; }
echo "ok"

publish-to-s3-guards:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7

# No AWS credentials here on purpose: this repository should not hold
# any. What is testable without them is the argument handling and the
# guards, which are where the damage would be.
- name: Required arguments are enforced
run: |
fail() { echo "::error::$1"; exit 1; }
bin/publish-to-s3.sh 2>/dev/null && fail "should require --bucket"
bin/publish-to-s3.sh --bucket b 2>/dev/null && fail "should require --prefix"
bin/publish-to-s3.sh --bucket b --prefix p 2>/dev/null && fail "should require --source"
echo "ok"

- name: An empty prefix publishes to the bucket root
run: |
fail() { echo "::error::$1"; exit 1; }
mkdir -p /tmp/pfx/dist && touch /tmp/pfx/dist/a.js
printf '#!/bin/sh\nexit 0\n' > /tmp/aws && chmod +x /tmp/aws
out="$(PATH=/tmp:$PATH bin/publish-to-s3.sh --bucket b --source /tmp/pfx/dist \
--distribution E1 --dry-run 2>&1)"
echo "$out" | grep -q 's3://b/$' || fail "no prefix should target the bucket root, got: $out"
echo "$out" | grep -q 'invalidate /\* ' || fail "no prefix should invalidate /*"
echo "$out" | grep -q 'no prefix, so this invalidates the whole distribution' \
|| fail "should say so when invalidating everything"
# and a prefix with stray slashes is normalized
out="$(PATH=/tmp:$PATH bin/publish-to-s3.sh --bucket b --prefix /wrapped/ \
--source /tmp/pfx/dist --distribution E1 --dry-run 2>&1)"
echo "$out" | grep -q 's3://b/wrapped/$' || fail "stray slashes not normalized: $out"
echo ok

- name: An empty or missing source is refused
run: |
fail() { echo "::error::$1"; exit 1; }
mkdir -p /tmp/empty-dist
bin/publish-to-s3.sh --bucket b --prefix p --source /tmp/empty-dist 2>/dev/null \
&& fail "publishing an empty directory should be an error"
bin/publish-to-s3.sh --bucket b --prefix p --source /tmp/nope 2>/dev/null \
&& fail "a missing directory should be an error"
echo "ok"

# deploy-site is the seam every generator meets, so its guards matter more
# than any single builder's.
deploy-guards:
Expand Down
75 changes: 75 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Consuming repositories reference a **moving major tag** such as `@v1`, which
advances with every backward-compatible change. This file is how you find out
what moved. See
[ADR-0006](docs/decisions/0006-version-with-moving-major-tags.md).

## [Unreleased]

### Added

- `CHANGELOG.md`, following
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- [ADR-0007](docs/decisions/0007-publish-without-destroying-what-you-did-not-create.md),
recording why publishing touches only what it published: copy rather than
sync with delete, no ACL by default, explicit content types, and invalidation
scoped to the prefix. Each looks like an oversight to a reader assuming a
dedicated bucket, and each is destructive if "corrected".

### Changed

- `publish-to-s3`: `prefix` is now optional. An empty prefix publishes to the
bucket root and invalidates the whole distribution, which is correct for a
bucket belonging to one project. The script reports when that happens.
- `publish-to-s3`: **the `acl` default is now `none`**, following AWS's
recommendation to disable ACLs and grant access by bucket policy. A bucket
in legacy `ObjectWriter` mode with no policy must now set
`acl: public-read` explicitly.
- `publish-to-s3`: when `public-base-url` is set, one published file is
fetched after a real publish and a non-200 fails the job. An object uploaded
without an ACL a legacy bucket needs otherwise succeeds and is silently
unreadable.

### Fixed

- `publish-to-s3` no longer fails after a successful publish when 40 or fewer
files are uploaded. The summary step ended on a test that returns non-zero
when false, and composite action steps run with `bash -eo pipefail`.

## [1.1.0] - 2026-09-01

### Added

- `publish-to-s3` action: uploads built assets to S3 and invalidates the
matching CloudFront paths, authenticating with OIDC so no long-lived AWS
keys are required. Replaces per-repository `publish_to_s3.bash` and
`invalidate_cdn.bash` scripts.
- `public-base-url` input on `publish-to-s3`, which turns the job summary into
links to each published file. S3 website endpoints serve no directory
listing, so the files have to be named individually.

## [1.0.0] - 2026-08-31

### Added

- `build-pandoc`, `build-zensical` and `build-sphinx` actions, one per
documentation generator. Only the build step differs between them; see
[ADR-0003](docs/decisions/0003-one-build-action-per-generator.md).
- `index-site` action, building a Pagefind index over already-built HTML, so
it works with any generator.
- `deploy-site` action, which refuses to publish a build containing no HTML or
no `index.html` before uploading it as a Pages artifact.
- `docs-pandoc.yml` reusable workflow, wrapping the common build, index and
deploy sequence.
- The shared Caltech Pandoc theme in `pandoc/`, overridable per project.
- Architecture decision records in `docs/decisions/`.

[Unreleased]: https://github.com/caltechlibrary/workflows/compare/v1.1.0...HEAD
[1.1.0]: https://github.com/caltechlibrary/workflows/compare/v1.0.0...v1.1.0
[1.0.0]: https://github.com/caltechlibrary/workflows/releases/tag/v1.0.0
34 changes: 34 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,3 +169,37 @@ rejected, which is the part the code cannot tell you.

Adding a generator does not need an ADR. Changing the contract every generator
satisfies does.

Every change that reaches a consumer gets a `CHANGELOG.md` entry under
`## [Unreleased]`, following
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Consumers reference a
moving major tag, so the changelog is the only way they learn what moved.

## Releasing

Entries accumulate under `## [Unreleased]` as pull requests merge. Releasing is
a separate, deliberate step — one commit and one tag per release, not per
change.

1. Rename `## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD`, and add a fresh empty
`## [Unreleased]` above it.
2. Update the link references at the bottom of `CHANGELOG.md`.
3. Commit that on its own: `Release X.Y.Z`.
4. Tag that commit: `git tag -a vX.Y.Z -m "vX.Y.Z — <summary>"`.
5. Move the major tag: `git tag -f vX` and force-push it.
6. Push the commit and both tags.

Tagging the release commit means the tag points at a changelog that describes
itself, rather than one still saying "Unreleased".

Choosing the number, per
[ADR-0006](docs/decisions/0006-version-with-moving-major-tags.md):

| Change | |
| --- | --- |
| An input renamed or removed, an action path changed, **a default changed** | major |
| A new action or input, backward compatible | minor |
| A fix with no interface change | patch |

Changing a default is breaking even though nothing in the interface moved:
existing callers get different behavior without editing anything.
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ decision instead of a rewrite.
| `build-sphinx` | Sphinx, using the project's own documentation requirements |
| `index-site` | Pagefind search index over built HTML — works with any generator |
| `deploy-site` | Checks the build produced a real site, uploads it for Pages |
| `publish-to-s3` | Uploads built assets to S3 over OIDC and invalidates CloudFront |

Adding a generator means one new `build-*` action; nothing downstream changes.

Expand Down Expand Up @@ -190,6 +191,54 @@ rather than at `sphinx-build` with a confusing error.

Skip this for generators with their own search.

### `publish-to-s3`

Authenticates with OIDC, so no long-lived AWS keys exist anywhere. The calling
job must declare `permissions: id-token: write` — a composite action cannot.

The role's trust policy has to name the **calling repository's** subject claim.
That claim is repo-specific, so a role that works for one repository will not
work for another until an entry is added.

| Input | Default | |
| --- | --- | --- |
| `role-to-assume` | — | ARN of the role to assume via OIDC |
| `aws-region` | `us-west-2` | |
| `bucket` | — | S3 bucket name |
| `prefix` | — | Key prefix within the bucket. Leave empty to publish to the bucket root |
| `sources` | `dist`, `css` | Directories to upload, one per line. `dist` uploads to the prefix root |
| `distribution-id` | — | CloudFront distribution to invalidate; omit to skip |
| `acl` | `none` | AWS recommends ACLs disabled. Set to `public-read` for a legacy bucket with no policy |
| `public-base-url` | — | Public URL the prefix is served from. When set, the summary links every published file and one is fetched afterwards to confirm it is readable |
| `dry-run` | `false` | Show what would happen and change nothing |

Run it with `dry-run: true` first. It refuses to publish an empty or missing
source directory, because a build that "succeeds" into nothing would otherwise
leave the CDN serving stale objects with no signal.

Content types are set explicitly per extension rather than left to the CLI's
guess, which omits the charset and can differ between CLI versions — otherwise
what the CDN serves could change for files nobody edited.

The `acl` default follows AWS's recommendation: ACLs disabled, access granted
by bucket policy. A bucket still in legacy `ObjectWriter` mode with no policy
needs `acl: public-read` — without it, objects upload successfully and are
unreadable. Setting `public-base-url` catches that: after a real publish one
file is fetched, and a 403 fails the job.

With no `prefix`, files go to the bucket root and the invalidation covers the
whole distribution — correct when the bucket belongs to one project, wrong when
it is shared. The script says so when it happens.

S3 website endpoints serve no directory listing — a URL ending in `/` returns
404 rather than an index — so the summary names each published file instead of
linking a parent directory. Set `public-base-url` to turn those into links.

It uses `aws s3 cp`, **not `sync --delete`**. Buckets often hold objects the
publishing repository does not manage, and a delete-enabled sync would remove
them with nothing to put them back. See
[ADR-0007](docs/decisions/0007-publish-without-destroying-what-you-did-not-create.md).

### `deploy-site`

| Input | Default | |
Expand Down Expand Up @@ -244,6 +293,10 @@ documentation requirements for `build-sphinx.sh`, and

## Versioning

Changes are recorded in [CHANGELOG.md](CHANGELOG.md), following
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Because `@v1` moves,
that file is how you find out what moved.

Reference a **major tag**:

```yaml
Expand Down
Loading