Skip to content

ci(service): run the service end to end on kind and docker compose; fix two Helm chart bugs (T09 follow-up) - #44

Open
nick-developer wants to merge 6 commits into
mainfrom
claude/project-thread-ifv3sd
Open

ci(service): run the service end to end on kind and docker compose; fix two Helm chart bugs (T09 follow-up)#44
nick-developer wants to merge 6 commits into
mainfrom
claude/project-thread-ifv3sd

Conversation

@nick-developer

@nick-developer nick-developer commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Requested by Cod3r · project thread

What and why

T09 left one open item: run the Kubernetes acceptance test on a real cluster. The sandbox could only dry-run the chart, and the docker compose test (31/31) had only been run by hand. This PR adds .github/workflows/service-e2e.yml. It runs when the service, chart or images change, weekly, and by hand. It is not a required check.

  • compose job: deploy/e2e/compose-e2e.sh (SSO, run, approval, download, audit, tenant isolation, and a worker killed mid-run).
  • kind job:
    • helm lint of the chart.
    • A server-side dry run of the chart in a namespace that enforces Pod Security restricted.
    • deploy/e2e/k8s-e2e.sh on a kind cluster. deploy/e2e/k8s/kind.yaml maps NodePorts 30080/30081 to localhost, as the e2e flow and the OIDC issuer expect.
  • k8s-e2e.sh now also:
    • deletes the worker pod that holds a job's lease (restart_check.py --k8s-namespace) and checks that the other worker finishes the run;
    • waits for the worker Deployment to recover;
    • on failure, prints pods, events, logs, each container's last termination state and its previous log.

The kind job found two real chart bugs in 2.0.0, both fixed here (CHANGELOG, Unreleased > Fixed):

  1. A fresh helm install timed out. The pre-install migrate hook ran as the release ServiceAccount, which Helm only creates after the hooks. The migrate job now has its own hook ServiceAccount (<release>-migrate, weight -5, same annotations).
  2. Workers lost every synthesis run. The data PVC was mounted at /var/lib/relsynth, so the runtime created /var/lib/relsynth/data (the parent of the work emptyDir) as root, and the non-root worker crashed with Permission denied: '/var/lib/relsynth/data/tenants'. The PVC is now mounted at <data_dir>/tenants, the only directory the service writes. Upgrade note: an existing volume keeps artefacts under data/tenants/; move them to the volume root.

Also: types failed on every PR after SQLAlchemy 2.1.8's stubs rejected sa.Column(**kw) in connectors/ddl.py; kw is now typed dict[str, Any]. The e2e DP release test's quality floor is 0.3 because its DP noise is unseeded.

Known limit: kind's default network plugin does not enforce NetworkPolicy, so this test does not prove the policy.

How it was tested

  • ruff check, bash -n deploy/e2e/k8s-e2e.sh, mypy ratchet, workflow YAML parses
  • kind and compose jobs pass on GitHub Actions (head 4e3026f), with all 29 checks green

Statistics and privacy

  • No new statistical or privacy claim; no engine change

Checklist

  • Conventional Commits, signed off
  • CHANGELOG: Unreleased > Fixed (both chart bugs)
  • Documentation updated
  • No data committed; no new dependencies

🤖 Generated with Claude Code

https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn

@claude
ci(service): run the service end to end on kind and docker compose
2574b18 
The T09 acceptance flow had only run by hand (compose 31/31), and the
Kubernetes script had never run on a real cluster: the sandbox could
only dry-run the chart. The new service-e2e workflow runs, when the
service, chart or images change, weekly and by hand:

- compose: deploy/e2e/compose-e2e.sh (SSO, run, approval, download,
  audit, tenant isolation, a worker killed mid-run);
- kind: lint and a server-side dry run of the chart under Pod Security
  'restricted', then deploy/e2e/k8s-e2e.sh on a kind cluster
  (deploy/e2e/k8s/kind.yaml maps NodePorts 30080/30081 to localhost).

k8s-e2e.sh now also deletes the worker pod that holds a job's lease
(restart_check.py --k8s-namespace; the lease owner starts with the pod
name) and checks the other worker finishes it, and prints pods, events
and logs on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@nick-developer nick-developer self-assigned this Oct 31, 2026
claude added 5 commits November 1, 2026 02:24
@claude
fix(e2e): use the local chart path when helm is installed; steady DP …
02abfcb 
…gate

k8s-e2e.sh defined a helm shell function in both branches, so the
check for a helm binary never matched and the chart path stayed
/src/... (the docker fallback's mount), which the kind job could not
find. The chart and values paths are now chosen together with helm.

test_governed_dp_run_charges_the_domain_and_releases_externally set
the organisation's quality floor at 0.5, but DP noise is never seeded
(by design), and the fixture's pass rate varied (0.472 in CI on this
PR, above 0.5 on #40). The test is about the budget and release flow;
the floor is now 0.3, which still exercises the quality rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@claude
fix(helm): give the migrate hook job its own pre-install ServiceAccount
c5db4d5 
A fresh helm install timed out: the pre-install migrate job referenced the
release ServiceAccount, which Helm creates only after the pre-install hooks
run. The job now uses a hook ServiceAccount (weight -5, same annotations)
created before it. The kind e2e job found this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@claude
test(e2e): print why restarted pods ended and their previous logs
b1ff90b 
The kind run's workers restarted during the synthesis job with an empty
current log, so the failure output now includes each container's last
termination state and its previous log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@claude
fix(helm): mount the data PVC at <data_dir>/tenants so non-root worke…
4639430 
…rs can write

The PVC was mounted at /var/lib/relsynth; the work emptyDir at
/var/lib/relsynth/data/work made the runtime create /var/lib/relsynth/data
as root, so workers crashed with Permission denied creating data/tenants and
every synthesis run was lost. The kind e2e job's new previous-log output
showed the traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@claude
fix(types): type the Column keyword arguments in ddl.py
4e3026f 
SQLAlchemy 2.1.8's stubs reject **dict[str, bool | None] for Column, which
failed the mypy ratchet on every PR (not caused by this PR's changes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFQpRaQG7JXjqkxnbn
Signed-off-by: Claude <noreply@anthropic.com>
@nick-developer nick-developer mentioned this pull request Nov 6, 2026
Open
11 of 12 tasks
nick-developer pushed a commit that referenced this pull request Nov 6, 2026
@claude
fix(types): annotate DDL column kwargs for SQLAlchemy 2.1.8 stubs (po…
ac54eac 
…rt of the PR #44 fix)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwFW6fhyVohUnuKd2fu
Signed-off-by: Claude <noreply@anthropic.com>
@nick-developer nick-developer changed the title ci(service): run the service end to end on kind and docker compose (T09 follow-up) ci(service): run the service end to end on kind and docker compose; fix two Helm chart bugs (T09 follow-up) Nov 6, 2026
@nick-developer nick-developer mentioned this pull request Nov 6, 2026
Draft
12 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@nick-developer nick-developer

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nick-developer @claude