Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ bootstrap apply or node provisioning.
| Domain | Path | TLS |
|---|---|---|
| `<app>.makeitwork.cloud` | HTTP via cloudflare-operator `TunnelBinding` | Cloudflare edge |
| `k3s.makeitwork.cloud` | TCP via `ClusterTunnel` to kube-apiserver, gated by Cloudflare Access | Cloudflare edge |
| `api.makeitwork.cloud` | HTTPS via `ClusterTunnel` to kube-apiserver; Kubernetes OIDC required | Cloudflare edge |
| `k3s.makeitwork.cloud` | TCP via `ClusterTunnel` to kube-apiserver, gated by Cloudflare Access (migration fallback) | Cloudflare edge |

There is no in-cluster ingress controller and no public IP. Every external entry point — public web, kubectl, everything — uses a Cloudflare Tunnel managed by cloudflare-operator. App CNAMEs are declared in `tfroot-cloudflare` and must stay aligned with the routes here. Legacy hostnames `api.makeitwork.cloud` and `*.apps.makeitwork.cloud` are not in use.
There is no in-cluster ingress controller and no public IP. Every external entry point — public web, kubectl, everything — uses a Cloudflare Tunnel managed by cloudflare-operator. App CNAMEs are declared in `tfroot-cloudflare` and must stay aligned with the routes here. The legacy `*.apps.makeitwork.cloud` hostnames are not in use.

## Key Namespaces

Expand Down Expand Up @@ -59,11 +60,12 @@ App DNS and routes have coordinated owners:
## kubectl Access

Use the dedicated `makeitworkcloud-k3s` kubeconfig and the Cloudflare/Dex OIDC
procedure in `README.md#kubectl-access`. Never use or modify an unrelated
production or staging context. If no Make IT Work Cloud context is configured,
ask for the approved public server CA and create the non-secret exec kubeconfig
from `docs/kubeconfig.example.yaml`; do not copy the k3s admin kubeconfig off
the node.
procedure in `README.md#kubectl-access`. Normal access connects directly to
`https://api.makeitwork.cloud`; Cloudflare provides public TLS and Dex provides
the Kubernetes identity. Never use or modify an unrelated production or
staging context. If no Make IT Work Cloud context is configured, create the
non-secret exec kubeconfig from `docs/kubeconfig.example.yaml`; do not copy the
k3s admin kubeconfig or private cluster CA off the node.

## SOPS / KSOPS

Expand Down
62 changes: 28 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ creation until every workload-required operator CRD is available.
| Domain | Path | TLS |
|---|---|---|
| `<app>.makeitwork.cloud` | HTTP via cloudflare-operator `TunnelBinding` | Cloudflare edge |
| `k3s.makeitwork.cloud` | TCP via `ClusterTunnel` to kube-apiserver, gated by Cloudflare Access | Cloudflare edge |
| `api.makeitwork.cloud` | HTTPS via `ClusterTunnel` to kube-apiserver; Kubernetes OIDC required | Cloudflare edge |
| `k3s.makeitwork.cloud` | TCP via `ClusterTunnel` to kube-apiserver, gated by Cloudflare Access (migration fallback) | Cloudflare edge |

There is no in-cluster ingress controller and no public IP. Every external entry point is a Cloudflare Tunnel.

Expand Down Expand Up @@ -60,35 +61,24 @@ the `makeitworkcloud:admins` GitHub team to cluster-admin through

### kubectl access

Access has two independent gates:
Normal access connects directly to `https://api.makeitwork.cloud`. Cloudflare
proxies HTTPS to the in-cluster API Service, while Kubernetes remains the
authentication and authorization boundary. `kubelogin` obtains a Dex token;
Dex includes the requested `email` and `groups` claims, which the API server
and Kubernetes RBAC validate.

1. Cloudflare Access authorizes the TCP connection to
`k3s.makeitwork.cloud` using the policy in `tfroot-cloudflare`.
2. `kubelogin` obtains a Dex token. Dex includes the requested `email` and
`groups` claims, which the API server and Kubernetes RBAC validate.

Install `cloudflared`, `kubectl`, and the `kubectl oidc-login` plugin. Confirm
all three commands are available before continuing. Start the local TCP proxy
in a dedicated terminal:

```bash
cloudflared access tcp --hostname k3s.makeitwork.cloud --url localhost:6443
```

If port 6443 is occupied, choose another local port and use the same port in
the kubeconfig server URL.
Install `kubectl` and the `kubectl oidc-login` plugin. Confirm both commands are
available before continuing.

Create a dedicated kubeconfig such as `~/.kube/makeitworkcloud-k3s.yaml`, mode
`0600`. Its cluster entry must point to `https://127.0.0.1:6443` and trust the
public k3s server CA supplied by an administrator. That CA may be copied from
`/var/lib/rancher/k3s/server/tls/server-ca.crt` on the k3s VM through an
approved channel. The kubeconfig must not contain a token, client certificate,
or client key. Do not copy `/etc/rancher/k3s/k3s.yaml` off the node: it contains
cluster-admin client credentials.
`0600`. Its cluster entry points to `https://api.makeitwork.cloud`; the public
Cloudflare certificate uses normal system CA trust. The kubeconfig must not
contain a token, client certificate, client key, or private cluster CA. Do not
copy `/etc/rancher/k3s/k3s.yaml` off the node: it contains cluster-admin client
credentials.

Copy `docs/kubeconfig.example.yaml` to that dedicated path, replace its
`certificate-authority` placeholder with the absolute path to the approved CA
file, and set mode `0600`. Its user exec credential is:
Copy `docs/kubeconfig.example.yaml` to that dedicated path and set mode `0600`.
Its user exec credential is:

```yaml
user:
Expand All @@ -108,9 +98,8 @@ user:
```

Give this cluster a distinct context name such as `makeitworkcloud-k3s`; never
reuse an unrelated production or staging context. With the proxy running,
first confirm the dedicated file's current context, server, and user name
without displaying credentials:
reuse an unrelated production or staging context. First confirm the dedicated
file's current context, server, and user name without displaying credentials:

```bash
export KUBECONFIG="$HOME/.kube/makeitworkcloud-k3s.yaml"
Expand All @@ -120,7 +109,7 @@ kubectl config view --minify \
```

The expected context is `makeitworkcloud-k3s`, the server is
`https://127.0.0.1:6443`, and the user is the dedicated OIDC exec user. Then
`https://api.makeitwork.cloud`, and the user is the dedicated OIDC exec user. Then
verify the authenticated identity before performing any change:

```bash
Expand All @@ -129,9 +118,14 @@ kubectl --context makeitworkcloud-k3s auth can-i '*' '*' --all-namespaces
```

`auth whoami` should show your email and the `makeitworkcloud:admins` group;
`auth can-i` should return `yes`. When finished, stop the local `cloudflared`
process. OIDC and Cloudflare Access sessions expire independently; do not run a
global credential-cache cleanup unless you have reviewed its scope.
`auth can-i` should return `yes`. The OIDC plugin caches tokens in the operating
system keyring; do not run a global credential-cache cleanup unless you have
reviewed its scope.

During migration, the previous Access-gated TCP path remains available at
`k3s.makeitwork.cloud` using `cloudflared access tcp`. Do not remove it until
direct API discovery, watches, logs, exec, copy, and port-forward have been
validated through the HTTPS endpoint.

For break-glass access, SSH to the k3s VM through `hero.makeitwork.cloud` and
run kubectl there with `/etc/rancher/k3s/k3s.yaml`; see `tfroot-libvirt`.
Expand All @@ -152,7 +146,7 @@ Secrets are age-encrypted with field-level selective encryption. The `.sops.yaml

Future KMS-backed `SopsSecret` manifests should use the SOPS KMS recipient managed by the [`makeitworkcloud/tfroot-aws`](https://github.com/makeitworkcloud/tfroot-aws) repository with `encrypted_suffix: Templates`. Do not copy raw KMS key identifiers into docs or chat; use the applied OpenTofu output locally when encrypting migration manifests.

For a full KSOPS deprecation with no secret bootstrap loop, the operator should use ambient AWS auth, not static AWS access keys in a Kubernetes Secret. Because the cluster API is not publicly reachable, the target design is a small public static Kubernetes ServiceAccount OIDC issuer endpoint for AWS STS discovery/JWKS, not public cluster access. The endpoint is hosted outside the cluster by the `makeitworkcloud/www` static site at `https://makeitwork.cloud/oidc`, with discovery metadata at `/oidc/.well-known/openid-configuration` and JWKS at `/oidc/openid/v1/jwks`.
For a full KSOPS deprecation with no secret bootstrap loop, the operator should use ambient AWS auth, not static AWS access keys in a Kubernetes Secret. The interactive public kubectl endpoint is not the stable static issuer AWS STS should depend on, so the target design remains a small public static Kubernetes ServiceAccount OIDC issuer endpoint. The endpoint is hosted outside the cluster by the `makeitworkcloud/www` static site at `https://makeitwork.cloud/oidc`, with discovery metadata at `/oidc/.well-known/openid-configuration` and JWKS at `/oidc/openid/v1/jwks`.

That endpoint serves only public metadata: `/.well-known/openid-configuration` and `/openid/v1/jwks`. The private ServiceAccount signing key remains host-local to k3s/provisioning, k3s issues projected ServiceAccount tokens with the public issuer URL, and AWS IAM reads only the public discovery document and JWKS. This removes the Kubernetes/GitOps AWS-credential bootstrap loop, but it does not eliminate the host-level trust root required for k3s to sign ServiceAccount tokens. Static access keys are acceptable only for short-lived validation, not as the target endstate.

Expand Down
3 changes: 1 addition & 2 deletions docs/kubeconfig.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ preferences: {}
clusters:
- name: makeitworkcloud-k3s
cluster:
server: https://127.0.0.1:6443
certificate-authority: /absolute/path/to/makeitworkcloud-k3s-server-ca.crt
server: https://api.makeitwork.cloud
users:
- name: makeitworkcloud-oidc
user:
Expand Down
16 changes: 12 additions & 4 deletions workloads/kubectl-tunnel/tunnel-binding.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
---
# Routes TCP traffic for k3s.makeitwork.cloud through the cluster-apps tunnel
# to the in-cluster kube-apiserver Service. Combined with the Cloudflare Access
# self-hosted app in tfroot-cloudflare, only authenticated org admins can
# connect via `cloudflared access tcp --hostname k3s.makeitwork.cloud …`.
# Publishes the kube-apiserver through the cluster-apps tunnel. The HTTPS route
# supports direct kubectl OIDC access; the existing Access-gated TCP route is
# retained as a migration fallback until the HTTPS path is validated.
apiVersion: networking.cfargotunnel.com/v1alpha1
kind: TunnelBinding
metadata:
Expand All @@ -16,6 +15,15 @@ subjects:
fqdn: k3s.makeitwork.cloud
protocol: tcp
target: tcp://kubernetes.default.svc:443
- name: kubernetes
spec:
fqdn: api.makeitwork.cloud
protocol: https
target: https://kubernetes.default.svc:443
# cloudflared cannot validate the private Kubernetes service CA until it
# is mounted through ClusterTunnel originCaPool. Traffic remains TLS
# encrypted inside the cluster; Cloudflare terminates public client TLS.
noTlsVerify: true
tunnelRef:
kind: ClusterTunnel
name: cluster-apps
Loading