docs(opa): the OPA surface on its own engines-only port

platform/data-governance/engines-and-policy/trino-on-opa.md

@@ -23,9 +23,11 @@ EGRET_GOVERNANCE__OPA__ENABLED=true
 EGRET_RANGER__ENABLED=true
 ```
 
+Egret serves the OPA routes on their own port, **8082** by default, not on the API port 8080 (see [Network exposure](#network-exposure)). On Kubernetes, the Helm value `opa.enabled=true` sets both keys of the OPA listener for you (see [Kubernetes](../../deploy-operate/kubernetes.md)).
+
 It also needs `ranger.enabled = true`, because the OPA routes check the engine credentials of the Ranger surface (see [Engines and Ranger setup](../../deploy-operate/engines-and-ranger.md)). With the Ranger surface off, OPA does not run and a switch to OPA is refused with the same message as a disabled gate.
 
-With the gate off, the OPA routes are not mounted (they answer 404), no snapshot refresher runs, and a switch to OPA is refused: *OPA is not enabled on this installation ([governance.opa] enabled)*. The two other keys of `[governance.opa]` are under [Limits](#limits).
+With the gate off, the OPA routes are not mounted (they answer 404), no snapshot refresher runs, and a switch to OPA is refused: *OPA is not enabled on this installation ([governance.opa] enabled)*. The keys that tune the request bodies are under [Limits](#limits); the listener's keys are under [Network exposure](#network-exposure).
 
 **Turning the gate off while profiles are on OPA makes their engines fail every query**, because the routes unmount. Switch those profiles back to Ranger first.
 
@@ -137,6 +139,8 @@ The engine's **Plugin** tab in **Governance → Engine & policy** has **Download
 
 Read `external_engines_unchanged` (or the section's *Not restarted (external)* line) after every switch.
 
+**After an upgrade to the OPA port.** An external engine that already runs the OPA plugin points at the old address on port 8080, which now answers 404 for `/opa/`, so it fails every query. Download its files again (the plugin-config route above now returns the OPA port's URL), install them in `/etc/trino/` and restart the coordinator. Egret re-applies the engines it provisions itself (see [Network exposure](#network-exposure)); it does not touch an external one.
+
 ## Never enable opa.log-requests
 
 The engine's Ranger plugin credential is the OPA secret, and the plugin sends it **inside every request body**, from `egret-opa-context.properties`. The generated files set `opa.log-requests` and `opa.log-responses` to `false`, and the files carry the warning *Do not enable opa.log-requests: every request carries egret.secret in input.context.properties, and the coordinator log would hold it.*
@@ -223,12 +227,33 @@ Explain follows the **profile's** mechanism, not each engine's. It also cannot s
 
 ## Network exposure
 
-The OPA routes live on Egret's main listener, under `/opa/v1/data/egret/trino/`: `allow`, `batch`, `row_filters`, `batch_column_masks` and `column_mask`. They have no CSRF protection and no OpenAPI entry.
+The OPA routes live on their own listener, under `/opa/v1/data/egret/trino/`: `allow`, `batch`, `row_filters`, `batch_column_masks` and `column_mask`. The listener serves nothing else. They have no CSRF protection and no OpenAPI entry. Port 8080 answers 404 for `/opa/`; `/ranger/`, `/oauth/token` and the Iceberg REST catalog stay on 8080, and engines on the Ranger mechanism are unchanged.
+
+The OPA surface reads a request body before it can authenticate it. On a shared port, any pod that could reach Egret could hold stalled requests and use up the budget that engines need; on its own port, a NetworkPolicy can admit only the engines.
+
+| Key | Environment variable | Default | Meaning |
+|---|---|---|---|
+| `[governance.opa] listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, as `host:port`. With OPA enabled, Egret refuses to start when its port equals `server.port` or, when apps are enabled (`apps.origin` is set), the apps listener's port. With OPA off it is not checked, because nothing binds it. |
+| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL; `http://egret:8080` when `public_url` is unset, so `http://egret:8082`) and replaces its port with the port of `listen`, then trims a trailing slash. With OPA enabled, a value that is not an absolute `http(s)` URL with a host stops Egret at start. |
+
+:::warning
+**Set `engine_url` when `[catalog] public_url` is not Egret's in-cluster or internal address.** If it is, for example, an ingress hostname, the derived URL points at that hostname on the OPA port and engines cannot reach it. Set `engine_url` to an address engines can reach, such as `http://egret.<namespace>.svc.cluster.local:8082`.
+:::
 
 :::danger
-**Restrict `/opa/` to the engines' network**, with an ingress rule or a NetworkPolicy, so that only Trino coordinators reach it. This is required, not advisory. Egret reads request bodies before it authenticates the caller, and a caller that opens connections and stalls them can hold most of the in-flight budget, renewing it every `body_read_timeout_ms`; the engines' queries then get 503 and fail.
+**Only the engines may reach the OPA port.** Allow it from the engines' network and from nothing else. On Kubernetes, `engineNetworkPolicy.enabled` renders `NetworkPolicy/egret-ingress`, which admits the OPA port from `engineNamespace` and from `opa.extraFrom` only (see [Kubernetes](../../deploy-operate/kubernetes.md#engine-network-policy)). Any firewall between the engines and Egret must allow the OPA port, 8082 by default, or every query on an OPA engine fails.
 :::
 
+### Engines that Egret provisions
+
+Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies, in the background, each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server; apply a failed engine again with any change that applies it. The pass runs after the new Egret starts serving, and each re-applied coordinator restarts.
+
+:::warning
+**Queries on these engines fail until each one is re-applied and restarted.** Until then an engine asks port 8080 for `/opa/`, which answers 404. During a rolling update an engine already moved to the OPA port is also refused whenever its request reaches an old Egret pod, which has no OPA listener. Upgrade in a maintenance window, or when the query load on OPA engines is low. See [Upgrade](../../deploy-operate/upgrade.md).
+:::
+
+External engines are not re-applied: see [External engines](#external-engines).
+
 ## Limits
 
 Two keys of `[governance.opa]` tune the request bodies:

platform/deploy-operate/kubernetes.md

@@ -25,14 +25,15 @@ With default values `helm template` renders these objects:
 | Object | Purpose |
 |---|---|
 | `Deployment/egret` | The Egret server. Probes: startup `GET /healthz`, readiness `GET /readyz`, liveness `GET /healthz`. Requests 500m CPU and 1 GiB, limits 2 CPU and 4 GiB. `terminationGracePeriodSeconds` is 65, which must stay above the server's 55 second drain. |
-| `Service/egret` | Port 8080 (and 8081 for Data Apps when `apps.origin` is set). |
+| `Service/egret` | Port 8080 (8081 for Data Apps when `apps.origin` is set, and the port named `opa` when `opa.enabled` is set). |
 | `Deployment/egret-frontend`, `Service/egret-frontend` | The built web app behind nginx, as a NodePort Service (30080). |
 | `Deployment/egret-renderer`, `Service/egret-renderer` | The screenshot service used by reports. |
 | `Job/egret-init` | A post-install and post-upgrade hook that runs `egret init`, creating the administrator and the `default` workspace. It is idempotent. |
 | `Secret/egret-secrets` | `master-key`, `secret-key`, `catalog-auth-token`, storage credentials, the admin password and the OIDC client secret. The three keys are generated at install and **reused on upgrade** (looked up from the existing Secret). |
 | `ConfigMap/egret-env` | Every `EGRET_*` setting that is not a secret. |
 | `ServiceAccount`, `Role` and `RoleBinding` `egret-engine-provisioner` | Lets Egret manage workloads in the engine namespace. |
 | `NetworkPolicy/egret-engine-isolation` | In the engine namespace: only that namespace's own pods, Egret's pods and the StarRocks operator (front end ports 8030 and 9030) may connect to the engines (since platform #828). See [Engine network policy](#engine-network-policy). |
+| `NetworkPolicy/egret-ingress` | In the release namespace, selecting Egret's backend pods (`app: egret`): 8080 open to every peer, 8081 open when `apps.origin` is set, and the OPA port only from `engineNamespace` and `opa.extraFrom`. Rendered when `engineNetworkPolicy.enabled`. See [Engine network policy](#engine-network-policy). |
 | `Deployment/postgres`, `PersistentVolumeClaim/postgres-data` (1 GiB), `Service/postgres` | Bundled PostgreSQL 16 (`deps.enabled`). |
 | `Deployment/rustfs`, `PersistentVolumeClaim/rustfs-data`, `Service/rustfs`, `Job/egret-rustfs-bucket` | Bundled RustFS and a job that creates the bucket. |
 | `Pod/mailpit`, `Service/mailpit` | A mail catcher so you can read report emails. Never for real use. |
@@ -177,10 +178,13 @@ Running a script task also needs the workspace feature `workflow_containers` (of
 |---|---|---|
 | `ranger.enabled` | `false` | Sets `EGRET_RANGER__ENABLED`. The chart sets `EGRET_RANGER__AUTH_TOKEN` to empty, so stock plugins can download policies. |
 | `engineNamespace` | `egret-engines` | Where Egret provisions engines. Normally pre-created by `just k8s-up`; set `createEngineNamespace=true` on a bare cluster. |
-| `engineNetworkPolicy.enabled` | `true` | Render `NetworkPolicy/egret-engine-isolation` in `engineNamespace`. |
+| `engineNetworkPolicy.enabled` | `true` | Render `NetworkPolicy/egret-engine-isolation` in `engineNamespace` and `NetworkPolicy/egret-ingress` in the release namespace. |
 | `engineNetworkPolicy.operatorNamespaces` | `[starrocks]` | Namespaces of the StarRocks operator, let in on the front end's 8030 and 9030. |
 | `engineNetworkPolicy.ingressFrom` | `[]` | NetworkPolicy peers an engine Ingress forwards from (your ingress controller), let in on 8030 (StarRocks front end) and 8080 and 8443 (Trino). **Required when an engine has an Ingress**, or the Ingress gets no traffic. |
 | `engineNetworkPolicy.extraFrom` | `[]` | Other NetworkPolicy peers, let in on every port, for example a metrics scraper. |
+| `opa.enabled` | `false` | Sets `EGRET_GOVERNANCE__OPA__ENABLED=true` and `EGRET_GOVERNANCE__OPA__LISTEN=0.0.0.0:<opa.port>`, and adds the `opa` port to the Deployment and the `Service/egret`. It still needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). |
+| `opa.port` | `8082` | The port of Egret's OPA listener. Egret derives the URL engines use from `catalog.public_url` with this port, so no other value is needed while that URL is Egret's in-cluster address. |
+| `opa.extraFrom` | `[]` | NetworkPolicy peers beyond `engineNamespace` that may reach the OPA port, for example the namespace of an external Trino on OPA. |
 | `apps.origin`, `apps.egretOrigin` | empty | The Data Apps origin (a sibling subdomain such as `apps.<host>`) and Egret's own origin. Empty turns the apps iframe and its listener off. |
 | `apps.pod.launcher` | `none` | `kubernetes` starts one hardened pod per author per workspace in `apps.pod.namespace` (`egret-apps`). Also `image`, `imagePullPolicy`, `resources`, `readyTimeoutSecs`, `idleSecs`, `networkPolicy.enabled` and `run.*`. |
 
@@ -212,6 +216,10 @@ Measured on kind 0.29 with a listener on 8030, 9030, 8080 and 9010: before the p
 after it, a task pod and a non-Egret pod of the release namespace were refused on all four, the operator's namespace got
 8030 and 9030 only, and an `ingressFrom` peer got 8030 and 8080 only.
 
+`NetworkPolicy/egret-ingress` protects Egret's own pods the other way round. It selects the backend pods (`app: egret`) and admits 8080 from every peer (the frontend, ingresses, app and task pods and the kubelet use it), 8081 when `apps.origin` is set, and the OPA port, 8082 by default, **only** from `engineNamespace` and `opa.extraFrom`. The OPA surface reads a request body before it can authenticate it, so a pod that could reach that port could hold stalled requests and exhaust the budget engines need. The policy renders whenever `engineNetworkPolicy.enabled` is set, with or without `opa.enabled`. This describes what the chart renders; it has not been run on a live cluster.
+
+An external engine on OPA lives outside `engineNamespace`: add its namespace to `opa.extraFrom`, for example `- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: trino-prod } }`. Any firewall between the engines and Egret must allow the OPA port too. When `catalog.public_url` is not Egret's in-cluster address, set `[governance.opa] engine_url` (see [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md#network-exposure)).
+
 The policy needs a network plugin that enforces NetworkPolicy (kind 0.24 and later, Calico, Cilium and most managed
 clusters do). An external engine is outside it: put it behind your own firewall.
 

platform/deploy-operate/upgrade.md

@@ -635,6 +635,36 @@ configuration of its last apply.
 See [Security & Exposure](../control-plane/query-engines/provision.md#security--exposure) and KI-355 in
 [Known issues](../known-issues.md).
 
+### The OPA surface moves to its own port (platform #843)
+
+Egret's OPA routes (`/opa/`) leave the API port 8080 and are served on their own listener, `[governance.opa] listen`, **8082** by
+default. Port 8080 now answers 404 for `/opa/`; `/ranger/`, `/oauth/token` and the Iceberg REST catalog stay there. There is a
+migration (`0216`, one column, `applied_opa_url`, that records the OPA URL each engine was last applied with). Nothing changes
+while `[governance.opa] enabled` is `false`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md#network-exposure).
+
+- **New keys:** `[governance.opa] listen` and `engine_url`, checked only while `[governance.opa] enabled` is `true`. Egret then
+  refuses to start when `listen` has the port of `server.port` or, with `apps.origin` set, of the apps listener, or when
+  `engine_url` is set but is not an absolute `http(s)` URL with a host. When `engine_url` is unset, engines get `[catalog] public_url` with the port of
+  `listen`; **if `public_url` is an ingress hostname rather than Egret's in-cluster address, set `engine_url`**.
+- **Provisioned engines re-apply themselves.** At start, with OPA enabled, Egret re-applies each provisioned Trino engine on an OPA
+  profile whose recorded OPA URL differs from the current one, including engines applied before the upgrade. A failure is logged
+  per engine as a warning and does not stop the server; apply that engine again with any change that applies it. The pass runs in
+  the background after the new Egret starts serving, and each re-applied coordinator restarts.
+- **Queries on provisioned OPA engines fail during the upgrade.** From the moment the new Egret serves until an engine has been
+  re-applied and its coordinator has restarted, that engine still asks port 8080 for `/opa/`, which now answers 404, so its queries
+  are refused. During a rolling update, old and new Egret pods serve side by side: an engine already re-applied to the OPA port is
+  also refused whenever its request reaches an old pod, which has no OPA listener. On Docker the window includes the Egret container
+  recreate. How long this lasts depends on the number of engines and on how fast each coordinator restarts. **Upgrade in a
+  maintenance window, or when the query load on OPA engines is low.** Engines on the Ranger mechanism are not affected.
+- **External engines re-install their files.** Download the OPA plugin files again (**Control plane → Engines → \<engine\> → Plugin
+  config**, or `GET /api/control-plane/engines/{id}/opa-plugin-config`, which now returns the OPA port's URL), install them and
+  restart the coordinator. From the upgrade until then the engine asks port 8080 and fails every query.
+- **Firewalls:** allow the OPA port (8082 by default) from the engines to Egret. Anything between them that only allowed 8080 blocks OPA.
+- **Helm:** new values `opa.enabled` (default `false`), `opa.port` (`8082`) and `opa.extraFrom`, and a new
+  `NetworkPolicy/egret-ingress`, rendered when `engineNetworkPolicy.enabled`, that admits the OPA port only from `engineNamespace`
+  and `opa.extraFrom`. With `engineNetworkPolicy.enabled`, `egret-ingress` admits only 8080, 8081 (apps) and the OPA port to the backend pod: anything that reached another port on it is now refused. Add the namespace of an external Trino on OPA to `opa.extraFrom`. See
+  [Kubernetes](./kubernetes.md#engine-network-policy).
+
 ## Further reading
 
 - [Backup and restore](./backup-restore)

platform/reference/configuration-keys.md

@@ -156,9 +156,11 @@ The S3-compatible object store used for the catalog warehouse, query results, wo
 
 | Key | Environment variable | Default | Meaning |
 |---|---|---|---|
-| `enabled` | `EGRET_GOVERNANCE__OPA__ENABLED` | `false` | Serve the OPA decision routes for Trino and allow a policy profile to switch to OPA. Needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). |
+| `enabled` | `EGRET_GOVERNANCE__OPA__ENABLED` | `false` | Serve the OPA decision routes for Trino, on their own listener (`listen`), and allow a policy profile to switch to OPA. Needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). |
 | `batch_body_limit_bytes` | `EGRET_GOVERNANCE__OPA__BATCH_BODY_LIMIT_BYTES` | `25165824` (24 MiB) | Largest request body the OPA `/batch` route accepts. Trino sends a listing of a whole schema as one batch; a larger body is answered 413 and the query fails. From 4194304 (4 MiB) to 67108864 (64 MiB); every other OPA route keeps 4 MiB. |
 | `body_read_timeout_ms` | `EGRET_GOVERNANCE__OPA__BODY_READ_TIMEOUT_MS` | `5000` | How long an admitted OPA request may take to send its whole body before it is answered 408 and its connection closed. From 1 to 60000. |
+| `listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, `host:port`. It serves only the OPA routes; port 8080 answers 404 for `/opa/`. With `enabled`, Egret refuses to start when the port equals `server.port` or, with `apps.origin` set, the apps listener's port; with OPA off it is not checked. |
+| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL; `http://egret:8080` if unset) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. With `enabled`, it must be an absolute `http(s)` URL with a host. |
 
 ## engine