From a14fed7029a4cb513779976c70215737ca22a228 Mon Sep 17 00:00:00 2001 From: 1Pro-L4R8 Date: Sun, 8 Nov 2026 08:58:03 +0700 Subject: [PATCH 1/4] docs(opa): the OPA surface on its own engines-only port Co-Authored-By: Claude Opus 5.5 --- .../engines-and-policy/trino-on-opa.md | 25 ++++++++++++++++--- platform/deploy-operate/kubernetes.md | 12 +++++++-- platform/deploy-operate/upgrade.md | 22 ++++++++++++++++ platform/reference/configuration-keys.md | 4 ++- 4 files changed, 57 insertions(+), 6 deletions(-) diff --git a/platform/data-governance/engines-and-policy/trino-on-opa.md b/platform/data-governance/engines-and-policy/trino-on-opa.md index 1f19696..fdcbc1b 100644 --- a/platform/data-governance/engines-and-policy/trino-on-opa.md +++ b/platform/data-governance/engines-and-policy/trino-on-opa.md @@ -23,9 +23,11 @@ EGRET_GOVERNANCE__OPA__ENABLED=true EGRET_RANGER__ENABLED=true ``` +Egret serves the OPA routes on their own port, **8082** by default, not on the API port 8080 (see [Network exposure](#network-exposure)). On Kubernetes, the Helm value `opa.enabled=true` sets both keys of the OPA listener for you (see [Kubernetes](../../deploy-operate/kubernetes.md)). + It also needs `ranger.enabled = true`, because the OPA routes check the engine credentials of the Ranger surface (see [Engines and Ranger setup](../../deploy-operate/engines-and-ranger.md)). With the Ranger surface off, OPA does not run and a switch to OPA is refused with the same message as a disabled gate. -With the gate off, the OPA routes are not mounted (they answer 404), no snapshot refresher runs, and a switch to OPA is refused: *OPA is not enabled on this installation ([governance.opa] enabled)*. The two other keys of `[governance.opa]` are under [Limits](#limits). +With the gate off, the OPA routes are not mounted (they answer 404), no snapshot refresher runs, and a switch to OPA is refused: *OPA is not enabled on this installation ([governance.opa] enabled)*. The keys that tune the request bodies are under [Limits](#limits); the listener's keys are under [Network exposure](#network-exposure). **Turning the gate off while profiles are on OPA makes their engines fail every query**, because the routes unmount. Switch those profiles back to Ranger first. @@ -137,6 +139,8 @@ The engine's **Plugin** tab in **Governance → Engine & policy** has **Download Read `external_engines_unchanged` (or the section's *Not restarted (external)* line) after every switch. +**After an upgrade to the OPA port.** An external engine that already runs the OPA plugin points at the old address on port 8080, which now answers 404 for `/opa/`, so it fails every query. Download its files again (the plugin-config route above now returns the OPA port's URL), install them in `/etc/trino/` and restart the coordinator. Egret re-applies the engines it provisions itself (see [Network exposure](#network-exposure)); it does not touch an external one. + ## Never enable opa.log-requests The engine's Ranger plugin credential is the OPA secret, and the plugin sends it **inside every request body**, from `egret-opa-context.properties`. The generated files set `opa.log-requests` and `opa.log-responses` to `false`, and the files carry the warning *Do not enable opa.log-requests: every request carries egret.secret in input.context.properties, and the coordinator log would hold it.* @@ -223,12 +227,27 @@ Explain follows the **profile's** mechanism, not each engine's. It also cannot s ## Network exposure -The OPA routes live on Egret's main listener, under `/opa/v1/data/egret/trino/`: `allow`, `batch`, `row_filters`, `batch_column_masks` and `column_mask`. They have no CSRF protection and no OpenAPI entry. +The OPA routes live on their own listener, under `/opa/v1/data/egret/trino/`: `allow`, `batch`, `row_filters`, `batch_column_masks` and `column_mask`. The listener serves nothing else. They have no CSRF protection and no OpenAPI entry. Port 8080 answers 404 for `/opa/`; `/ranger/`, `/oauth/token` and the Iceberg REST catalog stay on 8080, and engines on the Ranger mechanism are unchanged. + +The OPA surface reads a request body before it can authenticate it. On a shared port, any pod that could reach Egret could hold stalled requests and use up the budget that engines need; on its own port, a NetworkPolicy can admit only the engines. + +| Key | Environment variable | Default | Meaning | +|---|---|---|---| +| `[governance.opa] listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, as `host:port`. Egret refuses to start when its port equals `server.port` or, when apps are enabled (`apps.origin` is set), the apps listener's port. | +| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL) and replaces its port with the port of `listen`, then trims a trailing slash. | + +:::warning +**Set `engine_url` when `[catalog] public_url` is not Egret's in-cluster or internal address.** If it is, for example, an ingress hostname, the derived URL points at that hostname on the OPA port and engines cannot reach it. Set `engine_url` to an address engines can reach, such as `http://egret..svc.cluster.local:8082`. +::: :::danger -**Restrict `/opa/` to the engines' network**, with an ingress rule or a NetworkPolicy, so that only Trino coordinators reach it. This is required, not advisory. Egret reads request bodies before it authenticates the caller, and a caller that opens connections and stalls them can hold most of the in-flight budget, renewing it every `body_read_timeout_ms`; the engines' queries then get 503 and fail. +**Only the engines may reach the OPA port.** Allow it from the engines' network and from nothing else. On Kubernetes, `engineNetworkPolicy.enabled` renders `NetworkPolicy/egret-ingress`, which admits the OPA port from `engineNamespace` and from `opa.extraFrom` only (see [Kubernetes](../../deploy-operate/kubernetes.md#engine-network-policy)). Any firewall between the engines and Egret must allow the OPA port, 8082 by default, or every query on an OPA engine fails. ::: +### Engines that Egret provisions + +Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server; re-apply that engine from the console. External engines are not re-applied: see [External engines](#external-engines). + ## Limits Two keys of `[governance.opa]` tune the request bodies: diff --git a/platform/deploy-operate/kubernetes.md b/platform/deploy-operate/kubernetes.md index ad1bd7d..82f53e3 100644 --- a/platform/deploy-operate/kubernetes.md +++ b/platform/deploy-operate/kubernetes.md @@ -25,7 +25,7 @@ With default values `helm template` renders these objects: | Object | Purpose | |---|---| | `Deployment/egret` | The Egret server. Probes: startup `GET /healthz`, readiness `GET /readyz`, liveness `GET /healthz`. Requests 500m CPU and 1 GiB, limits 2 CPU and 4 GiB. `terminationGracePeriodSeconds` is 65, which must stay above the server's 55 second drain. | -| `Service/egret` | Port 8080 (and 8081 for Data Apps when `apps.origin` is set). | +| `Service/egret` | Port 8080 (8081 for Data Apps when `apps.origin` is set, and the port named `opa` when `opa.enabled` is set). | | `Deployment/egret-frontend`, `Service/egret-frontend` | The built web app behind nginx, as a NodePort Service (30080). | | `Deployment/egret-renderer`, `Service/egret-renderer` | The screenshot service used by reports. | | `Job/egret-init` | A post-install and post-upgrade hook that runs `egret init`, creating the administrator and the `default` workspace. It is idempotent. | @@ -33,6 +33,7 @@ With default values `helm template` renders these objects: | `ConfigMap/egret-env` | Every `EGRET_*` setting that is not a secret. | | `ServiceAccount`, `Role` and `RoleBinding` `egret-engine-provisioner` | Lets Egret manage workloads in the engine namespace. | | `NetworkPolicy/egret-engine-isolation` | In the engine namespace: only that namespace's own pods, Egret's pods and the StarRocks operator (front end ports 8030 and 9030) may connect to the engines (since platform #828). See [Engine network policy](#engine-network-policy). | +| `NetworkPolicy/egret-ingress` | In the release namespace, selecting Egret's backend pods (`app: egret`): 8080 open to every peer, 8081 open when `apps.origin` is set, and the OPA port only from `engineNamespace` and `opa.extraFrom`. Rendered when `engineNetworkPolicy.enabled`. See [Engine network policy](#engine-network-policy). | | `Deployment/postgres`, `PersistentVolumeClaim/postgres-data` (1 GiB), `Service/postgres` | Bundled PostgreSQL 16 (`deps.enabled`). | | `Deployment/rustfs`, `PersistentVolumeClaim/rustfs-data`, `Service/rustfs`, `Job/egret-rustfs-bucket` | Bundled RustFS and a job that creates the bucket. | | `Pod/mailpit`, `Service/mailpit` | A mail catcher so you can read report emails. Never for real use. | @@ -177,10 +178,13 @@ Running a script task also needs the workspace feature `workflow_containers` (of |---|---|---| | `ranger.enabled` | `false` | Sets `EGRET_RANGER__ENABLED`. The chart sets `EGRET_RANGER__AUTH_TOKEN` to empty, so stock plugins can download policies. | | `engineNamespace` | `egret-engines` | Where Egret provisions engines. Normally pre-created by `just k8s-up`; set `createEngineNamespace=true` on a bare cluster. | -| `engineNetworkPolicy.enabled` | `true` | Render `NetworkPolicy/egret-engine-isolation` in `engineNamespace`. | +| `engineNetworkPolicy.enabled` | `true` | Render `NetworkPolicy/egret-engine-isolation` in `engineNamespace` and `NetworkPolicy/egret-ingress` in the release namespace. | | `engineNetworkPolicy.operatorNamespaces` | `[starrocks]` | Namespaces of the StarRocks operator, let in on the front end's 8030 and 9030. | | `engineNetworkPolicy.ingressFrom` | `[]` | NetworkPolicy peers an engine Ingress forwards from (your ingress controller), let in on 8030 (StarRocks front end) and 8080 and 8443 (Trino). **Required when an engine has an Ingress**, or the Ingress gets no traffic. | | `engineNetworkPolicy.extraFrom` | `[]` | Other NetworkPolicy peers, let in on every port, for example a metrics scraper. | +| `opa.enabled` | `false` | Sets `EGRET_GOVERNANCE__OPA__ENABLED=true` and `EGRET_GOVERNANCE__OPA__LISTEN=0.0.0.0:`, and adds the `opa` port to the Deployment and the `Service/egret`. It still needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). | +| `opa.port` | `8082` | The port of Egret's OPA listener. Egret derives the URL engines use from `catalog.public_url` with this port, so no other value is needed while that URL is Egret's in-cluster address. | +| `opa.extraFrom` | `[]` | NetworkPolicy peers beyond `engineNamespace` that may reach the OPA port, for example the namespace of an external Trino on OPA. | | `apps.origin`, `apps.egretOrigin` | empty | The Data Apps origin (a sibling subdomain such as `apps.`) and Egret's own origin. Empty turns the apps iframe and its listener off. | | `apps.pod.launcher` | `none` | `kubernetes` starts one hardened pod per author per workspace in `apps.pod.namespace` (`egret-apps`). Also `image`, `imagePullPolicy`, `resources`, `readyTimeoutSecs`, `idleSecs`, `networkPolicy.enabled` and `run.*`. | @@ -212,6 +216,10 @@ Measured on kind 0.29 with a listener on 8030, 9030, 8080 and 9010: before the p after it, a task pod and a non-Egret pod of the release namespace were refused on all four, the operator's namespace got 8030 and 9030 only, and an `ingressFrom` peer got 8030 and 8080 only. +`NetworkPolicy/egret-ingress` protects Egret's own pods the other way round. It selects the backend pods (`app: egret`) and admits 8080 from every peer (the frontend, ingresses, app and task pods and the kubelet use it), 8081 when `apps.origin` is set, and the OPA port, 8082 by default, **only** from `engineNamespace` and `opa.extraFrom`. The OPA surface reads a request body before it can authenticate it, so a pod that could reach that port could hold stalled requests and exhaust the budget engines need. The policy renders whenever `engineNetworkPolicy.enabled` is set, with or without `opa.enabled`. This describes what the chart renders; it has not been run on a live cluster. + +An external engine on OPA lives outside `engineNamespace`: add its namespace to `opa.extraFrom`, for example `- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: trino-prod } }`. Any firewall between the engines and Egret must allow the OPA port too. When `catalog.public_url` is not Egret's in-cluster address, set `[governance.opa] engine_url` (see [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md#network-exposure)). + The policy needs a network plugin that enforces NetworkPolicy (kind 0.24 and later, Calico, Cilium and most managed clusters do). An external engine is outside it: put it behind your own firewall. diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index 1629d07..14a5afd 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -635,6 +635,28 @@ configuration of its last apply. See [Security & Exposure](../control-plane/query-engines/provision.md#security--exposure) and KI-355 in [Known issues](../known-issues.md). +### The OPA surface moves to its own port (platform #843) + +Egret's OPA routes (`/opa/`) leave the API port 8080 and are served on their own listener, `[governance.opa] listen`, **8082** by +default. Port 8080 now answers 404 for `/opa/`; `/ranger/`, `/oauth/token` and the Iceberg REST catalog stay there. There is a +migration (`0216`, one column, `applied_opa_url`, that records the OPA URL each engine was last applied with). Nothing changes +while `[governance.opa] enabled` is `false`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md#network-exposure). + +- **New keys:** `[governance.opa] listen` and `engine_url`. Egret refuses to start when `listen` has the port of `server.port` or, + with `apps.origin` set, of the apps listener. When `engine_url` is unset, engines get `[catalog] public_url` with the port of + `listen`; **if `public_url` is an ingress hostname rather than Egret's in-cluster address, set `engine_url`**. +- **Provisioned engines re-apply themselves.** At start, with OPA enabled, Egret re-applies each provisioned Trino engine on an OPA + profile whose recorded OPA URL differs from the current one, including engines applied before the upgrade. A failure is logged + per engine as a warning and does not stop the server; re-apply that engine from the console. Expect a coordinator restart for each. +- **External engines re-install their files.** Download the OPA plugin files again (**Control plane → Engines → \ → Plugin + config**, or `GET /api/control-plane/engines/{id}/opa-plugin-config`, which now returns the OPA port's URL), install them and + restart the coordinator. Until then the engine asks port 8080 and fails every query. +- **Firewalls:** allow the OPA port (8082 by default) from the engines to Egret. Anything between them that only allowed 8080 blocks OPA. +- **Helm:** new values `opa.enabled` (default `false`), `opa.port` (`8082`) and `opa.extraFrom`, and a new + `NetworkPolicy/egret-ingress`, rendered when `engineNetworkPolicy.enabled`, that admits the OPA port only from `engineNamespace` + and `opa.extraFrom`. Add the namespace of an external Trino on OPA to `opa.extraFrom`. See + [Kubernetes](./kubernetes.md#engine-network-policy). + ## Further reading - [Backup and restore](./backup-restore) diff --git a/platform/reference/configuration-keys.md b/platform/reference/configuration-keys.md index 8063954..87be7b1 100644 --- a/platform/reference/configuration-keys.md +++ b/platform/reference/configuration-keys.md @@ -156,9 +156,11 @@ The S3-compatible object store used for the catalog warehouse, query results, wo | Key | Environment variable | Default | Meaning | |---|---|---|---| -| `enabled` | `EGRET_GOVERNANCE__OPA__ENABLED` | `false` | Serve the OPA decision routes for Trino and allow a policy profile to switch to OPA. Needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). | +| `enabled` | `EGRET_GOVERNANCE__OPA__ENABLED` | `false` | Serve the OPA decision routes for Trino, on their own listener (`listen`), and allow a policy profile to switch to OPA. Needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). | | `batch_body_limit_bytes` | `EGRET_GOVERNANCE__OPA__BATCH_BODY_LIMIT_BYTES` | `25165824` (24 MiB) | Largest request body the OPA `/batch` route accepts. Trino sends a listing of a whole schema as one batch; a larger body is answered 413 and the query fails. From 4194304 (4 MiB) to 67108864 (64 MiB); every other OPA route keeps 4 MiB. | | `body_read_timeout_ms` | `EGRET_GOVERNANCE__OPA__BODY_READ_TIMEOUT_MS` | `5000` | How long an admitted OPA request may take to send its whole body before it is answered 408 and its connection closed. From 1 to 60000. | +| `listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, `host:port`. It serves only the OPA routes; port 8080 answers 404 for `/opa/`. Egret refuses to start when the port equals `server.port` or, with `apps.origin` set, the apps listener's port. | +| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. | ## engine From c4684d66de3655856a55da2ee0cfc18bca137bbc Mon Sep 17 00:00:00 2001 From: 1Pro-L4R8 Date: Sun, 8 Nov 2026 09:01:24 +0700 Subject: [PATCH 2/4] docs(opa): background re-apply, engine_url fallback, egret-ingress scope Co-Authored-By: Claude Opus 5.5 --- platform/data-governance/engines-and-policy/trino-on-opa.md | 4 ++-- platform/deploy-operate/upgrade.md | 4 ++-- platform/reference/configuration-keys.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/platform/data-governance/engines-and-policy/trino-on-opa.md b/platform/data-governance/engines-and-policy/trino-on-opa.md index fdcbc1b..11e7fdf 100644 --- a/platform/data-governance/engines-and-policy/trino-on-opa.md +++ b/platform/data-governance/engines-and-policy/trino-on-opa.md @@ -234,7 +234,7 @@ The OPA surface reads a request body before it can authenticate it. On a shared | Key | Environment variable | Default | Meaning | |---|---|---|---| | `[governance.opa] listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, as `host:port`. Egret refuses to start when its port equals `server.port` or, when apps are enabled (`apps.origin` is set), the apps listener's port. | -| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL) and replaces its port with the port of `listen`, then trims a trailing slash. | +| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL; `http://egret:8080` when `public_url` is unset, so `http://egret:8082`) and replaces its port with the port of `listen`, then trims a trailing slash. | :::warning **Set `engine_url` when `[catalog] public_url` is not Egret's in-cluster or internal address.** If it is, for example, an ingress hostname, the derived URL points at that hostname on the OPA port and engines cannot reach it. Set `engine_url` to an address engines can reach, such as `http://egret..svc.cluster.local:8082`. @@ -246,7 +246,7 @@ The OPA surface reads a request body before it can authenticate it. On a shared ### Engines that Egret provisions -Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server; re-apply that engine from the console. External engines are not re-applied: see [External engines](#external-engines). +Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies, in the background, each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server. Serving starts at once and the engines converge shortly after; apply a failed engine again with any change that applies it. External engines are not re-applied: see [External engines](#external-engines). ## Limits diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index 14a5afd..80ea14f 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -647,14 +647,14 @@ while `[governance.opa] enabled` is `false`. See [Trino on OPA](../data-governan `listen`; **if `public_url` is an ingress hostname rather than Egret's in-cluster address, set `engine_url`**. - **Provisioned engines re-apply themselves.** At start, with OPA enabled, Egret re-applies each provisioned Trino engine on an OPA profile whose recorded OPA URL differs from the current one, including engines applied before the upgrade. A failure is logged - per engine as a warning and does not stop the server; re-apply that engine from the console. Expect a coordinator restart for each. + per engine as a warning and does not stop the server; apply that engine again with any change that applies it. The pass runs in the background: serving starts at once and the engines converge shortly after. Expect a coordinator restart for each. - **External engines re-install their files.** Download the OPA plugin files again (**Control plane → Engines → \ → Plugin config**, or `GET /api/control-plane/engines/{id}/opa-plugin-config`, which now returns the OPA port's URL), install them and restart the coordinator. Until then the engine asks port 8080 and fails every query. - **Firewalls:** allow the OPA port (8082 by default) from the engines to Egret. Anything between them that only allowed 8080 blocks OPA. - **Helm:** new values `opa.enabled` (default `false`), `opa.port` (`8082`) and `opa.extraFrom`, and a new `NetworkPolicy/egret-ingress`, rendered when `engineNetworkPolicy.enabled`, that admits the OPA port only from `engineNamespace` - and `opa.extraFrom`. Add the namespace of an external Trino on OPA to `opa.extraFrom`. See + and `opa.extraFrom`. With `engineNetworkPolicy.enabled`, `egret-ingress` admits only 8080, 8081 (apps) and the OPA port to the backend pod: anything that reached another port on it is now refused. Add the namespace of an external Trino on OPA to `opa.extraFrom`. See [Kubernetes](./kubernetes.md#engine-network-policy). ## Further reading diff --git a/platform/reference/configuration-keys.md b/platform/reference/configuration-keys.md index 87be7b1..28d544c 100644 --- a/platform/reference/configuration-keys.md +++ b/platform/reference/configuration-keys.md @@ -160,7 +160,7 @@ The S3-compatible object store used for the catalog warehouse, query results, wo | `batch_body_limit_bytes` | `EGRET_GOVERNANCE__OPA__BATCH_BODY_LIMIT_BYTES` | `25165824` (24 MiB) | Largest request body the OPA `/batch` route accepts. Trino sends a listing of a whole schema as one batch; a larger body is answered 413 and the query fails. From 4194304 (4 MiB) to 67108864 (64 MiB); every other OPA route keeps 4 MiB. | | `body_read_timeout_ms` | `EGRET_GOVERNANCE__OPA__BODY_READ_TIMEOUT_MS` | `5000` | How long an admitted OPA request may take to send its whole body before it is answered 408 and its connection closed. From 1 to 60000. | | `listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, `host:port`. It serves only the OPA routes; port 8080 answers 404 for `/opa/`. Egret refuses to start when the port equals `server.port` or, with `apps.origin` set, the apps listener's port. | -| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. | +| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL; `http://egret:8080` if unset) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. | ## engine From 8909278f60a1b89433769b6a9864c22e34e51519 Mon Sep 17 00:00:00 2001 From: 1Pro-L4R8 Date: Sun, 8 Nov 2026 09:22:48 +0700 Subject: [PATCH 3/4] docs(opa): state the upgrade outage on OPA engines; OPA-off checks Queries on provisioned OPA engines are refused from the moment the new egret serves until each coordinator is re-applied and restarted, also during a rolling update with old and new pods serving; recommend a maintenance window. The listen/engine_url checks apply only with OPA enabled, and engine_url must be an absolute http(s) URL. Co-Authored-By: Claude Opus 5.5 --- .../engines-and-policy/trino-on-opa.md | 10 +++++++--- platform/deploy-operate/upgrade.md | 16 ++++++++++++---- platform/reference/configuration-keys.md | 4 ++-- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/platform/data-governance/engines-and-policy/trino-on-opa.md b/platform/data-governance/engines-and-policy/trino-on-opa.md index 11e7fdf..530e642 100644 --- a/platform/data-governance/engines-and-policy/trino-on-opa.md +++ b/platform/data-governance/engines-and-policy/trino-on-opa.md @@ -233,8 +233,8 @@ The OPA surface reads a request body before it can authenticate it. On a shared | Key | Environment variable | Default | Meaning | |---|---|---|---| -| `[governance.opa] listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, as `host:port`. Egret refuses to start when its port equals `server.port` or, when apps are enabled (`apps.origin` is set), the apps listener's port. | -| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL; `http://egret:8080` when `public_url` is unset, so `http://egret:8082`) and replaces its port with the port of `listen`, then trims a trailing slash. | +| `[governance.opa] listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, as `host:port`. With OPA enabled, Egret refuses to start when its port equals `server.port` or, when apps are enabled (`apps.origin` is set), the apps listener's port. With OPA off it is not checked, because nothing binds it. | +| `[governance.opa] engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | The base URL engines use for OPA. Unset, Egret takes `[catalog] public_url` (or the Docker callback URL; `http://egret:8080` when `public_url` is unset, so `http://egret:8082`) and replaces its port with the port of `listen`, then trims a trailing slash. With OPA enabled, a value that is not an absolute `http(s)` URL with a host stops Egret at start. | :::warning **Set `engine_url` when `[catalog] public_url` is not Egret's in-cluster or internal address.** If it is, for example, an ingress hostname, the derived URL points at that hostname on the OPA port and engines cannot reach it. Set `engine_url` to an address engines can reach, such as `http://egret..svc.cluster.local:8082`. @@ -246,7 +246,11 @@ The OPA surface reads a request body before it can authenticate it. On a shared ### Engines that Egret provisions -Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies, in the background, each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server. Serving starts at once and the engines converge shortly after; apply a failed engine again with any change that applies it. External engines are not re-applied: see [External engines](#external-engines). +Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wrote into its plugin files. When Egret starts with OPA enabled, it re-applies, in the background, each provisioned Trino engine on an OPA profile whose recorded URL differs from the current one, including engines applied before the upgrade (their value is empty). A failure is logged per engine as a warning and does not stop the server; apply a failed engine again with any change that applies it. The pass runs after the new Egret starts serving, and each re-applied coordinator restarts. + +:::warning +**Queries on these engines fail until each one is re-applied and restarted.** Until then an engine asks port 8080 for `/opa/`, which answers 404. During a rolling update an engine already moved to the OPA port is also refused whenever its request reaches an old Egret pod, which has no OPA listener. Upgrade in a maintenance window, or when the query load on OPA engines is low. See [Upgrade](../../deploy-operate/upgrade.md). +::: External engines are not re-applied: see [External engines](#external-engines). ## Limits diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index 80ea14f..ef5284a 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -642,15 +642,23 @@ default. Port 8080 now answers 404 for `/opa/`; `/ranger/`, `/oauth/token` and t migration (`0216`, one column, `applied_opa_url`, that records the OPA URL each engine was last applied with). Nothing changes while `[governance.opa] enabled` is `false`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md#network-exposure). -- **New keys:** `[governance.opa] listen` and `engine_url`. Egret refuses to start when `listen` has the port of `server.port` or, - with `apps.origin` set, of the apps listener. When `engine_url` is unset, engines get `[catalog] public_url` with the port of +- **New keys:** `[governance.opa] listen` and `engine_url`, checked only while `[governance.opa] enabled` is `true`. Egret then + refuses to start when `listen` has the port of `server.port` or, with `apps.origin` set, of the apps listener, or when + `engine_url` is set but is not an absolute `http(s)` URL with a host. When `engine_url` is unset, engines get `[catalog] public_url` with the port of `listen`; **if `public_url` is an ingress hostname rather than Egret's in-cluster address, set `engine_url`**. - **Provisioned engines re-apply themselves.** At start, with OPA enabled, Egret re-applies each provisioned Trino engine on an OPA profile whose recorded OPA URL differs from the current one, including engines applied before the upgrade. A failure is logged - per engine as a warning and does not stop the server; apply that engine again with any change that applies it. The pass runs in the background: serving starts at once and the engines converge shortly after. Expect a coordinator restart for each. + per engine as a warning and does not stop the server; apply that engine again with any change that applies it. The pass runs in + the background after the new Egret starts serving, and each re-applied coordinator restarts. +- **Queries on provisioned OPA engines fail during the upgrade.** From the moment the new Egret serves until an engine has been + re-applied and its coordinator has restarted, that engine still asks port 8080 for `/opa/`, which now answers 404, so its queries + are refused. During a rolling update, old and new Egret pods serve side by side: an engine already re-applied to the OPA port is + also refused whenever its request reaches an old pod, which has no OPA listener. On Docker the window includes the Egret container + recreate. How long this lasts depends on the number of engines and on how fast each coordinator restarts. **Upgrade in a + maintenance window, or when the query load on OPA engines is low.** Engines on the Ranger mechanism are not affected. - **External engines re-install their files.** Download the OPA plugin files again (**Control plane → Engines → \ → Plugin config**, or `GET /api/control-plane/engines/{id}/opa-plugin-config`, which now returns the OPA port's URL), install them and - restart the coordinator. Until then the engine asks port 8080 and fails every query. + restart the coordinator. From the upgrade until then the engine asks port 8080 and fails every query. - **Firewalls:** allow the OPA port (8082 by default) from the engines to Egret. Anything between them that only allowed 8080 blocks OPA. - **Helm:** new values `opa.enabled` (default `false`), `opa.port` (`8082`) and `opa.extraFrom`, and a new `NetworkPolicy/egret-ingress`, rendered when `engineNetworkPolicy.enabled`, that admits the OPA port only from `engineNamespace` diff --git a/platform/reference/configuration-keys.md b/platform/reference/configuration-keys.md index 28d544c..fc488a2 100644 --- a/platform/reference/configuration-keys.md +++ b/platform/reference/configuration-keys.md @@ -159,8 +159,8 @@ The S3-compatible object store used for the catalog warehouse, query results, wo | `enabled` | `EGRET_GOVERNANCE__OPA__ENABLED` | `false` | Serve the OPA decision routes for Trino, on their own listener (`listen`), and allow a policy profile to switch to OPA. Needs `ranger.enabled`. See [Trino on OPA](../data-governance/engines-and-policy/trino-on-opa.md). | | `batch_body_limit_bytes` | `EGRET_GOVERNANCE__OPA__BATCH_BODY_LIMIT_BYTES` | `25165824` (24 MiB) | Largest request body the OPA `/batch` route accepts. Trino sends a listing of a whole schema as one batch; a larger body is answered 413 and the query fails. From 4194304 (4 MiB) to 67108864 (64 MiB); every other OPA route keeps 4 MiB. | | `body_read_timeout_ms` | `EGRET_GOVERNANCE__OPA__BODY_READ_TIMEOUT_MS` | `5000` | How long an admitted OPA request may take to send its whole body before it is answered 408 and its connection closed. From 1 to 60000. | -| `listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, `host:port`. It serves only the OPA routes; port 8080 answers 404 for `/opa/`. Egret refuses to start when the port equals `server.port` or, with `apps.origin` set, the apps listener's port. | -| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL; `http://egret:8080` if unset) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. | +| `listen` | `EGRET_GOVERNANCE__OPA__LISTEN` | `0.0.0.0:8082` | Where the OPA listener binds, `host:port`. It serves only the OPA routes; port 8080 answers 404 for `/opa/`. With `enabled`, Egret refuses to start when the port equals `server.port` or, with `apps.origin` set, the apps listener's port; with OPA off it is not checked. | +| `engine_url` | `EGRET_GOVERNANCE__OPA__ENGINE_URL` | unset | Base URL engines use for OPA. Unset: `catalog.public_url` (or the Docker callback URL; `http://egret:8080` if unset) with its port replaced by the port of `listen`. Set it when `public_url` is not Egret's in-cluster address, for example an ingress hostname. With `enabled`, it must be an absolute `http(s)` URL with a host. | ## engine From 9aac85dc464bbaa13a79e2a79e5123921bfca581 Mon Sep 17 00:00:00 2001 From: 1Pro-L4R8 Date: Sun, 8 Nov 2026 09:37:16 +0700 Subject: [PATCH 4/4] docs(opa): close the upgrade warning on its own line Co-Authored-By: Claude Opus 5.5 --- platform/data-governance/engines-and-policy/trino-on-opa.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/platform/data-governance/engines-and-policy/trino-on-opa.md b/platform/data-governance/engines-and-policy/trino-on-opa.md index 530e642..eed44fc 100644 --- a/platform/data-governance/engines-and-policy/trino-on-opa.md +++ b/platform/data-governance/engines-and-policy/trino-on-opa.md @@ -250,7 +250,9 @@ Migration 0216 adds `applied_opa_url` to every engine: the OPA URL Egret last wr :::warning **Queries on these engines fail until each one is re-applied and restarted.** Until then an engine asks port 8080 for `/opa/`, which answers 404. During a rolling update an engine already moved to the OPA port is also refused whenever its request reaches an old Egret pod, which has no OPA listener. Upgrade in a maintenance window, or when the query load on OPA engines is low. See [Upgrade](../../deploy-operate/upgrade.md). -::: External engines are not re-applied: see [External engines](#external-engines). +::: + +External engines are not re-applied: see [External engines](#external-engines). ## Limits