Skip to content

feat(opa): serve the OPA surface on its own engines-only listener - #843

Merged
viethqb merged 12 commits into
mainfrom
feat/opa-listener
Nov 8, 2026
Merged

viethqb merged 12 commits into
mainfrom
feat/opa-listener

Conversation

@viethqb

@viethqb viethqb commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Summary

This is port 8080 hardening, PR B: egret's OPA decision surface gets its own listener, which only engines can reach. It is the second prerequisite for SP2b Task 26 (OPA on by default). PR A was #842.

The problem. /opa/ was served on egret's API port 8080.

  • That port is shared with app and task pods, which the Helm chart lets reach it.
  • The surface must read the request body before it can authenticate it, because the credential is in the body.
  • So any pod that can reach 8080 can open connections, declare a large Content-Length and stall. That exhausts the shared in-flight budget, and every engine gets 503.
  • A per-peer limit was rejected, because an ingress collapses peers into one address.

Spec: docs/en/specs/2026-11-07-port-8080-hardening-design.md, Part B (decisions Q2 and Q3). Plan: Tasks B1–B7.

Changes

  • Config
    • [governance.opa] listen (default 0.0.0.0:8082) and engine_url.
    • engine_url defaults to [catalog] public_url (or http://egret:8080) with the port swapped to the listener's port.
    • While OPA is on, egret refuses to start if the port collides with the API or apps listener, if listen has no port, or if engine_url is not an absolute http(s) URL. With OPA off, nothing changes.
  • Listener
    • egret_server::opa_router serves the surface only on that listener.
    • Port 8080 answers 404 for /opa/. /ranger/, /oauth/token and the Iceberg catalog stay on 8080.
    • The body limits, the in-flight and read-timeout gate, and refuse_same_site move with the surface.
    • On shutdown, the OPA listener and the apps listener now drain within the same DRAIN_DEADLINE as 8080. They used to be spawn-and-forget.
  • Engine URL
    • CpState and the policy admin state carry engine_opa_url.
    • ranger_binding, the opa-plugin-config route and the policy admin trino-config give OPA-profile engines the new URL.
    • Ranger engines keep 8080; a test pins this.
  • Migration 0216: applied_opa_url
    • Every successful apply records the URL rendered for a Trino engine on OPA, and NULL otherwise.
    • A failure to record it keeps the apply successful and names the engine in the "not recorded" note.
    • At start, with OPA on, reapply_stale_opa_engines runs in the background and re-applies provisioned Trino engines on an OPA profile whose recorded URL differs, NULL included.
  • Helm
    • opa.enabled, opa.port and opa.extraFrom; an opa port on the Deployment and the Service.
    • networkpolicy-egret.yaml (egret-ingress), rendered with engineNetworkPolicy.enabled: 8080 is open to all, 8081 is open when apps are on, and the OPA port admits only engineNamespace plus opa.extraFrom.
  • opa-e2e
    • egret listens on 8082.
    • opa_port_check (run by layer2.sh in CI) asserts that the rendered opa.policy.uri names :8082/opa/ and that POST egret:8080/opa/... answers exactly 404.

Upgrade impact (breaking for OPA engines)

  • Queries on provisioned OPA engines are refused from the moment the new egret serves until each engine is re-applied and its coordinator restarted. The start-up pass does both, in the background.
  • During a rolling update, old and new egret pods serve at the same time, and both can refuse queries:
    • an engine not yet re-applied still asks 8080 for /opa/, so the new pods answer 404;
    • an engine already re-applied asks the OPA port, which the old pods don't have.
  • External engines fail until their plugin files are re-installed.
  • The docs recommend a maintenance window. Installs without OPA, and Ranger engines, are unaffected.

Test plan

  • Unit and DB suites, each new behaviour with a test that fails without it and a mutant killed by a failing assertion:
    • egret-config 90, egret-server 77 (including a real-socket drain test), egret-engine-mgmt 514, egret-bin 112, egret-policy (admin_routes 55).
  • Helm: helm_opa_listener 5/5, plus helm template with OPA on and off, and helm lint.
  • fmt and clippy -D warnings are clean.
  • CI on the stacked draft (head 5d240bf): all jobs green, including opa-e2e with the new port check. The frontend job needed one rerun for a flake that this PR doesn't touch: a focusArrival.ts timer after jsdom teardown.
  • Reviews: a task review after every task, a final whole-branch review (opus), one fix wave and a re-review. The fix wave covered the fmt failure, OPA-off installs, the listener drain and the upgrade docs.

Known follow-ups (not in this PR)

  • A dead OPA listener leaves the pod Ready.
  • The Helm chart has no render-time guard for opa.enabled without Ranger, or for an opa.port clash.
  • The derived engine_url keeps public_url's scheme and path.
  • focusArrival timer flake in the frontend tests.

Documentation

egret-data/docs#70.

Checklist

  • Linked to an issue (no tracking issue)
  • Tests added or updated
  • Docs updated in egret-data/docs
  • Commit messages follow Conventional Commits
  • No secrets, tokens or real data added

🤖 Generated with Claude Code

viethqb and others added 12 commits November 8, 2026 10:32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing branches

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_router no longer merges the OPA surface; egret_server::opa_router
returns it for a dedicated listener bound from [governance.opa] listen,
drained on shutdown like the apps host. Also repairs opa_start.rs, which
B1's new OpaSettings fields had left uncompilable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engines on an OPA profile are now configured with the OPA listener's URL
(governance.opa.engine_url, or the engine URL with the listener's port),
while Ranger and catalog URLs stay on the main port. Start fails when the
OPA URL cannot be derived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An install that never enables [governance.opa] no longer has its
server.port / apps.listen checked against the OPA listen port, its listen
format checked, or its start aborted by an underivable engine OPA URL
(the URL then falls back to the engine egret URL). While OPA is on, an
explicit engine_url must be an absolute http(s) URL with a host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OPA listener's task handle was dropped, so at SIGTERM the process
exited as soon as the API listener drained, killing in-flight OPA
decisions (each a failed Trino query). main now keeps the handles of the
side listeners and waits for them after the API drain, bounded by
DRAIN_DEADLINE. The apps listener had the same flaw and shares the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@viethqb
viethqb changed the base branch from feat/port-8080-hardening to main November 8, 2026 03:42
@viethqb
viethqb marked this pull request as ready for review November 8, 2026 03:42
@viethqb viethqb closed this Nov 8, 2026
@viethqb viethqb reopened this Nov 8, 2026
@viethqb
viethqb merged commit 1d483e0 into main Nov 8, 2026
14 of 15 checks passed
@viethqb
viethqb deleted the feat/opa-listener branch November 8, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant