diff --git a/platform/control-plane/query-engines/engine-detail.md b/platform/control-plane/query-engines/engine-detail.md index 782bca7..35ca26a 100644 --- a/platform/control-plane/query-engines/engine-detail.md +++ b/platform/control-plane/query-engines/engine-detail.md @@ -10,7 +10,7 @@ Select an engine in **Control Plane → Engines** to open its page. The tabs dep ![The Overview tab of a provisioned engine on the Docker demo stack: phase running, one container, its catalogs and workspaces](/img/platform/admin/cp-engine-overview.png) -The header shows the engine name, its namespace (`egret-engines`), the image, the endpoint, the authentication (*JWT auth*) and the Ranger plugin version, with the type and status badges. The endpoint is the `host:port` a workspace connection to the engine dials, for example `docs-trino-trino.egret-engines.svc:8443` (port 8443 is the HTTPS listener of an engine with JWT delegation). **Open in Engine & policy** opens the engine in Governance, **Restart** restarts the engine, and the **More engine actions** menu (**⋯**) holds **Delete engine**. The tabs are **Overview**, **Catalogs**, **Workspaces**, **Settings** and **Monitor**. +The header shows the engine name, its namespace (`egret-engines`), the image, the endpoint, the authentication (*JWT auth*) and the Ranger plugin version, with the type and status badges. The endpoint is the `host:port` a workspace connection to the engine dials, for example `docs-trino-trino.egret-engines.svc:8443` (port 8443 is the HTTPS listener of an engine with JWT delegation). **Open in Engine & policy** opens the engine in Governance, **Restart** restarts the engine, and the **More engine actions** menu (**⋯**) holds **Delete engine**. The tabs are **Overview**, **Catalogs**, **Workspaces**, **Settings** and **Monitor**. A provisioned StarRocks engine has one more tab after Monitor, [**Query log**](#query-log). A provisioned StarRocks engine that a policy profile names shows, above the tabs, the warning that unless the engine runs `egret-starrocks-fe` `4.1.7-egret.4` or later its StarRocks lets users past the profile: official images serve the Iceberg metadata tables without asking Ranger and apply no mask or row filter to prepared statements, and official and earlier patched images apply none through the HTTP SQL API or Arrow Flight SQL, let anyone read object storage through `FILES()` and let a masked or filtered user store a table's rows in the clear in a materialized view, and images before `4.1.7-egret.4` show a filtered user the statistics of hidden rows through `EXPLAIN COSTS` and let anyone reach object storage with the cluster's own identity through `BROKER LOAD`, `EXPORT` and `SELECT INTO OUTFILE` without keys (since platform #822, widened in #826, #827 and #832; see [External engine](#overview-1), KI-340, KI-345 to KI-348, KI-350, KI-351). It is not shown while the profile list loads, or to a viewer who cannot read it. It is not shown for a provisioned engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.4` or later (the default for new engines since platform #832), which fixes all of them; it is shown for an engine still on `4.1.7-egret.1` or `4.1.7-egret.3`; an external engine always shows it, because Egret does not know its image. @@ -116,6 +116,75 @@ How Egret signs in to the coordinator depends on the engine: The node list comes from Trino's `GET /v1/node` where the coordinator serves it. Trino 483 does not by default, and then Egret reads `system.runtime.nodes` instead. That query runs as the same sign-in, so on a JWT engine it appears in the **Queries** table under your name (on the demo stack: `SELECT http_uri FROM system.runtime.nodes`, user `admin`, **FINISHED**). If it is refused too, **nodes** shows `—`: the count is unknown, not zero, and the query counts are still shown. +### Query log + +A provisioned StarRocks engine shows a **Query log** tab after Monitor (`?tab=query-log`). It lists the statements the engine ran, from every way in: the MySQL protocol, prepared statements, the HTTP SQL API, Arrow Flight SQL and materialized view refreshes. A Trino engine and an external StarRocks engine have no such tab: egret does not install anything on an engine it does not run, and Trino's **Monitor** shows live queries only. Anyone with `engine:read` sees the tab; the statement text and the error text need `engine:audit` (see [Who sees the statement](#who-sees-the-statement)). + +![The Query log tab of a StarRocks engine: the status line, the filters and the statements, newest first](/img/platform/admin/cp-engine-query-log.png) + +Above the table a status line says what state the log is in. + +| State | API value | What the tab shows | +|---|---|---| +| Ready | `ready` | *Kept 30 days. Latest event: \.* When nothing has been logged yet: *No events yet. Events reach the log about a minute after a statement runs.* | +| Not set up | `notSetUp` | A warning: *The query log is not set up on this engine: its audit plugin or its log table is missing.* with a **Set up query log** button. | +| Plugin not shipped | `pluginNotShipped` | A warning that the engine's image has no audit plugin (see [Engines on an older image](#engines-on-an-older-image)). | +| Unreachable | `unreachable` | The engine's message, with a **Retry** button. | + +The state is read from the engine each time the tab opens. Nothing about it is stored in egret. + +#### Filters and columns + +The filters sit above the table and stay in the page address, so a link shares them: + +- **Time range**: **Last hour**, **Last 24 hours** (the default), **Last 7 days**, **Last 30 days** or **Custom** with **From** and **To**. A range is at most 31 days. +- **User**: the person's name, exact and case-insensitive. It matches either of the two user columns below. +- **State**: **All**, **Succeeded** or **Failed**. + +The table is newest first, 50 rows at a time; **Load more** fetches the next page. The columns are **Time** (your local time; hover for the UTC instant), **User**, **Connected as**, **State**, **Command**, **Duration**, **Rows returned** and **Statement**, which shows the first line of the statement. + +**User** and **Connected as** differ for a delegated query. egret's engine queries on behalf of a person log in to StarRocks as a service user and then run `EXECUTE AS `. StarRocks records both. **User** is the person the statement ran for, so you read who asked first. **Connected as** is the session user and appears only when it differs, for example the service user behind a Data API endpoint or a workspace query. For a person who connects to StarRocks directly there is one name, and **Connected as** is empty. + +**State** is **Succeeded** or **Failed**. A statement StarRocks or its policies refused, such as a column a Ranger policy denies, is **Failed**. The audit row stores only an error *code*, not the message, so the **Error** field of the side panel shows a code such as `INTERNAL_ERR` or `ANALYSIS_ERR` (for `EXECUTE AS` a user that does not exist). A Ranger deny shows as `INTERNAL_ERR`, not as *Access denied*. + +Select a row to open a side panel with the whole statement (up to 1 MiB; the table cuts it at 2,000 characters), the error and the counters: catalog, database, client address, query source, rows and bytes scanned, rows returned, CPU time and memory. + +![The side panel of a statement with its error, counters and the full text](/img/platform/admin/cp-engine-query-log-panel.png) + +egret's own reads and probes, which run as `root` and carry the marker comment `/* egret-query-log-read */`, are left out of the table and of **Latest event**. Every other statement of the `root` user, egret's own set-up statements included, is shown. + +#### Who sees the statement + +The statement text can contain the values a person filtered on and the mask and row filter expressions of the policies that applied, and an error can refer to the statement that caused it. So **both the statement and the error need the `engine:audit` permission** ("Read statement text in engine query logs"). Without it the tab shows every other column, the state included, so a failed statement still reads as failed, and the statement and the error show *Hidden: needs engine:audit*. Migration 0218 grants `engine:audit` to every platform role that already holds `engine:update`. Grant it to other roles under **Control Plane → Roles**; see [Permissions](../../reference/permissions.md). What the log redacts is listed under [Security of a StarRocks engine](../../deploy-operate/engines-and-ranger.md#the-query-log-of-a-starrocks-engine). + +#### Set-up and retention + +The log is kept for **30 days**. StarRocks drops older partitions of the log table by itself. + +The set-up runs on its own, so most engines never need the button: + +- when a StarRocks engine is created. The setup runs after the delegation setup and waits for an alive compute node before it creates the table, so on a new engine the tab becomes ready about when the engine is **Running**; +- after each apply or upgrade that may restart the engine, including **Upgrade security**; +- once when egret starts, for every reachable provisioned StarRocks engine, so engines that existed before the Query log get it. + +**Set up query log** (needs `engine:update`) runs the same steps once, now. Use it to recover from a failed or lost set-up. If the plugin ever ends up in a state other than `INSTALLED`, the tab shows **Not set up** until the next set-up reinstalls it. Without the permission the button is disabled and says *Setting it up needs the engine:update permission.* The steps are safe to repeat. An error is shown as a message and logged; a failed set-up never fails a create, an apply or an upgrade. + +:::note Events that are not recorded +StarRocks hands events to its audit plugins only about a minute after the plugin is installed, and again about once a minute. Statements that ran before the plugin's first refresh are not recorded, and after that a statement shows in the log about a minute after it ran. + +The plugin survives an FE pod restart: the FE extracts it again from the image, and nothing needs to be set up. Roughly 2 minutes of events around a restart are not recorded, those queued in the old FE and those sent before the new plugin's first load. +::: + +![The Query log tab of an engine whose log is not set up, with the warning and the Set up query log button](/img/platform/admin/cp-engine-query-log-not-set-up.png) + +#### Engines on an older image + +Only egret's patched StarRocks image ships the audit plugin, from `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.4`. An engine on an official image or on `4.1.7-egret.1` to `4.1.7-egret.3` shows `pluginNotShipped`. egret still creates the log table and the loader user, and does not fail the apply. When the engine carries the **Security upgrade needed** warning, the tab says: *This engine's image has no audit plugin. **Upgrade security**, in the security warning on this page, moves it to egret's image, and the Query log starts after that.* Press [**Upgrade security**](#security-upgrade-needed); the set-up then runs again by itself and installs the plugin. Without that warning the tab says *The Query log starts once the engine runs egret's image.* + +![The Query log tab of an engine on an older image, with the warning that its image has no audit plugin](/img/platform/admin/cp-engine-query-log-plugin-not-shipped.png) + +The tab is also served by the [Engines API](../../reference/rest-api/control-plane-engines.md#get-apicontrol-planeenginesidquery-log). + ### Restart **Restart** asks *Restart engine?* (*This rolls the coordinator and worker pods; running queries are interrupted.* On the Docker provisioner: *This restarts the coordinator and worker containers; running queries are interrupted.*). For a Trino engine it adds *If the pods (containers) are gone, they are created again.* On the demo stack the status went to **Provisioning** and back to **Running** in 38 seconds, and the container was recreated. diff --git a/platform/deploy-operate/engines-and-ranger.md b/platform/deploy-operate/engines-and-ranger.md index e947426..3dade84 100644 --- a/platform/deploy-operate/engines-and-ranger.md +++ b/platform/deploy-operate/engines-and-ranger.md @@ -286,6 +286,32 @@ that cost time otherwise: the table and a mask per column). A schema change to such a table through egret's Iceberg catalog recompiles the profile at once. Hand-written tag policies are not enforced on StarRocks. +### The query log of a StarRocks engine + +A provisioned StarRocks engine on egret's patched image records every statement it runs (see [Query log](../control-plane/query-engines/engine-detail.md#query-log)). The pieces, so you know what egret adds to the engine: + +- **The audit database.** egret creates `starrocks_audit_db__.starrocks_audit_tbl__` on the engine. The StarRocks AuditLoader plugin, which the image ships at `/opt/egret/auditloader.zip`, writes one row per statement. The table keeps 30 days. +- **The loader user.** The plugin loads as the StarRocks user `egret_query_log`, which holds `INSERT` on that table and nothing else. Its password is a secret egret generates for the engine; egret never rotates it. The name is reserved: no person can be given it, and if a human already holds it the set-up stops and says so. On an engine bound to no policy profile egret grants `INSERT` natively; on a bound engine, which runs `access_control = ranger`, the next rows allow it. +- **The Ranger rows.** Two infrastructure rows on the engine's StarRocks profile: `egret-query-log-loader` allows `egret_query_log` to `insert` into the table, and the `egret-query-log-guard - …` rows (one per resource level under a database) deny every user all access to the audit database. They are egret's own, appear in the profile's policy list, and cannot be created, renamed or edited by hand. +- **The guard denies everyone.** That includes engine admins, whose `egret-admin - …` rows allow everything: a Ranger deny beats an allow. Nobody reads the audit database by SQL on a governed engine; only the loader user may insert. egret reads it itself as `root`, which Ranger does not check, and gates what it shows behind `engine:read` and `engine:audit`. An engine bound to no profile has no Ranger: its users have no privilege on a database they did not create unless someone grants one natively, which egret does not do. +- **No Ranger audit.** The Ranger plugin's own audit stays off (`xasecure.audit.is.enabled=false`). The Query log is the only audit surface. +- **Refused `feConfig` keys.** `plugin_enable` and `plugin_dir` are refused in an engine's `feConfig`, so an override cannot switch the audit plugin off or move its directory. + +:::warning Statements can show data +A statement run through `PREPARE` and `EXECUTE` or the MySQL binary protocol is recorded as the FE ran it: with its parameter values inlined and with the mask and row filter expressions of the policies that applied. So the log can show the values a person filtered on and the rules that restricted them. An error can refer to the statement too. That is why the statement and the error need `engine:audit`, which only roles that can update engines hold by default. Grant it sparingly. +::: + +**What is redacted.** StarRocks replaces the secrets in `FILES()` and in catalog properties with `***` before it records the event, and it does the same for passwords in `CREATE USER … IDENTIFIED BY`, `ALTER USER … IDENTIFIED BY`, `SET PASSWORD` and `INSTALL PLUGIN` (measured on `4.1.7-egret.4`). egret redacts a second time before text leaves the server, in the statement and the error: + +- the engine's `root` password and the loader password, wherever either appears, become `***`; +- the value of `IDENTIFIED BY '…'`, and of a quoted property pair whose key contains `password`, `passwd`, `secret`, `access_key` or `secret_key`, becomes `'***'`. + +Nothing else is redacted. A literal in a `WHERE` clause or in an `INSERT` is shown as it was sent. + +**Errors are codes.** The audit row stores only an error code, so the Query log shows `INTERNAL_ERR` or `ANALYSIS_ERR`, not the engine's message. A Ranger deny shows as `INTERNAL_ERR`. + +**Known limit: the row count is readable.** The guard denies reading the statement and other columns, `insert`, `drop` and the rest. StarRocks checks `SELECT` per column, and `SELECT count(*) FROM starrocks_audit_db__.starrocks_audit_tbl__` references none, so it is not blocked: anyone who can connect to the engine can learn how many statements the log holds, but not their contents. This is a StarRocks and Ranger property; closing it is a follow-up. + ### Verify the plugin is polling On the engine, the plugin logs each poll (StarRocks: `fe.log`, `RangerAdminRESTClient.getServicePoliciesIfUpdated`). diff --git a/platform/deploy-operate/security-hardening.md b/platform/deploy-operate/security-hardening.md index 1dd6aa2..9f00130 100644 --- a/platform/deploy-operate/security-hardening.md +++ b/platform/deploy-operate/security-hardening.md @@ -113,6 +113,7 @@ Expose only what clients need. - [ ] **Remember StarRocks' Ranger plugin enforces resource-based policies only.** Egret's rules and clearances by classification reach it as resource policies; hand-written tag policies do not. - [ ] **Re-check `access_control = ranger` after StarRocks container recreation.** It is lost silently. +- [ ] **Grant `engine:audit` sparingly.** It reveals the statement and error text of a StarRocks engine's [Query log](../control-plane/query-engines/engine-detail.md#query-log), which can include the values people filtered on and the mask and row filter expressions that applied to them. Only roles that hold `engine:update` get it by default. ## 7. Single sign-on diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index f894d88..c6fedcb 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -686,6 +686,19 @@ Nothing changes on its own when you upgrade: no migration rewrites an engine's i - Look for **Not governed** too: with Ranger on, a StarRocks engine with no policy profile enforces none of egret's policies. Bind a profile in its settings. - **API change:** the engine adds `notGoverned`, `securityUpgradeNeeded`, `securityUpgradeReasons`, `securityUpgradeFeImage` and `securityUpgradeBlockedRelease`, and a new route `POST /api/control-plane/engines/{id}/security-upgrade`. See [Engines API](../reference/rest-api/control-plane-engines.md). +### StarRocks Query log (migration 0218) + +A provisioned StarRocks engine gets a [Query log](../control-plane/query-engines/engine-detail.md#query-log) tab. Migration 0218 adds the column that holds the loader user's secret, reserves the user name `egret_query_log` so no person can be given it, and adds the permission `engine:audit`. + +- **Existing engines set themselves up.** When egret starts, it sets the Query log up once on every reachable provisioned StarRocks engine. It creates the audit database and table, the loader user and, on an engine whose image ships it, installs the plugin; the engine does not restart. Nothing runs on an engine that is not reachable at that moment: open its **Query log** tab and press **Set up query log** (`engine:update`), or `POST /api/control-plane/engines/{id}/query-log/setup`. +- **Engines on an image before `4.1.7-egret.4` need Upgrade security first.** Only that image ships the audit plugin. Such an engine shows `pluginNotShipped` on the tab. Press **Upgrade security** (see the [posture note above](#starrocks-engine-posture-platform-847)); the set-up then runs again by itself, and the Query log starts after the FE and CN pods are back. +- **Events from before the plugin's first refresh are not recorded.** StarRocks hands events to the plugin about a minute after it is installed, so a new Query log starts a minute or two after the set-up. +- **Who can read statements changes.** `engine:audit` is granted to every platform role that holds `engine:update`. Roles that only hold `engine:read` see the tab without the statement and error text. Grant `engine:audit` to other roles under **Control Plane → Roles** if they need it. +- **Direct SQL on the audit database is refused to everyone,** engine admins included, on an engine bound to a Ranger profile. Egret adds the guard rows to the profile on its own; they cannot be edited. +- **`plugin_enable` and `plugin_dir` are refused in `feConfig`.** An engine whose `feConfig` already overrides either key gets a 400 on its next update until you remove those keys. +- **API change:** four routes under `/api/control-plane/engines/{id}/query-log`. See [Engines API](../reference/rest-api/control-plane-engines.md#get-apicontrol-planeenginesidquery-log). +- Retention is 30 days, and there is no export and no rotation of the loader password. + ## Further reading - [Backup and restore](./backup-restore) diff --git a/platform/reference/permissions.md b/platform/reference/permissions.md index 6f20f82..a5b7222 100644 --- a/platform/reference/permissions.md +++ b/platform/reference/permissions.md @@ -58,6 +58,7 @@ The last column is what a fresh install grants; your roles may differ. | `engine:delete` | platform | Delete engines | none (only the `platform_admin` bypass) | | `engine:provision` | platform | Provision / restart engines on the cluster | none (only the `platform_admin` bypass) | | `engine:kill` | platform | Kill running queries | none (only the `platform_admin` bypass) | +| `engine:audit` | platform | Read statement text in engine query logs | none (only the `platform_admin` bypass); migration 0218 also grants it to every platform role that holds `engine:update` | ### Catalogs diff --git a/platform/reference/rest-api/control-plane-engines.md b/platform/reference/rest-api/control-plane-engines.md index dcbed2c..a9c5cf7 100644 --- a/platform/reference/rest-api/control-plane-engines.md +++ b/platform/reference/rest-api/control-plane-engines.md @@ -8,7 +8,7 @@ description: REST endpoints in the control-plane-engines group. Engine provisioning: Trino-on-Kubernetes clusters, their catalogs, native query monitor, and pod logs/exec. -This page lists 28 endpoints (OpenAPI tag `control-plane-engines`). It is generated from `openapi.json`; see the [REST API overview](../rest-api) for authentication, CSRF and error conventions. +This page lists 32 endpoints (OpenAPI tag `control-plane-engines`). It is generated from `openapi.json`; see the [REST API overview](../rest-api) for authentication, CSRF and error conventions. | Method | Path | Operation id | Notes | |---|---|---|---| @@ -32,6 +32,10 @@ This page lists 28 endpoints (OpenAPI tag `control-plane-engines`). It is genera | GET | `/api/control-plane/engines/{id}/queries` | `list_queries` | | | GET | `/api/control-plane/engines/{id}/queries/{qid}` | `get_query` | | | DELETE | `/api/control-plane/engines/{id}/queries/{qid}` | `kill_query` | | +| GET | `/api/control-plane/engines/{id}/query-log` | `list_query_log` | | +| POST | `/api/control-plane/engines/{id}/query-log/setup` | `setup_query_log` | | +| GET | `/api/control-plane/engines/{id}/query-log/status` | `query_log_status` | | +| GET | `/api/control-plane/engines/{id}/query-log/{queryId}` | `get_query_log_entry` | | | GET | `/api/control-plane/engines/{id}/ranger-plugin-config` | `ranger_plugin_config` | `GET /api/control-plane/engines/\{id\}/ranger-plugin-config`. | | POST | `/api/control-plane/engines/{id}/restart` | `restart_engine` | `POST /engines/\{id\}/restart` — roll the engine's pods without changing config (like `kubectl rollout restart`). | | POST | `/api/control-plane/engines/{id}/retire-previous-ranger-credential` | `retire_previous_ranger_credential` | `POST /engines/\{id\}/retire-previous-ranger-credential` — the operator's confirmation that an EXTERNAL engine's plugin now runs with the credential named `ins... | @@ -137,7 +141,7 @@ Operation id: `list_engines`. Auth: Session cookie. Forbidden (403): Missing eng | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | Status | Meaning | |---|---| @@ -195,7 +199,7 @@ Operation id: `create_engine`. Auth: Session cookie and `X-Egret-CSRF` header. F | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | `catalogAttachErrors` | CatalogAttachErrorDto[] | yes | Catalogs from the body's `catalogs` that could not be attached. Empty on success. The engine EXISTS and is provisioned; these catalogs simply are not on it, and each can be retried with `POST /api/control-plane/engines/\{id\}/catalogs/\{cid\}`.... | | Status | Meaning | @@ -249,7 +253,7 @@ Operation id: `register_external_engine`. Auth: Session cookie and `X-Egret-CSRF | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | `rangerCredential` | RangerCredentialDto | yes | The credential the engine's Ranger plugin presents to egret's `/ranger` surface (HTTP Basic). **Shown once**: egret never returns this password again; a lost one is replaced with `POST... | | Status | Meaning | @@ -313,7 +317,7 @@ Operation id: `get_engine`. Auth: Session cookie. Forbidden (403): Missing engin | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | Status | Meaning | |---|---| @@ -376,7 +380,7 @@ Operation id: `update_engine`. Auth: Session cookie and `X-Egret-CSRF` header. F | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | Status | Meaning | |---|---| @@ -510,7 +514,7 @@ Operation id: `update_external_engine`. Auth: Session cookie and `X-Egret-CSRF` | `securityUpgradeReasons` | SecurityUpgradeReason[] | yes | Why the engine needs the security upgrade, in order: `unpatchedImage` (the stored FE image is absent or not an egret FE with every security patch), `noRootPassword` (the FE root has no password secret) and `notApplied` (the last security... | | `securityUpgradeFeImage` | string, nullable | no | The FE image the security upgrade would switch the engine to. Set only when `unpatchedImage` is a reason and the swap is not a downgrade; absent otherwise. The security upgrade action (`POST... | | `securityUpgradeBlockedRelease` | string, nullable | no | The release (e.g. `4.1.8`) that blocks the FE image swap: set when `unpatchedImage` is a reason but the engine already runs a release newer than the default FE, which cannot be downgraded. Absent otherwise. The security upgrade action... | -| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by the... | +| `notGoverned` | boolean | yes | `true` when Ranger is on and this StarRocks engine (provisioned or external) is bound to no policy profile, so Ranger governs nothing on it (KI-336). A Trino engine is never flagged: it late-binds a profile on its next apply. Set by every... | | Status | Meaning | |---|---| @@ -706,6 +710,137 @@ Operation id: `kill_query`. Auth: Session cookie and `X-Egret-CSRF` header. Forb | 500 | Internal error | | 502 | Trino coordinator unreachable or rejected the cancel | +## GET `/api/control-plane/engines/{id}/query-log` + +Operation id: `list_query_log`. Auth: Session cookie. Forbidden (403): Missing engine:read permission. + +| Parameter | In | Type | Required | Description | +|---|---|---|---|---| +| `id` | path | string (uuid) | yes | Engine id | +| `from` | query | string (date-time) | no | RFC 3339; rows with `from <= timestamp`. Default: `to` minus 24 hours. | +| `to` | query | string (date-time) | no | RFC 3339; rows with `timestamp < to`. Default: now. At most 31 days after `from`. | +| `user` | query | string | no | The session user or the user inside `authorizedUser`, exact and case-insensitive; at most 128 characters. | +| `state` | query | string | no | `ok`, `error` or `all` (default). | +| `limit` | query | integer (int32) | no | Rows per page, 1 to 200 (default 50). | +| `cursor` | query | string | no | The previous page's `nextCursor`. | + +**Response 200 (application/json)**: `QueryLogPageDto` + +| Field | Type | Required | Description | +|---|---|---|---| +| `entries` | QueryLogEntryDto[] | yes | | +| `nextCursor` | string, nullable | no | Pass it as `cursor` for the next page; `null` on the last. | + +| Status | Meaning | +|---|---| +| 200 | A page of the engine's statements, newest first. `statement` and `errorMessage` are absent without engine:audit | +| 400 | A malformed parameter; the message names it | +| 401 | Not authenticated | +| 403 | Missing engine:read permission | +| 404 | No such engine | +| 409 | The engine is external (`code` = `engine_is_external`), not a StarRocks engine, or its query log table does not exist (`code` = `query_log_not_ready`, with `state`) | +| 500 | Internal error | +| 502 | The FE did not answer or refused the read (`code` = `engine_fe_error`, scrubbed message) | + +## POST `/api/control-plane/engines/{id}/query-log/setup` + +Operation id: `setup_query_log`. Auth: Session cookie and `X-Egret-CSRF` header. Forbidden (403): Missing engine:update permission. + +| Parameter | In | Type | Required | Description | +|---|---|---|---|---| +| `id` | path | string (uuid) | yes | Engine id | + +**Response 200 (application/json)**: `QueryLogStatusDto` + +| Field | Type | Required | Description | +|---|---|---|---| +| `state` | QueryLogState | yes | The query log's state, as the Query log tab shows it (design section 5). | +| `message` | string, nullable | no | Why the state is `unreachable` (the FE's message, scrubbed). `null` otherwise. | +| `lastEventAt` | string, nullable | no | The newest event in the last day (RFC 3339); `null` when there is none or the table does not exist. | +| `retentionDays` | integer (int32) | yes | How many days of events the table keeps. | + +| Status | Meaning | +|---|---| +| 200 | Set up: state `ready`, or `pluginNotShipped` when the engine's image has no plugin (the table and loader user were still created) | +| 401 | Not authenticated | +| 403 | Missing engine:update permission | +| 404 | No such engine | +| 409 | The engine is external (`code` = `engine_is_external`), not a StarRocks engine, or being deleted | +| 500 | Internal error | +| 502 | A setup step failed (`code` = `engine_fe_error`, scrubbed message) | + +## GET `/api/control-plane/engines/{id}/query-log/status` + +Operation id: `query_log_status`. Auth: Session cookie. Forbidden (403): Missing engine:read permission. + +| Parameter | In | Type | Required | Description | +|---|---|---|---|---| +| `id` | path | string (uuid) | yes | Engine id | + +**Response 200 (application/json)**: `QueryLogStatusDto` + +| Field | Type | Required | Description | +|---|---|---|---| +| `state` | QueryLogState | yes | The query log's state, as the Query log tab shows it (design section 5). | +| `message` | string, nullable | no | Why the state is `unreachable` (the FE's message, scrubbed). `null` otherwise. | +| `lastEventAt` | string, nullable | no | The newest event in the last day (RFC 3339); `null` when there is none or the table does not exist. | +| `retentionDays` | integer (int32) | yes | How many days of events the table keeps. | + +| Status | Meaning | +|---|---| +| 200 | The query log's state, read from the FE now. An FE that does not answer is state `unreachable` with its scrubbed message, still a 200 | +| 401 | Not authenticated | +| 403 | Missing engine:read permission | +| 404 | No such engine | +| 409 | The engine is external (`code` = `engine_is_external`), or not a StarRocks engine | +| 500 | Internal error | + +## GET `/api/control-plane/engines/{id}/query-log/{queryId}` + +Operation id: `get_query_log_entry`. Auth: Session cookie. Forbidden (403): Missing engine:read permission. + +| Parameter | In | Type | Required | Description | +|---|---|---|---|---| +| `id` | path | string (uuid) | yes | Engine id | +| `queryId` | path | string | yes | The statement's query id (a UUID) | + +**Response 200 (application/json)**: `QueryLogEntryDto` + +| Field | Type | Required | Description | +|---|---|---|---| +| `queryId` | string | yes | | +| `timestamp` | string | yes | When the statement ended (RFC 3339). | +| `user` | string, nullable | no | The session user ("Connected as"). | +| `authorizedUser` | string, nullable | no | The bare name of the user the statement ran for ("User"), parsed from `'bob'@'%'`; `null` when empty. | +| `clientIp` | string, nullable | no | | +| `catalog` | string, nullable | no | | +| `db` | string, nullable | no | | +| `state` | string, nullable | no | `EOF` or `OK` for a success, `ERR` for a failure. | +| `errorMessage` | string, nullable | no | The FE's error, redacted. Absent (not `null`) for a caller without `engine:audit`: StarRocks errors quote the statement's text. | +| `queryTimeMs` | integer (int64), nullable | no | | +| `scanRows` | integer (int64), nullable | no | | +| `scanBytes` | integer (int64), nullable | no | | +| `returnRows` | integer (int64), nullable | no | | +| `cpuCostNs` | integer (int64), nullable | no | | +| `memCostBytes` | integer (int64), nullable | no | | +| `command` | string, nullable | no | | +| `querySource` | string, nullable | no | | +| `isQuery` | boolean, nullable | no | | +| `digest` | string, nullable | no | | +| `statement` | string, nullable | no | The statement, redacted: its first 2,000 characters on a page, whole on the detail route. Absent for a caller without `engine:audit`. | +| `statementTruncated` | boolean | yes | Whether `statement` was cut. Always false when it is absent. | + +| Status | Meaning | +|---|---| +| 200 | One statement, whole. `statement` and `errorMessage` are absent without engine:audit | +| 400 | `queryId` is not a UUID | +| 401 | Not authenticated | +| 403 | Missing engine:read permission | +| 404 | No such engine, or no such statement in the log | +| 409 | The engine is external (`code` = `engine_is_external`), not a StarRocks engine, or its query log table does not exist (`code` = `query_log_not_ready`, with `state`) | +| 500 | Internal error | +| 502 | The FE did not answer or refused the read (`code` = `engine_fe_error`, scrubbed message) | + ## GET `/api/control-plane/engines/{id}/ranger-plugin-config` Operation id: `ranger_plugin_config`. Auth: Session cookie. Forbidden (403): Missing engine:read permission. diff --git a/platform/reference/rest-api/index.md b/platform/reference/rest-api/index.md index 46624c4..f5adbc3 100644 --- a/platform/reference/rest-api/index.md +++ b/platform/reference/rest-api/index.md @@ -133,7 +133,7 @@ described by their auth line here. | Data governance | [Governance data browser](rest-api/control-plane-governance) | 11 | | Data governance | [Classification tags](rest-api/control-plane-tags) | 8 | | Control plane | [Access control](rest-api/control-plane-access) | 26 | -| Control plane | [Engines](rest-api/control-plane-engines) | 28 | +| Control plane | [Engines](rest-api/control-plane-engines) | 32 | | Control plane | [Catalog registry](rest-api/control-plane-catalog-registry) | 14 | | Control plane | [Egret warehouses](rest-api/control-plane-catalog) | 30 | | Control plane | [Storage connections](rest-api/control-plane-storage) | 7 | diff --git a/static/img/platform/admin/cp-engine-query-log-not-set-up.png b/static/img/platform/admin/cp-engine-query-log-not-set-up.png new file mode 100644 index 0000000..c1d76e0 Binary files /dev/null and b/static/img/platform/admin/cp-engine-query-log-not-set-up.png differ diff --git a/static/img/platform/admin/cp-engine-query-log-panel.png b/static/img/platform/admin/cp-engine-query-log-panel.png new file mode 100644 index 0000000..66eede6 Binary files /dev/null and b/static/img/platform/admin/cp-engine-query-log-panel.png differ diff --git a/static/img/platform/admin/cp-engine-query-log-plugin-not-shipped.png b/static/img/platform/admin/cp-engine-query-log-plugin-not-shipped.png new file mode 100644 index 0000000..c59e14d Binary files /dev/null and b/static/img/platform/admin/cp-engine-query-log-plugin-not-shipped.png differ diff --git a/static/img/platform/admin/cp-engine-query-log.png b/static/img/platform/admin/cp-engine-query-log.png new file mode 100644 index 0000000..1ea5d1d Binary files /dev/null and b/static/img/platform/admin/cp-engine-query-log.png differ