Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions platform/control-plane/query-engines/engine-detail.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,12 @@ A Trino engine that has run before and now has no pod or container at all, for e

Type the engine name and press **Delete**. You return to the Engines list. The dialog says *This destroys the engine's Kubernetes cluster. It cannot be undone.*, or on the Docker provisioner *This removes the engine's containers. It cannot be undone.* (measured: 7 seconds). The engine's policy profile is kept and appears under **Profiles without an engine** in Governance.

:::warning Deleting a StarRocks engine destroys its front-end metadata
The delete removes the engine's initpwd Job pods and the front end's metadata volumes (`fe-meta-<name>-fe-0` and any further ordinals) along with the cluster. That metadata holds the StarRocks users, catalogs, grants, the root password and the query log, and it cannot be recovered. Table data stays in object storage under the engine's storage path. Creating an engine of the same name afterwards starts with a fresh front end.

The same holds outside Egret: deleting the front end's StatefulSet with `kubectl delete sts` (default cascade) now deletes its metadata volumes too. To have the operator re-create the StatefulSet and keep the metadata, delete it with `--cascade=orphan`.
:::

### Kubernetes-only features

**Logs** and **Exec** work only with the Kubernetes provisioner, and are not shown for an engine Docker runs:
Expand Down
6 changes: 6 additions & 0 deletions platform/control-plane/query-engines/provision.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,12 @@ On the demo stack the default mode ran queries as the signed-in person in both c

A StarRocks engine has its own configuration screen with sections such as **Replicas & resources**, **JVM**, **Per-node memory**, **Health probes**, **Graceful shutdown**, **Resource groups** and **Session properties**, plus a Security section for authentication and delegation. Attaching or detaching a catalog on StarRocks creates or drops it on the FE in place, without rolling the pods. StarRocks engines need the Kubernetes provisioner.

If an earlier StarRocks engine of the same name left its front-end metadata volume behind, **Create** is refused with *A previous StarRocks engine named "\<name\>" left its front-end metadata volume fe-meta-\<name\>-fe-0. A new engine would start with the old engine's users, catalogs and root password. Delete it first: kubectl delete pvc fe-meta-\<name\>-fe-0 -n egret-engines*. Run the command, then create the engine again. (A live engine of that name gets the usual "already exists" refusal.)

```bash
kubectl delete pvc fe-meta-<name>-fe-0 -n egret-engines
```

## What happens next

1. Egret starts the cluster, creates a **policy profile** with the engine's name (for example `docs-trino`, with the tag service `docs-trino_tag`) and binds it.
Expand Down
33 changes: 32 additions & 1 deletion platform/deploy-operate/engines-and-ranger.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ Egret writes its own policies into the profile of every engine that delegates qu
the engine, by engine username. A disabled platform admin is not in them.
- On Trino, three baseline policies: `egret-user-impersonate-self`, `egret-user-execute-query` and `egret-query-timeout`.
- On a Trino engine that impersonates with a delegator account, also `egret-delegator-impersonate`.
- On a StarRocks engine in impersonation mode, also `egret-delegator-impersonate`, on the resource `user`. It lets the delegator `egret_delegator` run `EXECUTE AS` for every user except `root`, `egret_delegator` and `egret_query_log`.

Egret derives this set from the engines bound to the profile and the active platform admins, and brings the stored
policies in line with it. It adds a missing policy, updates one in place (the policy id stays the same), and deletes one
Expand All @@ -166,7 +167,7 @@ finds nothing to change leaves the policy version where it was.

So a policy that is no longer wanted goes away:

- `egret-delegator-impersonate` goes when the engine moves from impersonation to JWT;
- `egret-delegator-impersonate` goes when the engine moves from impersonation to JWT (on StarRocks too);
- all of these policies go when the profile's last delegating engine is unbound or deleted;
- the `egret-admin` policies go when there is no active platform admin.

Expand All @@ -185,6 +186,36 @@ A change reaches an engine at the plugin's next download. Engine tokens already
valid until they expire (at most 15 minutes), but a disabled admin's tokens no longer carry all access once the new
policies are downloaded.

### StarRocks users for impersonation

In impersonation mode egret runs each query on StarRocks as the person who started it (`EXECUTE AS`), and StarRocks can only do that for a user that exists. So egret creates the users itself; nobody has to configure a service user by hand.

- **Who gets a user.** On an engine that serves at least one workspace, egret creates `'<username>'@'%'` for each member of each workspace the engine serves, and for each live platform admin. The password is random and nobody knows it, so nobody can log in as that user: it exists only for the delegator to become. A platform admin gets a user on every impersonation engine assigned to a workspace.
- **When a user is dropped.** egret drops a user it created when access ends: the person leaves the workspace, is disabled or deleted, loses `platform_admin`, or the engine is unassigned from every workspace or moved to JWT. Under JWT delegation egret does not use `EXECUTE AS` and keeps no users.
- **What it never touches.** egret drops only users it created, and keeps a record of them. A user it did not create is left as it is. The names `root`, `egret_delegator` and `egret_query_log` are never created, granted or dropped, and a person cannot be given the username `root` (they get `root-2`).
- **When it runs.** Database triggers start a sync on membership, role and user changes (a member added or removed, a group membership change, a role or platform-role grant, a user disabled, enabled or deleted). It also runs on every engine apply, once at server start, and every 10 minutes. A change takes effect within seconds; a failed step is logged and the next run tries again.
- **Native access control.** An engine bound to no policy profile has no Ranger row, so the delegator holds `GRANT IMPERSONATE ON USER '<username>'@'%'` for exactly the users egret manages, and never `ON ALL USERS` (which would let it become `root`). egret adds the per-user grants first and revokes `ON ALL USERS` only after every grant landed. Before it drops a user it revokes that user's grant. An engine bound to a profile runs Ranger, which ignores native grants; there the row above is the control.

#### Handing over a user egret did not create

If a member's name is already held by a StarRocks user egret did not create (for example a test user made by hand), egret leaves it untouched and logs a warning naming it: `starrocks users: a member's name is held by a StarRocks user egret did not create; it is left as it is`. To hand it over, drop it as `root` on the FE. The next sync creates egret's own user of that name.

```sql
DROP USER 'member-sso'@'%';
```

:::warning Known issue: orphan IMPERSONATE grants (KI-358)
On an engine bound to no policy profile, StarRocks keeps the delegator's `IMPERSONATE ON USER 'u'@'%'` after `DROP USER 'u'@'%'`, and refuses to revoke it once `u` no longer exists. egret revokes before its own drops, but it cannot revoke a grant on a name whose user is already gone: one dropped before this release, or by someone other than egret. Anyone who later creates a user of that name can then be impersonated by the delegator. To clean up, as `root`, for each name that `SHOW GRANTS FOR 'egret_delegator'@'%'` lists and `SHOW USERS` does not, create the user temporarily with a random password, revoke, then drop:

```sql
CREATE USER 'u'@'%' IDENTIFIED BY '<a long random password>';
REVOKE IMPERSONATE ON USER 'u'@'%' FROM USER 'egret_delegator'@'%';
DROP USER 'u'@'%';
```

Engines bound to a policy profile are not affected: Ranger does not read the native grant.
:::

### Authentication of the plugin

Every request under `/ranger/service/` is authenticated. `ranger.auth_mode` (`EGRET_RANGER__AUTH_MODE`, Helm
Expand Down
17 changes: 17 additions & 0 deletions platform/deploy-operate/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,23 @@ A provisioned StarRocks engine gets a [Query log](../control-plane/query-engines
- **API change:** four routes under `/api/control-plane/engines/{id}/query-log`. See [Engines API](../reference/rest-api/control-plane-engines.md#get-apicontrol-planeenginesidquery-log).
- Retention is 30 days, and there is no export and no rotation of the loader password.

### StarRocks impersonation, CN password and engine delete (platform #850)

- **StarRocks engines in impersonation mode start working under Ranger.** On upgrade, egret writes `egret-delegator-impersonate` to their profile and creates a StarRocks user for each member of each workspace the engine serves, within seconds of start. Users made by hand for earlier tests (for example a passwordless `member-sso`) are not egret's: they are left as they are and a warning names them. Drop them as `root` to let egret create its own; see [Handing over a user egret did not create](./engines-and-ranger.md#handing-over-a-user-egret-did-not-create).
- **Passworded StarRocks engines roll their FE and CN once** on their next apply, because `MYSQL_PWD` is now set on both. A CN that crash-loops today (KI-335) recovers then. An engine without a persistent front-end metadata volume loses its FE metadata on that roll, as on any roll.
- **Engine delete now removes the FE metadata volume.** The FE metadata of a deleted engine is gone; the table data stays in object storage.
- **Claims left by earlier deletes are not removed by the upgrade.** List them, and delete those whose engine no longer exists before re-creating an engine of the same name:

```bash
kubectl get pvc -n egret-engines | grep '^fe-meta-'
```

- **A user named `root` cannot be minted any more** (the new user gets `root-2`). One who already holds it keeps it and is not synced to StarRocks.
- **Platform admins get a StarRocks user** on every impersonation-mode engine assigned to a workspace.
- **Engines bound to no profile lose `GRANT IMPERSONATE ON ALL USERS`.** On their next apply or sweep (at most ten minutes after the upgrade) the delegator gets per-user grants for egret's users instead, and can no longer become `root`.
- **Orphan IMPERSONATE grants** on users dropped before the upgrade stay. See the manual cleanup under [StarRocks users for impersonation](./engines-and-ranger.md#starrocks-users-for-impersonation) (KI-358).
- **Creating a StarRocks engine whose name left a metadata claim** answers 409 `engine_metadata_left_over` with the `kubectl` command that removes it.

## Further reading

- [Backup and restore](./backup-restore)
Expand Down
Loading
Loading