diff --git a/README.md b/README.md index ae43aef..f1da810 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ This repository serves as the home for everything related to how we collaborate, This repo contains: - [Design proposals](./design-proposals/README.md) — cross-cutting and architectural changes +- [Decision records](./decisions/README.md) — what we decided and why, once a question is settled - [The community meeting](./community_meeting.md) — when it happens, how to join, how to add a topic - [The meeting archive](./meetings/README.md) — every past meeting, its topics and its recording - Community resources and helpful links (below) @@ -26,6 +27,7 @@ Cozystack work is split across two repositories. This repo is for **how we decid | A **bug** or a **concrete, scoped feature** | [cozystack/cozystack issues](https://github.com/cozystack/cozystack/issues/new/choose) | | A **usage question** or an early idea | [cozystack/cozystack Discussions](https://github.com/cozystack/cozystack/discussions) | | A **cross-cutting / architectural change** — affects multiple components or APIs, or needs a decision before code | A [design proposal](./design-proposals/README.md) (a PR) **here** | +| A **decision already made** whose reasoning would otherwise survive only in a pull-request thread | A [decision record](./decisions/README.md) (a PR) **here** | | **Governance, process, or community** matters | An issue **here** | ## Community Resources diff --git a/decisions/0001-computeplane-ships-as-an-operator-owned-module.md b/decisions/0001-computeplane-ships-as-an-operator-owned-module.md new file mode 100644 index 0000000..0936623 --- /dev/null +++ b/decisions/0001-computeplane-ships-as-an-operator-owned-module.md @@ -0,0 +1,44 @@ +# 0001. ComputePlane ships as an operator-owned module, not as preset fields on `kind: Kubernetes` + +- **Number:** `0001` +- **Date:** `2026-07-18` +- **Status:** Accepted +- **Deciders:** `@kvaps, @myasnikovdaniil` +- **Proposal:** [`design-proposals/compute-plane/README.md`](../design-proposals/compute-plane/README.md) +- **Implemented in:** not yet — see the proposal's Rollout section + +## Context + +The first ComputePlane revision ([#17](https://github.com/cozystack/community/pull/17), merged 2026-06-30) delivered a hardened, operator-controlled Kubernetes cluster for running code-executing catalog apps — notebooks, workflow code nodes, plugin systems — behind a per-VM kernel boundary instead of as shared-kernel pods on the management cluster. It was shaped as a tenant module selected by a single string (`computePlane: ""`), with the cluster's HelmRelease rendered inline in `apps/tenant/templates`. + +Two reviews converged on the same objection ([#26](https://github.com/cozystack/community/issues/26), and Timofei Larkin on [#17](https://github.com/cozystack/community/pull/17)): the cluster a ComputePlane runs on *is* an ordinary managed `kind: Kubernetes` — Kamaji control plane, KubeVirt-VM workers, operator-held kubeconfig — so ComputePlane should not become a distinct kind with a parallel cluster implementation, reconcile loop and RBAC surface. + +Revision [#27](https://github.com/cozystack/community/pull/27) answered that by putting the posture directly on the existing app as two orthogonal user-facing preset fields: `isolationProfile` (`standard` | `sandbox` | `cluster-meshed`) × `componentProfile` (`minimal` | `standard-addons` | `gpu`). A ComputePlane would then be nothing more than `kind: Kubernetes` with `isolationProfile: sandbox`. + +The isolation mechanism itself was never in question at any point — one-way remote Flux apply via `HelmRelease.spec.kubeConfig.secretRef`, untrusted code behind a disposable guest kernel, single-tenant, scoped egress. Only the delivery surface was contested. + +## Decision + +ComputePlane ships as a Cozystack-owned **tenant module** (`packages/extra/computeplane`) that provisions a hardened `kind: Kubernetes` by wrapping the unchanged `apps/kubernetes` chart with operator-fixed values, sourced through the existing PackageSource source-only-chart mechanism that NATS and SeaweedFS already use. + +Because a tenant module is registered through an `ApplicationDefinition`, ComputePlane does get its own `apps.cozystack.io` kind, with its own API endpoint and input schema — the same CRD-free way every managed service is registered. So the accurate claim is not "no new kind": it is **no new CRD, no new controller or reconcile path, and no new fields on `kind: Kubernetes`**. Catalog apps route onto the cluster through a `placement: ManagementPlane | ComputePlane` field on `ApplicationDefinition`. + +## Why not the alternatives + +- **User-facing preset fields on `kind: Kubernetes` ([#27](https://github.com/cozystack/community/pull/27)).** Putting the posture on the app object turns an operator's choice into tenant-editable API surface. The entire value of a sandbox is that the tenant *cannot* weaken it, and any field that configures the hardening is a field that can relax it. It also loads a hardened posture onto the one `ApplicationDefinition` that has to stay tenant-generic. +- **A distinct, heavyweight `kind: ComputePlane` with its own cluster implementation ([#17](https://github.com/cozystack/community/pull/17) as merged).** Duplicates the cluster reconcile and RBAC path, which was the substance of the #26 / #17-review objection. Note this decision does still register a `ComputePlane` kind — it departs from #26's literal "no new kind at all" — but as a thin wrapper over the unchanged app, which is what the objection was actually about. +- **The single-string `computePlane:` module rendered inline in `apps/tenant/templates` (the first revision's implementation).** Right delivery shape, wrong packaging: the cluster HelmRelease was inlined into the tenant chart instead of shipping as a first-class chart with its own PackageSource, the way `extra/seaweedfs` does. +- **Hardening containers in the tenant namespace instead.** Rejected as the primary boundary: hardening does not make container isolation multi-tenant, and it breaks the apps in scope. +- **gVisor or a sandboxed runtime as the primary boundary.** Rejected for incomplete syscall coverage and no blast-radius containment on a kernel panic. Still valid as an *inner* layer for per-task sandboxes inside the cluster, and tracked as a future runtime option rather than dismissed. + +## Consequences + +- The hardening is tamper-proof by construction. The cluster's values live entirely in an operator-owned chart and the tenant holds no admin kubeconfig, so "withhold admin" and "own the configuration" become one fact rather than two features to keep in sync. +- `kind: Kubernetes` gains no fields and stays tenant-generic. Node-provisioning and addon changes to the app apply to ComputePlane transparently. +- The cost is composability: several differently-hardened postures per tenant now require separate module variants rather than a combination of two fields. The `placement: ` path that would give a tenant N sandbox clusters is deferred. +- The `computeplane` release name is load-bearing (proposal, Design §1). Any later move of the chart between directories must preserve it. +- [#39](https://github.com/cozystack/community/pull/39) (folding `extra` into `apps` as declarative `ApplicationDefinition` capabilities) composes with this rather than conflicting: ComputePlane becomes a directory move plus `visibility: module` and `cardinality: {scope: tenant, max: 1}`, and the two-piece structure carries over unchanged. + +## Revisit if + +[#39](https://github.com/cozystack/community/pull/39) lands and changes how module kinds are registered, or a concrete need appears for several simultaneous, differently-hardened compute clusters per tenant — the case the single-module shape does not serve and the deferred `placement: ` path would. diff --git a/decisions/README.md b/decisions/README.md new file mode 100644 index 0000000..8a26b86 --- /dev/null +++ b/decisions/README.md @@ -0,0 +1,91 @@ +# Cozystack Decision Records + +This folder holds **decision records**: short, dated notes on architectural decisions the project has actually made, and why. + +A [design proposal](../design-proposals/README.md) is intent — what we think we should build, written before the work. A decision record is history — what we settled on, written once the question is closed. Both are useful; the second is the one we have been missing. + +## Why a separate folder + +Design proposals get edited. When implementation contradicts the design — and it regularly does — the proposal is revised in place, so it ends up reading as though it always said the current thing. The reasoning that changed our minds (*we tried X, hit a concrete constraint, moved to Y*) then survives only in a pull-request diff that nobody will find in a year. + +That reasoning is the most expensive thing we produce and the easiest to lose. A decision record is where it goes. + +| | Design proposal | Decision record | +|---|---|---| +| Written | before the work, to get agreement | once the question is settled | +| Answers | "should we, and how might we?" | "what did we decide, and why not the alternatives?" | +| Edited after merge | yes, as the design evolves | never — superseded by a new record | +| Length | as long as it needs to be | one page | +| Review | consensus from maintainers | one maintainer checks it for accuracy | + +Both have an alternatives section, and they are not the same list. A proposal weighs the options we could imagine *before* building. A record names the option that lost *during* building — frequently the proposal's own original design. + +## When to write one + +Write a decision record when a future contributor would otherwise have to reconstruct the reasoning from a pull-request thread. In practice: + +- Implementation contradicted an accepted proposal and the design changed course. +- Two viable approaches existed and we picked one for reasons that are not visible in the resulting code. +- We hit a constraint that now shapes the design — an upstream limitation, a Kubernetes semantic, a vulnerability class. +- The code enforces a contract it cannot explain: a field is immutable, a release name is load-bearing, an ordering is required. +- We deliberately decided *not* to do something, and the question keeps coming back. + +Do not write one for: + +- Routine code choices that the diff and the tests already explain. +- Operator-facing how-to — that is user documentation, and it belongs on [the website](https://cozystack.io/docs/). +- The mechanics of the product repository's own workflow (release process, changelog conventions) — those live next to the code in [cozystack/cozystack](https://github.com/cozystack/cozystack). +- A decision that has not been made yet. That is a design proposal, or an open question inside one. + +## How to write one + +1. Copy [`template.md`](./template.md) to `decisions/NNNN-short-slug.md`, taking the next free number: + + ``` + ./decisions/0007-etcd-is-per-cluster-not-per-tenant.md + ``` + +1. **Title the decision, not the topic.** `storageClass is immutable after creation`, not `storageClass immutability`. Someone scanning the folder should learn what we decided from the filename alone. + +1. **Keep it to a page.** Link out to the proposal, the code and the pull requests for detail. A record that grows into a second design document will not get read. + +1. **Link both ways.** The record links its proposal and the pull requests that implemented it; the proposal's `Decisions` section links back to the record. + +1. As with all commits in CNCF projects, sign the commit for the DCO check: + + ```bash + git commit --signoff + ``` + +Two pull requests can claim the same number. Whoever merges second renumbers on rebase — a file rename and a couple of link fixes. + +## Status and immutability + +| Status | Meaning | +|---|---| +| `Accepted` | Current. The decision stands. | +| `Superseded by NNNN` | A later record replaced it. The original text stays exactly as written. | +| `Reverted` | We undid it. Say what we do instead. | + +Once merged, **the body of an accepted record is not edited.** Fix typos and broken links; change nothing else. If the decision changes, write a new record and mark the old one `Superseded by NNNN`. + +This one rule is what makes the folder a history instead of a second set of documents to keep current. A record that gets quietly rewritten is worth no more than the proposal it was meant to supplement. + +## Review + +A decision record documents a decision that has already been made. Review therefore checks that the record is **accurate** — not whether the reviewer agrees with it. One maintainer's approval is enough, and it should be quick. + +If review turns into re-litigating the decision, that is a signal the decision was not actually settled. Close the pull request and open a design proposal or an issue instead. + +## Where this sits relative to everything else + +| The question you are answering | Where it is answered | +|---|---| +| "Why is it built this way, and not the obvious other way?" | A decision record, here | +| "What should we build?" | A [design proposal](../design-proposals/README.md), here | +| "How do I use it?" | [User documentation](https://cozystack.io/docs/) on the website | +| "How does this repository release, or write changelogs?" | [cozystack/cozystack](https://github.com/cozystack/cozystack), next to the code | + +## Inspiration + +The format follows [Michael Nygard's architecture decision records](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) and [MADR](https://adr.github.io/madr/), trimmed to the sections we will actually fill in. diff --git a/decisions/template.md b/decisions/template.md new file mode 100644 index 0000000..ae64a11 --- /dev/null +++ b/decisions/template.md @@ -0,0 +1,59 @@ + +# NNNN. State the decision here, as a fact + +- **Number:** `NNNN` +- **Date:** `YYYY-MM-DD` +- **Status:** Accepted | Superseded by `NNNN` | Reverted +- **Deciders:** `@your-github-handle, @codecider` +- **Proposal:** `design-proposals//README.md` — or `none` +- **Implemented in:** `cozystack/cozystack#NNNN` — or `not yet` + + + +## Context + + + +## Decision + + + +## Why not the alternatives + + + +## Consequences + + + +## Revisit if + + + +--- + + diff --git a/design-proposals/README.md b/design-proposals/README.md index 83fba99..e473ce3 100644 --- a/design-proposals/README.md +++ b/design-proposals/README.md @@ -58,7 +58,9 @@ Proposals are reviewed in GitHub PRs. Once a proposal receives consensus from co We understand that real-world implementation may diverge from initial designs. This is normal and expected. -While it's helpful to update the proposal if the divergence is significant, the **codebase and user documentation are the final source of truth**. +Update the proposal when the divergence is significant. When the divergence came from a decision worth remembering — an approach that failed, a constraint you discovered, an alternative you picked instead — also write a [decision record](../decisions/README.md) and link it from the proposal's `Decisions` section. Editing the proposal alone loses the reasoning: the revised text reads as though it always said the current thing, and *why* the design changed course survives only in the pull-request diff. + +The **codebase and user documentation remain the final source of truth** for what the system does. The decision records are the source of truth for why it does it that way. ## Inspiration diff --git a/design-proposals/compute-plane/README.md b/design-proposals/compute-plane/README.md index a64f978..27710f8 100644 --- a/design-proposals/compute-plane/README.md +++ b/design-proposals/compute-plane/README.md @@ -36,6 +36,10 @@ The capability is generic and intended to live in Cozystack core as a reusable p - **`design-proposals/kubernetes-nodes-split`** / **`kubernetes-nodes-hybrid-clusters`** (PR #8/#9): the substrate is the existing managed-`kubernetes` app (Kamaji + CAPI/KubeVirt); node-provisioning changes apply transparently. - **Deferred:** billing/metering of cluster resource and API consumption; secret delivery of managed-service connection strings into sandbox workloads; the per-instance/label granularity of the visibility control (Design §6). (Cross-tenant *sharing* of a cluster is **not** deferred — it is rejected by design; see Non-goals.) +## Decisions + +- [0001. ComputePlane ships as an operator-owned module, not as preset fields on `kind: Kubernetes`](../../decisions/0001-computeplane-ships-as-an-operator-owned-module.md) — why the preset-field revision (#27) was set aside. + ## Context Today Cozystack already has every primitive needed *except* the glue that ties them into "deploy this catalog app onto a hardened, operator-controlled `kind: Kubernetes` the tenant does not administer": diff --git a/design-proposals/template.md b/design-proposals/template.md index 176743a..ae2541c 100644 --- a/design-proposals/template.md +++ b/design-proposals/template.md @@ -20,6 +20,18 @@ A reader who stops here should know what you're asking for and roughly why. --> Link them by repo path or URL. If this proposal must land before or after another, say so. Omit the section if there are no related proposals. --> +## Decisions + + + ## Context