From 16432fb5d7d1c57e39895a03cc5fb4adba191737 Mon Sep 17 00:00:00 2001 From: Marian Koreniuk Date: Wed, 7 Oct 2026 20:26:15 +0200 Subject: [PATCH] docs(networking): describe Proxmox-backed VPC subnets cozystack/cozystack#4813 (draft) lets a tenant back a VPC subnet with a Proxmox VE VLAN by adding `proxmox: {}` to it, give the VPC an egress gateway, and place the workers of a Proxmox-backed Kubernetes cluster on such a subnet. The design is proposed in cozystack/community#93. The tenant page covers the new vpc fields (`subnets[].proxmox`, `egress`), the ProxmoxNetwork status and conditions, routing and egress, `proxmox.network` on Kubernetes and KubernetesNodes and `proxmox.additionalNetworks`, admission of attachments, deletion and the known limits. The operator page covers the opt-in `cozystack.proxmox-network` package of the iaas bundle: the zone bridge, QinQ and MTU on the hypervisors, the trunk NIC for Kube-OVN on the cluster nodes, the transit router, package values and zones, the admission policies, the OVN gateway management, conditions, metrics, alerts, day-2 operations and the changes that reach existing installations. The variants page names the new opt-in package, and the platform stack and licenses pages list the in-cluster Cluster API IPAM provider that it installs. Only `next/` is changed, and every page says the feature is not merged. Assisted-by: LLM Signed-off-by: Marian Koreniuk --- .../docs/next/guides/platform-stack/_index.md | 1 + .../next/networking/proxmox-vpc-subnets.md | 253 ++++++++++++ .../next/operations/configuration/licenses.md | 1 + .../next/operations/configuration/variants.md | 4 + .../next/operations/vpc/proxmox-network.md | 372 ++++++++++++++++++ 5 files changed, 631 insertions(+) create mode 100644 content/en/docs/next/networking/proxmox-vpc-subnets.md create mode 100644 content/en/docs/next/operations/vpc/proxmox-network.md diff --git a/content/en/docs/next/guides/platform-stack/_index.md b/content/en/docs/next/guides/platform-stack/_index.md index 1323c063a..05bf870cc 100644 --- a/content/en/docs/next/guides/platform-stack/_index.md +++ b/content/en/docs/next/guides/platform-stack/_index.md @@ -27,6 +27,7 @@ Cozystack-maintained charts, CRDs, controllers, and application APIs are license {{< oss-cards >}} {{< oss-card name="Kamaji" logo="kamaji" license="Apache-2.0" source="https://github.com/clastix/kamaji/blob/master/LICENSE" description="Deploys tenant Kubernetes control planes as pods in the management cluster. Enables multi-tenancy without dedicated control-plane VMs." >}} {{< oss-card name="Cluster API" logo="clusterapi" license="Apache-2.0" source="https://github.com/kubernetes-sigs/cluster-api/blob/main/LICENSE" description="Declarative cluster lifecycle management via Kamaji and KubeVirt providers. Enables consistent, reproducible tenant cluster provisioning and upgrades." >}} +{{< oss-card name="Cluster API IPAM Provider In-Cluster" logo="clusterapi" license="Apache-2.0" source="https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/blob/main/LICENSE" description="Cluster API IPAM provider that assigns static addresses from in-cluster pools. Installed with the opt-in Proxmox packages; gives Proxmox VE worker VMs their addresses, including from Proxmox-backed VPC subnets." >}} {{< oss-card name="KubeVirt" logo="kubevirt" license="Apache-2.0" source="https://github.com/kubevirt/kubevirt/blob/main/LICENSE" description="Virtual machine management as native Kubernetes workloads. Powers Cozystack's VM service and tenant cluster worker nodes via CDI disk management." >}} {{< /oss-cards >}} diff --git a/content/en/docs/next/networking/proxmox-vpc-subnets.md b/content/en/docs/next/networking/proxmox-vpc-subnets.md new file mode 100644 index 000000000..6dc791e58 --- /dev/null +++ b/content/en/docs/next/networking/proxmox-vpc-subnets.md @@ -0,0 +1,253 @@ +--- +title: "Proxmox-backed VPC Subnets" +linkTitle: "Proxmox VPC Subnets" +description: "Extend a VPC subnet onto a Proxmox VE VLAN, give it a way out of the VPC, and place the workers of a Proxmox-backed Kubernetes cluster on it." +weight: 12 +--- + +{{% alert title="Proposed feature" color="warning" %}} +This page describes a feature that is **not merged** into Cozystack and is not part of any release. +It is implemented in the draft pull request [cozystack/cozystack#4813](https://github.com/cozystack/cozystack/pull/4813), +its design is discussed in the proposal [cozystack/community#93](https://github.com/cozystack/community/pull/93), +and whether Proxmox VE becomes a supported platform is decided in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). +Field names and behaviour may change before it lands, or the feature may not land at all. +{{% /alert %}} + +## What this page covers + +How a tenant extends a subnet of its [VPC]({{% ref "/docs/next/networking/vpc" %}}) onto Proxmox VE, how to check that the subnet is ready, how virtual machines on it reach the rest of the VPC and the outside, and how to put the workers of a Proxmox-backed [Kubernetes]({{% ref "/docs/next/kubernetes" %}}) cluster on it. + +The platform side, which an administrator sets up once (hypervisor bridges, the trunk to the cluster nodes, zones and the transit network), is described in [Setting up Proxmox-backed VPC subnets]({{% ref "/docs/next/operations/vpc/proxmox-network" %}}). + +## How it works + +A Proxmox-backed subnet is an ordinary subnet of your VPC with one more field, `proxmox: {}`. +The platform then: + +- turns the Kube-OVN subnet into a VLAN-backed logical switch of your VPC's own router; +- allocates a VLAN for it from a range the administrator owns and carries that VLAN to the cluster nodes; +- creates an IP pool for the part of the subnet that Proxmox VMs take their addresses from; +- lets a Proxmox-backed `Kubernetes` cluster or `KubernetesNodes` pool refer to the subnet by name. + +VMs and pods on the subnet are L2 neighbours. The subnet's gateway is the VPC router port, so the VPC's isolation and routing cover the VMs as well. +You never choose the VLAN, the Proxmox bridge or the IP pool, and you cannot set them: they come from the platform. + +```text +VPC subnet "workers" (name, CIDR) <- what you write + | + +-- Kube-OVN logical switch + router port (gateway) + +-- VLAN from the administrator's zone <- allocated by the platform + +-- IP pool for the VMs' address range <- used by Cluster API + | +Proxmox VM NIC: bridge + VLAN tag + address from the pool +``` + +## Before you start + +- The administrator has enabled the `cozystack.proxmox-network` package and created at least one zone. Without it the VPC refuses the subnet with `the proxmox.cozystack.io API is not installed` or `no ProxmoxNetworkZone exists`. +- If the platform has more than one zone, ask the administrator which one your tenant may use. A zone can be limited to some namespaces. +- Proxmox-backed subnets are IPv4 only, with a prefix between `/8` and `/28`. + +## Add a Proxmox-backed subnet + +Add `proxmox: {}` to a subnet of a `VirtualPrivateCloud`: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: VirtualPrivateCloud +metadata: + name: main + namespace: tenant-example +spec: + subnets: + - name: workers + cidr: 172.20.10.0/24 + proxmox: {} + - name: data + cidr: 172.20.11.0/24 + proxmox: + zone: default + vmRange: 172.20.11.100-172.20.11.199 + - name: pods + cidr: 172.20.20.0/24 + allowSubnets: + - 172.20.10.0/24 + - 172.20.11.0/24 + egress: + enabled: true +``` + +| Field | Meaning | +| --- | --- | +| `subnets[].proxmox` | Back this subnet with a Proxmox VLAN. `{}` takes every default. | +| `subnets[].proxmox.zone` | The zone the VLAN comes from. Empty selects the only zone and fails when there are several. | +| `subnets[].proxmox.vmRange` | The addresses Proxmox VMs take, as `first-last`, inside the CIDR, after the gateway and before the broadcast address. Kube-OVN never hands these addresses to pods. Defaults to the middle half of the subnet: `172.20.10.64-172.20.10.191` for the `/24` above. | +| `egress.enabled` | Route traffic from the Proxmox-backed subnets to destinations outside the VPC through an egress gateway. See [Reaching the outside](#reaching-the-outside). | +| `egress.replicas` | Number of egress gateway pods. Default `1`. | +| `egress.internalCidr` | Optional CIDR, a `/29` is enough, for a dedicated overlay subnet that holds the gateway pods' primary interface. It must not overlap any other subnet in the cluster. By default they sit on the first Proxmox-backed subnet, which is the better choice; see the note in [Reaching the outside](#reaching-the-outside). | + +The subnet is split between the two address owners: + +```text +172.20.10.0/24 + .1 gateway: the VPC router port + .2 - .63 Kube-OVN: pods attached to the subnet + .64 - .191 VM pool: Proxmox VMs (the default vmRange) + .192 - .254 Kube-OVN: pods attached to the subnet +``` + +The VPC application refuses: + +- `allowSubnets` on a Proxmox-backed subnet. It is not a private subnet and accepts whatever its VPC routes to it, so the list would do nothing. Put `allowSubnets` on the overlay subnets that the Proxmox-backed ones should reach, as `pods` does above. +- A `vmRange` outside the CIDR or one that contains the gateway. +- Removing `proxmox` from a subnet and keeping its name. Remove the subnet and add it back under a new name. +- Removing a subnet while VM addresses are still allocated on it. Delete those machines first. + +## Check the subnet + +Every Proxmox-backed subnet has a read-only `ProxmoxNetwork` object in your namespace. Its name is generated, so select it by the VPC and subnet names: + +```bash +kubectl get proxmoxnetworks --namespace tenant-example --output wide \ + --selector cozystack.io/vpcName=virtualprivatecloud-main,cozystack.io/subnetName=workers +``` + +```console +NAME ZONE VLAN BRIDGE CIDR USED FREE READY REASON +subnet-1a2b3c4d default 214 vmbr1 172.20.10.0/24 0 128 True Ready +``` + +The `Ready` condition sums up the others: + +| Condition | Meaning | +| --- | --- | +| `VLANAllocated` | A VLAN was allocated from the zone. Reasons when it was not: `ZoneNotFound`, `NamespaceNotAllowed` (the zone does not admit your namespace), `ZoneExhausted` (no free VLAN left), `VLANConflict`. | +| `SubnetReady` | Kube-OVN created the logical switch for the subnet. | +| `IPPoolReady` | The VM pool lies inside the subnet and inside the range Kube-OVN keeps away from pods. `PoolExhausted` means every VM address is taken; VMs already on the network keep working. | +| `GatewayReady` | VMs can reach their gateway, the VPC router port. | +| `Ready` | All of the above. `GatewayReady` counts only while the platform manages gateways; with that management off it is `Unknown` with reason `Disabled`. A Kubernetes cluster or node pool is placed only on a Ready network. | + +When a condition is `False`, its message says why. Most of the reasons need the administrator: an exhausted zone, a zone that does not admit your namespace, or a gateway that is not ready. + +## Reaching other subnets + +VMs and pods on a Proxmox-backed subnet reach each other at L2. The VPC router routes between the VPC's subnets, with two limits: + +- Overlay subnets of a VPC are private. A Proxmox-backed subnet reaches an overlay subnet only when that overlay subnet lists its CIDR in `allowSubnets`. +- A pod attached to a Proxmox-backed subnet has to run on a node that carries the Proxmox trunk; which nodes do is up to the administrator. Keep overlay pods that the VMs need to reach on those nodes as well: traffic from a VM to an overlay pod on a node without the trunk was expected to fail, worked in the tests so far, and is still an open question. + +There is no route between two VPCs unless both declare a peering and route to each other, as for any VPC. Peering between VPCs with Proxmox-backed subnets has not been tested on a live cluster yet. + +## Reaching the outside + +Traffic from a Proxmox-backed subnet stays inside the VPC unless the VPC has an egress gateway. To let the VMs out, for example so that Kubernetes workers reach their control plane, DNS, NTP and image registries, set `egress.enabled: true`. + +The VPC then gets an egress gateway: pods with one leg in the VPC and one on the platform's transit network. The VPC router sends traffic from the Proxmox-backed subnets that leaves the VPC to these pods, which SNAT it onto the transit network. Where it can go from there is decided by the administrator's transit router: usually the platform's LoadBalancer range (where tenant Kubernetes control planes listen), DNS, NTP and the Internet. + +- One egress gateway serves one zone. A VPC whose Proxmox-backed subnets are in two zones cannot enable egress. +- `egress.enabled` needs at least one Proxmox-backed subnet. +- A static route of the VPC to something narrower than `0.0.0.0/0`, such as a peered VPC, keeps its own next hop and does not go through the gateway. +- Leave `egress.internalCidr` unset unless you have a reason. With it the gateway's primary interface sits on an overlay subnet, and on platforms where Cilium runs in front of the overlay tunnel, VM traffic to a LoadBalancer IP can be rewritten on the way and never arrive. +- Nothing reaches the VMs from outside the VPC: there is no inbound path (floating IP or DNAT) to a Proxmox VM yet, and VMs cannot reach the management cluster's ClusterIP Services. + +{{% alert color="info" %}} +Every VPC, whether it has Proxmox-backed subnets or not, now refuses a static route whose `nextHopIP` is outside the VPC's own subnets (or outside `169.254.0.0/16` when peers are declared). Such a VPC stops rendering with `route to : next hop is outside every subnet of this VPC`. +{{% /alert %}} + +## Put Kubernetes workers on the subnet + +A Proxmox-backed Kubernetes cluster (`substrate: proxmox`) can take its worker addresses from a Proxmox-backed subnet instead of a raw address range. Set `proxmox.network` on the cluster and on every node pool: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Kubernetes +metadata: + name: example + namespace: tenant-example +spec: + version: "v1.35" + substrate: proxmox + proxmox: + dnsServers: ["198.51.100.53"] + network: + vpc: main + subnet: workers + ccm: + credentialsSecretName: proxmox-credentials + csi: + credentialsSecretName: proxmox-credentials +--- +apiVersion: apps.cozystack.io/v1alpha1 +kind: KubernetesNodes +metadata: + name: example-md0 + namespace: tenant-example +spec: + cluster: example + minReplicas: 2 + maxReplicas: 2 + resources: + cpu: 2 + memory: 4Gi + substrate: proxmox + proxmox: + templateTags: [talos, worker-template] + dnsServers: ["198.51.100.53"] + network: + vpc: main + subnet: workers + additionalNetworks: + - name: net1 + vpc: main + subnet: data +``` + +| Field | Application | Meaning | +| --- | --- | --- | +| `proxmox.network.vpc`, `proxmox.network.subnet` | `Kubernetes` | The VPC application in this namespace and its Proxmox-backed subnet. Replaces `proxmox.ipv4Config`, which must then stay empty. | +| `proxmox.network.vpc`, `proxmox.network.subnet` | `KubernetesNodes` | The subnet for `net0` of every worker. Bridge, VLAN and IP pool come from the subnet. Cannot be combined with `proxmox.network.bridge` or `proxmox.network.vlan`. | +| `proxmox.network.mtu` | `KubernetesNodes` | NIC MTU. With a subnet, defaults to the zone's MTU. | +| `proxmox.additionalNetworks[]` | `KubernetesNodes` | Extra NICs, `net1` to `net31`, each on a Proxmox-backed subnet of this namespace, each with its own address from that subnet's pool. Changing the list rolls the pool. | + +As for any `KubernetesNodes` pool, the name is `-`, here pool `md0` of cluster `example`. + +Point the cluster and all of its pools at subnets of the same VPC, and enable egress on that VPC, or the workers cannot reach their control plane. + +The applications look the subnet up when they render. If its `ProxmoxNetwork` is not Ready, the render fails with the condition's message, and the HelmRelease of the cluster or pool shows why no machine was created. A subnet without `proxmox` fails with `is not Proxmox-backed`. + +In a namespace that has Proxmox-backed subnets, every Cluster API machine NIC must be on one of them: a pool with a raw `bridge`/`vlan` in the same namespace is refused by admission. + +The other Proxmox settings of these applications (credentials, the template, resource pools and privileges) are described in the [Kubernetes application reference]({{% ref "/docs/next/kubernetes" %}}) and in the `kubernetes-nodes` [README](https://github.com/cozystack/cozystack/tree/main/packages/apps/kubernetes-nodes). + +## Pods and KubeVirt VMs on the subnet + +Pods and KubeVirt VMs attach to a Proxmox-backed subnet as a secondary network, in the same way as to any VPC subnet, and must run on a node that carries the Proxmox trunk. + +With the package enabled, admission limits network attachments in tenant namespaces: + +- A pod or a KubeVirt VM may attach only NetworkAttachmentDefinitions and Kube-OVN subnets of its own namespace. +- A Multus default network must be written as `/`. +- Pinning a pod to a Kube-OVN logical switch with a `*.kubernetes.io/logical_switch` annotation is refused, except for the namespace's default switch. A copy of a running pod that keeps the annotations Kube-OVN wrote, such as `kubectl debug --copy-to` or a restored backup, is refused as well. + +## Remove a subnet + +Delete the Kubernetes clusters and node pools that use a subnet before you remove the subnet or the VPC. + +If VMs still hold addresses, the platform keeps the subnet, its VLAN and its pool until the last VM is gone, and the `ProxmoxNetwork` shows `Ready=False` with reason `InUse`. Traffic inside the VLAN keeps working meanwhile, but the egress gateway goes with the VPC, so the VMs lose their way out and their control plane. Deleting a VPC under a running cluster can also remove the VPC router while the held subnets stay. The VLAN returns to the zone only after Kube-OVN has removed the subnet's switch. + +## Limits + +- Between two VPCs' Proxmox-backed subnets there is no drop inside OVN yet. Isolation rests on one router per VPC, one VLAN per subnet and the administrator's transit router, which does not forward into VPC address ranges. +- VMs run at the zone MTU, pods on the overlay at a smaller one, and the router does not send "fragmentation needed". A full-size packet with the DF bit set from a VM to an overlay pod is dropped silently; TCP is not affected. +- IPv4 only. No inbound path to a VM and no ClusterIP access from VMs. +- capmox, which creates the VMs, and the Proxmox cloud-controller-manager still use one Proxmox API token for every tenant. +- Chassis failover and VPC peering with Proxmox-backed subnets have not been tested on a live cluster. + +The open questions are listed in the [design proposal](https://github.com/cozystack/community/pull/93). + +## See also + +- [Setting up Proxmox-backed VPC subnets]({{% ref "/docs/next/operations/vpc/proxmox-network" %}}): the administrator's guide. +- [VPC application reference]({{% ref "/docs/next/networking/vpc" %}}). +- [Managed Kubernetes]({{% ref "/docs/next/kubernetes" %}}). +- `docs/proxmox-networking.md` in [cozystack/cozystack#4813](https://github.com/cozystack/cozystack/pull/4813): the full design, failure handling and security model. diff --git a/content/en/docs/next/operations/configuration/licenses.md b/content/en/docs/next/operations/configuration/licenses.md index b5cc17aaa..73b709bff 100644 --- a/content/en/docs/next/operations/configuration/licenses.md +++ b/content/en/docs/next/operations/configuration/licenses.md @@ -22,6 +22,7 @@ Pinned upstream versions of managed runtimes (PostgreSQL, MariaDB, Kafka, etc.) {{< oss-card name="Kubernetes" logo="kubernetes" license="Apache-2.0" source="https://github.com/kubernetes/kubernetes/blob/master/LICENSE" description="Container orchestration kernel used for both the management cluster and tenant clusters." >}} {{< oss-card name="Kamaji" logo="kamaji" license="Apache-2.0" source="https://github.com/clastix/kamaji/blob/master/LICENSE" description="Hosted control planes for tenant Kubernetes clusters." >}} {{< oss-card name="Cluster API" logo="clusterapi" license="Apache-2.0" source="https://github.com/kubernetes-sigs/cluster-api/blob/main/LICENSE" description="Declarative provisioning of tenant Kubernetes clusters (core, operator, and Kamaji/KubeVirt providers)." >}} +{{< oss-card name="Cluster API IPAM Provider In-Cluster" logo="clusterapi" license="Apache-2.0" source="https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/blob/main/LICENSE" description="Static address pools for Cluster API machines, used by the opt-in Proxmox packages." >}} {{< oss-card name="KubeVirt" logo="kubevirt" license="Apache-2.0" source="https://github.com/kubevirt/kubevirt/blob/main/LICENSE" description="Virtual machines as Kubernetes-native workloads (core, CDI, CSI, and instancetypes)." >}} {{< /oss-cards >}} diff --git a/content/en/docs/next/operations/configuration/variants.md b/content/en/docs/next/operations/configuration/variants.md index 82ac58da6..6e1aeb0cc 100644 --- a/content/en/docs/next/operations/configuration/variants.md +++ b/content/en/docs/next/operations/configuration/variants.md @@ -124,6 +124,10 @@ spec: - vm-exportproxy ``` +Some packages of the `iaas` bundle are opt-in on `isp-full` and `isp-full-generic`: they are installed only when listed in `bundles.enabledPackages`. +One of them is `cozystack.proxmox-network`, which backs tenant VPC subnets with Proxmox VE VLANs and brings the in-cluster Cluster API IPAM provider along. +It is a proposed feature that is not in any release yet; see [Setting up Proxmox-backed VPC subnets]({{% ref "/docs/next/operations/vpc/proxmox-network" %}}). + ### `isp-full-generic` `isp-full-generic` provides the same full-featured PaaS and IaaS experience as `isp-full`, but is designed for generic Kubernetes distributions such as k3s, kubeadm, or RKE2. diff --git a/content/en/docs/next/operations/vpc/proxmox-network.md b/content/en/docs/next/operations/vpc/proxmox-network.md new file mode 100644 index 000000000..4113e7b60 --- /dev/null +++ b/content/en/docs/next/operations/vpc/proxmox-network.md @@ -0,0 +1,372 @@ +--- +title: "Setting up Proxmox-backed VPC Subnets" +linkTitle: "Proxmox VPC Subnets" +description: "Enable the opt-in proxmox-network package, prepare the Proxmox VE hosts, the cluster nodes and the transit network, and operate the zones that back tenant VPC subnets with Proxmox VLANs." +weight: 30 +--- + +{{% alert title="Proposed feature" color="warning" %}} +This page describes a feature that is **not merged** into Cozystack and is not part of any release. +It is implemented in the draft pull request [cozystack/cozystack#4813](https://github.com/cozystack/cozystack/pull/4813), +its design is discussed in the proposal [cozystack/community#93](https://github.com/cozystack/community/pull/93), +and whether Proxmox VE becomes a supported platform is decided in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). +Package names, values and behaviour may change before it lands, or the feature may not land at all. +{{% /alert %}} + +## What this page covers + +What a platform administrator sets up so that tenants can add `proxmox: {}` to a VPC subnet and put Proxmox VE virtual machines on it, the workers of Proxmox-backed Kubernetes clusters included: + +- the Proxmox VE hosts: a VLAN-aware zone bridge, QinQ on the uplink and the MTU; +- the cluster nodes: one trunk NIC each for Kube-OVN; +- the transit network and the router that gives the VPCs their way out; +- the `cozystack.proxmox-network` package and its zones; +- admission, status, metrics and alerts, and day-2 operations. + +The tenant side is described in [Proxmox-backed VPC subnets]({{% ref "/docs/next/networking/proxmox-vpc-subnets" %}}). + +## How the pieces fit + +A tenant's Proxmox-backed subnet becomes a VLAN-backed (underlay) logical switch of the tenant's own Kube-OVN VPC router. Each subnet gets one VLAN from a range the administrator owns. The VLANs reach the cluster nodes over a single trunk NIC per node, and Kube-OVN tags each subnet's localnet port with its VLAN. VMs and pods on a subnet are L2 neighbours, and the VPC router port is the VMs' gateway. + +```text + Proxmox VM (net0, tag 201) Proxmox VM (net0, tag 202) + | | + ---- zone bridge vmbr1 (VLAN-aware, on bond0.40, every Proxmox VE host) ---- + | trunk 199-299: one virtio NIC per cluster node, no address + v + node trunk NIC -> OVS bridge of the provider network + -> localnet port, tag 201 -> switch of tenant A's subnet -> VPC A router + -> localnet port, tag 202 -> switch of tenant B's subnet -> VPC B router + -> transit VLAN 199 -> egress gateways of every VPC -> transit router +``` + +| Object | Scope | Created by | Tenant access | +| --- | --- | --- | --- | +| `ProxmoxNetworkZone` | cluster | the administrator, through the package values | none | +| Kube-OVN `ProviderNetwork` (the trunk) | cluster | the package | none | +| Transit `Vlan`, `Subnet` and NetworkAttachmentDefinition | cluster, package namespace | the package | none | +| Kube-OVN `Subnet` with `vlan` set | cluster | the vpc application | through the application | +| `InClusterIPPool` for the VMs' range | tenant namespace | the vpc application | read | +| `ProxmoxNetwork` | tenant namespace | the vpc application; the controller fills its status | read | +| Kube-OVN `Vlan` | cluster | `proxmox-network-controller` | none | +| `VpcEgressGateway` | the zone's gateway namespace | the vpc application | none | + +The controller allocates the VLANs, holds the Subnet, the Vlan and the pool with finalizers until no VM holds an address, makes each router port reachable from the VLAN (see [Gateway management](#gateway-management)), maintains a namespace annotation that admission reads, and exports metrics. + +## Requirements + +- The `isp-full` or `isp-full-generic` variant with the `iaas` bundle. The package needs Kube-OVN and Multus. +- A Proxmox VE cluster that runs both the management cluster's nodes, as VMs, and the tenant VMs, so that both reach the zone bridge. This guide assumes that layout. The implementation was tested with Proxmox VE 9. +- For Proxmox-backed Kubernetes clusters, also the opt-in `cozystack.capi-provider-infra-proxmox` package and its credentials. The networks themselves do not need it: pods can use a zone before any Proxmox-backed cluster exists. +- A router for the transit network outside the cluster: a Linux host or VM, or a physical router. + +## Example topology + +The examples on this page use one topology. Every value is an example, not a default, unless it is named as the package default. + +- The hosts' uplink is a bond, `bond0`. Tenant traffic rides on an outer VLAN of it, `40`, as the VLAN device `bond0.40`. `vmbr0` is the hosts' own bridge directly on `bond0`. +- The zone bridge is `vmbr1`, VLAN-aware, on `bond0.40`. +- Tenant VLANs come from `200-299`. The transit VLAN is `199`, with the network `192.0.2.0/24` and the transit router at `192.0.2.1`. +- The Kube-OVN provider network is `pxtenant` on the trunk NIC `ens19` (both package defaults), and the transit subnet is `px-transit` (the package default). +- The uplink runs a 9000-byte MTU, so the zone MTU is `8996`. + +## Prepare the Proxmox VE hosts + +### Zone bridge + +On every host, create a VLAN-aware bridge on its own outer VLAN of the uplink, carrying the tenant VLANs and the transit VLAN: + +```text +auto bond0.40 +iface bond0.40 inet manual + mtu 9000 + +auto vmbr1 +iface vmbr1 inet manual + bridge-ports bond0.40 + bridge-stp off + bridge-fd 0 + bridge-vlan-aware yes + bridge-vids 199-299 + mtu 9000 + post-up sysctl -w net.ipv6.conf.vmbr1.disable_ipv6=1 +``` + +- Disabling IPv6 on the zone bridge keeps the host itself, with its link-local address, off the tenant transport. +- Bring up only the new interfaces, with `ifup bond0.40 vmbr1`. Do not run `ifreload -a` on a host whose interfaces file has stale entries. +- Add the outer VLAN to the switch ports of every path between the hosts, the backup path of the bond included. + +Tenant VLANs ride inside the outer VLAN, so a tag a guest sends on a tenant VLAN never reaches the hosts' own VLANs. That holds only while no guest NIC sits on a bridge that enslaves the uplink's parent device, such as `vmbr0` over `bond0`: such a NIC is a trunk on which the guest can set the outer tag itself and reach the zone bridge with any inner tag. Admission keeps Cluster API machines off those bridges (`uplinkBridges`, below). For any other guest on them, make the host drop guest 802.1Q and 802.1ad frames, for example with ebtables on the guest ports, or make that bridge VLAN-aware with `bridge-vids` that exclude the uplink and host VLANs. + +### QinQ on the uplink + +Between the hosts, tenant and transit frames carry two 802.1Q tags: the outer VLAN of the uplink and the inner tenant VLAN. The uplink must keep both. + +- A frame that loses its inner tag lands in the far bridge's VLAN 1 and reaches the trunk NICs untagged. No localnet port matches it, so everything that crosses hosts breaks, and the tenant VLANs of the other host are no longer apart on that bridge. +- Some NICs overwrite the inner tag when they insert the outer one in hardware. This was seen with Mellanox ConnectX-3 (`mlx4_en`) ports in an active-backup bond. Other NICs may behave the same, so test every pair of hosts, in both directions. +- To test a pair, send traffic on an inner VLAN from one host to the other, for example a ping from the transit router to an egress gateway. On the receiving host, `tcpdump -nei vlan` must show both tags, and `bridge fdb show br vmbr1` must list the sender's MAC under the inner VLAN, not `vlan 1`. A VM-to-VM ping across hosts on one tenant VLAN is the end-to-end check. +- If the inner tag is lost, turn off hardware VLAN insertion with `ethtool -K txvlan off` on every slave of the uplink bond, the backup slaves included, so the kernel inserts the outer tag. Persist it, for example as a `post-up` of the bond or of its VLAN device, and check it again after host changes. + +### MTU + +Choose one number, the zone MTU, and make every hop fit it: + +- The zone MTU is the uplink VLAN device's MTU minus 4, because the inner tag is payload of that device. In the example `bond0`, `bond0.40`, `vmbr1` and the trunk NICs run 9000, and the zone, the transit network and the transit router's interface 8996. +- A switch on the uplink path must pass 9000 bytes plus the Ethernet header, two tags and the FCS. +- The zone MTU goes into the VM NICs that capmox creates and, through the vpc application, into the Kube-OVN subnets of the Proxmox-backed subnets, so that the egress gateway pods run at the VMs' MTU. Set `transit.mtu` of the package to the same value. +- Overlay pods keep Kube-OVN's own MTU, typically 1400. OVN sends no "fragmentation needed" between the two, so a full-size packet with the DF bit set from a VM to an overlay pod is dropped silently. TCP is not affected. +- A changed zone MTU reaches only VMs that are created after the change, and pods that are recreated after it. + +The acceptance check is a DF ping of the VM's full MTU from a VM, through the egress gateway, to the transit router: it must arrive, or the VM must receive "fragmentation needed". + +## Prepare the cluster nodes + +Each management cluster node that should carry tenant networks needs one extra virtio NIC on the zone bridge, trunking the tenant and transit VLANs, with no address: + +```bash +qm set --net1 virtio,bridge=vmbr1,trunks=199-299,mtu=9000 +``` + +The NIC's name inside the node is the package's `providerNetwork.defaultInterface` (`ens19` by default). A node with another name goes into `providerNetwork.customInterfaces`, and a node without the NIC into `providerNetwork.excludeNodes`. + +On Talos Linux, tell Talos to leave the NIC alone. `dhcp: false` is not enough: + +```yaml +machine: + network: + interfaces: + - interface: ens19 + ignore: true +``` + +Without `ignore: true`, Talos keeps trying to take the NIC back from Open vSwitch, logs `error enslaving/unslaving link ... operation not supported`, and leaves the link down while Kube-OVN still reports the node ready. After patching a running node, restart its `kube-ovn-cni` pod, which brings the link up when it starts. + +Pods attached to a Proxmox-backed subnet, the egress gateways included, can run only on nodes with the trunk. + +## Prepare the transit router + +The egress gateways of all VPCs SNAT the traffic of their Proxmox-backed subnets onto the shared transit network. The router at the transit gateway address decides where it goes next, and it carries the L3 boundary between VPCs on its own. It must: + +- forward from the transit network, with SNAT, only to the LoadBalancer range the tenant control planes use, DNS (UDP 53), NTP (UDP 123) and the Internet; +- use as that LoadBalancer range the address pool the tenant Kubernetes control planes get their LoadBalancer addresses from, with a free address for each Proxmox-backed cluster; +- drop every other private destination and have no route into any VPC address range; +- decide transit traffic in both directions ahead of any other forwarding rule; +- keep its rules across restarts and netfilter reloads. A reload that removes them, under a forwarding policy of accept, leaves the router open. + +Inside the transit network, an ACL keeps each egress gateway to the router: one gateway cannot reach another VPC's gateway. + +## Enable the package + +Add `cozystack.proxmox-network` to `bundles.enabledPackages` of the Platform Package, and give the package its values. Enabling it also installs the in-cluster IPAM provider, `cozystack.capi-provider-ipam-in-cluster`. Listing that provider in `bundles.disabledPackages` at the same time fails the platform render. + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: Package +metadata: + name: cozystack.cozystack-platform +spec: + variant: isp-full + components: + platform: + values: + bundles: + enabledPackages: + - cozystack.proxmox-network +--- +apiVersion: cozystack.io/v1alpha1 +kind: Package +metadata: + name: cozystack.proxmox-network +spec: + variant: default + components: + proxmox-network: + values: + providerNetwork: + create: true + name: pxtenant + defaultInterface: ens19 + excludeNodes: [] + zones: + - name: default + bridge: vmbr1 + vlanRanges: ["200-299"] + mtu: 8996 + uplinkVLAN: 40 + uplinkBridges: [vmbr0] + transit: + enabled: true + vlan: 199 + cidr: 192.0.2.0/24 + gateway: 192.0.2.1 + mtu: 8996 +``` + +The package installs two components into `cozy-proxmox-network`: `proxmox-network-crds` first, which is kept on uninstall, and then `proxmox-network` with the controller. + +### Package values + +| Value | Default | Meaning | +| --- | --- | --- | +| `providerNetwork.create` | `false` | Create the Kube-OVN ProviderNetwork that carries the trunk. Set `true` unless you manage it yourself. | +| `providerNetwork.name` | `pxtenant` | ProviderNetwork name, at most 12 characters. | +| `providerNetwork.defaultInterface` | `ens19` | The trunk NIC inside the nodes. | +| `providerNetwork.customInterfaces` | `[]` | Nodes whose trunk NIC has another name, as `[{interface: ens20, nodes: [node-3]}]`. | +| `providerNetwork.excludeNodes` | `[]` | Node names without a trunk. A name that matches no Node excludes nothing, and the node it meant keeps the provider network from becoming ready. | +| `providerNetwork.nodeSelector` | `{}` | Select trunk nodes by label instead. | +| `zones` | `[]` | The `ProxmoxNetworkZone` objects, see below. | +| `transit.enabled` | `false` | Create the transit `Vlan`, `Subnet` and NetworkAttachmentDefinition. Zones without their own `egress` then use it. | +| `transit.name` | `px-transit` | Name of the transit subnet. | +| `transit.vlan`, `transit.cidr`, `transit.gateway` | | VLAN, network and router address of the transit network. | +| `transit.excludeIps` | `[]` | Addresses the router side keeps, in Kube-OVN notation (`a..b`). | +| `transit.mtu` | `""` | MTU of the egress gateways' transit interface. Match the zone MTU. | +| `ovn.manageGatewayChassis` | `true` | Make each Proxmox-backed router port reachable from its VLAN. See [Gateway management](#gateway-management). | +| `ovn.nbAddress` | `ssl:ovn-nb.cozy-kubeovn.svc:6641` | OVN northbound address(es), comma-separated. Writes must reach the RAFT leader, which the `ovn-nb` Service selects. | +| `ovn.tlsSecret.namespace`, `ovn.tlsSecret.name` | `cozy-kubeovn`, `kube-ovn-tls` | Kube-OVN's client TLS material. | +| `admission.enabled` | `true` | The machine binding policy. | +| `admission.vmNetworkNamespace`, `admission.podNetworkNamespace` | `true` | The attachment policies for KubeVirt VMs and for pods. | +| `admission.exemptNamespaces` | `[]` | Namespaces the attachment policies skip, on top of the platform's own. | +| `admission.validationActions` | `[Deny]` | Validation actions of every policy of the package. | +| `alerts.enabled` | `true` | Ship the PrometheusRule. | +| `proxmoxNetworkController.subnetPruneTimeout` | `""` (10m) | How long a deleted subnet may stay listed on its Kube-OVN Vlan before the VLAN is released anyway. Raise it above twice the longest Kube-OVN or OVN outage you expect. | + +### Zones + +A zone is the pool of VLANs on one Proxmox bridge that tenant subnets draw from. + +| Field | Meaning | +| --- | --- | +| `name` | Zone name. Tenants name it in `subnets[].proxmox.zone`; with a single zone they may leave it out. | +| `bridge` | The VLAN-aware bridge present on every Proxmox VE host the VMs may run on. | +| `providerNetwork` | The Kube-OVN ProviderNetwork that carries the zone's VLANs. Defaults to `providerNetwork.name`. | +| `vlanRanges` | The only VLANs the allocator may hand out, such as `["200-299"]`. | +| `reservedVLANs` | VLANs inside the ranges that are never allocated. | +| `mtu` | Written into the VM NICs and into the Kube-OVN subnets of the zone's networks. | +| `namespaceSelector` | Which namespaces may have networks in this zone. Empty admits every namespace. | +| `staticAssignments` | Explicit VLANs for given networks, as `{namespace, name, vlan}`. | +| `egress.externalSubnet`, `egress.gatewayNamespace` | The subnet the VPC egress gateways attach to, and the platform namespace they run in. Default to the transit subnet and the package namespace when `transit.enabled` is set. Tenant egress needs both. | +| `uplinkVLAN` | Admission only, not part of the zone object. The outer VLAN that carries the zone bridge between hosts (`40` in the example). A machine NIC on any other bridge may not use it as its tag. | +| `uplinkBridges` | Admission only. Every other bridge that enslaves the uplink's parent device (`vmbr0` in the example). No machine NIC may sit on them. | + +Leaving `uplinkVLAN` or `uplinkBridges` unset leaves that path into the zone open. + +Extending a zone's range is safe. Shrinking it never takes a VLAN away from a network that holds one; the zone's Ready message lists the VLANs stranded outside the new ranges until those networks are deleted. + +### Verify + +```bash +kubectl get proxmoxnetworkzones +kubectl get provider-networks.kubeovn.io pxtenant +kubectl get nodes --selector pxtenant.provider-network.kubernetes.io/ready=true +``` + +The zone must be Ready, and every node that should carry the trunk must have the ready label. On each of those nodes the trunk NIC must be up, and `ovs-vsctl get interface ens19 link_resets` must stay steady. + +## Admission policies + +The package installs three ValidatingAdmissionPolicies, all bound with `admission.validationActions`: + +- `proxmox-network-machine-binding` (`admission.enabled`), on CREATE of `ProxmoxMachine` and `ProxmoxMachineTemplate` in every namespace: + - a NIC on a zone bridge must be one of the namespace's own networks (bridge, VLAN and pool together), so a namespace without Proxmox networks cannot reach the trunk through a raw bridge and VLAN; + - a NIC on any other bridge may not be tagged with a zone's `uplinkVLAN`; + - no NIC may sit on a zone's `uplinkBridges`; + - in a namespace that owns networks, every NIC must be one of them. +- `proxmox-network-vm-attachments` (`admission.vmNetworkNamespace`): a KubeVirt VM or VMI attaches only NetworkAttachmentDefinitions and Kube-OVN subnets of its own namespace. +- `proxmox-network-pod-attachments` (`admission.podNetworkNamespace`): the same rule for pods, including those other controllers create. A `*.kubernetes.io/logical_switch` pin other than the namespace's default switch is refused on CREATE, so a copy of a running pod that keeps Kube-OVN's annotations (`kubectl debug --copy-to`, a restored backup) is refused too. + +The attachment policies hold in every namespace except `cozy-*`, `kube-system`, the package namespace, each zone's egress gateway namespace and `admission.exemptNamespaces`. They apply as soon as the package is enabled, whether or not a tenant uses a Proxmox-backed subnet. The bridges, uplink VLANs and uplink bridges the machine policy checks come only from the package values: a zone created outside them is not protected, and VMs created outside Cluster API are not covered at all. + +## Gateway management + +Kube-OVN's OVN drops ARP requests that arrive from a localnet port for the address of a router port, unless that router port is a gateway port. Without a change, a VM on a Proxmox VLAN never resolves its gateway: it reaches pods on its own subnet at L2, but nothing routed. + +With `ovn.manageGatewayChassis: true` (the default), the controller turns each Proxmox-backed router port into a distributed gateway port. It writes one OVN `HA_Chassis_Group` per port, named `px-`, listing the chassis of the zone's trunk nodes, and points the port at it. The chassis priorities are keyed by the VPC, not the network, so all Proxmox-backed ports of one VPC are active on the same chassis, and different VPCs spread over the trunk nodes. Routed traffic of a subnet passes through its active chassis; L2 traffic inside a VLAN does not. + +What this means for operations: + +- Kube-OVN has no field for this, so the controller writes the OVN northbound database directly. It writes only the groups it owns (`external_ids` `owner=proxmox-network`), their chassis rows and the `ha_chassis_group` pointer of its networks' router ports. It never takes a group or a port that someone else has set; such a network turns `GatewayReady=False/ForeignGroup`. +- It connects with Kube-OVN's client certificate, read from the `kube-ovn-tls` Secret through a Role limited to that one Secret by name. +- A network is Ready only once its gateway is (`GatewayReady`). When the northbound database cannot be reached, existing groups keep working, and a network whose gateway was up stays Ready for 10 minutes. +- `ovn.manageGatewayChassis: false` removes the Role and every northbound write. Existing groups stay and keep working, but nothing repairs or removes them, and the VMs of new networks have no gateway. +- This is a stopgap until Kube-OVN sets such groups itself. Before upgrading to a Kube-OVN version that manages `ha_chassis_group` on subnet router ports, set `ovn.manageGatewayChassis: false`, upgrade, check that the router ports point at Kube-OVN's groups, and delete the `px-*` groups. Otherwise every network whose port Kube-OVN takes over turns `ForeignGroup` and not Ready. + +To read the groups, run in an `ovn-central` pod: + +```bash +ovn-nbctl --no-leader-only --columns=name,external_ids,ha_chassis list ha_chassis_group +``` + +## Status, metrics and alerts + +Each tenant subnet has a `ProxmoxNetwork` in the tenant's namespace. `kubectl get proxmoxnetworks --all-namespaces` lists them with their zone, VLAN, bridge, CIDR, pool usage and readiness. The conditions: + +| Condition | Typical reasons when not True | +| --- | --- | +| `VLANAllocated` | `ZoneNotFound`, `NamespaceNotAllowed`, `ZoneExhausted`, `VLANConflict` | +| `SubnetReady` | the reason Kube-OVN reports for the subnet | +| `IPPoolReady` | `PoolOverlapsSubnet`; `True/PoolExhausted` when the pool is full | +| `GatewayReady` | `WaitingForVLAN`, `RouterPortMissing`, `NoTrunkNodes`, `NBUnavailable`, `ForeignGroup`; `Unknown/Disabled` with gateway management off | +| `Ready` | `InUse`, `SubnetDeleting`, `SwitchDeleting`, `GatewayReleasePending` while a network is being deleted | + +The controller serves Prometheus metrics, among them: + +| Metric | Labels | +| --- | --- | +| `cozy_proxmox_zone_vlans_total`, `_allocated`, `_free` | zone | +| `cozy_proxmox_zone_networks` | zone | +| `cozy_proxmox_network_ready` | namespace, network, zone, vlan | +| `cozy_proxmox_network_ippool_addresses` | namespace, network, state (`total`, `used`, `free`) | +| `cozy_proxmox_network_consumers` | namespace, network | +| `cozy_proxmox_network_deleting` | namespace, network | +| `cozy_proxmox_vlan_allocations_total` | zone, result | +| `cozy_proxmox_orphaned_vlans` | zone | +| `cozy_proxmox_network_gateway_ready`, `cozy_proxmox_network_gateway_chassis` | namespace, network | +| `cozy_proxmox_gateway_nb_errors_total` | | + +With `alerts.enabled`, the package ships these alerts: + +| Alert | Fires when | +| --- | --- | +| `ProxmoxNetworkNotReady` | a network that is not being deleted is not Ready for 15 minutes | +| `ProxmoxNetworkGatewayNotReady` | a network's VMs have had no gateway for 10 minutes | +| `ProxmoxNetworkGatewayNBErrors` | northbound errors keep growing for 10 minutes | +| `ProxmoxNetworkPoolNearlyFull` | a VM pool is more than 90% used | +| `ProxmoxNetworkZoneExhausted` | a zone has no free VLAN | +| `ProxmoxNetworkOrphanedVLANs` | a zone has had Kube-OVN Vlans whose network is gone for 30 minutes | +| `ProxmoxNetworkStuckDeleting` | a network has been deleting for an hour | +| `ProxmoxNetworkControllerDown` | the controller has not been scraped for 10 minutes | + +## Day-2 operations + +- A network stuck in `InUse` waits for machines. `kubectl get ipaddresses --namespace ` shows who holds addresses. Delete the clusters or pools on a network before the VPC: deleting a VPC under a running cluster cuts the workers off their control plane, and the network stays `InUse` meanwhile. +- A network in `SwitchDeleting` waits for Kube-OVN to remove the subnet's switch and is released, at the latest, at the time its Ready message gives. +- Never delete a Kube-OVN `Vlan` that carries zone labels by hand; delete the tenant's subnet instead. +- A `ForeignGroup` network needs a decision by hand; the condition's message names what the controller found. +- An egress gateway can stay Terminating with Kube-OVN's finalizer when its VPC router went first. Once `ovn-nbctl lr-list` no longer lists the VPC router, remove the finalizer by hand. + +## Changes that reach existing installations + +- Every VPC refuses a static route whose `nextHopIP` is outside the VPC's own subnets, or outside `169.254.0.0/16` when peers are declared. A VPC with such a route stops rendering after the upgrade. Check the routes of existing VPCs before upgrading. +- VPC peering link addresses carry their `/30`. +- The Proxmox cloud-controller-manager of Proxmox-backed Kubernetes clusters moves to v0.16.1. +- The admission policies apply only once the package is enabled. + +## Limits and open questions + +- There is no drop between VPCs inside OVN yet: a Proxmox-backed subnet is not private, and the transit router alone stops one VPC from reaching another VPC's address range at L3. Moving that boundary into OVN needs a platform-owned list of the tenant address space, which is an open question of the proposal. +- VMs run at the zone MTU next to overlay pods at a smaller one, with no "fragmentation needed" between them. +- Traffic from a VM to an overlay pod on a node without the trunk was expected to fail and worked in tests, without an explanation yet. +- Deleting a VPC may delete its router while the held subnets stay. +- Failover of the active gateway chassis and VPC peering with Proxmox-backed subnets have not been tested on a live cluster. +- IPv4 only. No inbound path (floating IP or DNAT) to a VM, no ClusterIP access from VMs, and capmox and the cloud-controller-manager use one Proxmox API token for every tenant. +- Management access to tenant VMs, and egress narrowed to a tenant's own control plane, are open questions. +- Proxmox SDN is not used. The proposal describes a phased plan for it. + +The full list, with the decisions the feature needs, is in the [design proposal](https://github.com/cozystack/community/pull/93) and in `docs/proxmox-networking.md` of [cozystack/cozystack#4813](https://github.com/cozystack/cozystack/pull/4813). + +## See also + +- [Proxmox-backed VPC subnets]({{% ref "/docs/next/networking/proxmox-vpc-subnets" %}}): the tenant side. +- [Enabling and disabling components]({{% ref "/docs/next/operations/configuration/components#enabling-and-disabling-components" %}}): the `bundles.enabledPackages` mechanism used here. +- [Cozystack variants]({{% ref "/docs/next/operations/configuration/variants" %}}).