Conversation
An egress shard no longer waits for an operator to hand-write the public address it translates to. A controller in the cell claims one address per shard from the addressing service and writes it into the shard's spec, where the data plane already reads it. The shard does not claim its own address. A shard runs on every translating node, including hardware at the edge, and the process that would make the claim is the one serving the datapath -- so the credential would sit on every such node, reachable from the process that also handles tenant packets, and the allocation would sit beside the path that attaches a workload. One controller per cell moves the credential count from the number of translating nodes to the number of cells and takes the allocation off that path: an address is claimed when a node is commissioned, not when a workload arrives. It is a binary of its own rather than a reconciler inside vpc-controller, following the split fabric-identity-controller already establishes for a controller needing an addressing credential. vpc-controller writes the attachment state of every workload in the cell and serves an admission webhook; a credential into the platform's own tenancy widens the blast radius of the one component a cell cannot run without, and one that expired would stop workloads attaching. spec.shardAddressIPv6 is write-once, so nothing here writes a value that is not an address the service has already bound. Every path that cannot produce one leaves the field untouched and retries. Key changes: - Add internal/egressaddress, which holds one address per shard under a claim named for that shard. Attribution rides on the name and on annotations because the service overwrites spec.ownerRef with the requesting project's identity - Add EgressShardAddressReconciler, which assigns the address and the IPv6 family label the selectors placing traffic on a shard match, and releases the claim only for a shard that is actually gone - Read the address from status.allocatedCIDR alone. The API carries a status.address holding the single-address form and no released version of the service writes it - Adopt a retained allocation the service refuses a same-named claim with, so an address held by an earlier claim is read rather than lost behind a conflict that never clears (milo-os/ipam #107) - Claim with reclaimPolicy Delete. Retain would hand a recreated shard the same address back and defeat the only remedy the write-once field leaves for a wrong one, and would never return the address at all - Add config/ipam/egress-shard-ipv6.yaml, the pool and class chain an operator applies. One pool per location, keyed on location rather than cell because two cells already serve one location. IPv6 only: the platform pairs no resolver with a translator, so an IPv4 egress address would be reachability the API reports and the data plane does not deliver - Leave the root pool's cidr empty. The service refuses a root pool without one, so applying the file unedited is rejected rather than seeding a permanent aggregate from a guess This cannot build until go.datum.net/network publishes the EgressShard spec address fields and label keys, which are on an unpushed branch. A local go.work resolves them for development and is gitignored; a committed replace directive would break the build for every checkout and would still need reverting after the dependency ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI checks out this repo alone, so a module replace pointing at a local path cannot resolve. Point both at the pushed commits instead, which CI can fetch, so the build reflects the branches this depends on. Key changes: - Replace the network module with the pushed egress address commit - Replace the operator module with the pushed egress API commit - Drop the local workspace file in favour of resolvable versions Revert both replaces once those modules release the fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claiming an address now reports the record in the addressing service that holds it, alongside the address itself. Nothing is written to the shard yet: the spec field that will carry the reference is not published, so this is the plumbing a correct reference needs and not the reference. The record has to be reported rather than recomputed by whoever wants it. The service overwrites spec.ownerRef on a claim with the requesting project's identity, so a claim cannot say which shard it was made for, and the only place that link can live is on the shard pointing back. The adopt path is why a holder carries its kind. When an allocation retained by an earlier claim already occupies a claim's identity, the service rolls its transaction back before refusing, so the claim it refused is never stored. The address is recovered from the IPAllocation the refusal names, and that name is a hash of the claim's namespace and name rather than the claim name, so it cannot be derived from the shard. A reference built from the shard's own name would, in exactly this case, name an object that does not exist. Key changes: - Return a Holding from egressaddress.Claim carrying the address and the namespace, kind and name of the record holding it, taken from the object actually stored rather than recomputed - Report a retained allocation as held by KindIPAllocation, under the name the refusal carried - Report nothing at all for an unbound claim, so there is no partial holder to write into a field that cannot be rewritten - Assign the address in one update, so a rule pairing it with the record holding it is satisfied by that write rather than by a second one that could fail on its own - Log the holder as a stopgap for auditability. A log line is not queryable and ages out; it is not the record Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A shard now carries the addressing-service record its public IPv6 address came from, written in the same update as the address itself. The address alone is unattributable: the service overwrites a claim's spec.ownerRef with the requesting project's identity, so a claim cannot say which shard it was made for and the trail has to run from the shard back. The reference names the record that exists, which is not always a claim. An address recovered from a retained allocation is held by an IPAllocation and by no claim at all -- the service rolls its transaction back before refusing the create, so the claim it refused was never stored, and its name is a hash of the claim's namespace and name rather than the claim name. Kind carries that difference instead of taking its default, because both fields are write-once and a reference naming a claim that does not exist would be permanent for the shard's lifetime. Nothing gains a reference it did not earn. An address assigned by hand has no claim behind it and stays unattributable rather than gaining one this controller invented, and a claim holding no address yet writes neither field. Key changes: - Write spec.shardAddressIPv6ClaimRef beside the address in one update. The API pairs neither field with the other, so this is for atomicity rather than to satisfy validation: two writes would leave a window where the address is recorded and its origin is not - Set Kind from the record actually read, IPClaim or IPAllocation, rather than accepting the IPClaim default - Carry the platform project on the reconciler and wire it from the existing -platform-project flag. The reference requires it: a claim is namespaced within a project and this reference is read from outside every project, so a namespace alone does not identify one - Refuse to start without it, alongside the existing required values - Repin go.datum.net/network to affb112, which publishes AddressClaimRef and the field Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The claimer mounted a credential that nothing had permitted to create claims. This adds the grant, gives the component an identity of its own, and states on the deployment what holding that credential reaches. The narrowest grant the platform can express is coarser than the one the component wants. Milo carries the project in the caller's user extras and RBAC does not read extras, so verbs are authorized on every project path at once; an iam.miloapis.com PolicyBinding naming the Project is the only way to narrow that, and both existing IPAM grants are wide for the same reason. Per-namespace scoping cannot be expressed at all: IPAM declares Project as the only parent of an IPClaim, so a namespaced Role would authorize that namespace name in every project and narrow nothing. The grant is not part of the cell overlay. It is applied to the Milo control plane, so carrying it here would invite applying it to a cell. Key changes: - Add config/egress-address/milo-grant.yaml: a ClusterRole granting get, create and delete on IPClaims and get on IPAllocations, bound to system:egress-address@cloud.datumapis.com. Deliberately excluded from the overlay's kustomization - Grant no classes and no pools under any verb, and no delete on IPAllocations. Claims carry reclaim policy Delete, so removing a claim frees its address without reaching past it; address space is operator inventory and nothing that allocates from it needs to reshape it - Specify the certificate and per-cell Secret that have to be created with it, including that the Secret must not be the cell controller's ipam-cluster-kubeconfig - Mount egress-address-ipam-kubeconfig rather than reusing that Secret name, which authenticates as system:nso-cell and additionally carries delete on IPAllocations. Sharing the name would have handed this component the broader identity silently - State the blast radius on the volume a reviewer reads to understand the credential: what a compromised holder could do, what it could not, and why the scope stops where it does Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The public address an egress shard translates to is hand written by an operator today, so no shard outside a lab has one and nothing records which address went where.
This claims an address from the addressing service for each shard and assigns it, running in the edge cluster that holds the shard rather than centrally, so the credential stays off the machines carrying tenant packets and off the path that attaches a workload.
It also ships the addressing inventory an operator applies to make the space real, deliberately without a prefix, so applying it unedited is refused rather than quietly reserving public space nobody assigned.
Reaching the internet still needs that prefix chosen and its route announced, neither of which this does.
Architecture
sequenceDiagram participant S as Egress shard participant C as Address controller, in the same edge cluster participant I as Addressing service, in the project S->>C: a shard exists with no public address C->>I: claim one address from the operator's pool I-->>C: a new address, or the one this shard already holds C->>S: assign it, and record the claim it came from Note over S: assignment is write-once, so a recovered claim has to be the real one S->>C: shard released C->>I: return the address to the poolTest plan
Related to datum-cloud/network-services-operator#475