Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Understudy

Understudy prepares temporary replacement capacity for announced node disruptions.

## Language

**Desired baseline**: The capacity the service author requests outside a disruption.
_Avoid_: Effective replicas, surge capacity

**Effective replicas**: The capacity currently requested from the workload controller,
including any temporary surge.

**Handover baseline**: The baseline committed to an active disruption handover;
later desired-baseline changes do not alter its readiness requirement.

**Handover**: The interval in which Understudy prepares replacement capacity and
allows departing pods to leave, ending when the disruption clears or a release takes over.

**Pending baseline**: The latest desired baseline awaiting completion of a handover.
Intermediate requests are superseded rather than queued.

**Release**: A change to the application's revision, managed by its workload controller.
_Avoid_: Handover, node drain
19 changes: 17 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ someone runs them on GKE and AKS.
|---|---|---|
| `targetRef.kind` | `Deployment` | `Deployment` (`apps/v1`) or `Rollout` (`argoproj.io/v1alpha1`) |
| `targetRef.name` | required | The workload to protect, in the same namespace |
| `baselineReplicas` | unset | Opt into capacity management: desired steady-state replicas; changes wait during handover. Supports zero. See [GitOps capacity](docs/gitops-capacity.md) |
| `surgeReplicas` | 1 | How many stand-ins to add |
| `minSurgeTimeSeconds` | 60 | How long this workload realistically takes to start serving. If a termination deadline is nearer than this, the eviction is released immediately instead of being held |
| `readinessDeadlineSeconds` | 600 | How long to hold an eviction with no progress before giving up and letting it through |
Expand Down Expand Up @@ -236,14 +237,28 @@ revision (or the blue/green active revision) count as stand-ins. Ready preview
pods and old revisions cannot release a production pod's eviction hold.

During an update, pause, or abort, Understudy stands down and admits evictions.
If a release begins during an Understudy surge, it restores the original replica
count and relinquishes the hold. Progressive delivery remains Argo's responsibility.
If a release begins during an Understudy surge, it removes the surge and
relinquishes the hold. With `baselineReplicas`, it applies the latest requested
baseline; otherwise it restores the original replica count. Progressive delivery
remains Argo's responsibility.

Argo Rollouts must already be installed. Its API is detected at operator startup;
restart Understudy if you install the Rollout CRD later. Clusters without Argo
Rollouts continue to support Deployments. As with Deployments, another reconciler
(such as an HPA or GitOps tool) must not overwrite the temporary replica increase.

### GitOps capacity

Argo CD keeps Kubernetes aligned with Git; Argo Rollouts manages application
releases. These are separate jobs. For either a Deployment or a Rollout managed
by Argo CD, set `spec.baselineReplicas` on the Understudy resource and delegate
the workload's effective `spec.replicas` to Understudy.

Git capacity changes then converge automatically. During a drain handover,
Understudy preserves its captured baseline plus surge and applies the latest
requested baseline afterward. `BaselineApplied=False` reports a pending change.
See [the complete ownership configuration and lifecycle](docs/gitops-capacity.md).

## How it works

Briefly: answer the eviction at the door with HTTP 429, surge when its node is
Expand Down
23 changes: 23 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

Version-to-version notes. Anything not mentioned here needs no action.

## 0.4.3 to 0.4.4

`spec.baselineReplicas` opts into continuous capacity management. Existing
resources without it retain reactive-only protection. Install the updated
operator and CRD together before enabling it; v0.4.3 does not implement this
contract. [GitOps capacity](docs/gitops-capacity.md) documents the exact Argo
ownership settings and lifecycle for Deployments and Rollouts.

Helm does not upgrade files in `crds/`. Apply the v0.4.4 schema first, then
upgrade the chart and operator:

```sh
kubectl apply --server-side -f \
https://raw.githubusercontent.com/mitis-cloud/understudy/v0.4.4/charts/understudy/crds/apps.understudy.sh_understudies.yaml

helm upgrade understudy oci://ghcr.io/kylan11/charts/understudy \
--namespace understudy-system --version 0.4.4
```

Then configure the desired baseline and Argo ignore rules together. Do not
enable only the ignore rules: without the new
baseline input, intentional Git capacity changes would still be ignored.

## 0.4.2 to 0.4.3

Argo Rollouts are now supported with `targetRef.kind: Rollout`. Both canary and
Expand Down
10 changes: 10 additions & 0 deletions api/v1alpha1/understudy_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ const (
ConditionProtected = "Protected"
ConditionSurgeReady = "SurgeReady"
ConditionZeroDowntimeStrategy = "ZeroDowntimeUpdateStrategy"
ConditionBaselineApplied = "BaselineApplied"
)

type TargetReference struct {
Expand All @@ -62,6 +63,11 @@ type UnderstudySpec struct {
// +required
TargetRef TargetReference `json:"targetRef"`

// BaselineReplicas opts into capacity management; changes wait during handover.
// +kubebuilder:validation:Minimum=0
// +optional
BaselineReplicas *int32 `json:"baselineReplicas,omitempty"`

// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
// +optional
Expand All @@ -85,6 +91,10 @@ type UnderstudySpec struct {
}

type UnderstudyStatus struct {
// BaselineReplicas is the applied baseline, frozen while a handover is active.
// +optional
BaselineReplicas *int32 `json:"baselineReplicas,omitempty"`

// +optional
Phase Phase `json:"phase,omitempty"`

Expand Down
10 changes: 10 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions charts/understudy/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ description: >-
downtime from announced node disruptions (drains, upgrades, consolidation,
spot reclaims) without a standing second replica.
type: application
version: 0.4.3
appVersion: "0.4.3"
version: 0.4.4
appVersion: "0.4.4"
kubeVersion: ">=1.34.0-0"
keywords:
- availability
Expand All @@ -17,6 +17,7 @@ keywords:
- karpenter
- cluster-autoscaler
- argo-rollouts
- gitops
home: https://github.com/kylan11/understudy
sources:
- https://github.com/kylan11/understudy
Expand All @@ -41,7 +42,7 @@ annotations:
url: https://github.com/kylan11/understudy/issues
artifacthub.io/images: |
- name: understudy
image: ghcr.io/kylan11/understudy:0.4.3
image: ghcr.io/kylan11/understudy:0.4.4
artifacthub.io/crds: |
- kind: Understudy
version: v1alpha1
Expand All @@ -65,6 +66,8 @@ annotations:
readinessDeadlineSeconds: 600
artifacthub.io/changes: |
- kind: added
description: Protect fully promoted Argo Rollouts with canary and blue/green strategies using targetRef.kind Rollout
description: Optional baselineReplicas separates Git-owned desired capacity from effective Deployment and Rollout replicas, deferring capacity changes until handover completes
- kind: added
description: BaselineApplied status and documented Argo CD field delegation expose and preserve pending capacity changes
- kind: fixed
description: Admit evictions while standing down for progressive delivery and exclude preview revisions from Rollout replacement readiness
description: Detect Deployment releases during an active surge and apply the latest baseline when application releases take over
11 changes: 11 additions & 0 deletions charts/understudy/crds/apps.understudy.sh_understudies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ spec:
type: object
spec:
properties:
baselineReplicas:
description: BaselineReplicas opts into capacity management; changes
wait during handover.
format: int32
minimum: 0
type: integer
holdMode:
enum:
- always
Expand Down Expand Up @@ -97,6 +103,11 @@ spec:
type: object
status:
properties:
baselineReplicas:
description: BaselineReplicas is the applied baseline, frozen while
a handover is active.
format: int32
type: integer
blockedSince:
format: date-time
type: string
Expand Down
11 changes: 11 additions & 0 deletions config/crd/bases/apps.understudy.sh_understudies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ spec:
type: object
spec:
properties:
baselineReplicas:
description: BaselineReplicas opts into capacity management; changes
wait during handover.
format: int32
minimum: 0
type: integer
holdMode:
enum:
- always
Expand Down Expand Up @@ -97,6 +103,11 @@ spec:
type: object
status:
properties:
baselineReplicas:
description: BaselineReplicas is the applied baseline, frozen while
a handover is active.
format: int32
type: integer
blockedSince:
format: date-time
type: string
Expand Down
27 changes: 27 additions & 0 deletions docs/adr/0001-declarative-baseline-capacity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
status: accepted
---

# Separate desired baseline from effective replicas

GitOps synchronization and Understudy's temporary surge both write workload
replicas. Permanently ignoring replicas prevents Git capacity changes, while
conditional ignores do not preserve the surge through Argo CD 3.5.5's sync
preprocessing. We therefore add opt-in `spec.baselineReplicas`: Git owns the
desired baseline, and Understudy continuously reconciles effective replicas.

During handover, the captured baseline remains the readiness threshold and the
latest desired baseline waits in the resource's spec. Completion, cancellation,
or release takeover applies that latest value. Application release stand-down
disables disruption surging and holding, not ordinary baseline reconciliation.

Argo delegates the effective replica field and Understudy's handover annotations
using its native, statically scoped `ignoreDifferences` and
`RespectIgnoreDifferences=true`. No synchronization wrapper or additional
controller is introduced. Omitting the baseline retains reactive-only
protection; the opt-in contract requires a single capacity writer.

This boundary expansion was approved in Q38, originating from DBRID-2851's
service-chart integration requirement. The operator owns its implementation;
consumer charts derive the desired baseline and first-install replica count
from the same service capacity value.
129 changes: 129 additions & 0 deletions docs/gitops-capacity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# GitOps capacity contract

Available starting with **Understudy v0.4.4**, Helm chart **0.4.4**, and its
`apps.understudy.sh/v1alpha1` CRD. Upgrade the CRD and operator together;
v0.4.3 does not support the field.

Git supplies the **desired baseline**. Understudy owns **effective replicas**:
the normal baseline plus temporary capacity while handing over departing pods.
This applies equally to Kubernetes Deployments and Argo Rollouts.

## Configure the two owners

Enable capacity management on the Understudy resource:

```yaml
apiVersion: apps.understudy.sh/v1alpha1
kind: Understudy
metadata:
name: worker
namespace: workers
spec:
targetRef:
kind: Deployment # or Rollout
name: worker
baselineReplicas: 1
```

In the Argo CD Application, delegate the effective field and handover markers
for that specific target:

```yaml
spec:
ignoreDifferences:
- group: apps # argoproj.io for a Rollout
kind: Deployment # Rollout for an Argo Rollout
name: worker
namespace: workers
jsonPointers:
- /spec/replicas
- /metadata/annotations/understudy.sh~1base-replicas
- /metadata/annotations/understudy.sh~1target-revision
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
```

Do not ignore `Understudy.spec.baselineReplicas`. Argo continues to synchronize
that input, application releases, and unrelated settings normally. The ignore
rules are static and scoped to the protected target: no conditional JQ,
managed-field ownership inference, or manual sync pause is needed.

Consumer charts should render the workload's initial `spec.replicas` and the
Understudy's `spec.baselineReplicas` from **one service capacity value**. On first
creation, Argo applies the workload as rendered; `RespectIgnoreDifferences`
only preserves ignored fields on resources that already exist. Understudy then
corrects any mismatch once both resources exist. A CR created before its target
waits for the target to appear.

## Lifecycle

| Situation | Effective replicas and protection |
|---|---|
| No handover | Understudy reconciles effective replicas to the desired baseline, including live drift. |
| Handover starts | Captures the currently applied capacity as the handover baseline and adds `surgeReplicas`. |
| Baseline changes during handover | Keeps the captured baseline and surge; the latest requested value waits in the CR spec. Intermediate requests are superseded. |
| Replacement becomes ready | The readiness threshold remains the captured baseline, not a pending larger or smaller request. |
| Departing pods leave, or the disruption clears | Removes handover markers and applies the latest desired baseline. |
| Readiness deadline expires | Relaxes the eviction hold; keeps surge capacity until departing pods leave or the disruption clears, then applies the latest baseline. |
| Application release starts during handover | Ends the disruption surge, clears deletion-cost hints, admits evictions, and applies the latest baseline. |
| Application release is updating, paused, or aborted | Disruption protection stands down; ordinary baseline reconciliation continues. Argo Rollouts still owns promotion and traffic routing. |
| Desired baseline is zero | Scales to zero. A zero requested during handover waits like any other change; an already-zero workload does not start a disruption surge. |

The baseline and revision captured for handover live in annotations on the
target. The pending request lives in `Understudy.spec.baselineReplicas`, so a
lost status update cannot discard it. On restart the operator reconstructs
disruption signals and resumes reconciliation. Webhook availability still has
the existing fail-open behavior while the operator is unavailable.

## Observe the result

- `spec.baselineReplicas`: latest Git request.
- `status.baselineReplicas`: applied baseline, frozen during handover.
- `BaselineApplied=False`, reason `HandoverInProgress`: desired baseline is pending.
- `BaselineApplied=True`: baseline has been applied to the workload's replica
request. This does **not** claim that every requested pod is Ready.
- `SurgeReady` and `Protected`: replacement readiness and eviction-hold behavior.
- Target `spec.replicas`: effective replicas, including a possible surge.

Conditions include `observedGeneration`, so a client can distinguish the latest
request from an older status observation.

## Enable and retire ownership

Use one Understudy resource per target and one capacity writer. An HPA, KEDA,
another Understudy resource, or another GitOps application must not concurrently
write that target's effective replicas. Workloads must still tolerate temporary
overlap; baseline management does not make overlap-unsafe singletons safe.

Omitting `baselineReplicas` keeps reactive-only protection. Removing it stops
continuous baseline reconciliation; an active handover finishes at its captured
baseline. Deleting the CR does not restore replicas: it removes the controller's
input. To retire protection, finish or cancel the handover first, align the Git
replica count with the intended capacity, then remove the CR and its Argo ignore
rules together. Keep a target reference fixed through handover; retargeting a CR
does not clean up the old target.

The documented Argo configuration uses its ordinary apply synchronization path.
Force/replace synchronization is outside this contract because it can recreate
resources and discard the runtime state. Argo's native documentation explains
[RespectIgnoreDifferences](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-differences-configs).

For Helm upgrades, update the CRD explicitly before using the new field: Helm
does not upgrade CRDs in a chart's `crds/` directory. See [upgrade notes](../UPGRADING.md).

## Verified integration

Validated with Argo CD **3.5.5**, Argo Rollouts **1.10.1**, and Kubernetes
**1.36.4** in isolated Kind clusters, with automatic sync and self-heal enabled.
Deployment, canary, and blue/green targets passed initial installation,
unrelated sync during surge, repeated pending baseline updates, cold operator
restart and leadership recovery, eviction hold/release, final convergence,
steady scaling, HTTP serving, and release takeover. Blue/green also converged a
pending scale-to-zero request and subsequently scaled up from Git.

The test used the ordinary Argo apply path and the scoped ignore rules above.
Server-side apply and force/replace sync were not part of this verification.
Loading
Loading