Skip to content

fix: stop committed resource controller reconcile storm #1089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juliusclausnitzer wants to merge 1 commit into
base: main
Choose a base branch
from
+51 −0
Open

fix: stop committed resource controller reconcile storm #1089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions internal/scheduling/reservations/commitments/committed_resource_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,16 @@ func (r *CommittedResourceController) checkChildReservationStatus(ctx context.Co
}

func (r *CommittedResourceController) setAccepted(ctx context.Context, cr *v1alpha1.CommittedResource) error {
// Skip the patch when the CR is already accepted for the current generation.
// Without this guard, every reconcile writes a new AcceptedAt timestamp and a
// time-varying StatusSummary, triggering the self-watch and causing a reconcile storm.
cond := meta.FindStatusCondition(cr.Status.Conditions, v1alpha1.CommittedResourceConditionReady)
if cond != nil &&
cond.Status == metav1.ConditionTrue &&
cond.Reason == v1alpha1.CommittedResourceReasonAccepted &&
cond.ObservedGeneration == cr.Generation {
return nil
}
now := metav1.Now()
old := cr.DeepCopy()
specCopy := cr.Spec.DeepCopy()
Expand Down
41 changes: 41 additions & 0 deletions internal/scheduling/reservations/commitments/committed_resource_controller_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1165,3 +1165,44 @@ func TestCheckChildReservationStatus_GenerationGuard(t *testing.T) {
})
}
}

func TestCommittedResourceController_SetAcceptedIdempotent(t *testing.T) {
// Reconciling an already-accepted cores CR must not patch the status again.
// Without the idempotency guard, every reconcile writes a new AcceptedAt timestamp,
// which triggers the self-watch and causes a reconcile storm.
scheme := newCRTestScheme(t)
cr := newTestCoresCR("test-cr", v1alpha1.CommitmentStatusConfirmed, 4, false)
fgc := newTestFlavorGroupCapacity("test-group", "test-az", 16)
k8sClient := newCRTestClient(scheme, cr, fgc)
Comment on lines +1173 to +1176

@coderabbitai coderabbitai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Exercise a non-zero generation in this test.

The fixture does not explicitly set metadata.generation, so a zero-generation test client could allow an implementation that incorrectly records ObservedGeneration=0 to pass. Seed cr.Generation = 1 and assert the accepted condition’s ObservedGeneration after the first reconcile.

Suggested test hardening 
 cr := newTestCoresCR("test-cr", v1alpha1.CommitmentStatusConfirmed, 4, false)
+cr.Generation = 1
 fgc := newTestFlavorGroupCapacity("test-group", "test-az", 16)
...
 rv1 := after1.ResourceVersion
+cond := meta.FindStatusCondition(after1.Status.Conditions, v1alpha1.CommittedResourceConditionReady)
+if cond == nil || cond.ObservedGeneration != after1.Generation {
+	t.Fatalf("expected ObservedGeneration=%d, got %#v", after1.Generation, cond)
+}

Also applies to: 1184-1194

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@internal/scheduling/reservations/commitments/committed_resource_controller_test.go`
around lines 1173 - 1176, Update the test setup around newTestCoresCR and the
first reconcile to set cr.Generation to 1 before creating the test client, then
assert that the accepted condition’s ObservedGeneration equals 1 after
reconciliation. Keep the existing condition assertions and test flow unchanged.


controller := &CommittedResourceController{
Client: k8sClient,
Scheme: scheme,
Conf: CommittedResourceControllerConfig{RequeueIntervalRetry: metav1.Duration{Duration: 1 * time.Minute}},
}

// First reconcile: CR transitions to Accepted and status is written.
if _, err := controller.Reconcile(context.Background(), reconcileReq(cr.Name)); err != nil {
t.Fatalf("first reconcile: %v", err)
}
assertCondition(t, k8sClient, cr.Name, metav1.ConditionTrue, v1alpha1.CommittedResourceReasonAccepted)

var after1 v1alpha1.CommittedResource
if err := k8sClient.Get(context.Background(), types.NamespacedName{Name: cr.Name}, &after1); err != nil {
t.Fatalf("get after first reconcile: %v", err)
}
rv1 := after1.ResourceVersion

// Second reconcile: CR is already accepted — must not patch status.
if _, err := controller.Reconcile(context.Background(), reconcileReq(cr.Name)); err != nil {
t.Fatalf("second reconcile: %v", err)
}

var after2 v1alpha1.CommittedResource
if err := k8sClient.Get(context.Background(), types.NamespacedName{Name: cr.Name}, &after2); err != nil {
t.Fatalf("get after second reconcile: %v", err)
}
if after2.ResourceVersion != rv1 {
t.Errorf("ResourceVersion changed on second reconcile (%s → %s): status was patched unnecessarily", rv1, after2.ResourceVersion)
}
}