Skip to content

Restore registry-server startup probe FailureThreshold to 15 #3876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emmahone wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Restore registry-server startup probe FailureThreshold to 15 #3876

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/controller/registry/reconciler/reconciler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ func Pod(source *operatorsv1alpha1.CatalogSource, name, opmImg, utilImage, img s
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,

@wking wking Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restores StartupProbe.FailureThreshold for the registry-server container from 10 back to 15, returning the startup budget from 100s to 150s...

But in #2919:

Also adjusted the failure threshold in order to maintain the ~150s startup time failure as originally designed (timeout of 5s + period of 10s = 15s per attempt * 10 failures = 150s).

So maybe you're going up to timeout of 5s + period of 10s = 15s per attempt * 15 failures = 225s)? I'm not clear on what "as originally designed" is talking about though, so not clear what the downsides would be of going too high here.

@emmahone emmahone Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today with the probe interval of 10s and failure threshold of 10, my understanding is the 10th probe is sent out at the 90s mark and times out at 5s. So technically the time to failure is 95s. After this change the time to failure would be 145s.

Versus what occurred prior to 401bfff where the last probe was sent out at 140s and times out based on the default timeout.

EDIT: Just to clarify, 'as originally designed' is just quoted from the commit message in 401bfff. The tuning of these probe interval and failure threshold is fairly arbitrary which is why I am pushing for increasing them slightly rather than simply telling customers to fix the underlying issues causing these issues and ignoring these values. This gives us some resiliency with a small cost in time to failure.

-mahoney

PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down
32 changes: 16 additions & 16 deletions pkg/controller/registry/reconciler/reconciler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestPodMemoryTarget(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "KpDeU9UesY5GlqG7ZUg9deo8Jp7mEGdpBhGJn", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "9ypiMzzIhmehTVss6ziefzhUy1oqi5HaKuuKnC", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -71,7 +71,7 @@ func TestPodMemoryTarget(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -110,7 +110,7 @@ func TestPodMemoryTarget(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "1Cl0FkZ1FdtjQ91Ojcj7pM0MphZFlOuLPtz2Cu", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "8YO5HNCBmJmib560diQLiRqZvRvoaKhF1wrPQd", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -144,7 +144,7 @@ func TestPodMemoryTarget(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -209,7 +209,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "KpDeU9UesY5GlqG7ZUg9deo8Jp7mEGdpBhGJn", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "9ypiMzzIhmehTVss6ziefzhUy1oqi5HaKuuKnC", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -242,7 +242,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -285,7 +285,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "9SbEBzbV5JmBIA6zza29T4lIo0ESVJ8SN6slOY", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "1qbvCkU20Obt17Z3B6bQtBeGyIm5wbpv5bkpgg", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -363,7 +363,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -406,7 +406,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "1B9AFU7EIoI0CgRaHbyBL3EnGzwrkBq968SSps", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "2UwfmcYOWijz1pC3xapgQnKK3yRkVFhZN1clrb", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -486,7 +486,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -522,7 +522,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "8qB6OcFt60v8HdhXnPkB1cjF39t7RkFx9K0JxW", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "2iRFHadLZxZOhZIrrbv9BWV8iBMZTzoebGNVbG", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -555,7 +555,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -605,7 +605,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "3xuLPXGJ2pzekw21PFU68XUKOYc7PTuW45M521", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "5Snaq6foF8dkSrxPHZGSAHm0497jiRrAp6nUT2", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -687,7 +687,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down Expand Up @@ -737,7 +737,7 @@ func TestPodExtractContent(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{
GenerateName: "test-",
Namespace: "testns",
Labels: map[string]string{"olm.pod-spec-hash": "7noQSgGmkI4BD1MPKe0pEFFfOE3jJtN2DUyZuD", "olm.managed": "true"},
Labels: map[string]string{"olm.pod-spec-hash": "czwUQnRXkqTsMLZO4QJUEAicYEM4O5ZCXagHS8", "olm.managed": "true"},
Annotations: map[string]string{"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -821,7 +821,7 @@ func TestPodExtractContent(t *testing.T) {
Command: []string{"grpc_health_probe", "-addr=:50051", "-rpc-timeout=5s"},
},
},
FailureThreshold: 10,
FailureThreshold: 15,
PeriodSeconds: 10,
TimeoutSeconds: 5,
},
Expand Down
Loading