[Debug] Test e2-standard-8 control plane for gcp-ovn-rt-upgrade by sdodson · Pull Request #82754 · openshift/release

sdodson · 2026-07-31T18:41:58Z

Summary

Overrides CONTROL_PLANE_NODE_TYPE from the default e2-custom-6-16384 to e2-standard-8 for e2e-gcp-ovn-rt-upgrade.
Purpose: test whether the extra CPU headroom (8 vs 6 vCPU) and higher persistent-disk throughput ceiling (800 vs 240 MiB/s) reduce the etcd disk-latency SLO breaches and control-plane CPU saturation found in prior investigation of this job's flakiness (see the case *) comment in ipi-conf-gcp-commands.sh noting e2-custom-6-16384 was already a "temporary test" for the same symptom).
Will trigger a rehearsal to collect comparison data (etcd fsync/commit latency, CPU p90) against the current default.

⚠️ DO NOT MERGE — rehearsal-only experiment, will be closed after data collection.

Summary by CodeRabbit

The e2e-gcp-ovn-rt-upgrade OpenShift CI rehearsal will use e2-custom-8-16384 control-plane nodes.
The rehearsal evaluates whether eight vCPUs with 16 GB memory reduce etcd latency SLO breaches and control-plane CPU saturation.
The change is for testing only and must not be merged.

Overrides CONTROL_PLANE_NODE_TYPE from the default e2-custom-6-16384 to e2-standard-8 to test whether the extra CPU headroom (8 vs 6 vCPU) and higher persistent-disk throughput ceiling (800 vs 240 MiB/s) reduce the etcd disk-latency SLO breaches and control-plane CPU saturation found in prior investigation of this job's flakiness. DO NOT MERGE - rehearsal-only experiment, will be closed after data collection. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED

openshift-ci · 2026-07-31T18:42:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift/release/OWNERS~~ [sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-07-31T18:42:43Z

Caution

Review failed

An error occurred during the review process. Please try again later.

Walkthrough

The GCP OVN realtime upgrade test now sets CONTROL_PLANE_NODE_TYPE to e2-custom-8-16384.

Changes

GCP OVN upgrade test

Layer / File(s)	Summary
Configure control plane node type `ci-operator/config/openshift/release/openshift-release-main__ci-5.0-upgrade-from-stable-4.22.yaml`	The test environment sets `CONTROL_PLANE_NODE_TYPE` to `e2-custom-8-16384`.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested labels: rehearsals-ack

Suggested reviewers: sosiouxme, xueqzhan

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title names `e2-standard-8`, but the change sets `CONTROL_PLANE_NODE_TYPE` to `e2-custom-8-16384`.	Update the title to reference `e2-custom-8-16384` or change the configuration to match the stated `e2-standard-8` test.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The patch changes only CONTROL_PLANE_NODE_TYPE; it adds no Ginkgo title and the existing e2e-gcp-ovn-rt-upgrade identifier is static.
Test Structure And Quality	✅ Passed	The PR changes only one YAML environment value; it adds no Ginkgo It blocks or test source to review for setup, waits, assertions, or cleanup.
Microshift Test Compatibility	✅ Passed	The PR changes only a CI YAML node-type value; it adds no Ginkgo tests or test declarations that require MicroShift compatibility review.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The commit changes only the existing YAML job's CONTROL_PLANE_NODE_TYPE; it adds no Ginkgo It, Describe, Context, or When test, so the SNO check does not apply.
Topology-Aware Scheduling Compatibility	✅ Passed	The only change is a CI test environment value, CONTROL_PLANE_NODE_TYPE; it adds no deployment manifest, operator code, controller, or scheduling constraint.
Ote Binary Stdout Contract	✅ Passed	The PR changes only one YAML environment value; it adds no OTE binary or process-level Go stdout code, so the stdout contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The patch changes only a YAML test configuration value; it adds no Ginkgo test, IPv4 assumption, or external connectivity requirement.
No-Weak-Crypto	✅ Passed	The commit only changes a YAML node-type value; no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparison is introduced.
Container-Privileges	✅ Passed	The only code change sets CONTROL_PLANE_NODE_TYPE; the changed manifest and referenced workflow contain no privileged, host namespace, SYS_ADMIN, or escalation settings.
No-Sensitive-Data-In-Logs	✅ Passed	The only changed line sets CONTROL_PLANE_NODE_TYPE to the non-sensitive machine type e2-custom-8-16384; no logging or credential, PII, hostname, or customer data was added.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

sdodson · 2026-07-31T18:45:46Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-07-31T18:45:48Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-01T00:27:49Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-01T00:27:52Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-01T05:49:43Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-01T05:49:46Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-01T11:19:45Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-01T11:19:48Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Switches CONTROL_PLANE_NODE_TYPE from e2-standard-8 (which showed strong improvement across 4 rehearsal runs) to n2-standard-4, to test the low end of the sizing curve. Expected to perform worse than the current e2-custom-6-16384 baseline: n2-standard-4 shares the same disk I/O ceiling (2-7 vCPU tier: 15,000 IOPS / 240 MiB/s) but has less CPU headroom (4 vCPU) than the already CPU-constrained 6 vCPU baseline. Running anyway to confirm the CPU-headroom theory from the other direction and establish the low end of the cost/performance curve. DO NOT MERGE - rehearsal-only experiment. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED

sdodson · 2026-08-01T16:35:15Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-01T16:35:18Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-01T22:03:06Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-01T22:03:09Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-02T03:05:39Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-02T03:05:41Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-02T03:31:44Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-02T03:31:46Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-02T09:07:02Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-02T09:07:04Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Third series in the control-plane sizing campaign. n2-standard-4 (4-run avg) performed clearly worse than the e2-custom-6-16384 baseline across CPU saturation and etcd disk-latency metrics (2/4 runs failed), confirming the CPU-headroom theory from the low end. e2-standard-8 (previous series, 4/4 passed) performed clearly better. This series tests e2-custom-8-16384: same 8 vCPU as e2-standard-8 but keeping the 16GB memory ratio matching the current e2-custom-6-16384 baseline, to isolate whether the memory ratio matters independent of vCPU count. DO NOT MERGE - rehearsal-only experiment. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED

sdodson · 2026-08-02T14:38:37Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-02T14:38:40Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-02T20:07:15Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-02T20:07:17Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-03T01:38:05Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-03T01:38:08Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-03T06:37:12Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-03T06:37:15Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

DO NOT MERGE. Part of an ongoing rehearsal campaign to find a better control-plane instance type for the flaky e2e-gcp-ovn-rt-upgrade job. This is the final series (n2-custom-8-16384) after n2-standard-4 and e2-custom-8-16384. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED

sdodson · 2026-08-03T11:38:50Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-03T11:38:52Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-08-03T11:42:27Z

[REHEARSALNOTIFIER]
@sdodson: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade	N/A	periodic	Ci-operator config changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

sdodson · 2026-08-03T16:37:17Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-03T16:37:39Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

sdodson · 2026-08-03T16:48:42Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade

openshift-merge-bot · 2026-08-03T16:48:44Z

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci · 2026-08-03T22:11:46Z

@sdodson: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade	`9dbf7f2`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

…o custom-8-16384 The default e2-custom-6-16384 (6 vCPU) GCP control-plane sizing has been associated with consistent high CPU alerts and elevated etcd disk-latency (WAL fsync / backend commit) across many GCP e2e jobs, contributing to flaky, non-reproducible test failures during upgrade and general e2e runs. An empirical rehearsal campaign on PR openshift#82754 (12 rehearsal runs across n2-standard-4, e2-standard-8, e2-custom-8-16384, and n2-custom-8-16384) found that control-plane vCPU count, not memory ratio or CPU platform, is the dominant driver of the improvement, and that e2-custom-8-16384 matches or beats e2-standard-8 on etcd/CPU metrics at a lower monthly cost by avoiding e2-standard-8's unneeded extra RAM. This changes the shared default in ipi-conf-gcp-commands.sh so every GCP job across every release branch that doesn't already set a larger CONTROL_PLANE_NODE_TYPE picks up the new sizing automatically, with no per-job config changes required. The 3 existing per-job overrides (nightly-4.21/4.22/5.0 e2e-gcp-custom-dns -> e2-custom-8-32768) are already >= 8 vCPU and are left untouched. The arm64 path (t2a-standard-4) is unaffected - custom sizes aren't supported on arm64, and this change was only validated on amd64 (e2/n2 GCP host pools). rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2026

openshift-ci Bot requested review from sosiouxme and xueqzhan July 31, 2026 18:42

sdodson mentioned this pull request Aug 3, 2026

gcp: change default control-plane instance size from custom-6-16384 to custom-8-16384 #82852

Open

4 tasks

sdodson mentioned this pull request Aug 4, 2026

[Debug] Study: e2-standard-4 workers + e2-standard-4 control plane for serial GCP jobs #82856

Open

Uh oh!

Conversation

sdodson commented Jul 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Summary by CodeRabbit

Uh oh!

openshift-ci Bot commented Jul 31, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

sdodson commented Jul 31, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 31, 2026

Uh oh!

sdodson commented Aug 1, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 1, 2026

Uh oh!

sdodson commented Aug 1, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 1, 2026

Uh oh!

sdodson commented Aug 1, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 1, 2026

Uh oh!

sdodson commented Aug 1, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 1, 2026

Uh oh!

sdodson commented Aug 1, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 1, 2026

Uh oh!

sdodson commented Aug 2, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 2, 2026

Uh oh!

sdodson commented Aug 2, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 2, 2026

Uh oh!

sdodson commented Aug 2, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 2, 2026

Uh oh!

sdodson commented Aug 2, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 2, 2026

Uh oh!

sdodson commented Aug 2, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 2, 2026

Uh oh!

sdodson commented Aug 3, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 3, 2026

Uh oh!

sdodson commented Aug 3, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 3, 2026

Uh oh!

sdodson commented Aug 3, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 3, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 3, 2026

Uh oh!

sdodson commented Aug 3, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 3, 2026

Uh oh!

sdodson commented Aug 3, 2026

Uh oh!

sdodson commented Jul 31, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading