Skip to content

[Debug] Study: e2-standard-4 workers + e2-standard-4 control plane for serial GCP jobs - #82856

Open
sdodson wants to merge 1 commit into
openshift:mainfrom
sdodson:gcp-e2-standard-4-worker-serial-cp-study
Open

[Debug] Study: e2-standard-4 workers + e2-standard-4 control plane for serial GCP jobs#82856
sdodson wants to merge 1 commit into
openshift:mainfrom
sdodson:gcp-e2-standard-4-worker-serial-cp-study

Conversation

@sdodson

@sdodson sdodson commented Aug 4, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Member

DO NOT MERGE - rehearsal-only study branch

Follow-up to the control-plane sizing investigation (#82754, #82852). A survey found:

  • GCP workers across many job types average well under 50% CPU utilization.
  • Serial (non-parallel test execution) GCP jobs show ~35-41% lower control-plane CPU than parallel counterparts, and in every single-run sample checked, spent 0 seconds above the 95% CPU threshold on both control-plane and worker nodes (vs up to 210s for parallel counterparts).
  • However, etcd disk-latency (fsync/backend-commit) does not reliably track CPU/parallelism - one serial/non-serial pair showed serial with worse latency despite lower CPU, suggesting disk latency is more storage/zone-driven than parallelism-driven.

This branch empirically tests via rehearsal:

  1. Whether GCP worker nodes across the board (serial and non-serial jobs) can be downsized from the default e2-custom-6-16384 (6 vCPU) to e2-standard-4 (4 vCPU) without introducing test or performance regressions.
  2. Whether, additionally, SERIAL jobs' control planes specifically can also be downsized to e2-standard-4, given their markedly lower measured CPU load - while deliberately leaving non-serial jobs' control planes untouched, since parallel jobs generate the etcd/apiserver load that motivated the 8-vCPU default bump in gcp: change default control-plane instance size from custom-6-16384 to custom-8-16384 #82852.

Jobs under study (4 serial + 4 non-serial, 4 release branches)

Job Variant Serial? Change
e2e-gcp-ovn-rhcos10-techpreview ci-5.0 No workers -> e2-standard-4
e2e-gcp-ovn-techpreview ci-4.15 No workers -> e2-standard-4
e2e-gcp-ovn-techpreview-serial ci-4.15 Yes workers + control plane -> e2-standard-4
e2e-gcp-ovn-runc nightly-4.22 No workers -> e2-standard-4
e2e-gcp-ovn-serial-runc nightly-4.22 Yes workers + control plane -> e2-standard-4
e2e-gcp-ovn-runc nightly-5.0 No workers -> e2-standard-4
e2e-gcp-ovn-serial-runc nightly-5.0 Yes workers + control plane -> e2-standard-4
e2e-gcp-ovn-serial nightly-5.0 Yes workers + control plane -> e2-standard-4

Note: e2e-gcp-ovn-rhcos10-techpreview-serial (ci-5.0) was excluded - it has been failing before reaching the e2e-test phase for weeks, unrelated to sizing, so rehearsing it would produce no usable data.

Plan

4 rehearsal rounds across all 8 jobs (running in parallel each round), comparing etcd/CPU metrics against each job's own recent baseline (already surveyed) plus the shared e2-custom-6-16384-workers / current-control-plane-default baseline.

Not rehearsed yet as of PR creation - rehearsals will be triggered via /pj-rehearse comments.

Summary by CodeRabbit

  • Updates GCP OpenShift CI rehearsal configurations across 4.15, 4.22, and 5.0 release jobs.
  • Changes selected worker nodes from e2-custom-6-16384 to e2-standard-4.
  • Applies e2-standard-4 to control-plane nodes for serial jobs only.
  • Keeps control-plane sizing unchanged for non-serial jobs.
  • Supports comparison of CPU usage, etcd latency, and test or performance regressions across the rehearsal rounds.

@sdodson
[Debug] Study: e2-standard-4 workers (all) + e2-standard-4 control pl…
181da44 
…ane (serial only)

DO NOT MERGE. Rehearsal-only study branch.

Follow-up to the control-plane sizing investigation (PR openshift#82754, openshift#82852).
A survey found GCP workers across many job types average well under
50% CPU utilization, and serial test-execution jobs specifically show
markedly lower control-plane CPU (and zero measured time above the
95% CPU threshold across control plane, and workers, in every sample)
compared to their parallel counterparts. This branch tests whether
worker nodes can be downsized to e2-standard-4 (4 vCPU) across the
board, and whether serial jobs' control planes can also be downsized
to e2-standard-4, without introducing etcd disk-latency or test
regressions.

8 jobs across 4 release variants, 4 serial + 4 non-serial:
- ci-5.0: e2e-gcp-ovn-rhcos10-techpreview (non-serial, workers only)
- ci-4.15: e2e-gcp-ovn-techpreview (non-serial, workers only)
          e2e-gcp-ovn-techpreview-serial (serial, workers+CP)
- nightly-4.22: e2e-gcp-ovn-runc (non-serial, workers only)
               e2e-gcp-ovn-serial-runc (serial, workers+CP)
- nightly-5.0: e2e-gcp-ovn-runc (non-serial, workers only)
              e2e-gcp-ovn-serial-runc (serial, workers+CP)
              e2e-gcp-ovn-serial (serial, workers+CP)

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
openshift-ci Bot requested review from stbenjam and xueqzhan August 4, 2026 00:39
@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 108b7211-5ba5-4a5e-9166-f1190f4561a1

📥 Commits

Reviewing files that changed from the base of the PR and between c7a033c and 181da44.

📒 Files selected for processing (4)
  • ci-operator/config/openshift/release/openshift-release-main__ci-4.15.yaml
  • ci-operator/config/openshift/release/openshift-release-main__ci-5.0.yaml
  • ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml
  • ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yaml

Walkthrough

Changes

The CI and nightly GCP test configurations now set e2-standard-4 for compute nodes and, where specified, control-plane nodes.

GCP node sizing

Layer / File(s) Summary
Configure GCP test node types
ci-operator/config/openshift/release/openshift-release-main__ci-4.15.yaml, ci-operator/config/openshift/release/openshift-release-main__ci-5.0.yaml, ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml, ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yaml		 GCP TechPreview, OVN, runC, and RHCOS test configurations add e2-standard-4 compute and control-plane node settings where applicable.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested labels: rehearsals-ack

Suggested reviewers: kasturinarra

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the study's GCP worker downsizing and serial control-plane changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only four YAML job configurations; no Ginkgo It, Describe, Context, or When test titles are added or modified.
Test Structure And Quality ✅ Passed The commit changes only four CI YAML files with machine-type environment entries; it adds no Ginkgo It blocks, resource setup, waits, or assertions to review.
Microshift Test Compatibility ✅ Passed The pull request changes only four YAML release configurations and adds node-type environment values; it adds no Ginkgo e2e tests or MicroShift-incompatible API usage.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The commit changes only four YAML job configurations (+15 lines); it adds no Ginkgo e2e tests, so SNO compatibility checks are not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The commit changes only four CI YAML env blocks; the variables select GCP machine types, and the diff adds no pod, affinity, topology, node-selector, toleration, replica, or PDB constraints.
Ote Binary Stdout Contract ✅ Passed The PR changes only four YAML CI configuration files; it adds no Go/OTE source, suite setup, or stdout/logging writes.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR changes only four YAML release configurations by adding node-type environment values; it adds no Ginkgo tests or networking/external-connectivity code.
No-Weak-Crypto ✅ Passed The PR adds only GCP node-type YAML settings; the added lines contain no weak-crypto algorithms, crypto implementations, or secret comparisons.
Container-Privileges ✅ Passed The PR adds only GCP node-type environment variables; the added lines contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed The patch changes only four YAML files by adding GCP machine-type values; it adds no logging or credentials, and existing logs expose only non-sensitive node types.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 4, 2026
@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@sdodson: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc N/A periodic Ci-operator config changed
periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview N/A periodic Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview 181da44 link unknown /pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview
ci/rehearse/periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc 181da44 link unknown /pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc
ci/rehearse/periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc 181da44 link unknown /pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview periodic-ci-openshift-release-main-ci-4.15-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial-runc periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-rhcos10-techpreview

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sdodson

sdodson commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-serial

@openshift-merge-bot

openshift-merge-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@sdodson: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stbenjam stbenjam Awaiting requested review from stbenjam
@xueqzhan xueqzhan Awaiting requested review from xueqzhan

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sdodson