gcp: change default control-plane instance size from custom-6-16384 to custom-8-16384 - #82852
gcp: change default control-plane instance size from custom-6-16384 to custom-8-16384#82852sdodson wants to merge 2 commits into
Conversation
openshift-ci
Bot
added
the
needs-rebase
WalkthroughAzure test jobs now use updated compute, control-plane, and additional-worker VM types. GCP fallback and custom DNS tests now use updated machine types. ChangesCloud VM sizing
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5 | ❌ 10❌ Failed checks (10 inconclusive)
✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@sdodson, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh`:
- Around line 41-44: Add representative AMD64 GCP job links and explicit
sign-off evidence for the e2-custom-8-16384 default in the surrounding comment
for master_type_suffix. Retain the existing PR reference and performance
rationale, and ensure the cited runs cover the exact instance type before merge.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d6eb6e1b-93fb-4b25-af3c-18060c46090a
📒 Files selected for processing (12)
ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly-4.22-upgrade-from-stable-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-rollback-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-stable-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-cpou-upgrade-from-4.20.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-upgrade-from-stable-4.21.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly-4.22-upgrade-from-stable-4.22.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yamlci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-stable.yamlci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
| # 8 vCPU / 16GB default: empirically validated (PR openshift/release#82754) to | ||
| # eliminate the CPU saturation / etcd disk-latency pattern seen with the prior | ||
| # 6 vCPU default, at a lower cost than e2-standard-8. | ||
| master_type_suffix="custom-8-16384" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 8 \
'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp \
ci-operator/configRepository: openshift/release
Length of output: 2110
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- repository state ---'
pwd
git status --short --branch
printf '%s\n' '--- target files ---'
git ls-files 'ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh' 'ci-operator/config/**' | head -80
printf '%s\n' '--- relevant references ---'
rg -n -C 8 'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp ci-operator/configRepository: openshift/release
Length of output: 6742
🌐 Web query:
openshift/release PR 82754 GCP custom-8-16384 empirical validation
💡 Result:
There is no evidence of a pull request numbered 82754 in the openshift/release repository [1][2][3][4]. The number 82754 is not associated with an OpenShift release or an empirical validation task for the GCP custom machine type custom-8-16384. While custom-8-16384 refers to a standard Google Cloud Platform (GCP) custom instance type configuration with 8 vCPUs and 16 GiB (16384 MB) of memory [5][6], it is a generic resource specification rather than a specific OpenShift validation target linked to the requested PR number [7]. Publicly available information regarding 82754 unrelated to OpenShift includes references to technical documentation for hardware components [8], security configuration guidelines [9][10], and unrelated software patches [11].
Citations:
- 1: c266a73
- 2: 5996017
- 3: 8514e4e
- 4: a30a9ba
- 5: https://github.com/openshift/openshift-docs/blob/3c1ab270/modules/sdpolicy-am-gcp-compute-types.adoc
- 6: https://discuss.ray.io/t/gcp-custom-machine-type-and-best-practices-for-node-choice/4732
- 7: openshift/installer@f8ab1fb...6723dfd
- 8: https://eu.raptorsupplies.com/pd/volkel/82754
- 9: https://ato-pathways.com/catalogs/references/cce/CCE-82754-3
- 10: https://static.open-scap.org/ssg-guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 11: http://mails.dpdk.org/archives/test-report/2020-October/163416.html
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target script ---'
sed -n '1,100p' ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- repository references ---'
rg -n -C 8 'master_type_suffix|SIZE_VARIANT|custom-6-16384|custom-8-16384' \
ci-operator/step-registry/ipi/conf/gcp ci-operator/config || true
printf '%s\n' '--- related GCP step files ---'
git ls-files 'ci-operator/step-registry/ipi/conf/gcp/*'Repository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
for url in \
'https://api.github.com/repos/openshift/release/pulls/82754' \
'https://api.github.com/repos/openshift/release/issues/82754' \
'https://api.github.com/repos/openshift/release/commits?per_page=100'
do
printf '\n--- %s ---\n' "$url"
curl -LfsS --max-time 20 \
-H 'Accept: application/vnd.github+json' \
"$url" | jq '{message,number,title,state,html_url,merged_at,body}'
doneRepository: openshift/release
Length of output: 274
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target script ---'
cat -n ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- exact symbol references in step registry ---'
rg -n -C 5 'master_type_suffix|custom-6-16384|custom-8-16384' ci-operator/step-registry || true
printf '%s\n' '--- GCP workflow and chain references ---'
rg -n -C 4 'ipi-conf-gcp|ipi-gcp|conf-gcp' ci-operator/step-registry ci-operator/config --glob '*.yaml' --glob '*.json' || true
printf '%s\n' '--- cited identifier in local repository ---'
rg -n '82754' . || trueRepository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- recent history for target script ---'
git log -n 12 --format='%h %ad %s' --date=short -- ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- repository remotes ---'
git remote -vRepository: openshift/release
Length of output: 441
🌐 Web query:
site:github.com/openshift/release/pull/82754
💡 Result:
PR #82754 in openshift/release is an open, rehearsal-only experiment titled:
[Debug] Test e2-standard-8 control plane for gcp-ovn-rt-upgrade
It tests larger GCP control-plane nodes to determine whether additional CPU and disk throughput reduce etcd latency and CPU-saturation issues. It should not be merged. The PR later also tests n2-standard-4 as a lower-sizing comparison. [1] (github.com)
Citations:
🌐 Web query:
"openshift/release#82754"
💡 Result:
There is no public information available regarding a pull request with the number #82754 in the openshift/release repository. Search results for this number returned references to a Common Configuration Enumeration (CCE) identifier, CCE-82754-3, associated with the secure configuration of Red Hat Enterprise Linux CoreOS rather than an OpenShift pull request [1][2]. Other pull requests in the openshift/release repository exist with different identifiers [3][4][5][6][7].
Citations:
- 1: https://static.open-scap.org/ssg-guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 2: https://complianceascode.github.io/content-pages/guides/ssg-rhcos4-guide-anssi_bp28_intermediary.html
- 3: 25e5022
- 4: 8844aed
- 5: 5996017
- 6: 23ab27d
- 7: CNTRLPLANE-3509: Deep-verify release component digests in nodepool resolve step #80576
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- current fallback logic ---'
sed -n '1,90p' ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- change commit ---'
git show --format=fuller --no-ext-diff --unified=25 \
2ca8c5d90 -- ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh
printf '%s\n' '--- exact machine-size references in GCP step files ---'
rg -n 'master_type_suffix|custom-[0-9]+-[0-9]+|e2-standard-8|n2-standard-4' \
ci-operator/step-registry/ipi/conf/gcpRepository: openshift/release
Length of output: 12924
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- commit and parent metadata ---'
git show -s --format='%H%n%P%n%B' 2ca8c5d90
printf '%s\n' '--- parent value ---'
git show 2ca8c5d90^:ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh \
| sed -n '30,55p'Repository: openshift/release
Length of output: 1698
🌐 Web query:
site:github.com/openshift/release/pull/82754 "custom-8-16384"
💡 Result:
No search results were found for:
site:github.com/openshift/release/pull/82754 "custom-8-16384"
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- control-plane override references ---'
rg -n -C 5 'CONTROL_PLANE_NODE_TYPE|e2-custom-8-32768' \
ci-operator/step-registry/ipi ci-operator/config || true
printf '%s\n' '--- files with the claimed override ---'
rg -l 'e2-custom-8-32768' ci-operator/config ci-operator/step-registry || true
printf '%s\n' '--- default-size inputs in the GCP configuration path ---'
rg -n -C 4 'SIZE_VARIANT|CONTROL_ARCH|OCP_ARCH|COMPUTE_NODE_TYPE' \
ci-operator/step-registry/ipi/conf/gcp \
ci-operator/step-registry/ipi/conf \
--glob '*.yaml' --glob '*.sh' || trueRepository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
needles = (
"chain: ipi-conf-gcp",
"ref: ipi-conf-gcp",
"workflow: ipi-gcp",
"e2-custom-8-32768",
"CONTROL_PLANE_NODE_TYPE",
)
for root in (Path("ci-operator/config"), Path("ci-operator/step-registry")):
counts = {needle: 0 for needle in needles}
files = {needle: set() for needle in needles}
for path in root.rglob("*"):
if not path.is_file():
continue
try:
text = path.read_text(errors="replace")
except OSError:
continue
for needle in needles:
if needle in text:
counts[needle] += text.count(needle)
files[needle].add(str(path))
print(root)
for needle in needles:
print(f"{needle}: {counts[needle]} matches in {len(files[needle])} files")
for path in sorted(files[needle])[:20]:
print(f" {path}")
PYRepository: openshift/release
Length of output: 18284
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- explicit control-plane override step ---'
cat -n ci-operator/step-registry/ipi/conf/gcp/control-plane-node-type/ipi-conf-gcp-control-plane-node-type-commands.sh
cat -n ci-operator/step-registry/ipi/conf/gcp/control-plane-node-type/ipi-conf-gcp-control-plane-node-type-ref.yaml
printf '%s\n' '--- three claimed override jobs ---'
rg -n -C 10 'e2-custom-8-32768|ipi-conf-gcp-control-plane-node-type' \
ci-operator/config/openshift/release/openshift-release-main__nightly-4.21.yaml \
ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml \
ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yamlRepository: openshift/release
Length of output: 10103
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- default GCP provision chain ---'
cat -n ci-operator/step-registry/cucushift/installer/rehearse/gcp/ipi/default/provision/cucushift-installer-rehearse-gcp-ipi-default-provision-chain.yaml
printf '%s\n' '--- GCP machine-type check ---'
cat -n ci-operator/step-registry/cucushift/installer/check/gcp/machine-type/cucushift-installer-check-gcp-machine-type-commands.shRepository: openshift/release
Length of output: 3733
Link the representative validation and sign-off for e2-custom-8-16384.
PR openshift/release#82754 is cited for 12 rehearsal runs, including this exact type, but no run links or sign-off are included. Add evidence for representative AMD64 GCP jobs before merge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ci-operator/step-registry/ipi/conf/gcp/ipi-conf-gcp-commands.sh` around lines
41 - 44, Add representative AMD64 GCP job links and explicit sign-off evidence
for the e2-custom-8-16384 default in the surrounding comment for
master_type_suffix. Retain the existing PR reference and performance rationale,
and ensure the cited runs cover the exact instance type before merge.
|
@sdodson, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
…o custom-8-16384 The default e2-custom-6-16384 (6 vCPU) GCP control-plane sizing has been associated with consistent high CPU alerts and elevated etcd disk-latency (WAL fsync / backend commit) across many GCP e2e jobs, contributing to flaky, non-reproducible test failures during upgrade and general e2e runs. An empirical rehearsal campaign on PR openshift#82754 (12 rehearsal runs across n2-standard-4, e2-standard-8, e2-custom-8-16384, and n2-custom-8-16384) found that control-plane vCPU count, not memory ratio or CPU platform, is the dominant driver of the improvement, and that e2-custom-8-16384 matches or beats e2-standard-8 on etcd/CPU metrics at a lower monthly cost by avoiding e2-standard-8's unneeded extra RAM. This changes the shared default in ipi-conf-gcp-commands.sh so every GCP job across every release branch that doesn't already set a larger CONTROL_PLANE_NODE_TYPE picks up the new sizing automatically, with no per-job config changes required. The 3 existing per-job overrides (nightly-4.21/4.22/5.0 e2e-gcp-custom-dns -> e2-custom-8-32768) are already >= 8 vCPU and are left untouched. The arm64 path (t2a-standard-4) is unaffected - custom sizes aren't supported on arm64, and this change was only validated on amd64 (e2/n2 GCP host pools). rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
e2-custom-8-32768 (8 vCPU, 32GB, custom) and e2-standard-8 (8 vCPU, 32GB, predefined) have identical vCPU/RAM specs, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly (confirmed via the GCP Cloud Billing Catalog API: $616.37/mo vs $587.02/mo for 3 nodes). Custom types only save money when trimming RAM below the nearest predefined tier, which isn't the case here. These were the only 3 CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPE overrides in the repo using this instance type (nightly-4.21/4.22/5.0 e2e-gcp-custom-dns), so switching to e2-standard-8 is a strict cost win with no behavior change. rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
8ec7704 to
5103c9d
Compare
openshift-ci
Bot
removed
the
needs-rebase
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: sdodson The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
[REHEARSALNOTIFIER]
A total of 6218 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@sdodson: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
Commit 1: Changes the shared GCP IPI default control-plane instance size from
e2-custom-6-16384(6 vCPU / 16GB) toe2-custom-8-16384(8 vCPU / 16GB) inipi-conf-gcp-commands.sh. Single-line default change affecting every GCP job across every release branch that doesn't already override with something larger - no per-job config edits needed for this part.Commit 2: Replaces the only 3 existing
CONTROL_PLANE_NODE_TYPE/COMPUTE_NODE_TYPEoverrides in the repo (nightly-4.21/nightly-4.22/nightly-5.0e2e-gcp-custom-dns, currentlye2-custom-8-32768) withe2-standard-8. Both are 8 vCPU/32GB, but GCP's custom machine type pricing carries a ~5% premium over the equivalent predefined shape when the ratio matches a predefined tier exactly - confirmed via the GCP Cloud Billing Catalog API ($616.37/mo vs $587.02/mo for 3 nodes). Custom types only save money when trimming RAM below the nearest predefined tier, which wasn't happening here. Strict cost win, no behavior change.Why (commit 1)
The 6 vCPU default has an existing code comment noting it was a "temporary test to see if this helps the consistent high CPU alerts and random test failures" - this same pattern was independently confirmed via:
e2-custom-8-16384matches or beatse2-standard-8on etcd fsync/commit latency, control-plane CPU saturation, and etcd error rates - at ~13% lower monthly cost thane2-standard-8and only ~24% above the current default's cost.n2-standard-4(4 vCPU) performed worse than the 6 vCPU baseline, while every 8-vCPU option tested performed better.Scope
*-case (amd64) default inipi-conf-gcp-commands.shis changed. Thexlarge/large/compactSIZE_VARIANTbranches (already >=8 vCPU equivalent) and thearm64path (t2a-standard-4, unvalidated by this work) are untouched.COMPUTE_NODE_TYPE) is unchanged for the shared default - out of scope for that part of this work (the commit-2 change toe2e-gcp-custom-dns's worker override is a separate, pricing-driven cleanup, not a sizing change).Validation
This affects ~722 GCP job configs across every active release branch (4.6-5.0, nightly, OKD). Before merging, this should be rehearsed against a representative sample of jobs (at minimum a 5.0 e2e job, an upgrade job, and one older stable branch) to confirm the shared step-registry change behaves as expected. Not rehearsed as part of this PR submission - please advise if you'd like specific jobs rehearsed before review.
Test plan
e2e-gcp-custom-dnson one branch to confirme2-standard-8behaves identically toe2-custom-8-32768e2family, only changes size within family, but flagging for visibility)Summary by CodeRabbit