Skip to content

MCO-2482: Revert "MCO-2470: Disable scale-up test support for AWS and vSphere" - #6356

Open
proietfb wants to merge 1 commit into
openshift:mainfrom
proietfb:revert-6344-MCO-2470_Disable_scale-up_test_support_for_AWS_and_vSphere
Open

MCO-2482: Revert "MCO-2470: Disable scale-up test support for AWS and vSphere"#6356
proietfb wants to merge 1 commit into
openshift:mainfrom
proietfb:revert-6344-MCO-2470_Disable_scale-up_test_support_for_AWS_and_vSphere

Conversation

@proietfb

@proietfb proietfb commented Jul 31, 2026

Copy link
Copy Markdown
Member

Reverts #6344

Summary by CodeRabbit

  • New Features
    • Added AWS NVMe disk path discovery for scale-up testing.
    • Scale-up tests now support AWS environments without platform-specific skips.
    • Improved disk filtering to exclude partitions and boot devices.
  • Tests
    • Enhanced scale-up coverage by discovering storage paths through a temporary probe node and retaining the original test node.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 31, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@proietfb: This pull request references MCO-2470 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Reverts #6344

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Walkthrough

The extended scale-up test now supports AWS. It discovers non-partition NVMe by-path devices from a temporary probe node, uses those paths in the MachineConfig, and selects a separate AWS test node.

Changes

AWS NVMe scale-up testing

Layer / File(s) Summary
NVMe by-path discovery
test/extended-priv/mco_irreconcilablechanges.go
The test adds a helper that runs a node-local command, filters partition and boot entries, sorts the remaining NVMe paths, and returns them.
AWS scale-up test flow
test/extended-priv/mco_irreconcilablechanges.go
The irreconcilable-changes test includes AWS. AWS uses a temporary probe node for disk discovery, creates the MachineConfig with two discovered paths, and scales two nodes before selecting the node distinct from the probe node.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScaleUpTest
  participant ProbeNode
  participant discoverNVMeByPathDisks
  participant MachineConfig
  participant TestNode
  ScaleUpTest->>ProbeNode: Provision temporary probe node
  ScaleUpTest->>discoverNVMeByPathDisks: Discover NVMe by-path devices
  discoverNVMeByPathDisks->>ProbeNode: Run node-local shell command
  ProbeNode-->>discoverNVMeByPathDisks: Return sorted device paths
  discoverNVMeByPathDisks-->>ScaleUpTest: Return two paths
  ScaleUpTest->>MachineConfig: Create configuration with discovered paths
  ScaleUpTest->>TestNode: Scale AWS nodes to two
  ScaleUpTest->>TestNode: Select node distinct from probe node
Loading

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The PR adds INFO logs that emit raw probe node names, which can be internal hostnames, and includes the discovered disk paths. Do not log raw node.GetName() values or disk details; log a redacted identifier and the disk count only.
Test Structure And Quality ⚠️ Warning 45 of 46 assertions lack meaningful failure messages. Tests must include diagnostic context in all Expect calls to aid troubleshooting. Add descriptive messages to all o.Expect assertions: use format Expect(value, "describes what should happen or the context of failure").
Microshift Test Compatibility ⚠️ Warning Test "Verify irreconcilable changes on new and existing nodes after scale up" uses machine.openshift.io MachineSet APIs for node scaling without static [apigroup:machine.openshift.io] or [Skipped:M... Add [apigroup:machine.openshift.io] tag to test name at line 166, or add [Skipped:MicroShift] label, since MicroShift lacks MachineAPI capability and cannot scale nodes.
✅ Passed checks (12 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All five Ginkgo test titles in the modified file use static, descriptive strings with no dynamic values, generated identifiers, node names, timestamps, or format specifiers.
Single Node Openshift (Sno) Test Compatibility ✅ Passed All tests in the Describe block are protected from SNO by JustBeforeEach calling SkipIfCompactOrSNO(oc), which skips the entire test suite on Single Node OpenShift deployments.
Topology-Aware Scheduling Compatibility ✅ Passed This PR modifies only test code in test/extended-priv/mco_irreconcilablechanges.go, not deployment manifests or operator code. The custom check applies only to deployment manifests, operator code,...
Ote Binary Stdout Contract ✅ Passed The patch adds no main/init/suite stdout writes; its only logger call runs inside It, uses GinkgoWriter, and remote shell output is captured by DebugNodeWithChrootStd.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No IPv4 assumptions or external connectivity found. Test uses local filesystem operations and cluster-internal MachineSet scaling with no hardcoded IPs or external URLs.
No-Weak-Crypto ✅ Passed The PR contains no weak cryptographic implementations, custom crypto, or non-constant-time secret comparisons. Changes are limited to adding NVMe disk discovery logic and test platform support for...
Container-Privileges ✅ Passed The PR modifies only a Go test file (test/extended-priv/mco_irreconcilablechanges.go) and does not create, modify, or deploy any Kubernetes manifests or container definitions with privileged settings.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the revert that restores scale-up test support for AWS and vSphere, matching the stated pull request objective.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended-priv/mco_irreconcilablechanges.go`:
- Around line 80-96: The discoverNVMeByPathDisks function selects all
non-partition NVMe devices from the glob pattern, but AWS instance-store volumes
also match this pattern, causing the HaveLen(2) assertion to fail on instances
with local NVMe storage. Update the logic after collecting the by-path entries
to filter and map each candidate to the expected block-device mappings /dev/sdb
or /dev/sdc. Only return the two MachineSet EBS data disks by validating each
discovered path corresponds to one of these two expected block devices rather
than returning all matched NVMe devices.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 07048f52-9c5d-4f2e-a0cc-d6ac1ed7ff25

📥 Commits

Reviewing files that changed from the base of the PR and between 75bdcd8 and 227d7f0.

📒 Files selected for processing (1)
  • test/extended-priv/mco_irreconcilablechanges.go

Comment on lines +80 to +96
// discoverNVMeByPathDisks discovers the /dev/disk/by-path/ entries for non-boot NVMe
// devices on an AWS node. These PCI-based paths are stable per instance type, unlike
// /dev/nvmeXn1 which depends on device enumeration order.
func discoverNVMeByPathDisks(node *Node) []string {
script := `for p in /dev/disk/by-path/*nvme*; do [[ $p == *part* ]] && continue; ls ${p}-part* &>/dev/null && continue; echo "$p"; done | sort`
stdout, _, err := node.DebugNodeWithChrootStd("bash", "-c", script)
o.ExpectWithOffset(1, err).NotTo(o.HaveOccurred(), "Failed to discover NVMe by-path disks on node %s", node.GetName())

var paths []string
for _, line := range strings.Split(strings.TrimSpace(stdout), "\n") {
line = strings.TrimSpace(line)
if line != "" {
paths = append(paths, line)
}
}
logger.Infof("Discovered %d non-boot NVMe by-path disks on node %s: %v", len(paths), node.GetName(), paths)
return paths

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Select only the two MachineSet EBS data disks.

The command selects every non-partition NVMe device. AWS can automatically enumerate NVMe instance-store volumes, and those devices also match this glob. The HaveLen(2) assertion then fails on supported instance types with local NVMe storage. (docs.aws.amazon.com)

Map each candidate to the expected /dev/sdb or /dev/sdc block-device mapping before returning its by-path entry. Do not use the total NVMe-device count as disk selection.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended-priv/mco_irreconcilablechanges.go` around lines 80 - 96, The
discoverNVMeByPathDisks function selects all non-partition NVMe devices from the
glob pattern, but AWS instance-store volumes also match this pattern, causing
the HaveLen(2) assertion to fail on instances with local NVMe storage. Update
the logic after collecting the by-path entries to filter and map each candidate
to the expected block-device mappings /dev/sdb or /dev/sdc. Only return the two
MachineSet EBS data disks by validating each discovered path corresponds to one
of these two expected block devices rather than returning all matched NVMe
devices.

@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@proietfb: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@proietfb

Copy link
Copy Markdown
Member Author

/payload-job periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-vsphere-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-azure-mco-disruptive-techpreview-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-gcp-mco-disruptive-techpreview-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-1of3

@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@proietfb: trigger 5 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview-1of3
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-vsphere-mco-disruptive-techpreview
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-azure-mco-disruptive-techpreview-1of3
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-gcp-mco-disruptive-techpreview-1of3
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-1of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/50db5c00-8cf7-11f1-97fe-1a7c1596ddc9-0

@proietfb

Copy link
Copy Markdown
Member Author

/test required

@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@proietfb: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test bootstrap-unit
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test images
/test okd-scos-images
/test tls-pqc-readiness
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test e2e-agent-compact-ipv4
/test e2e-agent-compact-ipv4-iso-no-registry
/test e2e-aws-disruptive
/test e2e-aws-mco-disruptive
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-fips-op
/test e2e-aws-ovn-ocb-techpreview
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-upgrade-ipsec
/test e2e-aws-ovn-upgrade-ocb-techpreview
/test e2e-aws-ovn-upgrade-out-of-change
/test e2e-aws-ovn-windows
/test e2e-aws-ovn-workers-rhel8
/test e2e-aws-proxy
/test e2e-aws-serial
/test e2e-aws-single-node
/test e2e-aws-upgrade-single-node
/test e2e-aws-workers-rhel8
/test e2e-azure
/test e2e-azure-ovn-multidisk-techpreview
/test e2e-azure-ovn-upgrade
/test e2e-azure-ovn-upgrade-out-of-change
/test e2e-azure-upgrade
/test e2e-gcp-mco-disruptive
/test e2e-gcp-op
/test e2e-gcp-op-techpreview
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-rt
/test e2e-gcp-rt-op
/test e2e-gcp-single-node
/test e2e-gcp-upgrade
/test e2e-hypershift-techpreview
/test e2e-metal-assisted
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-openstack
/test e2e-openstack-dualstack
/test e2e-openstack-externallb
/test e2e-openstack-hypershift
/test e2e-openstack-parallel
/test e2e-openstack-singlestackv6
/test e2e-ovirt
/test e2e-ovirt-upgrade
/test e2e-ovn-step-registry
/test e2e-vsphere
/test e2e-vsphere-ovn-disk-setup-techpreview
/test e2e-vsphere-ovn-upi
/test e2e-vsphere-ovn-upi-zones
/test e2e-vsphere-ovn-zones
/test e2e-vsphere-upgrade
/test okd-scos-e2e-aws-ovn
/test security

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-machine-config-operator-main-bootstrap-unit
pull-ci-openshift-machine-config-operator-main-images
pull-ci-openshift-machine-config-operator-main-okd-scos-images
pull-ci-openshift-machine-config-operator-main-security
pull-ci-openshift-machine-config-operator-main-unit
pull-ci-openshift-machine-config-operator-main-verify
pull-ci-openshift-machine-config-operator-main-verify-deps
Details

In response to this:

/test required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@proietfb

Copy link
Copy Markdown
Member Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

No second-stage tests were triggered for this PR.

This can happen when:

  • The changed files don't match any pipeline_run_if_changed patterns
  • All files match pipeline_skip_if_only_changed patterns
  • No pipeline-controlled jobs are defined for the main branch

Use /test ? to see all available tests.

@yuqi-zhang yuqi-zhang changed the title Revert "MCO-2470: Disable scale-up test support for AWS and vSphere" MCO-2482: Revert "MCO-2470: Disable scale-up test support for AWS and vSphere" Jul 31, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@proietfb: This pull request references MCO-2482 which is a valid jira issue.

Details

In response to this:

Reverts #6344

Summary by CodeRabbit

  • New Features
  • Added AWS NVMe disk path discovery for scale-up testing.
  • Scale-up tests now support AWS environments without platform-specific skips.
  • Improved disk filtering to exclude partitions and boot devices.
  • Tests
  • Enhanced scale-up coverage by discovering storage paths through a temporary probe node and retaining the original test node.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@yuqi-zhang

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

No second-stage tests were triggered for this PR.

This can happen when:

  • The changed files don't match any pipeline_run_if_changed patterns
  • All files match pipeline_skip_if_only_changed patterns
  • No pipeline-controlled jobs are defined for the main branch

Use /test ? to see all available tests.

@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: proietfb, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [proietfb,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants