Skip to content

AUTOSCALE-871: allow Karpenter Operator to run in ManagementCluster mode - #19

Open
maxcao13 wants to merge 1 commit into
openshift:mainfrom
maxcao13:target-kubeconfig
Open

AUTOSCALE-871: allow Karpenter Operator to run in ManagementCluster mode#19
maxcao13 wants to merge 1 commit into
openshift:mainfrom
maxcao13:target-kubeconfig

Conversation

@maxcao13

@maxcao13 maxcao13 commented Jul 30, 2026

Copy link
Copy Markdown
Member

Allows KarpenterOperator to run in 'ManagementCluster' mode. This means that when enabled, Karpenter Operator is assumed to run in a management cluster. Karpenter CRDs will live in a separate guest cluster that the management cluster oversees.

Introduces:

  • a new flag to the karpenter-operator binary called 'target-kubeconfig' which is the path to a guest cluster kubeconfig
  • an environment variable called MANAGEMENT_CLUSTER is read by karpenter-operator
    • if set to 'true', this enables Karpenter Operator to be deployed in management cluster mode

Currently target-kubeconfig is not wired up to any logic. That work is tracked in a separate issue. As of now, the flag only exists to not break HCP during the refactor which will pass that argument.

This is part 1 of the effort to allow HCP to deploy Karpenter Operator from this repo. Second part will reference this PR and will be a PR against openshift/hypershift.

Summary by CodeRabbit

  • New Features

    • Added management-cluster mode with support for target kubeconfig, platform, region, cluster name, and endpoint configuration.
    • Added configuration for deployment settings, health probes, webhook control, and AWS credentials.
    • Controllers now adjust automatically for management-cluster deployments.
  • Bug Fixes

    • Startup now reports environment-loading failures and exits cleanly.
    • Improved validation and clearer errors for missing configuration.
  • Tests

    • Updated validation and integration tests for management-cluster behavior.

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 30, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 30, 2026

Copy link
Copy Markdown

@maxcao13: This pull request references AUTOSCALE-871 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Allows KarpenterOperator to run in 'ManagementCluster' mode. This means that when enabled, Karpenter Operator is assumed to run in a management cluster. Karpenter CRDs will live in a separate guest cluster that the management cluster oversees.

Introduces:

  • a new flag to the karpenter-operator binary called 'target-kubeconfig' which is the path to a guest cluster kubeconfig
  • an environment variable called MANAGEMENT_CLUSTER is read by karpenter-operator
    • if set to 'true', this enables Karpenter Operator to be deployed in management cluster mode

Currently target-kubeconfig is not wired up to any logic. That work is tracked in a separate issue. As of now, the flag only exists to not break HCP during the refactor which will pass that argument.

This is part 1 of the effort to allow HCP to deploy Karpenter Operator from this repo. Second part will reference this PR and will be a PR against openshift/hypershift.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: maxcao13

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 30, 2026
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Walkthrough

Changes

Management cluster support

Layer / File(s) Summary
Startup configuration and validation
cmd/main.go, pkg/operator/options.go, pkg/operator/options_test.go
Adds target-kubeconfig support, management-cluster environment parsing, platform and region fields, error propagation, and conditional validation.
Mode-specific infrastructure discovery
pkg/cloudprovider/common/common.go, pkg/operator/operator.go
Uses environment-derived infrastructure in management-cluster mode and removes topology fields from discovered infrastructure data.
Controller selection and deployment environment
pkg/controllers/controllers.go, pkg/controllers/karpenter/controller.go, pkg/cloudprovider/common/consts.go, pkg/cloudprovider/aws/*
Skips controller creation for management clusters and centralizes Karpenter and AWS environment variable names.
Management-cluster test handling
test/pkg/environment/environment.go, test/suites/operator/operator_test.go
Updates the test environment and ClusterOperator checks to recognize management-cluster mode.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Options
  participant Operator
  participant Controllers
  CLI->>Options: Load environment and flags
  Options-->>CLI: Validated management-cluster settings
  CLI->>Operator: Run(options)
  Operator->>Operator: Build infrastructure from environment
  Operator->>Controllers: ResolveControllerConfig
  Controllers-->>Operator: No controllers for management cluster
Loading
🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning Management mode creates no controllers, but the suite still runs Karpenter/CRD reconciliation tests; the cluster-scoped Karpenter is cleaned only in AfterAll and several assertions lack messages. Skip the full resource/reconciliation suite in management mode; move cluster-resource cleanup to per-test teardown and add diagnostic messages to bare Gomega assertions.
✅ Passed checks (13 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All affected Ginkgo titles are static descriptive strings; the PR adds no dynamic title construction. New table-test names are also fixed literals and contain no run-varying values.
Microshift Test Compatibility ✅ Passed The diff adds no new Ginkgo e2e declarations; it only changes existing ClusterOperator skip conditions. Added tests are standard Go unit tests, so this check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the only Ginkgo change updates existing ClusterOperator skips for management-cluster mode, with no new multi-node assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed Added-line and source scans found no affinity, topology-spread, node-selector, toleration, taint, or PDB constraints; the existing one-replica workload is unchanged, and management mode disables co...
Ote Binary Stdout Contract ✅ Passed No process-level stdout writes found; main uses controller-runtime zap whose default DestWriter is os.Stderr, and suite/init setup emits no direct stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds no new Ginkgo e2e tests; the existing suite only changes skip conditions, with no IPv4 literals or external connectivity requirements introduced.
No-Weak-Crypto ✅ Passed The PR adds configuration and management-cluster wiring only; commit and tracked-source searches found no MD5/SHA1/DES/RC4/Blowfish/ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed No flagged settings found. The deployment and generated Karpenter pods require non-root, disable privilege escalation, and drop all capabilities; no host namespace, privileged, or SYS_ADMIN usage e...
No-Sensitive-Data-In-Logs ✅ Passed The only new log reports LoadEnv errors; those contain only an invalid MANAGEMENT_CLUSTER boolean value. Infrastructure logging is pre-existing, and no credentials, tokens, kubeconfig path, or secr...
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes enabling Karpenter Operator to run in ManagementCluster mode, matching the primary changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@maxcao13
maxcao13 force-pushed the target-kubeconfig branch from cab1f8f to 669c586 Compare July 30, 2026 23:12

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/operator/options_test.go`:
- Around line 14-17: Update the test around opts.LoadEnv to isolate the
environment contract by setting PLATFORM, REGION, and MANAGEMENT_CLUSTER with
t.Setenv before loading. After LoadEnv succeeds, assert that the corresponding
option fields contain those configured values, preventing ambient environment
values from affecting the test and covering the new fields.

In `@test/suites/operator/operator_test.go`:
- Around line 65-66: Gate the controller-dependent tests in the operator suite
on !env.IsManagementCluster(), including Karpenter reconciliation,
operand-readiness, spec-propagation, and drift-correction contexts. Update the
existing management-cluster skip logic around the ClusterOperator setup so these
assertions do not run when controller construction is suppressed, while
preserving normal-mode coverage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5ef0a368-28cf-42bc-8d70-0f212d729cce

📥 Commits

Reviewing files that changed from the base of the PR and between cab1f8f and 669c586.

📒 Files selected for processing (12)
  • cmd/main.go
  • pkg/cloudprovider/aws/consts.go
  • pkg/cloudprovider/aws/provider.go
  • pkg/cloudprovider/common/common.go
  • pkg/cloudprovider/common/consts.go
  • pkg/controllers/controllers.go
  • pkg/controllers/karpenter/controller.go
  • pkg/operator/operator.go
  • pkg/operator/options.go
  • pkg/operator/options_test.go
  • test/pkg/environment/environment.go
  • test/suites/operator/operator_test.go
💤 Files with no reviewable changes (1)
  • pkg/cloudprovider/common/common.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/controllers/karpenter/controller.go

Comment thread pkg/operator/options_test.go
Comment thread test/suites/operator/operator_test.go
Allows KarpenterOperator to run in 'ManagementCluster' mode.
This means that when enabled, Karpenter Operator is assumed to run in a management cluster.
Karpenter CRDs will live in a separate guest cluster that the management cluster oversees.

Introduces:
- a new flag to the karpenter-operator binary called 'target-kubeconfig' which is the path to a guest cluster kubeconfig
- an environment variable called MANAGEMENT_CLUSTER is read by karpenter-operator
    - if set to 'true', this enables Karpenter Operator to be deployed in management cluster mode

Currently target-kubeconfig is not wired up to any logic. That work is tracked in a separate issue.
As of now, the flag only exists to not break HCP during the refactor which will pass that argument.

Signed-off-by: Max Cao <macao@redhat.com>
@maxcao13
maxcao13 force-pushed the target-kubeconfig branch from 669c586 to 792c56c Compare July 30, 2026 23:18
@maxcao13
maxcao13 marked this pull request as ready for review July 30, 2026 23:34
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2026
@openshift-ci
openshift-ci Bot requested review from jkyros and joelsmith July 30, 2026 23:34
Comment thread pkg/operator/operator.go
if err != nil {
return fmt.Errorf("failed to discover infrastructure: %w", err)
}
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-thinking about this, it is probably better to just remove the discovering infrastructure logic altogether. I was only using it in standalone, but since we are going to use CAPI there is no use for it any longer, and we pass through important fields through env vars and flags on HCP.

@linkvt

linkvt commented Aug 3, 2026

Copy link
Copy Markdown
Member

/retest

LGTM, but I don't have enough background in the surrounding components yet to get all the details. Also not sure how the kubeconfig would be provided but that's something I can find out 😀 .

@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown

@maxcao13: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants