fix: ensure kubelet and containerd in kube.slice via systemd drop-ins by titilambert · Pull Request #8958 · Azure/AgentBaker

titilambert · 2026-07-15T19:38:16Z

What this PR does / why we need it:

When node hardening is enabled, ensureKubeletCgroupHierarchy() creates a dedicated kubelet.slice for kube-reserved cgroup enforcement. However the existing drop-in for kubelet.service only declared Wants= and After= ordering — it did not set Slice=kubelet.slice, so kubelet remained in system.slice. Additionally, containerd.service had no drop-in at all and was never placed in the slice.

This PR fixes both issues:

Adds Slice=kubelet.slice to the kubelet drop-in so kubelet actually runs inside the dedicated slice.
Creates an equivalent drop-in for containerd.service (10-kubelet-slice.conf) so containerd is co-located in the same slice.

Without this fix, cgroup resource accounting under kubelet.slice is incomplete — node hardening's kube-reserved budget doesn't capture the real resource usage of kubelet and containerd, which blocks nodes to be ready since they are affected to system.slice which has a low max memory limit

Testing:

Updated ShellSpec tests to validate both drop-ins are written with the
correct [Service] Slice=kubelet.slice directive.

 $ systemctl status containerd | grep "CGroup"
 CGroup: /kubelet.slice/containerd.service
 $ systemctl status kubelet | grep "CGroup"
 CGroup: /kubelet.slice/kubelet.service

Which issue(s) this PR fixes:

Fixes #

titilambert · 2026-07-15T19:38:17Z

This change is part of the following stack:

fix: ensure kubelet and containerd in kube.slice via systemd drop-ins #8958 ◀

_{Change managed by git-spice.}

Copilot

Pull request overview

This PR updates the Linux CSE helper responsible for Node Memory Hardening cgroup hierarchy setup so that both kubelet and containerd are configured to run under kubelet.slice, and adjusts ShellSpec coverage accordingly.

Changes:

Extend ensureKubeletCgroupHierarchy to set Slice=kubelet.slice for kubelet.service and add a matching drop-in for containerd.service.
Update ShellSpec assertions to validate the additional systemd drop-in content for containerd.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`parts/linux/cloud-init/artifacts/cse_helpers.sh`	Adds systemd drop-ins to place kubelet and containerd into `kubelet.slice` when hardening cgroups are enabled.
`spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh`	Updates tests to assert `Slice=` and validates the containerd drop-in is created.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

SriHarsha001

We need to look into the AI's comments, apart from that changes look good to me.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

aks-node-assistant · 2026-07-16T23:16:45Z

### AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172575900

Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E

Detective summary: The primary terminal failure is the known Ubuntu2204 HTTPSProxy PrivateDNS E2E proxy flake. CSE failed before kubelet startup while apt refreshed packages.microsoft.com through the scenario proxy.

Likely cause/signature: vmssCSE exited 99 after apt-get update to packages.microsoft.com via proxy 10.14.1.120:8888 returned connection refused. Wiki signature: e2e-httpsproxy-privatedns-err-outbound-conn-fail.

Confidence: High for E2E infra/proxy availability; low that this PR caused the failure.

Recommended owner/action: E2E/test-infra owner should track this under repair item #38837639 and harden proxy readiness/diagnostics for the HTTPSProxy PrivateDNS scenario. No PR action is expected from this signal.

Strongest alternative: PR CSE/helper changes broke provisioning. Less likely because the concrete failure is TCP connection refused to the scenario proxy during apt-get update, before kubelet/containerd slice behavior was reached.

Evidence:

Build timeline/log: https://dev.azure.com/msazure/CloudNativeCompute/_apis/build/builds/172575900/logs/562
Adjacent CIS signal is tracked separately as linux-vhd-prgate-cis-ubuntu2204-gen2-containerd-6131-pass-fail.

…de hardening is on

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

+    tee "/etc/systemd/system/kubelet.service.d/10-containerd-base-flag.conf" > /dev/null <<EOF
+[Service]
+Environment="KUBELET_CONTAINERD_FLAGS=--runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=${containerd_runtime_cgroups}"
+EOF
+


+// ValidateServiceInSlice asserts that the given systemd service is running in the expected slice.
+func ValidateServiceInSlice(ctx context.Context, s *Scenario, service, expectedSlice string) {
+	s.T.Helper()
+	result := execScriptOnVMForScenarioValidateExitCode(ctx, s,
+		fmt.Sprintf("systemctl show %s -p Slice --value", service), 0,
+		fmt.Sprintf("could not query Slice property of %s", service))
+	actual := strings.TrimSpace(result.stdout)
+	require.Equal(s.T, expectedSlice, actual,
+		"expected %s to be in %s, but got %s", service, expectedSlice, actual)
+}


… enabled

aks-node-assistant · 2026-07-17T19:36:41Z

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172680420

Failed job/stage/task: e2e / Run AgentBaker E2E / Go package compile

Detective summary: The E2E gate failed before running any tests because the e2e package did not compile.

Likely cause/signature: scenario_test.go references SkipScriptlessNBC in a Config literal, but Config does not define that field. Wiki signature: e2e-compile-config-unknown-field-skip-scriptless-nbc.

Confidence: Very high that this is PR/test-code caused; infra was not reached.

Recommended owner/action: PR author should fix the Config field definition or use the correct config struct/field and run an E2E compile check before retrying the gate.

Strongest alternative: stale generated source or checkout mismatch. Less likely because the compiler points at the PR-touched scenario_test.go and zero tests ran.

Evidence:

E2E log: https://dev.azure.com/msazure/CloudNativeCompute/_apis/build/builds/172680420/logs/596
Compiler error: ./scenario_test.go:3117:4: unknown field SkipScriptlessNBC in struct literal of type Config

…bled

github-actions · 2026-07-17T20:36:54Z

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jul 17, 2026, 9:28 PM

github-actions · 2026-07-17T20:36:54Z

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jul 17, 2026, 8:36 PM

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

+    # Validate supported values: only /kube.slice (or bare kube.slice) is
    # supported for KUBE_RESERVED_CGROUP, and only /system.slice for
    # SYSTEM_RESERVED_CGROUP (a built-in systemd slice). Reject any other value
    # explicitly so kubelet doesn't fail later with an opaque enforcement error.
    case "${KUBE_RESERVED_CGROUP:-}" in
-        ""|"/kubelet.slice"|"kubelet.slice") ;;
+        ""|"/kube.slice"|"kube.slice") ;;
        *)
-            echo "ensureKubeletCgroupHierarchy: unsupported KUBE_RESERVED_CGROUP=${KUBE_RESERVED_CGROUP}; only /kubelet.slice is supported"
+            echo "ensureKubeletCgroupHierarchy: unsupported KUBE_RESERVED_CGROUP=${KUBE_RESERVED_CGROUP}; only /kube.slice is supported"
            return 1


 	})
 }
+
+func Test_Ubuntu2204_NodeHardening_KubeletSlice(t *testing.T) {


Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/kubelet_config.pb.go: Generated file

+    # Validate supported values: only /kube-reserved.slice (or bare kube-reserved.slice) is
    # supported for KUBE_RESERVED_CGROUP, and only /system.slice for
    # SYSTEM_RESERVED_CGROUP (a built-in systemd slice). Reject any other value
    # explicitly so kubelet doesn't fail later with an opaque enforcement error.
    case "${KUBE_RESERVED_CGROUP:-}" in
-        ""|"/kubelet.slice"|"kubelet.slice") ;;
+        ""|"/kube-reserved.slice"|"kube-reserved.slice") ;;
        *)
-            echo "ensureKubeletCgroupHierarchy: unsupported KUBE_RESERVED_CGROUP=${KUBE_RESERVED_CGROUP}; only /kubelet.slice is supported"
+            echo "ensureKubeletCgroupHierarchy: unsupported KUBE_RESERVED_CGROUP=${KUBE_RESERVED_CGROUP}; only /kube-reserved.slice is supported"
            return 1


+    tee "/etc/systemd/system/kubelet.service.d/10-containerd-base-flag.conf" > /dev/null <<EOF
+[Service]
+Environment="KUBELET_CONTAINERD_FLAGS=--runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=${containerd_runtime_cgroups}"
+EOF
+


 	})
 }
+
+func Test_Ubuntu2204_NodeHardening_KubeletSlice(t *testing.T) {


Copilot AI review requested due to automatic review settings July 15, 2026 19:38

titilambert requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, karenychen, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, xuexu6666 and zachary-bailey as code owners July 15, 2026 19:38

Copilot started reviewing on behalf of titilambert July 15, 2026 19:39 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh Outdated

Comment thread spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh

Copilot AI review requested due to automatic review settings July 15, 2026 19:42

titilambert force-pushed the fix-test-ensure-kubelet-and branch from 39476df to 4045739 Compare July 15, 2026 19:42

Copilot started reviewing on behalf of titilambert July 15, 2026 19:43 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh Outdated

Comment thread spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh

titilambert changed the title ~~fix(test): ensure kubelet and containerd are in kubelet.slice when node hardening is on~~ fix: place kubelet and containerd in kubelet.slice via systemd drop-ins Jul 15, 2026

SriHarsha001 reviewed Jul 15, 2026

View reviewed changes

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh

titilambert enabled auto-merge (squash) July 16, 2026 20:55

mxj220 approved these changes Jul 16, 2026

View reviewed changes

titilambert added 8 commits July 17, 2026 09:13

fix(test): ensure kubelet and containerd are in kubelet.slice when no…

cd8d910

…de hardening is on

fix PR comments

17240f8

Make things simplier and idempotent

57494ca

Remove useless test

672bfa4

Add e2e tests

83b0cb3

Ensure runtime-cgroups match with containerd cgroups

1fb0911

Fix PR comments

6d6a02b

Fix PR comments

939cc03

Copilot AI review requested due to automatic review settings July 17, 2026 13:15

titilambert force-pushed the fix-test-ensure-kubelet-and branch from 4da0d05 to 939cc03 Compare July 17, 2026 13:15

Copilot started reviewing on behalf of titilambert July 17, 2026 13:15 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

titilambert disabled auto-merge July 17, 2026 13:24

Place containerd in dedicated containerd.slice when node hardening is…

93c48cc

… enabled

Place kubelet and containerd in kube.slice when node hardening is ena…

50681a8

…bled

Copilot AI review requested due to automatic review settings July 17, 2026 20:36

Copilot started reviewing on behalf of titilambert July 17, 2026 20:36 View session

titilambert changed the title ~~fix: ensure kubelet and containerd in kubelet.slice via systemd drop-ins~~ fix: ensure kubelet and containerd in kube.slice via systemd drop-ins Jul 17, 2026

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Rename kube.slice to kube-reserved.slice

dea3c28

Copilot AI review requested due to automatic review settings July 17, 2026 21:28

Copilot started reviewing on behalf of titilambert July 17, 2026 21:28 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Uh oh!

Conversation

titilambert commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

titilambert commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

SriHarsha001 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

aks-node-assistant Bot commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

aks-node-assistant Bot commented Jul 17, 2026

AgentBaker Linux gate detective

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

titilambert commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading