kernel version should be 4.19+; recommend version 5.8 for cgroup v2 #37

pacoxu · 2024-07-24T09:41:50Z

For kernel long term support, see https://wiki.linuxfoundation.org/civilinfrastructureplatform/start and https://endoflife.date/linux

4.4 & 4.19 are selected as kernel Super Long Term Support (SLTS), and the Civil Infrastructure Platform will provide support until at least 2026.
- EDITED removed 4.4 as CIP is not in https://www.kernel.org/category/releases.html and maintained in a different place.
For cgroup v2, Kubernetes recommends to use 5.8 and later, and in runc docs, the minimal version is 4.15 and 5.2+ is recommended.

Other comments that may be related:

In Kubernetes 1.31, cgroup v1 is moved to maintenance mode and 4.14 LTS EOF in Jan 2024 (linux, LTS) , besides, centos 7 is EOL in June 30, 2024. I chosen 4.15 as runc.

for kernel minimal version, choosing 4.15 as runc + cgroup v2 https://github.com/opencontainers/runc/blob/3778ae603c706494fd1e2c2faf83b406e38d687d/docs/cgroup-v2.md?plain=1#L24

kernel >= 4.15 with CONFIG_CGROUP_DEVICE and CONFIG_CGROUP_BPF is required. From support cgroup2 containerd/containerd#3799 (comment).

cpu (since Linux 4.15)

cgroup v2: Kubernetes recommended kernel version https://kubernetes.io/docs/concepts/architecture/cgroups/

More details can be found in kubernetes/kubernetes#116799.

The v1.31 KEP kubernetes/enhancements#4569

https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4569-cgroup-v1-maintenance-mode/README.md KEP-4569: Moving cgroup v1 support into maintenance mode

Other minimal kernel version candidates

kernel 4.5 announced that cgroup v2 is not experimental anymore, as it supports io/pids/memory.

runc recommends 5.2+ as 5.2 supports freezer.

pacoxu · 2024-07-24T09:46:40Z

/assign @neolit123 @SataQiu

neolit123

we could show an error for any version that is not in that list of Versions and we don't need a RecommendedVersions or warnings IMO.
the error message could be x is not in the list of recommended versions. %s, where %s is a VersionsNote from the KernelSpec API.

validators/kernel_validator.go

validators/kernel_validator_test.go

neolit123 · 2024-07-24T10:30:49Z

validators/types_unix.go

+		// cgroup v2: Kubernetes recommended kernel version
+		// https://kubernetes.io/docs/concepts/architecture/cgroups/
+		RecommendedVersions: []string{`^5\.[8-9].*$`, `^5\.[1-9][0-9].*$`, `^([6-9]|[1-9][0-9]+)\.([0-9]+)\.([0-9]+).*$`}, // Requires 5.8+, or newer for cgroup v2
+		RecommendedNote:     "kernel version should >= '4.15', and 5.8+ is recommended.",


can be called VersionsNote

from a discussion on slack we had:

the checks are regexp based so could be 4.4 == ok, 4.19 == ok, any other >= 4.19 version == ok, any other version == err.

so in this note we can say that all those 4.x LTS versions are supported but the recommendation is to use 4.15 or 5.8+.

VersionsNote: "kernel version should be in maintenance, at least 4.4 or 4.19+, and to use cgroup v2, 5.8+ is recommended.",

update it as above..

validators/kernel_validator.go

validators/types.go

validators/types_unix.go

validators/types_windows.go

validators/types_unix.go

validators/kernel_validator_test.go

neolit123

just one final nit
https://github.com/kubernetes/system-validators/pull/37/files#r1691979088
and LGTM

pacoxu · 2024-07-25T21:42:14Z

Updated.

neolit123

/lgtm
/approve

should we create a release after this PR merges?

neolit123 · 2024-07-31T06:14:31Z

/hold
drop the hold if needed.

pacoxu · 2024-07-31T07:46:32Z

should we create a release after this PR merges?

Yes.

/hold
drop the hold if needed.

We may wait for another week or two in case SIG-Node owners have some inputs.

neolit123 · 2024-07-31T08:11:11Z

should we create a release after this PR merges?

Yes.

/hold
drop the hold if needed.

We may wait for another week or two in case SIG-Node owners have some inputs.

you should probably give them a ping on slack, or they will miss this.

SergeyKanzhelev · 2024-08-01T20:51:39Z

validators/types_unix.go

-		Versions: []string{`^3\.[1-9][0-9].*$`, `^([4-9]|[1-9][0-9]+)\.([0-9]+)\.([0-9]+).*$`}, // Requires 3.10+, or newer
+		// 4.4 & 4.19 are selected as kernel Super Long Term Support (SLTS), and the Civil Infrastructure Platform will provide support until at least 2026.
+		Versions:     []string{`^4\.4.*$`, `^4\.19.*$`, `^4\.[2-9][0-9].*$`, `^([5-9]|[1-9][0-9]+)\.([0-9]+)\.([0-9]+).*$`},
+		VersionsNote: "Recommended LTS versions from the 4.x series are 4.4 and 4.19. Any 4.x version newer than 4.19 is also supported. For cgroups v2 support, the minimal version is 4.15 and the recommended version is 5.8+",


for this note, do we have a check on cgroupv2 that the minimal version is satisfied?

cc: @harche

The current statement is from

https://github.com/opencontainers/runc/blob/ad5b481dace5cda8ca7c659b7717a15517333198/docs/cgroup-v2.md?plain=1#L23-L24 Recommended version: 5.2 or later & Minimum version: 4.15

https://kubernetes.io/docs/concepts/architecture/cgroups/#requirements: Linux Kernel version is 5.8 or later

If the runc one is not accurate, we may change to the minimal version is 5.8

I'm fine with the message. Just was wondering if we need to add a validaiton in kubelet on start when cgroupv2 are used.

there are many kernel version validations in kube. Mainly related to some FG or sysctl.

We may add a warning first for kubelet using cgoup v2.

[1.32 placeholder for the cgroup v2 kernel version check in kubelet]kubernetes/kubernetes#126595

validators/kernel_validator_test.go

pacoxu · 2024-08-06T07:09:12Z

According to some discussions in endoflife-date/endoflife.date#5608 (comment), the CIP seems to be not in https://www.kernel.org/category/releases.html and maintained in a different place(gitlab) than torvalds/linux.

I prefer to remove the 4.4.* here.

neolit123 · 2024-08-06T07:12:50Z

According to some discussions in endoflife-date/endoflife.date#5608 (comment), the CIP seems to be not in https://www.kernel.org/category/releases.html and maintained in a different place(gitlab) than torvalds/linux.

I prefer to remove the 4.4.* here.

+1 to keep only official lts

pacoxu · 2024-08-06T08:12:11Z

Updated in commit: update according to kernel official active kernel releases.

neolit123

please squash the commits to 1 as well.

validators/types_unix.go

Signed-off-by: Paco Xu <paco.xu@daocloud.io>

neolit123

/lgtm

k8s-ci-robot · 2024-08-06T08:28:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neolit123, pacoxu, SataQiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SataQiu,neolit123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pacoxu · 2024-08-12T02:58:30Z

/unhold
@neolit123 Could you trigger a release after v1.31.0 is released this week?

k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Jul 24, 2024

k8s-ci-robot requested a review from fabriziopandini July 24, 2024 09:41

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 24, 2024

k8s-ci-robot requested a review from odinuge July 24, 2024 09:41

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 24, 2024

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from 02aa92c to d773f70 Compare July 24, 2024 09:46

k8s-ci-robot assigned neolit123 and SataQiu Jul 24, 2024

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch 2 times, most recently from a4259a9 to 8aaafa2 Compare July 24, 2024 09:59

pacoxu mentioned this pull request Jul 24, 2024

Update the kernel configuration for cgroup v2 #36

Open

neolit123 reviewed Jul 24, 2024

View reviewed changes

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from 8aaafa2 to f1c81c0 Compare July 25, 2024 02:53

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 25, 2024

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch 2 times, most recently from a313836 to 42443a8 Compare July 25, 2024 02:57

pacoxu changed the title ~~Cgroupv2 kernel min 4.15 & recommend version 5.8 for cgroup v2~~ kernel version can be 4.4 & 4.19+; recommend version 5.8 for cgroup v2 Jul 25, 2024

pacoxu mentioned this pull request Jul 25, 2024

Minimal kernel version plan: probably 4.18/4.19+ kubernetes/kubernetes#116799

Open

27 tasks

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch 2 times, most recently from c80cf2a to a9871d7 Compare July 25, 2024 03:51

pacoxu commented Jul 25, 2024

View reviewed changes

validators/kernel_validator.go Outdated Show resolved Hide resolved

neolit123 reviewed Jul 25, 2024

View reviewed changes

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from a9871d7 to 9778272 Compare July 25, 2024 13:29

neolit123 reviewed Jul 25, 2024

View reviewed changes

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from 9778272 to c1779ba Compare July 25, 2024 21:41

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from c1779ba to 8a3acbe Compare July 31, 2024 03:13

neolit123 reviewed Jul 31, 2024

View reviewed changes

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 31, 2024

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 31, 2024

SergeyKanzhelev reviewed Aug 1, 2024

View reviewed changes

SataQiu approved these changes Aug 2, 2024

View reviewed changes

validators/kernel_validator_test.go Show resolved Hide resolved

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from 8a3acbe to b7357e1 Compare August 4, 2024 11:32

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 4, 2024

pacoxu changed the title ~~kernel version can be 4.4 & 4.19+; recommend version 5.8 for cgroup v2~~ kernel version should be 4.19+; recommend version 5.8 for cgroup v2 Aug 6, 2024

neolit123 reviewed Aug 6, 2024

View reviewed changes

validators/types_unix.go Outdated Show resolved Hide resolved

validators/types_unix.go Outdated Show resolved Hide resolved

kernel should be active long support version 4.19 or later.

e6e7785

Signed-off-by: Paco Xu <paco.xu@daocloud.io>

pacoxu force-pushed the cgroupv2-kernel-min-4.15 branch from 020ec91 to e6e7785 Compare August 6, 2024 08:27

neolit123 reviewed Aug 6, 2024

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2024

pacoxu mentioned this pull request Aug 8, 2024

[1.32]kubelet: add log and event for cgroup v2 running on kernel < 5.8 kubernetes/kubernetes#126595

Merged

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2024

k8s-ci-robot merged commit 1e0d9c3 into kubernetes:master Aug 12, 2024
3 checks passed

pacoxu mentioned this pull request Sep 12, 2024

1.32: housekeeping tasks kubernetes/kubeadm#3097

Open

5 tasks

This was referenced Sep 23, 2024

Set sysctl net/ipv4/ip_unprivileged_port_start ? kubernetes/kubernetes#102612

Open

add kernel version check for kubeadm upgrade kubernetes/kubeadm#3114

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kernel version should be 4.19+; recommend version 5.8 for cgroup v2 #37

kernel version should be 4.19+; recommend version 5.8 for cgroup v2 #37

pacoxu commented Jul 24, 2024 •

edited

Loading

pacoxu commented Jul 24, 2024

neolit123 left a comment

neolit123 Jul 24, 2024

pacoxu Jul 25, 2024

neolit123 left a comment

pacoxu commented Jul 25, 2024

neolit123 left a comment

neolit123 commented Jul 31, 2024

pacoxu commented Jul 31, 2024

neolit123 commented Jul 31, 2024

SergeyKanzhelev Aug 1, 2024

pacoxu Aug 2, 2024 •

edited

Loading

SergeyKanzhelev Aug 2, 2024

pacoxu Aug 3, 2024

pacoxu Aug 8, 2024 •

edited

Loading

pacoxu commented Aug 6, 2024

neolit123 commented Aug 6, 2024

pacoxu commented Aug 6, 2024

neolit123 left a comment

neolit123 left a comment

k8s-ci-robot commented Aug 6, 2024

pacoxu commented Aug 12, 2024 •

edited

Loading

kernel version should be 4.19+; recommend version 5.8 for cgroup v2 #37

kernel version should be 4.19+; recommend version 5.8 for cgroup v2 #37

Conversation

pacoxu commented Jul 24, 2024 • edited Loading

pacoxu commented Jul 24, 2024

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 Jul 24, 2024

Choose a reason for hiding this comment

pacoxu Jul 25, 2024

Choose a reason for hiding this comment

neolit123 left a comment

Choose a reason for hiding this comment

pacoxu commented Jul 25, 2024

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 commented Jul 31, 2024

pacoxu commented Jul 31, 2024

neolit123 commented Jul 31, 2024

SergeyKanzhelev Aug 1, 2024

Choose a reason for hiding this comment

pacoxu Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

SergeyKanzhelev Aug 2, 2024

Choose a reason for hiding this comment

pacoxu Aug 3, 2024

Choose a reason for hiding this comment

pacoxu Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

pacoxu commented Aug 6, 2024

neolit123 commented Aug 6, 2024

pacoxu commented Aug 6, 2024

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 6, 2024

pacoxu commented Aug 12, 2024 • edited Loading

pacoxu commented Jul 24, 2024 •

edited

Loading

pacoxu Aug 2, 2024 •

edited

Loading

pacoxu Aug 8, 2024 •

edited

Loading

pacoxu commented Aug 12, 2024 •

edited

Loading