Switch to repos to maintaned pkgs.k8s.io #657

deric · 2023-09-22T08:18:07Z

Summary

New Kubernetes releases will be published to community owned repost pkgs.k8s.io instead of Google hosted repos.

up to v1.23 available only in packages.cloud.google.com
v1.24 to v1.27 can be installed from both repos
v1.28 onward should be installed from pkgs.k8s.io

Added tests to ensure backwards compatibility.

Related Issues (if any)

CAT-1005 - Updating latest version of K8S #655

Checklist

🟢 Spec tests.
🟢 Acceptance tests.
Manually verified. (For example puppet apply)

manifests/repos.pp

deric · 2023-09-25T09:55:12Z

The apt source must have following definition:

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /

Because otherwise we'd get following errors (as CDN is being used):

Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.

The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key

apt::source { 'kubernetes':
   ...
   key => {}
}

and keyring are mutually exclusive.

apt::source { 'kubernetes':
   ...
   keyring => '',
}

I'm not sure if we can write this in a better, backward compatible way 🤔

XSpielinbox · 2023-09-25T12:37:29Z

The apt source must have following definition:
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /
Because otherwise we'd get following errors (as CDN is being used):
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.
The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key
apt::source { 'kubernetes':
   ...
   key => {}
}
and keyring are mutually exclusive.
apt::source { 'kubernetes':
   ...
   keyring => '',
}
I'm not sure if we can write this in a better, backward compatible way 🤔

The signed-by option should be backwards compatible with the very old Debian 9 and Ubuntu 16.04 - that should be more than enough in my opinion. The newest versions of Debian and Ubuntu don't work with the old apt-key. Therefore I would definitely opt for the newer variant. See also: puppetlabs/puppetlabs-apt#1034

manifests/repos.pp

danifr · 2023-10-19T11:45:06Z

Thanks for working on this, I am also interested in having it merged as soon as possible.

For RedHat based OS, would it be possible to also add exclude in the repo definition?

Like in the official example provided here: https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/#how-to-migrate-rpm

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Thank you

deric · 2023-10-19T18:14:09Z

@danifr Sure. Just curious, what does the exclude mean? See baea6f9

XSpielinbox · 2023-10-20T10:17:41Z

@danifr Sure. Just curious, what does the exclude mean? See baea6f9

As far as I can find and understand it, this excludes updates or installs of these packages from this repository. It does not really make sense to me, however, as this would be the main packages you want to install from this repo or what am I missing?
I haven't tried the effect of this option, but the official documentation and man page for Fedora Linux 38 / RHEL 9 don't reference this option at all.

Sources I found about this option: 1, 2, 3, 4

danifr · 2023-10-20T13:13:16Z

Yeah disregard my previous comment. My idea was to exclude them from the repo definition so they won't be accidentally upgraded during a regular OS update.

But it seems that we exclude them there, puppet cannot installed them either.

Not sure why it is part of the configuration example...

deric · 2023-10-20T19:07:55Z

@danifr You might be able to use something like yum versionlock kubelet. But I think the whole point of having minor version in repo url:

https://pkgs.k8s.io/core:/stable:/v1.28/rpm/

is that the regular update could upgrade only to next patch version. Which also not ideal because the upgrade should be applied to control-plane nodes first. That's probably the reason why k8s packages are excluded, and when performing manual upgrade you should do this:

yum list --showduplicates kubeadm --disableexcludes=kubernetes

treydock · 2023-12-28T15:03:46Z

I think the recommendation by Kubernetes to exclude the Kubernetes packages is counterintuitive when configuration management is involved. To use the excludes the only real option with Puppet is install_options on the Package resources to force passing the --disableexcludes=kubernetes flag on install. Generally if someone is doing yum updates they need to do yum update --disablerepo=kubernetes unless they do intend to update Kubernetes since Kubernetes upgrades, even patch releases, require many extra steps.

Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream.

jadestorm · 2024-03-06T19:16:08Z

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

XSpielinbox · 2024-03-07T14:47:09Z

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

As announced back in August, the legacy repos have been frozen for month now and all Kubernetes version that where available through the old repos are long end of life now. As the end of February has passed the old repos are no longer available and if you are still running any version that was available in the old repos you should be looking into upgrading to a supported version as soon as possible.

[1] https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/
[2] https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/
[3] https://kubernetes.io/releases/patch-releases/

XSpielinbox

If you decide that you don't want to address my suggested changes soon I can also approve the PR as is.
Just please move forward as soon as possible. Kubernetes is moving fast and has very frequent releases. For the 1.28 branch 1.28.7 already released.

XSpielinbox · 2024-03-08T13:14:16Z

spec/acceptance/kubernetes_spec.rb

+              kubernetes_version => '1.28.2',
+              kubernetes_package_version => '1.28.2-1.1',


As 1.29.2 is already out by now this probably should get updated to that new version.

Sure, this is just version for tests. The point was to update the version, that is available in the new repos. We should definitely update the default version, because the current value is 1.10.2.

XSpielinbox · 2024-03-08T13:18:02Z

manifests/repos.pp

+          # For pkgs.k8s.io use GPG siging key
+          $_keyring = '/usr/share/keyrings/kubernetes-apt-keyring.gpg'
+          # TODO: Switch to apt::keyring once supported by puppetlabs-apt
+          # see: https://github.com/puppetlabs/puppetlabs-apt/pull/1128
+          archive { '/tmp/kubernetes-apt-keyring.gpg':
+            source          => "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key",
+            extract         => true,
+            extract_path    => '/usr/share/keyrings',
+            extract_command => 'gpg --dearmor < %s > kubernetes-apt-keyring.gpg',
+            creates         => $_keyring,
+          }
+
+          Apt::Source<| title == 'kubernetes' |> {
+            keyring  => $_keyring,
+            require  => Archive['/tmp/kubernetes-apt-keyring.gpg'],
+          }


The support for modern keyrings has been released in puppetlabs-apt since three month now in version 9.2. Version 9.4 is also already out. To me, it would make sense to switch to apt::keyring, but if you would like to keep it like this for compatibility for now this would also be fine for me.

The most important for me would be that this PR finally get's merged and there is a release of puppetlabs-kubernetes that has useful, modern defaults and works with the newest version of Kubernetes out of the box.

As I wrote earlier, it would be very nice to switch to modern apt::keyring. But we should make incremental changes. Adding dependency on puppetlabs-apt >= 9.2.0 means effectively bumping dependency on puppetlabs-stdlib from >= 4.25.0 to >= 9.0.0. This is rather huge step for the mankind, especially done in a single innocent PR called "Swich repos...".

Depending on puppet code base setup this might mean upgrading many (10 or 40) modules just because of this minor change. I would prefer to release this "old but working version" as a 8.1.0 version. And then change module dependencies in separate PR, while bumping the module version e.g. to 9.0.0.

As mentioned in #681, the actual puppetlabs/stdlib requirement is currently >= 8.4.0. So the change wouldn't be that huge.

XSpielinbox · 2024-03-08T13:22:07Z

spec/classes/repos_spec.rb

+            full: '16.04'
+          },
+          distro: {
+            codename: 'xenial'


This should perhaps also get updated to a newer version like 22.04, as 24.04 will soon be released, 16.04 is long EOL now and it doesn't help me at all, if it works on some ancient release, but not a current one.

This has nothing to do with newer distributions. Google was releasing all packages under xenial codename because the binary packages weren't dependent any specific distribution.

Yes, I know that everything was released under the kubernetes-xenial distribution (release in puppet), but I am not completely mistaken this is what got removed as default in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-b27acae942df5d6ec2b58798add49fff0f625bec33093d74ea3cb1ac39fd0c6aL69, https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L26 and tested against in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L50
but the standalone xenial should refer to the distro, not the kubernetes-repo.

XSpielinbox · 2024-03-08T13:27:22Z

spec/classes/repos_spec.rb

+        'kubernetes_yum_baseurl' => 'https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64',
+        'kubernetes_yum_gpgkey' => 'https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg',
+        'docker_apt_location' => 'https://download.docker.com/linux/ubuntu',
+        'docker_apt_release' => 'xenial',
+        'docker_apt_repos' => 'main',
+        'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable',
+        'docker_yum_gpgkey' => 'https://download.docker.com/linux/centos/gpg',


Same here: CentOS 7 / RHEL 7 will soon be EOL - CentOS Stream 9 / RHEL 9 have been out for almost 2 years now. For sure it would be good, if it still works on older releases, but people should be moving away from them anyway and ensuring that it works on newer versions is crucial to not hold people back and force them to stick with increasingly insecure OS versions.

Sorry, I've copy pasted those tests. But these test are garbage in & garbage out tests. It doesn't actually test these distributions. It only tests string getting passed to the class, that it gets propagated. Actual tests should be written in acceptance tests.

Yeah sure, you take some (more or less arbitrary base os) and then test, if 1. If you don't specify a specific repository, it now correctly defaults to the new repo and 2. If you override it with something (e.g. the old repo), it still includes the specified repo as desired, to e.g. still use the old repo for whatever reason.

As this could in theory be OS version dependent, this should in my opinion be tested on one recent release (Ubuntu 22.04 (as you changed it now - thank you!) or Debian 12 and CentOS Stream 9 / RHEL 9) and optionally additionally on some older version as well.

https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R5-R70 does 1. as far as I understand it: when testing on (now) Ubuntu 22.04 with Kubernetes version 1.28, no additional kubernetes_apt_* and for whatever reason questionable kubernetes_yum_* config, it correctly set's the apt source to the new 1.28 repo.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R71-R132 does 2., testing that on a Debian-based distro when overriding the repo, it set's it to the override.

https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R195-R246 tests 1., that on CentOS 7 the new default works.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 tests 2., that the override works for CentOS 7, while still configuring some questionable apt stuff, that should be absolutely irrelevant on dnf-based distros.

And finally https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 does 2. again on CentOS 7, but now with packaged containerd instead of docker archives.

XSpielinbox · 2024-03-08T13:28:17Z

spec/classes/repos_spec.rb

+        operatingsystemrelease: '7.0',
+        kernel: 'Linux',
+        os: {
+          family: 'RedHat',
+          name: 'RedHat',
+          release: {
+            full: '7.0'
+          }
+        }
+      }
+    end
+
+    let(:params) do
+      {
+        'container_runtime' => 'docker',
+        'kubernetes_version' => '1.28.1',
+        'kubernetes_apt_location' => '',
+        'kubernetes_apt_release' => '',
+        'kubernetes_apt_repos' => '',
+        'kubernetes_key_id' => '',
+        'kubernetes_key_source' => '',
+        'kubernetes_yum_baseurl' => '',
+        'kubernetes_yum_gpgkey' => '',
+        'docker_apt_location' => 'https://download.docker.com/linux/ubuntu',
+        'docker_apt_release' => 'xenial',
+        'docker_apt_repos' => 'main',
+        'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable',


jadestorm · 2024-03-08T16:21:03Z

Just wanted to add a couple of notes --

It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...
I can't explain why but when I used your fork for this PR as-is @deric , I got the following dependency loop which frankly makes zero sense to me after looking through the code:
Error: Found 1 dependency cycle:
(Exec[Install cni network provider] => Exec[Install cni network provider])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
Error: Failed to apply catalog: One or more resource dependency cycles detected in graph

I could not track down specifically what I might have been doing to cause this, if anything, but I just wanted to mention it. It also did not occur on ALL of our hosts using this module -- so there's a good chance it's something we caused. None-the-less, it does not occur in 8.0.0 so I wanted to bring it up not as a problem with this PR, but as a note. I believe your fork only has work-since-8.0.0 plus pkgs changes.

I can share the hiera method of working around this issue with 8.0.0 if anyone needs it.

XSpielinbox · 2024-03-08T17:44:19Z

1. It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...

@jadestorm from looking at the code something like setting

kubernetes::kubernetes_apt_location: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb"
kubernetes::kubernetes_apt_release: '/'
kubernetes::kubernetes_version: 1.29.2
kubernetes::kubernetes_package_version: 1.29.2-1.1
kubernetes::kubernetes_apt_repos: ''
kubernetes::kubernetes_key_id: '<the new fingerprint>'
kubernetes::kubernetes_key_source: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key"

in the hiera should work (given that your containerd, systemd, etc. config is updated already) , but haven't tested it directly.
${minor_version} of course must be substituted either directly or by doing something like

$kubernetes_version = lookup('kubernetes::kubernetes_version')

$parts = split($kubernetes_version, '[.]')
$minor_version = "${parts[0]}.${parts[1]}"

Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream. (cherry picked from commit 07acbc1)

Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream. (cherry picked from commit 07acbc1) Change-Id: I452a7e087cda5bb0089b5faa7b14c6c633ed1497

deric requested review from bastelfreak and a team as code owners September 22, 2023 08:18

ia-content added the community label Sep 22, 2023

deric force-pushed the pkgs branch 2 times, most recently from db17097 to 53b786c Compare September 22, 2023 08:35

XSpielinbox suggested changes Sep 24, 2023

View reviewed changes

manifests/repos.pp Outdated Show resolved Hide resolved

manifests/repos.pp Outdated Show resolved Hide resolved

deric force-pushed the pkgs branch 2 times, most recently from fcd52f8 to eb3f67c Compare September 25, 2023 08:32

XSpielinbox suggested changes Sep 25, 2023

View reviewed changes

manifests/repos.pp Outdated Show resolved Hide resolved

deric requested a review from XSpielinbox October 19, 2023 18:15

deric force-pushed the pkgs branch from 78008e4 to 23c6ced Compare February 8, 2024 09:54

deric requested review from GSPatton and removed request for XSpielinbox February 8, 2024 10:17

deric requested a review from XSpielinbox March 8, 2024 11:45

XSpielinbox reviewed Mar 8, 2024

View reviewed changes

deric added 4 commits June 27, 2024 10:50

Switch to repos to maintaned pkgs.k8s.io

54dc7af

Override 'main' value in apt::source::repo param

04a99db

Use GPG key signing

ea70044

Update test cases to more recent distributions

3074204

deric force-pushed the pkgs branch from f20690d to 3074204 Compare June 27, 2024 08:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to repos to maintaned pkgs.k8s.io #657

Switch to repos to maintaned pkgs.k8s.io #657

deric commented Sep 22, 2023 •

edited

Loading

deric commented Sep 25, 2023

XSpielinbox commented Sep 25, 2023

danifr commented Oct 19, 2023

deric commented Oct 19, 2023

XSpielinbox commented Oct 20, 2023

danifr commented Oct 20, 2023

deric commented Oct 20, 2023

treydock commented Dec 28, 2023

jadestorm commented Mar 6, 2024

XSpielinbox commented Mar 7, 2024

XSpielinbox left a comment

XSpielinbox Mar 8, 2024

deric Mar 8, 2024

XSpielinbox Mar 8, 2024

deric Mar 8, 2024

deric Jun 27, 2024

XSpielinbox Mar 8, 2024

deric Mar 8, 2024

XSpielinbox Mar 8, 2024

XSpielinbox Mar 8, 2024

deric Mar 8, 2024

XSpielinbox Mar 8, 2024

XSpielinbox Mar 8, 2024

jadestorm commented Mar 8, 2024

XSpielinbox commented Mar 8, 2024

		kubernetes_version => '1.28.2',
		kubernetes_package_version => '1.28.2-1.1',

Switch to repos to maintaned pkgs.k8s.io #657

Are you sure you want to change the base?

Switch to repos to maintaned pkgs.k8s.io #657

Conversation

deric commented Sep 22, 2023 • edited Loading

Summary

Related Issues (if any)

Checklist

deric commented Sep 25, 2023

XSpielinbox commented Sep 25, 2023

danifr commented Oct 19, 2023

deric commented Oct 19, 2023

XSpielinbox commented Oct 20, 2023

danifr commented Oct 20, 2023

deric commented Oct 20, 2023

treydock commented Dec 28, 2023

jadestorm commented Mar 6, 2024

XSpielinbox commented Mar 7, 2024

XSpielinbox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jadestorm commented Mar 8, 2024

XSpielinbox commented Mar 8, 2024

deric commented Sep 22, 2023 •

edited

Loading