ETCD not becoming healthy #1501

tuxpeople · 2024-05-30T08:07:10Z

tuxpeople
May 30, 2024

I tried to set up a Talos Cluster with this, and almost everything is working when bootstraping. Only the last step never finishes until it runs into the timeout:

task: [talos:bootstrap] talosctl health --server=false
◳ waiting for etcd to be healthy: expected a response with 0 node(s), got 1
context deadline exceeded
task: Failed to run task "talos:bootstrap": exit status 1

Any idea where to look for debugging? Kubernetes itself looks to be working:

$ kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
talos-test01   Ready    control-plane   13m   v1.30.1   192.168.13.11   <none>        Talos (v1.7.2)   6.6.30-talos     containerd://1.7.16
talos-test02   Ready    control-plane   13m   v1.30.1   192.168.13.12   <none>        Talos (v1.7.2)   6.6.30-talos     containerd://1.7.16
talos-test03   Ready    control-plane   13m   v1.30.1   192.168.13.13   <none>        Talos (v1.7.2)   6.6.30-talos     containerd://1.7.16
talos-test04   Ready    <none>          13m   v1.30.1   192.168.13.14   <none>        Talos (v1.7.2)   6.6.30-talos     containerd://1.7.16

$ kubectl get pods -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS      AGE
kube-system   cilium-5d2qx                           1/1     Running   0             13m
kube-system   cilium-mxg8p                           1/1     Running   0             13m
kube-system   cilium-operator-7b9946bf99-fkmlp       1/1     Running   2 (11m ago)   13m
kube-system   cilium-x66sn                           1/1     Running   0             13m
kube-system   cilium-z8rq7                           1/1     Running   0             13m
kube-system   coredns-6dd9dfd4b4-f8rmv               1/1     Running   0             11m
kube-system   kube-apiserver-talos-test01            1/1     Running   0             11m
kube-system   kube-apiserver-talos-test02            1/1     Running   0             12m
kube-system   kube-apiserver-talos-test03            1/1     Running   0             13m
kube-system   kube-controller-manager-talos-test01   1/1     Running   2 (12m ago)   12m
kube-system   kube-controller-manager-talos-test02   1/1     Running   2 (11m ago)   12m
kube-system   kube-controller-manager-talos-test03   1/1     Running   0             13m
kube-system   kube-scheduler-talos-test01            1/1     Running   2 (12m ago)   12m
kube-system   kube-scheduler-talos-test02            1/1     Running   2 (10m ago)   12m
kube-system   kube-scheduler-talos-test03            1/1     Running   0             13m
kube-system   kubelet-csr-approver-5f65dfc95-84pnt   1/1     Running   1 (11m ago)   11m
kube-system   kubelet-csr-approver-5f65dfc95-9rd45   1/1     Running   0             11m
kube-system   spegel-2mqzg                           1/1     Running   0             11m
kube-system   spegel-8mbjg                           1/1     Running   0             11m
kube-system   spegel-8r2ws                           1/1     Running   0             11m
kube-system   spegel-mgfpv                           1/1     Running   0             11m

Answered by tuxpeople

May 31, 2024

@onedr0p It's working now:

task: [talos:bootstrap] talosctl health --server=false
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: OK
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: SKIP
waiting for coredns t…

View full answer

tuxpeople · 2024-05-30T08:26:47Z

tuxpeople
May 30, 2024
Author

I might need to add:

I'm using this Talos Factory Image of 1.7.2:
https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Fiscsi-tools&extensions=siderolabs%2Futil-linux-tools&extensions=siderolabs%2Fvmtoolsd-guest-agent&platform=vmware&target=cloud&version=1.7.2

It's customized like this:

customization:
    systemExtensions:
        officialExtensions:
            - siderolabs/iscsi-tools
            - siderolabs/util-linux-tools
            - siderolabs/vmtoolsd-guest-agent

0 replies

onedr0p · 2024-05-30T12:00:49Z

onedr0p
May 30, 2024
Maintainer

Anything in the talos console or logs?

1 reply

tuxpeople May 30, 2024
Author

Hope those information are helpful:

This are the consoles of the VMs:

Full output of `task talos:bootstrap`

task: [talos:bootstrap] if [ ! -f "folder/path/kubernetes/bootstrap/talos/talsecret.sops.yaml" ]; then
    talhelper gensecret > folder/path/kubernetes/bootstrap/talos/talsecret.sops.yaml
    sops --encrypt --in-place folder/path/kubernetes/bootstrap/talos/talsecret.sops.yaml
fi

task: [talos:bootstrap] talhelper genconfig --config-file folder/path/kubernetes/bootstrap/talos/talconfig.yaml --secret-file folder/path/kubernetes/bootstrap/talos/talsecret.sops.yaml --out-dir folder/path/kubernetes/bootstrap/talos/clusterconfig
generated config for talos-test01 in folder/path/kubernetes/bootstrap/talos/clusterconfig/talos-test-talos-test01.yaml
generated config for talos-test02 in folder/path/kubernetes/bootstrap/talos/clusterconfig/talos-test-talos-test02.yaml
generated config for talos-test03 in folder/path/kubernetes/bootstrap/talos/clusterconfig/talos-test-talos-test03.yaml
generated config for talos-test04 in folder/path/kubernetes/bootstrap/talos/clusterconfig/talos-test-talos-test04.yaml
generated client config in folder/path/kubernetes/bootstrap/talos/clusterconfig/talosconfig
generated .gitignore file in folder/path/kubernetes/bootstrap/talos/clusterconfig/.gitignore
task: [talos:bootstrap] talhelper gencommand apply --config-file folder/path/kubernetes/bootstrap/talos/talconfig.yaml --out-dir folder/path/kubernetes/bootstrap/talos/clusterconfig --extra-flags="--insecure" | bash
task: [talos:bootstrap] until talhelper gencommand bootstrap --config-file folder/path/kubernetes/bootstrap/talos/talconfig.yaml --out-dir folder/path/kubernetes/bootstrap/talos/clusterconfig | bash; do sleep 10; done
error executing bootstrap: 1 error occurred:
        * 192.168.13.11: rpc error: code = FailedPrecondition desc = bootstrap is not available yet


task: [talos:fetch-kubeconfig] until talhelper gencommand kubeconfig --config-file folder/path/kubernetes/bootstrap/talos/talconfig.yaml --out-dir folder/path/kubernetes/bootstrap/talos/clusterconfig --extra-flags="folder/path --force" | bash; do sleep 10; done
task: [talos:install-helm-apps] until kubectl --kubeconfig folder/path/kubeconfig wait --for=condition=Ready=False nodes --all --timeout=600s; do sleep 10; done
Unable to connect to the server: dial tcp 192.168.13.10:6443: connect: network is unreachable
The connection to the server 192.168.13.10:6443 was refused - did you specify the right host or port?
The connection to the server 192.168.13.10:6443 was refused - did you specify the right host or port?
The connection to the server 192.168.13.10:6443 was refused - did you specify the right host or port?
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
node/talos-test03 condition met
task: [talos:install-helm-apps] helmfile --kubeconfig folder/path/kubeconfig --file folder/path/kubernetes/bootstrap/helmfile.yaml apply --skip-diff-on-install --suppress-diff
Adding repo cilium https://helm.cilium.io
"cilium" has been added to your repositories

Adding repo coredns https://coredns.github.io/helm
"coredns" has been added to your repositories

Adding repo postfinance https://postfinance.github.io/kubelet-csr-approver
"postfinance" has been added to your repositories

Pulling ghcr.io/spegel-org/helm-charts/spegel:v0.0.22
Pulling ghcr.io/prometheus-community/charts/prometheus-operator-crds:11.0.0
Listing releases matching ^coredns$
Listing releases matching ^kubelet-csr-approver$
Listing releases matching ^cilium$
Listing releases matching ^prometheus-operator-crds$
Listing releases matching ^spegel$
Upgrading release=prometheus-operator-crds, chart=/var/folders/w1/jr434jtd2jbf62vg2ldkxqp40000gp/T/helmfile3906169066/observability/prometheus-operator-crds/prometheus-operator-crds/11.0.0/prometheus-operator-crds
Release "prometheus-operator-crds" does not exist. Installing it now.
NAME: prometheus-operator-crds
LAST DEPLOYED: Thu May 30 14:15:20 2024
NAMESPACE: observability
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^prometheus-operator-crds$
prometheus-operator-crds        observability   1               2024-05-30 14:15:20.053904 +0200 CEST   deployed        prometheus-operator-crds-11.0.0 v0.73.0    

Upgrading release=cilium, chart=cilium/cilium
Release "cilium" does not exist. Installing it now.
NAME: cilium
LAST DEPLOYED: Thu May 30 14:15:23 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium.

Your release version is 1.15.5.

For any further help, visit https://docs.cilium.io/en/v1.15/gettinghelp

Listing releases matching ^cilium$
cilium  kube-system     1               2024-05-30 14:15:23.740795 +0200 CEST   deployed        cilium-1.15.5   1.15.5     

Upgrading release=coredns, chart=coredns/coredns
Release "coredns" does not exist. Installing it now.
NAME: coredns
LAST DEPLOYED: Thu May 30 14:15:28 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CoreDNS is now running in the cluster as a cluster-service.

It can be tested with the following:

1. Launch a Pod with DNS tools:

kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools

2. Query the DNS server:

/ # host kubernetes

Listing releases matching ^coredns$
coredns kube-system     1               2024-05-30 14:15:28.684479 +0200 CEST   deployed        coredns-1.29.0  1.11.1     

Upgrading release=kubelet-csr-approver, chart=postfinance/kubelet-csr-approver
Release "kubelet-csr-approver" does not exist. Installing it now.
NAME: kubelet-csr-approver
LAST DEPLOYED: Thu May 30 14:15:31 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1

Listing releases matching ^kubelet-csr-approver$
kubelet-csr-approver    kube-system     1               2024-05-30 14:15:31.262073 +0200 CEST   deployed        kubelet-csr-approver-1.2.1      v1.2.1     

Upgrading release=spegel, chart=/var/folders/w1/jr434jtd2jbf62vg2ldkxqp40000gp/T/helmfile3906169066/kube-system/spegel/spegel/v0.0.22/spegel
Release "spegel" does not exist. Installing it now.
NAME: spegel
LAST DEPLOYED: Thu May 30 14:16:05 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^spegel$
spegel  kube-system     1               2024-05-30 14:16:05.596814 +0200 CEST   deployed        spegel-v0.0.22  v0.0.22    


UPDATED RELEASES:
NAME                       NAMESPACE       CHART                                                                VERSION   DURATION
prometheus-operator-crds   observability   oci://ghcr.io/prometheus-community/charts/prometheus-operator-crds   11.0.0          4s
cilium                     kube-system     cilium/cilium                                                        1.15.5          3s
coredns                    kube-system     coredns/coredns                                                      1.29.0          5s
kubelet-csr-approver       kube-system     postfinance/kubelet-csr-approver                                     1.2.1          34s
spegel                     kube-system     oci://ghcr.io/spegel-org/helm-charts/spegel                          v0.0.22        31s

task: [talos:install-helm-apps] until kubectl --kubeconfig folder/path/kubeconfig wait --for=condition=Ready nodes --all --timeout=600s; do sleep 10; done
node/talos-test01 condition met
node/talos-test02 condition met
node/talos-test03 condition met
node/talos-test04 condition met
task: [talos:bootstrap] talosctl health --server=false
◳ waiting for etcd to be healthy: expected a response with 0 node(s), got 1
context deadline exceeded
task: Failed to run task "talos:bootstrap": exit status 1

$ talosctl etcd members
WARNING: 1 error occurred:
        * 192.168.13.14: rpc error: code = Unimplemented desc = member list is only available on control plane nodes


NODE            ID                 HOSTNAME       PEER URLS                    CLIENT URLS                  LEARNER
192.168.13.12   389541d3d653bfb5   talos-test02   https://192.168.13.12:2380   https://192.168.13.12:2379   false
192.168.13.12   7f3e4176e7e0ba29   talos-test01   https://192.168.13.11:2380   https://192.168.13.11:2379   false
192.168.13.12   dd5f68210756a175   talos-test03   https://192.168.13.13:2380   https://192.168.13.13:2379   false
192.168.13.13   389541d3d653bfb5   talos-test02   https://192.168.13.12:2380   https://192.168.13.12:2379   false
192.168.13.13   7f3e4176e7e0ba29   talos-test01   https://192.168.13.11:2380   https://192.168.13.11:2379   false
192.168.13.13   dd5f68210756a175   talos-test03   https://192.168.13.13:2380   https://192.168.13.13:2379   false
192.168.13.11   389541d3d653bfb5   talos-test02   https://192.168.13.12:2380   https://192.168.13.12:2379   false
192.168.13.11   7f3e4176e7e0ba29   talos-test01   https://192.168.13.11:2380   https://192.168.13.11:2379   false
192.168.13.11   dd5f68210756a175   talos-test03   https://192.168.13.13:2380   https://192.168.13.13:2379   false
(.venv)

$ for n in 1 2 3; do talosctl health --nodes 192.168.13.1$i; echo ""; done
discovered nodes: ["192.168.13.10" "192.168.13.12" "192.168.13.13" "192.168.13.14"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

discovered nodes: ["192.168.13.10" "192.168.13.12" "192.168.13.13" "192.168.13.14"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

discovered nodes: ["192.168.13.10" "192.168.13.12" "192.168.13.13" "192.168.13.14"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

(.venv)

tuxpeople · 2024-05-31T13:10:58Z

tuxpeople
May 31, 2024
Author

@onedr0p It's working now:

task: [talos:bootstrap] talosctl health --server=false
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: OK
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: OK

Issue was that I had a stoneage version of talosctl in my path, before the up-to-date version from Homebrew. Removing the ancient version fixed the issue.

Before:

$ talosctl version
Client:
        Tag:         v1.0.0
        SHA:         80167fd2
        Built:       
        Go version:  go1.17.8
        OS/Arch:     darwin/arm64
Server:
        NODE:        192.168.13.11
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.12
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.13
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.14
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC

After

$ /opt/homebrew/bin/talosctl version
Client:
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     darwin/arm64
Server:
        NODE:        192.168.13.11
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.12
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.13
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
        NODE:        192.168.13.14
        Tag:         v1.7.2
        SHA:         f876025b
        Built:       
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ETCD not becoming healthy #1501

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

ETCD not becoming healthy #1501

tuxpeople May 30, 2024

Replies: 3 comments · 1 reply

tuxpeople May 30, 2024 Author

onedr0p May 30, 2024 Maintainer

tuxpeople May 30, 2024 Author

tuxpeople May 31, 2024 Author

tuxpeople
May 30, 2024

Replies: 3 comments 1 reply

tuxpeople
May 30, 2024
Author

onedr0p
May 30, 2024
Maintainer

tuxpeople May 30, 2024
Author

tuxpeople
May 31, 2024
Author