Traffic within the cluster that crosses nodes doesn't work, everything is dropped

[nogweii]

Really not sure why this is. If I launch two pods on the same node, everything works. However, if those two pods are on 2 different hosts, then they can't talk to each other. I noticed the issue and started following Kubernetes' Debug Service guide and have attached details. Use a pod with host networking, I can verify each node can see each other. All of them report Ready in kubernetes' node list as well as Ready in talosctl dashboard.

• talosctl support.zip
• kubectl describe pods -l app=hostnames
• kubectl describe service/hostnames

Talos v1.7.2 & kubernetes v1.30.1 in my bare-metal homelab. Using flannel as the CNI.

[smira]

It's hard to say what exactly is wrong, try capturing traffic on the interface connecting the nodes and look for vxlan packets which Flannel should generate.

It might be confused by multiple IPs you're using, or it might be MTU issue.

[nogweii]

I tried doing a couple of packet captures, and I don't see any vxlan packets at all. Weird, they should always show up, right...?

Maybe I missed something, but here's been my process:

• Launch 2 netshoot containers with a node selector and hostnetworking enabled. (In the kube-system namespace, so the PodAdmissionPolicy is happy):

  kubectl run nodeX-host --rm -i --tty --image nicolaka/netshoot --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "nodeX" }, "hostNetwork": true } }'
  

• In both of these pods, I start a tcpdump capture: tcpdump -i any -w host.pcap

• Then launch a 3rd netshoot in the default namespace targeting the node 'redstone', to be my example client:

  kubectl run -n default test-client --rm -i --tty --image nicolaka/netshoot --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "redstone" } } }'
  

• In that test-client pod, I run ping, curl, etc, trying to reach the pod running on octarine (e.g. curl -vvv 'http://[fd0a:c031:a0af:cafe:4::2b]:9376')

• Then I finish the packet capture and copy them to my PC with kubectl cp and merge the two captures together in Wireshark.

But I don't see any evidence of the two nodes talking to each other. Lots of communication with the kubernetes endpoint, and with my router (DNS & NTP), but not much else of note.

[nogweii]

Hm, investigating it more, I hadn't even realized DNS doesn't work, since that too would require traversing nodes. And again, if I run a pod on that specific node, DNS will work just fine. (Both coredns pods ended up on the same node, so I can access either of them directly, or use the service kube-dns.kube-system.svc.cluster.local)

[nogweii]

Trying something else, I launched a new pod (in default namespace if that matters) and just tried pinging some things, and I'm getting some surprising results:

• The pod can ping my router (over ipv4 and ipv6), even get it's login page over HTTP
• It can ping other computers in my LAN, such as my laptop and NAS.
• But ping nor curl works in trying to access the Talos nodes over IPv4.
• However, the pod can ping the nodes, use curl against the apid GRPC port (getting an empty reply from the server, but still proving connection) when using IPv6.

Eh? What? Only IPv4 is broken, and I've noticed my router is adding a couple of odd IPv6 addresses to it's DNS response so I need to investigate that. Anyways, how does that happen? The IPv4 address is not static, it comes from DHCP.

[nogweii]

On a lark, I decided to enable KubeSpan hoping it might provide an alternative route that does allow the nodes to communicate with each other. It seems like no such luck. Dang.