Re-added node to talos cluster but it is not shown in kubectl get nodes

[dpasman]

Hi community,

Recently I had an issue on a node in our cluster which left all pods on terminating state after reboot, after rebooting static pods did not seem to be come up back again. Because of this I decided to reset the node, deleted the node from the kubernetes cluster and tried to re-add the node to the cluster.

I got the working configs from a working node by running talosctl -n 10.100.20.24 get machineconfig -o yaml | yq eval .spec
Then I changed the IP adresses to the desired IP adres the node had before resetting it.
After this apply Talos rebooted the node and joined the cluster succesfully. However, it seems to be a seperate kubernetes cluster now? Talos Health is showing the following output:

> `❯ talosctl -n 10.100.20.19 health
> discovered nodes: ["10.10.42.1" "10.10.42.2" "10.10.42.3"]
> waiting for etcd to be healthy: ...
> waiting for etcd to be healthy: OK
> waiting for etcd members to be consistent across nodes: ...
> waiting for etcd members to be consistent across nodes: 10.10.42.3: expected to have [controlplane-1] members, got [controlplane-2  controlplane-3]
> waiting for etcd members to be consistent across nodes: OK
> waiting for etcd members to be control plane nodes: ...
> waiting for etcd members to be control plane nodes: OK
> waiting for apid to be ready: ...
> waiting for apid to be ready: OK
> waiting for all nodes memory sizes: ...
> waiting for all nodes memory sizes: OK
> waiting for all nodes disk sizes: ...
> waiting for all nodes disk sizes: OK
> waiting for kubelet to be healthy: ...
> waiting for kubelet to be healthy: OK
> waiting for all nodes to finish boot sequence: ...
> waiting for all nodes to finish boot sequence: OK
> waiting for all k8s nodes to report: ...
> waiting for all k8s nodes to report: can't find expected node with IPs ["10.10.42.1" "10.100.20.19" "10.100.20.21"]
> healthcheck error: rpc error: code = DeadlineExceeded desc = context deadline exceeded`

Interesting to see is that the first node is looking on a different network (we are using this network for ceph storage). But I am 100% sure I added the right IP adresses to the right interfaces in the configuration file. In the talos dashboard I see the VIP is attached to both controlplane-1 and controlplane-3 (find output below as well).

Talos get members is showing the following output:

❯ talosctl -n 10.100.20.19 get members
NODE           NAMESPACE   TYPE     ID                                  VERSION   HOSTNAME                            MACHINE TYPE   OS               ADDRESSES
10.100.20.19   cluster     Member   controlplane-1   2         controlplane-1   controlplane   Talos (v1.4.5)   ["10.10.42.1","10.100.20.19","10.100.20.21"]
10.100.20.19   cluster     Member   controlplane-2   1         controlplane-2   controlplane   Talos (v1.5.3)   ["10.10.42.2","10.100.20.24"]
10.100.20.19   cluster     Member   controlplane-3   1         controlplane-3   controlplane   Talos (v1.5.3)   ["10.10.42.3","10.100.20.19","10.100.20.23"]

Important to mention is that the initial image is based on Talosv1.4.5 (we manage our environments via terraform) and the cluster is on v1.5.0.

Is anyone else having a issue like this? Please let me know if more logs/output is needed for further troubleshooting :)

Edit:
Hereby the config of the network configuration of the machine:

  network:
    hostname: controlplane-1
    interfaces:
      - deviceSelector:
          hardwareAddr: xx:xx:xx:xx:xx:xx
          driver: virtio_net
        addresses:
          - 10.100.20.21/16
        routes:
          - network: 10.100.0.0/16
            gateway: 10.100.0.1
        dhcp: false
        vip:
          ip: 10.100.20.19
      - deviceSelector:
          hardwareAddr: xx:xx:xx:xx:xx:xx
          driver: virtio_net
        addresses:
          - 10.10.42.1/24
        dhcp: false
    nameservers:
      - 10.100.20.13
    extraHostEntries:
      - ip: 127.0.0.1
        aliases:
          - https://10.100.20.19:6443
    kubespan:
      enabled: false

[smira]

The only error in your output is the mismatch on expected IPs in kubectl get nodes -o wide, you can see yourself why it doesn't match, there might be many reasons for it, but that is minor, and depends on kubelet configuration/multi-homed networking.

The etcd mismatch was transient - probably your machine was joining at that moment.