How to create a Kubernetes Secret for Talos ServiceAccount?

[zulh-civo]

We're running Talos ETCD backups using steps described here and it was working for us for > 1 year. However, it has stopped working with this error:

error reading file: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: bad certificate"

Looking at the CronJob manifest, the talosctl init container is using talos-secrets volume which is etcd-backup-talos-secrets Kubernetes Secret. And that Secret was created by the serviceaccounts.talos.dev (line 76).

When I decoded the etcd-backup-talos-secrets Kubernetes Secret and drill down to .contexts.default.crt key, I can confirmed the cert has expired a few days ago.

Validity
    Not Before: May 18 11:44:18 2024 UTC
    Not After : May 18 17:44:18 2024 UTC

I also noticed the gap between Not Before vs Not After is just 6 hours long — something that I wasn't aware and totally missed before. I was expecting the Not Before to show last year date i.e. May 18 hh:mm:ss 2023 UTC when I created the Talos ServiceAccount. That's different topic and let's not focus on that.

Back to topic, from memory, I can't recall creating that etcd-backup-talos-secrets Kubernetes Secret. I believe it was created when I apply the CronJob manifest above which contains serviceaccounts.talos.dev object. In other words, I believe the etcd-backup-talos-secrets Kubernetes Secret was created when etcd-backup-talos-secrets Talos ServiceAccount was created.

Now, when I deleted etcd-backup-talos-secrets Talos ServiceAccount, its etcd-backup-talos-secrets Kubernetes Secret also get deleted and that's expected. However, when I recreated the etcd-backup-talos-secrets Talos ServiceAccount, I don't see any Kubernetes Secret gets created and linked to it. I even ran k -n kube-system get secrets --sort-by=.metadata.creationTimestamp to clearly see what's the latest Kubernetes Secret in my namespace.

Am I missing something here? I've read the Talos API access from Kubernetes docs but I can't see any part about "recreating Talos ServiceAccount's Secret". I'd appreciate if someone could help me with this issue.

FWIW, I'm running Talos v1.2.4 and Kubernetes v1.24.8 for this cluster.

[smira]

First of all, please consider upgrading to the latest version of Talos, as it would bring bug fixes, security fixes and new features.

Considering your issue, this sounds like a bug, but I would ask @utkuozdemir to verify my words.

It would help if you could provide talosctl logs controller-runtime from the controlplane nodes to see what is going on.

[zulh-civo]

Thanks for your reply, @smira. It's fixed now. The issue was related to one of our control-plane — we restarted its kubelet and the ETCD backup is running now.