Can't connect to node after apply, Certificate errors

[btrepp]

I'm trying to automate my cluster creation, by keeping the secrets.yaml checked in and encrypted with pass

I seem to be able to generate all the required files, and the machine yamls appear stable. However when I apply a machine, I instantly can't connect, even using the same context.

pass clusters/talos1/secrets.yaml > obj/secrets.yaml
talosctl gen config -t controlplane \
        --config-patch @control1.yaml \
        --output obj/control1.yaml \
        --with-secrets obj/secrets.yaml \
        talos1 https://rpi1.tail65d51.ts.net
generating PKI and tokens
Created obj/control1.yaml
talosctl apply-config \
		--file obj/control1.yaml \
		--insecure \
		-n rpi1.localdomain
rm obj/control1.yaml obj/secrets.yaml
talosctl -n rpi1.localdomain get logs                       
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")"
beautrepp@beaus-Air talos1 % talosctl config contexts
CURRENT   NAME     ENDPOINTS   NODES
*         talos1   127.0.0.1   

I've thought that maybe bootstrap needs to be run, but also fails

talosctl bootstrap -n rpi1.localdomain           
error executing bootstrap: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")"

Other people seem to have a mismatch in talosconfig, but I only have the one, so I am bit confused as to what's going on.

machine:
    type: controlplane # Defines the role of the machine within the cluster.
    certSANs:
        - rpi1
        - rpi1.localdomain
        - rpi1.tail65d51.ts.net

I've tried adding the extra sans, but also not happy.

Does anyone know the relationship between the talos api server, and the certificates?. This is obviously some mTLS error, but I don't really know where the 'talos' authority is defined, or how to view it.

Here is the full makefile

CLUSTER_INFO:=talos.yaml
CLUSTER_NAME:=$(shell yq '.name' $(CLUSTER_INFO))
CLUSTER_SECRETS:=$(shell yq '.secrets' $(CLUSTER_INFO))
CLUSTER_IP:= $(shell yq '.addr' $(CLUSTER_INFO))
CLUSTER_ADDR:= https://$(CLUSTER_IP)
CONTROL:=$(patsubst ./%,./obj/%,$(shell find . -name 'control*.yaml'))
WORKERS:=$(patsubst ./%,./obj/%,$(shell find . -name 'worker*.yaml'))
NODES:= $(CONTROL) $(WORKERS)
.INTERMEDIATE: $(NODES)

default: $(NODES) obj/talosconfig

obj:
	mkdir $@

.PHONY: clean
clean:
	rm -r obj

# Extracts the secrets from password store
# These seem to need to be kept
.INTERMEDIATE: obj/secrets.yaml
obj/secrets.yaml: $(CLUSTER_INFO) | obj
	pass $(CLUSTER_SECRETS) > $@

# Creates a talosconfig from the secrets
.INTERMEDIATE: obj/talosconfig
obj/talosconfig: obj/secrets.yaml
	talosctl gen config -t talosconfig \
		--with-secrets obj/secrets.yaml \
		--output $@ \
		$(CLUSTER_NAME) $(CLUSTER_ADDR)

# Makes control plane nodes
.INTERMEDIATE: obj/control%.yaml
obj/control%.yaml: control%.yaml obj/secrets.yaml $(CLUSTER_INFO) | obj
	talosctl gen config -t controlplane \
        --config-patch @$< \
        --output $@ \
        --with-secrets obj/secrets.yaml \
        $(CLUSTER_NAME) $(CLUSTER_ADDR)

# Makes worker nodes
.INTERMEDIATE: obj/worker%.yaml
obj/worker%.yaml: worker%.yaml obj/secrets.yaml $(CLUSTER_INFO) | obj
	talosctl gen config -t worker \
		--config-patch @$< \
		--output $@ \
		--with-secrets obj/secrets.yaml \
		$(CLUSTER_NAME) $(CLUSTER_ADDR)

# Target to run when a node is a 'blank' imaged
# This will initally configure the node
.PHONY: init-%
init-%: obj/%.yaml
init-%: obj/%.yaml talos.yaml
	talosctl apply-config \
		--file $< \
		--insecure \
		-n $(shell yq '.nodes.$(patsubst obj/%.yaml,%,$<).init' talos.yaml)

# Run normally (uses tailscale, so you need to be in the tailnet)
.PHONY: apply-%
apply-%: obj/%.yaml
apply-%: obj/%.yaml talos.yaml
	talosctl apply-config \
		--file $< \
		-n $(shell yq '.nodes.$(patsubst obj/%.yaml,%,$<).secure' talos.yaml)

# Copy talos config to user space
.PHONY: merge
merge: obj/talosconfig
	talosctl config merge $<

[steverfrancis]

Try passing in the generated talosconfig file directly to eliminate that you aren’t picking up the wrong one… Append “—talosconfig=./talosconfig.yaml” to the ‘get logs’ command

…

On Sat, Apr 22, 2023 at 3:44 AM btrepp ***@***.***> wrote: I'm trying to automate my cluster creation, by keeping the secrets.yaml checked in and encrypted with pass This seems to generate all the required files, and the machine yamls appear stable. However when I apply a machine, I instantly can't connect, even using the same context. pass clusters/talos1/secrets.yaml > obj/secrets.yaml talosctl gen config -t controlplane \ --config-patch @control1.yaml \ --output obj/control1.yaml \ --with-secrets obj/secrets.yaml \ talos1 https://rpi1.tail65d51.ts.net generating PKI and tokens Created obj/control1.yaml talosctl apply-config \ --file obj/control1.yaml \ --insecure \ -n rpi1.localdomain rm obj/control1.yaml obj/secrets.yaml ***@***.*** talos1 % talosctl -n rpi1.localdomain get logs rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")" ***@***.*** talos1 % talosctl config contexts CURRENT NAME ENDPOINTS NODES * talos1 127.0.0.1 Other people seem to have a mismatch in talosconfig, but I only have the one, so I am bit confused as to what's going on. machine: type: controlplane # Defines the role of the machine within the cluster. certSANs: - rpi1 - rpi1.localdomain - rpi1.tail65d51.ts.net I've tried adding the extra sans, but also not happy. Does anyone know the relationship between the talos api server, and the certificates?. This is obviously some mTLS error, but I don't really know where the 'talos' authority is defined, or how to view it. Here is the full makefile CLUSTER_INFO:=talos.yaml CLUSTER_NAME:=$(shell yq '.name' $(CLUSTER_INFO)) CLUSTER_SECRETS:=$(shell yq '.secrets' $(CLUSTER_INFO)) CLUSTER_IP:= $(shell yq '.addr' $(CLUSTER_INFO)) CLUSTER_ADDR:= https://$(CLUSTER_IP) CONTROL:=$(patsubst ./%,./obj/%,$(shell find . -name 'control*.yaml')) WORKERS:=$(patsubst ./%,./obj/%,$(shell find . -name 'worker*.yaml')) NODES:= $(CONTROL) $(WORKERS) .INTERMEDIATE: $(NODES) default: $(NODES) obj/talosconfig obj: mkdir $@ .PHONY: clean clean: rm -r obj # Extracts the secrets from password store # These seem to need to be kept .INTERMEDIATE: obj/secrets.yaml obj/secrets.yaml: $(CLUSTER_INFO) | obj pass $(CLUSTER_SECRETS) > $@ # Creates a talosconfig from the secrets .INTERMEDIATE: obj/talosconfig obj/talosconfig: obj/secrets.yaml talosctl gen config -t talosconfig \ --with-secrets obj/secrets.yaml \ --output $@ \ $(CLUSTER_NAME) $(CLUSTER_ADDR) # Makes control plane nodes .INTERMEDIATE: obj/control%.yaml obj/control%.yaml: control%.yaml obj/secrets.yaml $(CLUSTER_INFO) | obj talosctl gen config -t controlplane \ --config-patch @$< \ --output $@ \ --with-secrets obj/secrets.yaml \ $(CLUSTER_NAME) $(CLUSTER_ADDR) # Makes worker nodes .INTERMEDIATE: obj/worker%.yaml obj/worker%.yaml: worker%.yaml obj/secrets.yaml $(CLUSTER_INFO) | obj talosctl gen config -t worker \ --config-patch @$< \ --output $@ \ --with-secrets obj/secrets.yaml \ $(CLUSTER_NAME) $(CLUSTER_ADDR) # Target to run when a node is a 'blank' imaged # This will initally configure the node .PHONY: init-% init-%: obj/%.yaml init-%: obj/%.yaml talos.yaml talosctl apply-config \ --file $< \ --insecure \ -n $(shell yq '.nodes.$(patsubst obj/%.yaml,%,$<).init' talos.yaml) # Run normally (uses tailscale, so you need to be in the tailnet) .PHONY: apply-% apply-%: obj/%.yaml apply-%: obj/%.yaml talos.yaml talosctl apply-config \ --file $< \ -n $(shell yq '.nodes.$(patsubst obj/%.yaml,%,$<).secure' talos.yaml) # Copy talos config to user space .PHONY: merge merge: obj/talosconfig talosctl config merge $< — Reply to this email directly, view it on GitHub <#7122>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5INTIZCBXZU26VJ773XCOZBVANCNFSM6AAAAAAXHYGSHI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[btrepp]

Ah awesome, that's a great lead.

talosctl -n rpi1.localdomain --talosconfig=~/.talos/config get logs
error constructing client: failed to determine endpoints
talosctl -n rpi1.localdomain get logs  
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")"

The below does seem to work

talosctl -n rpi1.localdomain -e rpi1.localdomain --talosconfig=/Users/beautrepp/.talos/config containers
NODE               NAMESPACE   ID       IMAGE   PID    STATUS
rpi1.localdomain   system      apid             3505   RUNNING
rpi1.localdomain   system      trustd           3507   RUNNING

Does not though

talosctl --talosconfig=~/.talos/config -n rpi1.localdomain  -e rpi1.localdomain get containers
error constructing client: failed to create client connection: failed to resolve configuration context: talos config file is empty

Which prompted me to look and it creates a ~ folder in the working directory

beautrepp@beaus-Air talos1 % ls -la
total 48
drwxr-xr-x  10 beautrepp  staff   320 23 Apr 09:20 .
drwxr-xr-x   3 beautrepp  staff    96  5 Dec 20:44 ..
-rw-r--r--   1 beautrepp  staff     5 15 Apr 14:41 .gitignore
-rw-r--r--   1 beautrepp  staff  2097 23 Apr 09:19 Makefile
-rw-r--r--   1 beautrepp  staff   111 22 Apr 17:44 control1.yaml
drwxr-xr-x   3 beautrepp  staff    96 23 Apr 09:20 obj
-rw-r--r--   1 beautrepp  staff   304 22 Apr 18:05 talos.yaml
-rw-r--r--   1 beautrepp  staff   111 22 Apr 17:44 worker1.yaml
-rw-r--r--   1 beautrepp  staff   111 22 Apr 17:44 worker2.yaml
drwx------   3 beautrepp  staff    96 23 Apr 09:20 ~

With an empty talos config.

I think maybe there is an issue with passing '~' and how talosctl is looking up home directories (at least on macOS).

talosctl  -n rpi1.localdomain  -e rpi1.localdomain containers 
NODE               NAMESPACE   ID       IMAGE   PID    STATUS
rpi1.localdomain   system      apid             3505   RUNNING
rpi1.localdomain   system      trustd           3507   RUNNING

Seems to work now after deleting that folder. I'm not able to as easily recreate, and it also seems a bit strange in consistency.

[smira]

~ should be resolved by your shell, not talosctl, so it's not related to Talos