bootstrap fails on Azure Stack Hub

[a01fe]

I'm trying to create a cluster following the Azure installation guide and I'm stuck at bootstrapping bootstrapping etcd with this error (where A.B.C.D is the talos-controlplane-public-ip-0 IP address):

$  talosctl --talosconfig ./talosconfig bootstrap
error executing bootstrap: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for 10.0.0.4, 127.0.0.1, ::1, not A.B.C.D"

10.0.0.4 is the private IP address assigned to the nic.

Could this be happening because the VM isn't seeing the public address on the nic? I'm guessing that ASH behaves with respect to public IP addresses on nics. Is there a way that I can eplicitly tell the Talos VM what it's public IP address is?

[smira]

Yes, see https://www.talos.dev/v1.6/reference/configuration/v1alpha1/config/#Config.machine

You need a patch like that:

machine:
   certSANs:
      - A.B.C.D

[a01fe]

I changed talosctl gen config to:

 talosctl gen config talos-k8s-azure-tutorial https://${LB_PUBLIC_IP}:6443 --additional-sans "$CONTROL_PLANE_0_IP"

Removed the VMs and reran steps starting with "Cluster Configuration". controlplane.yaml now starts with:

version: v1alpha1 # Indicates the schema used to decode the contents.
debug: false # Enable verbose logging to the console.
persist: true
# Provides machine specific configuration options.
machine:
    type: controlplane # Defines the role of the machine within the cluster.
    token: xxxxxxx # The `token` is used by a machine to join the PKI of the cluster.
    # The root certificate authority of the PKI.
    ca:
        crt: xxxxx
        key: xxxxxx
    # Extra certificate subject alternative names for the machine's certificate.
    certSANs:
        - A.B.C.D

I'm still getting the same error from talosctl bootstrap. Should certSANs go somewhere else or should I be doing something to apply this change?

error executing bootstrap: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for 10.0.0.4, 127.0.0.1, ::1, not A.B.C.D"

[smira]

connecting to a different node? hard to guess. the config looks correct.

[a01fe]

I see this in control plane boot logs, could this be relevant?

[    9.097928] [talos] fetching azure instance config from: "http://169.254.169.254/metadata/instance/compute?ai-version=2021-12-13&format=json"
[    9.127572] [talos] retrying error: failed to download config from "http://169.254.169.254/metadata/instance/compute?api-version=2021-12-13&format=json": Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-12-13&format=json": dial tcp 169.254.169.254:80: connect: network is unreachable

I'm not sure why talos gets network unreachable, I'm able to curl that endpoint from other ASH VMs. I had to change api-version=2021-05-01, it appears that's the latest version ASH supports.

[smira]

I'd guess it's an early boot error which later resolves itself.

[a01fe]

Yes, looks like that error happened before the network was ready. These messages repeat at the tail:

[ 1472.072821] [talos] fetching azure instance config from: "http://169.254.169.254/metadata/instance/compute?api-version=2021-12-13&format=json"
[ 1473.128590] [talos] retrying error: failed to download config from "http://169.254.169.254/metadata/instance/compute?api-version=2021-12-13&format=json": failed to download config, status code 400, body "{ \"error\": \"Bad request. api-ver"
[ 1652.095158] [talos] restarting platform network config {"component": "controller-runtime", "controller": "network.PlatformConfigController", "interval": "11.488821786s", "error": "error fetching metadata: 2 error(s) occurred:\n\tfailed to download config from \"http://169.254.169.254/metadata/instance/compute?api-version=2021-12-13&format=json\": failed to download config, status code 400, body \"{ \\\"error\\\": \\\"Bad request. api-ver\"\n\ttimeout"}

I tried putting this in controlplane.yaml and rebuilt VMs, hoping to change api-version in the request, but it didn't change anything. Is there a way to change this URL?

machine:
    install:
        extraKernelArgs:
          - "talos.platform=http://169.254.169.254/metadata/instance/compute?api-version=2021-05-01&format=json"

[smira]

what you did would be ignored, and it doesn't make sense.

please create a GitHub issue with the exact Talos version and the kernel logs above.