It is possible to set a private IP in endpoint for single node clusters?

[paucampana]

Hello,

I am trying to configure a single node clusters. I am wondering if it is possible to set a private IP in the cluster endpoint for a single node cluster.

I am creating Talos clusters using AWS. Here is my code (using sideroLabs terraform module for Talos):

resource "talos_machine_secrets" "machine_secrets" {}

resource "talos_machine_configuration_controlplane" "machineconfig_cp" {
  cluster_name     = "test-talos-tf"
  cluster_endpoint = "https://10.0.0.19:6443"
  machine_secrets  = talos_machine_secrets.machine_secrets.machine_secrets
  config_patch     = <<EOT
machine:
  type: init
EOT
  docs_enabled = false
  examples_enabled = false
  kubernetes_version = "1.23.9"
  talos_version = "v1.2.0"
}


data "talos_client_configuration" "talosconfig" {
  cluster_name    = "test-talos-tf"
  machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
}


output "talos_config_test_talosconfig" {
  value = data.talos_client_configuration.talosconfig
  sensitive = true
}

output "talos_config_test_controlplane" {
  value = talos_machine_configuration_controlplane.machineconfig_cp.machine_config
  sensitive = true
}


provider "aws" {
  region = "eu-central-1"
}


resource "aws_instance" "web" {
  ami           = "ami-0a567162d9680e404"
  instance_type = "m6a.xlarge"
  user_data = talos_machine_configuration_controlplane.machineconfig_cp.machine_config
  subnet_id   = "SUBNET-IDE"
  vpc_security_group_ids = ["SECURITY-GROUP-ID"]
  associate_public_ip_address = true
  tags = {
    Name = "test-talos"
  }
  private_ip = "10.0.0.19"
}

The same code using a load balancer dns as the cluster endpoint worked fine. Now, that I am trying to use a private IP (for this example is 10.0.0.19, but I also tried with 127.0.0.1), it looks like all talos services are okay:

talosctl health
discovered nodes: ["10.0.0.19"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: OK
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

talosctl service
NODE             SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
IP-PUBLIC           apid         Running   OK       2m15s ago     Health check successful
IP-PUBLIC           containerd   Running   OK       2m16s ago     Health check successful
IP-PUBLIC           cri          Running   OK       2m14s ago     Health check successful
IP-PUBLIC           etcd         Running   OK       2m3s ago      Health check successful
IP-PUBLIC           kubelet      Running   OK       1m57s ago     Health check successful
IP-PUBLIC           machined     Running   ?        2m18s ago     Service started as goroutine
IP-PUBLIC           trustd       Running   OK       2m12s ago     Health check successful
IP-PUBLIC           udevd        Running   OK       2m15s ago     Health check successful

But I can not connect to kubernetes cluster. I am getting the kubeconfig file using the command talosctl kubeconfig . and editing the cluster service IP to the external ip of the ec2 instance. But when running a kubectl command I am getting the following error:

kubectl get pod -A
Unable to connect to the server: x509: certificate signed by unknown authority

I have tried to add the external IP to the certSANs field, but I am getting the same error.

Any idea what I should do to be available to run kubectl commands? or it is not possible to set a private IP in the endpoint? Thanks!

[smira]

when you run kubectl without any additional flags, it uses kubeconfig from a default location

when you run talosctl kubeconfig . (notice the dot!), you save it to the file ./kubeconfig

so you either:

• run talosctl kubeconfig which saves to default location
• specify kubeconfig to kubectl with either kubectl --kubeconfig=./kubeconfig or KUBECONFIG=./kubeconfig kubectl

The error you showed means that your kubeconfig doesn't match the cluster.

[paucampana]

I am getting the correct kubeconfig file. I forgot to add in the tiket that I am running the command export KUBECONFIG=/talos/dir/kubeconfig

So when I run the talosctl kubeconfig . It is generating a kubeconfig file:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxxx
    server: https://10.0.0.19:6443
  name: test-talos-tf
contexts:
- context:
    cluster: test-talos-tf
    namespace: default
    user: admin@test-talos-tf
  name: admin@test-talos-tf
current-context: admin@test-talos-tf
kind: Config
preferences: {}
users:
- name: admin@test-talos-tf
  user:
    client-certificate-data: xxxxx
    client-key-data: xxxxx

As I am using the ec2 private IP in the endpoint when generating the talos configuration, it is also adding the private IP to clusters[0].cluster.server

As the EC2 is configured with the associate_public_ip_address = true flag, it has also a public IP. Using this IP looks okay for talos (I can connect to the cluster and run talosctl commands without problem), but I am having problems using kubectl.

So for an external IP 1.2.3.4, I edit the server field to server: https://1.2.3.4:6443. but as I have explained I can not connect to the cluster

[frezbo]

you'd need this: https://www.talos.dev/v1.2/reference/configuration/#apiserverconfig
/cluster/apiServer/certSANs and add the public one

[paucampana]

Yes, I also tried to add the external IP to certSANs, but same error:
Adding the public IP in machine config:

cluster:
    apiServer:
        certSANs:
            - 10.0.0.19
            - PUBLIC-IP

talosctl edit machineconfig -n PUBLIC-IP --mode=no-reboot
Applied configuration without a reboot

And I continue having the same error. Also after a reboot talosctl reboot -n PUBLIC-IP I have the same error.

certSANs can be modified after cluster creation?

[frezbo]

yes, it's updated on fly. seems some other issue. maybe try kubectl --kubeconfig explicitly, still feels like using wrong kubeconfig.

[paucampana]

Same with kubeconfig. I will continue doing some troubleshotting but I think it is correct..

Focussing in the problem, should I be available to use the private IP of the ec2 for the endpoint https:PRIVATE-IP:6443 used for talosconfig, and then access the cluster using the public ip of the ec2?

[smira]

your error is all about wrong PKI being used.

both talosconfig and kubeconfig need and endpoint that can be reached from your device.

Talos machine configuration allows for additional certSANs for both Talos API and Kubernetes API.

In fact Talos on AWS automatically detects external public IP and adds it to the certSANs.

[paucampana]

I do not have a clear idea what I was doing wrong..I started again using for all the commands --talosconfig and --kubeconfig and it worked. If I find what was the origin of the problem I will add a new comment here.

Apart from the problem of the connection to the cluster, the answer to the original question "It is possible to set a private IP in endpoint for single node clusters?" is yes. I have tried using the private IP and the localhost 127.0.0.1 and in both cases it have worked. The only thing I have modified in these cases is editting the server for the public ip of the ec2 (after getting the kubeconfig talosctl --talosconfig talosconfig kubeconfig .)

Thanks for your time! 🙇‍♂️