Talos air-gapped installation challenges

[ossfellow]

Hello folks,
The Talos installation in the air-gapped environments doesn't work.

I have setup 5 private registries, mirroring all the release-notes-identified public registries, with the following notables:

• I've used Docker's Registry 2 image for the mirror registries.
• Let's encrypt certs are used for TLS.
• No Auth is used for accessing the registries.
• I'm installing Talos on ESX, mimicking an air-gapped environment.
• I'm using Talos v1.0.3.

I have loaded all the identified Talos images into the corresponding mirror registries and configured Talos to use these endpoints (talosctl ls /etc/cri/conf.d/hosts shows they're configured).
I can manually pull images from the mirror registries; so, I know they're functional.

However, what I observe is that Talos still attempts to use the unreachable public repositories instead of the private mirrors, which breaks the air-gapped installation.

To compare, I used Docker cli and it demonstrates the same behaviour as Talos (i.e. pulling an image from a public registry wouldn't automatically pull it from its private mirror; one must explicitly refer to the private registry to use it).
So, if Talos relies on Docker APIs for the mirror registries, then this test proves it will not produce the expected outcome.

Below is the config file and docker run command for one of the mirror registries:
config.yaml:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :443
  host: https://docker.camabri.com:5003
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://k8s.gcr.io

docker run:

sudo docker run --name k8s_ghcr_io -d \
-v $PWD/k8s.gcr.io:/etc/docker/registry \
-v /etc/letsencrypt/archive/docker.camabri.com:/certs \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain1.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey1.pem \
-p 5003:443 \
--restart always \
registry:2

If you use Docker cli to validate, please make sure you add { "registry-mirrors": ["https://<my-docker-mirror-host>"] } to /etc/docker/daemon.json and restart docker daemon first.

I'm testing the air-gapped installation, in hopes that it could be used for Talos installation on edge devices, which do not have Internet access. But, it doesn't work as it currently is.

Haven't I done something important, or does this point to a bug in Talos installation?

[smira]

I guess I'm missing something in your question, but Talos doesn't use Docker or anything else to pull the images. Talos pulls the images either on its own (installer, etcd, kubelet), or the containerd CRI plugin does that (kube-apiserver, workload images, etc.).

For Talos it's critical to setup registry mirror configuration in the machine config (or via --registry-mirror flags if using talosctl cluster create). This should redirect image pull request for each registry. I don't see those in your question above, and these are the most crtiical.

[ossfellow]

I used Docker cli to compare its behaviour with Talos installer (they're doing the same).

Here's my Talos configuration for the registry mirrors:

  registries:
    # Specifies mirror configuration for each registry.
    mirrors:
      docker.io:
        endpoints:
          - https://docker.camabri.com:5000
      gcr.io:
        endpoints:
          - https://docker.camabri.com:5001
      ghcr.io:
        endpoints:
          - https://docker.camabri.com:5002
      k8s.gcr.io:
        endpoints:
          - https://docker.camabri.com:5003
      quay.io:
        endpoints:
          - https://docker.camabri.com:5004

Please note that I've configured all the registries as pull-through cache (as indicated above) and loaded all images there.

What I'm observing is that Talos installer attempts to pull the images from the public registries, not the mirror registries I've setup.

You can test this, by removing Internet access, which should cause installation failure.

[frezbo]

i think there's a confusion here, for air-gapped you'd have to use a private registry, not pull through cache, talos use containerd config to setup mirror config, it doesn't work the way docker does. docker and containerd uses completely different config

[frezbo]

marking this as answered, just tested the offline install docs and it works fine

[ossfellow]

Thank you for the test; tonight, I’m going to make the changes and try air-gapped installation again. I only used the regular private registry with “*”, not the individual public registries, which earlier conversations clarified is needed.
I’ll be thrilled if everything works as expected.