Custom registry certificate causes failure to pull installation image

[j-lgs]

I'm trying to ascertain whether the behaviour I've ran into is user error on my part or a bug within Talos. The Talos installation ISO fails to pull the installation image if a TLS secured pull through registry is specified. The error is as follows.

[talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.0.4": failed to resolve reference "ghcr.io/siderolabs/installer:v1.0.4": failed to do request: Head "https://10.0.1.8:5004/v2/siderolabs/installer/manifests/v1.0.4?ns=ghcr.io": x509: certificate signed by unknown authority.

The docker registry is properly set up and certs are valid.

curl \
  --cacert secrets/registry/ca/ca.pem \
  --cert secrets/registry/registry-client/crt.pem \
  --key secrets/registry/registry-client/key.pem \
  https://10.0.1.8:5004/v2/_catalog
{"repositories":["ghcr.io/siderolabs/installer","siderolabs/installer","siderolabs/kubelet"]}

This error persists when setting the insecureSkipVerify parameter in a RegistryTLSConfig for ghcr.io. Is this error unrelated to the TLS server certificate verification that's disabled by that flag?

Using self signed registry server and client certiciates gave the same error as using one signed by my internal CA. Again it is set up just fine on the registry side of things, confirming it is a Talos issue.

curl -k \
  --cert secrets/registry/registry-client/ss-crt.pem \
  --key secrets/registry/registry-client/key.pem https://10.0.1.8:5004/v2/_catalog
{"repositories":["ghcr.io/siderolabs/installer","siderolabs/installer","siderolabs/kubelet"]}

Additionally the documentation examples show the key and crt paramaters of clientIdentity as being doubly base64 encoded. I do not understand why this would be the case if it's being encoded and decoded once.

How would I go about inspecting the containerd registry hosts configuration located at /etc/cri/conf.d/01-registries.part that's being generated prior to the installation image being pulled?

I have included a dump of the VM's dmesg when attempting to perform the installation
debug-dump.txt

The following commands were performed to generate the registry and client keys.

# Generate CA password, key and certificate.
openssl rand -base64 32 > test-ca.pass
openssl genrsa -aes256 -out test-ca-key.pem -passout file:test-ca.pass 4096
openssl req -new -x509 -days 365 -key test-ca-key.pem -sha256 -subj "/C=NA/ST=NA/L=NA/O=NA/CN=NA" -out test-ca-crt.pem -passin file:test-ca.pass

# Generate registry server key, signed by previously created CA.
openssl genrsa -out test-server-key.pem 4096
openssl req  -sha256 -new -key test-server-key.pem -subj "/C=NA/ST=NA/L=NA/O=NA/CN=10.0.1.8" -out test-server.csr
tee server-extfile << EOF
subjectAltName=IP:10.0.1.8
echo extendedKeyUsage=serverAuth
EOF
openssl x509 -req -days 365 -sha256 -in test-server.csr \
  -CA test-ca-crt.pem -CAkey test-ca-key.pem -CAcreateserial \
  -out test-server-crt.pem -extfile server-extfile.cnf -passin file:test-ca.pass

# Generate client key and certificate signed by previously created CA.
openssl genrsa -out test-client-key.pem 4096
openssl req -subj '/[email protected]' -new -key test-client-key.pem -out test-client.csr
echo extendedKeyUsage = clientAuth > test-client-extfile.cnf
openssl x509 -req -days 365 -sha256 -in test-client.csr \
  -CA test-ca-crt.pem -CAkey test-ca-key.pem -CAcreateserial \
  -out test-client-crt.pem -extfile test-client-extfile.cnf -passin file:test-ca.pass

Any insights or obvious things I have overlooked?

Thanks.

[frezbo]

could you provide the machine config used ?

[smira]

to be more explicit, what you probably need is the following:

   registries:
        mirrors:
            ghcr.io:
                endpoints:
                    - https://10.0.1.8:5004
        config:
            10.0.1.8:5004:
                tls:
                    ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZZekNDQTB1Z0F3SUJBZ0lVYXptTWNITVdvVUIrK211VEk2aXQ0ZHBIaVhjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNUa0V4Q3pBSkJnTlZCQWdNQWs1Qk1Rc3dDUVlEVlFRSERBSk9RVEVMTUFrRwpBMVVFQ2d3Q1RrRXhDekFKQmdOVkJBTU1BazVCTUI0WERUSXlNRFV4TnpBMk1UZ3dNRm9YRFRJek1EVXhOekEyCk1UZ3dNRm93UVRFTE1Ba0dBMVVFQmhNQ1RrRXhDekFKQmdOVkJBZ01BazVCTVFzd0NRWURWUVFIREFKT1FURUwKTUFrR0ExVUVDZ3dDVGtFeEN6QUpCZ05WQkFNTUFrNUJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUExTmhXQ29odkZPMjg0R1ljUkhidDVwYVJobStGSTY3Q1NrYjdqeHNzVHhobWpQd2d5RE9CCmZUUnVDVWtLMXJQclJ5WjdJem93VWZyWGl5cFpvUFFFOFRnRFFlSmlvdDVmUm9SeENlL295QWZ3ei9sUnNKeE0Kd2RocnZ2SjRVeUhtY2xpSHFJZCtyUS94aEhuMmFCQThZV3dYVkxTRlY3NEJyTjFWZkw4SjJBSStNQjBRenBzWApDd1hlSXdhazZDRWIxZkk4QmVLMUhqc05LNFc2QjgwOXlJNXU4SS9DNVVOTnpnNU8yMDNzVng2cXROL2J2MmdLCmhwM216R3d3VDZQcENIcGttTURHbjU4ZWExOTBHWW9Rdlh4aFd5NE1WalVTTG50MnR1WFZ0U1lrbnQ2cjgxKzcKcVZTWXFmU05oanlsNTRDbEExeTRaM21lYisvTVVkMFdIRzQ1MTdrajlHbUVZaVZmSFNVSTdtMVI5YkVXNThuMwpNcmRvQlV6SnZETWZuVnhyNU5tY3JmRjdCU3MrOTBQbGRCZ2d1WlRrb0lLZUJ3bEg3aTZnY05jN29WQlUyNzJHCmN1SDJEUy92dmxZWkRhaFlaUEJ6N0lxWHl6cXQ3OEdmZitFM1J0bnhzN245YzBObkl3MDh5Yk5jcDl4bmt2c2oKUk0xalhjL2xmR25hNy8xSGU4WlZQaVBYOThaRDB5R3pnL1phQmRmUVEwMk1ORXA5RW41aVM2bG9kdlBYcTBWcwp0OVhGM3g2YTBNZkQwdTVxWFVKKzRhNTFTV3FRYXQ5b0tDakhDWXZ0YzZXYU5CN1RVQ0RFbVhUVWoxZHF0eitaCmFuSmlPZE1aamovUEMyK2xGNEZmYVpaTmxDM2gyL3R2TGFoSUd4QTd6SktTZGlyV250c1lFcWtDQXdFQUFhTlQKTUZFd0hRWURWUjBPQkJZRUZGUGJUUEhPb1VlTnEyMy9UdmxtZEpqV29LNUFNQjhHQTFVZEl3UVlNQmFBRkZQYgpUUEhPb1VlTnEyMy9UdmxtZEpqV29LNUFNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMCkJRQURnZ0lCQUIrVFVsK3ptbDNCdEJ2aitLWnUzeDgvOFFyNWYyc0VnVXBJeldPZ0lLRW4wQi9IVHd4VzFnNVoKTXZUb0FMNkxoTURFNW9KZTZ1R1VETTFvOHdxS044K3F2cVp4RFBwUGRaQlZtQ2dKV2lPdGhSOFppWXgzam8wYwo4dXpxUkpGOS9ZWERlWDMvakxZMSs5eCtQaktaWGVtYmZ6MW91WTlDT0VKSWVVb3ZFbW5YcnpkSnpQYlF4RmI2ClJMckdVQmpQM2RDM3hUbzRuUXVGd3o2MXkvb1VtV2FoMExJeUpMUWMySjBPM2tOWDVFMnJCL0ZFR1ZQWm9pcSsKMUkwODNRblBHTFdteUhxZTZ0RS90VS8zVkhsTnIyWjFTZVRPTWZIM3lJTU5WK210OW1xUWlVcWw0di9rQ0NEKwpRRHR5aVlFVEkyUmprYTc2VWNSbHRrekRXWVlSL1k0QnE1VEVlTGpDc0o4UExXVVMwQUFvaTd5L1p4U1pDaVhBCmJZdWhUb0U2SDUrNk1BTk5Md1FObGw5NEFndXFPYW9WeTRLbXd1Y3k3MmNGdUoyRU9DSXpIbGZ3TW5JdjkzK3IKV2ZSTWcwL1JPd2dzM202eGFlSno1UzUvaWdhelVudnNKd3h4RlFLYk9sbldVQ3dBdHUxUGtZbEJxY0M4NUttUgpLMEphcGlNTWN6bUcwMW1lTjJjMUVpOUtUQ1FTLy96eHhyV2hhTEs3bEZBSnBGNHhsU1craVYwd2dKcC9qeFQ1CkVVemhhZWtNU3l2SjFBUVJkaXYwOU13cjEvZmJ5Y0pHQjhSYTlnU0FlOGZVeUFSNnZhUmFNY1JSUUhNWXVlZFoKLzF6bHR6VWFZNnhFaVZ4cXU1YVgxbEJMbjN0dmVZRllrR3B5TWdGaUQzbld0SnAzVG9YeAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
                    clientIdentity:
                       crt: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVWlJWRU5EUVhsdFowRjNTVUpCWjBsVlFqTm9VM2h6VjJKclQwZEZNRXB2TjFWRFJrUlFWM0ZGVVRWWmQwUlJXVXBMYjFwSmFIWmpUa0ZSUlV3S1FsRkJkMUZVUlV4TlFXdEhRVEZWUlVKb1RVTlVhMFY0UTNwQlNrSm5UbFpDUVdkTlFXczFRazFSYzNkRFVWbEVWbEZSU0VSQlNrOVJWRVZNVFVGclJ3cEJNVlZGUTJkM1ExUnJSWGhEZWtGS1FtZE9Wa0pCVFUxQmF6VkNUVUkwV0VSVVNYbE5SRlY0VG5wQk1rMVVaM2ROVm05WVJGUkplazFFVlhoT2VrRXlDazFVWjNkTlZtOTNSM3BGV2sxQ1kwZEJNVlZGUVhkM1VWcEhWakpSUjFKc1pHMUtkbVZETlhOaU1rNW9Za1JEUTBGcFNYZEVVVmxLUzI5YVNXaDJZMDRLUVZGRlFrSlJRVVJuWjBsUVFVUkRRMEZuYjBOblowbENRVTV0UW5GcGFYbHFObFpUYldSYVVIRnFSRXRFVlRWTVVITTVXV040TkhvelRXa3ZUbWh0Y1FwWVl6ZE1RekZFTUdkelRXdzJlVm81Vm5aMllWQkxUSEZ3VjNCdVJGbEdkM1pZVWxwSWVrdFRibFJTYldNMlJUSXJRbU52WW5aeFVqZFBVbXQyTjJsdkNtSk5lSFJ1VFdsNU0zVkxkWGQ0TURoM2RHbHlVVEJDY3pKdldrVkpNQ3RJZURnd1FrMTFlRU5NYmxkQ0wyODFkbk40UjJjemEzcHVkM1p2U1ZSWk0wTUtXbkJhTlVaSlZGY3JUVmt5VVZneU1YUmtkemt5VUZKQ2JGTnZOVlZ6Tm1WWGFsRXhURTFsTjNJdlVYWmxiblZ4YTJJdllUQnhSMWhJVDFwNk0xVlRkQXBQVkU5clVuQlRka05GWVM5d2VsRnZTVTFvZWxSMlVWcENXVWRQVEdVMlRVNU5ibFpRU1VOeFkyVTJSVWRMZDNoeWRIUkhWMmhvVlU5d2FYcG1aRzU2Q21sMlFXMDJTWEZ0VmsxMVJWcFJURlkxVVdKdEwwMTRZMGhsU1hSc1pIbzNXWE5HUzFsMGJFMHJObWhyV1hSc1RFaDRPRkZzWXpSRlNuSlhZVzg1ZEhBS1pGWnRaRVpNVUZSNlVXRmFNalZWVEZWbFIwVklOMUYwTnpZNU1HRTVTVFZ3VEhjMFV6ZE5VSFJpZUhvdlpHbERNR05DUkVwQ2FXSnVXVE51YVVGSVFRcHZiblo2V0hJM1VrOHJRVWRXV2pSc09GaDRia053UlhwUVltVk5WRTVKU0c5UU1uTkdlazVxZVRkVVNuRktkakJYZW1Nd1YxbzVSa2Q2VVZWSlJHZE1DakpVUjA5TVZIQmphM2hFWkUwNGVrZHVVM3BWWlhRelRtd3hNRlJ5UTI5eU4wMVhXVzEwV2pGWWJpOUpOVmhZVG14c2VVSnBTSEpCYjJSd1ZGZDJSWE1LUVV4bk1UaDZRVEo2ZVV3NVEwdFliSEJzTTBaalRsQlJaRUpDYmpGaFUyVmtlVlZoYmtSNWNIa3pNMjFvUW1aMFVsZFBLMUV4WjB0cFpsTktMelJLU2dveFYwSm5TRlJYZEhJd1dtRm9kV1IxWVV3MGQwcHJURFV5VVdaSmJWZFNOVFZ6TUZOU2JtRm5NVFZuTkhGWEwxRldlRFZtUVRWbWVVdFVhVWxGZEc5c0NrNVVPVkJCWjAxQ1FVRkhhbFo2UWxaTlFrMUhRVEZWWkVwUlVVMU5RVzlIUTBOelIwRlJWVVpDZDAxRFRVSXdSMEV4VldSRVoxRlhRa0pTZURCUWNXc0thalkwVkdReVQyZENUa2hDZFRBclRUSlRiRlEzYWtGbVFtZE9Wa2hUVFVWSFJFRlhaMEpTVkRJd2VuaDZjVVpJYW1GMGRDOHdOelZhYmxOWk1YRkRkUXBSUkVGT1FtZHJjV2hyYVVjNWR6QkNRVkZ6UmtGQlQwTkJaMFZCVms5bmVHcFBWemd5YTJsSFozUkpVbEpOS3padWRYZERVMGxoU2psRWNWQklRalZqQ2tNdmRFVXZWSEI0TjJVd1VFSkJNMnBxYjBKa1RuaHZMemcwWVdoQ05YSndlSFI0Ums5d2IyeFNPV3hQWjA5MGNuZGphM1l5T1VoRE0yTlRRbTVvY0ZZS09UTkZZbFJtYjA1dWJGa3hUamhFZHpkcVJVdDZNUzlhZEhkWlMwUmpTM1p0Wkd4d1VrbGFka1ZhVEZBeWFra3djbU1yVjJKWVoxSXJSazloVjJwTVZBcHNjekpsTW5CUlRISjNRMGMzUkZWRFJHWnhWV0psUlhsU2RtcFNWVzAwVURoc1FtZHNTMUJ2VDBWS2JYSkdUVWhNTHl0RGVqRlROMkowV1ZCTFNYTXhDa3RsWVhCRVpubzBSM3B0VjJONVpISnVRVVZVYTBSMk5tUlhabWhxVkRoRU9YRTVWVUk0UnpkNlZEWmpiMGxKZDJWc2JYUnFjR014V2pSRlJtZGlZelFLZDNGRFFrSXhVMDF3TW1oaU5FTnZWRWxGYVV4VEsxRk9RVEl3UlRaNE5XZENWbTVZYVRkSmFEVXZWMFJxYzFObE9XZFViVkpQUWpGdU5taDRXRUkxYndwa04yRXJNRWx5SzBkcmRrY3JRVFUzSzNwelFteG5SVEJPTDA5bk0wWnZiRFU1UVVSS01GQnFWVFpvZEZOeGRHWjFSMHhWY1RaTWVWSmlRME5uYURCdkNsbExOMDFNTVZGVFlTdE9ZbnBOYVZST05EaFlPRFkxUVdseVFscGhiRnBUTXpkbldGZDVjblJ2VFhwbmFYaDJhRnBuWmpaYVdHbHVkblJWYURGR1RtY0tZbTlRWkZjeVVWYzFRV2wyV2xkVWJIZE5kVFZtWm1oVU0wMTFRazAxZEVWTVFUTm5ibFJzVWtoUGIyUXJkVVpNWVRCT1JVVXJORGh2TUVGYVRUUkNkUW92Tm1WT1F6SnFOVkpEU25SdmMzUTBSR0Z2ZUVOS05qUXJjV2xxWjBSTVdFdFFiV05aTDB4clJFbHVPR1Y1YTJwaFNWWkpkbVpFYXk5Tk1GQjZSRGhOQ2tSalIxWXZaV00zVFhONFExbGFjVUZGSzNVNVExUXdWRnBvY2xGalVtbHFjbGhKT1U5WE9XTnRWM0oyVlc1a0wxazRiRVZPWmtwNlpIWjVaWGRTYW5jS2RqUlljVk5SZHowS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvPQ==
                       key: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVWlJWRU5EUVhsdFowRjNTVUpCWjBsVlFqTm9VM2h6VjJKclQwZEZNRXB2TjFWRFJrUlFWM0ZGVVRWWmQwUlJXVXBMYjFwSmFIWmpUa0ZSUlV3S1FsRkJkMUZVUlV4TlFXdEhRVEZWUlVKb1RVTlVhMFY0UTNwQlNrSm5UbFpDUVdkTlFXczFRazFSYzNkRFVWbEVWbEZSU0VSQlNrOVJWRVZNVFVGclJ3cEJNVlZGUTJkM1ExUnJSWGhEZWtGS1FtZE9Wa0pCVFUxQmF6VkNUVUkwV0VSVVNYbE5SRlY0VG5wQk1rMVVaM2ROVm05WVJGUkplazFFVlhoT2VrRXlDazFVWjNkTlZtOTNSM3BGV2sxQ1kwZEJNVlZGUVhkM1VWcEhWakpSUjFKc1pHMUtkbVZETlhOaU1rNW9Za1JEUTBGcFNYZEVVVmxLUzI5YVNXaDJZMDRLUVZGRlFrSlJRVVJuWjBsUVFVUkRRMEZuYjBOblowbENRVTV0UW5GcGFYbHFObFpUYldSYVVIRnFSRXRFVlRWTVVITTVXV040TkhvelRXa3ZUbWh0Y1FwWVl6ZE1RekZFTUdkelRXdzJlVm81Vm5aMllWQkxUSEZ3VjNCdVJGbEdkM1pZVWxwSWVrdFRibFJTYldNMlJUSXJRbU52WW5aeFVqZFBVbXQyTjJsdkNtSk5lSFJ1VFdsNU0zVkxkWGQ0TURoM2RHbHlVVEJDY3pKdldrVkpNQ3RJZURnd1FrMTFlRU5NYmxkQ0wyODFkbk40UjJjemEzcHVkM1p2U1ZSWk0wTUtXbkJhTlVaSlZGY3JUVmt5VVZneU1YUmtkemt5VUZKQ2JGTnZOVlZ6Tm1WWGFsRXhURTFsTjNJdlVYWmxiblZ4YTJJdllUQnhSMWhJVDFwNk0xVlRkQXBQVkU5clVuQlRka05GWVM5d2VsRnZTVTFvZWxSMlVWcENXVWRQVEdVMlRVNU5ibFpRU1VOeFkyVTJSVWRMZDNoeWRIUkhWMmhvVlU5d2FYcG1aRzU2Q21sMlFXMDJTWEZ0VmsxMVJWcFJURlkxVVdKdEwwMTRZMGhsU1hSc1pIbzNXWE5HUzFsMGJFMHJObWhyV1hSc1RFaDRPRkZzWXpSRlNuSlhZVzg1ZEhBS1pGWnRaRVpNVUZSNlVXRmFNalZWVEZWbFIwVklOMUYwTnpZNU1HRTVTVFZ3VEhjMFV6ZE5VSFJpZUhvdlpHbERNR05DUkVwQ2FXSnVXVE51YVVGSVFRcHZiblo2V0hJM1VrOHJRVWRXV2pSc09GaDRia053UlhwUVltVk5WRTVKU0c5UU1uTkdlazVxZVRkVVNuRktkakJYZW1Nd1YxbzVSa2Q2VVZWSlJHZE1DakpVUjA5TVZIQmphM2hFWkUwNGVrZHVVM3BWWlhRelRtd3hNRlJ5UTI5eU4wMVhXVzEwV2pGWWJpOUpOVmhZVG14c2VVSnBTSEpCYjJSd1ZGZDJSWE1LUVV4bk1UaDZRVEo2ZVV3NVEwdFliSEJzTTBaalRsQlJaRUpDYmpGaFUyVmtlVlZoYmtSNWNIa3pNMjFvUW1aMFVsZFBLMUV4WjB0cFpsTktMelJLU2dveFYwSm5TRlJYZEhJd1dtRm9kV1IxWVV3MGQwcHJURFV5VVdaSmJWZFNOVFZ6TUZOU2JtRm5NVFZuTkhGWEwxRldlRFZtUVRWbWVVdFVhVWxGZEc5c0NrNVVPVkJCWjAxQ1FVRkhhbFo2UWxaTlFrMUhRVEZWWkVwUlVVMU5RVzlIUTBOelIwRlJWVVpDZDAxRFRVSXdSMEV4VldSRVoxRlhRa0pTZURCUWNXc0thalkwVkdReVQyZENUa2hDZFRBclRUSlRiRlEzYWtGbVFtZE9Wa2hUVFVWSFJFRlhaMEpTVkRJd2VuaDZjVVpJYW1GMGRDOHdOelZhYmxOWk1YRkRkUXBSUkVGT1FtZHJjV2hyYVVjNWR6QkNRVkZ6UmtGQlQwTkJaMFZCVms5bmVHcFBWemd5YTJsSFozUkpVbEpOS3padWRYZERVMGxoU2psRWNWQklRalZqQ2tNdmRFVXZWSEI0TjJVd1VFSkJNMnBxYjBKa1RuaHZMemcwWVdoQ05YSndlSFI0Ums5d2IyeFNPV3hQWjA5MGNuZGphM1l5T1VoRE0yTlRRbTVvY0ZZS09UTkZZbFJtYjA1dWJGa3hUamhFZHpkcVJVdDZNUzlhZEhkWlMwUmpTM1p0Wkd4d1VrbGFka1ZhVEZBeWFra3djbU1yVjJKWVoxSXJSazloVjJwTVZBcHNjekpsTW5CUlRISjNRMGMzUkZWRFJHWnhWV0psUlhsU2RtcFNWVzAwVURoc1FtZHNTMUJ2VDBWS2JYSkdUVWhNTHl0RGVqRlROMkowV1ZCTFNYTXhDa3RsWVhCRVpubzBSM3B0VjJONVpISnVRVVZVYTBSMk5tUlhabWhxVkRoRU9YRTVWVUk0UnpkNlZEWmpiMGxKZDJWc2JYUnFjR014V2pSRlJtZGlZelFLZDNGRFFrSXhVMDF3TW1oaU5FTnZWRWxGYVV4VEsxRk9RVEl3UlRaNE5XZENWbTVZYVRkSmFEVXZWMFJxYzFObE9XZFViVkpQUWpGdU5taDRXRUkxYndwa04yRXJNRWx5SzBkcmRrY3JRVFUzSzNwelFteG5SVEJPTDA5bk0wWnZiRFU1UVVSS01GQnFWVFpvZEZOeGRHWjFSMHhWY1RaTWVWSmlRME5uYURCdkNsbExOMDFNTVZGVFlTdE9ZbnBOYVZST05EaFlPRFkxUVdseVFscGhiRnBUTXpkbldGZDVjblJ2VFhwbmFYaDJhRnBuWmpaYVdHbHVkblJWYURGR1RtY0tZbTlRWkZjeVVWYzFRV2wyV2xkVWJIZE5kVFZtWm1oVU0wMTFRazAxZEVWTVFUTm5ibFJzVWtoUGIyUXJkVVpNWVRCT1JVVXJORGh2TUVGYVRUUkNkUW92Tm1WT1F6SnFOVkpEU25SdmMzUTBSR0Z2ZUVOS05qUXJjV2xxWjBSTVdFdFFiV05aTDB4clJFbHVPR1Y1YTJwaFNWWkpkbVpFYXk5Tk1GQjZSRGhOQ2tSalIxWXZaV00zVFhONFExbGFjVUZGSzNVNVExUXdWRnBvY2xGalVtbHFjbGhKT1U5WE9XTnRWM0oyVlc1a0wxazRiRVZPWmtwNlpIWjVaWGRTYW5jS2RqUlljVk5SZHowS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvPQ==

[frezbo]

@smira the code has json tags for the clientIdentiy struct as Crt and Key, i wonder if that's also an issue

[smira]

I think it should be all YAML, so it should be lowercase

[j-lgs]

Specifying the endpoint did the trick. On a second glance it is evident in the documentation for RegistriesConfig as registry.local is used as an endpoint and key for the RegistryConfig map. Once it was properly specified everything worked.

    registries:
        mirrors:
            ghcr.io:
                endpoints:
                    - https://10.0.1.8:5004
        config:
            "10.0.1.8:5004":
                tls:
                    ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZZekNDQTB1Z0F3SUJBZ0lVYXptTWNITVdvVUIrK211VEk2aXQ0ZHBIaVhjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNUa0V4Q3pBSkJnTlZCQWdNQWs1Qk1Rc3dDUVlEVlFRSERBSk9RVEVMTUFrRwpBMVVFQ2d3Q1RrRXhDekFKQmdOVkJBTU1BazVCTUI0WERUSXlNRFV4TnpBMk1UZ3dNRm9YRFRJek1EVXhOekEyCk1UZ3dNRm93UVRFTE1Ba0dBMVVFQmhNQ1RrRXhDekFKQmdOVkJBZ01BazVCTVFzd0NRWURWUVFIREFKT1FURUwKTUFrR0ExVUVDZ3dDVGtFeEN6QUpCZ05WQkFNTUFrNUJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUExTmhXQ29odkZPMjg0R1ljUkhidDVwYVJobStGSTY3Q1NrYjdqeHNzVHhobWpQd2d5RE9CCmZUUnVDVWtLMXJQclJ5WjdJem93VWZyWGl5cFpvUFFFOFRnRFFlSmlvdDVmUm9SeENlL295QWZ3ei9sUnNKeE0Kd2RocnZ2SjRVeUhtY2xpSHFJZCtyUS94aEhuMmFCQThZV3dYVkxTRlY3NEJyTjFWZkw4SjJBSStNQjBRenBzWApDd1hlSXdhazZDRWIxZkk4QmVLMUhqc05LNFc2QjgwOXlJNXU4SS9DNVVOTnpnNU8yMDNzVng2cXROL2J2MmdLCmhwM216R3d3VDZQcENIcGttTURHbjU4ZWExOTBHWW9Rdlh4aFd5NE1WalVTTG50MnR1WFZ0U1lrbnQ2cjgxKzcKcVZTWXFmU05oanlsNTRDbEExeTRaM21lYisvTVVkMFdIRzQ1MTdrajlHbUVZaVZmSFNVSTdtMVI5YkVXNThuMwpNcmRvQlV6SnZETWZuVnhyNU5tY3JmRjdCU3MrOTBQbGRCZ2d1WlRrb0lLZUJ3bEg3aTZnY05jN29WQlUyNzJHCmN1SDJEUy92dmxZWkRhaFlaUEJ6N0lxWHl6cXQ3OEdmZitFM1J0bnhzN245YzBObkl3MDh5Yk5jcDl4bmt2c2oKUk0xalhjL2xmR25hNy8xSGU4WlZQaVBYOThaRDB5R3pnL1phQmRmUVEwMk1ORXA5RW41aVM2bG9kdlBYcTBWcwp0OVhGM3g2YTBNZkQwdTVxWFVKKzRhNTFTV3FRYXQ5b0tDakhDWXZ0YzZXYU5CN1RVQ0RFbVhUVWoxZHF0eitaCmFuSmlPZE1aamovUEMyK2xGNEZmYVpaTmxDM2gyL3R2TGFoSUd4QTd6SktTZGlyV250c1lFcWtDQXdFQUFhTlQKTUZFd0hRWURWUjBPQkJZRUZGUGJUUEhPb1VlTnEyMy9UdmxtZEpqV29LNUFNQjhHQTFVZEl3UVlNQmFBRkZQYgpUUEhPb1VlTnEyMy9UdmxtZEpqV29LNUFNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMCkJRQURnZ0lCQUIrVFVsK3ptbDNCdEJ2aitLWnUzeDgvOFFyNWYyc0VnVXBJeldPZ0lLRW4wQi9IVHd4VzFnNVoKTXZUb0FMNkxoTURFNW9KZTZ1R1VETTFvOHdxS044K3F2cVp4RFBwUGRaQlZtQ2dKV2lPdGhSOFppWXgzam8wYwo4dXpxUkpGOS9ZWERlWDMvakxZMSs5eCtQaktaWGVtYmZ6MW91WTlDT0VKSWVVb3ZFbW5YcnpkSnpQYlF4RmI2ClJMckdVQmpQM2RDM3hUbzRuUXVGd3o2MXkvb1VtV2FoMExJeUpMUWMySjBPM2tOWDVFMnJCL0ZFR1ZQWm9pcSsKMUkwODNRblBHTFdteUhxZTZ0RS90VS8zVkhsTnIyWjFTZVRPTWZIM3lJTU5WK210OW1xUWlVcWw0di9rQ0NEKwpRRHR5aVlFVEkyUmprYTc2VWNSbHRrekRXWVlSL1k0QnE1VEVlTGpDc0o4UExXVVMwQUFvaTd5L1p4U1pDaVhBCmJZdWhUb0U2SDUrNk1BTk5Md1FObGw5NEFndXFPYW9WeTRLbXd1Y3k3MmNGdUoyRU9DSXpIbGZ3TW5JdjkzK3IKV2ZSTWcwL1JPd2dzM202eGFlSno1UzUvaWdhelVudnNKd3h4RlFLYk9sbldVQ3dBdHUxUGtZbEJxY0M4NUttUgpLMEphcGlNTWN6bUcwMW1lTjJjMUVpOUtUQ1FTLy96eHhyV2hhTEs3bEZBSnBGNHhsU1craVYwd2dKcC9qeFQ1CkVVemhhZWtNU3l2SjFBUVJkaXYwOU13cjEvZmJ5Y0pHQjhSYTlnU0FlOGZVeUFSNnZhUmFNY1JSUUhNWXVlZFoKLzF6bHR6VWFZNnhFaVZ4cXU1YVgxbEJMbjN0dmVZRllrR3B5TWdGaUQzbld0SnAzVG9YeAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
                    clientIdentity:
                      crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZRVENDQXltZ0F3SUJBZ0lVUkFjclZnZUlmOEMraTJ1T2l3ZS9OTncwSTdBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNUa0V4Q3pBSkJnTlZCQWdNQWs1Qk1Rc3dDUVlEVlFRSERBSk9RVEVMTUFrRwpBMVVFQ2d3Q1RrRXhDekFKQmdOVkJBTU1BazVCTUI0WERUSXlNRFV4TnpFd01EWXhOVm9YRFRJek1EVXhOekV3Ck1EWXhOVm93R3pFWk1CY0dBMVVFQXd3UVpHVjJRR1JsZG1KdmVDNXNiMk5oYkRDQ0FpSXdEUVlKS29aSWh2Y04KQVFFQkJRQURnZ0lQQURDQ0Fnb0NnZ0lCQU1zUzU5OS9TeTlXNXpWYmFudXd6KzgvNllRT3JiTVBVM2tYM0hZZQpSUE11YnpzUmM2aGFMUXUrS2hhNUpjSHJ6QisrREMrRzdlVStlRkYzSWtTdmYrNmFCd0FXZXBBN2h3eDd6V2VCCkVHL0dMcXhLU29Md2Y0Y2hVNnRvU0VVWDNNcndxR2kreERaeDFBenBXQ0FseVNDWDdTWWxZNEZocVNrdnFNUmIKK2RJdEVna3hGMjA1R05NM2pPS0UrVUtWTzdXVzBEMWxrdElEZHZSWHdLMGNFSVZtOGgwd0Y4WFVSdm1sazdDUQp3U1NHVlRsWlN2T1l1MEVNdlpzY3d4d3BFZ2tVZWtRdVBYYkN6NnFneVlJNkVZQWZ4ZzIrbkdHZ0owdEJ4Uk5CCkczYXhGbk9pZ2tCdTVhZnZMUHFLbmQra2hlNEllTEJPMWRzcjBzU3JHaDJLVFBpMHh1bEtpb2pRWldRcGxWNmEKVHpIUXU4d2ZUeU96QXQxQitNNGhWMVBpcnY0U0pkQ2lkNTBIb3VwYnduak9sNXJVRGw1ajRtNHBQTzdDTndNegovc044aHgxcDBYa0JmY1FXMkZRRTF6Q1l6ZkVpaEM0YXJONzNkWXVNZ2pqVEhvRWQ1eVZ0K0ROcmYzVVNMSUI4CmlITlUwSDlkaHI1ajF6ODVueThrbFpvOW12dlFiVmVKNzZZV3dPL3N1NWpJeWExQTkzKzkzby9sVzlMQ3EzSjUKQXRRTWc4SU02ZkRJcDZrd1Q1d3RTSGxLMldsNnoySmtia0UxNVhXUWNEQ3NHQUdUVGtGYkhDd2tjdjdQTlZ3bgpjRGxyRGFvdnp4RjhlUkVvUmFoSWUvY3VLKzR3VlNIOEZIMVFYc3Jocndhak9qYTFrSWF1VkxoQXdadDlVM05SCnZ5ZUZBZ01CQUFHalZ6QlZNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01DTUIwR0ExVWREZ1FXQkJRNTdGekUKQ1NDWGJEcnNzNTR0Mjk2TjNGc3VXREFmQmdOVkhTTUVHREFXZ0JSVDIwenh6cUZIamF0dC8wNzVablNZMXFDdQpRREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBZ0VBTm9VQ2hYelJsVzFENXRUMnlZbStwMXhaZ3Y0aHNDUUgvdTBqCloveS9sZFBGWUhOUkorZzc3NFZZQmRMOEgydzkrK0w2anJoYjJVV0MyOEY0S2NvNjVKUnNUd3JScXlwOG0zM20KdXlHbmczYnpvY1NuYklleEV2NDZnTENEc1FYcEQwdVV4cFZ0RXRxbHAvRjVSaTZtenZMNEZlcm9TZ05WbkRETwpIV1c5bEdyZnd1Z0JsUDFFSHpuM0M2QVg4TFhJVmtESzBaTDh1cHlZZmErbHlDMTF5L3I3ODlkL2kyUWlnQXZNCi9uUHdVT1dFTFRPaGdhQU1DODBlNm1Fa25pb2Y0QWZHS0E3cG45TzkyeWE3SDZ2Y25kN1pWRUwrUll5NWVwZVQKVWFxM2VYMU53WnlHdDcva3FKRWtGeEdtZXNDa29lOHhJRTM2RHJndjVRV1F4ekVmM093T0ZFQUd5Y2FmL3JTbworeXM0YUEyTmw1Vmp6Wlozblo0SUhlYXFrbW9TTVNYV3JPdzRacGx5M3R0ekF1NjFYakxhUWJ5OGQ0QUNWMWRrCmM4ZDA2TWsyb1EzZkZ6clJPVzUzWXoySmx6Mk1Zb3NlREtOUmFwamhEUVRqYUJCcDZaMFRmR3poV1pBTXdhY04KU1FpMWNCRTFuV0VLRmczdkhsNjJCSzNjQjZSY3JpZUV2WmQ3QnRWQTZrTDZsVWFMckQrbnRoQ0V1NEt6SUpMRQpYVlR0d1BvdkNDYU83MnpaTVRlSzJvNGpLMWt3S25IMkU4TS9tR3NIWFR0YmcrRktPTUp6SHF6aXNWblc5V0FCCmwrUS9YUjhNckx1Q2wrbGFWR3czRGg5Yi82UEZ1cmhRUTNwQVNGbkM4UzNVblZIK0ltTTdKemUweWIwSTVCRDYKdDN2SUM3dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
                      key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRd0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1Mwd2dna3BBZ0VBQW9JQ0FRRExFdWZmZjBzdlZ1YzEKVzJwN3NNL3ZQK21FRHEyekQxTjVGOXgySGtUekxtODdFWE9vV2kwTHZpb1d1U1hCNjh3ZnZnd3ZodTNsUG5oUgpkeUpFcjMvdW1nY0FGbnFRTzRjTWU4MW5nUkJ2eGk2c1NrcUM4SCtISVZPcmFFaEZGOXpLOEtob3ZzUTJjZFFNCjZWZ2dKY2tnbCswbUpXT0JZYWtwTDZqRVcvblNMUklKTVJkdE9SalRONHppaFBsQ2xUdTFsdEE5WlpMU0EzYjAKVjhDdEhCQ0ZadklkTUJmRjFFYjVwWk93a01Fa2hsVTVXVXJ6bUx0QkRMMmJITU1jS1JJSkZIcEVMajEyd3MrcQpvTW1DT2hHQUg4WU52cHhob0NkTFFjVVRRUnQyc1Jaem9vSkFidVduN3l6NmlwM2ZwSVh1Q0hpd1R0WGJLOUxFCnF4b2Rpa3o0dE1icFNvcUkwR1ZrS1pWZW1rOHgwTHZNSDA4anN3TGRRZmpPSVZkVDRxNytFaVhRb25lZEI2THEKVzhKNHpwZWExQTVlWStKdUtUenV3amNETS83RGZJY2RhZEY1QVgzRUZ0aFVCTmN3bU0zeElvUXVHcXplOTNXTApqSUk0MHg2QkhlY2xiZmd6YTM5MUVpeUFmSWh6Vk5CL1hZYStZOWMvT1o4dkpKV2FQWnI3MEcxWGllK21Gc0R2CjdMdVl5TW10UVBkL3ZkNlA1VnZTd3F0eWVRTFVESVBDRE9ud3lLZXBNRStjTFVoNVN0bHBlczlpWkc1Qk5lVjEKa0hBd3JCZ0JrMDVCV3h3c0pITCt6elZjSjNBNWF3MnFMODhSZkhrUktFV29TSHYzTGl2dU1GVWgvQlI5VUY3Swo0YThHb3pvMnRaQ0dybFM0UU1HYmZWTnpVYjhuaFFJREFRQUJBb0lDQUFJNEtXMFZvdEI0WW13QW5RUVBqV0kzCndGbFhZTG9iZ1lYS0J6aUtRYVludklMNGdKdFhLdlZkSlhBY28yZEFhTGx6RmdhQjhzRWw0dUkvQzFLVldYQlYKNGh3ejRyQ0hXZ1UrcUNMSXRCR0lxTE5zMTI0aGF6OWlrNXZFdHI5OGtJYnVST1RxS3RKT2dhc0hkTlZiSEZ6ZApGRFF0eGUzZ3o1eTUzdDRRaEVVeUJXZXd6K2o3YmdSRzcrVDgzNDJFL0lOU1MrSFQ1WndpZWc4WEUyM0E2QVJSCk9QV3ROQ0ZQSGR4MkhReFp0S2F0eWlvbmlleHFINmZKR0J4bEh0UEV0aVAvdzR0UnR3M0FlTTBNVjA5elM0UXIKTkxxck9lSE9Rb2V0eHhYY0NRME54K2xKSGlQT29uZG9vNFQwZFYrL21GQWU2eFRBTTlHaW83blBCZWpMUFdhWQpkVGdiOUFJZmM5VXk5aHQzL1NrV3h2Y21WRUxJVVdsQ3kvUmsvNFBzZG40bDk2amlxKzBuVUFVQUxTQXliV1VDCllqQ0ZzNjVuWlRqVDd3V2pFT3EwYUk1TERQTG4rMFhGdGpuRTNaWEtCNi9McXlXWWlOYlkwN0FnU0xYWHdRbzIKK1hlemN6VktKL1ZhNGw2V1F6YkxEdTFubHhRM3hPYTAvWXRtUVR2NDFndSt5Qi9aaHdjYS8zQVdHc0RNV2ZibApEaWZJbjVWMTR2VjJCTWlLVXJuOHBua3lTM3EwdjVMdGdhTkQ3d1grbnk3WlZOVHl4aS9LNGVISllmVDJydlZWCmo2cHdlL2VBbEFaa2ttQUQ0L2R2emNaRkZHYjlZV1NQUi95MUo0TUE0OCtIaDJvL1NqdHBudWtoeGRGSHNoTlQKQkZFTGVWdy9qeHZZLzVZK3hBVnhBb0lCQVFEZzhoVVRpUXcxdWk2NVRWaVM5TFpYTzh2MHdIQTlyTk1pU2dJVApzejIxZHJpdFczc2Q3eW04bDBSYmVTbHIxRWNOMFhDZ3k3amJlMVI1elJtS09acmpYa2NTK1g5VG1YbW5hZlN1ClFJSkg1ZWRpQUJ3ZEg5K2hUMmkyeUZjQlpIMVdVZFVnWm1QQjJYVmo5TTNnTDRWSm1rZHc3VzdyVG85VDk3RHgKTnpCaWFTM2pvanZpeE1OTVNXd0V0VlUvK3lXYXhDTjU2VHJZZ1NsL051clFpUWs5Tmw3eTJiWFB6dW13ejdQaQpZSWpHOHppSkdMa0V0TjRjWWlUeGI5R0VvQmhLczlxbVYrbFllZjRvdHdCRndrVmZMOHdIcWI3dzZ0K3V2YnhiClh6dVRhS3h5T1lUQlRGOVRxNngrUHgzdlNhd0IrTFIwU3RFRzdkOUY3TjM2dVh3VkFvSUJBUURuRzlkMW1lWGcKcGlhWENudkRaZVJQTjByWnVVaDVHKzVaZ2Q5MXhWcTRHekR5SU1yUGtZbWl1c0tkSmJQNno2NUh1d3FTck4vOAoySHltcFpOK2dPSnRvK1U1TGs2WE9LUExqbDQ3dnI3bEVIMFNIZTdtRXVSNGliR0xMSHB5OXJLTkszU2NPOXNtCkI3K1VsUDVzNGN6NWg3a0k3aFkvd0YwRDJ3QWFxTU80c29qcWJXcnlBRGRtZHNVMWpWQVlrc0RVTXowL3psZ0MKYmJRVldiUFp0RFZkTmdmME5DYUVsZ2JkQjdmV1NPb2VnZ2tXU0w3SDg4ME1sV1M0ZUVBdi9DWTNXNVcrQU8vegpBc0NqWEJycVduZ0Q3WkZkUUFPNUZDZ2VwQko2Q08zYlY4cmVrRFpQUUZESy9oWURUMTE3OTFOYnRSdDdrYnRLCjJ3R2JPNXN6WFNteEFvSUJBUUNCZlJtQTIxMTUrQ2s1WmpyY3JaM2hiWHlrOGJvcTVyZkxmMHJleUVsM2tsWVIKQlI1RVJ3NkJqNW84QmUxQU16eUx5Y3JKNVR3T3JGb2VtMkJlWDNhMzkrZDJGc1dpL0RBNW5SYkswRHV1NDd0awpnS2V1WkZDNWJScmNUVXM0Yy81cW1FVFZsOXRKRmNNcnZScGVPVHpDOTlxME9SbW12dHhtKzZpUjByRTVsVTVqClFWcXoyWXlLOTZFRXZOWE1OT2lheE5tZkxoRS8zR3BrM3RvaEhTR0NhVEFUSUJ3MitlY3did0FnQ20zeUNNUzgKdHN5YTF2c2MzNnBnaVh2VW1zd2hCSk9UT2Z3TGdINDhOY3drWEFiUEdwYWF1cTQ5eFJicVlqSnhic3ErdW55NgpiRDN0ZitkSUpRVHd6NmlOdkowVnJYTkRsbVhwVlRrMzZYaUhBZ3RaQW9JQkFDNVRmV2djZG1FR3pUdC9wWGx3CnBraGV6QWxDdzgxTEZRY3FLbklBSW9RL0p4dVNDVFY4ZGZ1QnRsc3I0SjBKSG1WUWova0E0RkZkei9iVHYvSWUKd2NTNmtzdFloZlo0S3hVMXI1d0lTMW05YVdURlRVc2ZSRjhKTXVhei9zT213Zm5jV1I2c1BPeXRwdkNRd1VOdwpLOWRsSm1rczJKYms1NEJMV1FUL0NXU3ZuUlJvNUlRb0dQTFlJZ1h2RXJ3cVJxOENxbWRzNUdWd09pWUJlalRNCms5bmdaODNDMm1tL2JSYkhZa2w1Q2owZFVkYit0QWQrem42VVA4RmVrWGRNOHhncnFxT1Fxb2lxNTVTWGRPbS8KUXNsK3lLSEpvRFZzWGZKdWJ5bTVPbnk2WjNjZnIxTkprQ2h3c0ZidmhNekJzaHJ5L2RSU1RSbmZOcjQ0UDQ4WApRTEVDZ2dFQkFNblhVRnRKNEdHQkdqL0ZjV3VpeE5DNXpSYk1qTEtMNHBETDdZakFJNmoyQXQzNm9QRFhXQnlsCmU3Mi84b3VIb1N4MUhWTlg1Qk03N2xhakRxbndGMVd6MFVuRHpXNFlnMndQNFVVR1ZVSFYrYjF1WFlLUWEvTUQKWFJHb0VTU3h0MysyL2RxOWN3TnNKcGFnVmZYcmRYSDk3MEs3VS9vUGN1YXBkclhOSElySlREaEQvTGZXQXJ4YgpXQlM3NzYvdXRmQTBueW50d1BlTGNMTURyVytnWHFCQ1k0dnhrWGcvQ05TTW94OVN0Znd0K1dWb3lOVlFwOFM0Cmk2QW9mMHduVUxWTkNaWm8weGlJWXd0dHh5SlZMS2FhbWdiNXNIVzBOeVl4QnorSXhGL2hnZUl5WFdjUzZzYjAKV2k3eXpXTjE4RldYVXgvUWIvMXg1TTBBS2tYWTdRbz0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=

However doubly base64 encoding the key and crt fields of clientIdentity is most surely an issue in the documentation. It causes the following error.

[   21.025331] [talos] pulling "ghcr.io/siderolabs/installer:v1.0.4"
[   21.026327] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.0.4": failed to resolve reference "ghcr.io/siderolabs/installer:v1.0.4": error preparing TLS config for "10.0.1.8:5004": error parsing client identity: tls: failed to find any PEM data in certificate input

The relevant code is here and most likely stems from being passed base64 encoded PEMs rather than the raw PEM.

For the sake of completion I tried empty string fields for crt and key and got the same error which makes sense considering that they are also invalid inputs for the tls.X509KeyPair function.

[smira]

Double encoding will be fixed via #5595, thanks for reporting it!