Error when trying to write the Kubernetes cluster configuration to Hetzner storage

[SergiiKishkar]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.30.1 (git-v1.30.1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

kubernetesVersion: 1.29.6

3. What cloud provider are you using?

Hetzner

4. What commands did you run? What is the simplest way to reproduce this issue?

kops replace --force -f kops-manifest.yaml --v=10

5. What happened after the commands executed?

I1105 16:45:10.411123 40 s3fs.go:323] Calling S3 PutObject Bucket="new" Key="int-k8s-stage.k8s.local/config" SSE="AES256" ACL=""
Error: error creating cluster: error writing Cluster "int-k8s-stage.k8s.local": error writing configuration file s3://new/int-k8s-stage.k8s.local/config: error writing s3://new/int-k8s-stage.k8s.local/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: tx00000f7a6d034f2c71e7b-00672a4b96-57491f7-fsn1-prod1-ceph3, HostID: 57491f7-fsn1-prod1-ceph3-fsn1-prod1, api error InvalidArgument: UnknownError

6. What did you expect to happen?

I am waiting for the configuration file to be correctly written to the Hetzner storage

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null

  # Clustername muss für Hetzner in diesem Format sein
  name: int-k8s-stage.k8s.local
spec:
  api:
    loadBalancer:
      # macht keinen Unterschied
      type: Public
  authorization:
    rbac: {}
  # Verwaltung vom Cert-Manager durch kOps, Issuer muss selbstständig installiert werden
  certManager:
    enabled: true
    defaultIssuer: letsencrypt
  channel: stable
  cloudProvider: hetzner
  # das Schema do (Digital Ocean) verwenden, 
  # um die AES256-Verschlüsselung beim Speichern auf dem Hetzner-Speicher zu deaktivieren
  configBase: do://int-k8s-stage-kops-state/int-k8s-stage.k8s.local
  # 1 etcd Member pro Control-Plane
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: control-plane-nbg1-1
      name: etcd-1
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: control-plane-nbg1-1
      name: etcd-1
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
 
  # muss deaktiviert werden für Cilium
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false

  # funktioniert bisher nicht
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.29.6
  metricsServer:
    enabled: true
  networkCIDR: 10.61.0.0/16

  # Cilium als CNI
  networking:
    cilium:
      enableNodePort: true
  nonMasqueradeCIDR: 100.64.0.0/10

  # sshAccess kann limitiert werden
  sshAccess:
  - 0.0.0.0/0 

  subnets:
  - name: nbg1
    type: Public
    zone: nbg1
  topology:
    dns:
      type: Private
    # auf private setzen macht keinen Unterschied
    masters: public
    nodes: public

---

# Control-Plane 1
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: int-k8s-stage.k8s.local
  name: control-plane-nbg1-1
spec:
  image: ubuntu-22.04
  machineType: cx22
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - nbg1
---


# Worker InstanceGroup, skalieren durch Anpassen der min- und maxSize
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: int-k8s-stage.k8s.local
    environment: staging
  name: nodes-nbg1
spec:
  image: ubuntu-22.04
  machineType: cx32
  maxSize: 3
  minSize: 3
  role: Node
  subnets:
  - nbg1

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
I1105 16:45:06.860431 40 factory.go:82] state store s3://new
I1105 16:45:06.866333 40 s3context.go:106] Found S3_ENDPOINT="https://new.fsn1.your-objectstorage.com/", using as non-AWS S3 backend
I1105 16:45:06.866910 40 s3fs.go:385] Reading file "s3://new/int-k8s-stage.k8s.local/config"
I1105 16:45:07.996227 40 defaults.go:262] Not setting up Proxy Excludes
I1105 16:45:08.001304 40 s3fs.go:385] Reading file "s3://new/int-k8s-stage.k8s.local/config"
I1105 16:45:09.192732 40 s3fs.go:303] Writing file "s3://new/int-k8s-stage.k8s.local/config"
I1105 16:45:09.192768 40 s3context.go:259] Checking default bucket encryption for "new"
I1105 16:45:09.192776 40 s3context.go:264] Calling S3 GetBucketEncryption Bucket="new"
I1105 16:45:10.411098 40 s3context.go:271] Unable to read bucket encryption policy for "new": will encrypt using AES256
I1105 16:45:10.411123 40 s3fs.go:323] Calling S3 PutObject Bucket="new" Key="int-k8s-stage.k8s.local/config" SSE="AES256" ACL=""
Error: error creating cluster: error writing Cluster "int-k8s-stage.k8s.local": error writing configuration file s3://new/int-k8s-stage.k8s.local/config: error writing s3://new/int-k8s-stage.k8s.local/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: tx00000f7a6d034f2c71e7b-00672a4b96-57491f7-fsn1-prod1-ceph3, HostID: 57491f7-fsn1-prod1-ceph3-fsn1-prod1, api error InvalidArgument: UnknownError

9. Anything else do we need to know?
Recently I have been working on a task to move kops kubernetes cluster backups from minio to hetzner storage.
Minio uses the AES256 key for encrypting its backups. So when the Minio buckets are accessed by Kops to save the Kubernetes cluster configuration file, the SSE=‘AES256’ flag is present on the command line and the save operation succeeds.
I have changed the pipeline variables to Hetzner access parameters:
S3_ENDPOINT=https://fsn1.your-objectstorage.com
S3_ACCESS_KEY_ID=***********
S3_SECRET_ACCESS_KEY=*************

After that I started getting the Kops error when trying to write the cluster configuration file to Hetzner storage
$ kops replace --force -f kops-manifest.yaml
27Error: error creating cluster: error writing Cluster "int-k8s-stage.k8s.local": error writing configuration file s3://int-k8s-stage-kops-state/int-k8s-stage.k8s.local/config: error writing s3://int-k8s-stage-kops-state/int-k8s-stage.k8s.local/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: tx0000082b990f3bed2d284-006729e67b-529ccd2-fsn1-prod1-ceph3, HostID: 529ccd2-fsn1-prod1-ceph3-fsn1-prod1, api error InvalidArgument: UnknownError

As you can see, Kops cannot write the config file to the Hetzner storage and gets error 400 and InvalidArgument.
I tried different options to write variables for the pipeline, but it was unsuccessful. Also the introduction of the SSE=‘’ and SSE=‘-’ variable did not change the situation. The default Kops variable SSE=‘AES256’ has a higher priority than the pipeline variables.

According to the Hetzner Storage documentation, Hetzner currently supports only one type of encryption: Encryption (nur SSE-C)
https://docs.hetzner.com/de/storage/object-storage/supported-actions/

According to the Kops documentation, Kops currently supports SSE-S3 (server-side encryption using Amazon S3 keys) and SSE-KMS (server-side encryption using AWS KMS keys). It is for this purpose that the AES256 key is used.
https://kops.sigs.k8s.io/getting_started/aws/

Unfortunately, I have to say that the Kops and Hezner encryption schemes are not the same at the moment.

Questions

1. Is it possible to ensure that kubernetes cluster data from kops is saved to Hetzner Storage while maintaining the existing encryption schemes?
2. If it is possible: what Kops settings should I change to avoid a file write error when using Kops?

The solution that helped me avoid the error but did not configure encryption:

Changing the KOPS_STATE_STORE variable
From: KOPS_STATE_STORE=s3://int-k8s-stage-kops-state
To: KOPS_STATE_STORE=do://int-k8s-stage-kops-state
In this case, do:// specifies the DigitalOcean scheme when saving the configuration file.
Reference: Kops Documentation: Digital Ocean State

DigitalOcean (do://)
DigitalOcean storage is configured as a flavor of an S3 store.
The difference is not described in the documentation, but in practice, when using this method, the Kops SSE variable is set to "-", meaning encryption is disabled.

Modifying kops-manifest.yaml
yaml

configBase: do://int-k8s-stage-kops-state/int-k8s-stage.k8s.local
These two changes allow using Hetzner storage for storing the configuration file and Kops backups.

Major Drawback:
The data remains unencrypted.

[safizn]

Getting the same error:

Error: error writing updated configuration: error writing Cluster "testing.k8s.local": error writing configuration file s3://bucket-name/testing.k8s.local/config: error writing s3://bucket-name/testing.k8s.local/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: tx00000<...>-fsn1-testing1-ceph3, HostID: <...>-fsn1-testing1-ceph3-fsn1-testing1, api error InvalidArgument: UnknownError

for now I'll rely on another S3-compatible object store.