Feature Request: Automatically upgrade too small volumes to the minimum size

[jpeimer]

/kind bug

What happened?
Trying to create a PVC with a requested size smaller than 4Gi, but it stays Pending

$ oc describe pvc simple-pvc
Name:          simple-pvc
Namespace:     default
StorageClass:  io2-csi
Status:        Pending
....
Events:
  Type     Reason                Age                         From                                                                                                 Message
  ----     ------                ----                        ----                                                                                                 -------
  Warning  ProvisioningFailed    2m51s                       ebs.csi.aws.com_aws-ebs-csi-driver-controller-7cd74b9994-x9kqp_470affa4-ba6e-4219-b2b9-76f83c07fb2b  failed to provision volume with StorageClass "io2-csi": rpc error: code = Internal desc = Could not create volume "pvc-7f726ea6-6b67-43cc-9f82-766d5b25c35c": could not create volume in EC2: operation error EC2: CreateVolume, https response error StatusCode: 400, RequestID: 48d34e2f-476a-41c4-a89d-c501aff23c92, api error InvalidParameterValue: The volume size is invalid for io2 volumes: 1 GiB. io2 volumes must be at least 4 GiB in size. Please specify a volume size above the minimum limit.

What you expected to happen?
Volume provisioned anyway, with the minimum possible size (4Gi in this case)

How to reproduce it (as minimally and precisely as possible)?
Create a PVC and a first consumer pod

$ cat simple-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: simple-pvc
spec:
  storageClassName: io2-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi


$  cat first-consumer.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    -
      command:
        - sleep
        - "3600000"
      image: "quay.io/fedora/fedora:40"
      imagePullPolicy: IfNotPresent
      name: test-pod
      volumeMounts:
      - mountPath: /foo
        name: data-vol
  volumes:
    -
      name: data-vol
      persistentVolumeClaim:
        claimName: simple-pvc

Anything else we need to know?:
Storage class:

$ oc get sc io2-csi -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true","storageclass.kubevirt.io/is-default-virt-class":"true"},"creationTimestamp":"2024-09-02T11:28:15Z","name":"io2-csi","resourceVersion":"136187","uid":"4357b1d7-e3c0-4ec1-a372-9807d3f946a8"},"parameters":{"iops":"4000","type":"io2"},"provisioner":"ebs.csi.aws.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
    storageclass.kubevirt.io/is-default-virt-class: "true"
  creationTimestamp: "2024-09-02T15:44:38Z"
  name: io2-csi
  resourceVersion: "207551"
  uid: 41747b86-46f4-4f7a-957f-33dff7886e77
parameters:
  iops: "4000"
  type: io2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Environment

• Kubernetes version (use kubectl version):
• Driver version:

[AndrewSirenko]

Hello @jpeimer, thank you for your issue.

Indeed the minimum size for io2 (and io1) volumes is 4Gi. Therefore this error is not exclusive to the EBS CSI Driver. See the official EBS documentation, specifically the Volume size row.

You can observe a similar error with the following AWS EC2 CLI command:

❯ aws ec2 create-volume --availability-zone us-west-2a --volume-type io2 --size 1

An error occurred (InvalidParameterCombination) when calling the CreateVolume operation: The parameter iops must be specified for io2 volumes.

Please increase the size of your volume to 4Gi, or switch to a different volume type likegp3.

[ConnorJC3]

/close

As @AndrewSirenko stated above, this is an EBS limitation, and thus not controllable by the EBS CSI Driver. Volumes of type gp3 or gp2 can be created as small as 1GiB, which may be a workaround that suits your usecase.

Please reach out to AWS Support (or if applicable, your assigned account manager/service architect/etc) if you wish to request a change to the minimum size of io2 volumes.

I'm going to close this issue as there is nothing that can be done in the driver, but please re-open it or open a new issue if you need further assistance.

[k8s-ci-robot]

@ConnorJC3: Closing this issue.

In response to this:

/close

As @AndrewSirenko stated above, this is an EBS limitation, and thus not controllable by the EBS CSI Driver. Volumes of type gp3 or gp2 can be created as small as 1GiB, which may be a workaround that suits your usecase.

Please reach out to AWS Support (or if applicable, your assigned account manager/service architect/etc) if you wish to request a change to the minimum size of io2 volumes.

I'm going to close this issue as there is nothing that can be done in the driver, but please re-open it or open a new issue if you need further assistance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jpeimer]

Thank you for your answer @AndrewSirenko and @ConnorJC3

I understand it's a documented behavior, however, according to the Kubernetes documentation, the storage is free to return a larger volume to the user: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding
If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC. Otherwise, the user will always get at least what they asked for, but the volume may be in excess of what was requested.

So currently, if the users have some automation flows that request a smaller size - they can't use this storage driver.

It looks reasonable to adjust the behavior to the kubernetes expectations. What do you think?

/reopen

[k8s-ci-robot]

@jpeimer: Reopened this issue.

In response to this:

Thank you for your answer @AndrewSirenko and @ConnorJC3

I understand it's a documented behavior, however, according to the Kubernetes documentation, the storage is free to return a larger volume to the user: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding
If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC. Otherwise, the user will always get at least what they asked for, but the volume may be in excess of what was requested.

So currently, if the users have some automation flows that request a smaller size - they can't use this storage driver.

It looks reasonable to adjust the behavior to the kubernetes expectations. What do you think?

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ConnorJC3]

That section of the Kubernetes docs refers to the KCM and does not affect CSI drivers. However, reviewing the CSI spec (which is the binding contract between COs like Kubernetes and CSI drivers like the EBS CSI Driver), it does appear that over-provisioning a volume is allowed as long as LimitBytes is unset (or the value chosen is within the limit).

I'll leave this open as a feature request for automatically increasing the size of too small volumes - we will need to evaluate if this is suitable, as it may result in users beings surprised and potentially paying more than they expect for volumes. For the moment, I would suggest increasing your volume size to the minimum EBS size as a workaround.

/retitle Feature Request: Automatically upgrade too small volumes to the minimum size
/remove-kind bug
/kind feature