Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request: Automatically upgrade too small volumes to the minimum size #2162

Open
jpeimer opened this issue Oct 1, 2024 · 6 comments · May be fixed by #2273
Open

Feature Request: Automatically upgrade too small volumes to the minimum size #2162

jpeimer opened this issue Oct 1, 2024 · 6 comments · May be fixed by #2273
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@jpeimer
Copy link

jpeimer commented Oct 1, 2024

/kind bug

What happened?
Trying to create a PVC with a requested size smaller than 4Gi, but it stays Pending

$ oc describe pvc simple-pvc
Name:          simple-pvc
Namespace:     default
StorageClass:  io2-csi
Status:        Pending
....
Events:
  Type     Reason                Age                         From                                                                                                 Message
  ----     ------                ----                        ----                                                                                                 -------
  Warning  ProvisioningFailed    2m51s                       ebs.csi.aws.com_aws-ebs-csi-driver-controller-7cd74b9994-x9kqp_470affa4-ba6e-4219-b2b9-76f83c07fb2b  failed to provision volume with StorageClass "io2-csi": rpc error: code = Internal desc = Could not create volume "pvc-7f726ea6-6b67-43cc-9f82-766d5b25c35c": could not create volume in EC2: operation error EC2: CreateVolume, https response error StatusCode: 400, RequestID: 48d34e2f-476a-41c4-a89d-c501aff23c92, api error InvalidParameterValue: The volume size is invalid for io2 volumes: 1 GiB. io2 volumes must be at least 4 GiB in size. Please specify a volume size above the minimum limit.

What you expected to happen?
Volume provisioned anyway, with the minimum possible size (4Gi in this case)

How to reproduce it (as minimally and precisely as possible)?
Create a PVC and a first consumer pod

$ cat simple-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: simple-pvc
spec:
  storageClassName: io2-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi


$  cat first-consumer.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    -
      command:
        - sleep
        - "3600000"
      image: "quay.io/fedora/fedora:40"
      imagePullPolicy: IfNotPresent
      name: test-pod
      volumeMounts:
      - mountPath: /foo
        name: data-vol
  volumes:
    -
      name: data-vol
      persistentVolumeClaim:
        claimName: simple-pvc

Anything else we need to know?:
Storage class:

$ oc get sc io2-csi -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true","storageclass.kubevirt.io/is-default-virt-class":"true"},"creationTimestamp":"2024-09-02T11:28:15Z","name":"io2-csi","resourceVersion":"136187","uid":"4357b1d7-e3c0-4ec1-a372-9807d3f946a8"},"parameters":{"iops":"4000","type":"io2"},"provisioner":"ebs.csi.aws.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
    storageclass.kubevirt.io/is-default-virt-class: "true"
  creationTimestamp: "2024-09-02T15:44:38Z"
  name: io2-csi
  resourceVersion: "207551"
  uid: 41747b86-46f4-4f7a-957f-33dff7886e77
parameters:
  iops: "4000"
  type: io2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Environment

  • Kubernetes version (use kubectl version):
  • Driver version:
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 1, 2024
@AndrewSirenko
Copy link
Contributor

AndrewSirenko commented Oct 1, 2024 •
edited
Loading

Hello @jpeimer, thank you for your issue.

Indeed the minimum size for io2 (and io1) volumes is 4Gi. Therefore this error is not exclusive to the EBS CSI Driver. See the official EBS documentation, specifically the Volume size row.

You can observe a similar error with the following AWS EC2 CLI command:

❯ aws ec2 create-volume --availability-zone us-west-2a --volume-type io2 --size 1

An error occurred (InvalidParameterCombination) when calling the CreateVolume operation: The parameter iops must be specified for io2 volumes.

Please increase the size of your volume to 4Gi, or switch to a different volume type likegp3.

@ConnorJC3
Copy link
Contributor

/close

As @AndrewSirenko stated above, this is an EBS limitation, and thus not controllable by the EBS CSI Driver. Volumes of type gp3 or gp2 can be created as small as 1GiB, which may be a workaround that suits your usecase.

Please reach out to AWS Support (or if applicable, your assigned account manager/service architect/etc) if you wish to request a change to the minimum size of io2 volumes.

I'm going to close this issue as there is nothing that can be done in the driver, but please re-open it or open a new issue if you need further assistance.

@k8s-ci-robot
Copy link
Contributor

@ConnorJC3: Closing this issue.

In response to this:

/close

As @AndrewSirenko stated above, this is an EBS limitation, and thus not controllable by the EBS CSI Driver. Volumes of type gp3 or gp2 can be created as small as 1GiB, which may be a workaround that suits your usecase.

Please reach out to AWS Support (or if applicable, your assigned account manager/service architect/etc) if you wish to request a change to the minimum size of io2 volumes.

I'm going to close this issue as there is nothing that can be done in the driver, but please re-open it or open a new issue if you need further assistance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jpeimer
Copy link
Author

jpeimer commented Oct 7, 2024

Thank you for your answer @AndrewSirenko and @ConnorJC3

I understand it's a documented behavior, however, according to the Kubernetes documentation, the storage is free to return a larger volume to the user: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding
If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC. Otherwise, the user will always get at least what they asked for, but the volume may be in excess of what was requested.

So currently, if the users have some automation flows that request a smaller size - they can't use this storage driver.

It looks reasonable to adjust the behavior to the kubernetes expectations. What do you think?

/reopen

@k8s-ci-robot
Copy link
Contributor

@jpeimer: Reopened this issue.

In response to this:

Thank you for your answer @AndrewSirenko and @ConnorJC3

I understand it's a documented behavior, however, according to the Kubernetes documentation, the storage is free to return a larger volume to the user: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding
If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC. Otherwise, the user will always get at least what they asked for, but the volume may be in excess of what was requested.

So currently, if the users have some automation flows that request a smaller size - they can't use this storage driver.

It looks reasonable to adjust the behavior to the kubernetes expectations. What do you think?

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot reopened this Oct 7, 2024
@ConnorJC3
Copy link
Contributor

That section of the Kubernetes docs refers to the KCM and does not affect CSI drivers. However, reviewing the CSI spec (which is the binding contract between COs like Kubernetes and CSI drivers like the EBS CSI Driver), it does appear that over-provisioning a volume is allowed as long as LimitBytes is unset (or the value chosen is within the limit).

I'll leave this open as a feature request for automatically increasing the size of too small volumes - we will need to evaluate if this is suitable, as it may result in users beings surprised and potentially paying more than they expect for volumes. For the moment, I would suggest increasing your volume size to the minimum EBS size as a workaround.

/retitle Feature Request: Automatically upgrade too small volumes to the minimum size
/remove-kind bug
/kind feature

@k8s-ci-robot k8s-ci-robot changed the title Can't provision a volume if requested size is less than 4Gi - EBS io2 Feature Request: Automatically upgrade too small volumes to the minimum size Oct 7, 2024
@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/bug Categorizes issue or PR as related to a bug. labels Oct 7, 2024
@phuhung273 phuhung273 linked a pull request Dec 25, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
4 participants
@ConnorJC3 @k8s-ci-robot @AndrewSirenko @jpeimer