allow to specify wait time for attach disk operation

[sagor999]

What type of PR is this?

/kind feature

What this PR does / why we need it:
Currently poll time for attach disk or global\regional\zonal operation is hard coded:

gcp-compute-persistent-disk-csi-driver/pkg/gce-cloud-provider/compute/gce-compute.go

Line 781 in 9681ab1

return wait.Poll(5*time.Second, 2*time.Minute, func() (bool, error) {

For example, for attach disk it is hard coded to wait at least 5 seconds before checking if disk was attached.

For our project we need to be able to attach disk as fast as possible, and usually it is attached extremely fast (less than 1 second). So we need to be able to poll faster.

This PR allows to do that via providing parameters as command line arguments that allow you to change how often we poll for changes.

Default parameters remain the same (for example AttachDisk is still polled every 5 seconds with timeout of 2 minutes as before), so existing installations will not notice any change.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Allow to specify how frequently to poll for AttachDisk operation status, or any other global\regional\zonal operation.

[k8s-ci-robot]

Welcome @sagor999!

It looks like this is your first PR to kubernetes-sigs/gcp-compute-persistent-disk-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gcp-compute-persistent-disk-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @sagor999. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sagor999]

/assign @saad-ali

[mattcary]

/ok-to-test

[mattcary]

/lgtm
/approve

Thanks for doing this, this is something I've been mulling in general. PD 99% latencies seem to be in the ~10s range and tail latencies can be 1-2m, so this won't make a big difference in the tail where we're usually concerned, but I have been wondering if there's an affect of op or attach times of a few seconds and just missing the first poll, doubling that attach time.

Anyway it hasn't been a priority since the long tail is more important for us but it will be useful to have some knobs and this seems as good a way to do it as any.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattcary, sagor999

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mattcary]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rzetelskik]

usually it is attached extremely fast (less than 1 second)

@sagor999 I hope you don't mind me digging this out, but were you actually able to achieve a subsecond atachment? Experimentally I wasn't able to go below roughly a 3 second threshold, even outside of GKE, i.e. with just a VM, PD and a CLI attach-disk command.

[mattcary]

I don't think you're going to achieve reliable sub-second attach time, at least according to our fleet numbers. P50 is still seconds.

I think the best way to achieve fast "attach" time currently on GCE PD is to use shared PD, attaching it to the two VMs you want to swap between in advance, and do a logical attach in the data plane.

This use case isn't supported by this CSI driver. I'd do it by creating a new CSI driver.

• The volume would be created with a toplogy of two nodes (ie, hostname). The volume only supports RWO.
• On create, the volume would be attached to its topologies (as a block device -- ext4 doesn't support multiwriter block devices)
• ControllerPublish/Unpublish (attach/detach) is a no-op
• Node stage/unstage would mount as usual. There may need to be scsi reservation handling going on (see PD --multi-writer doesn't seem to work with GKE/K8S #848 which I still haven't sorted out)

Dealing with node failure might be tricky. In theory one could attach the PD to a new node, but I don't know how updating a PV's topology works.

Also note that practically what this is doing is make PD operate like and NFS mount in that the "attach" is really done at mount time. So if fast attach is necessary some NFS solution like filestore might actually work better than PD.

[rzetelskik]

I don't think you're going to achieve reliable sub-second attach time, at least according to our fleet numbers. P50 is still seconds.

@mattcary thanks for the input. Are these metrics you're referring to available for the broader public anywhere?