PD --multi-writer doesn't seem to work with GKE/K8S

[scanfield]

TLDR: attaching a --multi-writer PD to 2 GCE VMs works as expected, but if using GKE & exposing the PD as a block device then each pod will not see the other's writes unless the disk is detached and re-attached (e.g. when the pod restarts).

Here's a git repo with reproduction steps: https://github.com/scanfield/mw-repro

I know this is probably on the bleeding edge since --multi-writer is in beta, but it seems like the K8S functionality must work with other types of multi writer volumes.

Here's a slack thread for some context:
https://kubernetes.slack.com/archives/C09QZFCE5/p1632980542168500

[mattcary]

/assign @mattcary

Thanks Steven! I'll work on a repro.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mattcary]

/lifecycle frozen

This is still in my queue

[scanfield]

Extremely gentle ping -- this would still be quite useful

[mattcary]

Thanks for the ping, still in my queue :-P

[mattcary]

I looked into this for a bit, but didn't make progress.

I verified your repro (thanks for such a clear example!) and then simultaneously attached the same shared disk to bare vms, and verified that while the devices on the VM were seeing data immediately appear back and forth, those same writes were at the same time not visible from within the k8s pod.

I thought it might have to do with the bind mount k8s is doing. See here for how the kubelet sets up block devices.

But making bind mounts on the bare vm and writing through them didn't have any affect---the writes were immediately visible across different VMs (except the k8s ones).

Hrm.

[msau42]

Next layer to check could be the container runtime

[mattcary]

Yeah good point. Maybe something with mount propagation.