Under certain conditions, a DeleteSnapshot will never get issued for a deleted volumesnapshot

[akalenyu]

What happened:
Some race that I have yet to figure out will cause the sidecar controller to opt out of sending a
DeleteSnapshot and thus the volumesnapshot/content to stay pending in deletion indefinitely
(including waiting out the resync period)

What you expected to happen:
volumesnapshot/content requeued on some condition so eventually deleted properly

How to reproduce it:
seems tough, but something along the lines of quickly deleting a recently created snapshot

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  annotations: ...
  creationTimestamp: 2024-12-23T08:01:21Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2024-12-23T08:01:34Z
  finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 2
  labels: ...
spec:
  source:
    persistentVolumeClaimName: ...
  volumeSnapshotClassName: ocs-...
status:
  boundVolumeSnapshotContentName: snapcontent-...
  readyToUse: false

Anything else we need to know?:

Environment:

• Driver version:
• Kubernetes version (use kubectl version):
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[xing-yang]

Can you show the details of the VolumeSnapshotContent in this case?

[akalenyu]

I actually had it backed up:

apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshotContent
  metadata:
    annotations:
      snapshot.storage.kubernetes.io/deletion-secret-name: ...
      snapshot.storage.kubernetes.io/deletion-secret-namespace: ...
      snapshot.storage.kubernetes.io/volumesnapshot-being-deleted: "yes"
    creationTimestamp: "2024-12-23T08:01:21Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2024-12-23T08:01:34Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
    generation: 2
    name: snapcontent-...
...
  spec:
    deletionPolicy: Delete
    driver: openshift-storage.rbd.csi.ceph.com
    source:
      volumeHandle: ...
    sourceVolumeMode: Block
    volumeSnapshotClassName: ocs-...
    volumeSnapshotRef:
      apiVersion: snapshot.storage.k8s.io/v1
      kind: VolumeSnapshot
      name: ...
      namespace: ...
...
  status:
    creationTime: 1734940895115234196
    readyToUse: true
    restoreSize: 32212254720
    snapshotHandle: ...

Actually, now that I think about it some more, it's possible #1259 fixes this even without waiting out the entire resync period
(Since some things like status changes don't result in a requeue)

[xing-yang]

Please provide logs if deletion does not happen.

[akalenyu]

Please provide logs if deletion does not happen.

I'll try to reproduce and increase verbosity. Which value should I set it to?

Actually, now that I think about it some more, it's possible #1259 fixes this even without waiting out the entire resync period
(Since some things like status changes don't result in a requeue)

Does this theory make sense?