Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using volumegroup snapshot requires a volumesnapshotclass? #1203

Closed
gnufied opened this issue Nov 20, 2024 · 9 comments · Fixed by #1204 or #1210
Closed

Using volumegroup snapshot requires a volumesnapshotclass? #1203

gnufied opened this issue Nov 20, 2024 · 9 comments · Fixed by #1204 or #1210

Comments

@gnufied
Copy link
Contributor

gnufied commented Nov 20, 2024

I was attempting to use groupsnapshots but I forgot to create volumesnapshotclass.

  1. Here is my groupsnapshotclass:
---
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
kind: VolumeGroupSnapshotClass
metadata:
  name: csi-hostpath-groupsnapclass
driver: hostpath.csi.k8s.io
deletionPolicy: Delete
  1. I created a groupsnapshot:
---
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
kind: VolumeGroupSnapshot
metadata:
  name: new-groupsnapshot-demo
spec:
  source:
    selector:
      matchLabels:
        # The PVCs will need to have this label for it to be
        # included in the VolumeGroupSnapshot
        app.kubernetes.io/name: postgresql
  volumeGroupSnapshotClassName: csi-hostpath-groupsnapclass
  1. I noticed that although groupsnapshot is readyForUse, volumeSnapshot object has errors:
spec:
  source:
    persistentVolumeClaimName: csi-pvc
status:
  boundVolumeSnapshotContentName: snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f
  error:
    message: Failed to set default snapshot class with error cannot find default snapshot
      class
    time: "2024-11-20T19:41:56Z"
  readyToUse: false

And there were errors in snapshot-controller logs:

    snapshot-controller : I1120 14:38:35.463359  867688 util.go:284] storeObjectUpdate updating snapshot "default/snapshot-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f" with version 8262                                                   csi-snapshotter : I1120 14:38:35.517898  867697 util.go:284] storeObjectUpdate updating content "snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f" with version 816
csi-snapshotter : I1120 14:38:35.517938  867697 snapshot_controller.go:321] createSnapshotWrapper: Creating snapshot for content snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f through the plugin ...
csi-snapshotter : E1120 14:38:35.517944  867697 snapshot_controller.go:239] failed to getCSISnapshotInput snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f without a snapshot class                                              csi-snapshotter : I1120 14:38:35.517957  867697 snapshot_controller.go:169] updateContentStatusWithEvent[snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f]: the same error &{2024-11-20 14:15:11 -0500 EST 0xc000585280} is alrea
dy set                                                                                                                                                                                                                                                       csi-snapshotter : E1120 14:38:35.517968  867697 snapshot_controller.go:147] checkandUpdateContentStatus [snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f]: error occurred failed to get input parameters to create snapshot for
content snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f: "failed to take snapshot snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f without a snapshot class"
csi-snapshotter : E1120 14:38:35.517978  867697 snapshot_controller_base.go:357] could not sync content "snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f": failed to get input parameters to create snapshot for content snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f: "failed to take snapshot snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f without a snapshot class"
csi-snapshotter : I1120 14:38:35.517985  867697 snapshot_controller_base.go:228] Failed to sync content "snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f", will retry again: failed to get input parameters to create snapshot f
or content snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f: "failed to take snapshot snapcontent-cd7955d4ac3c780f7982722d67de67a737fb5c6242b64d460a8c73a14903357f without a snapshot class"
@gnufied
Copy link
Contributor Author

gnufied commented Nov 20, 2024

This should be more detailed version of - #1201

This was referenced Nov 20, 2024
Merged
Open
@gnufied
Copy link
Contributor Author

gnufied commented Nov 22, 2024

Actually I saw following panic:

snapshot-controller : I1122 12:55:09.243687  855161 snapshot_controller.go:107] syncContent [snapcontent-eab81b1d941144f610f63abace29f2bf4db358c604fe72ba3cbe550174111473]: VolumeSnapshotContent is pre-bound to VolumeSnapshot default/snapshot-eab81b1d941144f610f63abace29f2bf4db358c604fe72ba3cbe550174111473
csi-snapshotter : I1122 12:55:09.243669  855173 util.go:284] storeObjectUpdate updating content "snapcontent-eab81b1d941144f610f63abace29f2bf4db358c604fe72ba3cbe550174111473" with version 466
csi-snapshotter : E1122 12:55:09.243792  855173 panic.go:262] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
csi-snapshotter :       goroutine 177 [running]:
csi-snapshotter :       k8s.io/apimachinery/pkg/util/runtime.logPanic({0x22df120, 0x343c000}, {0x1cb1760, 0x33e1240})
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc
csi-snapshotter :       k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x22df120, 0x343c000}, {0x1cb1760, 0x33e1240}, {0x343c000, 0x0, 0x10000c000812810?})
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e
csi-snapshotter :       k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000812810?})
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
csi-snapshotter :       panic({0x1cb1760?, 0x33e1240?})
csi-snapshotter :               /usr/local/go/src/runtime/panic.go:785 +0x132
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).createSnapshotWrapper(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:342 +0x28e
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).checkandUpdateContentStatusOperation(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:318 +0xc85
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).checkandUpdateContentStatus(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:144 +0x10d
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).syncContent(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:107 +0x56a
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).updateContentInInformerCache(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:350 +0x12d
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).syncContentByKey(0xc0003b5d40, {0xc00075e230, 0x4c})
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:259 +0x6e9
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).processNextItem(0xc0003b5d40)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:226 +0xd1
csi-snapshotter :       github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).contentWorker(...)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:215
csi-snapshotter :       k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
csi-snapshotter :       k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00012d830, {0x22bee60, 0xc0007aaab0}, 0x1, 0xc00005d3b0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
csi-snapshotter :       k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00012d830, 0x0, 0x0, 0x1, 0xc00005d3b0)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
csi-snapshotter :       k8s.io/apimachinery/pkg/util/wait.Until(...)
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
csi-snapshotter :       created by github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).Run in goroutine 84
csi-snapshotter :               /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:185 +0x306
csi-snapshotter :  >
csi-snapshotter : panic: runtime error: invalid memory address or nil pointer dereference [recovered]
csi-snapshotter :       panic: runtime error: invalid memory address or nil pointer dereference
csi-snapshotter : [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x1a7602e]
csi-snapshotter :
csi-snapshotter : goroutine 177 [running]:
csi-snapshotter : k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x22df120, 0x343c000}, {0x1cb1760, 0x33e1240}, {0x343c000, 0x0, 0x10000c000812810?})
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
csi-snapshotter : k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000812810?})
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
csi-snapshotter : panic({0x1cb1760?, 0x33e1240?})
csi-snapshotter :       /usr/local/go/src/runtime/panic.go:785 +0x132
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).createSnapshotWrapper(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:342 +0x28e
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).checkandUpdateContentStatusOperation(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:318 +0xc85
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).checkandUpdateContentStatus(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:144 +0x10d
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).syncContent(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller.go:107 +0x56a
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).updateContentInInformerCache(0xc0003b5d40, 0xc0006ea1c0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:350 +0x12d
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).syncContentByKey(0xc0003b5d40, {0xc00075e230, 0x4c})
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:259 +0x6e9
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).processNextItem(0xc0003b5d40)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:226 +0xd1
csi-snapshotter : github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).contentWorker(...)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:215
csi-snapshotter : k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
csi-snapshotter : k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00012d830, {0x22bee60, 0xc0007aaab0}, 0x1, 0xc00005d3b0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
csi-snapshotter : k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00012d830, 0x0, 0x0, 0x1, 0xc00005d3b0)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
csi-snapshotter : k8s.io/apimachinery/pkg/util/wait.Until(...)
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
csi-snapshotter : created by github.com/kubernetes-csi/external-snapshotter/v8/pkg/sidecar-controller.(*csiSnapshotSideCarController).Run in goroutine 84
csi-snapshotter :       /home/hekumar/redhat/external-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go:185 +0x306
.

When doing this. Not sure if this is related.

@xing-yang
Copy link
Collaborator

Which build are you using? What unmerged PRs have you included?

@gnufied
Copy link
Contributor Author

gnufied commented Nov 22, 2024

master and just the beta branch from @leonardoce lenordo/update-vgs-beta.

@dobsonj
Copy link
Member

dobsonj commented Nov 22, 2024

The panic happens at pkg/sidecar-controller/snapshot_controller.go:342

This recent PR changed if content.Spec.Source.VolumeHandle != nil to if content.Spec.Source.VolumeHandle != nil && !groupSnapshotMember:
https://github.com/kubernetes-csi/external-snapshotter/pull/1204/files#diff-de5c43a0b291e8521fb4224cb9291a290eb6a14e8d6b5de9c5847aced207ae00R239

So now it no longer reports an error if it's NOT a group snapshot member. createSnapshotWrapper calls that function here:

external-snapshotter/pkg/sidecar-controller/snapshot_controller.go

Line 325 in 07425bd

class, snapshotterCredentials, err := ctrl.getCSISnapshotInput(content)

Since that no longer reports an error, it looks like class is now nil when we derefence it for the RemovePrefixedParameters call:

external-snapshotter/pkg/sidecar-controller/snapshot_controller.go

Line 342 in 07425bd

parameters, err := utils.RemovePrefixedParameters(class.Parameters)

@leonardoce
Copy link
Contributor

Yes exactly @dobsonj, that's what is happening.

@gnufied
Copy link
Contributor Author

gnufied commented Nov 22, 2024

/reopen

@k8s-ci-robot
Copy link
Contributor

@gnufied: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot reopened this Nov 22, 2024
@leonardoce
Copy link
Contributor

leonardoce commented Nov 22, 2024 •
edited
Loading

I may be getting something wrong, but I think we report an error only when we don't have the utils.VolumeGroupSnapshotHandleLabel label.
That label is set only for the member of dynamically provisioned volume group snapshot.

So we report that error only when the relative content is not part of a dynamically provisioned volume group snapshot.

		_, groupSnapshotMember := content.Labels[utils.VolumeGroupSnapshotHandleLabel]
		if content.Spec.Source.VolumeHandle != nil && !groupSnapshotMember {
			klog.Errorf("failed to getCSISnapshotInput %s without a snapshot class", content.Name)
			return nil, nil, fmt.Errorf("failed to take snapshot %s without a snapshot class", content.Name)
		}

Edit: sorry, I got it now!

@leonardoce leonardoce mentioned this issue Nov 22, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@gnufied @xing-yang @leonardoce @dobsonj @k8s-ci-robot