VPC: Add Custom Image reconciliation #1942

cjschaef · 2024-09-04T21:36:12Z

Add support to reconcile a VPC Custom Image for the new v2 VPC Infrastructure reconcile logic.

Related: #1896

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

/area provider/ibmcloud

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

Add support to reconcile VPC Custom Images during Infrastructure creation.

k8s-ci-robot · 2024-09-04T21:36:21Z

Hi @cjschaef. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-09-04T21:36:29Z

✅ Deploy Preview for kubernetes-sigs-cluster-api-ibmcloud ready!

Name	Link
🔨 Latest commit	`141d72f`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-ibmcloud/deploys/66e47e5978b3b500082b7ec5
😎 Deploy Preview	https://deploy-preview-1942--kubernetes-sigs-cluster-api-ibmcloud.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dharaneeshvrd · 2024-09-05T07:00:17Z

cloud/scope/vpc_cluster.go

@@ -23,6 +23,7 @@ import (

 	"github.com/go-logr/logr"

+	"github.com/IBM-Cloud/bluemix-go/crn"


Can we please not use bluemix just for parsing crn? Since we are not using the main functionality(connecting to ibmcloud), it may create dependency issues in future for capi.
I can see they have just split the crn and directly access the id by index. We can add this into some util func and use it locally.

+1, we purposefully migrated from bluemix-go client to other individual clients from IBM org, wondering if there is any other alternative for the crn validation?!

Is there an alternative golang module I can use instead for parsing CRN's?
Is this a valid alternative?
https://github.com/IBM-Cloud/ibm-cloud-cli-sdk/tree/master/bluemix/crn

It would be helpful to keep the CRN for the image, for future enhancements regarding account details, etc. from a Custom Image CRN that gets supplied.

Should I reimplement the CRN logic within CAPI then?

Should I reimplement the CRN logic within CAPI then?

lets do it..

dharaneeshvrd · 2024-09-05T07:13:48Z

cloud/scope/vpc_cluster.go

-	if err = s.TagResource(s.IBMVPCCluster.Name, *vpcDetails.CRN); err != nil {
+
+	// NOTE: This tagging is only attempted once. We may wish to refactor in case this single attempt fails.
+	if err = s.TagResource(s.Name(), *vpcDetails.CRN); err != nil {


Do we need to make this change as part of this PR?
It seems not relevant to me. Please see whether we can create a separate issue and handle it separately.

I can create a second PR, if that PR can be reviewed and merged prior to this PR, to fix this invalid reference.

I have the separate PR opened for the necessary fixes.
#1949

dharaneeshvrd · 2024-09-05T07:25:27Z

cloud/scope/vpc_cluster.go

+		if err != nil {
+			return false, fmt.Errorf("error retrieving vpc custom image by id: %w", err)
+		}
+		if image == nil {


Since you are passing id won't it return an error if the image is not found?
Please see whether we need this nil check?

If the image is not found, I thought we got a 404 DetailedResponse back. I am defensively not expecting a non-error (error == nil) to result in image always being set.

dharaneeshvrd · 2024-09-05T07:26:50Z

cloud/scope/vpc_cluster.go

+
+// createCustomImage will create a new VPC Custom Image.
+func (s *VPCClusterScope) createCustomImage() (*vpcv1.Image, error) {
+	if s.IBMVPCCluster.Spec.Image == nil {


This nil check already happening in main Reconcile func, please remove here.

Sure I can remove.

dharaneeshvrd · 2024-09-05T07:40:00Z

cloud/scope/vpc_cluster.go

+	if err != nil {
+		return nil, fmt.Errorf("error unknown failure creating vpc custom image: %w", err)
+	}
+	if imageDetails == nil || imageDetails.ID == nil || imageDetails.Name == nil || imageDetails.CRN == nil {


Why do we need to check all the params separately?
If the error is nil, would imageDetails still be not nil?

This is done defensively, to make sure the required Image fields are populated from the API response.

dharaneeshvrd · 2024-09-05T07:45:15Z

cloud/scope/vpc_cluster.go

+	}
+
+	// NOTE: This tagging is only attempted once. We may wish to refactor in case this single attempt fails.
+	if err := s.TagResource(s.Name(), *imageDetails.CRN); err != nil {


What happens when it fails to tag?
It seems it would again come to createCustomImage func and it will try to create the image again.

If tagging fails, after creating the Custom Image (or any resource with this similar logic, like VPC), we return an error. Next loop through could then find the resource that was created the first attempt, and update the resource Status (in the VPC example, a VPC is likely "ready" in the second attempt), with the expectation it is found by the name used during creation. This will monitor and/or update the resource Status accordingly. As we expect to not try to recreate a resource that was already created, we don't expect to try to retag the resource.

Logic could be added to check tags and apply if missing, but that logic will be more cumbersome and time consuming that desired currently, and I only plan to worry about tags during Delete actions, which are a secondary concern currently.

I can see you have handled this in ReconcileVPC, but here, you don't have a func like GetVPCID. You are checking either from status or from spec crn. If it's there you are validating the image and set the status based on that. Please take care of this here.

Not sure I am following, but if the recommendation here is to update status (SetResourceStatus) prior to tagging, I can move that for Custom Images.

I can handle VPC in a similar fashion then, in a separate PR I expect.

dharaneeshvrd · 2024-09-05T07:48:59Z

cloud/scope/vpc_cluster.go

+
+	// We must have an OperatingSystem value supplied in order to create the Custom Image.
+	// NOTE(cjschaef): Perhaps we could try defaulting this value, so it isn't required for Custom Image creation.
+	if s.IBMVPCCluster.Spec.Image.OperatingSystem == nil {


If you are planning to keep this, you can add this check into webhook.

Sorry, I'm not sure what the webhook expected change would be, can you provide more details?

Is this additional API changes, such as enhancing this function?

cluster-api-provider-ibmcloud/api/v1beta2/ibmvpccluster_webhook.go

Line 72 in 5f99eed

func (r *IBMVPCCluster) validateIBMVPCCluster() (admission.Warnings, error) {

Yes. But you can take that as an separate issue.

May be for this PR scope you can consider making this check as first check in the function so we can fail fast.

I can do that until it gets removed in favor of webhook validation then.

mkumatag · 2024-09-06T06:59:23Z

/ok-to-test

dharaneeshvrd · 2024-09-06T10:38:10Z

cloud/scope/vpc_cluster.go

+
+	// We must have an OperatingSystem value supplied in order to create the Custom Image.
+	// NOTE(cjschaef): Perhaps we could try defaulting this value, so it isn't required for Custom Image creation.
+	if s.IBMVPCCluster.Spec.Image.OperatingSystem == nil {


Yes. But you can take that as an separate issue.

dharaneeshvrd · 2024-09-06T10:38:27Z

cloud/scope/vpc_cluster.go

+	}
+
+	// NOTE: This tagging is only attempted once. We may wish to refactor in case this single attempt fails.
+	if err := s.TagResource(s.Name(), *imageDetails.CRN); err != nil {


I can see you have handled this in ReconcileVPC, but here, you don't have a func like GetVPCID. You are checking either from status or from spec crn. If it's there you are validating the image and set the status based on that. Please take care of this here.

cjschaef · 2024-09-06T15:28:49Z

cloud/scope/vpc_cluster.go

+
+	// No VPC Custom Image exists or was found.
+	// So, check if the Image spec was defined, as it contains all the data necessary to reconcile.
+	if s.IBMVPCCluster.Spec.Image == nil {


I have moved this check earlier, will be reflected once I push updates.

Karthik-K-N

For using this CustomImage for machine creation, will you be submitting another PR or planning to add to this as separate commit.

Karthik-K-N · 2024-09-09T05:23:13Z

cloud/scope/vpc_cluster.go

+		if s.IBMVPCCluster.Status.Image.ID != "" {
+			imageID = ptr.To(s.IBMVPCCluster.Status.Image.ID)
+		} else if s.IBMVPCCluster.Status.Image.Name != nil {
+			image, err := s.VPCClient.GetImageByName(*s.IBMVPCCluster.Status.Image.Name)


Since we are only setting image status, Do you see any case where we set only Name not ID in status?

Karthik-K-N · 2024-09-09T05:35:13Z

cloud/scope/vpc_cluster.go

+
+	// We must have an OperatingSystem value supplied in order to create the Custom Image.
+	// NOTE(cjschaef): Perhaps we could try defaulting this value, so it isn't required for Custom Image creation.
+	if s.IBMVPCCluster.Spec.Image.OperatingSystem == nil {


May be for this PR scope you can consider making this check as first check in the function so we can fail fast.

Karthik-K-N · 2024-09-09T05:38:20Z

cloud/scope/vpc_cluster.go

+	image, err := s.createCustomImage()
+	if err != nil {
+		return false, fmt.Errorf("error failure trying to create vpc custom image: %w", err)
+	} else if image == nil {


In createCustomImage() func, No where I see you are return nil, nil, So do you think this check is needed?

I've refactored the function and return values, so this block has been removed (it will be reflected once I push up changes).

Karthik-K-N · 2024-09-09T05:56:46Z

cloud/scope/vpc_cluster.go

+// buildCOSObjectHRef will build the HRef path to a COS Object that can be used for VPC Custom Image creation.
+func (s *VPCClusterScope) buildCOSObjectHRef() (*string, error) {
+	// We need COS details in order to create the Custom Image from.
+	if s.IBMVPCCluster.Spec.Image.COSInstance == nil || s.IBMVPCCluster.Spec.Image.COSBucket == nil || s.IBMVPCCluster.Spec.Image.COSObject == nil {


Lets create an issue or make a note to create a validation webhook for this, It will enhance user experience.

I can create an issue.

Karthik-K-N · 2024-09-09T05:58:03Z

cloud/scope/vpc_cluster.go

+	fileHRef, err := s.buildCOSObjectHRef()
+	if err != nil {
+		return nil, fmt.Errorf("error building vpc custom image file href: %w", err)
+	} else if fileHRef == nil {


I think this check is not required.

I will remove.

dharaneeshvrd · 2024-09-12T06:17:16Z

cloud/scope/vpc_cluster.go

+	var imageID *string
+	// Attempt to collect VPC Custom Image info from Status.
+	if s.IBMVPCCluster.Status.Image != nil {
+		if s.IBMVPCCluster.Status.Image.ID != "" {
+			imageID = ptr.To(s.IBMVPCCluster.Status.Image.ID)
+		}
+	} else if s.IBMVPCCluster.Spec.Image == nil {
+		// If no Image spec was defined, we expect it is maintained externally and continue without reconciling. For example, using a Catalog Offering Custom Image, which may be in another account, which means it cannot be looked up, but can be used when creating Instances.
+		s.V(3).Info("No VPC Custom Image defined, skipping reconciliation")
+		return false, nil
+	} else if s.IBMVPCCluster.Spec.Image.Name != nil {
+		// Attempt to retrieve the image details via the name, if it already exists
+		imageDetails, err := s.VPCClient.GetImageByName(*s.IBMVPCCluster.Spec.Image.Name)
+		if err != nil {
+			return false, fmt.Errorf("error checking vpc custom image by name: %w", err)
+		} else if imageDetails != nil && imageDetails.ID != nil {
+			// Prevent relookup (API request) of VPC Custom Image if we already have the necessary data
+			requeue := true
+			if imageDetails.Status != nil && *imageDetails.Status == string(vpcv1.ImageStatusAvailableConst) {
+				requeue = false
+			}
+			s.SetResourceStatus(infrav1beta2.ResourceTypeCustomImage, &infrav1beta2.ResourceStatus{
+				ID:   *imageDetails.ID,
+				Name: s.IBMVPCCluster.Spec.Image.Name,
+				// Ready status will be invert of the need to requeue.
+				Ready: !requeue,
+			})
+			return requeue, nil
+		}
+	} else if s.IBMVPCCluster.Spec.Image.CRN != nil {
+		// Parse the supplied Image CRN for Id, to perform image lookup.
+		imageCRN, err := ParseCRN(*s.IBMVPCCluster.Spec.Image.CRN)
+		if err != nil {
+			return false, fmt.Errorf("error parsing vpc custom image crn: %w", err)
+		}
+		// If the value provided isn't a CRN or is missing the Resource ID, raise an error.
+		if imageCRN == nil || imageCRN.Resource == "" {
+			return false, fmt.Errorf("error parsing vpc custom image crn, missing resource id")
+		}
+		// If we didn't hit an error during parsing, and Resource was set, set that as the Image ID.
+		imageID = ptr.To(imageCRN.Resource)
+	}


Suggested change

var imageID *string

// Attempt to collect VPC Custom Image info from Status.

if s.IBMVPCCluster.Status.Image != nil {

if s.IBMVPCCluster.Status.Image.ID != "" {

imageID = ptr.To(s.IBMVPCCluster.Status.Image.ID)

}

} else if s.IBMVPCCluster.Spec.Image == nil {

// If no Image spec was defined, we expect it is maintained externally and continue without reconciling. For example, using a Catalog Offering Custom Image, which may be in another account, which means it cannot be looked up, but can be used when creating Instances.

s.V(3).Info("No VPC Custom Image defined, skipping reconciliation")

return false, nil

} else if s.IBMVPCCluster.Spec.Image.Name != nil {

// Attempt to retrieve the image details via the name, if it already exists

imageDetails, err := s.VPCClient.GetImageByName(*s.IBMVPCCluster.Spec.Image.Name)

if err != nil {

return false, fmt.Errorf("error checking vpc custom image by name: %w", err)

} else if imageDetails != nil && imageDetails.ID != nil {

// Prevent relookup (API request) of VPC Custom Image if we already have the necessary data

requeue := true

if imageDetails.Status != nil && *imageDetails.Status == string(vpcv1.ImageStatusAvailableConst) {

requeue = false

}

s.SetResourceStatus(infrav1beta2.ResourceTypeCustomImage, &infrav1beta2.ResourceStatus{

ID: *imageDetails.ID,

Name: s.IBMVPCCluster.Spec.Image.Name,

// Ready status will be invert of the need to requeue.

Ready: !requeue,

})

return requeue, nil

}

} else if s.IBMVPCCluster.Spec.Image.CRN != nil {

// Parse the supplied Image CRN for Id, to perform image lookup.

imageCRN, err := ParseCRN(*s.IBMVPCCluster.Spec.Image.CRN)

if err != nil {

return false, fmt.Errorf("error parsing vpc custom image crn: %w", err)

}

// If the value provided isn't a CRN or is missing the Resource ID, raise an error.

if imageCRN == nil || imageCRN.Resource == "" {

return false, fmt.Errorf("error parsing vpc custom image crn, missing resource id")

}

// If we didn't hit an error during parsing, and Resource was set, set that as the Image ID.

imageID = ptr.To(imageCRN.Resource)

}

if s.IBMVPCCluster.Spec.Image == nil {

// If no Image spec was defined, we expect it is maintained externally and continue without reconciling. For example, using a Catalog Offering Custom Image, which may be in another account, which means it cannot be looked up, but can be used when creating Instances.

s.V(3).Info("No VPC Custom Image defined, skipping reconciliation")

return false, nil

} else if s.IBMVPCCluster.Spec.Image.Name != nil {

// Attempt to retrieve the image details via the name, if it already exists

imageDetails, err := s.VPCClient.GetImageByName(*s.IBMVPCCluster.Spec.Image.Name)

if err != nil {

return false, fmt.Errorf("error checking vpc custom image by name: %w", err)

} else if imageDetails != nil && imageDetails.ID != nil {

// Prevent relookup (API request) of VPC Custom Image if we already have the necessary data

requeue := true

if imageDetails.Status != nil && *imageDetails.Status == string(vpcv1.ImageStatusAvailableConst) {

requeue = false

}

s.SetResourceStatus(infrav1beta2.ResourceTypeCustomImage, &infrav1beta2.ResourceStatus{

ID: *imageDetails.ID,

Name: s.IBMVPCCluster.Spec.Image.Name,

// Ready status will be invert of the need to requeue.

Ready: !requeue,

})

return requeue, nil

}

}

var imageID *string

// Attempt to collect VPC Custom Image info from Status.

if s.IBMVPCCluster.Status.Image != nil {

if s.IBMVPCCluster.Status.Image.ID != "" {

imageID = ptr.To(s.IBMVPCCluster.Status.Image.ID)

}

} else if s.IBMVPCCluster.Spec.Image.CRN != nil {

// Parse the supplied Image CRN for Id, to perform image lookup.

imageCRN, err := ParseCRN(*s.IBMVPCCluster.Spec.Image.CRN)

if err != nil {

return false, fmt.Errorf("error parsing vpc custom image crn: %w", err)

}

// If the value provided isn't a CRN or is missing the Resource ID, raise an error.

if imageCRN == nil || imageCRN.Resource == "" {

return false, fmt.Errorf("error parsing vpc custom image crn, missing resource id")

}

// If we didn't hit an error during parsing, and Resource was set, set that as the Image ID.

imageID = ptr.To(imageCRN.Resource)

}

Can we please rearrange something like this?
Checking existing image based on name can be a separate block and using id to check the image status in a different block.

No, I do not want to rearrange to check Spec prior to Status in this way.

I could move s.IBMVPCCluster.Spec.Image == nil first, prior to Status check, but that would be the only check I'd want to move prior to Status check.

Karthik-K-N

Couple small changes, Otherwise LGTM. Thank you.

Karthik-K-N · 2024-09-12T10:47:18Z

pkg/cloud/services/vpc/service.go

+		}
+
+		for i, v := range imagesList.Images {
+			if (*v.Name) == imageName {


Suggested change

if (*v.Name) == imageName {

if *v.Name == imageName {

Sure, then I'd recommend the other Get*ByName functions get updated later as well.

Karthik-K-N · 2024-09-12T10:48:52Z

controllers/ibmvpccluster_controller.go

+	clusterScope.Info("Reconciling VPC Custom Image")
+	if requeue, err := clusterScope.ReconcileVPCCustomImage(); err != nil {
+		clusterScope.Error(err, "failed to reconcile VPC Custom Image")
+		conditions.MarkFalse(clusterScope.IBMVPCCluster, infrav1beta2.ImageReadyCondition, infrav1beta2.VPCReconciliationFailedReason, capiv1beta1.ConditionSeverityError, err.Error())


Lets define another reason specifically for CustomImage rather than using VPCReconciliationFailedReason

oops, yes I will add a new reconciliation failed reason.

Add support to reconcile a VPC Custom Image for the new v2 VPC Infrastructure reconcile logic. Related: kubernetes-sigs#1896

dharaneeshvrd · 2024-09-16T04:24:04Z

LGTM!

Karthik-K-N

/lgtm

k8s-ci-robot · 2024-09-17T04:55:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cjschaef, mkumatag

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mkumatag]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added area/provider/ibmcloud Issues or PRs related to ibmcloud provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 4, 2024

k8s-ci-robot requested review from mkumatag and Prajyot-Parab September 4, 2024 21:36

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 4, 2024

dharaneeshvrd suggested changes Sep 5, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 5, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 6, 2024

dharaneeshvrd suggested changes Sep 6, 2024

View reviewed changes

cjschaef commented Sep 6, 2024

View reviewed changes

Karthik-K-N reviewed Sep 9, 2024

View reviewed changes

cjschaef force-pushed the vpc_reconcile_image branch from 4e23720 to 2fb5b57 Compare September 11, 2024 16:45

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 11, 2024

dharaneeshvrd reviewed Sep 12, 2024

View reviewed changes

Karthik-K-N reviewed Sep 12, 2024

View reviewed changes

cjschaef force-pushed the vpc_reconcile_image branch from 2fb5b57 to 9b28686 Compare September 12, 2024 19:07

VPC: Add Custom Image reconciliation

141d72f

Add support to reconcile a VPC Custom Image for the new v2 VPC Infrastructure reconcile logic. Related: kubernetes-sigs#1896

cjschaef force-pushed the vpc_reconcile_image branch from 9b28686 to 141d72f Compare September 13, 2024 18:03

Karthik-K-N reviewed Sep 16, 2024

View reviewed changes

k8s-ci-robot assigned Karthik-K-N Sep 16, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 16, 2024

mkumatag approved these changes Sep 17, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 17, 2024

k8s-ci-robot merged commit 9dcb066 into kubernetes-sigs:main Sep 17, 2024
13 checks passed

cjschaef deleted the vpc_reconcile_image branch September 17, 2024 14:00

cjschaef mentioned this pull request Sep 17, 2024

Extend VPC (x86_64) Infrastructure reconciliation support #1896

Closed

		@@ -23,6 +23,7 @@ import (

		"github.com/go-logr/logr"

		"github.com/IBM-Cloud/bluemix-go/crn"

VPC: Add Custom Image reconciliation #1942

VPC: Add Custom Image reconciliation #1942

Conversation

cjschaef commented Sep 4, 2024

k8s-ci-robot commented Sep 4, 2024

netlify bot commented Sep 4, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-cluster-api-ibmcloud ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkumatag commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Karthik-K-N left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Karthik-K-N left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dharaneeshvrd commented Sep 16, 2024

Karthik-K-N left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 17, 2024

netlify bot commented Sep 4, 2024 •

edited

Loading