chore: defork cloud-provider-azure #6947

comtalyst · 2024-06-20T00:45:26Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

In the past, Azure provider has borrowed some code from cloud-provider-azure, although some did not even exist in cloud-provider-azure at that time.
This results in several pieces of code being duplicated or branched away. This PR is delegating those functionalities back to cloud-provider-azure as much as possible in attempt to eliminate the duplicated logic.

Examples for instances where we miss cloud-provider-azure updates for the duplicated code:

The authentication logic using AAD certificate, and not to mention security concerns implied from outdated code in this category.
The config file format (i.e., azure.json) for autoscaler branches out from the format expected by other components using cloud-provider-azure. For example, the field VmssCacheTTL in autoscaler's duplicated code is named VmssCacheTTLInSeconds in cloud-provider-azure. This causes confusion and kills the purpose of reusable Azure configuration file that cloud-provider-azure is trying to achieve.

On the other way around:

azure: support CLI credentials instead of service principal #4295, which, arguably, should have been added to cloud-provider-azure instead if there was no duplication in the first place. Proposing changes there: Support Azure CLI authentication kubernetes-sigs/cloud-provider-azure#6730

Furthermore, we see no need to have a customized version of cloud-provider-azure library for Cluster Autoscaler. Thus, delegating it back is justified.

This attempt, however, will not be perfect, given we have yet to finish the deprecation for existing behavior differences (e.g., config file format, CLI creds support).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

azure_config.go and azure_client.go are the core changes of this PR. Changes in other files are more of consequences.

Does this PR introduce a user-facing change?

- Azure: from now on, users should refer to https://cloud-provider-azure.sigs.k8s.io/install/configs/ for configuration interface
- Azure: fixed an issue where environment variables were not being passed in when config file exists
- Azure: fixed an issue where some cloud provider configurations were not being validated when UseManagedIdentityExtension is set to true
- Azure: renamed several fields from config file, with old names are still acceptable and taking precedence: `useWorkloadIdentityExtension` to `useFederatedWorkloadIdentityExtension`, `vmssCacheTTL` to `vmssCacheTTLInSeconds`, `vmssVmsCacheTTL` to `vmssVirtualMachinesCacheTTLInSeconds`, `enableVmssFlex` to `enableVmssFlexNodes`
- Azure: renamed several environment variables, with old names are still acceptable and taking precedence: `ARM_USE_MANAGED_IDENTITY_EXTENSION` to `ARM_USE_FEDERATED_WORKLOAD_IDENTITY_EXTENSION`, `AZURE_VMSS_CACHE_TTL` to `AZURE_VMSS_CACHE_TTL_IN_SECONDS`, `AZURE_VMSS_VMS_CACHE_TTL` to `AZURE_VMSS_VMS_CACHE_TTL_IN_SECONDS`, `AZURE_ENABLE_VMSS_FLEX` to `AZURE_ENABLE_VMSS_FLEX_NODES`

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2024-06-20T00:45:34Z

Welcome @comtalyst!

It looks like this is your first PR to kubernetes/autoscaler 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/autoscaler has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-06-20T00:45:35Z

Hi @comtalyst. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

comtalyst · 2024-06-20T02:33:13Z

cluster-autoscaler/cloudprovider/azure/azure_client.go

 )

-// DeploymentsClient defines needed functions for azure network.DeploymentsClient.


Delegated it back to the one provided by cloud-provider-azure. Don't see any breaking things so far, but haven't test it yet.

comtalyst · 2024-06-20T02:34:29Z

cluster-autoscaler/cloudprovider/azure/azure_client.go

 	interfacesClient                interfaceclient.Interface
 	disksClient                     diskclient.Interface
 	storageAccountsClient           storageaccountclient.Interface
 	skuClient                       compute.ResourceSkusClient
 }

-// newServicePrincipalTokenFromCredentials creates a new ServicePrincipalToken using values of the
-// passed credentials map.
-func newServicePrincipalTokenFromCredentials(config *Config, env *azure.Environment) (*adal.ServicePrincipalToken, error) {


Delegated back to cloud-provider-azure.

comtalyst · 2024-06-20T02:37:50Z

cluster-autoscaler/cloudprovider/azure/azure_client.go

-
-	return nil, fmt.Errorf("no credentials provided for AAD application %s", config.AADClientID)
-}
-
 func newAuthorizer(config *Config, env *azure.Environment) (autorest.Authorizer, error) {
 	switch config.AuthMethod {
 	case authMethodCLI:
 		return auth.NewAuthorizerFromCLI()


This CLI authentication is the only real blocker for the near-perfect de-forking of cloud-provider-azure uses here. Its deprecation (or alternatively, supporting it in cloud-provider-azure) could be something up for discussion.

Let's propose supporting it in cloud-provider-azure?

comtalyst · 2024-06-20T02:45:55Z

cluster-autoscaler/cloudprovider/azure/azure_manager.go

 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 )

 const (
-	vmTypeVMSS     = "vmss"


Delegated back to same constants provided by cloud-provider-azure.

comtalyst · 2024-06-20T03:26:12Z

cluster-autoscaler/cloudprovider/azure/azure_config.go

-	SubscriptionID string `json:"subscriptionId" yaml:"subscriptionId"`
-	ResourceGroup  string `json:"resourceGroup" yaml:"resourceGroup"`
-	VMType         string `json:"vmType" yaml:"vmType"`
+	providerazure.Config `json:",inline" yaml:",inline"`


Delegated most of the config back to cloud-provider-azure.

However, this, along with other user-facing interfaces that got delegated back to cloud-provider-azure will be more hideous from the user. For example, if VM type string "vmss" changes to "vmscalesets" on the newer version of cloud-provider-azure, it won't be visible to the user that might not be aware of the usage of cloud-provider-azure.

Well-covered unit testing would definitely help still. But it might even be difficult for the maintainers to keep track of the true size of the exposed interface by the library (e.g., through this Config and cloud-config-file which will be unmarshalled into this struct).

Or, maybe it is by design that the user is supposed to know the existence of this module and can configure them? If so, is it intended for the maintainer to not having to worry about those interfaces, and treat them as black boxes?

Anyway, in this case (and likely in most cases--in practice), although not ideal, I think it is worth it to delegate it back given that changes are really rare, the size of the interface is not too large to have a proper coverage/know what to care about, and there should be a proper deprecation procedures if something changes.

"more hideous" -> "more hidden / less visible / less discoverable"? (And I don't think this is a major issue; can be addressed by documentation.)

It may also be worth capturing in some way whatever guarantees are intended regarding compatibility of this config with cloud-provider-azure config serialization. Though answer is probably just that this one is a superset?

"more hideous" -> "more hidden / less visible / less discoverable"?

Yes. And also less aware of changes.

can be addressed by documentation

How would you imagine the documentation to be like?
Something like A) "we are using cloud-provider-azure, go see their repo for how to use it" or B) "here are the configs" then 'fork' the docs from cloud-provider-azure?
A) would be easier to be maintained, but additional burden to the user. B) would be like a fork to us.

If you ask me I would prefer A). In practice it wouldn't hurt the experience that much given that it seems to always be one-layer.

(also discussed with cloud-provider-azure maintainers--we are allowed to import the config struct)

A) makes sense to me, since cloud-provider-azure should be keeping their docs up to date (doesn't make sense for us to duplicate the work here).

comtalyst · 2024-06-20T03:26:17Z

cluster-autoscaler/cloudprovider/azure/azure_config.go

@@ -42,27 +43,9 @@ const (

 	imdsServerURL = "http://169.254.169.254"

-	// backoff
-	backoffRetriesDefault  = 6


Delegated back to cloud-provider-azure

have we confirmed that the cloud-provider-azure defaults are the same as those listed here?

Yes I did.
Also considering about having unit tests to guarantee them in a long run, but may not be necessary if we rely on user to check cloud-provider-azure directly.

comtalyst · 2024-06-20T03:28:42Z

cluster-autoscaler/cloudprovider/azure/azure_config.go

+	if _, err = assignFromEnvIfExists(&cfg.UserAssignedIdentityID, "ARM_USER_ASSIGNED_IDENTITY_ID"); err != nil {
+		return nil, err
+	}
+	if _, err = assignIntFromEnvIfExists(&cfg.VmssCacheTTLInSeconds, "AZURE_VMSS_CACHE_TTL"); err != nil {


This is one of the fields that have a difference in naming (cloud-provider-azure version has "InSeconds"). Although we keep the environment variables the same to not introduce breaking changes. ~~Not that environment variable name needs to be changed either, given the minimal change.~~ Changed my mind--I introduced both the new one along with it.

comtalyst · 2024-06-20T03:31:31Z

cluster-autoscaler/cloudprovider/azure/azure_config.go

-	case authMethodCLI:
-		// Nothing to check at the moment.
-	default:
-		return fmt.Errorf("unsupported authorization method: %s", cfg.AuthMethod)
 	}

 	if cfg.CloudProviderBackoff && cfg.CloudProviderBackoffRetries == 0 {


This used to be unintentionally(?) neglected when UseManagedIdentityExtension is true, which should be completely unrelated. Current changes should fix this bug.

cluster-autoscaler/cloudprovider/azure/azure_config.go

tallaxes

I really like where this is going, don't see any blockers (though would need another round of close review). Would be good to find a way to prove there are no regressions - to what extent do you think the tests cover this?

comtalyst · 2024-06-22T19:24:37Z

I really like where this is going, don't see any blockers (though would need another round of close review). Would be good to find a way to prove there are no regressions - to what extent do you think the tests cover this?

For config changes, the existing unit tests could be a prove that the delegations are not breaking interface-wise. More unit tests capturing more corner cases will be added as well. I have confidence in those so far.

For service principal token, I think the E2Es are currently the best bet. Some manual long-running tests might help as well.
Looking at their code, I don't see if they have real differences from ours at the moment. But not that my eyes should be trusted that much.

comtalyst · 2024-08-10T01:12:25Z

/test pull-cluster-autoscaler-e2e-azure

k8s-ci-robot · 2024-08-10T01:12:39Z

@comtalyst: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-cluster-autoscaler-e2e-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

comtalyst · 2024-08-16T17:57:58Z

/ok-to-test

tallaxes

Looks great, mostly minor quips/questions.

May also need to check if updates/renames to config/environment variables warrant any updates to readme and/or charts (could be a follow-up).

cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go

cluster-autoscaler/cloudprovider/azure/azure_cloud_provider_test.go

tallaxes · 2024-08-16T19:18:44Z

cluster-autoscaler/cloudprovider/azure/azure_config.go

+	if cfg.VMType != providerazureconsts.VMTypeStandard && cfg.VMType != providerazureconsts.VMTypeVMSS {
+		return fmt.Errorf("unsupported VM type: %s", cfg.VMType)
 	}


The comment on VMType still says vmssflex value is supported. Also need to think through migration, if that's indeed how it was intended to be set, and is changing now (if anybody is using it to begin with)

func (m *AzureManager) buildNodeGroupFromSpec(spec string) (cloudprovider.NodeGroup, error) { scaleToZeroSupported := scaleToZeroSupportedStandard if strings.EqualFold(m.config.VMType, providerazureconsts.VMTypeVMSS) { scaleToZeroSupported = scaleToZeroSupportedVMSS } s, err := dynamic.SpecFromString(spec, scaleToZeroSupported) if err != nil { return nil, fmt.Errorf("failed to parse node group spec: %v", err) } vmsPoolSet := m.azureCache.getVMsPoolSet() if _, ok := vmsPoolSet[s.Name]; ok { return NewVMsPool(s, m), nil } switch m.config.VMType { case providerazureconsts.VMTypeStandard: return NewAgentPool(s, m) case providerazureconsts.VMTypeVMSS: return NewScaleSet(s, m, -1, false) default: return nil, fmt.Errorf("vmtype %s not supported", m.config.VMType) } }

From this (that has always been here), VM types other than these two will effectively disable CAS already, given this method is critical. I don't think anyone can use that right now.
I think it is fair to "notify users" through fail initialization like this, given the config is cluster-level.

For our support of flex, I think we do have it (that EnableVMSSFlexNodes?), but not through this. This sounds like something to revisit.

tallaxes · 2024-08-16T19:22:20Z

cluster-autoscaler/cloudprovider/azure/azure_config_test.go

Any more tests we could/should add here (e.g. to test the changes in the overrid logic, such as legacy overrides?) Or are these covered in the azure_manager_tests?

Those are already covered in azure_manager_test.go, I think the coverage is quite decent now.

tallaxes · 2024-08-16T19:23:09Z

cluster-autoscaler/cloudprovider/azure/azure_fakes.go

There is no suitable mock in the cloud provider?

Haven't look at that yet. But, given low traffic for this area (deprecated VMAS type pool), I don't think we should put effort on it now.

cluster-autoscaler/cloudprovider/azure/azure_scale_set_test.go

cluster-autoscaler/cloudprovider/azure/azure_util.go

comtalyst · 2024-08-16T23:00:13Z

May also need to check if updates/renames to config/environment variables warrant any updates to readme and/or charts (could be a follow-up).

As discussed, I believe we should delegate all those to cloud-provider-azure docs. We could sweep through the readmes if there is any left to begin with.
For now, I left a note in the release notes (in PR description).

Also, I give it backward compatibility for those who wants to use old names for the interface config fields/envs. Included in the release note as well.

tallaxes

LGTM but for error checking bug

tallaxes · 2024-08-18T04:54:36Z

cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go

@@ -258,9 +267,9 @@ func (as *AgentPool) deleteOutdatedDeployments() (err error) {
 	errList := make([]error, 0)
 	for _, deployment := range toBeDeleted {
 		klog.V(4).Infof("deleteOutdatedDeployments: starts deleting outdated deployment (%s)", *deployment.Name)
-		_, err := as.manager.azClient.deploymentsClient.Delete(ctx, as.manager.config.ResourceGroup, *deployment.Name)
+		rerr := as.manager.azClient.deploymentClient.Delete(ctx, as.manager.config.ResourceGroup, *deployment.Name)
 		if err != nil {


Should be checking rerr

Done, and looks like this one of the last of this kind of issue

tallaxes

/lgtm
/approve

tallaxes · 2024-08-20T22:16:06Z

/approve

comtalyst · 2024-08-23T05:25:21Z

/assign @towca
There are some module bumps in go.mod from this, thanks.

towca · 2024-08-26T14:52:03Z

CA go.mod changes LGTM

/approve

k8s-ci-robot · 2024-08-26T14:52:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: comtalyst, tallaxes, towca

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cluster-autoscaler/OWNERS~~ [towca]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

comtalyst · 2024-08-26T17:50:03Z

/cherry-pick cluster-autoscaler-release-1.30

k8s-infra-cherrypick-robot · 2024-08-26T17:51:01Z

@comtalyst: #6947 failed to apply on top of branch "cluster-autoscaler-release-1.30":

Applying: chore: defork cloud-provider-azure
Using index info to reconstruct a base tree...
M	cluster-autoscaler/cloudprovider/azure/azure_manager.go
M	cluster-autoscaler/cloudprovider/azure/azure_manager_test.go
M	cluster-autoscaler/go.mod
M	cluster-autoscaler/go.sum
Falling back to patching base and 3-way merge...
Auto-merging cluster-autoscaler/go.sum
CONFLICT (content): Merge conflict in cluster-autoscaler/go.sum
Auto-merging cluster-autoscaler/go.mod
CONFLICT (content): Merge conflict in cluster-autoscaler/go.mod
Auto-merging cluster-autoscaler/cloudprovider/azure/azure_manager_test.go
Auto-merging cluster-autoscaler/cloudprovider/azure/azure_manager.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 chore: defork cloud-provider-azure
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick cluster-autoscaler-release-1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 20, 2024

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 20, 2024

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 20, 2024

k8s-ci-robot requested a review from BigDarkClown June 20, 2024 00:46

k8s-ci-robot added the area/cluster-autoscaler label Jun 20, 2024

k8s-ci-robot requested a review from nilo19 June 20, 2024 00:46

k8s-ci-robot added the area/provider/azure Issues or PRs related to azure provider label Jun 20, 2024

comtalyst commented Jun 20, 2024

View reviewed changes

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from d07dbf2 to 24d4667 Compare June 20, 2024 20:26

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 20, 2024

tallaxes reviewed Jun 22, 2024

View reviewed changes

cluster-autoscaler/cloudprovider/azure/azure_config.go Outdated Show resolved Hide resolved

tallaxes reviewed Jun 22, 2024

View reviewed changes

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from db0fb68 to 1b54748 Compare July 17, 2024 21:53

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 1b54748 to ecd0123 Compare August 3, 2024 00:17

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from ecd0123 to 2b40ce5 Compare August 10, 2024 00:55

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2024

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 2b40ce5 to 7e9172d Compare August 10, 2024 00:57

comtalyst mentioned this pull request Aug 12, 2024

REQUEST: New membership for comtalyst kubernetes/org#5099

Closed

11 tasks

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 7e9172d to 4830b17 Compare August 16, 2024 17:53

comtalyst marked this pull request as ready for review August 16, 2024 17:57

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2024

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 16, 2024

k8s-ci-robot requested a review from x13n August 16, 2024 17:58

comtalyst requested a review from tallaxes August 16, 2024 17:58

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 4830b17 to 811f93c Compare August 16, 2024 18:13

tallaxes suggested changes Aug 16, 2024

View reviewed changes

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 811f93c to 2d55b65 Compare August 17, 2024 01:43

comtalyst requested a review from tallaxes August 17, 2024 01:44

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch 2 times, most recently from 2849664 to dfac3bb Compare August 17, 2024 02:43

tallaxes suggested changes Aug 18, 2024

View reviewed changes

comtalyst requested a review from tallaxes August 19, 2024 04:05

chore: defork cloud-provider-azure

4e12429

comtalyst force-pushed the comtalyst/cas-azure-defork-cloud-provider-azure branch from 52fe2da to 4e12429 Compare August 19, 2024 17:31

tallaxes approved these changes Aug 20, 2024

View reviewed changes

k8s-ci-robot assigned tallaxes Aug 20, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 20, 2024

k8s-ci-robot assigned towca Aug 23, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2024

k8s-ci-robot merged commit 527de12 into kubernetes:master Aug 26, 2024
7 checks passed

		)

		// DeploymentsClient defines needed functions for azure network.DeploymentsClient.

chore: defork cloud-provider-azure #6947

chore: defork cloud-provider-azure #6947

Conversation

comtalyst commented Jun 20, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jun 20, 2024

k8s-ci-robot commented Jun 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

comtalyst Jun 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

comtalyst Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallaxes left a comment

Choose a reason for hiding this comment

comtalyst commented Jun 22, 2024

comtalyst commented Aug 10, 2024

k8s-ci-robot commented Aug 10, 2024

comtalyst commented Aug 16, 2024

tallaxes left a comment • edited Loading

Choose a reason for hiding this comment

tallaxes Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

comtalyst commented Aug 16, 2024 • edited Loading

tallaxes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallaxes left a comment • edited Loading

Choose a reason for hiding this comment

tallaxes commented Aug 20, 2024

comtalyst commented Aug 23, 2024 • edited Loading

towca commented Aug 26, 2024

k8s-ci-robot commented Aug 26, 2024

comtalyst commented Aug 26, 2024

k8s-infra-cherrypick-robot commented Aug 26, 2024

comtalyst commented Jun 20, 2024 •

edited

Loading

comtalyst Jun 22, 2024 •

edited

Loading

comtalyst Jun 20, 2024 •

edited

Loading

tallaxes left a comment •

edited

Loading

tallaxes Aug 16, 2024 •

edited

Loading

comtalyst commented Aug 16, 2024 •

edited

Loading

tallaxes left a comment •

edited

Loading

comtalyst commented Aug 23, 2024 •

edited

Loading