🐛 ec2/byoip: fix EIP leak when creating machine #5039

mtulio · 2024-06-27T20:18:30Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

The instance creation flow is creating by default EIP to instances even if the BYO IP flow is set. BYO IPv4 creates and associates the EIP to instance after it is created, preventing paying for additional EIP (amazon-provided) when creating the instance when the BYO IPv4 Pool is defined to be used by the machine.

Furthermore, the fix provides additional checks to prevent duplicated EIP in the BYO IP reconciliation loop. The extra checks include running the EIP association many times, while the EIP is already associated, and failures in the log when running the EIP association prematurely - when the instance isn't ready, Eg ec2 in pending state.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5038

Special notes for your reviewer:

Checklist:

Release note:

fix duplicated/leaked EIP when using BYO IPv4 on Machines.

k8s-ci-robot · 2024-06-27T20:18:39Z

Hi @mtulio. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nrb · 2024-06-27T20:30:54Z

/ok-to-test

pkg/cloud/services/ec2/eip.go

pkg/cloud/services/ec2/instances.go

pkg/cloud/services/ec2/eip.go

mtulio · 2024-06-28T04:28:13Z

/test ?

k8s-ci-robot · 2024-06-28T04:28:28Z

@mtulio: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build
/test pull-cluster-api-provider-aws-build-docker
/test pull-cluster-api-provider-aws-test
/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-main
/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e-eks-gc
/test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-main
pull-cluster-api-provider-aws-build
pull-cluster-api-provider-aws-build-docker
pull-cluster-api-provider-aws-test
pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mtulio · 2024-06-28T04:28:51Z

/test pull-cluster-api-provider-aws-e2e

mtulio · 2024-06-28T14:25:49Z

Premature failure.

/test pull-cluster-api-provider-aws-e2e

mtulio · 2024-07-03T15:24:55Z

/test pull-cluster-api-provider-aws-e2e

mtulio · 2024-07-04T17:04:38Z

/test pull-cluster-api-provider-aws-e2e

Okay, previous test failures were flake. The latest run pass. OpenShift e2e BYOIP test is also passing install:

time="2024-06-28T19:16:32Z" level=debug msg="E0628 19:16:32.239704     331 
awsmachine_controller.go:544] \"failed to reconcile BYO Public IPv4\" 
err=\"unable to reconcile Elastic IP Pool to instance \\\"i-05af21b09d3552d0f\\\" with state: pending\""

[...]

time="2024-06-28T19:16:50Z" level=debug msg="I0628 19:16:50.432073     331 eip.go:44] 
\"machine is already associated with an Elastic IP with custom Public IPv4 pool\" 
controller=\"awsmachine\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" 
controllerKind=\"AWSMachine\" AWSMachine=\"openshift-cluster-api-guests/ci-op-f04mmlsn-88881-49rgt-bootstrap\" namespace=\"openshift-cluster-api-guests\" 
name=\"ci-op-f04mmlsn-88881-49rgt-bootstrap\" 
reconcileID=\"825b335b-569b-4a16-891d-c6b7fb5a9db6\" 
machine=\"openshift-cluster-api-guests/ci-op-f04mmlsn-88881-49rgt-bootstrap\" 
cluster=\"openshift-cluster-api-guests/ci-op-f04mmlsn-88881-49rgt\" 
eip=\"eipalloc-0678f0da2c1ecd771\" eip-address=\"157.254.217.22\" 
eip-associationID=\"eipassoc-0ea51b9c68cdfb171\" eip-instance=\"i-05af21b09d3552d0f\""

This PR is ready for review. PTAL?
/assign @Ankitasw @dlipovetsky
cc @r4f4 @nrb

/test pull-cluster-api-provider-aws-e2e-eks

controllers/awsmachine_controller.go

mtulio · 2024-07-05T00:27:25Z

e2e EKS test is passing.

I am holding this PR to address the @r4f4 's feedback to decrease the warnings in the code, silent re-queuing expected states.

/hold

mtulio · 2024-08-31T00:03:47Z

Hi @nrb , would you mind taking a review in this bug, please?

As you see in the last comments, we are struggling to run the job pull-cluster-api-provider-aws-e2e, but the e2e-eks, and ele-conformance are passing (not sure under the hood if it's all touching this change, but considering this change is changing Machine creation flow, it think it should).

Furthermore, in downstream/OpenShift we are running several presubmit jobs (Public IPv4 pool is default over aws jobs) across the PR openshift/installer#8676 (vendoring this PR). I also introduced a new presubmit (openshift/release#56114) to enforce to disable the pool in CAPA provisioning to test the non-pool flow and it is all passing: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_installer/8676/pull-ci-openshift-installer-master-e2e-aws-ovn-public-ipv4-pool-disabled/1829621227333881856

I will trigger again, but also open to hear from you if you could share your tougths of another job to validate it, or those presented is ok.

Looking forward to hear from you, thanks!

cc @r4f4 @patrickdillon

/test pull-cluster-api-provider-aws-e2e

mtulio · 2024-09-02T14:40:46Z

/assign @nrb

mtulio · 2024-09-02T14:41:02Z

/test pull-cluster-api-provider-aws-e2e

mtulio · 2024-09-03T17:50:21Z

tl;dr: looks like pull-cluster-api-provider-aws-e2e is failing for unrelated change in this PR, it is trying to create a cluster with an AMI that does not exist in the test account.

After some investigation with @nrb, we are seeing the job e2e always failing the test [It] [unmanaged] [Cluster API Framework] Clusterctl Upgrade Spec [from latest v1beta1 release to v1beta2] Should create a management cluster and then upgrade all the providers .

We are seeing in the Control Plane spec that the AMI for kube 1.25 isn't available:

  - lastTransitionTime: "2024-09-02T15:00:37Z"
    message: 'failed to create AWSMachine instance: failed to find ami: found no AMIs
      with the name: "capa-ami-ubuntu-18.04-?1.25.0-*"'

Looks like the CAPI e2e[1] is setting the following variable:

cluster-api-provider-aws/test/e2e/data/e2e_conf.yaml

Lines 207 to 209 in abe918c

    
           # INIT_WITH_KUBERNETES_VERSION are only used by the clusterctl upgrade test to initialize 
        
           # the management cluster to be upgraded. 
        
           INIT_WITH_KUBERNETES_VERSION: "v1.25.0"

in the test spec:

cluster-api-provider-aws/test/e2e/suites/unmanaged/unmanaged_CAPI_test.go

Lines 112 to 129 in abe918c

    
           ginkgo.Describe("Clusterctl Upgrade Spec [from latest v1beta1 release to v1beta2]", func() { 
        
           	ginkgo.BeforeEach(func() { 
        
           		// As the resources cannot be defined by the It() clause in CAPI tests, using the largest values required for all It() tests in this CAPI test. 
        
           		requiredResources = &shared.TestResource{EC2Normal: 5 * e2eCtx.Settings.InstanceVCPU, IGW: 2, NGW: 2, VPC: 2, ClassicLB: 2, EIP: 2, EventBridgeRules: 50} 
        
           		requiredResources.WriteRequestedResources(e2eCtx, "capi-clusterctl-upgrade-test-v1beta1-to-v1beta2") 
        
           		Expect(shared.AcquireResources(requiredResources, ginkgo.GinkgoParallelProcess(), flock.New(shared.ResourceQuotaFilePath))).To(Succeed()) 
        
           	}) 
        
           	capi_e2e.ClusterctlUpgradeSpec(ctx, func() capi_e2e.ClusterctlUpgradeSpecInput { 
        
           		return capi_e2e.ClusterctlUpgradeSpecInput{ 
        
           			E2EConfig:                       e2eCtx.E2EConfig, 
        
           			ClusterctlConfigPath:            e2eCtx.Environment.ClusterctlConfigPath, 
        
           			BootstrapClusterProxy:           e2eCtx.Environment.BootstrapClusterProxy, 
        
           			ArtifactFolder:                  e2eCtx.Settings.ArtifactFolder, 
        
           			SkipCleanup:                     e2eCtx.Settings.SkipCleanup, 
        
           			MgmtFlavor:                      "remote-management-cluster", 
        
           			InitWithBinary:                  e2eCtx.E2EConfig.GetVariable("INIT_WITH_BINARY_V1BETA1"), 
        
           			InitWithKubernetesVersion:       e2eCtx.E2EConfig.GetVariable("INIT_WITH_KUBERNETES_VERSION"),

Causing the failures when looking up for an AMI that does not exists in the test account. (maybe had been pruned or 1.25 is not supported and dont need anymore?)

[1] https://github.com/kubernetes-sigs/cluster-api/blob/main/test/e2e/clusterctl_upgrade.go#L237C33-L237C58

mtulio · 2024-09-04T19:01:32Z

For reviewers: this PR is general ready for review. Following my last comment, the failure is unrelated with this PR.

pkg/cloud/services/ec2/eip.go

nrb · 2024-09-04T19:13:34Z

/override pull-cluster-api-provider-aws-e2e

This test is failing due to something unrelated right now.

k8s-ci-robot · 2024-09-04T19:13:36Z

@nrb: nrb unauthorized: /override is restricted to Repo administrators.

In response to this:

/override pull-cluster-api-provider-aws-e2e

This test is failing due to something unrelated right now.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

The instance creation flow is creating by default EIP to instances even if the BYO IP flow is set. BYO IPv4 creates and associates the EIP to instance after it is created, preventing paying for additional EIP (amazon-provided) when creating the instance when the BYO IPv4 Pool is defined to be used by the machine. Furthermore, the fix provides additional checks to prevent duplicated EIP in the BYO IP reconciliation loop. The extra checks include running the EIP association many times, while the EIP is already associated, and failures in the log when running the EIP association prematurely - when the instance isn't ready, Eg ec2 in pending state.

mtulio · 2024-09-09T15:35:38Z

PR #5118 merged, PR rebased to re-test the failed upgrade test.
/test pull-cluster-api-provider-aws-e2e

pkg/cloud/services/ec2/eip.go

nrb · 2024-09-09T18:00:25Z

I don't think the failure is related to this PR, nor the previous image issues we were seeing; the AWSMachine resources get to a Ready state and don't have errors themselves.

/retest

mtulio · 2024-09-10T00:18:42Z

@mtulio: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-provider-aws-e2e 4626a6a link false /test pull-cluster-api-provider-aws-e2e
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

I am seeing a lot CloudFormation to provision required environment, I can't see if could be related.

/retest

mtulio · 2024-09-10T02:19:30Z

@nrb e2e passing now! 🎉

nrb · 2024-09-10T13:13:19Z

/approve

k8s-ci-robot · 2024-09-10T13:13:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nrb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rvanderp3 · 2024-09-10T14:11:21Z

/lgtm

mtulio · 2024-09-10T14:26:36Z

/hold cancel

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 27, 2024

k8s-ci-robot requested review from Ankitasw and dlipovetsky June 27, 2024 20:18

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jun 27, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 27, 2024

r4f4 reviewed Jun 27, 2024

View reviewed changes

mtulio force-pushed the OCPBUGS-36293-fix-byoip-eip branch from b58e3e0 to ace1bee Compare June 28, 2024 03:30

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 28, 2024

mtulio mentioned this pull request Jun 28, 2024

OCPBUGS-36293: aws: bump capa to fix EIP leak on bootstrap when BYOIP openshift/installer#8676

Merged

mtulio force-pushed the OCPBUGS-36293-fix-byoip-eip branch from ace1bee to d5882fa Compare June 28, 2024 04:26

k8s-ci-robot assigned Ankitasw and dlipovetsky Jul 4, 2024

r4f4 reviewed Jul 4, 2024

View reviewed changes

controllers/awsmachine_controller.go Outdated Show resolved Hide resolved

k8s-ci-robot assigned nrb Sep 2, 2024

sl1pm4t mentioned this pull request Sep 3, 2024

✨ feat: Add IRSA support for self-managed clusters (rebase) #5109

Open

5 tasks

nrb reviewed Sep 4, 2024

View reviewed changes

pkg/cloud/services/ec2/eip.go Show resolved Hide resolved

nrb reviewed Sep 4, 2024

View reviewed changes

pkg/cloud/services/ec2/eip.go Outdated Show resolved Hide resolved

mtulio force-pushed the OCPBUGS-36293-fix-byoip-eip branch from c804c11 to 16eeaa7 Compare September 5, 2024 19:01

mtulio force-pushed the OCPBUGS-36293-fix-byoip-eip branch from 16eeaa7 to 4626a6a Compare September 9, 2024 15:35

nrb reviewed Sep 9, 2024

View reviewed changes

pkg/cloud/services/ec2/eip.go Show resolved Hide resolved

mtulio changed the title ~~🐛: ec2/byoip: fix EIP leak when creating machine~~ 🐛 ec2/byoip: fix EIP leak when creating machine Sep 9, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 10, 2024

k8s-ci-robot assigned rvanderp3 Sep 10, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 10, 2024

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 10, 2024

k8s-ci-robot merged commit 3f3ce56 into kubernetes-sigs:main Sep 10, 2024
23 checks passed

mtulio deleted the OCPBUGS-36293-fix-byoip-eip branch September 10, 2024 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 ec2/byoip: fix EIP leak when creating machine #5039

🐛 ec2/byoip: fix EIP leak when creating machine #5039

mtulio commented Jun 27, 2024 •

edited

Loading

k8s-ci-robot commented Jun 27, 2024

nrb commented Jun 27, 2024

mtulio commented Jun 28, 2024

k8s-ci-robot commented Jun 28, 2024

mtulio commented Jun 28, 2024

mtulio commented Jun 28, 2024

mtulio commented Jul 3, 2024

mtulio commented Jul 4, 2024

mtulio commented Jul 5, 2024

mtulio commented Aug 31, 2024

mtulio commented Sep 2, 2024

mtulio commented Sep 2, 2024

mtulio commented Sep 3, 2024

mtulio commented Sep 4, 2024

nrb commented Sep 4, 2024

k8s-ci-robot commented Sep 4, 2024

mtulio commented Sep 9, 2024 •

edited

Loading

nrb commented Sep 9, 2024

mtulio commented Sep 10, 2024

mtulio commented Sep 10, 2024

nrb commented Sep 10, 2024

k8s-ci-robot commented Sep 10, 2024

rvanderp3 commented Sep 10, 2024

mtulio commented Sep 10, 2024

🐛 ec2/byoip: fix EIP leak when creating machine #5039

🐛 ec2/byoip: fix EIP leak when creating machine #5039

Conversation

mtulio commented Jun 27, 2024 • edited Loading

k8s-ci-robot commented Jun 27, 2024

nrb commented Jun 27, 2024

mtulio commented Jun 28, 2024

k8s-ci-robot commented Jun 28, 2024

mtulio commented Jun 28, 2024

mtulio commented Jun 28, 2024

mtulio commented Jul 3, 2024

mtulio commented Jul 4, 2024

mtulio commented Jul 5, 2024

mtulio commented Aug 31, 2024

mtulio commented Sep 2, 2024

mtulio commented Sep 2, 2024

mtulio commented Sep 3, 2024

mtulio commented Sep 4, 2024

nrb commented Sep 4, 2024

k8s-ci-robot commented Sep 4, 2024

mtulio commented Sep 9, 2024 • edited Loading

nrb commented Sep 9, 2024

mtulio commented Sep 10, 2024

mtulio commented Sep 10, 2024

nrb commented Sep 10, 2024

k8s-ci-robot commented Sep 10, 2024

rvanderp3 commented Sep 10, 2024

mtulio commented Sep 10, 2024

mtulio commented Jun 27, 2024 •

edited

Loading

mtulio commented Sep 9, 2024 •

edited

Loading