Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make all reconcile timeouts configurable #4406

Merged
merged 1 commit into from
Dec 21, 2023
Merged

Make all reconcile timeouts configurable #4406

merged 1 commit into from
Dec 21, 2023

Conversation

CecileRobertMichon
Copy link
Contributor

What type of PR is this?

What this PR does / why we need it: Makes all reconciler timeouts configurable via controller flags. In some environments, network latencies can be higher which causes transient failures due to context deadline exceeded errors. This change enables users to modify the default values in case the default values are not optimal for their use case.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4276

Special notes for your reviewer:

  • cherry-pick candidate

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

Release note:

Make all reconcile timeouts configurable

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 18, 2023
@codecov Codecov
Copy link

codecov bot commented Dec 18, 2023 •
edited
Loading

Codecov Report

Attention: 79 lines in your changes are missing coverage. Please review.

Comparison is base (1845be1) 61.91% compared to head (04a7561) 61.97%.

Files Patch % Lines
azure/services/scalesets/client.go 0.00% 6 Missing ⚠️
azure/services/inboundnatrules/client.go 0.00% 4 Missing ⚠️
azure/services/loadbalancers/client.go 0.00% 4 Missing ⚠️
azure/services/networkinterfaces/client.go 0.00% 4 Missing ⚠️
azure/services/privatedns/link_client.go 0.00% 4 Missing ⚠️
azure/services/privatedns/zone_client.go 0.00% 4 Missing ⚠️
azure/services/publicips/client.go 0.00% 4 Missing ⚠️
azure/services/routetables/client.go 0.00% 4 Missing ⚠️
azure/services/securitygroups/client.go 0.00% 4 Missing ⚠️
azure/services/virtualmachines/client.go 0.00% 4 Missing ⚠️
... and 21 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #4406      +/-   ##
==========================================
+ Coverage   61.91%   61.97%   +0.06%     
==========================================
  Files         188      188              
  Lines       18725    18761      +36     
==========================================
+ Hits        11593    11627      +34     
- Misses       6494     6496       +2     
  Partials      638      638

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 18, 2023
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@CecileRobertMichon CecileRobertMichon force-pushed the timeout-reconcile-config branch 3 times, most recently from 35ab5eb to 31e9453 Compare December 19, 2023 18:24
CecileRobertMichon
CecileRobertMichon commented Dec 19, 2023
View reviewed changes
Copy link
Contributor Author

@CecileRobertMichon CecileRobertMichon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign @mboersma @nojnhuh

CecileRobertMichon
CecileRobertMichon commented Dec 19, 2023
View reviewed changes
util/reconciler/defaults.go
@@ -27,8 +27,6 @@ const (
DefaultMappingTimeout = 60 * time.Second
// DefaultAzureServiceReconcileTimeout is the default timeout for an Azure service reconcile.
DefaultAzureServiceReconcileTimeout = 12 * time.Second
// DefaultAKSServiceReconcileTimeout is the default timeout for an AKS service reconcile.
DefaultAKSServiceReconcileTimeout = 30 * time.Second
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was unused since AKS moved to ASO

@CecileRobertMichon CecileRobertMichon removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 19, 2023
mboersma
mboersma reviewed Dec 19, 2023
View reviewed changes
Copy link
Contributor

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

azure/services/async/async_test.go Outdated Show resolved Hide resolved
util/reconciler/defaults.go Outdated Show resolved Hide resolved
azure/services/publicips/publicips.go
@@ -75,6 +75,9 @@ func (s *Service) Reconcile(ctx context.Context) error {
ctx, _, done := tele.StartSpanWithLogger(ctx, "publicips.Service.Reconcile")
defer done()

ctx, cancel := context.WithTimeout(ctx, s.Scope.DefaultedAzureServiceReconcileTimeout())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Publicips didn't have an explicit timeout before this? 👍🏻

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct, it must have been forgotten in the async refactor

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 19, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 9dc083a5ba3483652afe590f5e897e2097394269

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 19, 2023
@CecileRobertMichon
Copy link
Contributor Author

/hold for squash

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 19, 2023
@CecileRobertMichon
Copy link
Contributor Author

/retest

@CecileRobertMichon
Copy link
Contributor Author

squashed

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 20, 2023
mboersma
mboersma reviewed Dec 20, 2023
View reviewed changes
Copy link
Contributor

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 20, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: dbc2fd9277efcd14f8d4e5cb4e7020ab11d05ac1

@nawazkh
Copy link
Member

nawazkh commented Dec 21, 2023

/lgtm

mboersma
mboersma approved these changes Dec 21, 2023
View reviewed changes
Copy link
Contributor

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 21, 2023
@k8s-ci-robot k8s-ci-robot merged commit 2bfbdc9 into kubernetes-sigs:main Dec 21, 2023
20 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.13 milestone Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mboersma mboersma mboersma approved these changes

@jackfrancis jackfrancis Awaiting requested review from jackfrancis

@jsturtevant jsturtevant Awaiting requested review from jsturtevant

@nojnhuh nojnhuh Awaiting requested review from nojnhuh

Assignees

@mboersma mboersma

@nojnhuh nojnhuh

@nawazkh nawazkh

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
CAPZ Planning
Archived in project
Milestone
v1.13
Development

Successfully merging this pull request may close these issues.

2 seconds is too short for azure api call
5 participants
@CecileRobertMichon @k8s-ci-robot @nawazkh @mboersma @nojnhuh