Auto backoff catchup #106
Merged
Auto backoff catchup #106
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Matthew Whitehead <[email protected]>
Signed-off-by: Matthew Whitehead <[email protected]>
Signed-off-by: Matthew Whitehead <[email protected]>
Signed-off-by: Matthew Whitehead <[email protected]>
matthew1001
force-pushed
the
auto-backoff-catchup
branch
from
1c62ce0 to
321864d
Compare
nguyer
approved these changes
Jan 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is intended to resolve FireFly issue hyperledger/firefly#1370
The design is based on the assumption that different JSON/RPC providers will return slightly different errors for this scenario, so the event listener uses a (configurable) regex pattern when an error is returned from eth_getLogs to decide if it should be retried with a smaller catchupPageSize.
If eth_getLogs returns such an error, the page size is halved (the initial default being 500) until one of the following is true:
the calls are successful
the catchupPageSize reaches 1
(Reducing catchupPageSize to 1 is likely to mean the connector never catches up with main. It feels arbitrary to me however, to pick a higher value (10? 20?) that we set as a minimum to reduce the value down to so I've left the minimum as 1 for now.)
The value of catchupPageSize is only changed for the current runtime. This means that if the JSON/RPC endpoint is upgraded to support larger responses later then a restart of FireFly will return to the default catchupPageSize.
I haven't gone to the complexity of trying to slowly increase catchupPageSize back to a higher value, since this adds complexity for what feels like relatively little benefit. If the use knows the limit they can manually configure it more precisely, but this PR at least moves FF along to work correctly for limited-resource API providers. It's certainly an option for a future PR though.