Lettuce Connection Pool

[bdchatham]

I'm currently building an application that uses a connection pool to manage RedisClusterClient instances connected to many different clusters. We're running this in AWS and have seen IAM auth throttling when we run high load tests on just one cluster. It seems Lettuce is creating hundreds of connections per second and this is causing auth events per connection cereated on the server side.

We are following the guidance here: https://github.com/lettuce-io/lettuce-core/wiki/Connection-Pooling.

What could be going on here? We have a couple hundred EC2 instances calling the cluster, but each should only have a handful, or less, connections and they should be long-lived.

I ran a manual test with logs enabled to see the connection pool stats like active, created, borrowed and returned. I ran a small tests with a handful requests and saw it's not closing connections with each request, which would have been an obvious root cause for this issue.

Are there any hidden challenges we may be hitting that aren't immediately obvious until run at scale?

Thanks

[mp911de]

It seems Lettuce is creating hundreds of connections per second

Lettuce is certainly able to do so. Connection lifecycle management is something that resides in your application, as Lettuce cannot assume how you want to use connections. A single long-lived connection is the ideal scenario.

We have a couple hundred EC2 instances calling the cluster, but each should only have a handful, or less, connections and they should be long-lived.

It seems that in some cases you allocate more cluster connections than necessary. Unless you're using blocking operations, having a single connection (without a pool) is the recommended approach.

Hope this helps.