Redis Sentinel: LettuceConnectionFactory not auto detecting changes in cluster

[vasugowda]

Hi Everyone,

We have an Spring application deployed in CloudFoundry and using LettuceConnectionFactory to connect to redis. Started getting error "io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)" recently but however the application was deployed months back and there were no changes in application. After talking to our internal platform team, deleted the application and re-deployed, re-binded the redis service which made the connection to redis successful.

1. What could be the reason for timeout exception, for the same redis service when the application is re-binded it is working fine.
2. Could this be redis service issue?
3. I understand from the LettuceConnectionFactory that RedisSentinelConfiguration should automatically detect changes in the Redis master and slave nodes. Are there any issue w.r.t RedisSentinelConfiguration.

Request you share your feedback.

Environment details

• Spring-boot version 2.6.6

• Redis service is binded to the spring applications

• Deployed in CloudFoundry

• The connection factory is created on application boot up

• The redis service contains below details
  "credentials": {
  "hostname":
  "master_name":
  "password":
  "port":
  "sentinel_port":
  }

• Java code snippet

• var sentinelConfig = new RedisSentinelConfiguration().master(master_name).sentinel(hostname, sentinel_port); sentinelConfig.setPassword(RedisPassword.of(dbKey)); sentinelConfig.setSentinelPassword(RedisPassword.of(dbKey)); return new LettuceConnectionFactory(sentinelConfig);

[mp911de]

What has happened to your Redis instances meanwhile? Were there any DNS or IP address changes? These typically cause outcomes like that one you've described.

[vasugowda]

From the platform redis service, the hostname remained same. However internally the masterip and slaves got changed.

How to address the issue in the code.

[vasugowda]

@mp911de is there any other configuration to set to detect any changes in the cluster in java code. Can you please let me know,

[vasugowda]

Hi All,

Can you please let me know how to address the issue, when the master IP changes in redis cluster. Is this an issue in lettuce-core?

[mp911de]

A sentinel-based setup auto-detects node failovers because it asks the Sentinel server before connecting/reconnecting what the current master IP is. You can collect trace logs of the Lettuce driver to get insight what is happening.

[vasugowda]

Thanks. I will get back on the logs as it is in production environment(Also trying to simulate it in local environment). However can you please confirm from the below code, since it only connects to RedisSentinel there should not be any impact due to master IP change (as sentinel auto detects the node failovers). Is there any scope for improvements in below code.

var sentinelConfig = new RedisSentinelConfiguration().master(master_name).sentinel(hostname, sentinel_port); sentinelConfig.setPassword(RedisPassword.of(dbKey)); sentinelConfig.setSentinelPassword(RedisPassword.of(dbKey)); return new LettuceConnectionFactory(sentinelConfig);