Lua Script not working when one of the redis master node is down in cluster

[AdityaRanjan012]

Current Redis Cluster config is :
cluster nodes

edab538440e1d3a92562c50ad2950d2182ef91fc 127.0.0.1:7002@17002 slave 99b546c4c1670ea9c99bedd2f05dcf82af44b2c0 0 1676021511040 22 connected
ade83ad879323e82500cb5d7d37da495f17898d9 127.0.0.1:7001@17001 slave 893f6a0543387799ed30b81e5e2f1d48b4142bff 0 1676021511040 19 connected
ac66659fddd68ee88b3f5599778edbd9c8d12666 127.0.0.1:7003@17003 master - 0 1676021511141 24 connected 10923-16383
893f6a0543387799ed30b81e5e2f1d48b4142bff 127.0.0.1:7004@17004 master - 0 1676021511543 19 connected 0-5460
99b546c4c1670ea9c99bedd2f05dcf82af44b2c0 127.0.0.1:7005@17005 myself,master - 0 1676021511000 22 connected 5461-10922
799c51a03869c38d566191ef9ca16541cc94f7d8 127.0.0.1:7006@17006 slave ac66659fddd68ee88b3f5599778edbd9c8d12666 0 1676021511141 24 connected

(3 master 3 slave)

we have set a key value pair key : test2 and value test2 which is stored in node 7005
127.0.0.1:7005> get test2
"test2"
127.0.0.1:7005> EVAL "return redis.call('GET', KEYS[1])" 1 test2
"test2"

In our java-code we are using lettuce client library with this we are fetching data via direct get command and with lua-script

GET:
RedisAdvancedClusterCommands<String, String> syncCommands = getRedisClusterConnection().sync();
syncCommands.get(key);

SCRIPT :
public static String GET_VALUE_SCRIPT = "return redis.call('get',KEYS[1])";
List resultScript = lettuceUtil.executeTaggingScript(RedisScripts.GET_VALUE_SCRIPT , keys , "random");

@trace
public List executeTaggingScript(String luaScript, List keys, String... values) {
RedisAdvancedClusterCommands<String, String> syncCommands = getRedisClusterConnection().sync();
syncCommands.scriptLoad(luaScript);
return syncCommands.eval(luaScript, ScriptOutputType.MULTI, keys.toArray(new String[0]), values);
}

NOW WE FAILED ONE OF THE REDIS-NODE

./redis-cli -c -p 7005 debug segfault

POST THIS cluster node state is :

893f6a0543387799ed30b81e5e2f1d48b4142bff 127.0.0.1:7004@17004 master - 0 1676021573186 19 connected 0-5460
ac66659fddd68ee88b3f5599778edbd9c8d12666 127.0.0.1:7003@17003 master - 0 1676021573789 24 connected 10923-16383
99b546c4c1670ea9c99bedd2f05dcf82af44b2c0 127.0.0.1:7005@17005 master,fail - 1676021566438 1676021566138 22 disconnected
799c51a03869c38d566191ef9ca16541cc94f7d8 127.0.0.1:7006@17006 slave ac66659fddd68ee88b3f5599778edbd9c8d12666 0 1676021573086 24 connected
ade83ad879323e82500cb5d7d37da495f17898d9 127.0.0.1:7001@17001 myself,slave 893f6a0543387799ed30b81e5e2f1d48b4142bff 0 1676021573000 13 connected
edab538440e1d3a92562c50ad2950d2182ef91fc 127.0.0.1:7002@17002 master - 0 1676021573086 25 connected 5461-10922

So now as 7002 node was slave of 7005 , this node become the master (master/slave failover worked)
edab538440e1d3a92562c50ad2950d2182ef91fc 127.0.0.1:7002@17002 master - 0 1676021573086 25 connected 5461-10922

Now when on redis-cli we are fetching the same key via get command or lua script we are getting the data

But via code where we are using lettuce client :

and hitting get command it is getting executed
(TOPLOGOLY GOT REFRESHED OF LETTUCE CLIENT AS WE ARE USING ADAPTIVE REFRESH)
@bean
public ClusterTopologyRefreshOptions clusterTopologyRefreshOptions() {
return ClusterTopologyRefreshOptions
.builder()
.enablePeriodicRefresh(Duration.ofMinutes(1))
.enableAllAdaptiveRefreshTriggers()
.build();
}

BUT WHILE FETCHING FROM LETTUCE LUA-SCRIPT COMMAND WE GETTING REDIS-TIMEED OUT FOR KEY TEST-2

Unknown error occurred
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 5 second(s)
at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:53)
at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:246)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:130)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy233.scriptLoad(Unknown Source)
at com.myntra.coupon.cache.LettuceUtil.executeTaggingScript(LettuceUtil.java:361)
at com.myntra.coupon.cache.LettuceUtil$$FastClassBySpringCGLIB$$7e1372cc.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.myntra.coupon.cache.LettuceUtil$$EnhancerBySpringCGLIB$$c7a2a535.executeTaggingScript()
at com.myntra.coupon.controller.v2.CouponServiceController.testLettuce(CouponServiceController.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-02-10 15:19:53.352 uuid=[] [lettuce-eventExecutorLoop-1-4] INFO i.l.core.protocol.ConnectionWatchdog - Reconnecting, last destination was 127.0.0.1:7005

Redis version : 4.0.12
lettuce client version : 6.1.4.RELEASE

Is anyone else faced the same issue if one of the node inside the cluster is down in this case LUA Script not works even though master/slave failover was successful and topology refresh also happened on lettuce-client level ?

[AdityaRanjan012]

Please let us know , if anyone else also faced the same issue and how to resolve this

[mp911de]

Please enable debug logging, so you see to which node the Lua script execution is being routed to.