Error while validating pooled Java object (java.util.ArrayList cannot be cast to [B)

[kchristidis]

Dependencies

• spring-data-redis: 2.7.2
• jedis: 3.9.0 (but this also happens with 3.8.0)
• commons-pool2: 2.11.1

Configuration settings

• spring.redis.jedis.pool.max-active=60
• spring.redis.jedis.pool.max-idle=60
• spring.redis.jedis.pool.min-idle=10

What my code does

public void scan(RedisClusterNode redisClusterNode, RedisClusterConnection redisClusterConnection, int scanLimit) { // scanLimit = 50
    ScanOptions scanOptions = ScanOptions.scanOptions().count(scanLimit).build();
    try (Cursor<byte[]> cursor = redisClusterConnection.scan(redisClusterNode, scanOptions)) {
      do {
        cursor.next();
      } while (cursor.hasNext());
    }
  }

This is how I set the redisClusterNode and redisClousterConnection fields before I pass them to the function:

RedisConnection redisConnection = Objects.requireNonNull(stringRedisTemplate.getConnectionFactory()).getConnection();
if (redisConnection instanceof RedisClusterConnection) {
    RedisClusterConnection redisClusterConnection = Objects.requireNonNull(stringRedisTemplate.getConnectionFactory()).getClusterConnection();
    Iterable<RedisClusterNode> redisClusterNodes = redisClusterConnection.clusterGetNodes();
    StreamSupport.stream(redisClusterNodes.spliterator(), false).forEach(
        (redisClusterNode) -> { scan(redisClusterNode, redisClusterConnection, scanLimit);}
    );
} else {
// ...      
}

How things fail

Shortly after I invoke this method, I will get the following:

[redis.clients.jedis.JedisFactory] [commons-pool-evictor] ERROR - <<JedisFactory>> Error while validating pooled Jedis object.
java.lang.ClassCastException: java.util.ArrayList cannot be cast to [B
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:270) ~[jedis-3.8.0.jar:?]
	at redis.clients.jedis.BinaryJedis.ping(BinaryJedis.java:384) ~[jedis-3.8.0.jar:?]
	at redis.clients.jedis.JedisFactory.validateObject(JedisFactory.java:214) ~[jedis-3.8.0.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.evict(GenericObjectPool.java:745) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:160) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.EvictionTimer$WeakRunner.run(EvictionTimer.java:113) ~[commons-pool2-2.11.1.jar:2.11.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_362]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_362]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_362]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]

The timing of this exception differs. Sometimes it shows up right away, sometimes it takes a few seconds (and iterations of the cursor).

Note that the size of the shard is big: ~90M keys.

Things I've tried successfully

1. Running against a much smaller cluster (each shard with hundreds of keys)
2. Using lettuce

Things I've tried without success

1. Running with only one thread active in the pool.
2. Running with an unlimited number of idle threads in the pool.
3. Setting scanLimit to 1.
4. Disabling pooling: spring.redis.jedis.pool.enabled=false

I am running this on a Redis cluster that's on v6.2.5.

How do I go about debugging this? Thank you for your help in advance.

[sazzad16]

We could not pin-point the issue.

Would you be willing to spend some time to investigate which is the first version of Jedis or spring-data-redis with this issue? It would be a great help. Asking you as incidentally now you have a reproducible test scenario.

[kchristidis]

Sure. Is there a compatibility list between Redis versions and jedis version? I'll start with whatever the earliest supported jedis version is, and work my way up from there.

[sazzad16]

There isn't any compatibility list. We take that any Jedis version is compatible with any Redis version.

AFAICR, I didn't hear such issue before 3.8.0. My best guess, you won't need to test Jedis older than 3.6.0.