Best way to handle password rotations?

[doktorjw]

Been doing a lot of digging on how best to handle a rotating password on Redis. We don't really need a connection pool (just doing some simple get/set in a java app). Not ready to use spring-data-redis yet, either .... just basic plain jane lettuce/redis.

Our thinking is to catch the exception that happens when an invalid password is used, and replace the connection with a new one and retry the operation.

Is there a better way to handle a rotating password on Redis more cleanly? I see that people have done so but ... no details on how.

[doktorjw]

Oh, to follow up just a bit .... we removed the connection pooling, and we are using clustered redis. Without connection pooling, and using a RedisCredentialsProvider, it appears that the reconnect internal to lettuce is handling things for us, and creating a new connection before we invoke anything. Would be nice; but the use of the connection pool, we end up with connections that are configured with the old password and the auto-reconnect is NOT pretty, and we fail with the invalid username/password error message.

Scratching our heads, and testing more.

[mp911de]

RedisCredentialsProvider is the way to go. Auto-reconnect requests new credentials on reconnect.

Since introduction of the ACL commands, you can rotate passwords by adding a new password and removing the old password after the last client has received the new password.

[doktorjw]

Thx. Using RedisCredentialsProvider without a pool seems to do the job (and with our setup, we have only 1 client for this redis instance). What we just observed now is a failure to refresh the topology (wrong pass) as if that part of the internals didn't refresh the password. We're chatting with our systems guy about that, but he's out-of-pocket for a conference & travel.

We're still digging and running tests; still puzzling over why this didn't seem to work with a generic pool of connections, too.

Here is how we are currently configuring our connection, complete with diagnostic logging that we have added:

@Component
public class RedisConfiguration {

    private static final String X_509 = "X.509";
    private static final int REFRESH_PERIOD_IN_MINUTES = 1;
    private static final int UPPER_JITTER_LIMIT_IN_SECONDS = 5;
    private static final int LOWER_JITTER_LIMIT_IN_MILLISECONDS = 100;
    private static final int JITTER_BASE_IN_MILLISECONDS = 100;

    private String url;
    private int port;
    private String certPath;
    private String authTokenPath;

    public RedisConfiguration( @Value( "${redis.url}" ) String url,
                               @Value( "${redis.port}" ) int port,
                               @Value( "${redis.auth.cert}" ) String certPath,
                               @Value( "${redis.auth.token}" ) String authTokenPath ) {
        this.url = url;
        this.port = port;
        this.certPath = certPath;
        this.authTokenPath = authTokenPath;
    }

    public StatefulRedisClusterConnection<String, String> redisConnection() throws Exception {

        X509Certificate cert = getCertificate( certPath );
        ClientResources resources = getClientResources();
        ClusterTopologyRefreshOptions topologyRefreshOptions = getTopologyRefreshOptions();
        RedisURI redisUri = getRedisUri();
        SocketOptions socketOptions = getSocketOptions();

        ClusterClientOptions clientOptions = getClusterClientOptions( topologyRefreshOptions, socketOptions, cert );
        RedisClusterClient clusterClient = getClient( resources, redisUri );
        clusterClient.setOptions( clientOptions );

        return clusterClient.connect();
    }
    protected X509Certificate getCertificate( String certPath ) throws CertificateException, IOException {
        X509Certificate cert = null;

        try ( FileInputStream inputStream = new FileInputStream( new File( certPath ) ) ) {
            CertificateFactory factory = CertificateFactory.getInstance( X_509 );
            cert = (X509Certificate) factory.generateCertificate( inputStream );
        }
        return cert;
    }

    protected SocketOptions getSocketOptions() {
        // @formatter:off
        return SocketOptions.builder()
                .keepAlive( true )
                .build();
        // @formatter:on
    }

    // HWO: do not start TLS in the method below !!!
    protected RedisURI getRedisUri() {
        // @formatter:off
        return RedisURI.Builder
                .redis( url, port )
                .withSsl( true )
                .withAuthentication( new TokenProvider( authTokenPath ) )
                .build();
        // @formatter:on
    }

    protected ClusterTopologyRefreshOptions getTopologyRefreshOptions() {
        // @formatter:off
        return ClusterTopologyRefreshOptions.builder()
                .refreshPeriod( Duration.ofMinutes( REFRESH_PERIOD_IN_MINUTES ) )
                .enableAllAdaptiveRefreshTriggers()
                .dynamicRefreshSources( false )
                .closeStaleConnections( true )
                .build();
        // @formatter:on
    }

    protected ClientResources getClientResources() {
        // @formatter:off
        return DefaultClientResources.builder()
                .reconnectDelay( Delay.fullJitter( Duration.ofMillis( LOWER_JITTER_LIMIT_IN_MILLISECONDS ),
                                                   Duration.ofSeconds( UPPER_JITTER_LIMIT_IN_SECONDS ),
                                                   JITTER_BASE_IN_MILLISECONDS,
                                                   TimeUnit.MILLISECONDS ) )
                .build();
        // @formatter:on
    }

    protected ClusterClientOptions getClusterClientOptions( ClusterTopologyRefreshOptions topologyOptions, SocketOptions socketOptions,
                                                          X509Certificate cert ) {
        // @formatter:off
        return ClusterClientOptions.builder()
                .autoReconnect( true )
                .topologyRefreshOptions( topologyOptions )
                .socketOptions( socketOptions )
                .sslOptions( getSslOptions( cert ) )
                .validateClusterNodeMembership( false )
                .nodeFilter( it -> !( it.is( RedisClusterNode.NodeFlag.FAIL )
                        || it.is( RedisClusterNode.NodeFlag.EVENTUAL_FAIL )
                        || it.is( RedisClusterNode.NodeFlag.NOADDR ) ) )
                .build();
        // @formatter:on
    }

    protected SslOptions getSslOptions( X509Certificate cert ) {
        // @formatter:off
        return SslOptions.builder()
                .jdkSslProvider()
                .sslContext( scb ->
                    scb.trustManager( cert )
                )
                .build();
        // @formatter:on
    }

    protected RedisClusterClient getClient( ClientResources resources, RedisURI uri ) {
        try ( RedisClusterClient clusterClient = RedisClusterClient.create( resources, uri ) ) {
            return clusterClient;
        }
    }

    private class TokenProvider implements RedisCredentialsProvider {

        private static final Logger LOGGER = LoggerFactory.getLogger( TokenProvider.class );

        String authTokenPath;

        public TokenProvider( String authTokenPath ) {
            this.authTokenPath = authTokenPath;
        }

        @Override
        public Mono<RedisCredentials> resolveCredentials() {
            return Mono.just( new RedisCredentials() {

                @Override
                public boolean hasUsername() {
                    return false;
                }

                @Override
                public boolean hasPassword() {
                    return true;
                }

                @Override
                public String getUsername() {
                    return null;
                }

                @Override
                public char[] getPassword() {
                    return getAuthToken().toCharArray();
                }
            } );
        }

        protected String getAuthToken() {
            LOGGER.info( "TokenProvider reading authToken!!!" );
            Thread.dumpStack();
            try {
                Path path = Paths.get( authTokenPath );
                return Files.readAllLines( path ).get( 0 );
            } catch( IOException e ) {
                LOGGER.error( "Could not read credentials for redis from file system; returning an empty token." );
                return "";
            }
        }
    }
}

For now, we're wiring this into our delegate, which invokes redisConnection() to get a connection and hold as an attribute (still a holdover from when we were trying to use a pool .... and would get a connection from the pool instead). Our system guys have set up redis to rotate the password and write the password/token to a file every hour so our RedisCredentialsProvider can read the file and set it appropriately. And yes, we're not tidy on thread safety yet --- but we will be eventually.

If there is anything you see in here that we are doing wrong, please let us know. There is very little I could find out there how folks are handling rotating passwords from the client perspective.

[doktorjw]

Okay ... platform folks messed things up a bit and weren't actually writing the new creds to the credentials file. :( go figure; a permissions error they missed.

This does seem to be working just fine now.

Thx for the help!