No reachable node in cluster exception (JedisNoReachableClusterNodeException) in Spring boot with AWS Elasticache - spring-boot

Question

I have a spring-boot application which I am trying to connect with AWS Elasticache Redis for caching solution. I did create a Redis cluster in AWS and I deployed my application in an EC2 instance with proper IAM role to access the AWS Elasticache. I came across couple of issues and I was able to solve but this redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster, I couldn't solve. I am adding all the necessary information below for the issue.
POM file
I am using the spring version 2.6.0
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.modelmapper</groupId>
<artifactId>modelmapper</artifactId>
<version>2.3.9</version>
</dependency>
<dependency>
<groupId>jakarta.validation</groupId>
<artifactId>jakarta.validation-api</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
</dependency>
</dependencies>
Redis configuration
#Configuration
#EnableCaching
public class RedisConfiguration {
List<String> clusterNodes =
Arrays.asList(
"xxx-redis-cache-0001-001.h82rl2.0001.use5.cache.amazonaws.com:6379",
"xxx-redis-cache-0001-002.h82rl2.0001.use5.cache.amazonaws.com:6379",
"xxx-redis-cache-0001-003.h82rl2.0001.use5.cache.amazonaws.com:6379");
#Bean
RedisConnectionFactory connectionFactory() {
JedisConnectionFactory factory = new JedisConnectionFactory(new RedisClusterConfiguration(clusterNodes));
factory.setTimeout(10000);
factory.setUsePool(true);
JedisPoolConfig pool = new JedisPoolConfig();
pool.setMaxIdle(30);
pool.setMinIdle(10);
factory.setPoolConfig(pool);
return factory;
}
#Bean(value = "redisTemplate")
public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory factory) {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setDefaultSerializer(new GenericJackson2JsonRedisSerializer());
template.setKeySerializer(new StringRedisSerializer());
template.setHashKeySerializer(new StringRedisSerializer());
template.setConnectionFactory(factory);
return template;
}
#Primary
#Bean(name = "cacheManager1Hr") // Default cache manager is 1 hr
public CacheManager cacheManager(RedisConnectionFactory redisConnectionFactory) {
Duration expiration = Duration.ofHours(1);
RedisCacheManager redisCacheManager =
RedisCacheManager.builder(redisConnectionFactory)
.cacheDefaults(
RedisCacheConfiguration.defaultCacheConfig()
.disableCachingNullValues()
.serializeValuesWith(
RedisSerializationContext.SerializationPair.fromSerializer(
new GenericJackson2JsonRedisSerializer()))
.entryTtl(expiration))
.build();
redisCacheManager.setTransactionAware(false);
return redisCacheManager;
}
#Bean(name = "cacheManager1Minutes")
public CacheManager cacheManager1Minutes(RedisConnectionFactory redisConnectionFactory) {
Duration expiration = Duration.ofMinutes(1);
RedisCacheManager redisCacheManager =
RedisCacheManager.builder(redisConnectionFactory)
.cacheDefaults(
RedisCacheConfiguration.defaultCacheConfig()
.disableCachingNullValues()
.serializeValuesWith(
RedisSerializationContext.SerializationPair.fromSerializer(
new GenericJackson2JsonRedisSerializer()))
.entryTtl(expiration))
.build();
redisCacheManager.setTransactionAware(false);
return redisCacheManager;
}
}
Controller
#GetMapping(path = "/{id}")
#Cacheable(value = "xxx-redis-cache", key = "{#theUserId}", cacheManager = "cacheManager1Hr")
public User getBookSecondWay(#PathVariable("id") long theUserId) {
printLog("User by ID: " + theUserId);
//return userService.findByID(theUserId);
try {
Thread.sleep(1000);
} catch (InterruptedException exception) {
}
return userList.stream().filter(user -> user.getId() == theUserId).collect(Collectors.toList()).get(0);
}
Redis cluster
Exception
2021-11-25 19:41:48.231 ERROR 3477 --- [p-nio-80-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster] with root cause
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:129) ~[jedis-3.7.0.jar!/:na]
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:151) ~[jedis-3.7.0.jar!/:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:118) ~[jedis-3.7.0.jar!/:na]
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:69) ~[jedis-3.7.0.jar!/:na]
at redis.clients.jedis.BinaryJedisCluster.get(BinaryJedisCluster.java:235) ~[jedis-3.7.0.jar!/:na]
at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.get(JedisClusterStringCommands.java:66) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:278) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.lambda$get$1(DefaultRedisCacheWriter.java:130) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:308) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.get(DefaultRedisCacheWriter.java:130) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.data.redis.cache.RedisCache.lookup(RedisCache.java:89) ~[spring-data-redis-2.6.0.jar!/:2.6.0]
at org.springframework.cache.support.AbstractValueAdaptingCache.get(AbstractValueAdaptingCache.java:58) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:73) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:571) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:536) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:402) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64) ~[spring-context-5.3.13.jar!/:5.3.13]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.13.jar!/:5.3.13]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar!/:5.3.13]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.13.jar!/:5.3.13]
at com.cache.caching.controller.UserController$$EnhancerBySpringCGLIB$$b00f093.getBookSecondWay(<generated>) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.13.jar!/:5.3.13]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar!/:5.3.13]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.55.jar!/:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.55.jar!/:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
If any additional information is needed please inform me and I will add. Thank you!

Answer 1

I figure out the issue. In-fact it was Redis cluster's security group. The security group allows all traffic from default security group of the VPC but not the security group of my EC2 instance. Once I did change it to the EC2 security group, it works fine.