How to capture Redis connection failure on Spring Boot Redis Session implementation? - spring

I have implemented Redis session management using LettuceConnectionFactory on my Spring Boot java application. I couldn't figure out a way to capture Redis connection failure. Is there a way to capture the connection failure?
Spring Boot version: 2.2.6
Code:
#Configuration
#PropertySource("classpath:application.properties")
#EnableRedisHttpSession
public class HttpSessionConfig extends AbstractHttpSessionApplicationInitializer {
Logger logger = LoggerFactory.getLogger(HttpSessionConfig.class);
#Value("${spring.redis.host}")
private String redisHostName;
#Value("${spring.redis.port}")
private int redisPort;
#Value("${spring.redis.password}")
private String redisPassword;
private #Value("${spring.redis.custom.command.timeout}")
Duration redisCommandTimeout;
private #Value("${spring.redis.timeout}")
Duration socketTimeout;
#Bean
LettuceConnectionFactory lettuceConnectionFactory() {
final SocketOptions socketOptions = SocketOptions.builder().connectTimeout(socketTimeout).build();
final ClientOptions clientOptions = ClientOptions.builder()
.socketOptions(socketOptions)
.build();
LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
.commandTimeout(redisCommandTimeout)
.clientOptions(clientOptions)
.readFrom(ReadFrom.SLAVE_PREFERRED)
.build();
RedisStandaloneConfiguration serverConfig = new RedisStandaloneConfiguration(redisHostName,
redisPort);
final LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(serverConfig,
clientConfig);
return lettuceConnectionFactory;
}
#Bean
public RedisTemplate<Object, Object> sessionRedisTemplate() {
final RedisTemplate<Object, Object> template = new RedisTemplate<>();
template.setConnectionFactory(lettuceConnectionFactory());
return template;
}
#Bean
public ConfigureRedisAction configureRedisAction() {
return ConfigureRedisAction.NO_OP;
}
}

Related

Redis Cache Pool: How to validate if redis pooling configuration is working?

Folks,
I have a redis cache connection for my application. Which recently got stalled due to connections increase. So I have implimented connection pooling for my redis connection in my application. The configuration is as below:
#Configuration
public class CacheConfig {
#Value("${cache.host.name}")
private String cacheHostName;
#Value("${cache.database.number}")
private Integer database;
#Value("${cache.port.number}")
private Integer portNumber;
#Value("${cache.pool.max-total}")
private Integer maxTotal;
#Value("${cache.pool.max-idle}")
private Integer maxIdle;
#Value("${cache.pool.min-idle}")
private Integer minIdle;
#Bean
JedisConnectionFactory jedisConnectionFactory() {
RedisStandaloneConfiguration redisConfig =
new RedisStandaloneConfiguration(cacheHostName, portNumber);
redisConfig.setDatabase(database);
JedisConnectionFactory redisConnectionFactory =
new JedisConnectionFactory(redisConfig, jedisClientConfiguration());
return redisConnectionFactory;
}
#Bean
JedisClientConfiguration jedisClientConfiguration() {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); // GenericObjectPoolConfig
jedisPoolConfig.setMaxTotal(maxTotal);
jedisPoolConfig.setMaxIdle(maxIdle);
jedisPoolConfig.setMinIdle(minIdle);
return JedisClientConfiguration.builder().usePooling().poolConfig(jedisPoolConfig).build();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(jedisConnectionFactory());
return template;
}
}
But I'm not able to validate if my configuration is actually in effect. Any suggestion to validate this is appreciated!

Spring's #Cacheable unable to cache data - Redis implementation

I am trying to use spring cache abstraction with Redis cache. I am unable to see the values in cache. Please help me if I am missing something in config :
As I am making the call multiple times actual fetch is happening. I tried connecting to same redis host port, I cant find there any keys as well.
PFB the implementation details.
CacheUtils.java :
#Slf4j
public class CacheUtils {
private final CustomerManagementClient customerManagementClient;
#Autowired
public CacheUtils(CustomerManagementClient customerManagementClient) {
this.customerManagementClient = customerManagementClient;
}
#Cacheable(value = "merchant-details", key = "#merchantEntityId")
public MerchantDetails getOrFetchMerchantDetails(OrderItemStatusChangeEvent event, MerchantType merchantType, String merchantEntityId) {
if (BUYER == merchantType) {
log.info("test - get buyer details");
CustomerDetails customerDetails =
customerManagementClient.getData(merchantEntityId);
String businessId = customerDetails.getBusinessId();
String phoneNumber = customerDetails.getPhoneNumber();
return MerchantDetails
.builder()
.merchantEntityId(merchantEntityId)
.businessId(businessId)
.businessName(customerDetails.getBusinessName())
.merchantType(merchantType)
.contactNumber(phoneNumber)
.build();
}
throw new InvalidInputException();
}
}
MainClass.java
#Slf4j
#Component
public class MainClass implements LogisticsPlanningService {
private final CacheUtils cacheUtils;
#Autowired
public LogisticsPlanningServiceImpl(CacheUtils cacheUtils) {
this.cacheUtils = cacheUtils;
}
private Set<LogisticsPlanningRequest> testMethod(Event event) {
MerchantDetails senderDetails = cacheUtils.getOrFetchMerchantDetails(event, SELLER, orderItem.getSellerId());
MerchantDetails receiverDetails = cacheUtils.getOrFetchMerchantDetails(event, BUYER, orderItem.getBuyerId());
}
}
RedisConfiguration.java
#Configuration
#EnableCaching
public class RedisConfiguration {
private String hostName;
private int port;
#Autowired
MarketPlaceServiceProperties properties;
#PostConstruct
public void init() {
hostName = properties.getRedisHostName();
port = Integer.parseInt(properties.getRedisPort());
}
#Bean
protected JedisConnectionFactory jedisConnectionFactory() {
RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration(hostName, port);
JedisConnectionFactory factory = new JedisConnectionFactory(configuration);
factory.afterPropertiesSet();
return factory;
}
public RedisCacheConfiguration getTestCacheConfig() {
RedisCacheConfiguration cacheConfiguration = RedisCacheConfiguration.defaultCacheConfig();
cacheConfiguration.prefixCacheNameWith("marketplace");
cacheConfiguration.disableCachingNullValues();
return cacheConfiguration;
}
// #Bean
// public RedisTemplate<String, Object> redisTemplate() {
// final RedisTemplate<String, Object> redisTemplate = new RedisTemplate<String, Object>();
// redisTemplate.setKeySerializer(new StringRedisSerializer());
// redisTemplate.setHashKeySerializer(new GenericToStringSerializer<>(Object.class));
// redisTemplate.setHashValueSerializer(new JdkSerializationRedisSerializer());
// redisTemplate.setValueSerializer(new JdkSerializationRedisSerializer());
// redisTemplate.setConnectionFactory(jedisConnectionFactory());
// return redisTemplate;
// }
}
service.properties :
redisHostName: redis.domain.prod.xyz.com
redisPort: 5400

Spring boot Redis get null in another project

I have two projects that use Spring boot with Redis. In the first project, I have saved data into Redis cache.
Then, in the second project, when I get data from Redis that I saved in project one it is always null, and project one is also null when I try to get data from Redis that I have saved in project two.
Both projects use the dto as below.
#Data
#EqualsAndHashCode(callSuper = false)
#NoArgsConstructor
#AllArgsConstructor
#JsonInclude(content = Include.NON_NULL)
public class AccountDto implements Serializable {
#JsonIgnore
private static final long serialVersionUID = -2006838697507278668L;
#JsonProperty("account_no")
private String accountNo;
#JsonProperty("account_type")
private String accountType;
#JsonProperty("customer_no")
private String customerNO;
#JsonProperty("entity_code")
private String entityCode;
#JsonProperty("auth_stat")
private String authStat;
#JsonProperty("create_at")
#Temporal(TemporalType.TIMESTAMP)
private Date createAt;
}
Redis configuration
#Configuration
#RefreshScope
#EnableRedisRepositories
public class RedisConfiguration {
#Value("${spring.redis.host: ''}")
private String redisHost;
#Value("${spring.redis.port: ''}")
private int redisPort;
#Bean
public JedisConnectionFactory connectionFactory() {
RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration();
configuration.setHostName(redisHost);
configuration.setPort(redisPort);
return new JedisConnectionFactory(configuration);
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory());
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(new StringRedisSerializer());
return template;
}
}
This class I have created two methods, one for saving and getting data from Redis
#RequiredArgsConstructor
#Service
public class AccountDtoCache {
private static final String HASH_KEY_ACCOUNT = "Account";
private static HashOperations<String, String, AccountDto> hashOperations;
private final RedisTemplate<String, Object> redisTemplate;
#PostConstruct
private synchronized void init() {
hashOperations = redisTemplate.opsForHash();
}
public void save(AccountDto account) {
hashOperations.put(HASH_KEY_ACCOUNT, account.getAccountNo(), account);
}
public static AccountDto getByAccountNo(String account) {
return hashOperations.get(HASH_KEY_ACCOUNT, account);
}
}
I try to get data from Redis as below.
When I debug, it's always null in other projects, but it's working with the same project that I have saved data into Redis.
AccountDtoCache.getByAccountNo("accountNo");
I fixed the issue above with two steps as below
I changed both of the projects package names to the same package name
Redis config
#Bean
public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory redisConnectionFactory) {
RedisTemplate<String, Object> template = new RedisTemplate<>();
GenericJackson2JsonRedisSerializer serializer = new GenericJackson2JsonRedisSerializer();
template.setConnectionFactory(redisConnectionFactory);
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(new StringRedisSerializer());
template.setHashKeySerializer(serializer);
template.setHashValueSerializer(serializer);
return template;
}

How to use "li-apache-kafka-clients" in spring boot app to send large message (above 1MB) from Kafka producer?

How to use li-apache-kafka-clients in spring boot app to send large message (above 1MB) from Kafka producer to Kafka Consumer? Below is the GitHub link of li-apache-kafka-clients:
https://github.com/linkedin/li-apache-kafka-clients
I have imported .jar file of li-apache-kafka-clients and put the below configuration for producer:
props.put("large.message.enabled", "true");
props.put("max.message.segment.bytes", 1000 * 1024);
props.put("segment.serializer", DefaultSegmentSerializer.class.getName());
and for consumer:
message.assembler.buffer.capacity,
max.tracked.messages.per.partition,
exception.on.message.dropped,
segment.deserializer.class
but still getting error for large message. Please help me to solve error.
Below is my code, please let me know where I need to create LiKafkaProducer:
#Configuration
public class KafkaProducerConfig {
#Value("${kafka.boot.server}")
private String kafkaServer;
#Bean
public ProducerFactory<String, String> producerFactory() {
return new DefaultKafkaProducerFactory<>(producerConfig());
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<String, String>(producerFactory());
}
#Bean
public Map<String, Object> producerConfig() {
// TODO Auto-generated method stub
Map<String, Object> config = new HashMap<>();
config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put("bootstrap.servers", "localhost:9092");
config.put("acks", "all");
config.put("retries", 0);
config.put("batch.size", 16384);
config.put("linger.ms", 1);
config.put("buffer.memory", 33554432);
// The following properties are used by LiKafkaProducerImpl
config.put("large.message.enabled", "true");
config.put("max.message.segment.bytes", 1000 * 1024);
config.put("segment.serializer", DefaultSegmentSerializer.class.getName());
config.put("auditor.class", LoggingAuditor.class.getName());
return config;
}
}
#RestController
#RequestMapping("/kafkaProducer")
public class KafkaProducerController {
#Autowired
private KafkaSender sender;
#PostMapping
public ResponseEntity<List<Student>> sendData(#RequestBody List<Student> student){
sender.sendData(student);
return new ResponseEntity<List<Student>>(student, HttpStatus.OK);
}
}
#Service
public class KafkaSender {
private static final Logger LOGGER = LoggerFactory.getLogger(KafkaSender.class);
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
#Value("${kafka.topic.name}")
private String topicName;
public void sendData(List<Student> student) {
// TODO Auto-generated method stub
Map<String, Object> headers = new HashMap<>();
headers.put(KafkaHeaders.TOPIC, topicName);
headers.put("payload", student.get(0));
// Construct a JSONObject from a Map.
JSONObject HeaderObject = new JSONObject(headers);
System.out.println("\nMethod-2: Using new JSONObject() ==> " + HeaderObject);
final String record = HeaderObject.toString();
Message<String> message = MessageBuilder.withPayload(record).setHeader(KafkaHeaders.TOPIC, topicName)
.setHeader(KafkaHeaders.MESSAGE_KEY, "Message")
.build();
kafkaTemplate.send(topicName, message.toString());
}
}
You would need to implement your own ConsumerFactory and ProducerFactory to create the LiKafkaConsumer and LiKafkaProducer respectively.
You should be able to subclass the default factories provided by the framework.

Simple embedded Kafka test example with spring boot

Edit FYI: working gitHub example
I was searching the internet and couldn't find a working and simple example of an embedded Kafka test.
My setup is:
Spring boot
Multiple #KafkaListener with different topics in one class
Embedded Kafka for test which is starting fine
Test with Kafkatemplate which is sending to topic but the
#KafkaListener methods are not receiving anything even after a huge sleep time
No warnings or errors are shown, only info spam from Kafka in logs
Please help me. There are mostly over configured or overengineered examples. I am sure it can be done simple.
Thanks, guys!
#Controller
public class KafkaController {
private static final Logger LOG = getLogger(KafkaController.class);
#KafkaListener(topics = "test.kafka.topic")
public void receiveDunningHead(final String payload) {
LOG.debug("Receiving event with payload [{}]", payload);
//I will do database stuff here which i could check in db for testing
}
}
private static String SENDER_TOPIC = "test.kafka.topic";
#ClassRule
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, SENDER_TOPIC);
#Test
public void testSend() throws InterruptedException, ExecutionException {
Map<String, Object> senderProps = KafkaTestUtils.producerProps(embeddedKafka);
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 0, "message00")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 1, "message01")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 1, 0, "message10")).get();
Thread.sleep(10000);
}
Embedded Kafka tests work for me with below configs,
Annotation on test class
#EnableKafka
#SpringBootTest(classes = {KafkaController.class}) // Specify #KafkaListener class if its not the same class, or not loaded with test config
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false,
brokerProperties = {
"listeners=PLAINTEXT://localhost:3333",
"port=3333"
})
public class KafkaConsumerTest {
#Autowired
KafkaEmbedded kafkaEmbeded;
#Autowired
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
Before annotation for setup method
#Before
public void setUp() throws Exception {
for (MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(messageListenerContainer,
kafkaEmbeded.getPartitionsPerTopic());
}
}
Note: I am not using #ClassRule for creating embedded Kafka rather auto-wiring #Autowired embeddedKafka
#Test
public void testReceive() throws Exception {
kafkaTemplate.send(topic, data);
}
Hope this helps!
Edit: Test configuration class marked with #TestConfiguration
#TestConfiguration
public class TestConfig {
#Bean
public ProducerFactory<String, String> producerFactory() {
return new DefaultKafkaProducerFactory<>(KafkaTestUtils.producerProps(kafkaEmbedded));
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
KafkaTemplate<String, String> kafkaTemplate = new KafkaTemplate<>(producerFactory());
kafkaTemplate.setDefaultTopic(topic);
return kafkaTemplate;
}
Now #Test method will autowire KafkaTemplate and use is to send message
kafkaTemplate.send(topic, data);
Updated answer code block with above line
since the accepted answer doesn't compile or work for me. I find another solution based on https://blog.mimacom.com/testing-apache-kafka-with-spring-boot/ what I would like to share with you.
The dependency is 'spring-kafka-test' version: '2.2.7.RELEASE'
#RunWith(SpringRunner.class)
#EmbeddedKafka(partitions = 1, topics = { "testTopic" })
#SpringBootTest
public class SimpleKafkaTest {
private static final String TEST_TOPIC = "testTopic";
#Autowired
EmbeddedKafkaBroker embeddedKafkaBroker;
#Test
public void testReceivingKafkaEvents() {
Consumer<Integer, String> consumer = configureConsumer();
Producer<Integer, String> producer = configureProducer();
producer.send(new ProducerRecord<>(TEST_TOPIC, 123, "my-test-value"));
ConsumerRecord<Integer, String> singleRecord = KafkaTestUtils.getSingleRecord(consumer, TEST_TOPIC);
assertThat(singleRecord).isNotNull();
assertThat(singleRecord.key()).isEqualTo(123);
assertThat(singleRecord.value()).isEqualTo("my-test-value");
consumer.close();
producer.close();
}
private Consumer<Integer, String> configureConsumer() {
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("testGroup", "true", embeddedKafkaBroker);
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
Consumer<Integer, String> consumer = new DefaultKafkaConsumerFactory<Integer, String>(consumerProps)
.createConsumer();
consumer.subscribe(Collections.singleton(TEST_TOPIC));
return consumer;
}
private Producer<Integer, String> configureProducer() {
Map<String, Object> producerProps = new HashMap<>(KafkaTestUtils.producerProps(embeddedKafkaBroker));
return new DefaultKafkaProducerFactory<Integer, String>(producerProps).createProducer();
}
}
I solved the issue now
#BeforeClass
public static void setUpBeforeClass() {
System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString());
System.setProperty("spring.cloud.stream.kafka.binder.zkNodes", embeddedKafka.getZookeeperConnectionString());
}
while I was debugging, I saw that the embedded kaka server is taking a random port.
I couldn't find the configuration for it, so I am setting the kafka config same as the server. Looks still a bit ugly for me.
I would love to have just the #Mayur mentioned line
#EmbeddedKafka(partitions = 1, controlledShutdown = false, brokerProperties = {"listeners=PLAINTEXT://localhost:9092", "port=9092"})
but can't find the right dependency in the internet.
In integration testing, having fixed ports like 9092 is not recommended because multiple tests should have the flexibility to open their own ports from embedded instances. So, following implementation is something like that,
NB: this implementation is based on junit5(Jupiter:5.7.0) and spring-boot 2.3.4.RELEASE
TestClass:
#EnableKafka
#SpringBootTest(classes = {ConsumerTest.Config.class, Consumer.class})
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false)
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class ConsumerTest {
#Autowired
private EmbeddedKafkaBroker kafkaEmbedded;
#Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
#BeforeAll
public void setUp() throws Exception {
for (final MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(messageListenerContainer,
kafkaEmbedded.getPartitionsPerTopic());
}
}
#Value("${topic.name}")
private String topicName;
#Autowired
private KafkaTemplate<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestKafkaTemplate;
#Test
public void consume_success() {
requestKafkaTemplate.send(topicName, load);
}
#Configuration
#Import({
KafkaListenerConfig.class,
TopicConfig.class
})
public static class Config {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Bean
public ProducerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestProducerFactory() {
final Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestKafkaTemplate() {
return new KafkaTemplate<>(requestProducerFactory());
}
}
}
Listener Class:
#Component
public class Consumer {
#KafkaListener(
topics = "${topic.name}",
containerFactory = "listenerContainerFactory"
)
#Override
public void listener(
final ConsumerRecord<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> consumerRecord,
final #Payload Optional<Map<String, List<ImmutablePair<String, String>>>> payload
) {
}
}
Listner Config:
#Configuration
public class KafkaListenerConfig {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Value(value = "${topic.name}")
private String resolvedTreeQueueName;
#Bean
public ConsumerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> resolvedTreeConsumerFactory() {
final Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put(ConsumerConfig.GROUP_ID_CONFIG, resolvedTreeQueueName);
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new CustomDeserializer());
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> resolvedTreeListenerContainerFactory() {
final ConcurrentKafkaListenerContainerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(resolvedTreeConsumerFactory());
return factory;
}
}
TopicConfig:
#Configuration
public class TopicConfig {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Value(value = "${topic.name}")
private String requestQueue;
#Bean
public KafkaAdmin kafkaAdmin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic requestTopic() {
return new NewTopic(requestQueue, 1, (short) 1);
}
}
application.properties:
spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}
This assignment is the most important assignment that would bind the embedded instance port to the KafkaTemplate and, KafkaListners.
Following the above implementation, you could open dynamic ports per test class and, it would be more convenient.

Resources