spring boot with redis - spring

I worked with spring boot and redis to caching.I can cache my data that fetch from database(oracle) use #Cacheable(key = "{#input,#page,#size}",value = "on_test").
when i try to fetch data from key("on_test::0,0,10") with redisTemplate the result is 0
why??
Redis Config:
#Configuration
public class RedisConfig {
#Bean
JedisConnectionFactory jedisConnectionFactory() {
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration("localhost", 6379);
redisStandaloneConfiguration.setPassword(RedisPassword.of("admin#123"));
return new JedisConnectionFactory(redisStandaloneConfiguration);
}
#Bean
public RedisTemplate<String,Objects> redisTemplate() {
RedisTemplate<String,Objects> template = new RedisTemplate<>();
template.setStringSerializer(new StringRedisSerializer());
template.setValueSerializer(new StringRedisSerializer());
template.setConnectionFactory(jedisConnectionFactory());
return template;
}
//service
#Override
#Cacheable(key = "{#input,#page,#size}",value = "on_test")
public Page<?> getAllByZikaConfirmedClinicIs(Integer input,int page,int size) {
try {
Pageable newPage = PageRequest.of(page, size);
String fromCache = controlledCacheService.getFromCache();
if (fromCache == null && input!=null) {
log.info("cache is empty lets initials it!!!");
Page<DataSet> all = dataSetRepository.getAllByZikaConfirmedClinicIs(input,newPage);
List<DataSet> d = redisTemplate.opsForHash().values("on_test::0,0,10");
System.out.print(d);
return all;
}
return null;

The whole point of using #Cacheable is that you don't need to be using RedisTemplate directly. You just need to call getAllByZikaConfirmedClinicIs() (from outside of the class it is defined in) and Spring will automatically check first if a cached result is available and return that instead of calling the function.
If that's not working, have you annotated one of your Spring Boot configuration classes with #EnableCaching to enable caching?
You might also need to set spring.cache.type=REDIS in application.properties, or spring.cache.type: REDIS in application.yml to ensure Spring is using Redis and not some other cache provider.

Related

Leverage Spring boot Redis Auto configure logic for RedisConnectionFactory

Spring boot auto configures RedisConnectionFactory if spring-data-redis exists on classpath and RedisConnectionFactory is initialized in LettuceConnectionConfiguration if Lettuce-core available on classpath.
I've only one Redis store as of now, so leveraging Spring boot auto configuration.
Now I'm adding two redis stores, one redis store used as default and other is used when specified with parameter cacheManager = "secondayCacheManager" in #Cacheable annotation so, application should've capability to cache/cache-get on both redis stores.
To configure both Redis Stores, we've to configure both the primary and secondary RedisConnectionFactory and cacheManager using custom configuration. (because spring doesn't auto configure RedisConnectionFactory if it already exists in any custom configuration)
Now the above is custom configuration and missing lot of logic that is happening while configuring RedisConnectionFactory in LettuceConnectionConfiguration.
Auto configure logic for LettuceConnectionConfiguration is package private so, cannot be called directly from custom configuration.
We would like to leverage the auto configure logic in
LettuceConnectionConfiguration while configuring the custom
RedisConnectionFactory for both primary and secondary redis caches.
Is there a way to achieve this?
Reason being we would like keep the redis connection configurations as it is done by spring boot auto configure.
Currently using below code to configure both the primary and secondary RedisConnectionFactory with Pool configuration and some code copy pasted from LettuceConnectionConfiguration class.
public static LettuceConnectionFactory buildLettuceConnectionFactory(RedisProperties properties, ClientResources clientResources) {
RedisStandaloneConfiguration standaloneConfiguration = new RedisStandaloneConfiguration(properties.getHost(), properties.getPort());
standaloneConfiguration.setDatabase(properties.getDatabase());
if (properties.getPassword() != null) {
standaloneConfiguration.setPassword(RedisPassword.of(properties.getPassword()));
}
if (properties.getUsername() != null) {
standaloneConfiguration.setUsername(properties.getUsername());
}
LettucePoolingClientConfiguration poolingClientConfiguration = LettucePoolingClientConfiguration.builder()
.poolConfig(buildGenericObjectPoolConfig(properties))
.shutdownTimeout(properties.getLettuce().getShutdownTimeout())
.clientOptions(createClientOptions(properties))
.clientResources(clientResources)
.build();
LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(
standaloneConfiguration, poolingClientConfiguration);
lettuceConnectionFactory.afterPropertiesSet();
return lettuceConnectionFactory;
}
private static GenericObjectPoolConfig buildGenericObjectPoolConfig(RedisProperties properties) {
RedisProperties.Pool pool = properties.getLettuce().getPool();
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
if (Objects.nonNull(pool)) {
poolConfig.setMaxIdle(pool.getMaxIdle());
poolConfig.setMinIdle(pool.getMinIdle());
poolConfig.setMaxTotal(pool.getMaxActive());
poolConfig.setMaxWaitMillis(pool.getMaxWait().toMillis());
}
return poolConfig;
}
private static ClientOptions createClientOptions(RedisProperties properties) {
ClientOptions.Builder builder = initializeClientOptionsBuilder(properties);
Duration connectTimeout = properties.getConnectTimeout();
if (connectTimeout != null) {
builder.socketOptions(SocketOptions.builder().connectTimeout(connectTimeout).build());
}
return builder.timeoutOptions(TimeoutOptions.enabled()).build();
}
private static ClientOptions.Builder initializeClientOptionsBuilder(RedisProperties properties) {
if (properties.getCluster() != null) {
ClusterClientOptions.Builder builder = ClusterClientOptions.builder();
Refresh refreshProperties = properties.getLettuce().getCluster().getRefresh();
Builder refreshBuilder = ClusterTopologyRefreshOptions.builder()
.dynamicRefreshSources(refreshProperties.isDynamicRefreshSources());
if (refreshProperties.getPeriod() != null) {
refreshBuilder.enablePeriodicRefresh(refreshProperties.getPeriod());
}
if (refreshProperties.isAdaptive()) {
refreshBuilder.enableAllAdaptiveRefreshTriggers();
}
return builder.topologyRefreshOptions(refreshBuilder.build());
}
return ClientOptions.builder();
}

Adding Compression to Spring Data Redis with LettuceConnectionFactory

I see Lettuce can do compression for Redis serialized objects: https://lettuce.io/core/release/reference/#codecs.compression
Any way to set this config within Spring Boot Data LettuceConnectionFactory or in some other bean? I've seen this question asked here as well: https://github.com/lettuce-io/lettuce-core/issues/633
I'd like to compress all serialized objects being sent to Redis to reduce network traffic between boxes.
Thanks
I ended up solving it the following way.
Create a RedisTemplate Spring Bean. This allows us to set a custom serializer.
#Bean
public RedisTemplate<Object, Object> redisTemplate(LettuceConnectionFactory connectionFactory) {
RedisTemplate<Object, Object> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory);
// Set a custom serializer that will compress/decompress data to/from redis
RedisSerializerGzip serializerGzip = new RedisSerializerGzip();
template.setValueSerializer(serializerGzip);
template.setHashValueSerializer(serializerGzip);
return template;
}
Create a custom serializer. I decided to extend JdkSerializationRedisSerializer since that's what Spring was using by default for Redis. I added a compress/decompress in each respected method and use the super class serialization code.
public class RedisSerializerGzip extends JdkSerializationRedisSerializer {
#Override
public Object deserialize(byte[] bytes) {
return super.deserialize(decompress(bytes));
}
#Override
public byte[] serialize(Object object) {
return compress(super.serialize(object));
}
////////////////////////
// Helpers
////////////////////////
private byte[] compress(byte[] content) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream)) {
gzipOutputStream.write(content);
} catch (IOException e) {
throw new SerializationException("Unable to compress data", e);
}
return byteArrayOutputStream.toByteArray();
}
private byte[] decompress(byte[] contentBytes) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
try {
IOUtils.copy(new GZIPInputStream(new ByteArrayInputStream(contentBytes)), out);
} catch (IOException e) {
throw new SerializationException("Unable to decompress data", e);
}
return out.toByteArray();
}
}
Use spring dependency injection for the RedisTemplate Bean.
Here is a sample of a Java Spring Boot Redis Cluster Data configuration.
It is an implementation with Redis Cluster and Redis Cache Manager.
Snappy Compression
Kryo Serialization
Support ttl per cache key
Gradle configuration
spring-data-redis
snappy-java
kryo
commons-codec
Link to github https://github.com/cboursinos/java-spring-redis-compression-snappy-kryo

use spring boot data redis Connect to the redis cluster problem

I used spring boot data redis to connect to the redis cluster, using version 2.1.3 The configuration is as follows:
#Bean
#Primary
public RedisConnectionFactory myLettuceConnectionFactory(GenericObjectPoolConfig poolConfig) {
RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration();
final List<String> nodeList = redisProperties.getCluster().getNodes();
Set<RedisNode> nodes = new HashSet<RedisNode>();
for (String ipPort : nodeList) {
String[] ipAndPort = ipPort.split(":");
nodes.add(new RedisNode(ipAndPort[0].trim(), Integer.valueOf(ipAndPort[1])));
}
redisClusterConfiguration.setPassword(RedisPassword.of(redisProperties.getPassword()));
redisClusterConfiguration.setClusterNodes(nodes);
redisClusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());
LettuceClientConfiguration clientConfig = LettucePoolingClientConfiguration.builder()
.commandTimeout(redisProperties.getTimeout())
.poolConfig(poolConfig)
.build();
RedisClusterClient clusterClient ;
LettuceConnectionFactory factory = new LettuceConnectionFactory(redisClusterConfiguration,clientConfig);
return factory;
}
However, during the operation, a WARN exception message will always be received as follows:
Well, this seems to be a problem with lettuce, How to map remote host & port to localhost using Lettuce,but I don't know how to use it in spring boot data redis. Any solution is welcome, thank you
I've got the answer, so let's define a ClinentRourse like this:
MappingSocketAddressResolver resolver = MappingSocketAddressResolver.create(DnsResolvers.UNRESOLVED ,
hostAndPort -> {
if(hostAndPort.getHostText().startsWith("172.31")){
return HostAndPort.of(ipStr, hostAndPort.getPort());
}
return hostAndPort;
});
ClientResources clientResources = ClientResources.builder()
.socketAddressResolver(resolver)
.build();
Then through LettuceClientConfiguration.clientResources method set in, the normal work of the lettuce.

how to configure different ttl for each redis cache when using #cacheable in springboot2.0

I am using #cacheable in springboot2.0 with redis. I have configured RedisCacheManager as follow:
#Bean
public RedisCacheManager redisCacheManager(RedisConnectionFactory connectionFactory) {
RedisCacheWriter redisCacheWriter = RedisCacheWriter.lockingRedisCacheWriter(connectionFactory);
SerializationPair<Object> valueSerializationPair = RedisSerializationContext.SerializationPair
.fromSerializer(new GenericJackson2JsonRedisSerializer());
RedisCacheConfiguration cacheConfiguration = RedisCacheConfiguration.defaultCacheConfig();
cacheConfiguration = cacheConfiguration.serializeValuesWith(valueSerializationPair);
cacheConfiguration = cacheConfiguration.prefixKeysWith("myPrefix");
cacheConfiguration = cacheConfiguration.entryTtl(Duration.ofSeconds(30));
RedisCacheManager redisCacheManager = new RedisCacheManager(redisCacheWriter, cacheConfiguration);
return redisCacheManager;
}
but this make all key's ttl 30 second, how to configure different ttl for each redis cache with different cachename?
You can configure different expire time for each cache using only one CacheManager by creating different configurations for each cache and put them in a map with which you create the CacheManager.
For example:
#Bean
RedisCacheWriter redisCacheWriter() {
return RedisCacheWriter.lockingRedisCacheWriter(jedisConnectionFactory());
}
#Bean
RedisCacheConfiguration defaultRedisCacheConfiguration() {
return RedisCacheConfiguration.defaultCacheConfig().entryTtl(Duration.ofSeconds(defaultCacheExpiration));
}
#Bean
CacheManager cacheManager() {
Map<String, RedisCacheConfiguration> cacheNamesConfigurationMap = new HashMap<>();
cacheNamesConfigurationMap.put("cacheName1", RedisCacheConfiguration.defaultCacheConfig().entryTtl(Duration.ofSeconds(ttl1)));
cacheNamesConfigurationMap.put("cacheName2", RedisCacheConfiguration.defaultCacheConfig().entryTtl(Duration.ofSeconds(ttl2)));
cacheNamesConfigurationMap.put("cacheName3", RedisCacheConfiguration.defaultCacheConfig().entryTtl(Duration.ofSeconds(ttl3)));
return new RedisCacheManager(redisCacheWriter(), defaultRedisCacheConfiguration(), cacheNamesConfigurationMap);
}
If you need configure different expire time for cache when using #cacheable ,
you can configure different CacheManager with different ttl,and specify cacheManager when using cache in your service.
#Cacheable(cacheManager = "expireOneHour", value = "onehour", key = "'_onehour_'+#key", sync = true)
Here is how you can define multiple Redis based caches with different TTL and maxIdleTime using Redisson Java client:
#Bean(destroyMethod="shutdown")
RedissonClient redisson() throws IOException {
Config config = new Config();
config.useClusterServers()
.addNodeAddress("redis://127.0.0.1:7004", "redis://127.0.0.1:7001");
return Redisson.create(config);
}
#Bean
CacheManager cacheManager(RedissonClient redissonClient) {
Map<String, CacheConfig> config = new HashMap<String, CacheConfig>();
// create "myCache1" cache with ttl = 20 minutes and maxIdleTime = 12 minutes
config.put("myCache", new CacheConfig(24*60*1000, 12*60*1000));
// create "myCache2" cache with ttl = 35 minutes and maxIdleTime = 24 minutes
config.put("myCache2", new CacheConfig(35*60*1000, 24*60*1000));
return new RedissonSpringCacheManager(redissonClient, config);
}
This is my code:
The shared config in common module
#Bean
RedisCacheManagerBuilderCustomizer redisCacheManagerBuilderCustomizer(List<RedisTtlConfig> ttlConfigs) {
RedisCacheConfiguration defaultCacheConfig = RedisCacheConfiguration.defaultCacheConfig();
return (builder) -> {
Map<String, RedisCacheConfiguration> ttlConfigMap = new HashMap<>();
ttlConfigs.forEach( config -> {
config.forEach( (key, ttl) -> {
ttlConfigMap.put(key, defaultCacheConfig.entryTtl(Duration.ofSeconds(ttl)));
});
});
builder.withInitialCacheConfigurations(ttlConfigMap);
builder.cacheDefaults(defaultCacheConfig);
};
}
A custom class to collect ttl config by key
public class RedisTtlConfig extends HashMap<String, Long> {
public RedisTtlConfig setTTL(String key, Long ttl){
this.put(key, ttl);
return this;
}
}
3.Simple ttl config code in ref module
#Bean
RedisTtlConfig corpCacheTtlConfig(){
return new RedisTtlConfig()
.setTTL("test1", 300l)
.setTTL("test2", 300l);
}

Spring, autowire #Value from a database

I am using a properties File to store some configuration properties, that are accessed this way:
#Value("#{configuration.path_file}")
private String pathFile;
Is it possible (with Spring 3) to use the same #Value annotation, but loading the properties from a database instead of a file ?
Assuming you have a table in your database stored key/value pairs:
Define a new bean "applicationProperties" - psuedo-code follows...
public class ApplicationProperties {
#AutoWired
private DataSource datasource;
public getPropertyValue(String key) {
// transact on your datasource here to fetch value for key
// SNIPPED
}
}
Inject this bean where required in your application. If you already have a dao/service layer then you would just make use of that.
Yes, you can keep your #Value annotation, and use the database source with the help of EnvironmentPostProcessor.
As of Spring Boot 1.3, we're able to use the EnvironmentPostProcessor to customize the application's Environment before application context is refreshed.
For example, create a class which implements EnvironmentPostProcessor:
public class ReadDbPropertiesPostProcessor implements EnvironmentPostProcessor {
private static final String PROPERTY_SOURCE_NAME = "databaseProperties";
private String[] CONFIGS = {
"app.version"
// list your properties here
};
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
Map<String, Object> propertySource = new HashMap<>();
try {
// the following db connections properties must be defined in application.properties
DataSource ds = DataSourceBuilder
.create()
.username(environment.getProperty("spring.datasource.username"))
.password(environment.getProperty("spring.datasource.password"))
.url(environment.getProperty("spring.datasource.url"))
.driverClassName("com.mysql.jdbc.Driver")
.build();
try (Connection connection = ds.getConnection();
// suppose you have a config table storing the properties name/value pair
PreparedStatement preparedStatement = connection.prepareStatement("SELECT value FROM config WHERE name = ?")) {
for (int i = 0; i < CONFIGS.length; i++) {
String configName = CONFIGS[i];
preparedStatement.setString(1, configName);
ResultSet rs = preparedStatement.executeQuery();
while (rs.next()) {
propertySource.put(configName, rs.getString("value"));
}
// rs.close();
preparedStatement.clearParameters();
}
}
environment.getPropertySources().addFirst(new MapPropertySource(PROPERTY_SOURCE_NAME, propertySource));
} catch (Throwable e) {
throw new RuntimeException(e);
}
}
}
Finally, don't forget to put your spring.factories in META-INF. An example:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=
com.baeldung.environmentpostprocessor.autoconfig.PriceCalculationAutoConfig
Although not having used spring 3, I'd assume you can, if you make a bean that reads the properties from the database and exposes them with getters.

Resources