I have a spring cache requirement:
I need to request to a server to get some data and store the results in spring cache. The same request can give me different results every time so I decided to use #cachePut so that every time I can go inside my function and cache gets updated.
#CachePut(value="mycache", key="#url")
public String getData(String url){
try{
// get the data from server
// update the cache
// return data
} catch(){
// return data from cache
}
}
Now there is a twist. If the server is down and I am not able to get the response; I want my data from the cache (stored in previous requests).
If i use #Cacheable, I can't get the updated data. What is the clean way to do this? Something like catching an exception and return the data from cache.
you can get the cache implementation like this and handle the cache.
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.Cache;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CacheConfig;
import org.springframework.cache.annotation.CachePut;
import org.springframework.stereotype.Service;
#Service
#CacheConfig(cacheNames="mycache") // refer to cache/ehcache-xxxx.xml
public class CacheService {
#Autowired private CacheManager manager;
#CachePut(key="#url")
public String getData(String url) {
try {
//do something.
return null;
}catch(Exception e) {
Cache cache = manager.getCache("mycache");
return (String) cache.get(url).get();
}
}
}
Related
I am trying to implement read-only data source in my application.
According to the following repo implementation, this aspect method should be called when a transaction happens but it never triggers this method(This line never printed to the console - System.out.println("Aspect executed");
#Aspect
#Component
#Order(0)
public class TransactionReadonlyAspect {
#Around("#annotation(transactional)")
public Object proceed(ProceedingJoinPoint proceedingJoinPoint, org.springframework.transaction.annotation.Transactional transactional) throws Throwable {
System.out.println("Aspect executed");
try {
if (transactional.readOnly()) {
DatabaseContextHolder.set(DatabaseEnvironment.READONLY);
}
return proceedingJoinPoint.proceed();
} finally {
DatabaseContextHolder.reset();
}
}
}
And also in the following class it initializes the default datasource no matter what,
How can I make this works or what are the other confihgurations I need to add ?
Thanks.
package com.programmingsharing.demoreadwriterouting.conf;
import com.programmingsharing.demoreadwriterouting.context.DatabaseEnvironment;
import com.programmingsharing.demoreadwriterouting.datasource.MasterSlaveRoutingDataSource;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;
#Configuration
public class DataSourceConfiguration {
#Value("${jdbc.master.url}")
private String mstUrl;
#Value("${jdbc.master.username}")
private String mstUsername;
#Value("${jdbc.master.password}")
private String mstPassword;
#Value("${jdbc.slave.url}")
private String slaveUrl;
#Value("${jdbc.slave.username}")
private String slaveUsername;
#Value("${jdbc.slave.password}")
private String slavePassword;
#Bean
public DataSource dataSource(){
MasterSlaveRoutingDataSource masterSlaveRoutingDataSource = new MasterSlaveRoutingDataSource();
Map<Object, Object> targetDataSources = new HashMap<>();
targetDataSources.put(DatabaseEnvironment.UPDATABLE, masterDataSource());
targetDataSources.put(DatabaseEnvironment.READONLY, slaveDataSource());
masterSlaveRoutingDataSource.setTargetDataSources(targetDataSources);
// Set as all transaction point to master
masterSlaveRoutingDataSource.setDefaultTargetDataSource(masterDataSource());
return masterSlaveRoutingDataSource;
}
public DataSource slaveDataSource() {
HikariDataSource hikariDataSource = new HikariDataSource();
hikariDataSource.setJdbcUrl(slaveUrl);
hikariDataSource.setUsername(slaveUsername);
hikariDataSource.setPassword(slavePassword);
return hikariDataSource;
}
public DataSource masterDataSource() {
HikariDataSource hikariDataSource = new HikariDataSource();
hikariDataSource.setJdbcUrl(mstUrl);
hikariDataSource.setUsername(mstUsername);
hikariDataSource.setPassword(mstPassword);
return hikariDataSource;
}
}
https://programmingsharing.com/routing-read-write-datasource-in-spring-99bcc4468f94
Also
context is always printed null
CONTEXT.get() : null
public class DatabaseContextHolder {
private static final ThreadLocal<DatabaseEnvironment> CONTEXT = new ThreadLocal<>();
public static void set(DatabaseEnvironment databaseEnvironment) {
CONTEXT.set(databaseEnvironment);
}
public static DatabaseEnvironment getEnvironment() {
System.out.println("CONTEXT.get() : " + CONTEXT.get());
return CONTEXT.get();
}
public static void reset() {
CONTEXT.set(DatabaseEnvironment.UPDATABLE);
}
}
Also this is always null, none of the environment variables doe not set
DatabaseContextHolder.getEnvironment() : null
public class MasterSlaveRoutingDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
System.out.println("DatabaseContextHolder.getEnvironment() : " + DatabaseContextHolder.getEnvironment());
return DatabaseContextHolder.getEnvironment();
}
}
That obviously not the answer to your Q, however I would discourage you from using that datasource routing "solution" you are referring to.
The problem is from spring-tx perspective transaction is read-only if and only if the outermost transaction definition is readonly, please check some examples of execution stacks below:
#Transactional(readonly=true)
...
#Transactional(readonly=false)
// current tx is read-only regardless readonly=false definition
#Transactional(readonly=false)
...
#Transactional(readonly=true)
// current tx is not read-only regardless readonly=true definition
"AspectJ" solution does not take into account that spring-tx convention and thus it is basically wrong.
Technically, we may determine whether transaction is read-only or not via calling TransactionSynchronizationManager#isCurrentTransactionReadOnly method, unfortunately that won't help us much because spring-tx may acquire resources (jdbc connection) before marking transaction as read-only, this problem was mentioned by Vlad Mihalcea in Read-write and read-only transaction routing with Spring:
Not only that the hibernate.connection.provider_disables_autocommit allows you to make better use of database connections, but it’s the only way we can make this example work since, without this configuration, the connection is acquired prior to calling the determineCurrentLookupKey method TransactionRoutingDataSource.
There are two options:
if you are using Hibernate - just follow Vlad's recommendations
if you are not using Hibernate you need to take into account that you need to control outermost transaction definitions only - just place there your own annotations/aspects and do not depend on spring-tx stuff.
I need to process data from Rest web service. the following basic exemple is :
import org.springframework.batch.item.ItemReader;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;
import java.util.Arrays;
import java.util.List;
class RESTDataReader implements ItemReader<DataDTO> {
private final String apiUrl;
private final RestTemplate restTemplate;
private int nextDataIndex;
private List<DataDTO> data;
RESTDataReader(String apiUrl, RestTemplate restTemplate) {
this.apiUrl = apiUrl;
this.restTemplate = restTemplate;
nextDataIndex = 0;
}
#Override
public DataDTO read() throws Exception {
if (dataIsNotInitialized()) {
data = fetchDataFromAPI();
}
DataDTO nextData = null;
if (nextDataIndex < data.size()) {
nextData = data.get(nextDataIndex);
nextDataIndex++;
}
else {
nextDataIndex= 0;
data = null;
}
return nextData;
}
private boolean dataIsNotInitialized() {
return this.data == null;
}
private List<DataDTO> fetchDataFromAPI() {
ResponseEntity<DataDTO[]> response = restTemplate.getForEntity(apiUrl,
DataDTO[].class
);
DataDTO[] data= response.getBody();
return Arrays.asList(data);
}
}
However, my fetchDataFromAPI method is called with time slots and it could get more than 20 Millions objects.
For example : if i call it between 01012020 and 01012021 i'll get 80 Millions data.
PS : the web service works by pagination of a single day, i.e. if I want to retrieve the data between 01/09/2020 and 07/09/2020 I have to call it several times (between 01/09-02/09 then between 02/09-03/09 and so on until 06/09-07/09)
My problem in this case is a heap space memory if the data is bulky.
I had to create a step for each month to avoid this problem in my BatchConfiguration (12 steps). The first step which will call the web service between 01/01/2020 and 01/02/2020 etc
Is there a solution to read all this volume of data with only one step before going to the processor ??
Thanks in advance
Since your web service does not provide pagination within a single day, you need to ensure that the process that calls this web service (ie your Spring Batch job) has enough memory to store all items returned by this service.
For example : if i call it between 01012020 and 01012021 i'll get 80 Millions data.
This means that if you call this web service with curl on a machine that does not have enough memory to hold the result, then the curl command will fail. The point I want to make here is that the only way to solve this issue is to give enough memory to the JVM that runs your Spring Batch job to hold such a big result set.
As a side note: if you have control over this web service, I highly recommend you to improve it by introducing a more granular pagination mechanism.
I am trying to cache Kafka Records within 3 minutes of interval post that it will get expired and removed from the cache.
Each incoming records which is fetched using kafka consumer written in springboot needs to be updated in cache first then if it is present i need to discard the next duplicate records if it matches the cache record.
I have tried using Caffeine cache as below,
#EnableCaching
public class AppCacheManagerConfig {
#Bean
public CacheManager cacheManager(Ticker ticker) {
CaffeineCache bookCache = buildCache("declineRecords", ticker, 3);
SimpleCacheManager cacheManager = new SimpleCacheManager();
cacheManager.setCaches(Collections.singletonList(bookCache));
return cacheManager;
}
private CaffeineCache buildCache(String name, Ticker ticker, int minutesToExpire) {
return new CaffeineCache(name, Caffeine.newBuilder().expireAfterWrite(minutesToExpire, TimeUnit.MINUTES)
.maximumSize(100).ticker(ticker).build());
}
#Bean
public Ticker ticker() {
return Ticker.systemTicker();
}
}
and my Kafka Consumer is as below,
#Autowired
CachingServiceImpl cachingService;
#KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'}", concurrency = "#{'${spring.kafka.consumer.concurrentConsumers}'}", errorHandler = "#{'${spring.kafka.consumer.errorHandler}'}")
public void consume(Message<?> message, Acknowledgment acknowledgment,
#Header(KafkaHeaders.RECEIVED_TIMESTAMP) long createTime) {
logger.info("Recieved Message: " + message.getPayload());
try {
boolean approveTopic = false;
boolean duplicateRecord = false;
if (cachingService.isDuplicateCheck(declineRecord)) {
//do something with records
}
else
{
//do something with records
}
cachingService.putInCache(xmlJSONObj, declineRecord, time);
and my caching service is as below,
#Component
public class CachingServiceImpl {
private static final Logger logger = LoggerFactory.getLogger(CachingServiceImpl.class);
#Autowired
CacheManager cacheManager;
#Cacheable(value = "declineRecords", key = "#declineRecord", sync = true)
public String putInCache(JSONObject xmlJSONObj, String declineRecord, String time) {
logger.info("Record is Cached for 3 minutes interval check", declineRecord);
cacheManager.getCache("declineRecords").put(declineRecord, time);
return declineRecord;
}
public boolean isDuplicateCheck(String declineRecord) {
if (null != cacheManager.getCache("declineRecords").get(declineRecord)) {
return true;
}
return false;
}
}
But Each time a record comes in consumer my cache is always empty. Its not holding the records.
Modifications Done:
I have added Configuration file as below after going through the suggestions and more kind of R&D removed some of the earlier logic and now the caching is working as expected but duplicate check is failing when all the three consumers are sending the same records.
`
#Configuration
public class AppCacheManagerConfig {
public static Cache<String, Object> jsonCache =
Caffeine.newBuilder().expireAfterWrite(3, TimeUnit.MINUTES)
.maximumSize(10000).recordStats().build();
#Bean
public CacheLoader<Object, Object> cacheLoader() {
CacheLoader<Object, Object> cacheLoader = new CacheLoader<Object, Object>() {
#Override
public Object load(Object key) throws Exception {
return null;
}
#Override
public Object reload(Object key, Object oldValue) throws Exception {
return oldValue;
}
};
return cacheLoader;
}
`
Now i am using the above cache as manual put and get.
I guess you're trying to implement records deduplication for Kafka.
Here is the similar discussion:
https://github.com/spring-projects/spring-kafka/issues/80
Here is the current abstract class which you may extend to achieve the necessary result:
https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/listener/adapter/AbstractFilteringMessageListener.java
Your caching service is definitely incorrect: Cacheable annotation allows marking the data getters and setters, to add caching through AOP. While in the code you clearly implement some low-level cache updating logic of your own.
At least next possible changes may help you:
Remove #Cacheable. You don't need it because you work with cache manually, so it may be the source of conflicts (especially as soon as you use sync = true). If it helps, remove #EnableCaching as well - it enables support for cache-related Spring annotations which you don't need here.
Try removing Ticker bean with the appropriate parameters for other beans. It should not be harmful as per your configuration, but usually it's helpful only for tests, no need to define it otherwise.
Double-check what is declineRecord. If it's a serialized object, ensure that serialization works properly.
Add recordStats() for cache and output stats() to log for further analysis.
So, I was looking at caching methods in Java (Spring). And Guava looked like it would solve the purpose.
This is the usecase -
I query for some data from a remote service. Kind of configuration field for my application. This field will be used by every inbound request to my application. And it would be expensive to call the remote service everytime as it's kind of constant which changes periodically.
So, on the first request inbound to my application, when I call remote service, I would cache the value. I set an expiry time of this cache as 30 mins. After 30 mins when the cache is expired and there is a request to retrieve the key, I would like a callback or something to do the operation of calling the remote service and setting the cache and return the value for that key.
How can I do it in Guava cache?
Here i give a example how to use guava cache. If you want to handle removal listener then need to call cleanUp. Here i run a thread which one call clean up every 30 minutes.
import com.google.common.cache.*;
import org.springframework.stereotype.Component;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
#Component
public class Cache {
public static LoadingCache<String, String> REQUIRED_CACHE;
public Cache(){
RemovalListener<String,String> REMOVAL_LISTENER = new RemovalListener<String, String>() {
#Override
public void onRemoval(RemovalNotification<String, String> notification) {
if(notification.getCause() == RemovalCause.EXPIRED){
//do as per your requirement
}
}
};
CacheLoader<String,String> LOADER = new CacheLoader<String, String>() {
#Override
public String load(String key) throws Exception {
return null; // return as per your requirement. if key value is not found
}
};
REQUIRED_CACHE = CacheBuilder.newBuilder().maximumSize(100000000)
.expireAfterWrite(30, TimeUnit.MINUTES)
.removalListener(REMOVAL_LISTENER)
.build(LOADER);
Executors.newSingleThreadExecutor().submit(()->{
while (true) {
REQUIRED_CACHE.cleanUp(); // need to call clean up for removal listener
TimeUnit.MINUTES.sleep(30L);
}
});
}
}
put & get data:
Cache.REQUIRED_CACHE.get("key");
Cache.REQUIRED_CACHE.put("key","value");
I am dealing with a custom implementation for wicket session store, data store, page store. I have cu cluster wicket and make it work in the following situation:
There are 2 nodes in the cluster, node one fails and the user should be able to continue the flow without noticing, the pages a statefull, with a lot of ajax requests. For now I'm storing the wicket session in a custom storage over rmi, and I'm trying to extend the DiskPageStore. The new challenge is SessionEntry inner class, it is still hold by a ConcurrentMap.
My question is: Has anyone done this before? Do you have any suggestions on how to accomplish this?
My suggestion is forget about DiskPageStore and SessionEntry in your situation. The ConcurrentMap you mentioned is held in the heap locally. Once one of the nodes fails, there is no way to get access to its ConcurrentMap, and Wicket resources referred to from the ConcurrentMap will be impossible to be released.
Therefore, in a clustered environment, you need to cluster the Wicket page store. Page versions can be expired based on certain policy, or deliberately removed when their corresponding session expires.
I've enabled web session and data store clustering for Apache Wicket used in an enterprise web application in production, and it has been working very well. The software I use are:
JDK 1.8.0_60
Apache Tomcat 8.0.33 (Tomcat 7 works too)
Wicket 6.16 (versions 6.22.0 and 7.2.0 should also work)
Apache Ignite 1.7.0
Load balancer: Crossroads
Ubuntu 14.04.1
The idea is use Apache Ignite for web session clustering, and it is pretty straightforward following its instructions for Web Session Clustering.
Once I got the web session clustered, I then put the data store (which includes the page store already) into the Ignite distributed data grid, while at the same time I disabled the Wicket application scoped cache (so as to make sure all data is clustered). Take a look at the documentation on Wicket's page store to find out how to configure the data store.
Alternatively you should be able to use Wicket HttpSessionDataStore to put the data store into the session. As the session is clustered, the data store is clustered automatically. But this approach does not work compatibly with Apache Ignite for me. So I use my own implementation of the IDataStore interface, which puts the data store into the Ignite distributed data grid. See below the implementation.
import java.util.concurrent.TimeUnit;
import javax.cache.expiry.Duration;
import javax.cache.expiry.TouchedExpiryPolicy;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMemoryMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.wicket.pageStore.IDataStore;
import org.apache.wicket.pageStore.memory.IDataStoreEvictionStrategy;
import org.apache.wicket.pageStore.memory.PageTable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class IgniteDataStore implements IDataStore {
private static final Logger log = LoggerFactory.getLogger(IgniteDataStore.class);
private final IDataStoreEvictionStrategy evictionStrategy;
private Ignite ignite;
IgniteCache<String, PageTable> igniteCache;
public IgniteDataStore(IDataStoreEvictionStrategy evictionStrategy) {
this.evictionStrategy = evictionStrategy;
CacheConfiguration<String, PageTable> cacheCfg = new CacheConfiguration<String, PageTable>("wicket-data-store");
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(1);
cacheCfg.setMemoryMode(CacheMemoryMode.OFFHEAP_VALUES);
cacheCfg.setOffHeapMaxMemory(2 * 1024L * 1024L * 1024L); // 2 Gigabytes.
cacheCfg.setEvictionPolicy(new LruEvictionPolicy<String, PageTable>(10000));
cacheCfg.setExpiryPolicyFactory(TouchedExpiryPolicy.factoryOf(new Duration(TimeUnit.SECONDS, 14400)));
log.info("IgniteDataStore timeout is set to 14400 seconds.");
ignite = Ignition.ignite();
igniteCache = ignite.getOrCreateCache(cacheCfg);
}
#Override
public synchronized byte[] getData(String sessionId, int id) {
PageTable pageTable = getPageTable(sessionId, false);
byte[] pageAsBytes = null;
if (pageTable != null) {
pageAsBytes = pageTable.getPage(id);
}
return pageAsBytes;
}
#Override
public synchronized void removeData(String sessionId, int id) {
PageTable pageTable = getPageTable(sessionId, false);
if (pageTable != null) {
pageTable.removePage(id);
}
}
#Override
public synchronized void removeData(String sessionId) {
PageTable pageTable = getPageTable(sessionId, false);
if (pageTable != null) {
pageTable.clear();
}
igniteCache.remove(sessionId);
}
#Override
public synchronized void storeData(String sessionId, int id, byte[] data) {
PageTable pageTable = getPageTable(sessionId, true);
if (pageTable != null) {
pageTable.storePage(id, data);
evictionStrategy.evict(pageTable);
igniteCache.put(sessionId, pageTable);
} else {
log.error("Cannot store the data for page with id '{}' in session with id '{}'", id, sessionId);
}
}
#Override
public synchronized void destroy() {
igniteCache.clear();
}
#Override
public boolean isReplicated() {
return true;
}
#Override
public boolean canBeAsynchronous() {
return false;
}
private PageTable getPageTable(String sessionId, boolean create) {
if (igniteCache.containsKey(sessionId)) {
return igniteCache.get(sessionId);
}
if (!create) {
return null;
}
PageTable pageTable = new PageTable();
igniteCache.put(sessionId, pageTable);
return pageTable;
}
}
Hope it helps.