I am working on a local classifieds website and right now every time a page loads the database gets queried.
I have noticed that other popular classifieds websites serve a cached version of their site, which would greatly reduce the load time and server load.
How can I achieve this with Spring Boot or Tomcat? I want the website's cache to update every X minutes.
I am using Thymeleaf as my template engine
First you should add org.springframework.boot:spring-boot-starter-cache to your dependencies in build.gradle or pom.xml.
Let's say you're using DataService to get data to feed your view. You can put #Cacheable annotation on it.
#Service
class DataService {
#Cacheable("cache")
String compute() {
return "something"
}
}
Then you should add the following configuration:
#EnableCaching
#Configuration
public class CacheConfiguration {
public static final String CACHE_NAME = "cache";
#Bean
public CacheManager cacheManager() {
ConcurrentMapCacheManager cacheManager = new ConcurrentMapCacheManager(CACHE_NAME);
return cacheManager;
}
#CacheEvict(allEntries = true, value = CACHE_NAME)
#Scheduled(fixedDelay = 10* 60 * 1000 , initialDelay = 500)
public void evictCache() {}
}
Every 10 minutes cache will be cleared.
Related
can anyone suggest any tutorial/sample project for Implementing Snapshot in AXON 4.4.2 with springBoot 2.3.3.
i went through the documentation(https://docs.axoniq.io/reference-guide/axon-framework/tuning/event-snapshots#snapshotting) and did below:
The AxonConfig.class
#Configuration
public class AxonConfig {
#Bean
public SnapshotTriggerDefinition app1SnapshotTrigger(Snapshotter snapshotter) {
return new EventCountSnapshotTriggerDefinition(snapshotter, 10);
}
}
The Aggregate
#Aggregate(snapshotTriggerDefinition = "app1SnapshotTrigger")
public class MyAggregate {
#AggregateIdentifier
private String id;
private String name;
#AggregateMember
private List<Address> addresses = new ArrayList<>();
private MyAggregate () {
}
#CommandHandler
private MyAggregate (CreateNameCommand createNameCommand) {
-----
}
#EventSourcingHandler
private void on(NameCreatedEvent nameCreatedEvent) {
----
}
Am i missing something. Will it create a snapshot at the threshold value 10.
Thanks.
unfortunately we have no sample demo ready to show in this case.
From your code snippet looks that all is in place. Maybe there is some other configuration that is taking over your annotation.
To give a try, I applied your configuration to our https://github.com/AxonIQ/giftcard-demo/
First note that can guide is the following
if you declared a Repository as we did in https://github.com/AxonIQ/giftcard-demo/blob/master/src/main/java/io/axoniq/demo/giftcard/command/GcCommandConfiguration.java#L17 this configuration will take over Annotation placed into your aggregate. If you prefer annotation, you can remove this Bean definition.
Here the piece of code, instead, to have this configured as a Bean
#Bean
public Repository<GiftCard> giftCardRepository(EventStore eventStore, SnapshotTriggerDefinition giftCardSnapshotTrigger) {
return EventSourcingRepository.builder(GiftCard.class)
.snapshotTriggerDefinition(giftCardSnapshotTrigger)
.eventStore(eventStore)
.build();
}
#Bean
public SpringAggregateSnapshotterFactoryBean snapshotter() {
var springAggregateSnapshotterFactoryBean = new SpringAggregateSnapshotterFactoryBean();
//Setting async executors
springAggregateSnapshotterFactoryBean.setExecutor(Executors.newSingleThreadExecutor());
return springAggregateSnapshotterFactoryBean;
}
#Bean("giftCardSnapshotTrigger")
public SnapshotTriggerDefinition giftCardSnapshotTriggerDefinition(Snapshotter snapshotter) {
return new EventCountSnapshotTriggerDefinition(snapshotter, 10);
}
You can check that your snapshot is working fine looking at client log : after 10 events on the same agggregateId, you should find this info log entry
o.a.a.c.event.axon.AxonServerEventStore : Snapshot created
To check you can use the REST api to retrieve the events from an aggregate
curl -X GET "http://localhost:8024/v1/events?aggregateId=A01"
This will produce a stream containing events starting from the latest Snapshot: you will have nine events listed until the tenth event will be processed. After that, the endpoint will list events from the snapshot.
You can also check /actuator/health endpoint: it will shows the last snapshot token if the showDetails is enabled (enabled by default in EE, not enabled by default in SE).
Corrado.
I am trying to cache Kafka Records within 3 minutes of interval post that it will get expired and removed from the cache.
Each incoming records which is fetched using kafka consumer written in springboot needs to be updated in cache first then if it is present i need to discard the next duplicate records if it matches the cache record.
I have tried using Caffeine cache as below,
#EnableCaching
public class AppCacheManagerConfig {
#Bean
public CacheManager cacheManager(Ticker ticker) {
CaffeineCache bookCache = buildCache("declineRecords", ticker, 3);
SimpleCacheManager cacheManager = new SimpleCacheManager();
cacheManager.setCaches(Collections.singletonList(bookCache));
return cacheManager;
}
private CaffeineCache buildCache(String name, Ticker ticker, int minutesToExpire) {
return new CaffeineCache(name, Caffeine.newBuilder().expireAfterWrite(minutesToExpire, TimeUnit.MINUTES)
.maximumSize(100).ticker(ticker).build());
}
#Bean
public Ticker ticker() {
return Ticker.systemTicker();
}
}
and my Kafka Consumer is as below,
#Autowired
CachingServiceImpl cachingService;
#KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'}", concurrency = "#{'${spring.kafka.consumer.concurrentConsumers}'}", errorHandler = "#{'${spring.kafka.consumer.errorHandler}'}")
public void consume(Message<?> message, Acknowledgment acknowledgment,
#Header(KafkaHeaders.RECEIVED_TIMESTAMP) long createTime) {
logger.info("Recieved Message: " + message.getPayload());
try {
boolean approveTopic = false;
boolean duplicateRecord = false;
if (cachingService.isDuplicateCheck(declineRecord)) {
//do something with records
}
else
{
//do something with records
}
cachingService.putInCache(xmlJSONObj, declineRecord, time);
and my caching service is as below,
#Component
public class CachingServiceImpl {
private static final Logger logger = LoggerFactory.getLogger(CachingServiceImpl.class);
#Autowired
CacheManager cacheManager;
#Cacheable(value = "declineRecords", key = "#declineRecord", sync = true)
public String putInCache(JSONObject xmlJSONObj, String declineRecord, String time) {
logger.info("Record is Cached for 3 minutes interval check", declineRecord);
cacheManager.getCache("declineRecords").put(declineRecord, time);
return declineRecord;
}
public boolean isDuplicateCheck(String declineRecord) {
if (null != cacheManager.getCache("declineRecords").get(declineRecord)) {
return true;
}
return false;
}
}
But Each time a record comes in consumer my cache is always empty. Its not holding the records.
Modifications Done:
I have added Configuration file as below after going through the suggestions and more kind of R&D removed some of the earlier logic and now the caching is working as expected but duplicate check is failing when all the three consumers are sending the same records.
`
#Configuration
public class AppCacheManagerConfig {
public static Cache<String, Object> jsonCache =
Caffeine.newBuilder().expireAfterWrite(3, TimeUnit.MINUTES)
.maximumSize(10000).recordStats().build();
#Bean
public CacheLoader<Object, Object> cacheLoader() {
CacheLoader<Object, Object> cacheLoader = new CacheLoader<Object, Object>() {
#Override
public Object load(Object key) throws Exception {
return null;
}
#Override
public Object reload(Object key, Object oldValue) throws Exception {
return oldValue;
}
};
return cacheLoader;
}
`
Now i am using the above cache as manual put and get.
I guess you're trying to implement records deduplication for Kafka.
Here is the similar discussion:
https://github.com/spring-projects/spring-kafka/issues/80
Here is the current abstract class which you may extend to achieve the necessary result:
https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/listener/adapter/AbstractFilteringMessageListener.java
Your caching service is definitely incorrect: Cacheable annotation allows marking the data getters and setters, to add caching through AOP. While in the code you clearly implement some low-level cache updating logic of your own.
At least next possible changes may help you:
Remove #Cacheable. You don't need it because you work with cache manually, so it may be the source of conflicts (especially as soon as you use sync = true). If it helps, remove #EnableCaching as well - it enables support for cache-related Spring annotations which you don't need here.
Try removing Ticker bean with the appropriate parameters for other beans. It should not be harmful as per your configuration, but usually it's helpful only for tests, no need to define it otherwise.
Double-check what is declineRecord. If it's a serialized object, ensure that serialization works properly.
Add recordStats() for cache and output stats() to log for further analysis.
I need to figure out if the following scenario is possible in Spring.
If we have different services / databases per region, can Spring facilitate directing calls to those services / databases per request from a single deployment? To give an example, all requests from user X will be directed to services / databases in the EAST region while all requests from user Y will be directed to services / databases in the WEST region.
Obviously connections to each database will use connection pooling, so the configuration will need to differ, not just properties. When other services are initialized, there is authentication done, so it's not just about databases connections.
This being Spring, I'd like to avoid having to pass implementations around. Can I direct Spring to use a specific configuration per request? Is there a better way to accomplish this?
-- Edit --
Technically it can be done like this, though this isn't exactly easily maintainable.
#Configuration
#PropertySource("classpath:region1.properties")
public class TestIndependentConfigurationRegion1Configuration {
#Bean
public String sampleServiceUrl(#Value("${sample.service.url}") String value) {
return value;
}
#Bean
public TestIndependentConfigurationSampleService testSampleService() {
return new TestIndependentConfigurationSampleService();
}
}
#Configuration
#PropertySource("classpath:region2.properties")
public class TestIndependentConfigurationRegion2Configuration {
#Bean
public String sampleServiceUrl(#Value("${sample.service.url}") String value) {
return value;
}
#Bean
public TestIndependentConfigurationSampleService testSampleService() {
return new TestIndependentConfigurationSampleService();
}
}
#Controller
public class TestIndependentConfigurationController {
protected ApplicationContext testRegion1ApplicationContext = new AnnotationConfigApplicationContext(TestIndependentConfigurationRegion1Configuration.class);
protected ApplicationContext testRegion2ApplicationContext = new AnnotationConfigApplicationContext(TestIndependentConfigurationRegion2Configuration.class);
#RequestMapping("/sample/service")
#ResponseBody
public String testSampleService() {
TestIndependentConfigurationSampleService testSampleService = null;
if(/* region 1 */) {
testSampleService = (TestIndependentConfigurationSampleService) testRegion1ApplicationContext.getBean("testSampleService");
}
if(/* region 2 */) {
testSampleService = (TestIndependentConfigurationSampleService) testRegion2ApplicationContext.getBean("testSampleService");
}
testSampleService.executeSampleService();
return "SUCCESS";
}
}
I don't think you can do that with properties. BUT, you should look at (netflix) ribbon client that is integrated with spring. Some of the ribbon's features allow you to load balance request's between regions. You could customize the ribbon client to do what you want.
Some readings here :
https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.html
I am trying to write a Spring Boot app that talks to two databases - primary which is read-write, and secondary which is read only.
This is also using spring-data-jpa for the repositories.
Roughly speaking this giude describes what I am doing: https://www.baeldung.com/spring-data-jpa-multiple-databases
And this documentation from spring:
https://docs.spring.io/spring-boot/docs/current/reference/html/howto-data-access.html#howto-two-datasources
I am having trouble understanding some things - I think about the transaction managers - which is making my program error out either during normal operation, or during unit tests.
I am running into two issues, with two different TransactionManagers that I do not understand well
1)
When I use JPATransactionManager, my secondary entities become detached between function calls. This happens in my application running in full Spring boot Tomcat, and when running the JUnit test as SpringRunner.
2)
When I use DataSourceTransactionManager which was given in some tutorial my application works correctly, but when I try to run a test case with SpringRunner, without running the full server, spring/hibernate will not perform any inserts or updates on the primaryDataSource.
--
Here is a snippet of code for (1) form a service class.
#Transactional
public List<PrimaryGroup> synchronizeAllGroups(){
Iterable<SecondarySite> secondarySiteList = secondarySiteRepo.findAll();
List<PrimaryGroup> allUserGroups = new ArrayList<PrimaryGroup>(0);
for( SecondarySite secondarySite: secondarySiteList){
allUserGroups.addAll(synchronizeSiteGroups( secondarySite.getName(), secondarySite));
}
return allUserGroups;
}
#Transactional
public List<PrimaryGroup> synchronizeSiteGroups(String sitename, SecondarySite secondarySite){
// GET all secondary groups
if( secondarySite == null){
secondarySite = secondarySiteRepo.getSiteByName(sitename);
}
logger.debug("synchronizeGroups started - siteId:{}", secondarySite.getLuid().toString());
List<SecondaryGroup> secondaryGroups = secondarySite.getGroups();// This shows the error because secondarySite passed in is detached
List<PrimaryGroup> primaryUserGroups = primaryGroupRepository.findAllByAppName(sitename);
...
// modify existingUserGroups to have new data from secondary
...
primaryGroupRepository.save(primaryUserGroups );
logger.debug("synchronizeGroups complete");
return existingUserGroups;
}
I am pretty sure I understand what is going on with the detached entities with JPATransactionManager -- When populateAllUsers calls populateSiteUser, it is only carrying over the primaryTransactionManager and the secondary one gets left out, so the entities become detached. I can probably work around that, but I'd like to see if there is any way to have this work, without putting all calls to secondary* into a seperate service layer, that returns non-managed entities.
--
Here is a snippet of code for (2) from a controller class
#GetMapping("synchronize/secondary")
public String synchronizesecondary() throws UnsupportedEncodingException{
synchronizeAllGroups(); // pull all the groups
synchronizeAllUsers(); // pull all the users
synchronizeAllUserGroupMaps(); // add the mapping table
return "true";
}
This references that same synchronizeAllGroups from above. But when I am useing DataSourceTransactionManager I do not get that detached entity error.
What I do get instead, is that the primaryGroupRepository.save(primaryUserGroups ); call does not generate any insert or update statement - when running a JUnit test that calls the controller directly. So when synchronizeAllUserGroupMaps gets called, primaryUserRepository.findAll() returns 0 rows, and same with primaryGroupRepository
That is to say - it works when I run this test case:
#RunWith(SpringRunner.class)
#SpringBootTest(classes=app.DApplication.class, properties={"spring.profiles.active=local,embedded"})
#AutoConfigureMockMvc
public class MockTest {
#Autowired
private MockMvc mockMvc;
#Test
public void shouldSync() throws Exception {
this.mockMvc.perform(get("/admin/synchronize/secondary")).andDo(print()).andExpect(status().isOk());
}
}
But it does not do any inserts or updates when I run this test case:
#RunWith(SpringRunner.class)
#SpringBootTest(classes=app.DApplication.class, properties={"spring.profiles.active=local,embedded"}, webEnvironment=WebEnvironment.MOCK)
#AutoConfigureMockMvc
public class ControllerTest {
#Autowired AdminController adminController;
#Test
public void shouldSync() throws Exception {
String o = adminController.synchronizesecondary();
}
}
Here are the two configuration classes
Primary:
#Configuration
#EnableTransactionManagement
#EntityScan(basePackageClasses = app.primary.dao.BasePackageMarker.class )
#EnableJpaRepositories(
transactionManagerRef = "dataSourceTransactionManager",
entityManagerFactoryRef = "primaryEntityManagerFactory",
basePackageClasses = { app.primary.dao.BasePackageMarker.class }
)
public class PrimaryConfig {
#Bean(name = "primaryDataSourceProperties")
#Primary
#ConfigurationProperties("app.primary.datasource")
public DataSourceProperties primaryDataSourceProperties() {
return new DataSourceProperties();
}
#Bean(name = "primaryDataSource")
#Primary
public DataSource primaryDataSourceEmbedded() {
return primaryDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
#Primary
public LocalContainerEntityManagerFactoryBean primaryEntityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("primaryDataSource") DataSource primaryDataSource) {
return builder
.dataSource(primaryDataSource)
.packages(app.primary.dao.BasePackageMarker.class)
.persistenceUnit("primary")
.build();
}
#Bean
#Primary
public DataSourceTransactionManager dataSourceTransactionManager( #Qualifier("primaryDataSource") DataSource primaryDataSource) {
DataSourceTransactionManager txm = new DataSourceTransactionManager(primaryDataSource);
return txm;
}
}
And Secondary:
#Configuration
#EnableTransactionManagement
#EntityScan(basePackageClasses=app.secondary.dao.BasePackageMarker.class ) /* scan secondary as secondary database */
#EnableJpaRepositories(
transactionManagerRef = "secondaryTransactionManager",
entityManagerFactoryRef = "secondaryEntityManagerFactory",
basePackageClasses={app.secondary.dao.BasePackageMarker.class}
)
public class SecondaryConfig {
private static final Logger log = LoggerFactory.getLogger(SecondaryConfig.class);
#Bean(name = "secondaryDataSourceProperties")
#ConfigurationProperties("app.secondary.datasource")
public DataSourceProperties secondaryDataSourceProperties() {
return new DataSourceProperties();
}
#Bean(name = "secondaryDataSource")
public DataSource secondaryDataSourceEmbedded() {
return secondaryDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
public LocalContainerEntityManagerFactoryBean secondaryEntityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("secondaryDataSource") DataSource secondaryDataSource) {
return builder
.dataSource(secondaryDataSource)
.packages(app.secondary.dao.BasePackageMarker.class)
.persistenceUnit("secondary")
.build();
}
#Bean
public DataSourceTransactionManager secondaryTransactionManager( #Qualifier("secondaryDataSource") DataSource secondaryDataSource) {
DataSourceTransactionManager txm = new DataSourceTransactionManager(secondaryDataSource);
return txm;
}
}
In my real application, the secondary data source - since it is read-only - is used during real run time, and during the unit test I am writing.
I have been having trouble getting spring to initialize both data sources, so I have not attached a complete example.
thanks for any insight people an give me.
Edit: I have read some things that say to use Jta Transaction Manager when using multiple databases, and I have tried that. I get an error when it tries to run the transaction on my second read-only database when I go to commit to the first database
Caused by: org.postgresql.util.PSQLException: ERROR: prepared transactions are disabled
Hint: Set max_prepared_transactions to a nonzero value.
In my case, I cannot set that, because the database is a read-only datbase provided by a vendor, we cannot change anything, and I really sho9udn't be trying to include this database as part of transactions, I just want to be able to call both databases in one service call.
Using Springboot 1.5.x, Spring Cloud, and JAX-RS:
I could use a second pair of eyes since it is not clear to me whether the Spring configured, Javanica HystrixCommand works for all use cases or whether I may have an error in my code. Below is an approximation of what I'm doing, the code below will not actually compile.
From below WebService lives in a library with separate package path to the main application(s). Meanwhile MyWebService lives in the application that is in the same context path as the Springboot application. Also MyWebService is functional, no issues there. This just has to do with the visibility of HystrixCommand annotation in regards to Springboot based configuration.
At runtime, what I notice is that when a code like the one below runs, I do see "commandKey=A" in my response. This one I did not quite expect since it's still running while the data is obtained. And since we log the HystrixRequestLog, I also see this command key in my logs.
But all the other Command keys are not visible at all, regardless of where I place them in the file. If I remove CommandKey-A then no commands are visible whatsoever.
Thoughts?
// Example WebService that we use as a shared component for performing a backend call that is the same across different resources
#RequiredArgsConstructor
#Accessors(fluent = true)
#Setter
public abstract class WebService {
private final #Nonnull Supplier<X> backendFactory;
#Setter(AccessLevel.PACKAGE)
private #Nonnull Supplier<BackendComponent> backendComponentSupplier = () -> new BackendComponent();
#GET
#Produces("application/json")
#HystrixCommand(commandKey="A")
public Response mainCall() {
Object obj = new Object();
try {
otherCommandMethod();
} catch (Exception commandException) {
// do nothing (for this example)
}
// get the hystrix request information so that we can determine what was executed
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = hystrixExecutedCommands();
// set the hystrix data, viewable in the response
obj.setData("hystrix", executedCommands.orElse(Collections.emptyList()));
if(hasError(obj)) {
return Response.serverError()
.entity(obj)
.build();
}
return Response.ok()
.entity(healthObject)
.build();
}
#HystrixCommand(commandKey="B")
private void otherCommandMethod() {
backendComponentSupplier
.get()
.observe()
.toBlocking()
.subscribe();
}
Optional<Collection<HystrixInvokableInfo<?>>> hystrixExecutedCommands() {
Optional<HystrixRequestLog> hystrixRequest = Optional
.ofNullable(HystrixRequestLog.getCurrentRequest());
// get the hystrix executed commands
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = Optional.empty();
if (hystrixRequest.isPresent()) {
executedCommands = Optional.of(hystrixRequest.get()
.getAllExecutedCommands());
}
return executedCommands;
}
#Setter
#RequiredArgsConstructor
public class BackendComponent implements ObservableCommand<Void> {
#Override
#HystrixCommand(commandKey="Y")
public Observable<Void> observe() {
// make some backend call
return backendFactory.get()
.observe();
}
}
}
// then later this component gets configured in the specific applications with sample configuraiton that looks like this:
#SuppressWarnings({ "unchecked", "rawtypes" })
#Path("resource/somepath")
#Component
public class MyWebService extends WebService {
#Inject
public MyWebService(Supplier<X> backendSupplier) {
super((Supplier)backendSupplier);
}
}
There is an issue with mainCall() calling otherCommandMethod(). Methods with #HystrixCommand can not be called from within the same class.
As discussed in the answers to this question this is a limitation of Spring's AOP.