I'm trying to test a service which's trying to communicate with other one.
One of them generates auditories which are stored on memory until an scheduled task flushs them on a redis node:
#Component
public class AuditFlushTask {
private AuditService auditService;
private AuditFlushTask(AuditService auditService) {
this.auditService = auditService;
}
#Scheduled(fixedDelayString = "${fo.audit-flush-interval}")
public void flushAudits() {
this.auditService.flush();
}
}
By other hand, this service provide an endpoint stands for providing those flushed auditories:
public Collection<String> listAudits(
) {
return this.boService.listRawAudits(deadlineTimestamp);
}
The problem is I'm building an integration test in order to check if this process works right, I mean, if audits are well provided.
So, I don't know how to "wait until audits has been flushed on microservice".
Any ideas?
Don't test the framework: Spring almost certainly has tests which test fixed delays.
Instead, keep all logic within the service itself, and integration test that in isolation from the Spring #Scheduled function.
Related
I have been looking into using JobRunr for starting background jobs on my Spring MVC application, as I really like the simplicity of it, and the ease of integrating it into an IoC container.
I am trying to create a simple test scheduled job that writes a line of text to my configured logger every minute, but I'm struggling to figure out how to get the JobRunr background job server to detect it and queue it up. I am not using Spring Boot so I am just using the generic jobrunr Maven artifact rather than the "Spring Boot Starter". My setup is as follows:
pom.xml
<dependency>
<groupId>org.jobrunr</groupId>
<artifactId>jobrunr</artifactId>
<version>2.0.0</version>
</dependency>
ApplicationConfig.java
#Bean
public JobMapper jobMapper() {
return new JobMapper(new JacksonJsonMapper());
}
#Bean
#DependsOn("jobMapper")
public StorageProvider storageProvider(JobMapper jobMapper) {
InMemoryStorageProvider storageProvider = new InMemoryStorageProvider();
storageProvider.setJobMapper(jobMapper);
return storageProvider;
}
#Bean
#DependsOn("storageProvider")
public JobScheduler jobScheduler(StorageProvider storageProvider, ApplicationContext applicationContext) {
return JobRunr.configure().useStorageProvider(storageProvider)
.useJobActivator(applicationContext::getBean)
.useDefaultBackgroundJobServer()
.useDashboard()
.useJmxExtensions()
.initialize();
}
BackgroundJobsController.java
#Controller
public class BackgroundJobsController {
private final Logger logger = LoggerFactory.getLogger(getClass());
private #Autowired JobScheduler jobScheduler;
#Job(name = "Test")
public void executeJob() {
BackgroundJob.scheduleRecurrently(Cron.minutely(), () -> logger.debug("It works!"));
jobScheduler.scheduleRecurrently(Cron.minutely(), () -> logger.debug("It works too!"));
}
}
As you can see, I have tried both methods of initiating the background job in the executeJob method. The issue is basically getting Jobrunr to detect the jobs - is it simply a case of somehow triggering the executeJob method upon startup of the application? If so, does anyone know the most simple way to do that? Previously I have used the Spring #Scheduled annotation to automatically run through methods in a Service/Controller class upon startup of the application, so I was hoping there was a straightforward way to get Jobrunr to pick up the scheduled tasks I am trying to create. Apologies if it is something stupid that I have overlooked. I've spent a good few hours trying different things and reading through the documentation!
Thanks in advance!
There are different ways for doing so:
This is one, annotating a method with #PostConstruct is indeed another.
#SpringBootApplication
#Import(JobRunrExampleConfiguration.class)
public class JobRunrApplication {
public static void main(String[] args) {
ConfigurableApplicationContext applicationContext = SpringApplication.run(JobRunrApplication.class, args);
JobScheduler jobScheduler = applicationContext.getBean(JobScheduler.class);
jobScheduler.<SampleJobService>scheduleRecurrently("recurring-sample-job", every5minutes(), x -> x.executeSampleJob("Hello from recurring job"));
}
}
You can see an example here: https://github.com/jobrunr/example-java-mag/blob/main/src/main/java/org/jobrunr/examples/JobRunrApplication.java
Have you tried annotating your executeJob Method with a #PostConstruct ? That way upon initialisation of your application, the jobs would be registered to the JobServer.
I believe the #Job annotation is meant fo the method of the job itself. (In your case the debug method).
There is now a new way to do so:
You can add #Recurring to any Spring Boot, Micronaut or Quarkus bean method. A Spring Boot example:
#Component
public class SomeService {
#Recurring(id="recurring-job-every-5-min" interval = "PT5M")
#Job(name="job name for the dashboard")
public void runEvery5Minutes() {
// business logic comes here
}
}
For more info, see the JobRunr documentation.
I am trying Spring Boot and think about scalabilty.
Lets say I have a component that does a job (e.g. checking for new mails).
It is done by a scheduled method.
e.g.
#Component
public class MailMan
{
#Scheduled (fixedRateString = "5000")
private void run () throws Exception
{ //... }
}
Now the application gets a new customer. So the job has to be done twice.
How can I scale this component to exist or run twice?
Interesting question but why Multiple components per customer? Can scheduler not pull the data for every customer on scheduled run and process the record for each customer? You component scaling should not be decided based on the entities evolved in your application but the resources utilization by the component. You can have dedicated components type for processing the messages for queues and same for REST. Scale them based on how much each of them is getting utilized.
Instead of using annotations to schedule a task, you could do the same thing programmatically by using a ScheduledTaskRegistrar. You can register the same bean multiple time, even if it is a singleton.
public class SomeSchedulingConfigurer implements SchedulingConfigurer {
private final SomeJob someJob; <-- a bean that is Runnable
public SomeSchedulingConfigurer(SomeJob someJob) {
this.someJob = someJob;
}
#Override
public void configureTasks(#NonNull ScheduledTaskRegistrar taskRegistrar) {
int concurrency = 2;
IntStream.range(0, concurrency)).forEach(
__ -> taskRegistrar.addFixedDelayTask(someJob, 5000));
}
}
Make sure the thread executor you are using is large enough to process the amount of jobs concurrently. The default executor has exactly one thead :-). Be aware that this approach has scaling limits.
I also recommend to add a delay or skew between jobs, so that not all jobs run at exactly the same moment.
See SchedulingConfigurer
and
ScheduledTaskRegistrar
for reference.
The job needs to run only once even with multiple customers. The component itself doesn't need to scale at all. It just a mechanism to "signal" that some logic needs to be run at some moment in time. I would keep the component really thin and just call the desired business logic that handles all the rest e.g.
#Component
public class MailMan {
#Autowired
private NewMailCollector newMailCollector;
#Scheduled (fixedRateString = "5000")
private void run () throws Exception {
// Collects emails for customers
newMailCollector.collect();
}
}
If you want to check for new e-mails per customer you might want to avoid using scheduled tasks in a backend service as it will make the implementation very inflexible.
Better make an endpoint available for clients to call to trigger that logic.
Note: read the end of the answer for the way I implemented #Nonika's suggestions
What's the "right way" to send a websocket event on data insert?
I'm using a Spring Boot server with SQL/JPA and non-stomp websockets. I need to use "plain" websockets as I'm using Java clients where (AFAIK) there's no stomp support.
When I make a change to the database I need to send the event to the client so I ended up with an implementation like this:
#Transactional
public void addEntity(...) {
performActualEntityAdding();
sendEntityAddedEvent(eventData);
}
#Transactional
public void sendEntityAddedEvent(String eventData) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
#Override
public void afterCommit() {
sendEntityAddedEventAsync(eventData);
}
});
}
#Async
public void sendEntityAddedEventAsync(String eventData) {
// does the websocket session sending...
}
This works. If I would just call the sendEntityAddedEventAsync it would also work for real world scenarios but it fails on unit tests because the event would arrive before transaction commit. As such when the unit test invokes a list of the entities after the event it fails.
This feels like a hack that shouldn't be here. Is there a better way to ensure a commit?
I tried multiple alternative approaches and the problem is that they often worked for 10 runs of the unit tests yet failed every once in a while. That isn't acceptable.
I tried multiple approaches to solve this such as different transaction annotations and splitting the method to accommodate them. E.g read uncommitted, not supported (to force a commit) etc. Nothing worked for all cases and I couldn't find an authoritative answer for this (probably common) use case that wasn't about STOMP (which is pretty different).
Edit
One of my original attempts looked something like this:
// this shouldn't be in a transaction
public void addEntity(...) {
performActualEntityAdding();
sendEntityAddedEvent(eventData);
}
#Transactional
public void performActualEntityAdding(...) {
//....
}
#Async
public void sendEntityAddedEventAsync(String eventData) {
// does the websocket session sending...
}
The assumption here is that when sendEntityAddedEventAsync is invoked the data would already be in the database. It wasn't for a couple of additional milliseconds.
A few additional details:
Test environment is based on h2 (initially I mistakenly wrote hsql)
Project is generated by JHipster
Level 2 cache is used but disabled as NONE for these entities
Solution (based on #Nonika's answer):
The solution for me included something similar to this:
public class WebEvent extends ApplicationEvent {
private ServerEventDAO event;
public WebEvent(Object source, ServerEventDAO event) {
super(source);
this.event = event;
}
public ServerEventDAO getEvent() {
return event;
}
}
#Transactional
public void addEntity(...) {
performActualEntityAdding();
applicationEventPublisher.publishEvent(new WebEvent(this, evtDao));
}
#Async
#TransactionalEventListener
public void sendEntityAddedEventAsync(WebEvent eventData) {
// does the websocket session sending...
}
This effectively guarantees that the data is committed properly before sending the event and it runs asynchronously to boot. Very nice and simple.
Spring is using AdviceMode.PROXY for both #Async and #Transactional this is quote from the javadoc:
The default is AdviceMode.PROXY. Please note that proxy mode allows
for interception of calls through the proxy only. Local calls within
the same class cannot get intercepted that way; an Async annotation on
such a method within a local call will be ignored since Spring's
interceptor does not even kick in for such a runtime scenario. For a
more advanced mode of interception, consider switching this to
AdviceMode.ASPECTJ.
This rule is common for almost all spring annotations which requires proxy to operate.
Into your first example, you have a #Transactional annotation on both addEntity(..) and performActualEntityAdding(..). I suppose you call addEntity from another class so #Transactional works as expected. process in this scenario can be described in this flow
// -> N1 transaction starts
addEntity(){
performActualEntityAdding()//-> we are still in transaction N1
sendEntityAddedEvent() // -> call to this #Async is a class local call, so this advice is ignored. But if this was an async call this would not work either.
}
//N1 transaction commits;
That's why the test fails. it gets an event that there is a change into the db, but there is nothing because the transaction has not been committed yet.
Scenario 2.
When you don't have a #Transactional addEntity(..) then second transaction for performActualEntityAdding not starts as there is a local call too.
Options:
You can use some middleware class to call these methods to trigger
spring interceptors.
you can use Self injection with Spring
if you have Spring 5.0 there is handy #TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
EDIT: As C. Weber suggested in the comments, the solution is to add #Transactional to the test class.
I have some tests that use an H2 in-memory DB. I need to reset the DB before each test. Although my SQL scripts are run each a test is executed, the DB is not properly reset, resulting in a missing needed entry after a delete test.
Test class:
#RunWith(SpringRunner.class)
#SpringBootTest
#AutoConfigureTestDatabase(replace=Replace.ANY, connection=EmbeddedDatabaseConnection.H2)
public class RepositoryTests {
#Autowired
private Repository repository;
#Autowired
private DataSource dataSource;
#Before
public void populateDb() {
Resource initSchema = new ClassPathResource("database/schema.sql");
Resource initData = new ClassPathResource("database/data.sql");
DatabasePopulator dbPopulator = new ResourceDatabasePopulator(initSchema, initData);
DatabasePopulatorUtils.execute(dbPopulator, dataSource);
}
#Test
public void testMethod1() {
// ...
repository.delete("testdata");
}
#Test
public void testMethod2() {
// ...
Object test = repository.get("testdata");
// is null but should be an instance
}
}
schema.sql drops all tables before recreating them. data.sql inserts all needed test data into the DB.
Running the testMethod2 alone succeeds. However, running all tests makes the test fail with a NullPointerException.
I have successfully tried to use #DirtiesContext, however this is not an option because I can't afford to have a 20 second startup for each 0.1 second test.
Is there another solution?
The Spring Test Framework provides a mechanism for the behaviour you want for your tests. Simply annotate your Test class with #Transactional to get the default rollback behaviour for each test method.
There are ways to configure the transactional behaviour of tests and also some pitfalls (like using RestTemplate inside test method), which you can read more about in the corresponding chapter of the Spring manual.
Spring Test Framework
I am trying to write my first integration test with spring boot application. So to write proper test cases I am using #Transactional provided by spring so that state of my in memory data base remains same before and after execution the test.
Thing is #Transactional is working as expected my test execution starts it inserts some data for testing into db and when execution finish it rollbacks the changes done by it but the problem is when the test execution goes to the code I am testing it tries to fetch data persisted by test but does not find any data there. I've shared my code for better understanding.
#RunWith(SpringRunner.class)
#SpringBootTest
#Transactional
public class MyFirstControllerTest {
#Autowired
private TestRestTemplate restTemplate;
#Autowired
private MyFirstRepository myFirstRepo;
#Test
public void testMethod() {
//insert some data.
Dummy data = new DummyData();
myFirstRepo.save(data);
//hits the rest route to get data.
ResponseEntity<String> response =
this.restTemplate.getForEntity("/dummys", String.class); // did not find any data in db.
myFirstRepo.findAll(); //working - fetched the data inserted in test.
}
}
Below is my controller.
#Controller
#Validated
public class MyFirstController {
#Autowired
private MyFirstRepository myFirstRepo;
#GetMapping("/dummys")
public ResponseEntity<List<DummyDataDto>> getDummyData() {
List<DummyData> data = myFirstRepo.findAll(); //does not finds any data
//convert list of dummy data to list of dummydata dto.
return new ResponseEntity<DummyDataDto>(List<DummyDataDto>, HttpStatus.OK);
}
}
This is occurring because of the isolation level of in memory database that you have been using. I assume that default isolation level of that database in Read Committed.
If you are using H2 then you may find details of isolation level here.
All you need to use Read Uncommited isolation level for integration test, if your business requirement says to do so.