Is there an sample code available to capture Spring Batch Micro Metrics? - spring-boot

Developing a spring boot batch application, wanted to know if there is a sample code to how to get the micro metrics discussed in the spring batch document?
I am looking way of getting these details per execution. Also, since I run the application using cron task schedule, can we get a separation of this data per execution?

Found the solution
You need not do any coding, all readily available.
Include actuator dependency in pom
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
In the property, file add this line (instead of * you can add comma separated and security) refrence
management.endpoints.web.exposure.include=*
If your application is running on port 8080
http://localhost:8080/actuator/metrics/spring.batch.job
Will give job stats. You can query on any other metrics parameters. Reference
You can calculate metrics for all execution in the corn task schedule by
First create a custom metric endpoint.
#Component
#Endpoint(id = "spring.batch.executions")
public class BatchMetricEndpoint {
#Autowired
BatchAllJobsMetricContext batchAllJobsMetricContext;
#ReadOperation
public Map<String, BatchPerJobMetricContext> features() {
return batchAllJobsMetricContext.getAll();
}
}
Second
Create a job listener
Tap into local metric endpoint after execution.
#Component
public class BatchJobListener implements JobExecutionListener {
#Autowired
private MetricsEndpoint metricsEndpoint;
#Autowired
BatchAllJobsMetricContext batchAllJobsMetricContext;
#Autowired
BatchPerJobMetricContext batchPerJobMetricContext;
#Override
public void beforeJob(JobExecution jobExecution) {
}
#Override
public void afterJob(JobExecution jobExecution) {
MetricsEndpoint.MetricResponse metricResponse = metricsEndpoint.metric("spring.batch.job",null);
String key = jobExecution.getJobParameters().getString("jobId");
String execution = "Execution "+jobExecution.getJobParameters().getString("executionCout");
if(batchAllJobsMetricContext.hasKey(key)){
batchAllJobsMetricContext.get(key).put(execution,metricResponse);
}else{
batchPerJobMetricContext.put(execution,metricResponse);
batchAllJobsMetricContext.put(key,batchPerJobMetricContext);
}
}
}
Third
Tap into the local metric to aggrigate data.
Please note this way of holding metic per iteration would be expensive on memory, you would like to keep some limit and push this data to time series data source.
Sample code

Related

spring boot with spark-submit cluster mode

We are trying to achieve parallelism through the spark executors below are the steps which we are following -
Read from the hive
Data transformation (custom spring library).
Ship it via rest endpoint in batches (1000 records per batch).
Problem - We want to do all these steps in parallel and want to use a spring-boot based library.
Understanding - If we are using the custom code for transformation (the code that we want to run parallelly most probably inside rdd.map() method) then our classes and composite dependencies need to be serialized or those classes need to implement serialization.
We know we can achieve this by performing these tasks in sequence over the driver in such case we need to collect the data over the driver again + again and then pass it to the next step. In this case, we are not leveraging the power of executors.
Needs your assistance here -
If we ship this spring boot dependency to executors then is there any way that the executor understands the spring boot code and resolves the annotations over there ?
sample code -
Code from spring boot library -
public class Process{
String convert(Row row) {
return row.mkString();
}
}
#Component
#ConditionalOnProperty(name = "process.dummy.serialize", havingValue = "true")
class ProcessNotSerialized extends Process {
#Autowired
private RecordService recorService; //not-serialized
public void setName(String name) {
this.name = name;
}
private String name;
#Override
public String toString() {
return "ProcessNotSerialized";
}
}
Code from my spark spring boot application -
Dataset<Row> sqlDf = sparkSession.sql(sqlQuery); // millions of data
ProcessNotSerialized process = new ProcessNotSerialized();
System.out.println("object name=>" + process.toString());
List<String> listColumns = sqlDf.select(column)
.javaRDD()
.map(row -> {
return process.convert(row);
})
.collect();
here is the code inside map() will execute in parallel.
Please let me know if you have any better way other than serialization or running over a driver.

How do I get JobRunr to detect my scheduled background job in a Spring controller/service?

I have been looking into using JobRunr for starting background jobs on my Spring MVC application, as I really like the simplicity of it, and the ease of integrating it into an IoC container.
I am trying to create a simple test scheduled job that writes a line of text to my configured logger every minute, but I'm struggling to figure out how to get the JobRunr background job server to detect it and queue it up. I am not using Spring Boot so I am just using the generic jobrunr Maven artifact rather than the "Spring Boot Starter". My setup is as follows:
pom.xml
<dependency>
<groupId>org.jobrunr</groupId>
<artifactId>jobrunr</artifactId>
<version>2.0.0</version>
</dependency>
ApplicationConfig.java
#Bean
public JobMapper jobMapper() {
return new JobMapper(new JacksonJsonMapper());
}
#Bean
#DependsOn("jobMapper")
public StorageProvider storageProvider(JobMapper jobMapper) {
InMemoryStorageProvider storageProvider = new InMemoryStorageProvider();
storageProvider.setJobMapper(jobMapper);
return storageProvider;
}
#Bean
#DependsOn("storageProvider")
public JobScheduler jobScheduler(StorageProvider storageProvider, ApplicationContext applicationContext) {
return JobRunr.configure().useStorageProvider(storageProvider)
.useJobActivator(applicationContext::getBean)
.useDefaultBackgroundJobServer()
.useDashboard()
.useJmxExtensions()
.initialize();
}
BackgroundJobsController.java
#Controller
public class BackgroundJobsController {
private final Logger logger = LoggerFactory.getLogger(getClass());
private #Autowired JobScheduler jobScheduler;
#Job(name = "Test")
public void executeJob() {
BackgroundJob.scheduleRecurrently(Cron.minutely(), () -> logger.debug("It works!"));
jobScheduler.scheduleRecurrently(Cron.minutely(), () -> logger.debug("It works too!"));
}
}
As you can see, I have tried both methods of initiating the background job in the executeJob method. The issue is basically getting Jobrunr to detect the jobs - is it simply a case of somehow triggering the executeJob method upon startup of the application? If so, does anyone know the most simple way to do that? Previously I have used the Spring #Scheduled annotation to automatically run through methods in a Service/Controller class upon startup of the application, so I was hoping there was a straightforward way to get Jobrunr to pick up the scheduled tasks I am trying to create. Apologies if it is something stupid that I have overlooked. I've spent a good few hours trying different things and reading through the documentation!
Thanks in advance!
There are different ways for doing so:
This is one, annotating a method with #PostConstruct is indeed another.
#SpringBootApplication
#Import(JobRunrExampleConfiguration.class)
public class JobRunrApplication {
public static void main(String[] args) {
ConfigurableApplicationContext applicationContext = SpringApplication.run(JobRunrApplication.class, args);
JobScheduler jobScheduler = applicationContext.getBean(JobScheduler.class);
jobScheduler.<SampleJobService>scheduleRecurrently("recurring-sample-job", every5minutes(), x -> x.executeSampleJob("Hello from recurring job"));
}
}
You can see an example here: https://github.com/jobrunr/example-java-mag/blob/main/src/main/java/org/jobrunr/examples/JobRunrApplication.java
Have you tried annotating your executeJob Method with a #PostConstruct ? That way upon initialisation of your application, the jobs would be registered to the JobServer.
I believe the #Job annotation is meant fo the method of the job itself. (In your case the debug method).
There is now a new way to do so:
You can add #Recurring to any Spring Boot, Micronaut or Quarkus bean method. A Spring Boot example:
#Component
public class SomeService {
#Recurring(id="recurring-job-every-5-min" interval = "PT5M")
#Job(name="job name for the dashboard")
public void runEvery5Minutes() {
// business logic comes here
}
}
For more info, see the JobRunr documentation.

Сamunda replace behaviour for external tasks in tests

I created simple Camunda spring boot project and also created simple BPMN process with switcher. (5.5 KB)
I used service task with external implementation as a spring beans. I want to write tests for process but I don't want to test how beans works. Because in general I use external implementation for connection to DB and save parameter to context or REST call to internal apps. For example I want skip execute service task(one) and instead set variables for switcher. I tried to use camunda-bpm-assert-scenario for test process and wrote simple test WorkflowTest.
I noticed if I use #MockBean for One.class then Camunda skip delegate execution. If use #Mock then Camunda execute delegate execution.
PS Sorry for bad english
One
#Service
public class One implements JavaDelegate {
private final Random random = new Random();
#Override
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello, One!");
execution.setVariable("check", isValue());
}
public boolean isValue() {
return random.nextBoolean();
}
}
WorkflowTest
#SpringBootTest
#RunWith(SpringRunner.class)
#Deployment(resources = "process.bpmn")
public class WorkflowTest extends AbstractProcessEngineRuleTest {
#Mock
private ProcessScenario insuranceApplication;
#MockBean
private One one;
#Before
public void init() {
MockitoAnnotations.initMocks(this);
Mocks.register("one", one);
}
#Test
public void shouldExecuteHappyPath() throws Exception {
// given
when(insuranceApplication.waitsAtServiceTask("Task_generator")).thenReturn(externalTaskDelegate -> {
externalTaskDelegate.complete(withVariables("check", true));
}
);
String processDefinitionKey = "camunda-test-process";
Scenario scenario = Scenario.run(insuranceApplication)
.startByKey(processDefinitionKey) // either just start process by key ...
.execute();
verify(insuranceApplication).hasFinished("end_true");
verify(insuranceApplication, never()).hasStarted("three");
verify(insuranceApplication, atLeastOnce()).hasStarted("two");
assertThat(scenario.instance(insuranceApplication)).variables().containsEntry("check", true);
}
}
I found two solutions:
It's a little hack. If user #MockBean for delegate in a test. The delegate will be skipped but you have trouble with process engine variables.
Create two beans with one qualifier and use profiles for testing and production. I used to default profile for local start and test profile for testing.

Spring boot test: Wait for microservices schedulder task

I'm trying to test a service which's trying to communicate with other one.
One of them generates auditories which are stored on memory until an scheduled task flushs them on a redis node:
#Component
public class AuditFlushTask {
private AuditService auditService;
private AuditFlushTask(AuditService auditService) {
this.auditService = auditService;
}
#Scheduled(fixedDelayString = "${fo.audit-flush-interval}")
public void flushAudits() {
this.auditService.flush();
}
}
By other hand, this service provide an endpoint stands for providing those flushed auditories:
public Collection<String> listAudits(
) {
return this.boService.listRawAudits(deadlineTimestamp);
}
The problem is I'm building an integration test in order to check if this process works right, I mean, if audits are well provided.
So, I don't know how to "wait until audits has been flushed on microservice".
Any ideas?
Don't test the framework: Spring almost certainly has tests which test fixed delays.
Instead, keep all logic within the service itself, and integration test that in isolation from the Spring #Scheduled function.

how to calculate the starting time and end time for apache camel router?

i am trying to capture the start time and end time of the apache camel router.
We are using camel for messaging and we have implemented soap serviceand quartz are in camel router like below
public class BbdDmsBatchRoute extends RouteBuilder{
#BeanInject
private BbdImageProcessorBean bbdImageProcessorBean;
#BeanInject
private SwtDEAStockProcessorBean swtDEAStockProcessorBean;
#Value("${bbd.batch.enabled}")
private String isBatchEnabled;
#Value("${bbd.dmsStockBatch.interval}")
private String bbdDmsBatchInterval ;
private static final Logger LOG = LoggerFactory.getLogger(BbdDmsBatchRoute.class);
#Override
public void configure() throws Exception {
//***********Batch Job to get BBD images**************//
// This Batch Job will run based on time given in properties file for "bbd.batch.interval" property (bbd.batch.interval=0+0+12+1/1+*+?+*)
LOG.info("BBD Batch Job Cron expression {}",bbdDmsBatchInterval);
from("quartz2://STouchDmsBatch/bbdDailyJob?cron="+ bbdDmsBatchInterval)
.choice()
.when(simple(isBatchEnabled))
.doTry()
.bean(bbdImageProcessorBean, "bbdReRunDmsBatchInvocation")
.split(bodyAs(List.class)) //Split each object in list and process in parallel
//.executorServiceRef("executerRefProfile").parallelProcessing()
.to("direct:batchJobRouteUnitWork")
.end()
.endDoTry()
.doCatch(Exception.class)
.to("direct:BbdReRunError")
.end();
}
}
i want to calculate startime when my quartz2:StouchDmsBatch gets triggered and endTime when the route get completed without any exception.
i red few examples, told to use camel interceptor, EventSupport and so on.
i would like to know did anybody have implemented the same?
can somebody help me on this?
Refer this .. if you just want to capture current time at some point and then use it in your logging somewhere .. you can store the current time in a exchange property and then use it anywhere you like..
How to time portions of an apache camel route?
You can add this class as a bean. Then enable the events you are interested. You can then see how long each step takes.
http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html

Resources