I have a spring batch jobs in which I want to push metrics to Prometheus as suggested here
I have a requirement to push only default metrics given by Spring batch via micrometer.There is a easy way in which I dont have to anything and just add following dependency:
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
</dependency>
as mentioned here
But the second way is setting up PushGateway server and then writing Configuration Class
Is there any difference in these two approaches?
In second approach how to write it when all we need is default metrics?Pushgatewayurl is fine ,I understood, but if I have dozen of jobs, what to set in jobname and grouping key.?
Is there any difference in these two approaches?
No, there is no difference. Please note that not all Spring Batch users are Spring Boot users. That's why the sample in Spring Batch's repository shows how to configure a task that pushes metrics to the gateway. Now if you use Spring Boot, you don't have to write that class since an equivalent is configured by Spring Boot automatically.
In second approach how to write it when all we need is default metrics?Pushgatewayurl is fine ,I understood, but if I have dozen of jobs, what to set in jobname and grouping key.?
Spring Batch metrics are tagged by job name, step name, etc, see the Tags column in the Built-in Metrics table. So even if you have multiple jobs, the metrics will be distinct. You can do the filtering on Prometheus side with the corresponding tag.
Related
I am working on component that is based on spring framework. We have not yet moved to spring boot.
My requirement is to capture metrics(JVM/http/disk space) for my component which runs on an application server.
I came across micrometer library which can be utilized to capture such metrics and it can be integrated very well with Promotheus.
What I did was that I added the below dependency
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.7.5</version>
</dependency>
After adding the dependency I exposed a rest end point and added some simple logic to pull the metrics. Doing that I was able to fetch some basic JVM metrics. I referred the below link for this which explains how to capture metrics.
(https://micrometer.io/docs/ref/jvm)
However in addition to JVM metrics I also want to capture http request metrics(eg. the no of requests, time taken on the http calls to the rest services).
In my application there are quite many rest endpoints. Is there any way to do that. I was not able find any good documentation on that.
Any help would be highly appreciated.
Thanks
Sachin
As you said, Spring Boot does this out of the box so if you can move there, you don't need to do anything.
In order to instrument your web endpoints you can do a few things:
You can create a Filter and instrument all of your calls there.
This is what Spring Boot does, you can take a look/copy WebMvcMetricsFilter
You can add #Timer for your controllers and set-up TimedAspect
You can manually instrument your controllers see the docs
After following the above suggestions I was actually able to see my http metrics
I Simply created a configuration class annotated with #EnableAspectJAutoProxy and defined a bean inside the class as below
#Bean
public TimedAspect timedAspect() {
return new TimedAspect(registry);
}
And then added the #Timed annotation on my REST api POST methods and then I was able to see the statistics in Prometheus dashboard.
This really works!
I have multiple jobs for my Spring Batch application, but only a single job uses some specific Spring Boot auto configuration features:
a job that uses spring-data-jpa auto configuration, to configure a database for business transactions (not for Spring Batch management)
a job that does not use the database at all
I have packaged both jobs in the same unit because it makes sense from business perspective. Both jobs will work together and the output of one job will be the input of the other job.
Is it possible to disable database specific auto configuration when I run the second job?
I just tried using profiles and I have disabled the autoconfiguration for a specific profile. I am pretty happy with this solution but I wonder if there are other solutions?
This is similar to trying to lazy load beans specific to a given job: How to apply something like #Lazy to Spring Batch?. While the Spring profiles feature may fix your issue, I believe it is working around the root issue which is packaging all jobs in a monolithic way.
I would package each job separately and this problem (and many others) disappears by design. There are several advantages to this approach:
Independent lifecyle management (bugs, features, etc)
Flexible deployment
Separate logs
Separate configurations (as in the current issue)
Easier/Better scalability
And all the good reasons to make one thing do one thing and do it well.
I have one #Scheduler job which runs on multiple servers in a clustered environment. However I want to restrict the job to run in only in one server and other servers should not run the same job once any other server has started it .
I have explored Spring Batch has lock mechanism using some Database table , but looking for any a solution only in spring task:scheduler.
I had the same problem and the solution what I implemented was a the Lock mechanism with Hazelcast and to made it easy to use I also added a proper annotation and a bit of spring AOP for that. So with this trick I was able to enforce a single schedule over the cluster done with a single annotation.
Spring Batch has this nice functionality that it would not run the job with same job arguments twice.
You can use this feature so that when a spring batch job kicks start in another server it does not run.
Usually people pass a timestamp as argument so it will by pass this logic, which you can change it.
I am a newbie in Java and trying to implement a Spring Boot batch application.
My requirement is like to check some data in database (one part) and delete if found (another part).
I am planning to implement Spring Boot batch for this.
I will have one job which will have 2 steps. If Step 1 find some data then only execute step 2? Can I achieve in Spring Boot Batch? Or what is the best way to achieve this keeping in mind I have to schedule this to run weekly.
With just the scheduled job for find and delete records from DB, I don't suggest using Spring Batch. Spring has nice good way of doing it without Batch using scheduling-tasks. You can see example here. Use Spring Batch only if you need to run jobs in batch that can't be handled with normal operation.
If you need complex scheduler, you can use Spring Quartz scheduler.
Just looking for some information if others have solved this pattern. I want to use Spring Integration and Spring Batch together. Both of these are SpringBoot applications and ideally I'd like to keep them and their respective configuration separated, so they are both their own executable jar. I'm having problems executing them in their own process space and I believe I want, unless someone can convince me otherwise, each to run like they are their own Spring Boot app and initialize themselves with their own profiles and properties. What I'm having trouble with though is the invocation of the job in my SpringBatch project from my SpringIntegration project. At first I couldn't get the properties loaded from the batch project, so I realized I need to pass the spring.active.profiles as a Job Parameter and that seemed to solve that. But there are other things in the Spring Boot Batch application that aren't loading correctly like the schema-platform.sql file and the database isn't getting initialized, etc.
On this initial launch of the job I might want the response to go back to Spring Integration for some messaging on Job Status. There might be times when I want to run a job without Spring Integration kicking off the job, but still take advantage of sending statuses back to the Spring Integration project providing its listening on a channel or something.
I've reviewed quite a few Spring samples and have yet to find my exact scenario, most are with the two dependencies in the same project, so maybe I'm doing something that's not possible, but I'm sure I'm just missing a little something in the Spring configuration.
My questions/issues are:
I don't want the Spring Integration project to know anything about the SpringBatch configuration other than the job its kicking off. I have found a good way to do that reference to the Job Bean without getting my entire batch configuration loading.
Should I keep these two projects separated or would it be better to combine them since I have two-way communication between both.
How should the Job be launch from the integration project. We're using the spring-batch-integration project with JobLaunchRequest and JobLauncher. This seems to run it in the same process as the Spring Integration project and I'm missing a lot of my SpringBootBatch projects initialization
Should I be using a CommandLineRunner instead to force it to another process.
Is SpringApplication.run(BatchConfiguration.class) the answer?
Looking for some general project configuration setup to meet these requirements.
Spring Cloud Data Flow in combination with Spring Cloud Task does exactly what you're asking. It launches Spring Cloud Task applications (which can contain batch jobs) as new processes on the platform you choose. I'd encourage you to check out that project here: http://cloud.spring.io/spring-cloud-dataflow/