Just looking for some information if others have solved this pattern. I want to use Spring Integration and Spring Batch together. Both of these are SpringBoot applications and ideally I'd like to keep them and their respective configuration separated, so they are both their own executable jar. I'm having problems executing them in their own process space and I believe I want, unless someone can convince me otherwise, each to run like they are their own Spring Boot app and initialize themselves with their own profiles and properties. What I'm having trouble with though is the invocation of the job in my SpringBatch project from my SpringIntegration project. At first I couldn't get the properties loaded from the batch project, so I realized I need to pass the spring.active.profiles as a Job Parameter and that seemed to solve that. But there are other things in the Spring Boot Batch application that aren't loading correctly like the schema-platform.sql file and the database isn't getting initialized, etc.
On this initial launch of the job I might want the response to go back to Spring Integration for some messaging on Job Status. There might be times when I want to run a job without Spring Integration kicking off the job, but still take advantage of sending statuses back to the Spring Integration project providing its listening on a channel or something.
I've reviewed quite a few Spring samples and have yet to find my exact scenario, most are with the two dependencies in the same project, so maybe I'm doing something that's not possible, but I'm sure I'm just missing a little something in the Spring configuration.
My questions/issues are:
I don't want the Spring Integration project to know anything about the SpringBatch configuration other than the job its kicking off. I have found a good way to do that reference to the Job Bean without getting my entire batch configuration loading.
Should I keep these two projects separated or would it be better to combine them since I have two-way communication between both.
How should the Job be launch from the integration project. We're using the spring-batch-integration project with JobLaunchRequest and JobLauncher. This seems to run it in the same process as the Spring Integration project and I'm missing a lot of my SpringBootBatch projects initialization
Should I be using a CommandLineRunner instead to force it to another process.
Is SpringApplication.run(BatchConfiguration.class) the answer?
Looking for some general project configuration setup to meet these requirements.
Spring Cloud Data Flow in combination with Spring Cloud Task does exactly what you're asking. It launches Spring Cloud Task applications (which can contain batch jobs) as new processes on the platform you choose. I'd encourage you to check out that project here: http://cloud.spring.io/spring-cloud-dataflow/
Related
I have multiple jobs for my Spring Batch application, but only a single job uses some specific Spring Boot auto configuration features:
a job that uses spring-data-jpa auto configuration, to configure a database for business transactions (not for Spring Batch management)
a job that does not use the database at all
I have packaged both jobs in the same unit because it makes sense from business perspective. Both jobs will work together and the output of one job will be the input of the other job.
Is it possible to disable database specific auto configuration when I run the second job?
I just tried using profiles and I have disabled the autoconfiguration for a specific profile. I am pretty happy with this solution but I wonder if there are other solutions?
This is similar to trying to lazy load beans specific to a given job: How to apply something like #Lazy to Spring Batch?. While the Spring profiles feature may fix your issue, I believe it is working around the root issue which is packaging all jobs in a monolithic way.
I would package each job separately and this problem (and many others) disappears by design. There are several advantages to this approach:
Independent lifecyle management (bugs, features, etc)
Flexible deployment
Separate logs
Separate configurations (as in the current issue)
Easier/Better scalability
And all the good reasons to make one thing do one thing and do it well.
I'm dipping my toes into the microservices, is spring boot batch applicable to the following requirements?
Files of one or multiple are read from a specific directory in Linux.
Several operations like regex, build new files, write the file and ftp to a location
Send email during a process fail
Using spring boot is confirmed, now the question is
Should I use spring batch or just core spring framework?
I need to integrate with Control-M to trigger the job. Can the Control-M be completely removed by using Spring batch library? As we don't know when to expect the files in the directory.
I've not seen a POC with these requirements. Would someone provide an example POC or an affirmation this could be achieved with Spring batch?
I would use Spring Batch for that use case. Not only does it provide out of the box components for reading, processing, and writing files, it adds a lot more for error handling, scalability, etc. All of those things you'd probably end up wiring up by yourself if you go without Spring Batch.
As for being launched via Control-M, yes MANY large customers use Control-M to launch their jobs. Unfortunately, I've never done it myself so I cannot provide any details on the mechanics, but if Control-M can either launch a script or call a REST API, you can launch a job with it.
I would suggest you, go for spring batch as it has much-inbuilt functionality which will be provided to you for file reading and writing to your required location. Even you will be able to handle record skipping requirement. Your mail triggering requirement will be handled by Control M. You just need to decide one exit code for your handled exception and on the basis of that exit code you can trigger the mail to respective members. And there are many other features which will be helpful if you go for spring batch.
I have a large spring project, using xml configuration. I'm looking for a quick way to verify changes to the xml configuration.
I can load the whole project locally - the problem is this takes more than 5 minutes, loads a huge amount of data.
My XML editor catches XML formatting errors.
I'm looking for something intermediate - to catch obvious problems like references to beans that aren't defined, or calling constructors with the wrong arguments. Is there a quick way to do this, without having to actually invoke all the constructors and bring up the whole environment?
I'm building with Maven and editing with Eclipse, although my question isn't specific to either.
Since you already use Eclipse, you could try Spring Tool Suite (comes either standalone or as an add-on). It's essentially Eclipse with extra Spring-specific features, like Beans Validator. I'm not sure how thorough the validation is, but it should catch most configuration problems.
It's maintained by SpringSource so its integration with Spring "just works" and it's guaranteed not be more or less in sync with Spring Framework's release cycle.
Beanoh :
http://beanoh.org/overview.html#Verify
this project does exactly what I'm looking for. Verify obvious problems with spring config, but without the overhead of initializing everything.
You can use a Spring testing support to integration test your Spring configuration. However if the loading of the context is taking 5 mins, then the tests will also take the same amount of time. Spring does cache the context so if you have multiple tests using the same set of Spring contexts, then once cached the tests should be very quick.
I can suggest a few ways to more efficiently test your configuration:
Organize your project in modules, with each module being responsible for its own Spring configuration - this way, each module can be independently developed and tested.
If you have a modular structure, the testing can be more localized by mocking out the dependent modules, again this is for speed.
I am new to Spring Integration and I am considering using it in order to poll a directory for new files in order to process those files.
My question is: is Spring Integration some sort of daemon one can launch and that one can use in order to poll a directory?
Is this is possible can someone please direct me to relevant section of the official documentation on how to launch Spring Integration?
All you need is to have a main method (or a WAR file if you want to deploy to Tomcat or another servlet container) that creates a Spring ApplicationContext (e.g. new ClassPathXmlApplicationContext("file-poller.xml"))
It can run with a cron trigger, fixed-rate or fixed-delay trigger.
JMX operations can be exposed on Spring Integration's File adapter (or any adapter) by simply adding a single config element (e.g. <mbean-export>).
Bottom line: you REALLY do not need an ESB if you simply want a File poller to run continuously. You can have a single small config file and one line of code in a main method.
Visit the samples for more info: https://github.com/springsource/spring-integration-samples (look under basic/file specifically)
Hope that helps,
Mark
Spring Integration is a part of framework, its not a programm or daemon.
What you cant do — is to configure Spring Integration to poll a directory, lunch JVM with Spring onboard and poller will do what you want.
You can start with this blog post.
More samples
Relevant section of documentation
Am a total beginner with spring framework and trying to know if it even fits my use case, before investing time learning it.
I'm responsible for a stand alone java project(used as a jar by a server) which basically serves requests from a server, and in turn makes service calls to various internal services.
This standalone java project, currently has all of its service calls hard coded. I want to use Spring to inject dependencies so I can make this stuff testable.
I have no idea how spring works. Does it even hold for standalone jars or is it only for 'running applications'?
If I make my standalone project 'spring enabled', when the server uses my jar, will it automagically work by creating beans or is there some requirement from the server side?
In short, yes, you can use Spring in a standalone jar-application ("console application", if you will), we do it all the time at work. You just need to create the ApplicationContext yourself when your application starts, see for example here: http://www.devdaily.com/blog/post/java/load-spring-application-context-file-java-swing-application
This is just one example I pulled straight out of Google, there are probably numerous others. Still, you really need to read at least the basics from the Spring documentation to get started, otherwise you'll probably hit a wall pretty soon.