I have a Spring boot MVC and batch application. Both the batch and MVC share the DAO and Service layers so they are in the same war file. They are deployed into 4 cloud servers and there is a load balance and vip configured for the UI application. So the MVC application is fine.
The problem is as part of the batch i do FTP of a file to an external server and that external server FTPs the processed file back. The processed file comes back only to one among the 4 servers. So I want the batch to run only on 1 server. How do i suppress the batch from executing in the other servers.
Solution becomes easier as your 4 instances are running on 4 different cloud severs. The starting point of the batch can be a file poller. So if the file is dropped into the polled directory on server 1, the batch job on server 1 will be invoked. The other instances do nothing as there is no file dropped on that server.
You need to integrate file poller before spring batch. Something like this - http://docs.spring.io/spring-batch/reference/html/springBatchIntegration.html
<int:channel id="inboundFileChannel"/>
<int:channel id="outboundJobRequestChannel"/>
<int:channel id="jobLaunchReplyChannel"/>
<int-file:inbound-channel-adapter id="filePoller"
channel="inboundFileChannel"
directory="file:/tmp/myfiles/"
filename-pattern="*.csv">
<int:poller fixed-rate="1000"/>
</int-file:inbound-channel-adapter>
<int:transformer input-channel="inboundFileChannel"
output-channel="outboundJobRequestChannel">. <bean class="io.spring.sbi.FileMessageToJobRequest">
<property name="job" ref="personJob"/>
<property name="fileParameterName" value="input.file.name"/>
</bean>
</int:transformer>
<batch-int:job-launching-gateway request-channel="outboundJobRequestChannel"
reply-channel="jobLaunchReplyChannel"/>
This can be one of many approaches but a way to achieve is keep a value in property file and set it's value to Boolean true
Now handle your batch to run only if property file value is true.
This way it gives you flexibility to change the server you want to handle batch job.
Related
Having deployed the activemq-web-console war into a Tomcat embedded application how can one make it connect to an existing broker rather than create a new one?
The war comes with a set of predefined configurations, in particular, the WEB-INF/activemq.xml contains a configuration for the BrokerService
<broker brokerName="web-console" useJmx="true" xmlns="http://activemq.apache.org/schema/core">
<persistenceAdapter><kahaDB directory="target/kahadb"/></persistenceAdapter>
<transportConnectors>
<transportConnector uri="tcp://localhost:12345"/>
</transportConnectors>
</broker>
used from webconsole-embedded.xml in the following manner:
<bean id="brokerService" class="org.apache.activemq.xbean.BrokerFactoryBean">
<property name="config" value="/WEB-INF/activemq.xml"/>
</bean>
This configuration creates a new instance of BrokerService and tries to start the broker.
It is reported that the web console can be used to monitor an existing broker service rather than creating a new one. For this one should set the following properties somewhere:
webconsole.type=properties
webconsole.jms.url=tcp://localhost:61616
webconsole.jmx.url=service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-trun
The questions is, where does one have to set these properties within the Tomcat embedded app and which XML changes in the above have to be performed for them to be used. I cannot find any sensible explanation how to configure it, and a BrokerService instance seems to be required by the remaining spring config.
Any ideas?
Please do not suggest to use hawtio instead!
I had the same problem today. You can start the webconsole in "properties" mode which gives you the oppertunity to connect over jmx.
I added following java arguments to our Jboss 6.1 and it worked immediatley. I didn't change any of the xmls (works out of the box)...
Example:
-Dwebconsole.type=properties -Dwebconsole.jms.url=tcp://<hostname>:61616 -Dwebconsole.jmx.url=service:jmx:rmi:///jndi/rmi://<hostname>:1090/jmxrmi -Dwebconsole.jmx.user=admin -Dwebconsole.jmx.password=123456
Also discussed here: https://svn.apache.org/repos/infra/websites/production/activemq/content/5.7.0/web-console.html
I am new to batch processing. I am trying to start with simple scheduler and job. But i am confused b/w
spring batch vs quartz jobs. My understanding is
Quartz :- quartz provides both frameworks i.e scheduler framework and job framework(in case I do not want to use spring batch jobs). Right ?
Spring Batch :- It only provides the job framework . I have always send using Quatz schecduler to schedule spring batch jobs.
Does spring provides its own scheduler also ?
Quartz is a scheduling framework. Like "execute something every hour or every last Friday of the month"
Spring Batch is a framework that defines that "something" that will be executed.
You can define a job, that consists of steps. Usually, a step is something that consists of an item reader, an optional item processor, and an item writer, but you can define a custom step. You can also tell Spring Batch to commit every 10 items and a lot of other stuff.
You can use Quartz to start Spring Batch jobs.
So basically Spring Batch defines what should be done, and Quartz defines when it should be done.
There is answer for this question in official FAQ
How does Spring Batch differ from Quartz?
Is there a place for them both in a solution?
Spring Batch and Quartz have different goals. Spring Batch provides functionality for processing large volumes of data and Quartz provides functionality for scheduling tasks. So Quartz could complement Spring Batch, but are not excluding technologies. A common combination would be to use Quartz as a trigger for a Spring Batch job using a Cron expression and the Spring Core convenience SchedulerFactoryBean.
Does spring provides its own scheduler also?
Yes, using Spring TaskScheduler as follows:
<task:scheduled-tasks>
<task:scheduled ref="runScheduler" method="run" fixed-delay="5000" />
</task:scheduled-tasks>
<task:scheduled-tasks>
<task:scheduled ref="runScheduler" method="run" cron="*/5 * * * * *" />
</task:scheduled-tasks>
full example
With Quartz Scheduler as follows:
<!-- run every 10 seconds -->
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<bean id="cronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="jobDetail" />
<property name="cronExpression" value="*/10 * * * * ?" />
</bean>
</property>
</bean>
full example
Spring Batch: reads data from a datasource (table in a database, flat file, etc), processes that data. Then stores the data in another datasource and may be in another format.
I have made a tutorial in my blog on how to integrate Spring Boot 2, Spring batch and Quartz.
You can integrate Spring boot and spring batch and skip the Quartz integration.
Quartz is a scheduler that schedules a task in the future and it has its own metadata tables to manage the state of the jobs.
I have config xml based spring application for which I have moved proprties required at start up time in database. It was very difficult to manage hundreds in property file and that is why database is introduced. To read properties a spring restful service is developed to return a map of all properties required at start up time.
I want to know how to replace properties reading from a map to spring context file e.g. ${config.service.url} should be polulated from a map read via web service.
One option I considered is to upgrade to Annotation based and start using MapPropertySource and Environment interface as environment.getRequiredProperty("config.service.url"). However upgrading to Annotation based is a big impact on project and is no at this time.
Second option that I am looking forward is to have a customised PropertyPlaceholderConfigurer.
Any pointer/help on this will be great.
Cheers,
Amber
You could define a PropertyPlaceholderConfigurer, but instead of specifying a file location, you can pass the properties directly as returned by your restful service.
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="properties" .../>
</bean>
We have a spring integration application which will monitor on an incoming folder then process the files.
When the application is down for maintenance or some other reason the incoming folder is filled with 100K files by upstream application.
When restart application it is getting frozen it is not processing incoming files may be trying to load all the incoming files.
Here is configuration
<file:inbound-channel-adapter id="inFiles" channel="inFilesin" directory="file:${incoming.folder}"
queue-size="300" filename-regex="(?i)^(?!.*writing) " auto-startup="true" auto-create-directory="false" >
<int:poller id="fw.fileInboudPoller" fixed-rate="1" receive-timeout="3" time-unit="SECONDS"
max-messages-per-poll="10" task-executor="taskExecutor" />
</file:inbound-channel-adapter>
<task:executor id="taskExecutor" pool-size="10-20" queue-capacity="20" rejection-policy="CALLER_RUNS" />
Appreciate your help.
Thanks,
Mohan
Suggest change fixed-rate to fixed-delay.
Your files are processed very slow and the first option sais that the new task should be started just afer that time (in your case 1 sec.).
Another problem - rejection-policy="CALLER_RUNS". In this casem if your thead queue will be exhausted (and it is in you case of 100K files), the scheduled thread does the real work.
Poller, to schedule tasks, uses ThreadPoolTaskScheduler with size 10. So, with this "havy-load" your app may be frozen, because that pool is shared for all application.
So, try to use fixed-delay. In this case your app won't be frozen, but files will be processed slower.
Maybe this can help you: <int:resource-inbound-channel-adapter> ?
I am using Spring Integration to poll a directory for a File, process this file in a service class, write this file to an output directory and then delete the original file.
I have the following XML configuration:
<int-file:inbound-channel-adapter id="filesInChannel"
directory="file:${java.io.tmpdir}/input"
auto-create-directory="true" >
<int:poller id="poller" fixed-delay="1000" />
</int-file:inbound-channel-adapter>
<int:service-activator id="servicActivator"
input-channel="filesInChannel"
output-channel="filesOut"
ref="my_file_processing_service">
</int:service-activator>
<int-file:outbound-channel-adapter id="filesOut" auto-create-directory="true" delete-source-files="true" directory="file:${java.io.tmpdir}/output"/>
This polls the file, passes it to my processing_service and copies it to the outbound directory. However the original file is not being deleted. Does anyone have any idea as to why not?
I know that the question was asked a long time ago but maybe the answer will be useful to someone else.
The reason why the input file is not deleted is provided in the Spring Integration Reference:
The delete-source-files attribute will only have an effect if the
inbound Message has a File payload or if the FileHeaders.ORIGINAL_FILE
header value contains either the source File instance or a String
representing the original file path.
Your message does not contain this particular header. If you use one of the standard file transformers (FileToStringTransformer and FileToByteArrayTransformer) it will be set automatically. Alternatively you can set it manually using a header enricher.
Behind the scenes something like this is happening in the file transformers:
...
Message<?> transformedMessage = MessageBuilder.withPayload(result)
.copyHeaders(message.getHeaders())
.setHeaderIfAbsent(FileHeaders.ORIGINAL_FILE, file)
.setHeaderIfAbsent(FileHeaders.FILENAME, file.getName())
.build();
...
From the documentation http://static.springsource.org/spring-integration/reference/html/files.html
<int-file:outbound-gateway id="mover" request-channel="moveInput"
reply-channel="output"
directory="${output.directory}"
mode="REPLACE" delete-source-files="true"/>
I don't know how to do this on the inbound-channel-adapter(which I think makes sense)