I am developing Spring batch application on Spring Boot. Batch process is based on fire and forget, which means migration process is triggered by Http GET method and doesn't wait for result. There are 6 (six) csv files which containes migration data and these data must be processed and transferred to DB. Source files path is provided by application.yml file. Batch config (template) looks like as follows:
#Configuration
public class BeginCureBtachConfig {
#Value("$batch-files.begincure-source")
private String dataSourceFilePath;
#Autowired
IteamReader<BeginCure> itemReader;
#Autowired
ItemProcessor<BeginCure, BeginCure> itemProcessor;
#Bean Job beginCureJob(JobBuilderFactory jbFactory, StepBuilderDactory stp, ItemWriter<BeginCure> begnCure {
//// code goes here
}
#Bean
public FlatFileItemReader<BeginCure> beginCureItemReader() {
FlatFileItemReader<BeginCure> ffReader = new FlatFileItemReader<>();
ffReader.setResource(new FileSystemResource(dataSourceFilePath));
ffReader.setLineMapper(lineMapper());
return ffReader;
}
#Bean
public LineMapper<BeginCure> lineMapper () {
//..
}
}
The problem is when project is deployed on openshift, batch looks for files even migratin is not triggered.
In the code, I think this part of code giving some error ffReader.setResource(new FileSystemResource(dataSourceFilePath));
Is there any workaround or solution? Did anybody have this kind of problem?
Thank you in advance^^
Related
Below showing the project structure
Core Project
|-config project
|
|-Service project
After building the core project we get Service.jar file.
While running the service.jar am passing spring.config.additional.location as command line argument.
java -jar Service-1.0.jar --spring.config.additional-location=C:/Users/Administrator/Desktop/Springboot/
above spring.config.additional.location path having application.property file and some xml files.
I can able to read application property file in service project ,following logic
Application.propertes
external.config=C:/Users/Administrator/Desktop/Springboot/config/
Mian Class
#ImportResource(locations = {
"${external.config}"+"/spring/service-config.xml",
"${external.config}"+"/spring/datasource-config.xml"
})
public class ServiceMain {
public static void main(String[] args) {
ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(ServiceMain.class)
.build()
.run(args);
for (String name : applicationContext.getBeanDefinitionNames()) {
}
}
}
Similar kind of logic applied in config project is given below,its not working
#Configuration
public class ConfigurationFactory
{
#Value("${external.config}")
public String extConfPath;
public String REQ_CONF = extConfPath+"/Configuration.xml";
public static final String FILTER_XML_CONF = extConfPath+"/DocFilter.xml";
}
Is there any better way to do this? How can i read external application.properties in config project
Do we have any better way to do this in spring boot
As you are cleary developing a distributed web system the best practice is to used externalised configuration used by your different services allowing you to update settings without redeployment. Take a look at Spring Cloud Config
I have came across a situation where I need to fetch cron expression from database and then schedule it in Spring boot. I am fetching the data using JPA. Now the problem is in spring boot when I use #Scheduled annotation it does not allow me to use the db value directly as it is taken only constant value. So, what I am planning to do is to dynamically generate properties file and read cron expression from properties file. But here also I am facing one problem.The dynamically generated properties file created in target directory.
So I cant use it the time of program loading.
So can anyone assist me to read the dynamically generated file from the resource folder or how to schedule cron expression fetching from DB in spring boot?
If I placed all the details of corn expression in properties file I can schedule the job.
Latest try with dynamically generate properties file.
#Configuration
public class CronConfiguration {
#Autowired
private JobRepository jobRepository;
#Autowired
private ResourceLoader resourceLoader;
#PostConstruct
protected void initialize() {
updateConfiguration();
}
private void updateConfiguration() {
Properties properties = new Properties();
List<Job> morningJobList=new ArrayList<Job>();
List<String> morningJobCornExp=new ArrayList<String>();
// Map<String,String> map=new HashMap<>();
int num=1;
System.out.println("started");
morningJobList= jobRepository.findByDescriptionContaining("Morning Job");
for(Job job:morningJobList) {
//morningJobURL.add(job.getJobUrl());
morningJobCornExp.add(job.getCronExp());
}
for(String cron:morningJobCornExp ) {
properties.setProperty("cron.expression"+num+"=", cron);
num++;
}
Resource propertiesResource = resourceLoader.getResource("classpath:application1.properties");
try (OutputStream out = new BufferedOutputStream(new FileOutputStream(propertiesResource.getFile()))) {
properties.store(out, null);
} catch (Exception ex) {
// Handle error
ex.printStackTrace();
}
}
}
Still it is not able to write in properties file under resource folder.
Consider using Quartz Scheduler framework. It stores scheduler info in DB. No need to implement own DB communication, it is already provided.
Found this example: https://www.callicoder.com/spring-boot-quartz-scheduler-email-scheduling-example/
im developing a Project using Spring Batch and JavaConfig (no XML).
I am creating Jobs using an Autowired jobBuilderFactory.
is it somehow possible to set the Incrementer for the Factory globally ?
return jobBuilderFactory.get("jobName").incrementer(new RunIdIncrementer()).start(stepOne()).next(lastStep()).build();
sorry if this is a dump question but i am new to Spring Batch and did not find a working solution.
With XML config you would use bean definition inheritance, but you said you don't use XML.
Since there is no equivalent of XML bean definition inheritance with Java config (See details here: https://stackoverflow.com/a/23266686/5019386), you can create the RunIdIncrementer globally in your config and use it in job definitions:
public JobParametersIncrementer jobParametersIncrementer() {
return new RunIdIncrementer();
}
public JobBuilder getJobBuilder(String jobName) {
return jobBuilderFactory.get(jobName)
.incrementer(jobParametersIncrementer());
}
#Bean
public Job job1() {
return getJobBuilder("job1")
.start(step())
.build();
}
#Bean
public Job job2() {
return getJobBuilder("job2")
.start(step())
.build();
}
But again and as said in comments, you will end up having run.id values that are not consecutive for each job.
I'm new to Spring Framework and, indeed, I'm learning and using Spring Boot. Recently, in the app I'm developing, I made Quartz Scheduler work, and now I want to make Spring Integration work there: FTP connection to a server to write and read files from.
What I want is really simple (as I've been able to do so in a previous java application). I've got two Quartz Jobs scheduled to fired in different times daily: one of them reads a file from a FTP server and another one writes a file to a FTP server.
I'll detail what I've developed so far.
#SpringBootApplication
#ImportResource("classpath:ws-config.xml")
#EnableIntegration
#EnableScheduling
public class MyApp extends SpringBootServletInitializer {
#Autowired
private Configuration configuration;
//...
#Bean
public DefaultFtpsSessionFactory myFtpsSessionFactory(){
DefaultFtpsSessionFactory sess = new DefaultFtpsSessionFactory();
Ftp ftp = configuration.getFtp();
sess.setHost(ftp.getServer());
sess.setPort(ftp.getPort());
sess.setUsername(ftp.getUsername());
sess.setPassword(ftp.getPassword());
return sess;
}
}
The following class I've named it as a FtpGateway, as follows:
#Component
public class FtpGateway {
#Autowired
private DefaultFtpsSessionFactory sess;
public void sendFile(){
// todo
}
public void readFile(){
// todo
}
}
I'm reading this documentation to learn to do so. Spring Integration's FTP seems to be event driven, so I don't know how can I execute either of the sendFile() and readFile() from by Jobs when the trigger is fired at an exact time.
The documentation tells me something about using Inbound Channel Adapter (to read files from a FTP?), Outbound Channel Adapter (to write files to a FTP?) and Outbound Gateway (to do what?):
Spring Integration supports sending and receiving files over FTP/FTPS by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway. It also provides convenient namespace-based configuration options for defining these client components.
So, I haven't got it clear as how to follow.
Please, could anybody give me a hint?
Thank you!
EDIT:
Thank you #M. Deinum. First, I'll try a simple task: read a file from the FTP, the poller will run every 5 seconds. This is what I've added:
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(myFtpsSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/Entrada");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.csv"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source = new FtpInboundFileSynchronizingMessageSource(inbound);
source.setLocalDirectory(new File(configuracion.getDirFicherosDescargados()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
if(payload instanceof File){
File f = (File) payload;
System.out.println(f.getName());
}else{
System.out.println(message.getPayload());
}
}
};
}
Then, when the app is running, I put a new csv file intro "Entrada" remote folder, but the handler() method isn't run after 5 seconds... I'm doing something wrong?
Please add #Scheduled(fixedDelay = 5000) over your poller method.
You should use SPRING BATCH with tasklet. It is far easier to configure bean, crone time, input source with existing interfaces provided by Spring.
https://www.baeldung.com/introduction-to-spring-batch
Above example is annotation and xml based both, you can use either.
Other benefit Take use of listeners and parallel steps. This framework can be used in Reader - Processor - Writer manner as well.
I have a project and in it I'm using spring-boot-jdbc-starter and it automatically configures a DataSource for me.
Now I added camel-spring-boot to project and I was able to successfully create routes from Beans of type RouteBuilder.
But when I'm using sql component of camel it can not find datasource. Is there any simple way to add Spring configured datasource to CamelContext? In samples of camel project they use spring xml for datasource configuration but I'm looking for a way with java config. This is what I tried:
#Configuration
public class SqlRouteBuilder extends RouteBuilder {
#Bean
public SqlComponent sqlComponent(DataSource dataSource) {
SqlComponent sqlComponent = new SqlComponent();
sqlComponent.setDataSource(dataSource);
return sqlComponent;
}
#Override
public void configure() throws Exception {
from("sql:SELECT * FROM tasks WHERE STATUS NOT LIKE 'completed'")
.to("mock:sql");
}
}
I have to publish it because although the answer is in the commentary, you may not notice it, and in my case such a configuration was necessary to run the process.
The use of the SQL component should look like this:
from("timer://dbQueryTimer?period=10s")
.routeId("DATABASE_QUERY_TIMER_ROUTE")
.to("sql:SELECT * FROM event_queue?dataSource=#dataSource")
.process(xchg -> {
List<Map<String, Object>> row = xchg.getIn().getBody(List.class);
row.stream()
.map((x) -> {
EventQueue eventQueue = new EventQueue();
eventQueue.setId((Long)x.get("id"));
eventQueue.setData((String)x.get("data"));
return eventQueue;
}).collect(Collectors.toList());
})
.log(LoggingLevel.INFO,"******Database query executed - body:${body}******");
Note the use of ?dataSource=#dataSource. The dataSource name points to the DataSource object configured by Spring, it can be changed to another one and thus use different DataSource in different routes.
Here is the sample/example code (Java DSL). For this I used
Spring boot
H2 embedded Database
Camel
on startup spring-boot, creates table and loads data. Then camel route, runs "select" to pull the data.
Here is the code:
public void configure() throws Exception {
from("timer://timer1?period=1000")
.setBody(constant("select * from Employee"))
.to("jdbc:dataSource")
.split().simple("${body}")
.log("process row ${body}")
full example in github