I have read this forum with lots of examples on how to use jobparameters but still got stuck as i am getting null value when retrieving jobparameter
Below is my code so far
Controller :
#RestController
#CrossOrigin
#RequestMapping("/bulkimport/api/")
public class BatchController {
#Autowired
public JobLauncher jobLauncher;
#Autowired
public Job bulkImportJob;
#RequestMapping(value = "/bulkProcessjob", method = RequestMethod.POST)
public void batchImportProcess(#RequestHeader(value = "X-Auth-Token") String jwtToken,
#RequestParam("excelfile") final MultipartFile excelfile
) throws IOException{
String path = new ClassPathResource("importFileFolder/").getURL().getPath();
File fileToImport = new File(path + excelfile.getOriginalFilename());
try {
Map<String, JobParameter> jobParametersMap = new HashMap<String, JobParameter>();
jobParametersMap.put("processName", new JobParameter("processName"));
jobParametersMap.put("operation", new JobParameter("operation"));
jobParametersMap.put("filePath", new JobParameter(fileToImport.getAbsolutePath()));
System.out.println(fileToImport.getAbsolutePath()+"filePath:::::::::::");
jobLauncher.run(bulkImportJob, new JobParameters(jobParametersMap));
}catch(Exception e){
}
}
}
Config class:
public String filePath;
#Bean(name = "bulkImportJob")
public Job bulkImportJob() {
return jobBuilderFactory.get("bulkImportJob").incrementer(new RunIdIncrementer()).flow(bulkImportProcessStep()).end().build();
}
#Bean
public Step bulkImportProcessStep() {
return stepBuilderFactory.get("bulkImportProcessStep").<String, String> chunk(1).reader(validateFileHeader(filePath)).writer(new TempWriter()).build();
}
#Bean
#StepScope
BulkUploadExcelHeader validateFileHeader(#Value("#{jobParameters[filePath]}") String filePath) {
System.out.println("filePath value::::::::::::::::::::::::::::::::"+filePath);
return new BulkUploadExcelHeader(filePath);
}
in above print statement filePath value is printed as null
can anyone please suggest what is wrong in above code
Most probably you need to enclose filePath with single quotes:
validateFileHeader(#Value("#{jobParameters['filePath']}")
Related
I have a directory having text files. I want to process files and write data into db. I did that by using MultiResourceItemReader.
I have a scenario like whenever file is coming, the first step is to save file info, like filename, record count in file in a log table(custom table).
Since i used MultiResourceItemReader, It's loading all files once and the code which i wrote is executing once in server startup. I tried with getCurrentResource() method but its returning null.
Please refer below code.
NetFileProcessController.java
#Slf4j
#RestController
#RequestMapping("/netProcess")
public class NetFileProcessController {
#Autowired
private JobLauncher jobLauncher;
#Autowired
#Qualifier("netFileParseJob")
private Job job;
#GetMapping(path = "/process")
public #ResponseBody StatusResponse process() throws ServiceException {
try {
Map<String, JobParameter> parameters = new HashMap<>();
parameters.put("date", new JobParameter(new Date()));
jobLauncher.run(job, new JobParameters(parameters));
return new StatusResponse(true);
} catch (Exception e) {
log.error("Exception", e);
Throwable rootException = ExceptionUtils.getRootCause(e);
String errMessage = rootException.getMessage();
log.info("Root cause is instance of JobInstanceAlreadyCompleteException --> "+(rootException instanceof JobInstanceAlreadyCompleteException));
if(rootException instanceof JobInstanceAlreadyCompleteException){
log.info(errMessage);
return new StatusResponse(false, "This job has been completed already!");
} else{
throw new ServiceException(errMessage);
}
}
}
}
BatchConfig.java
#Configuration
#EnableBatchProcessing
public class BatchConfig {
private JobBuilderFactory jobBuilderFactory;
#Autowired
public void setJobBuilderFactory(JobBuilderFactory jobBuilderFactory) {
this.jobBuilderFactory = jobBuilderFactory;
}
#Autowired
StepBuilderFactory stepBuilderFactory;
#Value("file:${input.files.location}${input.file.pattern}")
private Resource[] netFileInputs;
#Value("${net.file.column.names}")
private String netFilecolumnNames;
#Value("${net.file.column.lengths}")
private String netFileColumnLengths;
#Autowired
NetFileInfoTasklet netFileInfoTasklet;
#Autowired
NetFlatFileProcessor netFlatFileProcessor;
#Autowired
NetFlatFileWriter netFlatFileWriter;
#Bean
public Job netFileParseJob() {
return jobBuilderFactory.get("netFileParseJob")
.incrementer(new RunIdIncrementer())
.start(netFileStep())
.build();
}
public Step netFileStep() {
return stepBuilderFactory.get("netFileStep")
.<NetDetailsDTO, NetDetailsDTO>chunk(1)
.reader(new NetFlatFileReader(netFileInputs, netFilecolumnNames, netFileColumnLengths))
.processor(netFlatFileProcessor)
.writer(netFlatFileWriter)
.build();
}
}
NetFlatFileReader.java
#Slf4j
public class NetFlatFileReader extends MultiResourceItemReader<NetDetailsDTO> {
public netFlatFileReader(Resource[] netFileInputs, String netFilecolumnNames, String netFileColumnLengths) {
setResources(netFileInputs);
setDelegate(reader(netFilecolumnNames, netFileColumnLengths));
}
private FlatFileItemReader<NetDetailsDTO> reader(String netFilecolumnNames, String netFileColumnLengths) {
FlatFileItemReader<NetDetailsDTO> flatFileItemReader = new FlatFileItemReader<>();
FixedLengthTokenizer tokenizer = CommonUtil.fixedLengthTokenizer(netFilecolumnNames, netFileColumnLengths);
FieldSetMapper<NetDetailsDTO> mapper = createMapper();
DefaultLineMapper<NetDetailsDTO> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(tokenizer);
lineMapper.setFieldSetMapper(mapper);
flatFileItemReader.setLineMapper(lineMapper);
return flatFileItemReader;
}
/*
* Mapping column data to DTO
*/
private FieldSetMapper<NetDetailsDTO> createMapper() {
BeanWrapperFieldSetMapper<NetDetailsDTO> mapper = new BeanWrapperFieldSetMapper<>();
try {
mapper.setTargetType(NetDetailsDTO.class);
} catch(Exception e) {
log.error("Exception in mapping column data to dto ", e);
}
return mapper;
}
}
I am stuck on this scenario, Any help appreciated
I don't think MultiResourceItemReader is appropriate in your case. I would run a job per file for all the reasons of making one thing do one thing and do it well:
Your preparatory step will work by design
It would be easier to run multiple jobs in parallel and improve your file ingestion throughput
In case of failure, you would only restart the job for the failed file
EDIT: add an example
Resource[] netFileInputs = ... // same code that looks for file as currently in your reader
for (Resource netFileInput : netFileInputs) {
Map<String, JobParameter> parameters = new HashMap<>();
parameters.put("netFileInput", new JobParameter(netFileInput.getFilename()));
jobLauncher.run(job, new JobParameters(parameters));
}
I have a scenario for Springboot batch application where I need to read the CSV-11 and write to another CSV-2, the problem arises when I need to get the CSV-1 from a REST endpoint, for e.g. when I initiate the job it should fetch a CSV-1 from an endpoint and then continue the batch processing to write to CSV-2.
But it seems like to process a CSV-1 we need to feed the 'Resource' in advance while application startup [I am fairly new to batch so not sure if it's 100% correct].
Can anyone please guide me the right approach to resolve this?
EDIT : Adding code, I was able to resolve it but need advice if it's the right way to do things (please ignore the hard-coded data).
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
private JobBuilderFactory jobbuilder;
#Autowired
private StepBuilderFactory stepbuilder;
#Autowired
private CSVProcessor processor;
#Autowired
private CustomSkipPolicy customSkipPolicy;
#Autowired
private CSVResponse response;
#Bean(name="csv")
public Job job_csv() throws Exception {
Step step = stepbuilder
.get("csv-step")
.<Person, Person>chunk(5)
.reader(new CSVReader(response.getResource()))
.processor(processor)
.writer(new CSVWriter().write(response.getFilename()))
.faultTolerant()
.skipPolicy(customSkipPolicy)
.listener(new StepListener())
.build();
return jobbuilder
.get("csv-job")
.incrementer(new RunIdIncrementer())
.listener(new JobListener())
.flow(step)
.end()
.build();
}
}
#Slf4j
public class CSVReader extends FlatFileItemReader<Person> {
public CSVReader(Resource resource) throws Exception {
super();
setResource(resource);
setStrict(false);
setLinesToSkip(1);
doOpen();
DelimitedLineTokenizer dlt = new DelimitedLineTokenizer();
dlt.setNames(new String[] {"id","first_name","last_name","email","gender","ip_address","dob"});
dlt.setDelimiter(",");
dlt.setStrict(false);
BeanWrapperFieldSetMapper<Person> fsp = new BeanWrapperFieldSetMapper<>();
fsp.setTargetType(Person.class);
DefaultLineMapper<Person> dlp = new DefaultLineMapper<>();
dlp.setLineTokenizer(dlt);
dlp.setFieldSetMapper(fsp);
setLineMapper(dlp);
}
}
public class CSVWriter {
private static final String DATA_PROCESSED = "C:/data/processed";
public FlatFileItemWriter<Person> write(String filename) throws Exception {
FlatFileItemWriter<Person> writer = new FlatFileItemWriter<>();
writer.setResource(resource(filename));
writer.setHeaderCallback(new Header());
BeanWrapperFieldExtractor<Person> fe = new BeanWrapperFieldExtractor<>();
fe.setNames(new String[] { "id", "first_name", "last_name", "age" });
DelimitedLineAggregator<Person> dla = new DelimitedLineAggregator<>();
dla.setDelimiter(",");
dla.setFieldExtractor(fe);
writer.setLineAggregator(dla);
return writer;
}
private Resource resource(String filename) throws IOException {
String processed_file = filename.replace(".csv", "").concat("_PROCESSED").concat(".csv");
if (Files.notExists(Paths.get(DATA_PROCESSED), new LinkOption[] { LinkOption.NOFOLLOW_LINKS })) {
Files.createDirectory(Paths.get(DATA_PROCESSED));
}
if (Files.notExists(Paths.get(DATA_PROCESSED + "/" + processed_file),
new LinkOption[] { LinkOption.NOFOLLOW_LINKS })) {
Files.createFile(Paths.get(DATA_PROCESSED + "/" + processed_file));
}
return new FileSystemResource(DATA_PROCESSED + "/" + processed_file);
}
}
#Component
#Slf4j
public class CSVResponse {
private static final String DATA_RECEIVE = "C:/data/receive";
private String filename;
public Resource getResource() throws IOException {
ResponseEntity<Resource> resp = new RestTemplate().getForEntity("http://localhost:8081/csv", Resource.class);
log.info("File Received : "+resp.getBody().getFilename());
setFilename(resp.getBody().getFilename());
Path path = Paths.get(DATA_RECEIVE);
if(!Files.exists(path, new LinkOption[]{ LinkOption.NOFOLLOW_LINKS})) {
Files.createDirectory(path);
}
Files.copy(resp.getBody().getInputStream(), Paths.get(DATA_RECEIVE + "/" +resp.getBody().getFilename()), StandardCopyOption.REPLACE_EXISTING);
return resp.getBody();
}
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
}
You can implement item reader with StepExecutionListener interface, and you have to implement the fetching CSV-1 file via RestTemplate in beforeStep() method of the item reader.
You can refer how to use RestTemplate from https://www.baeldung.com/rest-template
I'm creating a Spring Batch Job to populate Data into a Database table from a given CSV file.
I created a customized FlatFileItemReader.
my problem is that the read() method is invoked only one time so only the first line of my CSV file is inserted into the database.
#Configuration
#EnableBatchProcessing
public class SpringBatchConfig {
private MultipartFile[] files;
#Bean
public Job job(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
ItemReader<MyModelEntity> itemReader,
ItemWriter<MyModelEntity> itemWriter) {
Step step = stepBuilderFactory.get("Load-CSV-file_STP")
.<MyModelEntity, MyModelEntity > chunk(12)
.reader(itemReader)
.writer(itemWriter).build();
return jobBuilderFactory.get("Load-CSV-Files").
incrementer(new RunIdIncrementer()) /
.start(step)
.build();
}
#Bean
ItemReader<MyModelEntity> myModelCsvReader() throws Exception {
return new MyModelCsvReader();
}
}
the myModelCsvReader
#Component
#StepScope
public class MyModelCsvReader implements ItemReader<MyModelEntity>{
#Value("#{jobParameters['SDH']}")
private String sdhPath;
private boolean batchJobState= false;
#Autowired
MyModelFieldSetMapper myModelFieldSetMapper;
public LineMapper<MyModelEntity> lineMapper() throws Exception {
DefaultLineMapper<MyModelEntity> defaultLineMapper = new
DefaultLineMapper<MyModelEntity>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]
{
"clientId","ddId","institName","progName",
"qual","startDate","endDate","eType", "country","comments"
});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(myModelFieldSetMapper);
return defaultLineMapper;}
#Override
public MyModelEntity read()
throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
//if(!batchJobState )
{
FlatFileItemReader<MyModelEntity> flatFileItemReader = new
FlatFileItemReader<MyModelEntity>();
flatFileItemReader.setMaxItemCount(2000);
flatFileItemReader.setResource(new UrlResource("file:\\"+sdhPath));
flatFileItemReader.setName("CSV-Reader");
flatFileItemReader.setLinesToSkip(1);
flatFileItemReader.setLineMapper(lineMapper());
flatFileItemReader.open(new ExecutionContext());
batchJobState=true;
return flatFileItemReader.read();
}
// return null;
}
}
the FieldSetMapper Implementation
#Component
public class MyModelFieldSetMapper implements FieldSetMapper<MyModelEntity> {
//private SiteService siteService =BeanUtil.getBean(SiteServiceImpl.class);
#Autowired
private SiteService siteService;
#Override
public MyModelEntity mapFieldSet(FieldSet fieldSet ) throws BindException {
if(fieldSet == null){
return null;
}
MyModelEntity educationHistory = new MyModelEntity();
// setting MyModelAttributes Values
return myModel;
}
}
any conribution is welcomed . thanks
// thereader after extending FlatFileItemReader
#Component
#StepScope
public class CustomUserItemReader extends FlatFileItemReader<User> {
#Value("#{jobParameters['UserCSVPath']}")
private String UserCSVPath;
private boolean batchJobState;
public CustomUserItemReader() throws Exception {
super();
setResource(new UrlResource("file:\\"+UserCSVPath));
setLineMapper(lineMapper());
afterPropertiesSet();
setStrict(false);
}
public LineMapper<User> lineMapper() throws Exception {
DefaultLineMapper<User> defaultLineMapper =
new DefaultLineMapper<User>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]{"name", "dept",
"salary","endDate"});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(new CustomUserFieldSetMapper());
//defaultLineMapper.setFieldSetMapper(fieldSetMapper);
return defaultLineMapper;}
#Override
public User read()
throws Exception, UnexpectedInputException, ParseException,
NonTransientResourceException {
//if(!batchJobState )
{
flatFileItemReader).size())
// flatFileItemReader.setMaxItemCount(2000);
this.setResource(new UrlResource("file:\\"+UserCSVPath));
this.setName("CSV-Reader");
this.setLinesToSkip(1);
//flatFileItemReader.setLineMapper(lineMapper());
this.open(new ExecutionContext());
User e = this.read();
batchJobState = true;
return e ;
}
// return null;
}
public
String getUserCSVPath() {
return UserCSVPath;
}
public
void setUserCSVPath(String userCSVPath) {
UserCSVPath = userCSVPath;
}
}
Thanks for all your suggestions, even if i have implemented ItemReader<>. I fixed the problem by moving the instantiation of the FlatFileItemReader Out from the read() method.
that was creating a new FlatFileItemReader in each loop and reading only the first line for each object created .
thanks
I have Spring Batch job where I am passing some values between two stpes. I set the value in Job Context in Step1 and now trying to read from RepositoryItemReader in Step2. There is #BeforeStep method where I am able to read value set in context. But I am setting up my repository along with method name and args in #PostConstruct annotated method which is executed before #BeforeStep annotated method.
What is the best way to read param in ReposiotryItem from JobExecution Context?
#Component
#JobScope
public class MyItemReader extends RepositoryItemReader<Scan> {
#Autowired
private MyRepository repository;
private Integer lastIdPulled = null;
public MyItemReader() {
super();
}
#BeforeStep
public void initializeValues(StepExecution stepExecution) {
Integer value = stepExecution.getJobExecution().getExecutionContext().getInt("lastIdPulled");
System.out.println(">>>>>>>> last_pulled_id = " + value);
}
#PostConstruct
protected void init() {
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("id", Direction.ASC);
this.setRepository(this.repository);
this.setSort(sorts);
this.setMethodName("findByGreaterThanId"); // You should sepcify the method which
//spring batch should call in your repository to fetch
// data and the arguments it needs needs to be
//specified with the below method.
List<Object> methodArgs = new ArrayList<Object>();
if(lastIdPulled== null || lastIdPulled<=0 ){
lastScanIdPulled = 0;
}
methodArgs.add(lastIdPulled);
this.setArguments(methodArgs);
}
}
Your reader needs to be #StepScoped instead of #JobScoped. Even though you're accessing the job context, the value is not available in the context until the previous step finishes. If you #StepScope your reader then it won't initialize until the step it is part of starts up and the value is available.
Another option is to construct the reader as a #Bean definition in a #Configuration file but the idea is the same. This uses SpEL for late binding.
#Configuration
public class JobConfig {
// Your item reader will get autowired into this method
// so you don't have to call it
#Bean
public Step myStep(MyItemReader myItemReader) {
//build your step
}
#Bean
#StepScope
public MyItemReader myItemReader(#Value("#{jobExecutionContext[partitionKey]}") Integer lastIdPulled) {
MyItemReader reader = new MyItemReader();
// Perform #PostConstruct tasks
return reader;
}
}
I was able to figure out how to solve your problem, and mine, without having to create a #Bean definition in #Configuration file and without using #PostConstruct.
Instead of using #PostConstruct, just set them in the constructor of the class and in your #BeforeStep set the arguments as shown below:
#Component
#StepScope
public class MyItemReader extends RepositoryItemReader<Scan> {
#Autowired
public MyItemReader(MyRepository myRepository) {
super();
this.setRepository(MyRepository myRepository);
this.setMethodName("findByGreaterThanId");
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("id", Direction.ASC);
this.setSort(sorts);
}
#BeforeStep
public void initializeValues(StepExecution stepExecution) {
Integer value = stepExecution.getJobExecution().getExecutionContext().getInt("lastIdPulled");
System.out.println(">>>>>>>> last_pulled_id = " + value);
List<Object> methodArgs = new ArrayList<Object>();
if(lastIdPulled== null || lastIdPulled<=0 ){
lastScanIdPulled = 0;
}
methodArgs.add(lastIdPulled);
this.setArguments(methodArgs);
}
}
I am creating a simple spring boot application in which I am trying to access an external config.properties file.
IndexController.java
#Controller
public class IndexController {
XmlOperation xmlOperation = new XmlOperation();
#RequestMapping("/")
public String greeting() {
return "greeting";
}
#RequestMapping(params = "btnOpen", method = RequestMethod.POST)
public String uploadFile(#RequestParam("file") MultipartFile file, Model model) {
try {
InputStream is = file.getInputStream();
model.addAttribute("fileContent", xmlOperation.readXml(is));
} catch (IOException e) {
System.out.println(e.getMessage());
}
return "greeting";
}
}
XmlOperation.java
#PropertySource("classpath:config.properties")
public class XmlOperation {
#Autowired
Environment env;
public String readXml(InputStream is) throws IOException {
System.out.println(env.getProperty("filepath"));
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, StandardCharsets.UTF_8);
String fileContent = writer.toString();
return fileContent;
}
The config.properties file is situated in src/main/resources. I am not able to get the value from the property file.
Any help would be appreciated...
Well config.properties file in src/main/resources is alright but why do you initialize:
XmlOperation xmlOperation = new XmlOperation();
in IndexController? And I am also not sure whether XmlOperation is spring component (you have only #PropertySource over XmlOperation in the question).
Basically I would make XmlOperation as a spring #Component and with IoC inject this component into IndexController.
public String readXml(InputStream is) in XmlOperation behaves like standard service and I would create property filepath and inject value from configuration file (config.properties) with #Value annotation.
FULL EXAMPLE:
#Controller
public class IndexController {
#Autowired
private XmlOperation xmlOperation;
#RequestMapping("/")
public String greeting() {
return "greeting";
}
#RequestMapping(params = "btnOpen", method = RequestMethod.POST)
public String uploadFile(#RequestParam("file") MultipartFile file, Model model) {
try {
InputStream is = file.getInputStream();
model.addAttribute("fileContent", xmlOperation.readXml(is));
} catch (IOException e) {
System.out.println(e.getMessage());
}
return "greeting";
}
}
#Component
#PropertySource("classpath:config.properties")
public class XmlOperation {
// use this when XmlOperation is #Configuration bean and you want to create #Bean-s e.g
// #Autowired
// Environment env;
// for your case inject property like this
#Value("${filepath}")
private String filepath;
public String readXml(InputStream is) throws IOException {
// dont use this
//System.out.println(env.getProperty("filepath"));
// rather this
System.out.println(filepath);
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, StandardCharsets.UTF_8);
String fileContent = writer.toString();
return fileContent;
}
}