i'm doing a simple batch job with Spring Batch and Spring Boot.
I need to read a flat file, separate the header data (first line) from the body data (rest of lines) for individual business logic processing and then write everything into a single file.
As you can see, the header has 5 params that have to be mapped to one class, and the body has 12 which have to be mapped to a different one.
I first thought of using FlatFileItemReader and skip the header. Then use the skippedLinesCallback to handle that line, but i couldn't figure out how to do it.
I'm new to Spring Batch and Java Config. If someone can help me writing a solution for my problem i would really aprecciate it!
I leave here the input file:
01.01.2017|SUBDCOBR|12:21:23|01/12/2016|31/12/2016
01.01.2017|12345678231234|0002342434|BORGIA RUBEN|27-32548987-9|FA|A|2062-
00010443/444/445|142,12|30/08/2017|142,01
01.01.2017|12345673201234|2342434|ALVAREZ ESTHER|27-32533987-9|FA|A|2062-
00010443/444/445|142,12|30/08/2017|142,02
01.01.2017|12345673201234|0002342434|LOPEZ LUCRECIA|27-32553387-9|FA|A|2062-
00010443/444/445|142,12|30/08/2017|142,12
01.01.2017|12345672301234|0002342434|SILVA JESUS|27-32558657-9|NC|A|2062-
00010443|142,12|30/08/2017|142,12
Cheers!
EDIT 1:
This would be my first attepmt . My "body" POJO is called DetalleFacturacion and my "header" POJO is CabeceraFacturacion. The reader I thought to do it with DetalleFacturacion pojo, so i can skip the header and treat it later... however i'm not sure how to assign header's data into CabeceraFacturacion.
public FlatFileItemReader<DetalleFacturacion> readerDetalleFacturacion(){
FlatFileItemReader<DetalleFacturacion> reader = new FlatFileItemReader<>();
reader.setLinesToSkip(1);
reader.setResource(new ClassPathResource("/inputFiles/GLEO-MN170100-PROCESO01-SUBDFACT-000001.txt"));
DefaultLineMapper<DetalleFacturacion> detalleLineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer tokenizerDet = new DelimitedLineTokenizer("|");
tokenizerDet.setNames(new String[] {"fechaEmision", "tipoDocumento", "letra", "nroComprobante",
"nroCliente", "razonSocial", "cuit", "montoNetoGP", "montoNetoG3",
"montoExento", "impuestos", "montoTotal"});
LineCallbackHandler skippedLineCallback = new LineCallbackHandler() {
#Override
public void handleLine(String line) {
String[] headerSeparado = line.split("|");
String printDate = headerSeparado[0];
String reportIdentifier = headerSeparado[1];
String tituloReporte = headerSeparado[2];
String fechaDesde = headerSeparado[3];
String fechaHasta = headerSeparado[4];
CabeceraFacturacion cabeceraFacturacion = new CabeceraFacturacion();
cabeceraFacturacion.setPrintDate(printDate);
cabeceraFacturacion.setReportIdentifier(reportIdentifier);
cabeceraFacturacion.setTituloReporte(tituloReporte);
cabeceraFacturacion.setFechaDesde(fechaDesde);
cabeceraFacturacion.setFechaHasta(fechaHasta);
}
};
reader.setSkippedLinesCallback(skippedLineCallback);
detalleLineMapper.setLineTokenizer(tokenizerDet);
detalleLineMapper.setFieldSetMapper(new DetalleFieldSetMapper());
detalleLineMapper.afterPropertiesSet();
reader.setLineMapper(detalleLineMapper);
// Test to check if it is saving correctly data in CabeceraFacturacion
CabeceraFacturacion cabeceraFacturacion = new CabeceraFacturacion();
System.out.println("Print Date:"+cabeceraFacturacion.getPrintDate());
System.out.println("Report Identif:
"+cabeceraFacturacion.getReportIdentifier());
return reader;
}
You are correct . You need to use skippedLinesCallback to handle skip lines.
You need to implement LineCallbackHandler interface and add you processing in handleLine method.
LineCallbackHandler Interface passes the raw line content of the lines in the file to be skipped. If linesToSkip is set to 2, then this interface is called twice.
This is how you can define Reader for the same.
Java Config - Spring Batch 4
#Bean
public FlatFileItemReader<POJO> myReader() {
return FlatFileItemReader<pojo>().
.setResource(new FileSystemResource("resources/players.csv"));
.name("myReader")
.delimited()
.delimiter(",")
.names("pro1,pro2,pro3")
.targetType(POJO.class)
.skippedLinesCallback(skippedLinesCallback)
.build();
}
Related
I have a problem with spring integration.
I want to make a request on an ftp server to retrieve the name of a file
(at the command line: ls "filename")
But I cannot recover the file name dynamically.
I understood that there was a story with payload or header but I can not
This is what I have:
Review my controller, I use this :
private FtpConfig.MyGateway gateway;
...
gateway.fichierExist(filename);
in my FTP file :
#Bean
public SessionFactory<FTPFile> ftpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("");
sf.setPort(21);
sf.setUsername("");
sf.setPassword("");
return new CachingSessionFactory<FTPFile>(sf);
}
#Bean
#ServiceActivator(inputChannel = "ftpChannelExist")
public MessageHandler handler2() {
FtpOutboundGateway ftpOutboundGateway =
new FtpOutboundGateway(ftpSessionFactory(), "ls");
ftpOutboundGateway.setOptions("-a -1")
FtpSimplePatternFileListFilter filter = new FtpSimplePatternFileListFilter("filename"); //on filtre sur le nom
return ftpOutboundGateway;
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "ftpChannelExist")
ArrayList<String> fichierExist(String filename);
}
I tried with header too, but I can not do anything ...
Thanks.
(Sorry for my english, i'm french)
See LS command description in the Reference Manual:
In addition, filename filtering is provided, in the same manner as the inbound-channel-adapter.
The message payload resulting from an ls operation is a list of file names, or a list of FileInfo objects. These objects provide information such as modified time, permissions etc.
The remote directory that the ls command acted on is provided in the file_remoteDirectory header.
What you are missing in your configuration is a fact of the remote directory to fetch files from. Typically we suggest to have such a directory in the payload as you do with your fichierExist(String filename) and configure the third ctor arg for the FtpOutboundGateway:
FtpOutboundGateway ftpOutboundGateway =
new FtpOutboundGateway(ftpSessionFactory(), "ls", "payload");
According the logic in the FtpOutboundGateway that expression is serving as a source for the remote directory in the LS command. In your case this one is going to be an argument of your fichierExist(String filename) gateway.
You indeed can use there a FtpSimplePatternFileListFilter, but be sure to specify a proper pattern to filter remote files.
In the end the names of the remote files in the requested directory, after filtering are going to be returned to the ArrayList<String> of your gateway. That's correct.
Otherwise your question isn't clear.
Thanks for your reply.
I have change my FtpOutboundGateway for add "payload" but I can't use payload for my FtpSimplePatternFileListFilter.
I've try :
FtpSimplePatternFileListFilter filter = new FtpSimplePatternFileListFilter("filename");
FtpSimplePatternFileListFilter filter = new FtpSimplePatternFileListFilter("payload");
FtpSimplePatternFileListFilter filter = new FtpSimplePatternFileListFilter("payload.filename");
FtpSimplePatternFileListFilter filter = new FtpSimplePatternFileListFilter("payload['filename']");
I tried the solution applied in the post here : Spring Batch - create a new file each time instead of overriding it for transferring data from CSV to XML, but it didn't worked for the annotation based approached I used.
fileItemWriter.setResource(new FileSystemResource("csv/employees-#{new java.text.SimpleDateFormat("Mddyyyyhhmmss").format(new java.util.GregorianCalendar().getTime())}.csv"));
My Batch job is scheduled to run in every 1 hours, this batch jobs reads table and write data to CSV file. When data writes I need to create new file altogether..will be good if file name is unique, so I was looking to implement the date etc as per post.
Could anyone guide what's wrong going on ?
#Bean(destroyMethod="")
public FlatFileItemWriter<Employees> employeesWriter(){
FlatFileItemWriter<Employees> fileItemWriter = new FlatFileItemWriter<>();
//fileItemWriter.setResource(new FileSystemResource("csv/employees.csv"));
fileItemWriter.setResource(new FileSystemResource("csv/employees-#{new java.text.SimpleDateFormat("Mddyyyyhhmmss").format(new java.util.GregorianCalendar().getTime())}.csv"));
fileItemWriter.setHeaderCallback(headerCallback());
BeanWrapperFieldExtractor<Employees> fieldExtractor = new BeanWrapperFieldExtractor<>();
fieldExtractor.setNames(new String[] {"employeeNumber", "lastName", "firstName", "extension", "email", "officeCode", "reportsTo", "jobTitle"});
DelimitedLineAggregator<Employees> lineAggregator = new DelimitedLineAggregator<>();
lineAggregator.setDelimiter(",");
lineAggregator.setFieldExtractor(fieldExtractor);
fileItemWriter.setLineAggregator(lineAggregator);
fileItemWriter.setShouldDeleteIfEmpty(true);
return fileItemWriter;
}
Could anyone guide what's wrong going on ?
Three things:
SpEL expressions are not interpreted when used like you do
The " copied from the xml sample will not work in Java config
The / in csv/... is not a valid character in a file name
You need to declare your writer as follows:
#Bean
public FlatFileItemWriter itemWriter(#Value("employees-#{new java.text.SimpleDateFormat('Mddyyyyhhmmss').format(new java.util.GregorianCalendar().getTime())}.csv") String filename) {
FlatFileItemWriter<Employees> fileItemWriter = new FlatFileItemWriter<>();
fileItemWriter.setResource(new FileSystemResource(filename));
...
return fileItemWriter;
}
But I would recommend using a step scoped item writer and pass the file name as a job parameter rather than using a SpEL expression.
I need the Jersey Multipart Client to Upload more than one file.
I am able to upload a Single file but how can i upload more than one file.
In the client i set the two filedatabody parts.
final FileDataBodyPart filePart = new FileDataBodyPart("file", new File("path"));
FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
FileDataBodyPart filePart2 = new FileDataBodyPart("file", new File("path2"));
final FormDataMultiPart multipart =
(FormDataMultiPart) formDataMultiPart.field("foo", "bar").bodyPart(filePart).bodyPart(filePart2);
How to write the server side code.
The "file" you're using here new FileDataBodyPart("file", new File("path2")); is the name of the body part. If you are going to name them the same (which is allowed), then use a List for your parameter type
public Response upload(#FormDataParam("file") List<InputSream> files)
Otherwise if you want to change the name of one of the parts, then just add another #FormDataParam parameter using that part's name
public Response upload(#FormDataParam("file1") InputStream file1,
#FormDataParam("file2") InputStream file2)
I am writing a spring batch job to process the below mentioned data file and write it into a db.
Sample data file is of this format where I have multiple headers and
each header has a bunch of rows associated with it .
I can have million of records for each header and I can have n number
of headers in a flat file that am processing.My requirement is to
pick a few readers which am concerned .
For all the picked readers I need to pick all the data rows .Each
header and its data format is also different .I can receive either of
these data in my processor and need to write them into my DB.
HDR01
A|41|57|Data1|S|62|Data2|9|N|2017-02-01 18:01:05|2017-02-01 00:00:00
A|41|57|Data1|S|62|Data2|9|N|2017-02-01 18:01:05|2017-02-01 00:00:00
HDR02
A|41|57|Data1|S|62|Data2|9|N|
A|41|57|Data1|S|62|Data2|9|N|
I tried exploring the PatternMatchingCompositeLineMapper where I can
map the different header pattern I have to a tokenizer and
corresponding FieldSetMapper but I need to read the body and not the
header here .
Don't have any footer to Crete a end of line policy of my own as well .
Also tried using AggregateItemReader but don't want to club all the
records of a header before I process them .
Each rows corresponding a header should be processed parallel .
#Bean
public LineMapper myLineMapper() {
PatternMatchingCompositeLineMapper< Domain > mapper = new PatternMatchingCompositeLineMapper<>();
final Map<String, LineTokenizer> tokenizers = new HashMap<String, LineTokenizer>();
tokenizers.put("* HDR01*", new DelimitedLineTokenizer());
tokenizers.put("*HDR02*", new DelimitedLineTokenizer());
tokenizers.put("*", new DelimitedLineTokenizer("|"));
mapper.setTokenizers(tokenizers);
Map<String, FieldSetMapper<VMSFeedStyleInfo>> mappers = new HashMap<String, FieldSetMapper<VMSFeedStyleInfo>>();
try {
mappers.put("* HDR01*", customMapper());
mappers.put("*HDR02*", customMapper());
mappers.put("*", customMapper() );
} catch (Exception e) {
e.printStackTrace();
}
mapper.setFieldSetMappers(mappers);
return mapper;
}
Can somebody help me provide some inputs as to how should I achieve this .
I am using Jersey JAX-RS.
I want to return a Response with a Map containing Image Files and JSON String values.
Is this the right way to do this:
Map<String,Object> map = new HashMap........
GenericEntity entity = new GenericEntity<Map<String,Object>>(map) {};
return Response.ok(entity).build();
Or is this better.I plan to use JAX-RS with Jersey only.
JResponse.ok(map).build();
I am basing this on this article:
http://aruld.info/handling-generified-collections-in-jersey-jax-rs/
I am not sure what to specify for #Produces too(planning to leave it out).
TIA,
Vijay
You better produce a multipart response:
import static com.sun.jersey.multipart.MultiPartMediaTypes.MULTIPART_MIXED_TYPE;
import static javax.ws.rs.core.MediaType.APPLICATION_XML_TYPE
#GET
#Produces(MULTIPART_MIXED_TYPE)
public Response get()
{
FileDataSource image = ... (gets the image file)
String info = ... (gets the xml structured information)
MultiPart multiPart = new MultiPart().
bodyPart(new BodyPart(info, APPLICATION_XML_TYPE)).
bodyPart(new BodyPart(image, new MediaType("image", "png")));
return Response.ok(multiPart, MULTIPART_MIXED_TYPE).build();
}
This example was taken from there.