I have data in a csv file that I want to read into a Map using Spring batch. The format of the data is like this:
1, "data1", 2, "data2", 3, "data3"
This format lends itself easily to a Map, but I can't seem to do it. I am currently using a PassThroughLineMapper and then tokenizing the String in the processor. However, since I have a couple of processors, I am having to do this in all of them. This seems very inefficient to me. Here's my current FlatFileItemReader code below.
#Bean
public FlatFileItemReader<String> reader() {
return new FlatFileItemReaderBuilder<String>()
.name("fileLineReader").linesToSkip(1)
.resource(new FileSystemResource(inputCsv))
.lineMapper(new PassThroughLineMapper())
.build();
}
I would like it to return Map<Integer, String>
Turned out to be a simple task in the end. I wrote a custom lineMapper. Not deleting the question because it might help somebody else.
#Override
public Map<Integer, String> mapLine(String s, int i) throws Exception {
Map<Integer, String> map = new HashMap<>();
String[] tokens = s.split(",");
String key = "";
for (int j = 0; j < tokens.length; j++) {
if (tokens[j].equals("9999")) {
break;
} else {
if (j % 2 == 0)
key = tokens[j];
else
map.putIfAbsent(Integer.valueOf(key), tokens[j]);
}
}
return map;
}
Related
here's my code how to declare queueName in here.
//RabbitmqConfig.java
#Getter
public List<String> queueNameList = new ArrayList<>();
#Bean
public DirectExchange exchange(RabbitAdmin rabbitAdmin) {
DirectExchange directExchange = new DirectExchange(exchange);
for (int num = 1; num <= 3; num++) {
String newQueueName = String.format(queueName + "-%s", num);
String newRoutingKey = String.format(routingKey + "-%s", num);
Queue queue = new Queue(newQueueName, false);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(directExchange).with(newRoutingKey));
queueNameList.add(newQueueName);
}
return new DirectExchange(exchange);
}
Then, my question is how to get these queue name on rabbitListener?
I got some answer using split.
#RabbitListener(queues = {"#{'${spring.rabbitmq.test}'.split(',')}"},
But I want to use RabbitmqConfig.queueNameList. Because the number of queue can be increase, so I want to deal it by setting variable num.
Maybe SPeL? or anything else?
I have following code.
public class ComparingTwoLists {
public static void main(String[] args) {
List<String> small = Arrays.asList("AA", "BB");
List<String> big = Arrays.asList("a.b.AA", "a.b.BB", "a.b.CC");
List<String> list = new ArrayList<String>();
Consumer<String> consumer = (String outer) -> {
Stream<String> stream1 = small.stream();
Consumer<String> innerConsumer = (String inner) -> {
if (outer.endsWith(inner)) {
list.add(outer);
}
};
stream1.forEach(innerConsumer);
};
Stream<String> stream2 = big.stream();
stream2.forEach(consumer);
System.out.println(list);
}
}
I am trying to find out if string values in small is present in string values of big at end of string, using method endWith() method.
Is this efficient way of doing it?
Another problem I am facing in this code is that when I am trying to debug this code, I am able to view/inspect the value of inner, but cannot view the value of outer. Is there a way to check the value of outer while debugging.
You can do like this:
big.stream()
.filter(s -> small.stream().anyMatch(s::endsWith))
.collect(Collectors.toList());
Is it possible to get the total number of rows written from FlatFileItemWriter.headerCallback()?
I am a spring-batch nubee and I looked at putting count of lines into header of flat file and Spring Batch - Counting Processed Rows.
However I can't seem to implement the logic using the advice given there. It makes sense the writer count will only be available after the file is processed. However I am trying to get the row-count just before the file is officially written.
I tried to look for a hook like #AfterStep and grab the total rows, but I keep going in circles.
#Bean
#StepScope
public FlatFileItemWriter<MyFile> generateMyFileWriter(Long jobId,Date eventDate) {
String filePath = "C:\MYFILE\COMPLETED";
Resource file = new FileSystemResource(filePath);
DelimitedLineAggregator<MyFile> myFileLineAggregator = new DelimitedLineAggregator<>();
myFileLineAggregator.setDelimiter(",");
myFileLineAggregator.setFieldExtractor(getMyFileFieldExtractor());
return new FlatFileItemWriterBuilder<MyFile>()
.name("my-file-writer")
.resource(file)
.headerCallback(new MyFileHeaderWriter(file.getFilename()))
.lineAggregator(myFileLineAggregator)
.build();
}
private FieldExtractor<MyFile> getMyFileFieldExtractor() {
final String[] fieldNames = new String[]{
"typeRecord",
"idSystem"
};
return item -> {
BeanWrapperFieldExtractor<MyFile> extractor = new BeanWrapperFieldExtractor<>();
extractor.setNames(fieldNames);
return extractor.extract(item);
};
}
Notice I am using the MyFileHeaderWriter.java class(below) in the headerCallback(new MyFileHeaderWriter(file.getFilename())) (above). I am trying to initialize the value of qtyRecordsCreated below.
class MyFileHeaderWriter implements FlatFileHeaderCallback {
private final String header;
private String dtxCreated;
private String tmxCreated;
private String fileName;//15 byte file name private String qtyRecordsCreated;//number of rows in file including the header row
MyFileHeaderWriter(String sbfFileName) {
SimpleDateFormat dateCreated = new SimpleDateFormat("YYDDD");
SimpleDateFormat timeCreated = new SimpleDateFormat("HHMM");
Date now = new Date();
this.dtxCreated = dateCreated.format(now);
this.tmxCreated = timeCreated.format(now);
this.fileName = sbfFileName; this.qtyRecordsCreated="";
String[] headerValues = {dtxCreated,tmxCreated,fileName,qtyRecordsCreated};
this.header = String.join(",", headerValues);
}
#Override
public void writeHeader(Writer writer) throws IOException {
writer.write(header);
}
}
How can I get the number of rows in the header row?
Can the FlatFileFooterCallback be used to fetch the number of rows and then update the header with number of rows in the file afterwards?
You can achieve this in ItemProcessor, try this it work for me
public class EmployeeProcessor implements ItemProcessor<Employee, Employee> {
#Override
public Employee process(Employee employee) throws Exception {
return employee;
}
#AfterStep
public void afterStep(StepExecution stepExecution) {
ExecutionContext stepContext = stepExecution.getExecutionContext();
stepContext.put("count", stepExecution.getReadCount());
System.out.println("COUNT" + stepExecution.getReadCount());
}
}
And in you writer to get value
int count = stepContext.getInt("count");
Hope work for you
Raw CSV is like this:
First line: Name, StudentID, comment
Data:
Name, StudentId, Comment
Jake, 12312, poor
Emma, 12324, good
Mary, 13214, need more work on programming
and math.
The comment cell of the last entry of the csv data contains two lines. I want to treat it as one line data.
When I read the file using flatItemReader, it throws error about "expected token 3 but actual 1" I guess it treat the second line as a new line.
Is there a way to treat them as one line?
Have your reader just return the raw string for each line without trying to split on the delimiter. Make a processor (has to be stateful) to handle the parsing. The only tricky part is you'll have to signal to the processor when you've reached the EOF somehow so it isn't waiting to see if it should aggregate the next line. Something like this:
public class AggregatingItemProcessor<T> implements ItemProcessor<T, T>, InitializingBean {
private BiPredicate<T, T> aggregatePredicate;
private BiFunction<T, T, T> aggregator;
public void setAggregatePredicate(BiPredicate<T, T> aggregatePredicate) {
this.aggregatePredicate = aggregatePredicate;
}
public void setAggregator(BiFunction<T, T, T> aggregator) {
this.aggregator = aggregator;
}
private T cur;
#Override
public T process(T item) throws Exception {
if(cur == null) {
cur = item;
return null;
}
if(aggregatePredicate.test(cur, item)) {
cur = aggregator.apply(cur, item);
return null;
} else {
T toRet = cur;
cur = item;
return toRet;
}
}
#Override
public void afterPropertiesSet() throws Exception {
Assert.notNull(aggregatePredicate, "Predicate to determine if records should be aggregated must not be null.");
Assert.notNull(aggregator, "Function for aggregating items must not be null.");
}
}
Then the config...
static final String EOF_MARKER = "\0";
#Bean
public FlatFileItemReader<String> reader() {
final FlatFileItemReader<String> reader = new FlatFileItemReader<String>() {
private boolean finished = false;
#Override
public String read() throws Exception, UnexpectedInputException, ParseException {
if(finished) return null;
String next = super.read();
if(next == null) {
finished = true;
return EOF_MARKER;
}
return next;
}
};
reader.setLineMapper((s, i) -> s);
return reader;
}
#Bean
public AggregatingItemProcessor<String> processor() {
final AggregatingItemProcessor<String> processor = new AggregatingItemProcessor<>();
processor.setAggregatePredicate((s1, s2) -> !EOF_MARKER.equals(s2) && StringUtils.countOccurrencesOf(s2, ",") < 2);
processor.setAggregator(String::concat);
return processor;
}
Problem is that the eclipse recommendation makes little sense -- to me. I checked my blocks several times.
This is my first time posting code so I'm hoping it's formatted ok.
The eclipse error is on line 73: below the ; "anagrams.add(current);" This is the second line below the last for loop in the code.
package anagrecur2;
import java.util.ArrayList;
import java.util.List;
import java.util.Arrays;
import java.io.*;
public class AnagRecur2Main {
public static void main(String[] args) {
List<String> anagrams = new ArrayList<String>();
List<String> preString = new ArrayList<String>();
List<String> postString = new ArrayList<String>();
List<String> word = new ArrayList<String>();
word.add("t");
word.add("e");
word.add("a");
word.add("m");
preString.add("");
postString.add("");
String c;
String current;
}
ArrayList<String> getAnagrams (String word, String preString){
// remove the first occurrence of each
// character in preString from word and
// stores it in postString
}
ArrayList<String> removePreStringFromWord(List word){
ArrayList<String> postString;
ArrayList<String> preString;
for (int i = 0; i < preString.size(); i++) {
for (int j = 0; j < word.size(); j++) {
if (preString.get(i)== word.get(j)){
word.remove(j);
}else {
postString.add((String) word.get(j));
}
}
return postString;
}
}
// using a string as if it is a c# string.
// for java you need to convert the string to
// char array
//ArrayList<String> word;
//String current;
//char poststrCharA;
ArrayList<String> anagrams;
ArrayList<String> preString;
ArrayList<String> postString;
String[] prestrAr = new String[preString.size()];{
prestrAr = preString.toArray(prestrAr);
for(String s : prestrAr){
System.out.println(s);
}
for (String poststrCharA : postString){
poststrCharA.toCharArray();
System.out.println("postStringtoCharAr"+poststrCharA);
}
// adds string combo to anagrams if
// it is a true word
for (String c : prestrAr) {
String current = prestrAr + c;
anagrams.add(current);
}
void getAnagrams(char word, char current){
System.out.println("word= "+word+"current= "+current);
}
}
First off, your getAnagrams doesn't return a string Array list. Also, what are you trying to accomplish with this file?