I am new to spring batch . I have to validate header and footer before processing . I am able to get header but not footer.
You can use a ItemProcessor, something like below;
public class DataProcessor implements ItemProcessor<String, String> {
#Override
public String process(String line) throws Exception {
if (line != null && line.contains("footer")) {
return "Footer Found";
} else {
return line;
}
}
}
Related
MultiResourceItemReader reads all files sequentially.
I want once one file read completely, it should call processor/writer.it should not read next file.
Since file content is not constant, i can't go with chunk size.
Any idea on chunk policy to decide end of file content?
I think you should write a step which read/process/write only one file with a "single file item reader" (like FlatFileItemReader). And repeat the step while there are files remainig.
Spring batch gives you a feature to do so : conditional flows and in particular the programmatic flow decision which gives you a smart way to decide when to stop a loop between steps (when there is not file any more)
And since you will not be able to give a constant input file name to your reader, you should also have a look at Late binding section.
Hope this will be enough to help you. Please, make comments if you need more details.
Using MultiResourceItemReader, assigning multiple file reasources.
Using custom file reader as delegate, reading a file completely
For reading file completely, come up with a logic
#Bean
public MultiResourceItemReader<SimpleFileBean> simpleReader()
{
Resource[] resourceList = getFileResources();
if(resourceList == null) {
System.out.println("No input files available");
}
MultiResourceItemReader<SimpleFileBean> resourceItemReader = new MultiResourceItemReader<SimpleFileBean>();
resourceItemReader.setResources(resourceList);
resourceItemReader.setDelegate(simpleFileReader());
return resourceItemReader;
}
#Bean
SimpleInboundReader simpleFileReader() {
return new SimpleInboundReader(customSimpleFileReader());
}
#Bean
public FlatFileItemReader customSimpleFileReader() {
return new FlatFileItemReaderBuilder()
.name("customFileItemReader")
.lineMapper(new PassThroughLineMapper())
.build();
}
public class SimpleInboundReader implements ResourceAwareItemReaderItemStream{
private Object currentItem = null;
private FileModel fileModel = null;
private String fileName = null;
private boolean fileRead = false;
private ResourceAwareItemReaderItemStream<String> delegate;
public SimpleInboundReader(ResourceAwareItemReaderItemStream<String> delegate) {
this.delegate = delegate;
}
#Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
delegate.open(executionContext);
}
#Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
delegate.update(executionContext);
}
#Override
public void close() throws ItemStreamException {
delegate.close();
}
#Override
public void setResource(Resource resource) {
fileName = resource.getFilename();
this.delegate.setResource(resource);
}
String getNextLine() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {
return delegate.read();
}
#Override
public SimpleFileBean read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
SimpleFileBean simpleFileBean = null;
String currentLine = null;
currentLine=delegate.read();
if(currentLine != null) {
simpleFileBean = new SimpleFileBean();
simpleFileBean.getLines().add(currentLine);
while ((currentLine = getNextLine()) != null) {
simpleFileBean.getLines().add(currentLine);
}
return simpleFileBean;
}
return null;
}
}
According to documentation, I can use something like this in exceptionExpression: #Retryable(exceptionExpression="message.contains('this can be retried')")
But I want to get response body and check message inside it (from RestClientResponseException), something similar to this: exceptionExpression = "getResponseBodyAsString().contains('important message')"
I tried like that but it doesn't work. So, is it possible to do something similar and check info from responseBody?
Edit: Adding whole #Retryable annotation parameters with Gary Russell's suggestion:
#Retryable(value = HttpClientErrorException.class, exceptionExpression = "#{#root instanceof T(org.springframework.web.client.HttpClientErrorException) AND responseBodyAsString.contains('important message')}")
I'm using actual RestClientResponseException subclass that I'm catching but is still not triggering retry.
With the current release, the expression incorrectly requires static template markers; they will not be needed in 1.3.
#Retryable(exceptionExpression = "#{responseBodyAsString.contains('foo')}")
However, you can't use this expression if there are include or exclude properties so the expression should check the type:
#Retryable(exceptionExpression =
"#{#root instanceof T(org.springframework.web.client.RestClientResponseException) "
+ "AND responseBodyAsString.contains('foo')}")
EDIT
#SpringBootApplication
#EnableRetry
public class So61488237Application {
public static void main(String[] args) {
SpringApplication.run(So61488237Application.class, args).close();
}
#Bean
public ApplicationRunner runner(Foo foo) {
return args -> {
try {
foo.test(1, "foo.");
}
catch (Exception e) {
}
};
}
}
#Component
class Foo {
#Retryable(exceptionExpression =
"#{#root instanceof T(org.springframework.web.client.RestClientException) "
+ "AND responseBodyAsString.contains('foo')}")
public void test(int val, String str) {
System.out.println(val + ":" + str);
throw new RestClientResponseException("foo", 500, "bar", new HttpHeaders(), "foo".getBytes(),
StandardCharsets.UTF_8);
}
}
1:foo.
1:foo.
1:foo.
I've implemented the following approach, which in my opinion is much more convenient.
#Retryable(value = WebClientException.class,
exceptionExpression = RetryCheckerService.EXPRESSION,
maxAttempts = 5,
backoff = #Backoff(delay = 500))
public List<ResultDto> getSomeResource () {}
Here the RetryCheckerService encapsulates all needed logic.
#Service
public class RetryCheckerService {
public static final String EXPRESSION = "#retryCheckerService.shouldRetry(#root)";
public boolean shouldRetry(WebClientException ex) {
if (ex instanceof WebClientResponseException responseException) {
return responseException.getStatusCode().is5xxServerError()
|| responseException.getStatusCode().equals(HttpStatus.NOT_FOUND);
}
if (ex instanceof WebClientRequestException requestException) {
String message = requestException.getMessage();
if (message == null) {
return false;
}
return message.contains("HttpConnectionOverHTTP");
}
return false;
}
}
Raw CSV is like this:
First line: Name, StudentID, comment
Data:
Name, StudentId, Comment
Jake, 12312, poor
Emma, 12324, good
Mary, 13214, need more work on programming
and math.
The comment cell of the last entry of the csv data contains two lines. I want to treat it as one line data.
When I read the file using flatItemReader, it throws error about "expected token 3 but actual 1" I guess it treat the second line as a new line.
Is there a way to treat them as one line?
Have your reader just return the raw string for each line without trying to split on the delimiter. Make a processor (has to be stateful) to handle the parsing. The only tricky part is you'll have to signal to the processor when you've reached the EOF somehow so it isn't waiting to see if it should aggregate the next line. Something like this:
public class AggregatingItemProcessor<T> implements ItemProcessor<T, T>, InitializingBean {
private BiPredicate<T, T> aggregatePredicate;
private BiFunction<T, T, T> aggregator;
public void setAggregatePredicate(BiPredicate<T, T> aggregatePredicate) {
this.aggregatePredicate = aggregatePredicate;
}
public void setAggregator(BiFunction<T, T, T> aggregator) {
this.aggregator = aggregator;
}
private T cur;
#Override
public T process(T item) throws Exception {
if(cur == null) {
cur = item;
return null;
}
if(aggregatePredicate.test(cur, item)) {
cur = aggregator.apply(cur, item);
return null;
} else {
T toRet = cur;
cur = item;
return toRet;
}
}
#Override
public void afterPropertiesSet() throws Exception {
Assert.notNull(aggregatePredicate, "Predicate to determine if records should be aggregated must not be null.");
Assert.notNull(aggregator, "Function for aggregating items must not be null.");
}
}
Then the config...
static final String EOF_MARKER = "\0";
#Bean
public FlatFileItemReader<String> reader() {
final FlatFileItemReader<String> reader = new FlatFileItemReader<String>() {
private boolean finished = false;
#Override
public String read() throws Exception, UnexpectedInputException, ParseException {
if(finished) return null;
String next = super.read();
if(next == null) {
finished = true;
return EOF_MARKER;
}
return next;
}
};
reader.setLineMapper((s, i) -> s);
return reader;
}
#Bean
public AggregatingItemProcessor<String> processor() {
final AggregatingItemProcessor<String> processor = new AggregatingItemProcessor<>();
processor.setAggregatePredicate((s1, s2) -> !EOF_MARKER.equals(s2) && StringUtils.countOccurrencesOf(s2, ",") < 2);
processor.setAggregator(String::concat);
return processor;
}
I am newbee to Spring integration. i am trying to implement customer sftp filter to list the files in SFTP server. I am getting "The blank final field seen may not have been initialized" at the constructor.Can you please suggest me to get list of file names from sftp server.
I dont have any idea what went wrong in my code.
Thanks in Advance
java code
public class SFTPFileFilter extends SftpSimplePatternFileListFilter {
public SFTPFileFilter(String pattern) {
super(pattern);
// TODO Auto-generated constructor stub
}
final static Logger logger = LoggerFactory.getLogger(SFTPFileFilter.class);
private final Queue<File> seen;
private final Set<File> seenSet = new HashSet<File>();
private final Object monitor = new Object();
public static int fileCount = 0;
#Autowired
private SourcePollingChannelAdapter sftpInbondAdapter;
public List<File> filterFiles(File[] files)
{
List<File> accepted = new ArrayList<File>();
for (File file : files) {
System.out.println(file.getName());
accepted.add(file);
}
return accepted;
}
public boolean accept(File file) {
synchronized (this.monitor) {
if (this.seenSet.contains(file)) {
logger.info(file.getName()+" is already copied earlier");
return false;
}
if (this.seen != null) {
if (!this.seen.offer(file)) {
File removed = this.seen.poll();
this.seenSet.remove(removed);
this.seen.add(file);
}
}
this.seenSet.add(file);
return true;
}
}
}
private final Queue<File> seen;
You are not initializing that field in a constructor.
You can't extend it like that; simply override the method like this...
public List<File> filterFiles(File[] files) {
for (File file : files) {
System.out.println("received:" + file.getName());
}
List<File> filtered = super.filterFiles(files);
for (File file : flteredFiles) {
System.out.println("after filter:" + file.getName());
}
return filteredFiles;
}
Hi My thread class is showing null pointer exception please help me to resolve
#Component
public class AlertsToProfile extends Thread {
public final Map<Integer, List<String>> userMessages = Collections.synchronizedMap(new HashMap<Integer, List<String>>());
#Autowired
ProfileDAO profileDAO;
private String categoryType;
private String dataMessage;
public String getCategoryType() {
return categoryType;
}
public void setCategoryType(String categoryType) {
this.categoryType = categoryType;
}
public String getDataMessage() {
return dataMessage;
}
public void setDataMessage(String dataMessage) {
this.dataMessage = dataMessage;
}
public void run() {
String category=getCategoryType();
String data= getDataMessage();
List<Profile> all = profileDAO.findAll();
if (all != null) {
if (category == "All" || category.equalsIgnoreCase("All")) {
for (Profile profile : all) {
List<String> list = userMessages.get(profile.getId());
if (list == null ) {
ArrayList<String> strings = new ArrayList<String>();
strings.add(data);
userMessages.put(profile.getId(), strings);
} else {
list.add(data);
}
}
}
}
}
and my service method is as follows
#Service
public class NoteManager
{
#Autowired AlertsToProfile alertsToProfile;
public void addNote(String type, String message, String category) {
alertsToProfile.setCategoryType(category);
String data = type + "," + message;
alertsToProfile.setDataMessage(data);
alertsToProfile.start();
System.out.println("addNotes is done");
}
But when i call start() method am getting null pointer exception please help me. I am new to spring with thread concept
It pretty obvious: you instantiate your thread directly, as opposed to letting spring create AlertsToProfile and auto wire your instance.
To fix this, create a Runnable around your run() method and embed that into a method, something like this:
public void startThread() {
new Thread(new Runnable() {
#Override
public void run() {
// your code in here
}}).start();
}
you will want to bind the Thread instance to a field in AlertsToProfile in order to avoid leaks and stop the thread when you're done.