I have a header that looks like this:
// Writer
#Bean(name = "cms200Writer")
#StepScope
public FlatFileItemWriter<Cms200Item> cmsWriter(#Value("#{jobExecutionContext}") Map<Object, Object> ec, //
#Qualifier("cms200LineAggregator") FormatterLineAggregator<Cms200Item> lineAgg) throws IOException {
#SuppressWarnings("unchecked")
String fileName = ((Map<String, MccFtpFile>) ec.get(AbstractSetupTasklet.BATCH_FTP_FILES)).get("cms").getLocalFile();
//Ensure the file can exist.
PrintWriter fos = getIoHarness().getFileOutputStream(fileName);
fos.close();
FlatFileItemWriter<Cms200Item> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(fileName));
writer.setLineAggregator(lineAgg);
Calendar cal = Calendar.getInstance();
Date date = cal.getTime();
DateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
String formattedDate=dateFormat.format(date);
writer.setHeaderCallback(new FlatFileHeaderCallback() {
public void writeHeader(Writer writer) throws IOException {
writer.write(" Test Company. " + formattedDate);
writer.write("\n CMS200 CUSTOMER SHIPMENT MANIFEST AUTHORIZATION BY CUSTOMER NAME Page 1");
writer.write("\n\n");
writer.write(" CUSTOMER NAME CITY ST CONTROL MNFST ID AUTH CODE I03 CLS EDI EXPRESS POV MOST CURRENT DEACTIVE");
writer.write("\n");
writer.write(" NBR TRL 214 WORK ACCESS DATE ");
}
});
return writer;
}
I want to print this header everytime 53 records are processed. I can't figure out how to implement that logic into my Spring Batch job. I have the writeCount added to my execution context, but not sure how to access that here, or if that's the correct approach.
The writer I posted is in my BatchConfiguration.java file
EDIT:
Below I have my filestep and added chunk size
#Bean(name = "cms200FileStep")
public Step createFileStep(StepBuilderFactory stepFactory, //
#Qualifier("cms200Reader") ItemReader<Cms200Item> reader, //
Cms200Processor processor, //
#Qualifier("cms200Writer") ItemWriter<Cms200Item> writer) {
return stepFactory.get("cms200FileStep") //
.<Cms200Item, Cms200Item>chunk(100000) //
.reader(reader) //
.processor(processor) //
.writer(writer).chunk(53) //
.allowStartIfComplete(true)//
.build();//
}
Edit: Added job config
// Job
#Bean(name = "mccCMSCLRPTjob")
public Job mccCmsclrptjob(JobBuilderFactory jobFactory, //
#Qualifier("cms200SetupStep") Step setupStep, //
#Qualifier("cms200FileStep") Step fileStep, //
#Qualifier("putFtpFilesStep") Step putFtpStep, //
#Qualifier("cms200TeardownStep") Step teardownStep, //
#Autowired SingleInstanceListener listener,
#Autowired ChunkSizeListener chunkListener) { //
return jobFactory.get("mccCMSCLRPTjob") //
.incrementer(new RunIdIncrementer()) //
.listener(listener) //
.start(setupStep) //
.next(fileStep) //
.next(putFtpStep) //
.next(teardownStep) //
.build();
}
Edit: adding the listener
#Bean(name = "cms200FileStep")
public Step createFileStep(StepBuilderFactory stepFactory, //
#Qualifier("cms200Reader") ItemReader<Cms200Item> reader, //
Cms200Processor processor, //
#Qualifier("cms200Writer") ItemWriter<Cms200Item> writer,
#Autowired ChunkSizeListener listener) {
return stepFactory.get("cms200FileStep") //
.<Cms200Item, Cms200Item>chunk(100000) //
.reader(reader) //
.processor(processor) //
.writer(writer).chunk(53) //
.allowStartIfComplete(true)//
.listener(listener) //
.build();//
}
EDIT: After a lot of back and forth this is where I'm at
// Utility Methods
#Bean(name = "cms200FileStep")
public Step createFileStep(StepBuilderFactory stepFactory, Map<Object, Object> ec, //
#Qualifier("cms200Reader") ItemReader<Cms200Item> reader, //
Cms200Processor processor, //
#Qualifier("cms200Writer") ItemWriter<Cms200Item> writer) throws IOException {
#SuppressWarnings("unchecked")
String fileName = ((Map<String, MccFtpFile>) ec.get(AbstractSetupTasklet.BATCH_FTP_FILES)).get("cms").getLocalFile();
return stepFactory.get("cms200FileStep") //
.<Cms200Item, Cms200Item>chunk(100000) //
.reader(reader) //
.processor(processor) //
.writer(writer).chunk(53) //
.allowStartIfComplete(true)//
// .listener((ChunkListener) listener) //
.listener((ChunkListener) new ChunkSizeListener(new File(fileName))) //
.build();//
}
The FlatFileHeaderCallback is called only once before the chunk-oriented step, aka before all chunks.
I want to print this header everytime 53 records are processed
What you can do is set the chunk-size to 53 and use a ChunkListener or ItemWriteListener to write the required data.
EDIT: Add an example
class MyChunkListener extends StepListenerSupport {
private FileWriter fileWriter;
public MyChunkListener(File file) throws IOException {
this.fileWriter = new FileWriter(file, true);
}
#Override
public void beforeChunk(ChunkContext context) {
try {
fileWriter.write("your custom header");
fileWriter.flush();
} catch (IOException e) {
System.err.println("Unable to write header to file");
}
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
try {
fileWriter.close();
} catch (IOException e) {
System.err.println("Unable to close writer");
}
return super.afterStep(stepExecution);
}
}
Related
I have working flow for getting files from single folder present in remote server using inbound adopter but i want for get files for all subfolder present in any remote server parent folder
I have code like this
#Bean
public SessionFactory<SftpClient.DirEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(port);
factory.setUser("foo");
factory.setPassword("foo");
factory.setAllowUnknownKeys(true);
factory.setTestSession(true);
return new CachingSessionFactory<>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("foo");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("sftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(1);
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}`
but instead of single folder i want get files for all subfolder present in foo directory
if possible please help with full code
#GaryRussell
Thanks you so much for you early response .I have done some changes according to your suggested code app is started but files is not getting picked up by application.
CompositeFileListFilter<LsEntry> compositeFileListFilter = new CompositeFileListFilter<>();
SftpPersistentAcceptOnceFileListFilter fileListFilter =
new SftpPersistentAcceptOnceFileListFilter(
(JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
if (Constants.APP1.equals(appName) || Constants.APP2.equals(appName)) {
SftpRegexPatternFileListFilter regexPatternFileListFilter =
new SftpRegexPatternFileListFilter(Pattern.compile("^IL.*"));
compositeFileListFilter.addFilter(regexPatternFileListFilter);
}
compositeFileListFilter.addFilter(fileListFilter);
return IntegrationFlows.fromSupplier(
() -> sftpEnvironment.getSftpGLSIncomingDir(), // remote dir
e -> e.autoStartup(true).poller(pollerMetada()))
.handle(
Sftp.outboundGateway(sftpSessionFactory(), Command.MGET, "payload")
.options(Option.RECURSIVE)
.filter(compositeFileListFilter)
.fileExistsMode(FileExistsMode.IGNORE)
.localDirectoryExpression("'/tmp/' + #remoteDirectory")) // re-create tree locally
.split()
.log()
.get();
#GaryRussell
I have changed my code this new way it is partially processing files mean one example out of 10 file only process 5 or 6 files. I am not able to figure the main issue in that.and I also have also some open challenges which i am mentioning below
Its able to read files form remote subdirectories and store in local directory but I want to process these file in some other sftpChannel, if posible without storing locally
I also want to apply some deduplication technique using data base which will help me to avoid duplicate file processing.
public class SFTPPollerService {
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
//code
return factory;
}
//OLD code
// #Bean
// public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
// SftpInboundFileSynchronizer fileSynchronizer =
// new SftpInboundFileSynchronizer(sftpSessionFactory());
// fileSynchronizer.setDeleteRemoteFiles(sftpEnvironment.isDeleteRemoteFiles());
// fileSynchronizer.setRemoteDirectory(sftpEnvironment.getSftpGLSIncomingDir());
// fileSynchronizer.setPreserveTimestamp(true);
// CompositeFileListFilter<LsEntry> compositeFileListFilter = new
// CompositeFileListFilter<>();
// SftpPersistentAcceptOnceFileListFilter fileListFilter =
// new SftpPersistentAcceptOnceFileListFilter(
// (JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
// if (Constants.app2.equals(appName)
// || Constants.app1.equals(appName)) {
// SftpRegexPatternFileListFilter regexPatternFileListFilter =
// new SftpRegexPatternFileListFilter(Pattern.compile("*.txt"));
// compositeFileListFilter.addFilter(regexPatternFileListFilter);
// }
// compositeFileListFilter.addFilter(fileListFilter);
// fileSynchronizer.setFilter(compositeFileListFilter);
// return fileSynchronizer;
// }
//
// #Bean
// #InboundChannelAdapter(channel = "sftpChannel", poller = #Poller("pollerMetada"))
// public MessageSource<File> sftpMessageSource() {
// SftpInboundFileSynchronizingMessageSource source =
// new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
// source.setLocalDirectory(new File(sftpEnvironment.getSftpLocalDir()));
//
// source.setAutoCreateLocalDirectory(true);
//
// try {
// source.setLocalFilter(
// (FileSystemPersistentAcceptOnceFileListFilter)
// context.getBean("filelistFilter"));
// } catch (Exception e) {
// LOG.error(
// "Exception caught while setting local filter on
// SftpInboundFileSynchronizingMessageSource",
// e);
// }
// source.setMaxFetchSize(sftpEnvironment.getMaxFetchFileSize());
//
// return source;
// }
//new Code
#Bean
public IntegrationFlow sftpInboundFlow() {
CompositeFileListFilter<LsEntry> compositeFileListFilter = new CompositeFileListFilter<>();
SftpPersistentAcceptOnceFileListFilter fileListFilter =
new SftpPersistentAcceptOnceFileListFilter(
(JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
if (Constants.app2.equals(appName) || Constants.app1.equals(appName)) {
SftpRegexPatternFileListFilter regexPatternFileListFilter =
new SftpRegexPatternFileListFilter(Pattern.compile("(subDir | *.txt)"));
compositeFileListFilter.addFilter(regexPatternFileListFilter);
}
fileListFilter.setForRecursion(true);
FileSystemPersistentAcceptOnceFileListFilter fileSystemPersistentAcceptOnceFileListFilter = (FileSystemPersistentAcceptOnceFileListFilter) context.getBean(
"filelistFilter");
compositeFileListFilter.addFilter(fileListFilter);
// IntegrationFlow ir =
// IntegrationFlows.from(
// Sftp.inboundAdapter(sftpSessionFactory())
// .preserveTimestamp(true)
// .remoteDirectory(sftpEnvironment.getSftpGLSIncomingDir())
// .deleteRemoteFiles(sftpEnvironment.isDeleteRemoteFiles())
// .filter(compositeFileListFilter)
// .autoCreateLocalDirectory(true)
// .localDirectory(new File(sftpEnvironment.getSftpLocalDir())),
// e -> e.autoStartup(true).poller(pollerMetada()))
// .handle(handler())
// .get();
return IntegrationFlows.fromSupplier(
() -> sftpEnvironment.getSftpGLSIncomingDir(), // remote dir
e -> e.autoStartup(true).poller(pollerMetada()))
.handle(
Sftp.outboundGateway(sftpSessionFactory(), Command.MGET, "payload")
.options(Option.RECURSIVE)
.fileExistsMode(FileExistsMode.IGNORE)
.regexFileNameFilter("(dsv[0-9]|.*.xml)")
// .filter(compositeFileListFilter)
.localDirectoryExpression("'user/localDir/test/'"))
// .handle(handler())
// .patternFileNameFilter(".*\\.xml")) // re-create tree locally
.split()
.channel("sftpChannel")
// .handle(handler())
.log()
.get();
}
#Bean
public PollerMetadata pollerMetada() {
PollerMetadata pm = new PollerMetadata();
ExpressionEvaluatingTransactionSynchronizationProcessor processor =
new ExpressionEvaluatingTransactionSynchronizationProcessor();
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression("payload.delete()");
processor.setAfterRollbackExpression(exp);
TransactionSynchronizationFactory tsf = new DefaultTransactionSynchronizationFactory(processor);
pm.setTransactionSynchronizationFactory(tsf);
List<Advice> advices = new ArrayList<>();
advices.add(compoundTriggerAdvice());
pm.setAdviceChain(advices);
pm.setTrigger(compoundTrigger());
pm.setMaxMessagesPerPoll(sftpEnvironment.getMaxMessagesPerPoll());
return pm;
}
#Bean
public CronTrigger cronTrigger() {
if (LOG.isDebugEnabled()) {
return new CronTrigger(sftpEnvironment.getPollerCronExpressionWhenDebugModeIsEnabled());
} else {
return new CronTrigger(sftpEnvironment.getPollerCronExpression());
}
}
#Bean
public PeriodicTrigger periodicTrigger() {
return new PeriodicTrigger(sftpEnvironment.getPeriodicTriggerInMillis());
}
#Bean
public CompoundTrigger compoundTrigger() {
return new CompoundTrigger(cronTrigger());
}
#Bean
public CompoundTriggerAdvice compoundTriggerAdvice() {
return new CompoundTriggerAdvice(compoundTrigger(), periodicTrigger());
}
#Bean
public FileSystemPersistentAcceptOnceFileListFilter filelistFilter(MetadataStore datastore) {
return new FileSystemPersistentAcceptOnceFileListFilter((JdbcMetadataStore) datastore, "INT");
}
#Bean
public PlatformTransactionManager transactionManager() {
return new org.springframework.integration.transaction.PseudoTransactionManager();
}
#Bean
DataSource dataSource() throws SQLException {
OracleDataSource dataSource = new OracleDataSource();
dataSource.setUser(databaseProperties.getOracleUsername());
dataSource.setPassword(databaseProperties.getOraclePassword());
dataSource.setURL(databaseProperties.getOracleUrl());
dataSource.setImplicitCachingEnabled(true);
dataSource.setFastConnectionFailoverEnabled(true);
return dataSource;
}
/**
* Creates a {#link JdbcMetadataStore} for the de-duplication logic.
*
* <p>This method uses the "REGION" column of the metadatastore table to differentiate between
* multiple apps. The value of the "REGION" column is set equal to the app-name.
*
* #return a JDBC metadata store
* #throws SQLException in case an exception occurs during connection to SQL database
*/
#Bean
public MetadataStore metadataStore() throws SQLException {
JdbcMetadataStore jdbcMetadataStore = new JdbcMetadataStore(dataSource());
if (!Constants.app2.equals(appName)) {
jdbcMetadataStore.setRegion(appName);
}
return jdbcMetadataStore;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return message -> {
File file = (File) message.getPayload();
FileDto fileDto = new FileDto(file);
fileHandler.handle(fileDto);
LOG.info("controller is here ");
try {
if (sftpEnvironment.isDeleteLocalFiles()) {
Files.deleteIfExists(Paths.get(file.toString()));
}
} catch (IOException e) {
// TODO retry/report/handle gracefully
LOG.error(String.format("MessageHandler had error message=%s", message), e);
}
};
}
}
The synchronizer doesn't support recursion. Use the outbound gateway with a recursive mget command instead.
https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound-gateway
Using the mget Command
mget retrieves multiple remote files based on a pattern and supports the following options:
-P: Preserve the timestamps of the remote files.
-R: Retrieve the entire directory tree recursively.
-x: Throw an exception if no files match the pattern (otherwise, an empty list is returned).
-D: Delete each remote file after successful transfer. If the transfer is ignored, the remote file is not deleted, because the FileExistsMode is IGNORE and the local file already exists.
The message payload resulting from an mget operation is a List<File> object (that is, a List of File objects, each representing a retrieved file).
EDIT
Here is an example using the java DSL...
#SpringBootApplication
public class So75180789Application {
public static void main(String[] args) {
SpringApplication.run(So75180789Application.class, args);
}
#Bean
IntegrationFlow flow(DefaultSftpSessionFactory sf) {
return IntegrationFlows.fromSupplier(() -> "foo/*", // remote dir
e -> e.poller(Pollers.fixedDelay(5000)))
.handle(Sftp.outboundGateway(sf, Command.MGET, "payload")
.options(Option.RECURSIVE)
.fileExistsMode(FileExistsMode.IGNORE)
.localDirectoryExpression("'/tmp/' + #remoteDirectory")) // re-create tree locally
.split()
.log()
.get();
}
#Bean
DefaultSftpSessionFactory sf(#Value("${host}") String host,
#Value("${username}") String user, #Value("${pw}") String pw) {
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(host);
sf.setUser(user);
sf.setPassword(pw);
sf.setAllowUnknownKeys(true);
return sf;
}
}
I have a fixedlength input file reading by using SPRING BATCH.
I have already implemented Job, Step, Processor, etc.
Here are the sample code.
#Configuration
public class BatchConfig {
private JobBuilderFactory jobBuilderFactory;
private StepBuilderFactory stepBuilderFactory;
#Value("${inputFile}")
private Resource resource;
#Autowired
public BatchConfig(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory) {
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
}
#Bean
public Job job() {
return this.jobBuilderFactory.get("JOB-Load")
.start(fileReadingStep())
.build();
}
#Bean
public Step fileReadingStep() {
return stepBuilderFactory.get("File-Read-Step1")
.<Employee,EmpOutput>chunk(1000)
.reader(itemReader())
.processor(new CustomFileProcesser())
.writer(new CustomFileWriter())
.faultTolerant()
.skipPolicy(skipPolicy())
.build();
}
#Bean
public FlatFileItemReader<Employee> itemReader() {
FlatFileItemReader<Employee> flatFileItemReader = new FlatFileItemReader<Employee>();
flatFileItemReader.setResource(resource);
flatFileItemReader.setName("File-Reader");
flatFileItemReader.setLineMapper(LineMapper());
return flatFileItemReader;
}
#Bean
public LineMapper<Employee> LineMapper() {
DefaultLineMapper<Employee> defaultLineMapper = new DefaultLineMapper<Employee>();
FixedLengthTokenizer fixedLengthTokenizer = new FixedLengthTokenizer();
fixedLengthTokenizer.setNames(new String[] { "employeeId", "employeeName", "employeeSalary" });
fixedLengthTokenizer.setColumns(new Range[] { new Range(1, 9), new Range(10, 20), new Range(20, 30)});
fixedLengthTokenizer.setStrict(false);
defaultLineMapper.setLineTokenizer(fixedLengthTokenizer);
defaultLineMapper.setFieldSetMapper(new CustomFieldSetMapper());
return defaultLineMapper;
}
#Bean
public JobSkipPolicy skipPolicy() {
return new JobSkipPolicy();
}
}
For Processing I have added some sample code What I need, But if I add BufferedReader here then it's taking more times to do the job.
#Component
public class CustomFileProcesser implements ItemProcessor<Employee, EmpOutput> {
#Override
public EmpOutput process(Employee item) throws Exception {
EmpOutput emp = new EmpOutput();
emp.setEmployeeSalary(checkSal(item.getEmployeeSalary()));
return emp;
}
public String checkSal(String sal) {
// need to read the another file
// required to do some kind of validation
// after that final result need to return
File f1 = new File("C:\\Users\\John\\New\\salary.txt");
FileReader fr;
try {
fr = new FileReader(f1);
BufferedReader br = new BufferedReader(fr);
String s = br.readLine();
while (s != null) {
String value = s.substring(5, 7);
if(value.equals(sal))
sal = value;
else
sal = "5000";
s = br.readLine();
}
} catch (Exception e) {
e.printStackTrace();
}
return sal;
}
// other fields need to check by reading different different file.
// These new files contains more than 30k records.
// all are fixedlength file.
// I need to get the field by giving the index
}
While doing the processing for one or more field, I need to check In another file by reading that file (it's a file I will read from fileSystem/Cloud).
While processing the data for 5 fields I need to read 5 different different file again, I will check the fields details inside those file and then I will gererate the result , That result will process forther.
You can cache the content of the file in memory and do your check against the cache instead of re-reading the entire file from disk for each item.
You can find an example here: Spring Batch With Annotation and Caching.
MultiResourceItemReader reads all files sequentially.
I want once one file read completely, it should call processor/writer.it should not read next file.
Since file content is not constant, i can't go with chunk size.
Any idea on chunk policy to decide end of file content?
I think you should write a step which read/process/write only one file with a "single file item reader" (like FlatFileItemReader). And repeat the step while there are files remainig.
Spring batch gives you a feature to do so : conditional flows and in particular the programmatic flow decision which gives you a smart way to decide when to stop a loop between steps (when there is not file any more)
And since you will not be able to give a constant input file name to your reader, you should also have a look at Late binding section.
Hope this will be enough to help you. Please, make comments if you need more details.
Using MultiResourceItemReader, assigning multiple file reasources.
Using custom file reader as delegate, reading a file completely
For reading file completely, come up with a logic
#Bean
public MultiResourceItemReader<SimpleFileBean> simpleReader()
{
Resource[] resourceList = getFileResources();
if(resourceList == null) {
System.out.println("No input files available");
}
MultiResourceItemReader<SimpleFileBean> resourceItemReader = new MultiResourceItemReader<SimpleFileBean>();
resourceItemReader.setResources(resourceList);
resourceItemReader.setDelegate(simpleFileReader());
return resourceItemReader;
}
#Bean
SimpleInboundReader simpleFileReader() {
return new SimpleInboundReader(customSimpleFileReader());
}
#Bean
public FlatFileItemReader customSimpleFileReader() {
return new FlatFileItemReaderBuilder()
.name("customFileItemReader")
.lineMapper(new PassThroughLineMapper())
.build();
}
public class SimpleInboundReader implements ResourceAwareItemReaderItemStream{
private Object currentItem = null;
private FileModel fileModel = null;
private String fileName = null;
private boolean fileRead = false;
private ResourceAwareItemReaderItemStream<String> delegate;
public SimpleInboundReader(ResourceAwareItemReaderItemStream<String> delegate) {
this.delegate = delegate;
}
#Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
delegate.open(executionContext);
}
#Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
delegate.update(executionContext);
}
#Override
public void close() throws ItemStreamException {
delegate.close();
}
#Override
public void setResource(Resource resource) {
fileName = resource.getFilename();
this.delegate.setResource(resource);
}
String getNextLine() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {
return delegate.read();
}
#Override
public SimpleFileBean read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
SimpleFileBean simpleFileBean = null;
String currentLine = null;
currentLine=delegate.read();
if(currentLine != null) {
simpleFileBean = new SimpleFileBean();
simpleFileBean.getLines().add(currentLine);
while ((currentLine = getNextLine()) != null) {
simpleFileBean.getLines().add(currentLine);
}
return simpleFileBean;
}
return null;
}
}
There seem to be an issue when I use AggregatingReplyingKafkaTemplate with template.setReturnPartialOnTimeout(true) in that, it returns timeout exception even if partial results are available from consumers.
In example below, I have 3 consumers to reply to the request topic and i've set the reply timeout at 10 seconds. I've explicitly delayed the response of Consumer 3 to 11 seconds, however, I expect the response back from Consumer 1 and 2, so, I can return partial results. However, I am getting KafkaReplyTimeoutException. Appreciate your inputs. Thanks.
I follow the code based on the Unit Test below.
[ReplyingKafkaTemplateTests][1]
I've provided the actual code below:
#RestController
public class SumController {
#Value("${kafka.bootstrap-servers}")
private String bootstrapServers;
public static final String D_REPLY = "dReply";
public static final String D_REQUEST = "dRequest";
#ResponseBody
#PostMapping(value="/sum")
public String sum(#RequestParam("message") String message) throws InterruptedException, ExecutionException {
AggregatingReplyingKafkaTemplate<Integer, String, String> template = aggregatingTemplate(
new TopicPartitionOffset(D_REPLY, 0), 3, new AtomicInteger());
String resultValue ="";
String currentValue ="";
try {
template.setDefaultReplyTimeout(Duration.ofSeconds(10));
template.setReturnPartialOnTimeout(true);
ProducerRecord<Integer, String> record = new ProducerRecord<>(D_REQUEST, null, null, null, message);
RequestReplyFuture<Integer, String, Collection<ConsumerRecord<Integer, String>>> future =
template.sendAndReceive(record);
future.getSendFuture().get(5, TimeUnit.SECONDS); // send ok
System.out.println("Send Completed Successfully");
ConsumerRecord<Integer, Collection<ConsumerRecord<Integer, String>>> consumerRecord = future.get(10, TimeUnit.SECONDS);
System.out.println("Consumer record size "+consumerRecord.value().size());
Iterator<ConsumerRecord<Integer, String>> iterator = consumerRecord.value().iterator();
while (iterator.hasNext()) {
currentValue = iterator.next().value();
System.out.println("response " + currentValue);
System.out.println("Record header " + consumerRecord.headers().toString());
resultValue = resultValue + currentValue + "\r\n";
}
} catch (Exception e) {
System.out.println("Error Message is "+e.getMessage());
}
return resultValue;
}
public AggregatingReplyingKafkaTemplate<Integer, String, String> aggregatingTemplate(
TopicPartitionOffset topic, int releaseSize, AtomicInteger releaseCount) {
//Create Container Properties
ContainerProperties containerProperties = new ContainerProperties(topic);
containerProperties.setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
//Set the consumer Config
//Create Consumer Factory with Consumer Config
DefaultKafkaConsumerFactory<Integer, Collection<ConsumerRecord<Integer, String>>> cf =
new DefaultKafkaConsumerFactory<>(consumerConfigs());
//Create Listener Container with Consumer Factory and Container Property
KafkaMessageListenerContainer<Integer, Collection<ConsumerRecord<Integer, String>>> container =
new KafkaMessageListenerContainer<>(cf, containerProperties);
// container.setBeanName(this.testName);
AggregatingReplyingKafkaTemplate<Integer, String, String> template =
new AggregatingReplyingKafkaTemplate<>(new DefaultKafkaProducerFactory<>(producerConfigs()), container,
(list, timeout) -> {
releaseCount.incrementAndGet();
return list.size() == releaseSize;
});
template.setSharedReplyTopic(true);
template.start();
return template;
}
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test_id");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringDeserializer.class);
return props;
}
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
// list of host:port pairs used for establishing the initial connections to the Kakfa cluster
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
org.apache.kafka.common.serialization.StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.class);
return props;
}
public ProducerFactory<Integer,String> producerFactory() {
return new DefaultKafkaProducerFactory<>(producerConfigs());
}
#KafkaListener(id = "def1", topics = { D_REQUEST}, groupId = "D_REQUEST1")
#SendTo // default REPLY_TOPIC header
public String dListener1(String in) throws InterruptedException {
return "First Consumer : "+ in.toUpperCase();
}
#KafkaListener(id = "def2", topics = { D_REQUEST}, groupId = "D_REQUEST2")
#SendTo // default REPLY_TOPIC header
public String dListener2(String in) throws InterruptedException {
return "Second Consumer : "+ in.toLowerCase();
}
#KafkaListener(id = "def3", topics = { D_REQUEST}, groupId = "D_REQUEST3")
#SendTo // default REPLY_TOPIC header
public String dListener3(String in) throws InterruptedException {
Thread.sleep(11000);
return "Third Consumer : "+ in;
}
}
'''
[1]: https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/test/java/org/springframework/kafka/requestreply/ReplyingKafkaTemplateTests.java
template.setReturnPartialOnTimeout(true) simply means the template will consult the release strategy on timeout (with the timeout argument = true, to tell the strategy it's a timeout rather than a delivery call).
It must return true to release the partial result.
This is to allow you to look at (and possibly modify) the list to decide whether you want to release or discard.
Your strategy ignores the timeout parameter:
(list, timeout) -> {
releaseCount.incrementAndGet();
return list.size() == releaseSize;
});
You need return timeout ? true : { ... }.
We are working with project reactor and having a huge problem right now. This is how we produce (publish our data):
public Flux<String> getAllFlux() {
return Flux.<String>create(sink -> {
new Thread(){
public void run(){
Iterator<Cache.Entry<String, MyObject>> iterator = getAllIterator();
ObjectMapper mapper = new ObjectMapper();
while(iterator.hasNext()) {
try {
sink.next(mapper.writeValueAsString(iterator.next().getValue()));
} catch (IOException e) {
e.printStackTrace();
}
}
sink.complete();
}
} .start();
});
}
As you can see we are taking data from an iterator and are publishing each item in that iterator as a json string. Our subscriber does the following:
flux.subscribe(new Subscriber<String>() {
private Subscription s;
int amount = 1; // the amount of received flux payload at a time
int onNextAmount;
String completeItem="";
ObjectMapper mapper = new ObjectMapper();
#Override
public void onSubscribe(Subscription s) {
System.out.println("subscribe");
this.s = s;
this.s.request(amount);
}
#Override
public void onNext(String item) {
MyObject myObject = null;
try {
System.out.println(item);
myObject = mapper.readValue(completeItem, MyObject.class);
System.out.println(myObject.toString());
} catch (IOException e) {
System.out.println(item);
System.out.println("failed: " + e.getLocalizedMessage());
}
onNextAmount++;
if (onNextAmount % amount == 0) {
this.s.request(amount);
}
}
#Override
public void onError(Throwable t) {
System.out.println(t.getLocalizedMessage())
}
#Override
public void onComplete() {
System.out.println("completed");
});
}
As you can see we are simply printing the String item which we receive and parsing it into an object using jackson wrapper. The problem we got now is that for most of our items everything works fine:
{"itemId": "someId", "itemDesc", "some description"}
But for some items the String is cut off like this for example:
{"itemId": "some"
And the next item after that would be
"Id", "itemDesc", "some description"}
There is no pattern for those cuts. It is completely random and it is different everytime we run that code. Ofcourse our jackson is gettin an error Unexpected end of Input with that behaviour.
So what is causing such a behaviour and how can we solve it?
Solution:
Send the Object inside the flux instead of the String:
public Flux<ItemIgnite> getAllFlux() {
return Flux.create(sink -> {
new Thread(){
public void run(){
Iterator<Cache.Entry<String, ItemIgnite>> iterator = getAllIterator();
while(iterator.hasNext()) {
sink.next(iterator.next().getValue());
}
}
} .start();
});
}
and use the following produces type:
#RequestMapping(value="/allFlux", method=RequestMethod.GET, produces="application/stream+json")
The key here is to use stream+json and not only json.