I am implementing one Spring Integration work flow as below.
IntegrationFlows.from("inputFileProcessorChannel")
.split(fileSplitterSpec, spec -> {})
.transform(lineItemTransformer)
.handle(httpRequestExecutingMessageHandler)
.transform(reportDataAggregator)
.aggregate(aggregatorSpec -> aggregatorSpec.requiresReply(false))
.channel("reportGeneratorChannel")
.get();
Now, once the above flow is completed, I need to move the input file to a archive directory. The decision to decide on the destination directory is based on a message header processingFailed and this header is added in .transform(reportDataAggregator) step in the flow. To move this files I have create another flow as in below code
IntegrationFlows.from(MessageChannels.direct("inputFileProcessorChannel"))
.routeToRecipients(routerSpec -> {
routerSpec.recipient("processedFileMoverChannel", createMessageSelector(Boolean.FALSE))
.recipient("failedFileMoverChannel", createMessageSelector(Boolean.TRUE));
})
.get();
Selector method
private MessageSelector createMessageSelector(Boolean ruleBoolean) {
return message -> ruleBoolean.equals(message.getHeaders().get("processingFailed"));
}
Report Channel flow below
IntegrationFlows.from("reportGeneratorChannel")
.transform(executionReportTransformer)
.handle(reportWritingMessageHandlerSpec)
.get();
But, as expected with this flow, File movement is not done as the said header is not present into the flow execution.
So, How to achieve this goal to Execute the file mover flow after the report file is created?
The FileSplitter populates for us this headers for each line to produce:
#Override
protected boolean willAddHeaders(Message<?> message) {
Object payload = message.getPayload();
return payload instanceof File || payload instanceof String;
}
#Override
protected void addHeaders(Message<?> message, Map<String, Object> headers) {
File file = null;
if (message.getPayload() instanceof File) {
file = (File) message.getPayload();
}
else if (message.getPayload() instanceof String) {
file = new File((String) message.getPayload());
}
if (file != null) {
if (!headers.containsKey(FileHeaders.ORIGINAL_FILE)) {
headers.put(FileHeaders.ORIGINAL_FILE, file);
}
if (!headers.containsKey(FileHeaders.FILENAME)) {
headers.put(FileHeaders.FILENAME, file.getName());
}
}
}
So, even if you done with an aggregation and ready to send a message into that .channel("reportGeneratorChannel"), you still have access to those file-related headers.
Making this reportGeneratorChannel as a PublishSubscribeChannel and move that "file mover flow" over there, would do the trick for you.
By the way: what you have so far with a IntegrationFlows.from(MessageChannels.direct("inputFileProcessorChannel")) and a second flow on the same channel would lead you to round-robin dispatching. That's not pub-sub distribution. See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel
Related
I am aware of the TransferManager and the .uploadFileList() and .uploadFileDirectory() methods, however they accept java.io.File types as arguments. I have a collection of byte array input streams containing jpeg image data. I don't want to create in-memory files to store this data before I upload it either.
So what I need is essentially what the S3 client's PutObjectRequest does but for a collection of InputStream objects. Also, if one upload fails, I want to abort the whole thing and not upload anything, much like how a database transaction will reverse the changes if something goes wrong along the way.
Is this possible with the Java SDK?
Before I share an answer, please consider upgrading...
fyi - TransferManager is deprecated, now supported as TransferManagerBuilder in JAVA AWS SDK, please consider upgrading if TransferManagerBuilder Object suits your needs.
now since you asked about TransferManager, you could either 1) copy the code below and replace the functionality/arguments with your custom in memory handling of the input stream and handle it in your custom function... or; 2) further below is another sample, try to use this as-is...
Github source modify with with inputstream and issue listed here
private def uploadFile(is: InputStream, s3ObjectName: String, metadata: ObjectMetadata) = {
try {
val putObjectRequest = new PutObjectRequest(bucketName, s3ObjectName,
is, metadata)
// TransferManager supports asynchronous uploads and downloads
val upload = transferManager.upload(putObjectRequest)
upload.addProgressListener(ExceptionReporter.wrap(UploadProgressListener(putObjectRequest)))
} catch {
case e: Exception => throw new RuntimeException(e)
}
}
Bonus, Nice custom answer here using sequence input streams
public void combineFiles() {
List<String> files = getFiles();
long totalFileSize = files.stream()
.map(this::getContentLength)
.reduce(0L, (f, s) -> f + s);
try {
try (InputStream partialFile = new SequenceInputStream(getInputStreamEnumeration(files))) {
ObjectMetadata resultFileMetadata = new ObjectMetadata();
resultFileMetadata.setContentLength(totalFileSize);
s3Client.putObject("bucketName", "resultFilePath", partialFile, resultFileMetadata);
}
} catch (IOException e) {
LOG.error("An error occurred while combining files. {}", e);
}
}
private Enumeration<? extends InputStream> getInputStreamEnumeration(List<String> files) {
return new Enumeration<InputStream>() {
private Iterator<String> fileNamesIterator = files.iterator();
#Override
public boolean hasMoreElements() {
return fileNamesIterator.hasNext();
}
#Override
public InputStream nextElement() {
try {
return new FileInputStream(Paths.get(fileNamesIterator.next()).toFile());
} catch (FileNotFoundException e) {
System.err.println(e.getMessage());
throw new RuntimeException(e);
}
}
};
}
Staring with the tutorial code at benwilcock/spring-rsocket-demo I am trying to write a server that replicates messages to a second server before responding to a client.
To try to debug my issues I am only attempting a trivial ping-pong exchange between servers. Only when the second server responds to the pong message should the first server reply to the client:
#MessageMapping("request-response")
Mono<Message> requestResponse(final Message request) {
// register a mono that will be completed when replication to another server has happened
String uuid = UUID.randomUUID().toString();
Mono<Message> deferred = Mono.create(sink -> replicationNexus.registerRequest(uuid, sink));
// FIXME attempt to send a nested request-response message that will complete the outer message later
this.requesterMono.flatMap(requester -> requester.route("pong")
.data(uuid)
.retrieveMono(String.class))
.subscribeOn(Schedulers.elastic())
.subscribe( uuid2 -> replicationNexus.complete(uuid2, new Message(SERVER, RESPONSE)));
// return the deferred work that will be completed by the pong response
return deferred;
}
That logic is trying to use this answer to create a connection to the second server that will reconnect:
this.requesterMono = builder.rsocketConnector(connector -> connector
.reconnect(Retry.fixedDelay(Integer.MAX_VALUE, Duration.ofSeconds(1))))
.connectTcp("localhost", otherPort).cache();
To complete the picture here is the trivial ping-pong logic:
#MessageMapping("pong")
public Mono<String> pong(String m) {
return Mono.just(m);
}
and here is the logic that holds the state of the outer client response that is completed when the other server responds:
public class ReplicationNexus<T> {
final Map<String, MonoSink<T>> requests = new ConcurrentHashMap<>();
public void registerRequest(String v, MonoSink<T> sink) {
requests.put(v, sink);
}
public boolean complete(String uuid, T message) {
Optional<MonoSink<T>> sink = Optional.of(requests.get(uuid));
if( sink.isPresent() ){
sink.get().success(message);
}
return sink.isPresent();
}
}
Debugging the second server it never runs the pong method. It seems that the first server does not actually send the inner request message.
What is the correct way to run an inner request-response exchange that completes an outer message exchange with automated reconnection logic?
Not sure if I'm missing some of the complexity of your question, but if the middle server is just activing like a proxy I'd start with the simplest case of chaining through the calls. I feel like I'm missing some nuance of the question, so let's work through that next.
#MessageMapping("runCommand")
suspend fun runCommandX(
request: CommandRequest,
): Mono<String> {
val uuid = UUID.randomUUID().toString()
return requesterMono
.flatMap { requester: RSocketRequester ->
requester.route("pong")
.data("TEST")
.retrieveMono(String::class.java)
}
.doOnSubscribe {
// register request with uuid
}
.doOnSuccess {
// register completion
}
.doOnError {
// register failure
}
}
Generally if you can avoid calling subscribe yourself in typical spring/reactive/rsocket code. You want the framework to do this for you.
I am trying to read an Excel file in manipulate it or add new data to it and write it back out. I am also trying to do this a complete reactive process using Flux and Mono. The Idea is to return the resulting file or bytearray via a webservice.
My question is how do I get a InputStream and OutputStream in a non blocking way?
I am using the Apache Poi library to read and generate the Excel File.
I currently have a solution based around a mix of Mono.fromCallable() and Blocking code getting the Input Stream.
For example the webservice part is as follows.
#GetMapping(value = API_BASE_PATH + "/download", produces = "application/vnd.ms-excel")
public Mono<ByteArrayResource> download() {
Flux<TimeKeepingEntry> createExcel = excelExport.createDocument(false);
return createExcel.then(Mono.fromCallable(() -> {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
excelExport.getWb().write(outputStream);
return new ByteArrayResource(outputStream.toByteArray());
}).subscribeOn(Schedulers.elastic()));
}
And the Processing of the file:
public Flux<TimeKeepingEntry> createDocument(boolean all) {
Flux<TimeKeepingEntry> entries = null;
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
entries = service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").sort(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntry-> {
this.populateEntry(sheet, timeKeepingEntry);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return entries;
}
This works well enough but not very in line with Flux and Mono. Some guidance here would be good. I would prefer to have the whole sequence non-blocking.
Unfortunately the WorkbookFactory.create() operation is blocking, so you have to perform that operation using imperative code. However fetching each timeKeepingEntry can be done reactively. Your code would looks something like this:
public Flux<TimeKeepingEntry> createDocument() {
return Flux.generate(
this::getWorkbookSheet,
(sheet, sink) -> {
sink.next(getNextTimeKeepingEntryFrom(sheet));
},
this::closeWorkbook);
}
This will keep the workbook in memory, but will fetch each entry on demand when the elements of the Flux are requested.
I'm trying to write a custom Nifi processor which will take in the contents of the incoming flow file, perform some math operations on it, then write the results into an outgoing flow file. Is there a way to dump the contents of the incoming flow file into a string or something? I've been searching for a while now and it doesn't seem that simple. If anyone could point me toward a good tutorial that deals with doing something like that it would be greatly appreciated.
The Apache NiFi Developer Guide documents the process of creating a custom processor very well. In your specific case, I would start with the Component Lifecycle section and the Enrich/Modify Content pattern. Any other processor which does similar work (like ReplaceText or Base64EncodeContent) would be good examples to learn from; all of the source code is available on GitHub.
Essentially you need to implement the #onTrigger() method in your processor class, read the flowfile content and parse it into your expected format, perform your operations, and then re-populate the resulting flowfile content. Your source code will look something like this:
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile flowFile = session.get();
if (flowFile == null) {
return;
}
final ComponentLog logger = getLogger();
AtomicBoolean error = new AtomicBoolean();
AtomicReference<String> result = new AtomicReference<>(null);
// This uses a lambda function in place of a callback for InputStreamCallback#process()
processSession.read(flowFile, in -> {
long start = System.nanoTime();
// Read the flowfile content into a String
// TODO: May need to buffer this if the content is large
try {
final String contents = IOUtils.toString(in, StandardCharsets.UTF_8);
result.set(new MyMathOperationService().performSomeOperation(contents));
long stop = System.nanoTime();
if (getLogger().isDebugEnabled()) {
final long durationNanos = stop - start;
DecimalFormat df = new DecimalFormat("#.###");
getLogger().debug("Performed operation in " + durationNanos + " nanoseconds (" + df.format(durationNanos / 1_000_000_000.0) + " seconds).");
}
} catch (Exception e) {
error.set(true);
getLogger().error(e.getMessage() + " Routing to failure.", e);
}
});
if (error.get()) {
processSession.transfer(flowFile, REL_FAILURE);
} else {
// Again, a lambda takes the place of the OutputStreamCallback#process()
FlowFile updatedFlowFile = session.write(flowFile, (in, out) -> {
final String resultString = result.get();
final byte[] resultBytes = resultString.getBytes(StandardCharsets.UTF_8);
// TODO: This can use a while loop for performance
out.write(resultBytes, 0, resultBytes.length);
out.flush();
});
processSession.transfer(updatedFlowFile, REL_SUCCESS);
}
}
Daggett is right that the ExecuteScript processor is a good place to start because it will shorten the development lifecycle (no building NARs, deploying, and restarting NiFi to use it) and when you have the correct behavior, you can easily copy/paste into the generated skeleton and deploy it once.
Trying to upload an image and it keeps sending as just bytes, not an image file. This is a very simple call, I don't need to send any params other than the image itself. I don't know how to format logs so I won't post the error here unless requested to.
The service:
public interface FileUploadService {
#Multipart
#POST("upload_profile_picture")
Call<ResponseBody> uploadProfilePicture(#Part("profile_picture") RequestBody file);
}
The call being made (a file is generated earlier, had to remove this code because SO needs the post to be mainly words..dumb..):
// Generate the service from interface
FileUploadService service = ServiceGenerator.createService(FileUploadService.class, this);
// Create RequestBody instance from file
RequestBody requestFile =
RequestBody.create(MediaType.parse("image/*"), imageFile);
Log.d(LOG_TAG, "formed file");
// finally, execute the request
Call<ResponseBody> call = service.uploadProfilePicture(requestFile);
Log.d(LOG_TAG, "sending call");
call.enqueue(new Callback<ResponseBody>() {
#Override
public void onResponse(Call<ResponseBody> call,
Response<ResponseBody> response) {
Log.d(LOG_TAG, "success");
Log.d(LOG_TAG, response.toString());
}
#Override
public void onFailure(Call<ResponseBody> call, Throwable t) {
Log.d(LOG_TAG, "failure");
Log.e(LOG_TAG, t.getMessage());
}
});
Is the issue with the MediaType.parse method? I've tried "multipart/form-data", "image/jpeg", and the above as well and nothing has worked.
The server team has said they are receiving the call, just as bytes and no image file.
I keep getting a 400 because it's sending all bytes. How can I just send this? Do I need to send as a multipart or what? From what I've seen, you just need to tag the param in the method with #Body and do the above and it should all work. Can anybody tell me why this is happening? Thanks!
This is a known issue in Retrofit 2.
Edit: Support for OkHttp's MultipartBody.Part has been added in the final 2.0 release.
In order to get it working, you need to change your interface a little bit first:
#Multipart
#POST("upload_profile_picture")
Call<ResponseBody> uploadProfilePicture(#Part MultipartBody.Part file);
Then you have to create the Part and make the call like this:
MultipartBody.Part file = MultipartBody.Part.createFormData(
"file",
imageFile.getName(),
RequestBody.create(MediaType.parse("image/*"), imageFile));
Call<ResponseBody> call = service.uploadProfilePicture(file);