Set S3 bucket dynamically using S3 Message Handler and Spring Integration - spring

How can I pass in an 'asset' POJO to the S3MessageHandler using Spring Integration and change the bucket?
I would like to be able to change the bucket, based on a folder in the asset, to something like 'root-bucket/a/b/c.'
Each asset can potentially have a different sub-path.
#Bean
IntegrationFlow flowUploadAssetToS3()
{
return flow -> flow
.channel(this.channelUploadAsset)
.channel(c -> c.queue(10))
.publishSubscribeChannel(c -> c
.subscribe(s -> s
// Publish the asset?
.<File>handle((p, h) -> this.publishS3Asset(
p,
(Asset)h.get("asset")))
// Set the action
.enrichHeaders(e -> e
.header("s3Command", Command.UPLOAD.name()))
// Upload the file
.handle(this.s3MessageHandler())));
}
MessageHandler s3MessageHandler()
{
return new S3MessageHandler(amazonS3, "root-bucket");
}
Thanks to the answer below, I got this working like so:
final String bucket = "'my-root";
final Expression bucketExpression = PARSER.parseExpression(bucket);
final Expression keyExpression = PARSER.parseExpression("headers['asset'].bucket");
final S3MessageHandler handler = new S3MessageHandler(amazonS3, bucketExpression);
handler.setKeyExpression(keyExpression);
return handler;

There is bucketExpression option which is SpEL to let to resolve a bucket from the requestMessage:
private static SpelExpressionParser PARSER = new SpelExpressionParser();
MessageHandler s3MessageHandler() {
Expression bucketExpression =
PARSER.parseExpression("payload.bucketProperty");
return new S3MessageHandler(amazonS3, bucketExpression);
}
Also be aware that root-bucket/a/b/c. isn't bucket name. You should consider to distinguish to bucket and the key to build that complex sub-folder path. For that purpose there is keyExpression option with similar functionality to resolve the key in bucket against requestMessage.

Related

Conditional File movement in Spring Integration File Support

I am implementing one Spring Integration work flow as below.
IntegrationFlows.from("inputFileProcessorChannel")
.split(fileSplitterSpec, spec -> {})
.transform(lineItemTransformer)
.handle(httpRequestExecutingMessageHandler)
.transform(reportDataAggregator)
.aggregate(aggregatorSpec -> aggregatorSpec.requiresReply(false))
.channel("reportGeneratorChannel")
.get();
Now, once the above flow is completed, I need to move the input file to a archive directory. The decision to decide on the destination directory is based on a message header processingFailed and this header is added in .transform(reportDataAggregator) step in the flow. To move this files I have create another flow as in below code
IntegrationFlows.from(MessageChannels.direct("inputFileProcessorChannel"))
.routeToRecipients(routerSpec -> {
routerSpec.recipient("processedFileMoverChannel", createMessageSelector(Boolean.FALSE))
.recipient("failedFileMoverChannel", createMessageSelector(Boolean.TRUE));
})
.get();
Selector method
private MessageSelector createMessageSelector(Boolean ruleBoolean) {
return message -> ruleBoolean.equals(message.getHeaders().get("processingFailed"));
}
Report Channel flow below
IntegrationFlows.from("reportGeneratorChannel")
.transform(executionReportTransformer)
.handle(reportWritingMessageHandlerSpec)
.get();
But, as expected with this flow, File movement is not done as the said header is not present into the flow execution.
So, How to achieve this goal to Execute the file mover flow after the report file is created?
The FileSplitter populates for us this headers for each line to produce:
#Override
protected boolean willAddHeaders(Message<?> message) {
Object payload = message.getPayload();
return payload instanceof File || payload instanceof String;
}
#Override
protected void addHeaders(Message<?> message, Map<String, Object> headers) {
File file = null;
if (message.getPayload() instanceof File) {
file = (File) message.getPayload();
}
else if (message.getPayload() instanceof String) {
file = new File((String) message.getPayload());
}
if (file != null) {
if (!headers.containsKey(FileHeaders.ORIGINAL_FILE)) {
headers.put(FileHeaders.ORIGINAL_FILE, file);
}
if (!headers.containsKey(FileHeaders.FILENAME)) {
headers.put(FileHeaders.FILENAME, file.getName());
}
}
}
So, even if you done with an aggregation and ready to send a message into that .channel("reportGeneratorChannel"), you still have access to those file-related headers.
Making this reportGeneratorChannel as a PublishSubscribeChannel and move that "file mover flow" over there, would do the trick for you.
By the way: what you have so far with a IntegrationFlows.from(MessageChannels.direct("inputFileProcessorChannel")) and a second flow on the same channel would lead you to round-robin dispatching. That's not pub-sub distribution. See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel

Stream an object, Send request with completable future and assign result to the object

I have a list of Object Signers. For each of this signers I need to make a ReST request to get their signing URL. I am trying to do it with completable futures so all ReST requests can be send in parallel, then I need to set that URL in each of the signers, so this operation will not return new signers just update the ones that I am already iterating.
I have this code that is already working, but I think that could be improved.
List<Signer> signers=......
List<CompletableFuture> futures = signers.stream()
.map(signer -> CompletableFuture.completedFuture(signer))
.map(future -> future.thenCombine( CompletableFuture.supplyAsync(()-> signatureService.getSigningUrl(future.join().getSignerId())),
(signer, url) -> {
signer.setUrl(url);
return url;
}
)).collect(toList());
futures.stream()
.map(CompletableFuture::join)
.collect(toList());
Could I replace this
futures.stream()
.map(CompletableFuture::join)
.collect(toList());
With this?
futures.stream().forEach(CompletableFuture::join)
I don't like to return that because it was already used setting it in the signer. and don't like the second collect(toList()) because I am not trying to collect anything at that time.
What other implementation would you use?
No. futures.stream().forEach(CompletableFuture::join) returns void whereas futures.stream().map(CompletableFuture::join).collect(toList()); returns CompletableFuture<List<?>>.
They both are meant for different purposes. But both does one thing common(i.e, blocks the main thread till all the completablefutures are finished).
I would write your same code bit differently using CompletableFuture.allOf.
Stream<CompletableFuture<String>> streamFutures = signers.stream()
.map(signer -> CompletableFuture.completedFuture(signer))
.map(future -> future.thenCombine(CompletableFuture.supplyAsync(() -> signatureService.getSigningUrl(future.join().getSignerId())),
(signer, url) -> {
signer.setUrl(url);
return url;
}
));
CompletableFuture<String> [] futureArr = streamFutures.toArray(size -> new CompletableFuture[size]);
List<CompletableFuture<String>> futures = Arrays.asList(futureArr);
CompletableFuture<Void> allFuturesVoid = CompletableFuture.allOf(futureArr);
allFuturesVoid.join();
CompletableFuture<List<?>> allCompletableFuture = allFuturesVoid.thenApply(future -> futures.stream().map(completableFuture -> completableFuture.join()).collect(Collectors.toList()));
There is good tutorial here https://m-hewedy.blogspot.com/2017/02/completablefutureallof-that-doenst.html and here https://medium.com/#senanayake.kalpa/fantastic-completablefuture-allof-and-how-to-handle-errors-27e8a97144a0.

How to extract and manipulate data within a Nifi processor

I'm trying to write a custom Nifi processor which will take in the contents of the incoming flow file, perform some math operations on it, then write the results into an outgoing flow file. Is there a way to dump the contents of the incoming flow file into a string or something? I've been searching for a while now and it doesn't seem that simple. If anyone could point me toward a good tutorial that deals with doing something like that it would be greatly appreciated.
The Apache NiFi Developer Guide documents the process of creating a custom processor very well. In your specific case, I would start with the Component Lifecycle section and the Enrich/Modify Content pattern. Any other processor which does similar work (like ReplaceText or Base64EncodeContent) would be good examples to learn from; all of the source code is available on GitHub.
Essentially you need to implement the #onTrigger() method in your processor class, read the flowfile content and parse it into your expected format, perform your operations, and then re-populate the resulting flowfile content. Your source code will look something like this:
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile flowFile = session.get();
if (flowFile == null) {
return;
}
final ComponentLog logger = getLogger();
AtomicBoolean error = new AtomicBoolean();
AtomicReference<String> result = new AtomicReference<>(null);
// This uses a lambda function in place of a callback for InputStreamCallback#process()
processSession.read(flowFile, in -> {
long start = System.nanoTime();
// Read the flowfile content into a String
// TODO: May need to buffer this if the content is large
try {
final String contents = IOUtils.toString(in, StandardCharsets.UTF_8);
result.set(new MyMathOperationService().performSomeOperation(contents));
long stop = System.nanoTime();
if (getLogger().isDebugEnabled()) {
final long durationNanos = stop - start;
DecimalFormat df = new DecimalFormat("#.###");
getLogger().debug("Performed operation in " + durationNanos + " nanoseconds (" + df.format(durationNanos / 1_000_000_000.0) + " seconds).");
}
} catch (Exception e) {
error.set(true);
getLogger().error(e.getMessage() + " Routing to failure.", e);
}
});
if (error.get()) {
processSession.transfer(flowFile, REL_FAILURE);
} else {
// Again, a lambda takes the place of the OutputStreamCallback#process()
FlowFile updatedFlowFile = session.write(flowFile, (in, out) -> {
final String resultString = result.get();
final byte[] resultBytes = resultString.getBytes(StandardCharsets.UTF_8);
// TODO: This can use a while loop for performance
out.write(resultBytes, 0, resultBytes.length);
out.flush();
});
processSession.transfer(updatedFlowFile, REL_SUCCESS);
}
}
Daggett is right that the ExecuteScript processor is a good place to start because it will shorten the development lifecycle (no building NARs, deploying, and restarting NiFi to use it) and when you have the correct behavior, you can easily copy/paste into the generated skeleton and deploy it once.

Spring webflux - multi Mono

I do not know or ask this question except here, if it's not the place I apologize.
I am currently working on an application using spring webflux and I have a problem with the use of Mono and Flux.
Here I have a REST request that comes with a simple bean that contains attributes including a list. This list is iterated to use a responsive mongo call that returns a Mono (findOne).
But I do not think I've found the right way to do it:
#PostMapping
#RequestMapping("/check")
public Mono<ContactCheckResponse> check(#RequestBody List<ContactCheckRequest> list) {
final ContactCheckResponse response = new ContactCheckResponse();
response.setRsnCode("00");
response.setRspnCode("0000");
LOG.debug("o--> person - check {} items", list.size());
final List<ContactCheckResponse.Contact> contacts = new ArrayList<>();
response.setContacts(contacts);
return Mono.fromCallable(() -> {
list.stream().forEach( c -> {
Boolean exists = contactRespository.findOneByThumbprint(c.getIdentifiant()).block() != null;
ContactCheckResponse.Contact responseContact = new ContactCheckResponse.Contact();
responseContact.setExist(exists);
responseContact.setIdentifiant(c.getIdentifiant());
responseContact.setRsnCode("00");
responseContact.setRspnCode("0000");
response.getContacts().add(responseContact);
});
return response;
});
}
the fact of having to make a block does not seem to me in the idea "reactive" but I did not find how to do otherwise.
Could someone help me find the best way to do this task?
Thank you
Something along these lines:
return Flux.fromIterable(list)
.flatMap(c -> contactRespository.findOneByThumbprint(c.getIdentifiant())
.map(r -> r != null)
.map(exists -> {
ContactCheckResponse.Contact responseContact = new ContactCheckResponse.Contact();
...
return responseContact;
})
)
.reduce(response, (r,c) -> {
response.getContacts().add(responseContact);
return response;
});
Create a Flux from the list, create a contact for each entry and reduce everything to a Mono.

How to create SubCommunities using the Social Business Toolkit Java API?

In the SDK Javadoc, the Community class does not have a "setParentCommunity" method but the CommunityList class does have a getSubCommunities method so there must be a programmatic way to set a parent Community's Uuid on new Community creation. The REST API mentions a "rel="http://www.ibm.com/xmlns/prod/sn/parentcommunity" element". While looking for clues I check an existing Subcommunity's XmlDataHandler's nodes and found a link element. I tried getting the XmlDataHandler for a newly-created Community and adding a link node with href, rel and type nodes similar to those in the existing Community but when trying to update or re-save the Community I got a bad request error. Actually even when I tried calling dataHandler.setData(n) where n was set as Node n=dataHandler.getData(); without any changes, then calling updateCommunity or save I got the same error, so it appears that manipulating the dataHandler XML is not valid.
What is the recommended way to specify a parent Community when creating a new Community so that it is created as a SubCommunity ?
The correct way to create a sub-community programatically is to modify the POST request body for community creation - here is the link to the Connections 45 infocenter - http://www-10.lotus.com/ldd/appdevwiki.nsf/xpDocViewer.xsp?lookupName=IBM+Connections+4.5+API+Documentation#action=openDocument&res_title=Creating_subcommunities_programmatically_ic45&content=pdcontent
We do not have support in the SBT SDK to do this using CommunityService APIs. We need to use low level Java APIs using Endpoint and ClientService classes to directly call the REST APIs with the appropriate request body.
I'd go ahead and extend the class CommunityService
then go ahead and add CommunityService
https://github.com/OpenNTF/SocialSDK/blob/master/src/eclipse/plugins/com.ibm.sbt.core/src/com/ibm/sbt/services/client/connections/communities/CommunityService.java
Line 605
public String createCommunity(Community community) throws CommunityServiceException {
if (null == community){
throw new CommunityServiceException(null, Messages.NullCommunityObjectException);
}
try {
Object communityPayload;
try {
communityPayload = community.constructCreateRequestBody();
} catch (TransformerException e) {
throw new CommunityServiceException(e, Messages.CreateCommunityPayloadException);
}
String communityPostUrl = resolveCommunityUrl(CommunityEntity.COMMUNITIES.getCommunityEntityType(),CommunityType.MY.getCommunityType());
Response requestData = createData(communityPostUrl, null, communityPayload,ClientService.FORMAT_CONNECTIONS_OUTPUT);
community.clearFieldsMap();
return extractCommunityIdFromHeaders(requestData);
} catch (ClientServicesException e) {
throw new CommunityServiceException(e, Messages.CreateCommunityException);
} catch (IOException e) {
throw new CommunityServiceException(e, Messages.CreateCommunityException);
}
}
You'll want to change your communityPostUrl to match...
https://greenhouse.lotus.com/communities/service/atom/community/subcommunities?communityUuid=2fba29fd-adfa-4d28-98cc-05cab12a7c43
and where the Uuid here is the parent uuid.
I followed #PaulBastide 's recommendation and created a SubCommunityService class, currently only containing a method for creation. It wraps the CommunityService rather than subclassing it, since I found that preferrable. Here's the code in case you want to reuse it:
public class SubCommunityService {
private final CommunityService communityService;
public SubCommunityService(CommunityService communityService) {
this.communityService = communityService;
}
public Community createCommunity(Community community, String superCommunityId) throws ClientServicesException {
Object constructCreateRequestBody = community.constructCreateRequestBody();
ClientService clientService = communityService.getEndpoint().getClientService();
String entityType = CommunityEntity.COMMUNITY.getCommunityEntityType();
Map<String, String> params = new HashMap<>();
params.put("communityUuid", superCommunityId);
String postUrl = communityService.resolveCommunityUrl(entityType,
CommunityType.SUBCOMMUNITIES.getCommunityType(), params);
String newCommunityUrl = (String) clientService.post(postUrl, null, constructCreateRequestBody,
ClientService.FORMAT_CONNECTIONS_OUTPUT);
String communityId = newCommunityUrl.substring(newCommunityUrl.indexOf("communityUuid=")
+ "communityUuid=".length());
community.setCommunityUuid(communityId);
return community;
}
}

Resources