Moving files to archive directory using apache camel ftp component - spring

I have few files stored in a database, these files were fetched from a FTP server and processed.
I am trying to move these files to archive directory on FTP server and to do that I am using producerTemplate.
This is what I have done so far.
try {
DefaultCamelContext camelContext = new DefaultCamelContext();
ProducerTemplate template = camelContext.createProducerTemplate();
template.request("ftp://xxxxx/include=fileFromDb.xml&move=archive/fileFromDb.xml", outExchange -> {
String body = outExchange.getIn().getBody(String.class);
});
} catch (Exception ex) {
// update the status in database to indicate archive failed
}
But this is failing with following error and can't get this resolved.
WARN o.a.c.c.f.remote.RemoteFileProducer - Writing file failed with: Cannot write null body to file: archive/ID-192-168-1-113-tpgi-com-au-1627667653835-1-2
I have tried implementing solutions from other answers but they didn't worked for either.
If I write a custom route like below, it works fine but because I need perform post move actions I prefere using producer template.
from("ftp://xxxxx/include=fileFromDb.xml&move=archive/fileFromDb.xml")
.log("file moved successfully").

If I write a custom route like below, it works fine but because I need perform post move actions I prefere using producer template.
You could use onCompletion to call another route to handle the post move actions.
String sftpURI = "sftp:localhost:2222/upload" +
"?username="+username+"&password="+password
+ "&move=done";
String sftpPostMoveURI = "sftp:localhost:2222/upload/done" +
"?username="+username+"&password="+password
+ "&fileName=${headers.CamelFileName}";
from(sftpURI)
.routeId("ReadFileUsingSFTPRoute")
.onCompletion().onCompleteOnly()
.setBody().constant("")
.to("direct:postMoveActions")
.end()
.log("file ${headers.CamelFileName} moved");
from("direct:postMoveActions")
.routeId("PostMoveActionsRoute")
.log("doing post move actions!")
.pollEnrich().simple(sftpPostMoveURI).timeout(5000)
.log("File: ${body}");

Related

Updating Apache Camel JPA object in database triggers deadlock

So I have a Apache Camel route that reads Data elements from a JPA endpoint, converts them to DataConverted elements and stores them into a different database via a second JPA endpoint. Both endpoints are Oracle databases.
Now I want to set a flag on the original Data element that it got copied successfully. What is the best way to achieve that?
I tried it like that: saving the ID in the context and then reading it and accessing a dao method in the .onCompletion().onCompleteOnly().
from("jpa://Data")
.onCompletion().onCompleteOnly().process(ex -> {
var id = Long.valueOf(getContext().getGlobalOption("id"));
myDao().setFlag(id);
}).end()
.process(ex -> {
Data data = ex.getIn().getBody(Data.class);
DataConverted dataConverted = convertData(data);
ex.getMessage().setBody(data);
var globalOptions = getContext().getGlobalOptions();
globalOptions.put("id", data.getId().toString());
getContext().setGlobalOptions(globalOptions);
})
.to("jpa://DataConverted").end();
However, this seems to trigger a deadlock, the dao method is stalling on the commit of the update. The only explanation could be that the Data object gets locked by Camel and is still locked in the .onCompletion().onCompleteOnly() part of the route, therefore it can't get updated there.
Is there a better way to do it?
Have you tried using the recipient list EIP where first destination is the jpa:DataConverted endpoint and the second destination will be the endpoint to set the flag. This way both get the same message and will be executed sequentially.
https://camel.apache.org/components/3.17.x/eips/recipientList-eip.html
from("jpa://Data")
.process(ex -> {
Data data = ex.getIn().getBody(Data.class);
DataConverted dataConverted = convertData(data);
ex.getIn().setBody(data);
})
.recipientList(constant("direct:DataConverted","direct:updateFlag"))
.end();
from("direct:DataConverted")
.to("jpa://DataConverted")
.end();
from("direct:updateFlag")
.process(ex -> {
var id = ((MessageConverted) ex.getIn().getBody()).getId();
myDao().setFlag(id);
})
.end();
Keep in mind, you might want to make the route transactional by adding .transacted()
https://camel.apache.org/components/3.17.x/eips/transactional-client.html

Spring integration - Use afterCommit expression to write file to a processed directory

My spring integration application reads files from a fileshare, does some processing including api calls etc. In case something goes wrong in between I would like to use a afterRollbakExpression to write the file to a failed directory in the file share/ftp/sftp etc.
I found an example of doing the same to a local file directory as follows,
#Bean
TransactionSynchronizationFactory transactionSynchronizationFactory() {
ExpressionParser parser = new SpelExpressionParser();
ExpressionEvaluatingTransactionSynchronizationProcessor syncProcessor =
new ExpressionEvaluatingTransactionSynchronizationProcessor();
syncProcessor.setBeanFactory(applicationContext.getAutowireCapableBeanFactory());
//afterCommit expression moves the file to a processed directory
syncProcessor.setAfterCommitExpression(parser.parseExpression("payload.renameTo(new java.io.File(#inboundProcessedDirectory.path "
+ " + T(java.io.File).separator + payload.name))"));
//afterRollback expression moves the file to a failed directory
syncProcessor.setAfterRollbackExpression(parser.parseExpression("payload.renameTo(new java.io.File(#inboundFailedDirectory.path "
+ " + T(java.io.File).separator + payload.name))"));
return new DefaultTransactionSynchronizationFactory(syncProcessor);
}
I would like to do the same thing but write the file to the fileshare/ftp/sftp during a rollback scenario and not to a local directory.
I have a messageHandler which I invoke to write files using the integration flow to the smb fileshare. I dont know how do i invoke the following messagehandler as a AfterRollbackExpression,
#Bean(name = SMB_FILE_ERROR_MESSAGE_HANDLER)
public MessageHandler smbMessageHandler() {
FileTransferringMessageHandler<SmbFile> handler =
new FileTransferringMessageHandler<>(smbSessionFactory);
handler.setRemoteDirectoryExpression(
new LiteralExpression("/INPUT/ERROR"));
handler.setFileNameGenerator(m ->
m.getHeaders().get(FileHeaders.FILENAME, String.class) + "." + DateTimeFormatter.ofPattern(dateFormat).format(LocalDateTime.now()));
handler.setAutoCreateDirectory(true);
return handler;
}
You just mark that smbMessageHandler bean with the #ServiceActivator(inputChannel = "smbStoreChannel") and this ExpressionEvaluatingTransactionSynchronizationProcessor may just have a setAfterRollbackChannel(smbStoreChannel). So, when rollback happens a failed message is going to be sent to that channel where your FileTransferringMessageHandler will consume it from the channel and probably send to SMB. Consider also to use a TransactionSynchronizationFactoryBean for convenience.

Using http.GetFile how to prevent bad url requests from creating new files

I am using the http.getFile function to download files from an api. I am having a issue where files are still created, even though the url passed to getFile is invalid or returning errors. After some research it appears the getFile will always create a new file, is there a way to prevent getFile from creating a new file if the url is invalid?
The only work around I can think is to check the file size after calling the getFile and deleting it if there is no data?
In the example below I was tying to use the File.exists, but it always returns true.
return http.getFile(fullUrl, filePath)
.then(function(r){
// Test - Check if file Exists
console.log("Check File Exist: " + fs.File.exists(filePath));
}, function(error) {
});
Just check if the "fullUrl" is a valid url before requesting:
var regexp = /(ftp|http|https):\/\/(\w+:{0,1}\w*#)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%#!\-\/]))?/
var isUrlValid = regexp.test(fullUrl);
if(isUrlValid){
http.getFile(fullUrl, filePath)
}

Spring xd stream web service responses

I am trying to use spring xd to ingest text/xml responses from a web service using HTTP/1.1 protocol. The goal is to convert the xml response to json and insert into mongodb. But right now I am unable to get any responses from the stream. I want to do this programatically rather than in the shell, below is my code
public static void main(String[] args) {
SpringXDTemplate template = null;
try {
template = new SpringXDTemplate(new URI("http://localhost:9393"));
} catch (URISyntaxException e) {
e.printStackTrace();
}
String name = "test";
String definition = "time --fixedDelay=5 | http-client --url='''http://www.ctabustracker.com/bustime/api/v2/getvehicles?key=key&vid=1''' | file";
template.streamOperations().destroy(name);
template.streamOperations().createStream(name, definition, true);
}
I'm expecting to find the responses written to C:\tmp\xd\output\test.out but no file was created. I see that the stream was created in the admin ui, I'm not seeing any exceptions. How do I obtain the responses from sending requests to this url?
Turn on DEBUG logging; you should see the time source emit a message every 5 seconds; follow the messages through the flow to figure out what's going wrong.

Logic hook displaying some information from parent record raises error

I've got an hosted instance of SugarCRM 6.5 CE, and one of the requirements I have to fulfil is to display some information--contact phone number, contact email address--of the parent record in an associated task/activity record.
All I found so far was pointing towards the creation of a logic hook for pulling the contact information from the parent record (Contacts) and display these in custom fields in the child record (Tasks).
Following some instructions and examples found I came up with the following as outlined below.
Under "custom/modules/Tasks" I've create a file called "logic_hooks.php"
<?php// $Id$
$hook_version = 1;
$hook_array = Array();
// debug
$GLOBALS['log'] = LoggerManager::getLogger('SugarCRM');
$GLOBALS['log']->debug("Task: logic hook invoked");
// position, file, function
$hook_array['after_retrieve'] = Array();
$hook_array['after_retrieve'][] = Array('1', 'contact_info', 'custom/modules/Tasks/hooks/contact_info.php','contact_info_class', 'contact_info_method');
?>
and under "custom/modules/Tasks/hooks" I've create a file called "contact_info.phplogic_hooks.php"
<?php
class contact_info_class {
// retrieve contact information from parent record
function contact_info_method($bean, $event, $arguments) {
// debug
$GLOBALS['log'] = LoggerManager::getLogger('SugarCRM');
$GLOBALS['log']->debug("Tasks: contact_info_method called for event ".$event . "(BeanID: " . $bean->id . ")");
// fetch data
if ($bean->fetched_row['id'] != $bean->id) {
// load Task
//$bean = BeanFactory::getBean('Tasks', $id);
// check if relationship is loaded
//if ($bean->load_relationship('contact_tasks_parent'))
if ($bean->load_relationship('contact_tasks')) {
// fetch related beans
//$relatedBeans = $bean->contact_tasks_parent->getBeans();
$relatedBeans = $bean->contact_tasks->getBeans();
$parentBean = false;
if (!empty($relatedBeans)) {
// order the results
reset($relatedBeans);
// first record in the list is the parent
$parentBean = current($relatedBeans);
// retrieve data from parent bean
$bean->contact_phone_c = $parentBean->phone_work
$bean->contact_primary_email_c = $parentBean->email1
}
}
}
} // contact_info_method
} // contact_info_class
?>
With this hook in place I can create new tasks without any problem at all, but when opening up an existing one, I'm receiving a message, reading
There was an error processing your request, please try again at a later time.
Being completely new to SugarCRM (btw. 6.5.20 CE it is I'm dealing with), I've got not the faintest idea as what is going wrong here.
I also cannot find any of the debug messages which are supposed to be written somewhere to.
--Sil68
The "contact_info.phplogic_hooks.php" file should be in the same folder as logic_hooks.php (custom/modules/< module-name>). And there's no need to name it that way (in fact I think it might cause problems). Try naming it just contact_info.php and changing the path given in the logic_hooks.php file to custom/modules/Tasks/contact_info.php.
As for where you can find the error log, assuming you're using apache for your web server (since you didn't specify) for linux/OS X, the error log is located at
/var/log/apache2/error.log
or
/var/log/apache2/error_log
In windows it'll be in
'C:\Program Files\Apache Software Foundation\Apache2.2\logs'.
Now that you know where the error log is, you can put
error_log('some helpful message');
inside your contact_info.php file and see which messages (if any) get sent to the error log. This can tell you if it even starts the logic hook and if so, how far it gets through the logic hook

Resources