Failover when reading from FTP sites using Camel - ftp

We need to download multiple files on a hourly basis from a vendors FTP site. The vendor provides two ftp sites for fault tolerance with both sites having identical files. I would like to setup a Camel route to download the files from ftp site A if its available and if not try ftp site B. The following code is incorrect but it may highlight what I am trying to achieve.
from("timer://timer1?fixedRate=true&period=60m")
.loadBalance()
.failover(-1, false, true)
.to("direct:ftp-symbolguides-1")
.to("direct:ftp-symbolguides-2")
.end();
from("direct:ftp-symbolguides-1")
.to("ftp://SiteA?localWorkDirectory=c:/temp&passiveMode=true&noop=true&idempotentKey=${file:name}-${file:size}&idempotentRepository=#idempotentRepository")
.to("file:/c:/temp/inbox");
from("direct:ftp-symbolguides-2")
.to("ftp://SiteB?password=publicftp&localWorkDirectory=c:/temp&passiveMode=true&noop=true&idempotentKey=${file:name}-${file:size}&idempotentRepository=#idempotentRepository")
.to("file:/c:/temp/inbox");
Does anyone have any thoughts on how I can achieve this?

When the connection fails by default ftp consumer produces a WARN message instead of throwing an exception, but by the throwExceptionOnConnectionFailed property you can change to throw exception, and handle exception from the
PollingConsumerPollStrategy[1] rollback() method.
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/PollingConsumerPollStrategy.html

Related

Nifi Error: Failed to establish the connection with AMQP broker

I am trying to read the data from following cap files.
everyhing from alerts folder
http://dd2.weather.gc.ca/alerts/cap/20180205/CWHX/14/
I am using AMQP from http://metpx.sourceforge.net. And when I am trying to connect to subscriber from nifi, I am getting the following error.
Failed to establish the connection with AMQP broker
this is my cap.conf file.
broker amqp://anonymous:anonymous#dd.weather.gc.ca
directory /data
subtopic alerts.cap.#
accept .*
mirror True
Over the summer, the broker migrated to SSL, so the current URL would be: amqps://anonymous:anonymous#dd.weather.gc.ca
The web page has also moved to: https://github.com/MetPX/sarracenia
best practice to put authentication info in ~/.config/sarra/credentials.conf
a line like: amqps://anonymous:anonymous#dd.weather.gc.ca
Installing a version from the past year is likely to be a much
better experience. It now comes with sample configurations,
one of them being ddc_cap-xml.conf which is the same data you
are trying to download.
So the work is:
blacklab% sr_subscribe add ddc_cap-xml.conf
blacklab% sr_subscribe edit ddc_cap-xml.conf
# Change the directory option to suit your case.
blacklab% sr_subscribe foreground ddc_cap-xml.conf
And it should work. It could take many hours to prove it because this particular set (severe weather warnings in Common Alerting Protocol format) is produced only when needed, rather than continuously. (Use start instead of foreground to run as a background daemon.)
To test things, it might be easier to start with dd_swob, which will be a continuous feed.
blacklab% sr_subscribe list dd_swob
broker amqp://anonymous#dd.weather.gc.ca
exchange xpublic
#msg_skip_threshold 60
#on_msg ../msg_skip_old.py
subtopic observations.swob-ml.#
accept .*
In this configuration you need to add a directory option just before the accept line. and should start downloading data immediately. Once you know it works, switch back to the data set you actually want.

Not able to publish a message to ActiveMQ using JMeter

I would like publish some messages and perform load testing of my docker-based ActiveMQ server using JMeter. I have the configuration in place but for some reason I am not able to see the message getting published in ActiveMQ.
I have the admin console up and running and see that the queues is not getting created either. I also tried to manually create the queue and see if the Jmeter script works, but no luck
I have already added activemq-all-5.15.0 jar in my jmeter/lib/ext folder of JMeter 3.2.
I am also seeing following error in JMeter Results Tree.
javax.naming.NameNotFoundException: jmeter-q
at org.apache.activemq.jndi.ReadOnlyContext.lookup(ReadOnlyContext.java:235)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at org.apache.jmeter.protocol.jms.Utils.lookupDestination(Utils.java:148)
at org.apache.jmeter.protocol.jms.client.Publisher.<init>(Publisher.java:134)
at org.apache.jmeter.protocol.jms.sampler.PublisherSampler.initClient(PublisherSampler.java:181)
at org.apache.jmeter.protocol.jms.sampler.PublisherSampler.sample(PublisherSampler.java:206)
at org.apache.jmeter.protocol.jms.sampler.BaseJMSSampler.sample(BaseJMSSampler.java:98)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:491)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:425)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:254)
at java.lang.Thread.run(Thread.java:748)
The correct format to provide destination under Destination textbox is
dynamicQueues/jmeter-q
Also please make sure that there should be no spaces at the end of each fields like provideURl, connectionf factory etc.

OSB File Read/Write Proxy Services

I'm doing a Proxy Service with Messaging transportation for Any XML and defining the stage, archive and error folders for a xml processing when configuring the proxy service.
I had included an error handler and when reading the file and an error happens, the file must be moved for error folder.
Currently when doing my test and an error happens, the file is moved to archive instead of error folder.
Are there something that I need to add and get this working?
Basically the complete scenario is a Proxy Service polling a folder, reading the XML and sending to Business Services as a XML messaging.
Could you give me directions to make this working or tips that I need to check?
Thanks
Edson
A wild guess - Is it because you have an error handler and you may be doing a success reply from the handler. Have you tested removing the error handler or reply-with-failure from the handler ?

Mule Connect to remote flat files

I am new to Mule and I have been struggling with a simple issue for a while now. I am trying to connect to flat files (.MDB, .DBF) located on a remote desktop through my Mule application using the generic database connector of Mule. I have tried different things here:
I am using StelsDBF and StelsMDB drivers for the JDBC connectivity. I tried connecting directly using jdbc URL - jdbc:jstels:mdb:host/path
I have also tried to access through FTP by using FileZilla server on remote desktop and using jdbc URL in my app - jdbc:jstels:dbf:ftp://user:password#host:21/path
None of these seem to be working as I am always getting Connection exceptions. If anyone has tried this before, what is the best way to go about it? Connecting a remote flat file with Mule? Your response on this will be greatly appreciated!
If you want to load the contents of the file inside a Mule flow you should use the file or FTP connector, i don't know for sure about your JDBC option.
With the File connector you can access local files (files on the server where mule is running), you could try to mount the folders as a share.
Or run an FTP server like you already tried, that should work.
There is probably an error in your syntax / connection.
Please paste the complete XML of your Mule flow so we can see what you are trying to do.
Your usecase is still not really clear to me, are you really planning to use http to trigger the DB everytime? Anyway did you try putting the file on a local path and use that path in your database url. Here is someone that says he had it working, he created a separate bean.
http://forums.mulesoft.com/questions/6422/setting_property_dynamically_on_jdbcdatasource.html
I think a local path is maybe possible and it's better to test that first.
Also take note of how to refer to a file path, look at the examples for the file connector: https://docs.mulesoft.com/mule-user-guide/v/3.7/file-transport-reference#namespace-and-syntax
If you manage to get it working and you can use the path directly in the JDBC url, you should have a look at the poll scope.
https://docs.mulesoft.com/mule-user-guide/v/3.7/poll-reference
You can use your DB connector as an inbound endpoint when wrapped in a poll scope.
I experienced the same issue when connect to Microsoft Access Database (*.mdb, *.accdb) using Mule Database Connector. After further investigation, it's solved by installing Microsoft Access Database Engine
Another issue, I couldn't pass parameter to construct a query as same as I do for other databases. e.g.: SELECT * FROM emplcopy WHERE id = #[payload.id]
To solve this issue:
I changed the Query type from Parameterized into Dynamic.
I generated the query inside Set Payload transformer (generate the query in form of String, e.g.: SELECT * FROM emplcopy WHERE id = '1').
Finally, put it into the Dynamic query area: #[payload]

how to delete file from the server after downloading it ?or how can i store the file to client machine directly from output stream?

am using liferay custom portlet and in that am using jasper report now my problem is that how can i download the pdf report directly on the client machine
right now am storing the file at server first.then provide url for downloading the pdf to user.but how can i directly store the file to client machine if i have pdf file's outputstream .
ot if i can know some how when user click on the download link and after downloading the file if i want to delete the donlowded file from the server then how can i do it.?if any one can guide me...
I'm not sure what you're asking for is possible, but I would be interested in seeing someone correct that statement though.
Servers really shouldn't be directly storing files on a client machine as that violates the intent of the client server relationship. A client has to make a request for the file and then the client can save that file (eg like a ftp download). Servers just don't manipulate client machines as they see fit.
As far as knowing when a file is downloaded, there isn't anything in a portlet you can do to detect that. You can use ResourceRequest and serveResource method to serve a file, but nothing in the portlet API will inform your portlet that the download is complete or that it wasn't interrupted by something.
As an alternative you might try simply having a cron job that will clean out old files. In this case, make sure to inform users how long they have to successfully download the file.

Resources