Camel http4 download file using Basic authentication over Https - https

Question

I am trying to download a file from a Https url which requires Basic authentication. I am using HTTP4
I am trying to download from url - https://ebc.cybersource.com/ebc/DownloadReport/xxx.csv?authMethod=Basic&authUsername=scott&authPassword=tiger
After the file is downloaded I need to save it to a folder. Here's what my code looks like
from(xxx)
.to("http4://ebc.cybersource.com/ebc/DownloadReport/xxx.csv?authMethod=Basic&authUsername=scott&authPassword=tiger")
.to("file:target/messages/download");
Here's the error I get -
java.lang.UnsupportedOperationException: Cannot consume from http endpoint
at org.apache.camel.component.http4.HttpEndpoint.createConsumer(HttpEndpoint.java:120)
at org.apache.camel.impl.EventDrivenConsumerRoute.addServices(EventDrivenConsumerRoute.java:65)
at org.apache.camel.impl.DefaultRoute.onStartingServices(DefaultRoute.java:85)
at org.apache.camel.impl.RouteService.warmUp(RouteService.java:158)
at org.apache.camel.impl.DefaultCamelContext.doWarmUpRoutes(DefaultCamelContext.java:3090)
at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:3020)
at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:2797)
at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:2653)
at org.apache.camel.impl.DefaultCamelContext.access$000(DefaultCamelContext.java:167)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2467)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2463)
at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:2486)
at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:2463)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:2432)
at com.dinesh.MainApp.main(MainApp.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Now the documentation says I need to use HttpContext,
public class HttpContextFactory {
private String httpHost = "localhost";
private String httpPort = 9001;
private BasicHttpContext httpContext = new BasicHttpContext();
private BasicAuthCache authCache = new BasicAuthCache();
private BasicScheme basicAuth = new BasicScheme();
public HttpContext getObject() {
authCache.put(new HttpHost(httpHost, httpPort), basicAuth);
httpContext.setAttribute(ClientContext.AUTH_CACHE, authCache);
return httpContext;
}
// getter and setter
}
I tried that but I am still getting errors.
UPDATE
I am actually using a quartz scheduler to trigger
from("quartz2://foo?cron=0/2+*+*+?+*+MON-FRI").multicast().stopOnException().to("direct:downLoadFile");
from("direct:downLoadFile")
.to("http4://ebc.cybersource.com/ebc/DownloadReport/2015/04/16/us_voip/us_voip.au.response.ss.csv?authMethod=Basic&authUserName=xxxx&authPassword=yyyy")
.log("File Downloaded");
When I run this I get the below error.
[ main] QuartzComponent INFO Starting scheduler.
[ main] QuartzScheduler INFO Scheduler DefaultQuartzScheduler-camel-1_$_NON_CLUSTERED started.
[ main] DefaultCamelContext INFO Total 2 routes, of which 2 is started.
[ main] DefaultCamelContext INFO Apache Camel 2.15.1 (CamelContext: camel-1) started in 0.938 seconds
[artzScheduler-camel-1_Worker-1] RetryExec INFO I/O exception (java.net.SocketException) caught when processing request to {}->http://ebc.cybersource.com:80: Connection reset
[artzScheduler-camel-1_Worker-1] RetryExec INFO Retrying request to {}->http://ebc.cybersource.com:80
[artzScheduler-camel-1_Worker-1] RetryExec INFO I/O exception (java.net.SocketException) caught when processing request to {}->http://ebc.cybersource.com:80: Connection reset
[artzScheduler-camel-1_Worker-1] RetryExec INFO Retrying request to {}->http://ebc.cybersource.com:80
[artzScheduler-camel-1_Worker-1] RetryExec INFO I/O exception (java.net.SocketException) caught when processing request to {}->http://ebc.cybersource.com:80: Connection reset
[artzScheduler-camel-1_Worker-1] RetryExec INFO Retrying request to {}->http://ebc.cybersource.com:80
[artzScheduler-camel-1_Worker-1] DefaultErrorHandler ERROR Failed delivery for (MessageId: ID-CN-AHONNAVA-LP-49362-1430485742705-0-1 on ExchangeId: ID-CN-AHONNAVA-LP-49362-1430485742705-0-3). Exhausted after delivery attempt: 1 caught: java.net.SocketException: Connection reset
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [quartz2://foo?cron=0%2F2+*+*+%3F+*+MON-FRI ] [ 523]
[route1 ] [multicast1 ] [multicast ] [ 520]
[route1 ] [to1 ] [direct:downLoadFile ] [ 513]
[route2 ] [to2 ] [http4://ebc.cybersource.com/ebc:443/DownloadReport/2015/04/16/us_voip/us_voip.] [ 510]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id ID-CN-AHONNAVA-LP-49362-1430485742705-0-3
ExchangePattern InOnly
Headers {breadcrumbId=ID-CN-AHONNAVA-LP-49362-1430485742705-0-1, calendar=null, CamelRedelivered=false, CamelRedeliveryCounter=0, fireTime=Fri May 01 09:09:04 EDT 2015, jobDetail=JobDetail 'Camel_camel-1.foo': jobClass: 'org.apache.camel.component.quartz2.CamelJob concurrentExectionDisallowed: false persistJobDataAfterExecution: false isDurable: false requestsRecovers: false, jobInstance=org.apache.camel.component.quartz2.CamelJob#6e61a2e3, jobRunTime=-1, mergedJobDataMap=org.quartz.JobDataMap#2a207ff6, nextFireTime=Fri May 01 09:09:06 EDT 2015, previousFireTime=null, refireCount=0, result=null, scheduledFireTime=Fri May 01 09:09:04 EDT 2015, scheduler=org.quartz.impl.StdScheduler#1e79d438, trigger=Trigger 'Camel_camel-1.foo': triggerClass: 'org.quartz.impl.triggers.CronTriggerImpl calendar: 'null' misfireInstruction: 1 nextFireTime: Fri May 01 09:09:06 EDT 2015, triggerGroup=Camel_camel-1, triggerName=foo}
BodyType null
Body [Body is null]
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.apache.camel.component.http4.HttpProducer.executeMethod(HttpProducer.java:258)
at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:158)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:590)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:518)
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:227)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[artzScheduler-camel-1_Worker-1] CamelJob ERROR Error processing exchange. Exchange[Message: [Body is null]]. Caused by: [org.quartz.JobExecutionException - org.apache.camel.CamelExchangeException: Sequential processing failed for number 0. Exchange[Message: [Body is null]]. Caused by: [java.net.SocketException - Connection reset]]
[artzScheduler-camel-1_Worker-1] JobRunShell INFO Job Camel_camel-1.foo threw a JobExecutionException:
org.quartz.JobExecutionException: org.apache.camel.CamelExchangeException: Sequential processing failed for number 0. Exchange[Message: [Body is null]]. Caused by: [java.net.SocketException - Connection reset] [See nested exception: org.apache.camel.CamelExchangeException: Sequential processing failed for number 0. Exchange[Message: [Body is null]]. Caused by: [java.net.SocketException - Connection reset]]
at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
FINAL UPDATE: SOLVED
Thanks to #Claus Ibsen who provided the correct component. All I had to was use use https4 instead of http. Note this information is missing in camel documentation. Here's the final code, that download the file from the website and saves in in target/download folder.
import org.apache.camel.builder.RouteBuilder;
import org.apache.commons.codec.binary.Base64;
/**
* Created by darora on 4/30/2015.
*/
public class SchedulerRoute extends RouteBuilder {
//TODO Move this to properties file
String userCredentials ="userName:Password";
String basicAuth = "Basic " + new String(new Base64().encode(userCredentials.getBytes()));
#Override
public void configure() throws Exception {
//from("timer://foo?fixedRate=true&period=6000").to("bean:logBean?method=sayHello");
//from("scheduler://foo?delay=500").to("bean:logBean?method=sayHello");
from("quartz2://foo?cron=0/2+*+*+?+*+MON-FRI").multicast().stopOnException().to("direct:downLoadFile","direct:myBean");
from("direct:myBean").log("Scheduler Started one more time .....")
.to("bean:downLoadFileBean?method=updatedatabase");
from("direct:downLoadFile").log("Scheduler Started one more time .....")
.setHeader("Authorization",constant(basicAuth))
.to("https4://ebc.cybersource.com/ebc/DownloadReport/2015/04/16/us_voip/us_voip.au.response.ss.csv")
.to("file:target/download")
.log("file is downloaded ..........");
}
}

Answer 1

You cannot start from http4 as the exception tells you, you need a timer or something to trigger the route
from("timer:foo?period=5000")
.to("http4://ebc.cybersource.com/ebc/DownloadReport/xxx.csv?authMethod=Basic&authUsername=scott&authPassword=tiger")
.to("file:target/messages/download");
In this example the timer triggers every 5th second.