Hey StackOverflow Community,
I have some problems with my Nifi flow. I made one to take my data from my Azure blob to put them into my HDFSCluster ( still in Azure).
My configuration in the item PutHDFS in Nifi is :
PutHDFSConfiguration
But when I inform the field "Hadoop ressources", i have this following error:
PutHDFS[id=89381b69-015d-1000-deb7-50b6cf485d28] org.apache.hadoop.fs.adl.HdiAdlFileSystem: java.lang.ClassNotFoundException: org.apache.hadoop.fs.adl.HdiAdlFileSystem
PutHDFS[id=89381b69-015d-1000-deb7-50b6cf485d28] PutHDFS[id=89381b69-015d-1000-deb7-50b6cf485d28] failed to invoke #OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
How can i resolve this and put my data into my clusters.
Thanks for your answer.
Apache NiFi does not bundle any of the Azure related libraries, it only bundles the standard Apache Hadoop client, currently 2.7.3 if using a recent NiFi release.
You can specify the location of the additional Azure JARs through the PutHDFS processor property called "Additional Classpath Resources".
Related
I am following this article
https://sematext.com/blog/2017/03/06/kafka-connect-elasticsearch-how-to/
I am getting error as
Caused by: org.apache.kafka.connect.errors.ConnectException:
Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector [...]
As Sönke says in the comments, this is a classpath problem. You can side-step this by just using the Confluent Open Source distribution which includes the Kafka Connect Elasticsearch plugin automatically with paths already in place.
I have a Topology that reads data from kafka using kafka-spout and pushes the processed data to elasticsearch. I have used, elasticsearch-2.3.3 client. And it requires guava-18.0 jar dependency. I have formed a jar-with-dependencies using maven. After submitting the topology to storm, i got below error
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; at org.elasticsearch.threadpool.ThreadPool.(ThreadPool.java:190)
After searching about the above error, got to know that this is due to mismatch of guava jar. I checked the storm server, where i found guava-15 in its /lib folder. Since other topologies are using guava-15.jar, i cannot upgrade it directly to guava-18.
Is there a way, where I can resolve this dependencies, by specifying guava-18 path exclusively for my topology while submitting.
I have been working with different versions of both hadoop and oozie to get them working together but haven't suceeded. Recently i considered the two versions(hadoop 1.2.0 and oozie 3.3.2) which are put together under hortonworks 1.3 data platform so i figured they will work together. I configured and run the hadoop on a single node in pseudo distributed mode.
Although i was able to build the oozie successfully including the jar files from hadoop, while i try to run it i'm faced with the following error on http://hostname:11000/oozie. Any help is greatly appreciated, it's so frustrating to haven't figured it out for weeks.
HTTP Status 500 - Filter execution threw an exception
type Exception report
message Filter execution threw an exception
description The server encountered an internal error that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Filter execution threw an exception
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84)
root cause
java.lang.NoSuchMethodError: org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:124)
org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:160)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:349)
org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84)
note The full stack trace of the root cause is available in the Apache Tomcat/6.0.36 logs. Apache Tomcat/6.0.36
I'm trying to perform mapreduce in hadoop-2.2.0 for the data stored in cassandra using pig. I was able run the script in pig local mode but I couldn't run in mapreduce mode.Kindly help me to resolve it.Please find the stack trace for more info.
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
ERROR pigstats.SimplePigStats: ERROR: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1390194645126_0008' doesn't exist in RM.
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103)
The Hadoop-2 (YARN) interfaces aren't supported in the Cassandra source tree yet (see issue CASSANDRA-5201). Based on the activity in that issue, it looks like this will be resolved relatively shortly. There has been some work done out of tree to support YARN that might work for you in the meantime. Or you could use Hadoop-1.x.
My goal is to run an elastic map reduce job that queries a Solr index in the map phase and writes the result to S3. Solr and Hadoop worked fine together when building a Solr index within a Hadoop job (ie writing to Solr index). When I run the a job to query a Solr index I get an error when trying to initiate the Solr client. I suspect that there's a dependency issue between Hadoop and Solr, I recall they both use different versions of http clients and the error is a method not found issue. Here's the stack trace
2013-07-24 03:17:47,082 FATAL org.apache.hadoop.mapred.Child (main): Error running child : java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry;
at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445)
at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179)
at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33)
at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115)
at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105)
at org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:154)
at org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:127)
Adding this opt did the trick
--args -s,mapreduce.user.classpath.first=true
Using the user defined classpath worked out the dependency issue between the hadoop and solr jar