Why is my Oozie installation failing? - hadoop

I'm trying to install Apache Oozie on my local machine (Mac OS High Sierra 10.13.3) but receive a "java.net.ConnectException Connection refused" error when checking the status of the installation.
$ oozied.sh start
Setting OOZIE_HOME: /Users/akirna/oozie
Setting OOZIE_CONFIG: /Users/akirna/oozie/conf
Sourcing: /Users/akirna/oozie/conf/oozie-env.sh
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE: oozie-site.xml
Setting OOZIE_DATA: /Users/akirna/oozie/data
Setting OOZIE_LOG: /Users/akirna/oozie/logs
Setting OOZIE_LOG4J_FILE: oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD: 10
Setting OOZIE_HTTP_HOSTNAME: LAMU02T76ZCGTFM.uhc.com
Setting OOZIE_HTTP_PORT: 11000
Setting OOZIE_ADMIN_PORT: 11001
Setting OOZIE_HTTPS_PORT: 11443
Setting OOZIE_BASE_URL: http://LAMU02T76ZCGTFM.uhc.com:11000/oozie
Setting CATALINA_BASE: /Users/akirna/oozie/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE: /Users/akirna/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS: password
Setting OOZIE_INSTANCE_ID: LAMU02T76ZCGTFM.uhc.com
Setting CATALINA_OUT: /Users/akirna/oozie/logs/catalina.out
Setting CATALINA_PID: /Users/akirna/oozie/oozie-server/temp/oozie.pid
Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/Users/akirna/oozie/logs/derby.log
Adding to CATALINA_OPTS: -Doozie.home.dir=/Users/akirna/oozie -Doozie.config.dir=/Users/akirna/oozie/conf -Doozie.log.dir=/Users/akirna/oozie/logs -Doozie.data.dir=/Users/akirna/oozie/data -Doozie.instance.id=LAMU02T76ZCGTFM.uhc.com -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=LAMU02T76ZCGTFM.uhc.com -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://LAMU02T76ZCGTFM.uhc.com:11000/oozie -Doozie.https.keystore.file=/Users/akirna/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
Setting up oozie DB
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
DONE
DB schema exists
The SQL commands have been written to: /var/folders/bx/5shvx3ln5wx4lp7c7j_l_f65gmq63g/T/ooziedb-4834223640212041380.sql
Using CATALINA_BASE: /Users/akirna/oozie/oozie-server
Using CATALINA_HOME: /Users/akirna/oozie/oozie-server
Using CATALINA_TMPDIR: /Users/akirna/oozie/oozie-server/temp
Using JRE_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
Using CLASSPATH: /Users/akirna/oozie/oozie-server/bin/bootstrap.jar
Using CATALINA_PID: /Users/akirna/oozie/oozie-server/temp/oozie.pid
Existing PID file found during start.
Removing/clearing stale PID file.
$ oozie admin -oozie http://localhost:11000/oozie -status output
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 1 sec. Retry count = 1
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 2 sec. Retry count = 2
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 4 sec. Retry count = 3
...
oozie/logs/oozie.log output:
*******************************************************************************
STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [akirna] on [2018.04.18-19:11:20GMT]
STARTUP MSG: revision [unavailable]#[unavailable]
*******************************************************************************
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j configuration file [oozie-log4j.properties]
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j configuration file loaded from [/Users/akirna/oozie/conf]
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j reload interval [10 sec]
2018-04-18 22:56:52,084 INFO ConfigurationService:520 - SERVER[] Oozie home dir [/Users/akirna/oozie]
2018-04-18 22:56:52,085 INFO ConfigurationService:520 - SERVER[] Oozie conf dir [/Users/akirna/oozie/conf]
2018-04-18 22:56:52,085 INFO ConfigurationService:520 - SERVER[] Oozie conf file [oozie-site.xml]
2018-04-18 22:56:52,250 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.akirna.hosts]
2018-04-18 22:56:52,251 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.akirna.groups]
2018-04-18 22:56:52,254 WARN Services:523 - SERVER[] System ID [oozie-akir] exceeds maximum length [10], trimming
2018-04-18 22:56:52,255 INFO Services:520 - SERVER[] Exiting null Entering NORMAL
2018-04-18 22:56:52,256 INFO Services:520 - SERVER[] Initialized runtime directory [/var/folders/bx/5shvx3ln5wx4lp7c7j_l_f65gmq63g/T/oozie-akir1419795458214876479.dir]
2018-04-18 22:56:52,256 INFO ConfigurationService:520 - SERVER[] Programmatic configuration change, property[oozie.services]=[org.apache.oozie.service.LiteWorkflowAppService, org.apache.oozie.service.HadoopAccessorService]
2018-04-18 22:56:52,258 INFO ConfigurationService:520 - SERVER[] Programmatic configuration change, property[oozie.services.ext]=[]
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] JOB_TRACKER_WHITELIST :[], Total entries :0
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] NAME_NODE_WHITELIST :[], Total entries :0
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] Oozie Kerberos Authentication [disabled]
2018-04-18 22:56:52,458 WARN NativeCodeLoader:60 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-18 22:56:52,556 INFO Services:520 - SERVER[] Initialized
2018-04-18 22:56:52,562 INFO Services:520 - SERVER[] Running with JARs for Hadoop version [3.0.0]
2018-04-18 22:56:52,563 INFO Services:520 - SERVER[] Oozie System ID [oozie-akir] started!
oozie/logs/catalina.out output:
ERROR: Oozie could not be started
REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/permission/AccessControlException
Stacktrace:
-----------------------------------------------------------------
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/permission/AccessControlException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2416)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2381)
at org.apache.oozie.util.XConfiguration.getClassByName(XConfiguration.java:220)
at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2454)
at org.apache.oozie.service.ConfigurationService$LogChangesConfiguration.getClasses(ConfigurationService.java:411)
at org.apache.oozie.service.ConfigurationService.getClasses(ConfigurationService.java:584)
at org.apache.oozie.service.Services.loadServices(Services.java:288)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:761)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.permission.AccessControlException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
... 35 more
-----------------------------------------------------------------

Related

geomesa add-attribute-index fails

I am trying to add an index to my existing table using the following command (run inside accumulo-master docker image)
geomesa add-attribute-index -u root -p secret -i gis -z SERVER_IP -c posiciones -f posicion -a id_posicion --coverage join
But it does not work and produce this output:
INFO Running map reduce index job for attributes: [id_posicion] with coverage: join...
ERROR Error encountered running attribute index command. Check hadoop's job history logs for more information.
The hadoop job log is the following:
2017-09-17 20:39:48,253 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1505353025896_0020_000002
2017-09-17 20:39:48,706 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-09-17 20:39:48,757 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-09-17 20:39:49,079 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 20 cluster_timestamp: 1505353025896 } attemptId: 2 } keyId: -1893920016)
2017-09-17 20:39:49,094 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2017-09-17 20:39:49,095 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.mapred.DirectFileOutputCommitter
2017-09-17 20:39:49,173 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat not found
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:519)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:499)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1594)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:499)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:284)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1552)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1549)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1482)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:223)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:516)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 13 more
Any idea?
This is probably a bug - the jars to load are defined in this file. Likely the file needs to be updated for newer versions of accumulo - the missing class now appears to be in the accumulo-core jar. You should be able to fix it by adding the line accumulo-core to that file, which ends up in lib/geomesa-accumulo-jobs-<version>.jar in the tools distribution.
Is $ACCUMULO_HOME set? And are other geomesa commands working?
Setting $ACCUMULO_HOME to point to a copy of the Accumulo distribution would likely help. If you are using the GeoMesa tools from a machine which is not part of the cluster, then you can use the install-hadoop-accumulo.sh script in the tools distribution to download a copy of the necessary dependencies to $GEOMESA_HOME/lib.

How to setup JanusGraph using Docker for Cassandra and Elasticsearch?

I'm trying to setup JanusGraph for development on my local machine. My goal is to have a setup similar to the Cassandra remote server mode. As storage backend, I want to use Cassandra and as index backend I planned to use Elasticsearch.
For both, I'm using Docker containers (Cassandra, Elasticsearch).
My janusgraph-server.properties file looks like this:
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandra
storage.hostname=127.0.0.1
storage.cassandra.astyanax.cluster-name=cassandra_test_cluster
index.search.backend=elasticsearch
index.search.hostname=127.0.0.1
index.search.port=9300
index.search.elasticsearch.cluster-name=elasticsearch_test_cluster
Starting the gremlin-server leads to this failures:
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
162 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
256 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics ConsoleReporter configured with report interval=180000ms
263 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics CsvReporter configured with report interval=180000ms to fileName=/tmp/gremlin-server-metrics.csv
343 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics JmxReporter configured with domain= and agentId=
345 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
800 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
807 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
884 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
884 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1070 [main] INFO org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=c0a8000424833-XXX-MacBook-Pro-local1
1078 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
1079 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1099 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
1179 [main] INFO org.elasticsearch.plugins - [General Orwell Taylor] loaded [], sites []
1655 [main] INFO org.janusgraph.diskstorage.es.ElasticSearchIndex - Configured remote host: 127.0.0.1 : 9300
1738 [elasticsearch[General Orwell Taylor][generic][T#2]] INFO org.elasticsearch.client.transport - [General Orwell Taylor] failed to get local cluster state for [#transport#-1][XXX-MacBook-Pro.local][inet[/127.0.0.1:9300]], disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][inet[/127.0.0.1:9300]][cluster:monitor/state] disconnected
1743 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [conf/gremlin-server/janusgraph-server.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.<init>(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:107)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:87)
... 13 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 20 more
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:279)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:198)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:86)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:127)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:92)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:215)
... 25 more
1745 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
2190 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded gremlin-groovy ScriptEngine
2836 [main] WARN org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/empty-sample.groovy as script could not be evaluated - javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
None of the configured nodes are available: [] why?
What can I do to make them available?
Have you verified whether Elasticsearch and Cassandra are running on those ports on localhost? If not, I would recommend checking that you're forwarding to those ports when starting your containers.
I would also recommend checking the logs for Cassandra and Elasticsearch and seeing if there is any errors in those.

Oozie launching MR jobs as YARN user instead of given user name

I have designed a Oozie workflow to run a Sqoop script.
I'm submitting the workflow using the user name given by hadoop admin team.
Script is failing because oozie is launching MR jobs as YARN user which is not able to access my userid directory in HDFS i.e /user/cv1100.
I checked the MR log there I can see the property as "user.name=yarn"
How can I change this? I have mentioned "user.name" in job.properties file of Oozie.
Below is the error I’m getting in logs, check if it helps
log4j:ERROR Could not find value for key log4j.appender.CLA
log4j:ERROR Could not instantiate appender named "CLA".
log4j:ERROR Could not find value for key log4j.appender.CLA
log4j:ERROR Could not instantiate appender named "CLA".
error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/aster-networking.jar; /usr/lib/hadoop/lib/aster-networking.jar (Permission denied)
Note: /tmp/sqoop-yarn/compile/a9988b6ea5448f4cc962b625361feb1a/test.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
31075 [Thread-35] INFO org.apache.sqoop.hive.HiveImport - Loading data to table ecsdashboard.test_oozie
31175 [Thread-35] INFO org.apache.sqoop.hive.HiveImport - Failed with exception Unable to move sourcehdfs:///user/qjdht93/test/_SUCCESS to destination hdfs://apps/hive/warehouse/ecsdashboard.db/test_oozie/_SUCCESS
31176 [Thread-35] INFO org.apache.sqoop.hive.HiveImport - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
31728 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Hive exited with status 1
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:425)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://Had1:8020/user/qjdht93/oozie-oozi/0000054-150404155202480-oozie-oozi-W/sqoop2hive--sqoop/action-data.seq
Oozie Launcher ends
Log Type: syslog
Log Length: 2914
2015-04-14 02:49:11,912 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-04-14 02:49:11,943 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started
2015-04-14 02:49:12,012 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-04-14 02:49:12,012 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-04-14 02:49:12,024 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-04-14 02:49:12,024 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1428177121154_1938, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier#45d1f40c)
2015-04-14 02:49:12,048 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 39.7.48.3:8050, Ident: (owner=qjdht93, renewer=oozie mr token, realUser=oozie, issueDate=1428994143490, maxDate=1429598943490, sequenceNumber=253, masterKeyId=11)
2015-04-14 02:49:12,100 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-04-14 02:49:12,383 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data2/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data3/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data4/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data5/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data6/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data7/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data8/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data9/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data10/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data11/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938,/data12/hadoop/yarn/local/usercache/qjdht93/appcache/application_1428177121154_1938
2015-04-14 02:49:12,821 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-04-14 02:49:13,273 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-04-14 02:49:13,596 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://Had1:8020/user/qjdht93/oozie-oozi/0000054-150404155202480-oozie-oozi-W/sqoop2hive--sqoop/input/dummy.txt:0+5
2015-04-14 02:49:13,622 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2015-04-14 02:49:13,659 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id

Error when crawling with Nutch - Input path does not exist: hdfs://.../urls/seed.txt

I have installed Apache Nutch and running a crawl using:
bin/crawl ./urls/seed.txt crawl http://localhost:8983/solr/ 5
Works fine from runtime/local. When I run that same command from runtime/deploy I get:
14/07/16 19:43:35 INFO crawl.InjectorJob: InjectorJob: starting at 2014-07-16 19:43:35
14/07/16 19:43:35 INFO crawl.InjectorJob: InjectorJob: Injecting urlDir: urls/seed.txt
14/07/16 19:43:37 INFO connection.CassandraHostRetryService: Downed Host Retry service started with queue size -1 and retry delay 10s
14/07/16 19:43:37 INFO service.JmxMonitor: Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
14/07/16 19:43:37 INFO crawl.InjectorJob: InjectorJob: Using class org.apache.gora.cassandra.store.CassandraStore as the Gora storage class.
14/07/16 19:43:37 INFO mapred.JobClient: Default number of map tasks: null
14/07/16 19:43:37 INFO mapred.JobClient: Setting default number of map tasks based on cluster size to : 12
14/07/16 19:43:37 INFO mapred.JobClient: Default number of reduce tasks: 0
14/07/16 19:43:38 INFO security.ShellBasedUnixGroupsMapping: add hadoop to shell userGroupsCache
14/07/16 19:43:38 INFO mapred.JobClient: Setting group to hadoop
14/07/16 19:43:39 INFO mapred.JobClient: Cleaning up the staging area hdfs://172.31.13.61:9000/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201407161337_0024
14/07/16 19:43:39 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://172.31.13.61:9000/user/hadoop/urls/seed.txt
14/07/16 19:43:39 ERROR crawl.InjectorJob: InjectorJob: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://172.31.13.61:9000/user/hadoop/urls/seed.txt
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1016)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1033)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:904)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1140)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:904)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:501)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:531)
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:50)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:233)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:251)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
It is not locating the file seed.txt and yes it does exist in $HOME/urls/seed.txt. I am using AWS EMR and Cassandra. Any help would be greatly appreciated.

Error on starting Pig

I configured Pig on my Hadoop system, but when I start it I get an error related to log4j. Am I missing something?
Thanks!
$ pig
log4j:ERROR Could not instantiate class [org.apache.hadoop.log.metrics.EventCounter].
java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
...
log4j:ERROR Could not instantiate appender named "EventCounter".
2014-02-14 10:45:46,512 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2014-02-14 10:45:46,513 [main] INFO org.apache.pig.Main - Logging error messages to: /usr/local/hadoop/pig_1392381946511.log
2014-02-14 10:45:46,541 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hduser/.pigbootup not found
2014-02-14 10:45:46,695 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: maprfs:///
2014-02-14 10:45:46,767 [main] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
2014-02-14 10:45:46,768 [main] INFO org.apache.hadoop.security.JniBasedUnixGroupsMapping - Using JniBasedUnixGroupsMapping for Group resolution
2014-02-14 10:45:46,853 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: maprfs:///
grunt>
First try running a pig script locally by
pig -x local <filename>.pig
If some error message shows up for running it locally also try
www.youtube.com/watch?v=BSsVvZnGz0M setup video,it is proper if you are using ubuntu 12.04LTS‎

Resources