I am started learning Hadoop. I followed this tutorial https://www.tutorialspoint.com/hadoop/hadoop_enviornment_setup.htm
When I am trying to execute the below command
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduceexamples-2.2.0.jar wordcount input output
getting this error
17/05/23 10:11:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
java.net.ConnectException: Call From LAPTOP-9BNLKT1E/192.168.1.11 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Help me where I am doing wrong.
Related
I have installed Hadoop on Windows according to this artile and now am able to run test application hadoop-mapreduce-examples-X.Y.Z.jar.
Unfortunately, when I am starting full-scale application, it starts to access some strange address 0.0.0.0:10020. Have changed my DFS config to <value>hdfs://0.0.0.0</value> but this didn't help.
Exception is following:
[Thread-14] INFO org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob - Job status available at: http://lagrangian:8088/proxy/application_1525212500911_0002/
[Thread-14] ERROR org.apache.crunch.impl.mr.exec.MRExecutor - Pipeline failed due to exception
java.io.IOException: java.io.IOException: java.net.ConnectException: Call From lagrangian/169.254.105.43 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:92)
at org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.run(CrunchJobHooks.java:79)
at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkRunningState(CrunchControlledJob.java:288)
at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkState(CrunchControlledJob.java:299)
at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.checkRunningJobs(CrunchJobControl.java:193)
at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:313)
at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:131)
at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:58)
at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:90)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.net.ConnectException: Call From lagrangian/169.254.105.43 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:429)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:617)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:616)
at org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:84)
... 9 more
Caused by: java.net.ConnectException: Call From lagrangian/169.254.105.43 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy20.getJobReport(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:325)
... 19 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
... 28 more
I read it is probably related with Job History Server, but I am not sure, how to run it on Windows.
Probably because the JobHistory server isn't started. You can run it using
mapred historyserver
Should be very similar between Windows and Linux. Check log output and jps to verify it's running.
Your service addresses should ideally be a hostname (but not localhost), while 0.0.0.0 will make them listen on all addresses
I'm using Hortonworks Sandbox HDP 2.4 and I can't use the list-databases tool, although import is working.
sqoop list-databases --connect jdbc:mysql://<ip>:<port>/ --driver com.mysql.jdbc.Driver --username=myusername -P
This the result that I get
16/08/11 07:44:47 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
16/08/11 07:44:47 INFO manager.SqlManager: Using default fetchSize of 1000
16/08/11 07:44:47 ERROR manager.SqlManager: Generic SqlManager.listDatabases() not implemented.
Could not retrieve database list from server
16/08/11 07:44:47 ERROR tool.ListDatabasesTool: manager.listDatabases() returned null
The response says that I need to set the --connection-manager parameter, which I do not know.
If I will take out the --driver argument, I get exception:
16/08/11 07:48:12 ERROR manager.CatalogQueryManager: Failed to list databases
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2370)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2154)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:241)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:295)
... 24 more
16/08/11 07:48:12 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2370)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2154)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
... 7 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:241)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:295)
... 24 more
What's the value for --connection-manager?
What worked for me is this:
$ sqoop list-databases
--connect jdbc:mysql://<ip>:<port>/
--connection-manager org.apache.sqoop.manager.MySQLManager
--username=myusername -P
Same thing goes for list-tables tool
$ sqoop list-tables
--connect jdbc:mysql://<ip>:<port>/mydb
--connection-manager org.apache.sqoop.manager.MySQLManager
--username=myusername -P
I removed the --driver flag.
I think the issue is not about the connection manager.
This exception usually occurs when there is no service listening on the port you are trying to connect to. A couple of things could be happening:
You have not started your server.
Your server is not waiting to accept connections.
You are trying to connect to the wrong port number.
When I run a notebook I get this error
java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
PLease if you could help
I am getting ConnectionRefused error when I am running pig in mapreduce mode.
Details:
I have installed Pig from tarball( pig-0.14), and exported the classpath in bashrc.
I have all the Hadoop (hadoop-2.5) daemons up and running (confirmed by JPS).
[root#localhost sbin]# jps
2272 Jps
2130 DataNode
2022 NameNode
2073 SecondaryNameNode
2238 NodeManager
2190 ResourceManager
I am running pig in mapreduce mode:
[root#localhost sbin]# pig
grunt> file = LOAD '/input/pig_input.csv' USING PigStorage(',') AS (col1,col2,col3);
grunt> dump file;
And then I am getting the error:
java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:231)
at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:352)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:233)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
at org.apache.pig.PigServer.storeEx(PigServer.java:1034)
at org.apache.pig.PigServer.store(PigServer.java:997)
at org.apache.pig.PigServer.openIterator(PigServer.java:910)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:558)
at org.apache.pig.Main.main(Main.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy15.getJobReport(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:320)
... 26 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 34 more
I haven't configured any config file in pig tarball.Do I need to make any changes in the pig configuration to make it identify the hadoop installation?
In bashrc the HADOOP_HOME is set properly.
Please help.
To start it,go to hadoop's sbin dir and then type the command
mr-jobhistory-server.sh start historyserver --config $HADOOP_CONF_DIR
$HADOOP_CONF_DIR is the directory where hadoop's config files like hdfs-site.xml etc. reside.
Found the solution.
The line:
Caused by: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
comes when JobHistoryServer is not up.
So starting the JobHistoryServer would eliminate this issue.
To start it, go to hadoop's sbin dir and then the command:
mr-jobhistory-server.sh start
Do a jps to check if JobHistoryServer is up, and then re-execute your Pig commands.
Updated command(as of 2017):
./mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver
check
jps
~/src/storm-0.8.1/bin/storm jar /root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology demo
I tried to run this and it said that the problem is with the nimbus connection , but my storm client (and supervisor in the same time ) is connected with my nimbus (shown in Strom UI )
Running: java -client -Dstorm.options= -Dstorm.home=/root/src/storm-0.8.1 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp /root/src/storm-0.8.1/storm-0.8.1.jar:/root/src/storm-0.8.1/lib/asm-4.0.jar:/root/src/storm-0.8.1/lib/commons-codec-1.4.jar:/root/src/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/src/storm-0.8.1/lib/kryo-2.17.jar:/root/src/storm-0.8.1/lib/clout-0.4.1.jar:/root/src/storm-0.8.1/lib/clojure-1.4.0.jar:/root/src/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/src/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/src/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/src/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/src/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/src/storm-0.8.1/lib/joda-time-2.0.jar:/root/src/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/src/storm-0.8.1/lib/commons-exec-1.1.jar:/root/src/storm-0.8.1/lib/jetty-6.1.26.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5.jar:/root/src/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/src/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/src/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/src/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/src/storm-0.8.1/lib/clj-time-0.4.1.jar:/root/src/storm-0.8.1/lib/commons-lang-2.5.jar:/root/src/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/src/storm-0.8.1/lib/log4j-1.2.16.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/src/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/src/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/src/storm-0.8.1/lib/minlog-1.2.jar:/root/src/storm-0.8.1/lib/objenesis-1.2.jar:/root/src/storm-0.8.1/lib/jline-0.9.94.jar:/root/src/storm-0.8.1/lib/commons-io-1.4.jar:/root/src/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/src/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/src/storm-0.8.1/lib/json-simple-1.1.jar:/root/src/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/src/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/src/storm-0.8.1/lib/compojure-0.6.4.jar:/root/src/storm-0.8.1/lib/httpcore-4.1.jar:/root/src/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/src/storm-0.8.1/lib/guava-13.0.jar:/root/src/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/src/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/root/src/storm-0.8.1/lib/junit-3.8.1.jar:/root/src/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/src/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/src/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/src/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/root/src/storm-0.8.1/bin -Dstorm.jar=/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology demo Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused at backtype.storm.utils.NimbusClient.(NimbusClient.java:36) at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:53) at storm.starter.ExclamationTopology.main(ExclamationTopology.java:59) Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) at backtype.storm.utils.NimbusClient.(NimbusClient.java:34) ... 3 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
This seems to suggest that you are having trouble connecting to nimbus from the machine where you are trying to run this command from. Most likely there is something wrong in your ~/.storm/storm.yaml. Make sure you have this correctly configured to point to your nimbus server.