Error Log :
Listening on 0.0.0.0:4567
2014-11-27 00:37:47,570 WARN - log - FAILED SocketConnector#0.0.0.0:4567: java.net.BindException: Address already in use: JVM_Bind
2014-11-27 00:37:47,572 WARN - log - FAILED org.eclipse.jetty.server.Server#5595a5a9: java.net.BindException: Address already in use: JVM_Bind
you didn't terminate spark properly and it's still running and bound to the default port 4567.
Just close the previous process and restart. If you can't find it or it's running in the background you can use NETSTAT to find the process and kill it.
on windows netstat -b to find the process
Related
As I'm trying to exploit the Camunda Enterprise version, now I'm stuck at the Optimize server. I'm not able to start the Optimize server followed by instruction. Here is the log file
16:59:37.322 [main] DEBUG o.e.j.u.component.AbstractLifeCycle - starting ServerConnector#48974e45{HTTP/1.1,[http/1.1]}{0.0.0.0:8095}
16:59:37.325 [main] WARN o.e.j.u.component.AbstractLifeCycle - FAILED ServerConnector#48974e45{HTTP/1.1,[http/1.1]}{0.0.0.0:8095}: java.io.IOException: Failed to bind to /0.0.0.0:8095
java.io.IOException: Failed to bind to /0.0.0.0:8095
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.camunda.optimize.jetty.EmbeddedCamundaOptimize.startOptimize(EmbeddedCamundaOptimize.java:169)
at org.camunda.optimize.Main.main(Main.java:17)
Caused by: java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
... 8 common frames omitted
I know it is obvious but I tried to changing different port. It's still showing the same error, no matter which port I choose. Please help to take a look.
The port is already in use, the message is obvious. All you can do is to kill the process that is listening to this thread.
I use Windows, but you can search for Linux way to do, the principle is the same.
Windows:
Find a process that uses the port 8095 using netstat -aon | find "8095". The result looks like:
TCP 0.0.0.0:8003 0.0.0.0:0 LISTENING 23332
TCP [::]:8003 [::]:0 LISTENING 23332
Find in the Task Manager in the Details tab what runs under the PID (process ID) number 23332.
End such a task. It will be probably java.exe since there is a pending process listening to the port. Restart the IDE and everything should work well. If it is ex. a database or anything else, you have to use a different port than 8095 either at your side or the application that already uses it.
Linux: https://unix.stackexchange.com/questions/140482/kill-any-service-running-at-a-specific-port
When I tried to stop the ZooKeeper with command "zkServer stop", I got the following result:
call "C:\Program Files\Java\jdk1.8.0_121"\bin\java "-Dzookeeper.log.dir=C:\zookeeper-3.4.10\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "C:\zookeeper-3.4.10\bin\..\build\classes;C:\zookeeper-3.4.10\bin\..\build\lib\*;C:\zookeeper-3.4.10\bin\..\*;C:\zookeeper-3.4.10\bin\..\lib\*;C:\zookeeper-3.4.10\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "C:\zookeeper-3.4.10\bin\..\conf\zoo.cfg" stop
Output:
2017-09-01 13:55:22,070 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2017-09-01 13:55:22,072 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2017-09-01 13:55:22,072 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2017-09-01 13:55:22,072 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, running in standalone mode
2017-09-01 13:55:22,145 [myid:] - ERROR [main:ZooKeeperServerMain#55] - Invalid arguments, exiting abnormally
java.lang.NumberFormatException: For input string: "C:\zookeeper-3.4.10\bin\..\conf\zoo.cfg"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.zookeeper.server.ServerConfig.parse(ServerConfig.java:59)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:84)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2017-09-01 13:55:22,148 [myid:] - INFO [main:ZooKeeperServerMain#56] - Usage: ZooKeeperServerMain configfile | port datadir [ticktime] [maxcnxns]
I am sure I have started the Zookeeper, because when I tried to start a new one, it shows "java.net.BindException: Address already in use: bind"
Another strange problem is that I cannot find Zookeeper in the Windows Service list. However, when I tried to show all port usage in Windows PowerShell by netstat -and, I found the 2181 is in use:
Proto Local Address Foreign Address State
TCP 0.0.0.0:2181 0.0.0.0:0 LISTENING
[java.exe]
TCP [::1]:2181 [::1]:62268 ESTABLISHED
[java.exe]
TCP [::1]:2181 [::1]:62279 ESTABLISHED
[java.exe]
TCP [::1]:2181 [::1]:62280 ESTABLISHED
[java.exe]
TCP [::1]:2181 [::1]:62281 ESTABLISHED
[java.exe]
I was running ZooKeeper on Windows and wasn't able to stop ZooKeeper running at 2181 port using zookeeper-stop.sh, so I tried this double slash "//" method to taskkill. It worked
1. netstat -ano | findstr :2181
TCP 0.0.0.0:2181 0.0.0.0:0 LISTENING 8876
TCP [::]:2181 [::]:0 LISTENING 8876
2. taskkill //PID 8876 //F
SUCCESS: The process with PID 8876 has been terminated.
Credit goes to: How do I kill the process currently using a port on localhost in Windows?
It looks like there is an open bug concerning the start and stop commands in ZooKeeper
To start ZooKeeper, omit the start parameter and call bin\zkServer instead.
To stop it, if you don't see the process from the task manager. You need to connect to ZooKeeper server as an administrator and perform the kill commands.
More details are here.
I'm running a Pig job that fails to connect to the Hadoop job history server.
The task (usually any task with GROUP BY) runs for a while and then it starts with a message like:
2015-04-21 19:05:22,825 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-04-21 19:05:26,721 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-04-21 19:05:29,721 [main] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
It then continues for a while retrying the connection. Sometimes it precedes further with the job. Othertimes it throws this exception:
2015-04-21 19:05:55,822 [main] WARN org.apache.pig.tools.pigstats.mapreduce.MRJobStats - Unable to get job counters
java.io.IOException: java.io.IOException: java.net.NoRouteToHostException: No Route to Host from cluster-01/10.10.10.11 to 0.0.0.0:10020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getCounters(HadoopShims.java:132)
at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addCounters(MRJobStats.java:284)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:235)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
I found this question here but in my case the job history server is started. If I run netstat, I find:
tcp 0 0 0.0.0.0:10020 0.0.0.0:* LISTEN 12073/java off (0.00/0/0)
Where 12073 is ...
12073 pts/4 Sl 0:07 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_historyserver -Xmx1000m -Djava.library.path=/data/hadoop/hadoop/lib -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/data/hadoop/hadoop-2.3.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/data/hadoop/hadoop-2.3.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/data/hadoop/hadoop/logs -Dhadoop.log.file=mapred-hadoop-historyserver-cluster-01.log -Dhadoop.root.logger=INFO,RFA -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
I tried opening the port 10200 in case it was a firewall issue:
ACCEPT tcp -- anywhere anywhere tcp dpt:10020
... but no luck.
After a few minutes, some of the tasks just arbitrarily continue to the next part.
I'm using Hadoop 2.3 and Pig 0.14.
My question is:
1) What are the possible reasons why Pig cannot connect to the job history server (JHS) given that the JHS is running on the same port that Pig looks for it?
... or failing that ...
2) Is there any way to just tell Pig to stop trying to connect to the JHS and continue with the task?
It seems that most Hadoop installation/configuration guides neglect to mention configuring the Job History Server. It seems that Pig, in particular, relies on this server. It also seems like the default (local) settings for the JHS won't work in a multi-node cluster.
The solution was to add the hostname of the server into the configuration in mapred-site.xml to make sure it could be accesses from the other machines. (In my version of the file, the lines had to be added as "new" ... there were no previous settings.)
<property>
<name>mapreduce.jobhistory.address</name>
<value>cm:10020</value>
<description>Host and port for Job History Server (default 0.0.0.0:10020)</description>
</property>
Then restart the job history server:
mr-jobhistory-daemon.sh stop historyserver
mr-jobhistory-daemon.sh start historyserver
If you get a bind exception (port in use), it means the stop didn't work. Either
Use ps ax | grep -e JobHistory to get the process and kill it manually with kill -9 [pid]. Then call the start command above again. Or
Use a different port in the configuration
Pig should pick up the new settings automatically. Run a Pig script and hope for the best.
start history server in hadoop bin using the below command
bin$ ./mr-jobhistory-daemon.sh start historyserver
run pig using the below command
$pig
Config mapreduce.jobhistory.address in hadoop/etc/hadoop/mapred-site.xml,
then:
mapred --daemon start
The solution was the History server was not running:
[user#vm9 sbin]$ ./mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/user/hadoop-2.7.7/logs/mapred-user-historyserver-vm9.out
[user#vm9 sbin]$ jps
5683 NameNode
6309 NodeManager
5974 SecondaryNameNode
8075 RunJar
6204 ResourceManager
8509 JobHistoryServer
5821 DataNode
8542 Jps
[user#vm9 sbin]$
Now pig can run properly and it will connect to the job history server and the dump command is working fine.
I downloaded CDH 4.5 quick start vm from here. Each service looks good except for below error was seen after I opened Activities tab to view mapreduce activities:
The Activity Monitor server (activitymonitor (localhost)) is unavailable or not responding to connections.
The problem remains after I tried to restart Activity Monitor service, then I found following error message in the log. Can anybody help take a look?
11:24:35.862 PM WARN org.mortbay.log
failed SelectChannelConnector#localhost.localdomain:9999: java.net.BindException: Address already in use
11:24:35.864 PM WARN org.mortbay.log
failed Server#59cc2f42: java.net.BindException: Address already in use
11:24:35.869 PM ERROR com.cloudera.cmon.firehose.Main
Failed to start Firehose
com.cloudera.enterprise.EnterpriseServiceException: java.net.BindException: Address already in use
at com.cloudera.cmon.firehose.AgentMessageService.startService(AgentMessageService.java:144)
at com.cloudera.enterprise.EnterpriseService.start(EnterpriseService.java:71)
at com.cloudera.enterprise.EnterpriseService.start(EnterpriseService.java:68)
at com.cloudera.cmon.firehose.Main.main(Main.java:371)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.cloudera.cmon.firehose.AgentMessageService.startService(AgentMessageService.java:142)
... 3 more
"java.net.BindException: Address already in use" clearly states that port 9999 is already occupied by some other service. You have to check the PID of the service and stop it:
lsof -P | grep LISTEN | grep 9999
use the PID to kill or if you know the service then stop gracefully
while installing Hadoop 0.23.0 on the cluster, node manager is unable to start and getting the following error.
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: 0.0.0.0/0.0.0.0:8080
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:303)
at org.apache.hadoop.mapred.ShuffleHandler.start(ShuffleHandler.java:255)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.start(AuxServices.java:123)
at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
... 4 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
It means 8080 is already in use.
so
sudo netstat -nvvpa |grep 8080 and see the o/p.
if it listens to any java, then if possible stop the process. and then try to start the nodemanager again.
This made my problem solved. thankyou.