j just want to install hadoop single node mode in ubuntu i formatted hdfs succesfully but when i start the job with start-all.sh I am getting error message as
rochdi#127:~$ start-all.sh
/usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-rochdi-namenode-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected
localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-rochdi-datanode-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected
localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-rochdi-secondarynamenode-127.0.0.1
/usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-rochdi-jobtracker-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected
localhost: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-rochdi-tasktracker-127.0.0.1
localhost: Erreur : impossible de trouver ou charger la classe principale localhost
and this is my bashrc config file
export HADOOP_PREFIX=/usr/local/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
and when i use jps command:
> rochdi#127:~$ jps
3427 Jps
help me please
Related
amtex#amtex-desktop:~$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
amtex#localhost's password:
localhost: starting namenode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-namenode-amtex-desktop.out
amtex#localhost's password:
localhost: starting datanode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-datanode-amtex-desktop.out
Starting secondary namenodes [0.0.0.0]
amtex#0.0.0.0's password:
0.0.0.0: starting secondarynamenode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-secondarynamenode-amtex-desktop.out
starting yarn daemons
starting resourcemanager, logging to /home/amtex/Documents/installed/hadoop/logs/yarn-amtex-resourcemanager-amtex-desktop.out
amtex#localhost's password:
localhost: starting nodemanager, logging to /home/amtex/Documents/installed/hadoop/logs/yarn-amtex-nodemanager-amtex-desktop.out
amtex#amtex-desktop:~$ jps
2404 Startup
18244 DataNode
18580 ResourceManager
18101 NameNode
18889 NodeManager
18425 SecondaryNameNode
18924 Jps
You need set password less login between machine
Below link has the step by step procedure to setup ssh password less login
http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/
Hope this helps!!!...
Based on my research i follow these steps to avoid the above problem
step 1: ssh-keygen -t rsa -P ""
step 2: cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
and now i started hadoop
amtex#amtex-desktop:~$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-namenode-amtex-desktop.out
localhost: starting datanode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-datanode-amtex-desktop.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/amtex/Documents/installed/hadoop/logs/hadoop-amtex-secondarynamenode-amtex-desktop.out
starting yarn daemons
starting resourcemanager, logging to /home/amtex/Documents/installed/hadoop/logs/yarn-amtex-resourcemanager-amtex-desktop.out
localhost: starting nodemanager, logging to /home/amtex/Documents/installed/hadoop/logs/yarn-amtex-nodemanager-amtex-desktop.out
amtex#amtex-desktop:~$ start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/amtex/Documents/installed/spark/logs/spark-amtex-org.apache.spark.deploy.master.Master-1-amtex-desktop.out
amtex#amtex-desktop:~$ start-slaves.sh
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/amtex/Documents/installed/spark/logs/spark-amtex-org.apache.spark.deploy.worker.Worker-1-amtex-desktop.out
amtex#amtex-desktop:~$ jps
21523 Jps
2404 Startup
21029 NodeManager
20581 DataNode
20439 NameNode
20760 SecondaryNameNode
21353 Master
21466 Worker
20911 ResourceManager
hduser#manoj-VirtualBox:/usr/local/hadoop$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: mkdir: cannot create directory ‘/tmp’: Permission denied
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-namenode.pid: No such file or directory
localhost: mkdir: cannot create directory ‘/tmp’: Permission denied
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-datanode.pid: No such file or directory
Starting secondary namenodes [0.0.0.0]
0.0.0.0: mkdir: cannot create directory ‘/tmp’: Permission denied
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-manoj-VirtualBox.out
0.0.0.0: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-secondarynamenode.pid: No such file or directory
starting yarn daemons
mkdir: cannot create directory ‘/tmp’: Permission denied
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-manoj-VirtualBox.out
/usr/local/hadoop/sbin/yarn-daemon.sh: line 125: /tmp/yarn-hduser-resourcemanager.pid: No such file or directory
localhost: mkdir: cannot create directory ‘/tmp’: Permission denied
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/yarn-daemon.sh: line 125: /tmp/yarn-hduser-nodemanager.pid: No such file or directory
I tried running
sudo chown -R hduser /usr/local/hadoop/
to give permissions to hduser. but still the same.
Also tried running sbin/start-dfs.sh & sbin/start-yarn.sh these also resulting same permisssion problems.
After Adding to sudoers some other permission denied errors are coming.
hduser#manoj-VirtualBox:/usr/local/hadoop$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-namenode.pid: Permission denied
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-datanode.pid: Permission denied
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-manoj-VirtualBox.out
0.0.0.0: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 165: /tmp/hadoop-hduser-secondarynamenode.pid: Permission denied
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-manoj-VirtualBox.out
/usr/local/hadoop/sbin/yarn-daemon.sh: line 125: /tmp/yarn-hduser-resourcemanager.pid: Permission denied
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-manoj-VirtualBox.out
localhost: /usr/local/hadoop/sbin/yarn-daemon.sh: line 125: /tmp/yarn-hduser-nodemanager.pid: Permission denied
I am trying to run Hadoop in Pseudo-Distributed mode. For this I am trying to follow this tutorial http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
I can ssh to my localhost and Format the filesystem. However, I can't start NameNode daemon and DataNode daemon by this command :
sbin/start-dfs.sh
When I execute it with sudo I get:
ubuntu#ip-172-31-42-67:/usr/local/hadoop-2.6.0$ sudo sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: Permission denied (publickey).
localhost: Permission denied (publickey).
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Permission denied (publickey).
and when executed without sudo:
ubuntu#ip-172-31-42-67:/usr/local/hadoop-2.6.0$ sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: mkdir: cannot create directory ‘/usr/local/hadoop-2.6.0/logs’: Permission denied
localhost: chown: cannot access ‘/usr/local/hadoop-2.6.0/logs’: No such file or directory
localhost: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-ip-172-31-42-67.out
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 159: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-ip-172-31-42-67.out: No such file or directory
localhost: head: cannot open ‘/usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-ip-172-31-42-67.out’ for reading: No such file or directory
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 177: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-ip-172-31-42-67.out: No such file or directory
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 178: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-ip-172-31-42-67.out: No such file or directory
localhost: mkdir: cannot create directory ‘/usr/local/hadoop-2.6.0/logs’: Permission denied
localhost: chown: cannot access ‘/usr/local/hadoop-2.6.0/logs’: No such file or directory
localhost: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-ip-172-31-42-67.out
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 159: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-ip-172-31-42-67.out: No such file or directory
localhost: head: cannot open ‘/usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-ip-172-31-42-67.out’ for reading: No such file or directory
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 177: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-ip-172-31-42-67.out: No such file or directory
localhost: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 178: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-ip-172-31-42-67.out: No such file or directory
Starting secondary namenodes [0.0.0.0]
0.0.0.0: mkdir: cannot create directory ‘/usr/local/hadoop-2.6.0/logs’: Permission denied
0.0.0.0: chown: cannot access ‘/usr/local/hadoop-2.6.0/logs’: No such file or directory
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-ip-172-31-42-67.out
0.0.0.0: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 159: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-ip-172-31-42-67.out: No such file or directory
0.0.0.0: head: cannot open ‘/usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-ip-172-31-42-67.out’ for reading: No such file or directory
0.0.0.0: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 177: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-ip-172-31-42-67.out: No such file or directory
0.0.0.0: /usr/local/hadoop-2.6.0/sbin/hadoop-daemon.sh: line 178: /usr/local/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-ip-172-31-42-67.out: No such file or directory
I also notice now that when executing ls to check content of hfs directories like here, it fails:
ubuntu#ip-172-31-42-67:~/dir$ hdfs dfs -ls output/
ls: Call From ip-172-31-42-67.us-west-2.compute.internal/172.31.42.67 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Can anyone tell me what could be the problem ?
I had the same problem and the only solution I found was:
https://anuragsoni.wordpress.com/2015/07/05/hadoop-start-dfs-sh-localhost-permission-denied-how-to-fix/
Which suggest you to generate a new ssh-rsa key
The errors above suggest a permissions problem.
You have to make sure that the hadoop user has the proper privileges to /usr/local/hadoop.
For this purpose you can try:
sudo chown -R hadoop /usr/local/hadoop/
Or
sudo chmod 777 /usr/local/hadoop/
Please make sure that you do the following "Configuration" correctly, you need to edit 4 ".xml" files:
Edit the file hadoop-2.6.0/etc/hadoop/core-site.xml , between , put in :
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
Edit the file hadoop-2.6.0/etc/hadoop/hdfs-site.xml, between put in:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
Edit the file hadoop-2.6.0/etc/hadoop/mapred-site.xm, between paste the following and save
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
Edit the file hadoop-2.6.0/etc/hadoop/yarn-site.xml, between paste the following and save
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
This is the error I get while starting resource manager daemon in Hadoop 2.6:
[hadoop#master hadoop]$ sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh: line 60: [: master.out: integer expression expected
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master.phd.com
Error: Could not find or load main class master.log
Hi i can't resolve my problem when running hadoop with start-all.sh
rochdi#127:~$ start-all.sh
/usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer
expression expected
starting namenode, logging to
/usr/local/hadoop/libexec/../logs/hadoop-rochdi-namenode-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [:
localhost: integer expression expected
localhost: starting datanode, logging to
/usr/local/hadoop/libexec/../logs/hadoop-rochdi-datanode-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [:
localhost: integer expression expected
localhost: starting secondarynamenode, logging to
/usr/local/hadoop/libexec/../logs/hadoop-rochdi-secondarynamenode-127.0.0.1
/usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer
expression expected
starting jobtracker, logging to
/usr/local/hadoop/libexec/../logs/hadoop-rochdi-jobtracker-127.0.0.1
localhost: /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [:
localhost: integer expression expected
localhost: starting tasktracker, logging to
/usr/local/hadoop/libexec/../logs/hadoop-rochdi-tasktracker-127.0.0.1
localhost: Erreur : impossible de trouver ou charger la classe
principale localhost
path:
rochdi#127:~$ echo "$PATH"
/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/hadoop/bin:/usr/local/hadoop/lib
before coming error i change hostname file as:
127.0.0.1 localhost
127.0.1.1 ubuntu.local ubuntu
and i configured my bashrc file as
export HADOOP_PREFIX=/usr/local/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
and jps command
rochdi#127:~$ jps
3427 Jps
help me please
i resolve the problem i just change my hostname but and all nodes start but when i stop them i have this message:
rochdi#acer:~$ jps
4605 NameNode
5084 SecondaryNameNode
5171 JobTracker
5460 Jps
5410 TaskTracker
rochdi#acer:~$ stop-all.sh
stopping jobtracker
localhost: no tasktracker to stop
stopping namenode
localhost: no datanode to stop
localhost: stopping secondarynamenode
after extract hadoop tar file
open ~/bashrc file and add following at the end of file
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
then,
edit file $HADOOP_HOME/etc/hadoop/core-site.xml add following config then start hadoop
<configuration>
<property>
<name>fs.default.name </name>
<value> hdfs://localhost:9000 </value>
</property>
</configuration>
still problem the use this link click here
Check your hosts file and *-site.xml files for host names. This error occurs when the hostnames not defined properly.
Use server IP address instead of using localhost in core-site.xml and check your entries in etc/hosts and slaves file.
The OP posted that the main error was fixed by changing the hostname (answered Dec 6 '13 at 14:19). It suggests issues with the file /etc/hosts and the file 'slaves' in the master. Remember that each host name in the cluster must match the values in those files. When an xml is wrongly configured, it normally throw up Connectivity issues between the ports of the services.
From the message "no ${SERVICE} to stop" most probably the previous start-all.sh leaved the java processes orphaned. The solutions is to stop manually each process, e.g.
$ kill -9 4605
and then execute again the start-all.sh command
It's important to mention that this is an old question, and currently we have the version 2 and 3 of Hadoop. And I strongly recommend using on of the latest version.