When a namenode starts up, it reads HDFS state from an image file, fsimage and then applies the edits from the edit log file.
If I am not wrong , the Name node starts up means when we write start-all.sh. So during this start up time I think it read the fsimage and edit logs and merge them . But from which folder or from which location it actually read both these things?
In hadoop-1.x start-all.sh script internally performs two operation start-dfs.sh and start-mapred.sh. start-dfs.sh will start all daemons required for hdfs ie : datanode, namenode, secondary namenode
The checkpoint operation(applying edit logs to fsimage) happens during namenode start and this activity can be configured during namenode runs by tuning the parameter hdfs-site.xml --> dfs.namenode.checkpoint.period .
During namenode starts namenode daemons loads fsimage from the directory specified in hdfs-site.xml -> dfs.name.dir. This property should have been overridden otherwise it would take the default value (file:///tmp/dfs/name/)
Location of the edit logs can be found by checking the value of hdfs-site.xml -> dfs.name.edits.dir. default value of dfs.name.edits.dir is ${dfs.name.dir}.
The above property names are changed in hadoop-2.0
When you start daemons,namenode will check the configuration xml file named
core-site.xml
located in your hadoop conf folder. In my system it is located at usr/lib/hadoop/conf which is hadoop installed directory.
In that configuration file you can see,
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-0.20/cache/${user.name}</value>
</property>
</configuration>
In this code /var/lib/hadoop-0.20/cache/ is the location of fsimage, fstime and edits log.
If in case of namenode failure for some particular time fsimage data will be stored in secondynamenode temporarily, and after namenode get recovered temporary data will be stored in fsimage.
below is the location where you can find edits and fsimage files..
/app/hadoop/tmp/dfs/name/current
[core-site.xml]
<property><name>hadoop.tmp.dir</name><value>/app/hadoop/tmp</value>
i believe this will help you..
Related
everytime my hadoop server reboots, I have to format the namenode to start the hadoop. This removes all of the files in my hadoop installation.
I need to move my hadoop hdfs location from /tmp file to permenant location where whenever the server reboots, I don't have to format the namenode etc.
I am very new to hadoop.
How do I create a hdfs file in another directory?
How do I reference this data directory in config file so that I don't have to format the namenode?
These two properties of the hdfs-site.xml determine where local files are stored.
The defaults are under /tmp
dfs.namenode.name.dir
dfs.datanode.data.dir
You typically have to format a namenode only when the HDFS processes failed to terminate correctly (such as a power failure or forced shutdown). It is encouraged to run a standby Namenode to prevent these scenarios.
We have 4 datanode HDFS cluster ...there is large amount of space available on each data node of about 98gb ...but when i look at the datanode information .. it's only using about 10gb and running out of space ...
How can we make it use all the 98gb and not run out of space as indicated in image
this is the disk space configuration
this is the hdfs-site.xml on name node
<property>
<name>dfs.name.dir</name>
<value>/test/hadoop/hadoopinfra/hdfs/namenode</value>
</property>
this is the hdfs-site.xml under data node
<property>
<name>dfs.data.dir</name>
<value>/test/hadoop/hadoopinfra/hdfs/datanode</value>
</property>
Eventhough /test has 98GB and hdfs is configured to use it it's not using it
Am I missing anything while doing the configuration changes? And how can we make sure 98GB is used?
According to this Hortonworks Community Portal link, the steps to amend your Data Node directory are as follows:
Stop the cluster.
Go to the ambari HDFS configuration and edit the datanode directory configuration: Remove /hadoop/hdfs/data and /hadoop/hdfs/data1. Add [new directory location].
Login into each datanode (via SSH) and copy the contents of /data and /data1 into the new directory.
Change the ownership of the new directory and everything under it to “hdfs”.
Start the cluster.
I'm assuming that you're technically already up to Step 2 since you've displayed your correctly configured core-site.xml files in the original question. Make sure you've done the other steps and that all Hadoop services have been stopped. From there, change the ownership to the user running Hadoop (typically hdfs but I've worked in a place where root was running the Hadoop processes) and you should be good to go :)
I tried to start namenode using bin/start-all.sh. But, this command doesnt start namenode. I know if I do bin/hadoop namenode -format , namenode will start but in that case, I will lose all my data. Is there a way to start namenode without formatting it?
Your problem might be related to the following:
Hadoop writes its NameNode data in /tmp/hadoop- folder by default which is cleaned after every reboot.
Add following property to conf/hdfs-site.xml
<property>
<name>dfs.name.dir</name>
<value><path to your desired folder></value>
</property>
The "dfs.name.dir" property allows you to control where Hadoop writes NameNode metadata.
bin/start-all.sh should start the namenode, as well as the datanodes, the jobtracker and the tasktrackers. So, check the log of the namenode for possible errors.
An alternative way to skip starting the jobtracker and the tasktrackers and just start the namenode (and the datanodes) is by using the command:
bin/start-dfs.sh
Actually, bin/start-all.sh is equivalent to using the commands:
bin/start-dfs.sh, which starts the namenode and datanodes and
bin/start-mapred.sh, which starts the jobtracker and the tasktrackers.
For more details, visit this page.
I have installed Hadoop in pseudo distributed mode on my laptop, OS is Ubuntu.
I have changed paths where hadoop will store its data (by default hadoop stores data in /tmp folder)
hdfs-site.xml file looks as below :
<property>
<name>dfs.data.dir</name>
<value>/HADOOP_CLUSTER_DATA/data</value>
</property>
Now whenever I restart machine and try to start hadoop cluster using start-all.sh script, data node never starts. I confirmed that data node is not start by checking logs and by using jps command.
Then I
Stopped cluster using stop-all.sh script.
Formatted HDFS using hadoop namenode -format command.
Started cluster using start-all.sh script.
Now everything works fine even if I stop and start cluster again. Problem occurs only when I restart machine and try to start the cluster.
Has anyone encountered similar problem?
Why this is happening and
How can we solve this problem?
By changing dfs.datanode.data.dir away from /tmp you indeed made the data (the blocks) survive across a reboot. However there is more to HDFS than just blocks. You need to make sure all the relevant dirs point away from /tmp, most notably dfs.namenode.name.dir (I can't tell what other dirs you have to change, it depends on your config, but the namenode dir is mandatory, could be also sufficient).
I would also recommend using a more recent Hadoop distribution. BTW, the 1.1 namenode dir setting is dfs.name.dir.
For those who use hadoop 2.0 or above versions config file names may be different.
As this answer points out, go to the /etc/hadoop directory of your hadoop installation.
Open the file hdfs-site.xml. This user configuration will override the default hadoop configurations, that are loaded by the java classloader before.
Add dfs.namenode.name.dir property and set a new namenode dir (default is file://${hadoop.tmp.dir}/dfs/name).
Do the same for dfs.datanode.data.dir property (default is file://${hadoop.tmp.dir}/dfs/data).
For example:
<property>
<name>dfs.namenode.name.dir</name>
<value>/Users/samuel/Documents/hadoop_data/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/Users/samuel/Documents/hadoop_data/data</value>
</property>
Other property where a tmp dir appears is dfs.namenode.checkpoint.dir. Its default value is: file://${hadoop.tmp.dir}/dfs/namesecondary.
If you want, you can easily also add this property:
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/Users/samuel/Documents/hadoop_data/namesecondary</value>
</property>
I am trying to run hadoop as a root user, i executed namenode format command hadoop namenode -format when the Hadoop file system is running.
After this, when i try to start the name node server, it shows error like below
13/05/23 04:11:37 ERROR namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:411)
I tried to search for any solution, but cannot find any clear solution.
Can anyone suggest?
Thanks.
DFS needs to be formatted. Just issue the following command after stopping all and then restart.
hadoop namenode -format
Cool, i have found the solution.
Stop all running server
1) stop-all.sh
Edit the file /usr/local/hadoop/conf/hdfs-site.xml and add below configuration if its missing
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/tmp/dfs/name/data</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/tmp/dfs/name</value>
<final>true</final>
</property>
Start both HDFS and MapReduce Daemons
2) start-dfs.sh
3) start-mapred.sh
Then now run the rest of the steps to run the map reduce sample given in this link
Note : You should be running the command bin/start-all.sh if the direct command is not running.
format hdfs when namenode stop.(just like the top answer).
I add some more details.
FORMAT command will check or create path/dfs/name, and initialize or reinitalize it.
then running start-dfs.sh would run namenode, datanode, then namesecondary.
when namenode check not exist path/dfs/name or not initialize, it occurs a fatal error, then exit.
that's why namenode not start up.
more details you can check HADOOP_COMMON/logs/XXX.namenode.log
Make sure the directory you've specified for your namenode is completely empty. Something like a "lost+found" folder in said directory will trigger this error.
hdfs-site.xml your value is wrong. You input the wrong folder that's why is not starting the name node.
First mkdir [folder], then set hdfs-site.xml then format
make sure that the directory to name(dfs.name.dir) and data (dfs.data.dir) folder is correctly listed in hdfs-site.xml
Formatting namenode worked for me
bin/hadoop namenode -format