When connecting to Hadoop cluster, how can I know which version of Hadoop this cluster is running? In particular this is important for proper configuration of libraries when compiling and packaging Hadoop Java jobs with Maven.
The simplest way if you have ssh access to hadoop node is by running command
$ hadoop version
If you are looking for CDH version then check /usr/lib/hadoop/cloudera/cdh_version.properties
In cdh, in the cluster I am using, there is not any cdh_version.properties (or I couldn't find it)
If your cluster uses "Parcels", you could check which version of cdh is used by doing:
/opt/cloudera/parcels
And you could see the version as the name of the folder:
CDH-5.5.1-1.cdh5.5.1.p0.11
Note: I know that this is a not a general rule for getting which cdh version is used. I am trying to show an alternative way that it worked to me.
We can check the installed version with the help of following command:
cat /usr/lib/hadoop/cloudera/cdh_version.properties
Hope this may help you.
Related
How do I get the cluster versions on Solana?
For example, I'd like to know the current version of the devnet cluster.
Is there a command line or website which tells me the current versions?
Yes there is a CLI and here is a link to how to get the cluster version.
https://docs.solana.com/cli/choose-a-cluster#ensure-versions-match
I want to installed the hadoop(3.2) in my linux system which has installed the hadoop(2.7).When I execute hadoop , I can only get the information of hadoop 2.7 ,even if I change the environment variable. And the most confused thing is when I run echo $HADOOP_HOME , sometimes I can get the path of hadoop 2.7,sometimes hadoop 3.2. I hope someone can help me.
enter image description here
enter image description here
I'm writing a spark program that download different jars from maven based on the environment it runs on, each for a different version of Hadoop distribution (e.g. CDH, HDP, MapR).
This is necessary because some low-level APIs of HDFS and YARN are not shared between these distributions. However, I cannot find any public API of HDFS and YARN that tells their version.
Is it possible to do it only in Java? Or I have to run an external shell to know it?
In Java org.apache.hadoop.util.VersionInfo.getVersion() should work.
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/util/VersionInfo.html
For the CLIs, you can use:
$ hadoop version
$ hdfs version
$ yarn version
I am new to HBase and installed the 0.20.6 version on Cygwin as that is the requirement of my project. The master is running along with Hregionserver. However, when I try to create a table in the HBase shell, there does not seem to be any response.
When I see the list of commands, none of the table related commands are seen. How do I resolve this? Please see the screenshot.
As reported on the documentation
"HBase requires that a JDK be installed"
But there is no Java on Cygwin, so you can not have a cygwin version of HBase.
I'm new to hadoop. I want to install phoenix with hbase but I have installed hadoop cluster using ambari 1.7 on ubuntu. I'm not able to find any tutorial to do so.
If you build up your own Hadoop stack:
https://phoenix.apache.org/download.html
https://phoenix.apache.org/installation.html
If you use e.g. IBM Open Platform (which is for free btw):
https://developer.ibm.com/hadoop/blog/2015/10/21/installing-apache-phoenix-ibm-open-platform-apache-hadoop-4-1/
hbase should be available as service under add service button on home page.
For installing phoenix i used this link
http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2-trunk/bk_installing_manually_book/content/upgrade-22-7-a.html
basically yum install phoenix on each node and then create soft links to the phoenix server jar file
hth