I am trying to setup pig on windows 7. I already have hadoop 2.7 single node cluster running on windows 7.
To setup pig, I have taken following steps as of now.
Downloaded the tar: http://mirror.metrocast.net/apache/pig/
Extracted tar to: C:\Users\zeba\Desktop\pig
Have set the Environment (User) Variable to:
PIG_HOME = C:\Users\zeba\Desktop\pig
PATH = C:\Users\zeba\Desktop\pig\bin
PIG_CLASSPATH = C:\Users\zeba\Desktop\hadoop\conf
Also changed HADOOP_BIN_PATH in pig.cmd to %HADOOP_HOME%\libexec as suggested by (Apache pig on windows gives "hadoop-config.cmd' is not recognized as an internal or external command" error when running "pig -x local") as was getting the same error
When I enter pig, I encounter the following error:
xmx1000m is not recognized as an internal or external command
Please help!
The error went away by installing pig-0.17.0. I was working with pig-0.16.0 previously.
Finally i got it. I changed HADOOP BIN PATH in pig.cmd to "HADOOP_HOME%\hadoop-2.9.2\libexec", as you can see "hadoop-2.9.2" is a subfile where "libexec" from my hadoop version is located..
Fix your "HADOOP_HOME" according to given image don't provide bin path only provide hadoop path.
Related
I have installed Hadoop 2.7.3 on Windows and I am able to start the cluster. Now I would like to have hive and went through the steps below:
1. Downloaded db-derby-10.12.1.1-bin.zip, unpacked it and started the startNetworkServer -h 0.0.0.0.
2. Downloaded apache-hive-1.1.1-bin.tar.gz from mirror site and unpacked it. Created hive-site.xml to have below properties:
javax.jdo.option.ConnectionURL
javax.jdo.option.ConnectionDriverName
hive.server2.enable.impersonation
hive.server2.authentication
datanucleus.autoCreateTables
hive.metastore.schema.verification
I have also setup HIVE_HOME and updated path. Also set HIVE_LIB and HIVE_BIN_PATH.
When I run hive from bin I get
'hive' is not recognized as an internal or external command,
operable program or batch file.
The bin/hive appears as filetype File.
Please suggest. Not sure if the hive version is correct one.
Thank you.
If someone is still going through this problem; here's what i did to solve hive installation on windows.
My configurations are as below (latest as of date):
I am using Windows 10
Hadoop 2.9.1
derby 10.14
hive 2.3.4 (my hive version does not contain bin/hive.cmd; the necessary file to run hive on windows)
#wheeler above mentioned that Hive is for Linux. Here's the hack to make it work for windows.
My Hive installation version did not come with windows executable files. Hence the hack!
STEP 1
There are 3 files which you need to specifically download from *https://svn.apache.org/repos/
https://svn.apache.org/repos/asf/hive/trunk/bin/hive.cmd
save it in your %HIVE_HOME%/bin/ as hive.cmd
https://svn.apache.org/repos/asf/hive/trunk/bin/ext/cli.cmd
save it in your %HIVE_HOME%/bin/ext/ as cli.cmd
https://svn.apache.org/repos/asf/hive/trunk/bin/ext/util/execHiveCmd.cmd
save it in your %HIVE_HOME%/bin/ext/util/ as execHiveCmd.cmd*
where %HIVE_HOME% is where Hive is installed.
STEP 2
Create tmp dir under your HIVE_HOME (on local machine and not on HDFS)
give 777 permissions to this tmp dir
STEP 3
Open your conf/hive-default.xml.template save it as conf/hive-site.xml
Then in this hive-site.xml, paste below properties at the top under
<property>
<name>system:java.io.tmpdir</name>
<value>{PUT YOUR HIVE HOME DIR PATH HERE}/tmp</value>
<!-- MY PATH WAS C:/BigData/hive/tmp -->
</property>
<property>
<name>system:user.name</name>
<value>${user.name}</value>
</property>
(check the indents)
STEP 4
- Run Hadoop services
start-dfs
start-yarn
Run derby
StartNetworkServer -h 0.0.0.0
Make sure you have all above services running
- go to cmd for HIVE_HOME/bin and run hive command
hive
Version 1.1.1 of Apache Hive does not contain a version that can be executed on Windows (only Linux binaries):
However, version 2.1.1 does have Windows capabilities:
So even if you had your path correctly set, cmd wouldn't be able to find an executable it could run, since one doesn't exist in 1.1.1.
i also run into this problem. to get necessary file to run hive on windows i have downloaded hive-2.3.9 and hive-3.1.2 but none of them have this files.so, we have two option:
Option 1: install hive-2.1.0 and set it up as i have tried,
Hadoop 2.8.0
derby 10.12.1.1
hive 2.1.0
Option 2: download whole bin directory and replace with yours hive bin directory. for downloading bin we need wget utility for windows.
after that run this command(to understand how it works):
wget -r -np -nH --cut-dirs=3 -R index.html
https://svn.apache.org/repos/asf/hive/trunk/bin/
your downloaded bin looks like:
after replacing it you are ready to go. so now my configurations are as below:
Hadoop 3.3.1
derby 10.13.1.1
hive 2.3.9
I am new to Hadoop and trying to run it on Windows 7.
Whenever I am trying to run hadoop bash script, I get the following error :
'-Xmx32m' is not recognized as an internal or external command,
operable program or batch file.
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
fs run a generic filesystem user client
version print the version
jar <jar> run a jar file
checknative [-a|-h] check native hadoop and compression libraries availability
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath prints the class path needed to get the
Hadoop jar and the required libraries
credential interact with credential providers
key manage keys via the KeyProvider
daemonlog get/set the log level for each daemon
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.
Also, when I run hdfs command ,
I get the following error :
-Xms1000m is not recognized as in internal or external command.
When I try to pass -Xmx and -Xms arguments, I get the following message :
Error occurred during initialization of VM
Could not reserve enough space for object heap
Can anyone help me out on this ?
The error message
is not recognized as an internal or external command
indicates that you attempted to run from the command line a program that Windows doesn't recognize. This likely has nothing to do with -Xms and -Xmx. The problem is Windows cannot find java.
Make sure you can ran java -version no matter what's the current folder you are in. If you can't, you need to add the java at the PATH environment variable.
This could also be an issue of installing java or hadoop in a folder that has spaces in the path e.g. C:\Program Files has a space in the folder and that can be a problem. If that's the cause then install java and hadoop on a different folder without spaces in the path.
i have installed hadoop single node on ubuntu 12.04. Now I am trying to install hbase over it (version 0.94.18). But i get the following errors(even though i have extracted it in the /usr/local/hbase):
Error: Could not find or load main class org.apache.hadoop.hbase.util.HBaseConfTool
Error: Could not find or load main class org.apache.hadoop.hbase.zookeeper.ZKServerTool
starting master, logging to /usr/lib/hbase/hbase-0.94.8/logs/hbase-hduser-master-ubuntu.out
nice: /usr/lib/hbase/hbase-0.94.8/bin/hbase: No such file or directory
cat: /usr/lib/hbase/hbase-0.94.8/conf/regionservers: No such file or directory
To resolve This Error
Download binary version of hbase
Edit conf file hbase-env.sh and hbase-site.xml
Set Up Hbase Home Directory
Start hbase By - Start-hbase.sh
Explanation To above Error:
Could not find or load main class your downloaded version does not have required jar
Hi can you tell when it is coming this error.
I think you gave environment set wrong
You should enter bellow command:
export HBASE_HOME="/usr/lib/hbase/hbase-0.94.18"
Then try hbase it will work.
If you want shell script you can download this lik :: https://github.com/tonyreddy/Apache-Hadoop1.2.1-SingleNode-installation-shellscript
It have hadoop, hive, hbase, pig.
Thank
Tony.
It is not recommended to run hbase from the source distribution directly instead you have to download the binary distribution as they have mentioned in their official site, follow the same instructions and you will get it up.
You could try installing the version 0.94.27
Download it from : h-base 0.94.27 dowload
This one worked for me.
Follow the instruction specified in :
Hbase installation guide
sed "s/<\/configuration>/<property>\n<name>hbase.rootdir<\/name>\n<value>hdfs:\/\/'$c':54310\/hbase<\/value>\n<\/property>\n<property>\n<name>hbase.cluster.distributed<\/name>\n<value>true<\/value>\n<\/property>\n<property>\n<name>hbase.zookeeper.property.clientPort<\/name>\n<value>2181<\/value>\n<\/property>\n<property>\n<name>hbase.zookeeper.quorum<\/name>\n<value>'$c'<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hbase/conf/hbase-site.xml
sed 's/localhost/'$c'/g' hbase/conf/regionservers -i
sed 's/#\ export\ HBASE_MANAGES_ZK=true/export\ HBASE_MANAGES_ZK=true/g' hbase/conf/hbase-env.sh -i
Yes just type this tree commands and you need change replace $c to your hostname.
Then try it will work.
I'm trying to set up a fully-distributed 4-node dev cluster with Hadoop 2.20 and HBase 0.98 on Windows. I've built Hadoop on Windows successfully, and more recently, also build HBase on Windows.
We have successfully ran the wordcount example from the Hadoop installation guide, as well as a custom WebHDFS job. As HBase fully-distributed on Windows isn't supported yet, I'm running HBase under cygwin.
When trying to start hbase from my master (./bin/start-hbase.sh), I get the following error:
2014-04-17 16:22:08,599 ERROR [main] util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
at org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:46)
Looking at the Shell.java source, what is here set as null, seems to be the HADOOP_HOME environment variable. With hadoop under D:/hadoop, and HBase under cygwin root at C:/cygwin/root/usr/local/hbase, the cygwin $HADOOP_HOME variable is /cygdrive/d/hadoop/, and the Windows system environment variable %HADOOP_HOME% is D:\hadoop . Seems to me like with those two variables, the variable should be found correctly...
Also potentially relevant: I'm running Windows Server 2012 x64.
Edit: I have verified that there actually is a winutils.exe in D:\hadoop\bin\ .
We've found it. So, in Hadoop's Shell.java, you'll find that there are two options to communicate the Hadoop-path.
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir");
// fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
After trial and error, we found that in the HBase options (HBase's hbase-env.sh, HBASE_OPTS variable), you'll need to add in this option with the Windows(!) path to Hadoop. In our case, we needed to add -Dhadoop.home.dir=D:/hadoop .
Good luck to anyone else who happens to stumble across this ;).
I have installed Pig 0.12.0 on my box. I have also installed Java and Hadoop and have set JAVA_HOME and HADOOP_HOME paths. When Igo to the bin directory of pig installation and type the following command on my command prompt:
pig -help
it errors out with the following message:
The system cannot find the path specified.
'-Xmx1000M' is not recognized as an internal or external command,
operable program or batch file.
Whats wrong?
Should I be using cygwin? (that didnt work either)
I just installed Pig 0.12.1 on Windows 7 without Hadoop installed. I also got this error message and resolved it by setting the "JAVA" environment variable to point to the java.exe executable.
In my case, I set JAVA=C:\Progra~1\Java\jdk1.8.0_05\bin\java.exe
I also set:
JAVA_HOME=C:\Progra~1\Java\jdk1.8.0_05
PIG_HOME=C:\pig-0.12.1 (This is where I extracted pig-0.12.1.tar.gz)
and added C:\pig-0.12.1\bin to my PATH environment variable.
Hope this helps anyone else with this issue!
I know this is a very late reply, but hope this will help someone to configure pig
Using OS Windows 8.1 ProN x64
Note -> Life will be easier in configuration, if your directory path doesn't contains a whitespace.
Steps to configure Pig with Hadoop
java path location (jdk1.8.0_151)
JAVA_HOME C:\Java\jdk1.8.0_151
python path location (Python27)
C:\Python27
ant path location (apache-ant-1.10.1)
ANT_HOME F:\Hadoop\apache-ant-1.10.1
hadoop path location (hadoop-2.8.2)
HADOOP_HOME F:\Hadoop\hadoop-2.8.2
HADOOP_COMMON_LIB_NATIVE_DIR %HADOOP_HOME%\lib\native
HADOOP_CONF_DIR %HADOOP_HOME%\etc\hadoop
pig path location (pig-0.17.0)
PIG_HOME F:\Hadoop\pig-0.17.0
System Variables Path
C:\Python27\;C:\Python27\Scripts;C:\Java\jdk1.8.0_151\bin;F:\Hadoop\hadoop-2.8.2\bin;F:\Hadoop\hadoop-2.8.2\sbin;F:\Hadoop\pig-0.17.0\bin;
explore winutils-master.zip on github, pull and download, extract to %HADOOP_HOME%\bin directory
open %PIG_HOME%\bin\pig.cmd using notepad/notepad++ (recommended notepad++)
change the below said line and later save it, close it -
"set HADOOP_BIN_PATH=%HADOOP_HOME%\bin" to
"set HADOOP_BIN_PATH=%HADOOP_HOME%\libexec"
so now the pig will access hadoop-config.cmd inside %HADOOP_HOME% path as we configured earlier.
start-all.cmd from hadoop to start the cluster with all dependencies.
go to %PIG_HOME%/bin, check with pig -help (if results are as parameterized).
pig (Enter to grunt shell.)
Note -> there are much possibilities to get below exception, if you don't configure as above.
'F:\Hadoop\hadoop-2.8.2\bin\hadoop-config.cmd' is not recognized as an internal or external command, operable program or batch file.
'-Xmx1000M' is not recognized as an internal or external command, operable program or batch file.
hope, these illustrated steps will help you to configure and start the pig grunt shell, thanks.
This will solve your problem...
1.Download PIG ->http://mirrors.estointernet.in/apache/pig/pig-0.16.0/
2.Set properties->
PIG_HOME=C:\Users\lenovo\Downloads\pig-0.16.0\pig-0.16.0
path=C:\Users\lenovo\Downloads\pig-0.16.0\pig-0.16.0\bin
PIG_CLASSPATH=C:\Users\lenovo\Downloads\hadoop-2.7.3\hadoop-2.7.3\etc\hadoop(Where
core-site.xml,mapred-site.xml are present)
3.
->open the file pig.cmd(From bin directory of PIG)
->look for the line set HADOOP_BIN_PATH=%HADOOP_HOME%\bin
->replace this with set HADOOP_BIN_PATH=%HADOOP_HOME%\libexec
4.Now in command prompt give->pig `enter code here`