I have installed Hadoop 3.1.1 and it is working. However, when I try to compile the WordCount example, I am receiving this error:
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2358: HADOOP_COM.SUN.TOOLS.JAVAC.MAIN_USER: bad substitution
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2453: HADOOP_COM.SUN.TOOLS.JAVAC.MAIN_OPTS: bad substitution
To compile, I used the next line:
hadoop com.sun.tools.javac.Main WordCount.java
I have the next variables in the .bashrc:
#Hadoop variables
export HADOOP_HOME=/usr/local/hadoop
export CONF=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
#Java home
export JAVA_HOME=${JAVA_HOME}/java-8-oracle
#Path Java Tools
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
This time I am using Java 8 of Oracle because the apt-get of Ubuntu 18.08 LTS does not give me the option of installing the OpenJDK8. I have updated and upgraded Ubuntu.
I have read a lot of different post and possible solutions, but I cannot solve it.
This is a viable solution I found in https://janzhou.org/2014/how-to-compile-hadoop.html
Set the HADOOP_CLASSPATH:
export HADOOP_CLASSPATH=$(bin/hadoop classpath)
Compile:
javac -classpath ${HADOOP_CLASSPATH} -d WordCount/ WordCount.java
Related
I’m having an issue with instantclient. I was following the exact instructions in the http://www.oracle.com/technetwork/topics/intel-macsoft-096467.html#ic_osx_inst to install Instant Client 11.2 on macOS.
However, when starting the server in the command line it return the following error:
DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib,
What is your version of Python? Is it 32-bit or 64-bit?
64-bit
What is your cx_Oracle version?
The latest version as of today, I believe it's 6.4.1
What exact command caused the problem (e.g. what command did you try to install with)?
When executing this command ‘python manage.py’ to run the application
What error(s) you are seeing?
DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos
In my bash profile I have these established:
export PATH="/usr/local/mysql/bin:$PATH"
export PATH=/usr/local/sbin:/usr/local/bin:$PATH
# Load .bashrc if it exists
test -f ~/.bashrc && source ~/.bashrc
export WORKON_HOME=$HOME/Virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages'
export VIRTUALENVWRAPPER_PYTHON=/usr/local/Cellar/python#2/2.7.15_1/bin/python2.7
export VIRTUALENVWRAPPER_VIRTUALENV=usr/local/Cellar/python#2/2.7.15_1/bin/virtualenv
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true
export TNS_ADMIN=/usr/local/oracle/instantclient_11_2
source /usr/local/bin/virtualenvwrapper.sh
export PATH="/usr/local/opt/gettext/bin:$PATH"
if [ -f $(brew --prefix)/etc/bash_completion ]; then
source $(brew --prefix)/etc/bash_completion
fi
export PATH=~/instantclient_11_2:$PATH
# Setting PATH for Python 2.7
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH
Review the installation notes for cx_Oracle: https://cx-oracle.readthedocs.io/en/latest/installation.html#installing-cx-oracle-on-macos. Note especially the first sentence -- you cannot use the OS installation of Python!
If anyone is still getting the error DPI-1047 in Python, you might want to upgrade cx_Oracle. The latest major version (which supersedes cx_Oracle 8.3) has a new name python-oracledb. By default this new version doesn't need Oracle Client libraries. Check the installation documentation.
I tried to start hadoop, but it failed with nothing started. Following the console log.
Mac:sbin lqs2$ sh start-all.sh
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-functions.sh: line 398:
syntax error near unexpected token `<'
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-functions.sh: line 398:
`done < <(for text in "${input[#]}"; do'
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-config.sh: line 70:
hadoop_deprecate_envvar: command not found
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-config.sh: line 87:
hadoop_bootstrap: command not found
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-config.sh: line 104:
hadoop_parse_args: command not found
/Users/lqs2/Library/hadoop-3.1.1/libexec/hadoop-config.sh: line 105:
shift: : numeric argument required
WARNING: Attempting to start all Apache Hadoop daemons as lqs2 in 10
seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
I have tried any ways to solve it but nothing woked. Even I reinstalled the latest version. But the error is the same. It almost drives me mad.
Any answer is helpful. Thanks.
Hadoop scripts require bash, not sh
$ chmod +x start-all.sh
$ ./start-all.sh
Though I would suggest starting HDFS and YARN separately so that you can isolate other issues
You also need to downgrade Hadoop to at least the latest 2.7 release for Spark to work
In my case, I was seeing this error in OSX after installing Hadoop using HomeBrew. The solution was to do a fresh install after downloading the Hadoop (3.2.1) binary directly from the official website. While installing, I had set HADOOP_HOME and JAVA_HOME environment variables.
A word of caution: I found that the issue can occur if the following environment variables are defined in hadoop-env.sh :
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
I had initially added these variables while trying to fix the issue. Ultimately I removed them and the error disappeared.
Note, I executed all the Hadoop commands and scripts as non-root user, and also upgraded bash to version 5.0.17.
I want to run hadoop 2.6.1 on cygwin64. (Later I want to run hBsae,too)
The problem is, that I start start-dfs.sh, I will get this exception:
15/10/07 21:13:54 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:331)
Okay, I know that this is a well known problem and some solutions in this forum say something about adding HADOOP_HOME to the envirements. But the problem is still available.
this is a snippet of my bash_profile:
# cygwin path translation
export JAVA_HOME=$(cygpath -u $(cygpath -m -s "${JAVA_HOME}"))
export HADOOP_PREFIX=/home/user/hadoop-2.6.1
export PATH=$PATH:$(cygpath -u $(cygpath -m -s "${JAVA_HOME}\bin")):/home/user/hadoop-2.6.1/bin
export HADOOP_INSTALL=/home/user/hadoop-2.6.1
export HADOOP_HOME=/home/user/hadoop-2.6.1/
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=/home/user/hadoop-2.6.1/lib/native
export PATH=$PATH:$HADOOP_INSTALL/sbin
export PATH=$PATH:$HADOOP_INSTALL/bin
an echo $HADOOP_HOME also prints the path. But why is the path of winutils incomplete? Or is there a variable that I ferget?
greetings
You should set up winutils.exe in your PATH (environment variable).
I am trying to install pig 0.13 in ubuntu
I downloaded the pig-0.13.0.tar.gz and installed in $HOME path
After that I edit .bashrc file with below export commands
Please look at my .bashrc file
export HADOOP_HOME=$HOME/hadoop-1.2.1
export HADOOP_CONF_DIR=$HOME/hadoop-1.2.1/conf
export HADOOP_MAPRED_HOME=$HOME/hadoop-1.2.1/conf
export HADOOP_COMMON_HOME=$HOME/hadoop-1.2.1
export HADOOP_HDFS_HOME=$HOME/hadoop-1.2.1
###################JAVA############################
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i36
export PATH=$PATH:$HOME/hadoop-1.2.1/bin
########################PIG#######################
export PIG_HOME=/home/user/pig-0.13.0
export PATH=$PATH:$PIG_HOME/bin
But when i try to test whether pig is launced correctly or not.. it is not activated..
I reboot my machine and applied source .bashrc as well but nothing helped
user#ubuntu:~$ ls
Desktop examples.desktop Music pig-0.13.0.tar.gz Templates
Documents hadoop-1.2.1 Pictures Public Videos
Downloads hadoop-1.2.1.tar.gz pig-0.13.0 surender
user#ubuntu:~$ hadoop version
Warning: $HADOOP_HOME is deprecated.
Hadoop 1.2.1
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches /branch-1.2 -r 1503152
Compiled by mattf on Mon Jul 22 15:23:09 PDT 2013
From source with checksum 6923c86528809c4e7e6f493b6b413a9a
This command was run using /home/user/hadoop-1.2.1/hadoop-core-1.2.1.jar
user#ubuntu:~$ pig -help
pig: invalid option -- 'h'
usage: pig
user#ubuntu:~$
my $HOME is /home/user
Could somebody help me on this?
I downloaded pig 0.14.0 and I am running Hadoop 2.6.0 on MAC OSX. I followed all the installation steps for PIG at https://github.com/ucbtwitter/getting-started/wiki/Installing-Pig .I had set JAVA_HOME correctly as mentioned.
Even after running the ant "-Dhadoopversion=23 jar" command I am getting the same error "Cannot locate pig-core-h2.jar. do 'ant -Dhadoopversion=23 jar', and try again".
This error constantly arising
Cannot locate pig-core-h2.jar. do 'ant -Dhadoopversion=23 jar', and try again.
I studied the shell script by opening pig-0.14.0/bin/pig file and found that this error is related to the setting of CLASSPATH and PIG_HOME and JAVA_HOME variables.
Then I found that I mispelled the PIG_HOME then I corrected it.
Next I ran that specified command('ant -Dhadoopversion=23 jar') in the pig installation directory.
Then I got this error
Not a valid JAR: /Users/../../../pig-0.14.0/pig-0.14.0-SNAPSHOT-core-h2.jar /Users/../../../pig-0.14.0/pig-0.14.0-core-h2.jar
To resolve it remove that jar file in that location.
Then I got it working.
Find the path to the file pig-*-core-h2.jar.
I installed pig using brew install pig and found the jar in the path /usr/local/Cellar/pig/0.17.0/libexec
Run export PIG_HOME=/usr/local/Cellar/pig/0.17.0/libexec
This will fix your error.
i did this to fix the pig
mv /data/mapr/pig/pig-0.14/pig-0.14.0-mapr-1603-core-h2.jar /data/mapr/pig/pig-0.14/pig-0.14.0-mapr-1603-core-h2.jar.orig
The following solutions works:
Please make sure in your .bash_profile or .bashrc you have following environment variables:
export PIG_HOME="/Library/apache-pig-0.15.0"
export PATH="/Library/apache-pig-0.15.0/bin:${PATH}"
restart the machine or restart the unix terminal;
I replaced /Library/apache-pig-0.15.0/ with "home/cwu/Downloads/pig-0.15.0-src
"