I just started learning hadoop (CentOS7), here is a question about environment variable:
In my VM, rxie is the logged on user:
HADOOP_CONF_DIR = /opt/hadoop/hadoop-2.7.2/etc/rxie
There is no path of /opt/hadoop/hadoop-2.7.2/etc/rxie
it should be /opt/hadoop/hadoop-2.7.2/etc/hadoop
I am trying to find out where the variable is defined so that I can correct it.
here is what I come up:
bash-4.2# grep -r HADOOP_CONF_DIR ~/.*
/root/./.bashrc:export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
Please note HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop is correct.
I don't see where the erroneous HADOOP_CONF_DIR is defined?
It would be appreciated if anyone can enlighten me and offer a fix, thank you very much.
Run the following:
PS4='+ $BASH_SOURCE:$LINENO:' BASH_XTRACEFD=7 bash -xlic "" 7>trace.out
When that completes, run:
grep HADOOP_CONF_DIR trace.out
The output will show you every time that HADOOP_CONF_DIR was set, the file that set it, and the line number in that file.
Example
Let's take as an example something that I have in my .bashrc file:
$ grep HISTFILESIZE trace.out
++ /home/john1024/.bashrc:193:export HISTFILESIZE=20000
This shows that is was line 193 of ~/.bashrc that set HISTFILESIZE.
How it works
This command starts a bash login shell with diagnostics turned on. The PS4 variable sets the prefix for the diagnostic output so that it contains the file name and line number. For more details, see here.
Other method
Although it is less reliable, one can also try:
grep HADOOP_CONF_DIR ~/.bashrc ~/.bash_profile ~/.profile /etc/profile.d/* /etc/profile /etc/bashrc
Related
I installed Maven on macOS 10.12.6.
Added environment to .bash_profile file:
export PS1="\h --- \t *************************** \w $ "
export JAVA_HOME=$(/usr/libexec/java_home)
export MAVEN_HOME=/Users/{user}/Documents/Path/Maven/maven_3.3.9
export GRADLE_HOME=/Users/{user}/Documents/Path/Gradle/gradle_6.1.1
export PATH=$PATH:$MAVEN_HOME/bin
export PATH=$GRADLE_HOME/bin:$PATH
After that I save it and set source .bash_profile command in Terminal.
When I try in Terminal input mvn -v command I see:
-bash: mvn: command not found
When I input $MAVEN_HOME I see:
: No such file or directory/Path/Maven/maven_3.3.9
If I use echo $MAVEN_HOME command I see:
/Users/{user}/Documents/Path/Maven/maven_3.3.9
If I input this mvn I see:
-bash: mvn: command not found
JDK on Mac - $JAVA_HOME command in Terminal:
-bash: /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home: is a directory
Can you help me please to understand where I missed.
Thank you very much for your answers, I fixed my problem just like that:
export PATH=/Users/{user}/Documents/Path/Maven/maven_3.3.9/bin:$PATH
After that everything worked as expected.
If you think my method is not good, please tell me why?
Since it is a maven 2+, export M2_HOME (see "What is the difference between M2_HOME and MAVEN_HOME")
And double check what echo $PATH returns after sourcing the .bashrc.
Make sure it:
includes $M2_HOME/bin
ls $M2_HOME/bin does exists.
Check this
1) Review your path, I think you are missing '$' character right before {user}
2) Review execution permission for 'maven_3.3.9/bin/mvn'
3) Make sure you are using Bash as Unix shell (and not zsh or another)
echo $SHELL
# output should look like: '/bin/bash'
4) If you are using bash then add maven's ENV variables to the file ~/.bashrc (or ~/.zshrc if you are using zsh)
5) 'source' your configuration
source ~/.bashrc
Notes
I recommend understand differences between .bashrc and .bash_profile: http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html
If you want to keep your configuration on .bash_profile then just rebooting your system should be enough (to start a login shell which load .bash_profile)
This question already has an answer here:
How to restore .bash_profile on a mac? None of my unix terminal are working [closed]
(1 answer)
Closed 5 years ago.
I tried installing Anaconda to get many python packages at once but had some issues with python IDLE where it said No package found so had to manually set the path in ~/.bashrc.
Once I set the path in ~/.bashrc the IMPORT ERROR in python IDLE was solved but I'm unable to use commands on terminal now.
I'm getting this error all the time.
sid#sids-ubuntu:~$ ls
Command 'ls' is available in '/bin/ls'
The command could not be located because '/bin' is not included in the PATH environment variable.
ls: command not found
sid#sids-ubuntu:~$ sudo
Command 'sudo' is available in '/usr/bin/sudo'
The command could not be located because '/usr/bin' is not included in the PATH environment variable.
sudo: command not found
sid#sids-ubuntu:~$ mkdir aa
Command 'mkdir' is available in '/bin/mkdir'
The command could not be located because '/bin' is not included in the PATH environment variable.
mkdir: command not found
I did export PATH=/usr/bin:/bin to find out my $PATH and content of /etc/environment. It seems both are different.
sid#sids-ubuntu:~$ export PATH=/usr/bin:/bin
sid#sids-ubuntu:~$ echo $PATH
/usr/bin:/bin
sid#sids-ubuntu:~$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
Doesn't Ubuntu look for $PATH in /etc/environment?
If yes, what could be the reason my $PATH is different from /etc/environment? and please help me fix it!
If not, where does Ubuntu look for $PATH? please help me fix it!
Be aware that your /etc/environment is only re-read at reboot.
When you want to change your path, be sure to include the existing part as well. To do that, add $PATH in the new path definition.
export PATH="$PATH:/usr/bin"
Looking at your problems, adding the $PATH in your ~/.bashrc should do the trick. If not, open a new terminal and show us the output of
echo $PATH
When adding some directory to PATH it's good idea not to overwrite previous value, just append desired directory (e.g. $HOME/bin), in your ~/.bashrc add at the end line (and remove any previous tampering with PATH)
export PATH="$PATH:$HOME/bin"
and run:
source ~/.bashrc
(or just open new session of terminal).
PATH is an environment variable, and therefore it is not looked up in any file.
There are several files which are sourced when bash is invoked (see the section named INVOCATION in the bash man page), and while sourcing these files, the environment variable PATH can be set, respectively manipulated. Note that .bashrc is not always processed; please read the bash man-page carefully to understand, which files are included under which condition.
I have read several answers on how to set environment variables on OSX permanently.
First, I tried this, How to permanently set $PATH on Linux/Unix but I had an error message saying no such file and directory, so I thought I could try ~/.bash_profile instead of ~/.profile but it did not work.
Second, I found this solution How to set the $PATH as used by applications in os x , which advises to make changes in
~/.MacOSX/environment.plist
but again I had no such file and directory error.
I need a way to set these variables such that it won't require setting them again and again each time I open a new terminal session.
You have to add it to /etc/paths.
Reference (which works for me) : Here
I've found that there are some files that may affect the $PATH variable in macOS (works for me, 10.11 El Capitan), listed below:
As the top voted answer said, vi /etc/paths, which is recommended from my point of view.
Also don't forget the /etc/paths.d directory, which contains files may affect the $PATH variable, set the git and mono-command path in my case. You can ls -l /etc/paths.d to list items and rm /etc/paths.d/path_you_dislike to remove items.
If you're using a "bash" environment (the default Terminal.app, for example), you should check out ~/.bash_profile or ~/.bashrc. There may be not that file yet, but these two files have effects on the $PATH.
If you're using a "zsh" environment (Oh-My-Zsh, for example), you should check out ~./zshrc instead of ~/.bash* thing.
And don't forget to restart all the terminal windows, then echo $PATH. The $PATH string will be PATH_SET_IN_3&4:PATH_SET_IN_1:PATH_SET_IN_2.
Noticed that the first two ways (/etc/paths and /etc/path.d) is in / directory which will affect all the accounts in your computer while the last two ways (~/.bash* or ~/.zsh*) is in ~/ directory (aka, /Users/yourusername/) which will only affect your account settings.
Read more: Mac OS X: Set / Change $PATH Variable - nixCraft
For a new path to be added to PATH environment variable in MacOS just create a new file under /etc/paths.d directory and add write path to be set in the file. Restart the terminal. You can check with echo $PATH at the prompt to confirm if the path was added to the environment variable.
For example: to add a new path /usr/local/sbin to the PATH variable:
cd /etc/paths.d
sudo vi newfile
Add the path to the newfile and save it.
Restart the terminal and type echo $PATH to confirm
You can open any of the following files:
/etc/profile
~/.bash_profile
~/.bash_login (if .bash_profile does not exist)
~/.profile (if .bash_login does not exist)
And add:
export PATH="$PATH:your/new/path/here"
You could also add this
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
to ~/.bash_profile, then create ~/.bashrc where you can just add more paths to PATH. An example with .
export PATH=$PATH:.
If you are using zsh do the following.
Open .zshrc file nano $HOME/.zshrc
You will see the commented $PATH variable here
# If you come from bash you might have to change your $PATH.
# export PATH=$HOME/bin:/usr/local/...
Remove the comment symbol(#) and append your new path using a separator(:) like this.
export
PATH=$HOME/bin:/usr/local/bin:/Users/ebin/Documents/Softwares/mongoDB/bin:$PATH
Activate the change source $HOME/.zshrc
You're done !!!
sudo nano /etc/paths
now find the path of command i am giving an example of setting path for flutter.
/Users/username/development/flutter/bin
now cntrol+x and then y . reopen the terminal and check.
launchctl setenv environmentvariablename environmentvariablevalue
or
launchctl setenv environmentvariablename `command that will generate value`
use proper ` and remember to restart application or terminal for the environment variable to take effect.
you can check environment variable by printenv command.
note : environment variable named path is already set by someone else so we are not appending anything to that path at all here.
shows all hidden files like .bash_profile and .zshrc
$ ls -a
Starting with macOS Catalina, mac uses zsh instead of bash. so by default mac uses zsh.
Check which shell running:
$ echo $SHELL
/usr/zsh
$ cd $HOME
$ open -e .zshrc
or if using vim
$ vi .zshrc
Then add it like this
$ export my_var="/path/where/it/exists"
$ export PATH=$PATH:/$my_var/bin
For example: if installed app named: myapp in /Applications
Then
export MYAPP_HOME=/Applications/myapp
export PATH=$PATH:$MYAPP_HOME/bin
or shortcut
export PATH=${PATH}:/Applications/myapp/bin
TADA you set for life !!! Thank me later
19 October 2021.
Confirming iplus26's answer with one correction.
Test environment
OS: macOS 11.6 (Big Sur) x86_64
Shell: zsh 5.8
Below is the order in which the $PATH environment variable is modified:
each line in etc/paths text file gets appended
each line in each text file inside etc/paths.d directory gets appended
finally, the $PATH is further modified in ~/.zshrc
iplus26's answer stated "when (you run) echo $PATH, The $PATH string will be PATH_SET_IN_3&4:PATH_SET_IN_1:PATH_SET_IN_2" but this isn't always the case. It will have to depend on what the script is doing inside .zshrc. E.g. If we do something like
PATH="/new/path:${PATH}"
then, the new path will be in the beginning of the path list. However, if we do something like
PATH="${PATH}:/new/path"
then, the new path will be appended at the end of the path list.
Of course, you'll have to make sure you export the modified path in the ~/.zshrc file.
export PATH=$PATH
One handy command you could use to "pretty print" your path list is
print -l $path
This will print each path on a new line for better readability. Note $path is like $PATH except it's delimited by a single space, instead of a colon, :.
Hopefully this can further clarify for newcomers to this thread.
For setting up path in Mac two methods can be followed.
Creating a file for variable name and paste the path there under
/etc/paths.d and source the file to profile_bashrc.
Export path variable in ~/.profile_bashrc as
export VARIABLE_NAME = $(PATH_VALUE)
AND source the the path. Its simple and stable.
You can set any path variable by Mac terminal or in linux also.
I'm working with Ubuntu 12.04 LTS.
I'm going through the hadoop quickstart manual to make a pseudo-distributed operation. It seems simple and straightforward (easy!).
However, when I try to run start-all.sh I get:
localhost: Error: JAVA_HOME is not set.
I've read all the other advice on stackoverflow for this issue and have done the following to ensure JAVA_HOME is set:
In /etc/hadoop/conf/hadoop-env.sh I have set
JAVA_HOME=/usr/lib/jvm/java-6-oracle
export JAVA_HOME
In /etc/bash.bashrc I have set
JAVA_HOME=/usr/lib/jvm/java-6-oracle
export JAVA_HOME
PATH=$PATH:$JAVA_HOME/bin
export PATH
which java returns:
/usr/bin/java
java –version works
echo $JAVA_HOME returns:
/usr/lib/jvm/java-6-oracle
I've even tried becoming root and explicitly writing the in the terminal:
$ JAVA_HOME=/usr/lib/jvm/java-6-oracle
$ export JAVA_HOME
$ start-all.sh
If you could show me how to resolve this error it would be greatly appreciated.
I'm thinking that my JAVA_HOME is being overridden somehow. If that is the case, could you explain to me how to make my exports global?
I am using hadoop 1.1, and faced the same problem.
I got it solved through changing JAVA_HOME variable in /etc/hadoop/hadoop-env.sh as:
export JAVA_HOME=/usr/lib/jvm/<jdk folder>
The way to solve this problem is to export the JAVA_HOME variable inside the conf/hadoop-env.sh file.
It doesn't matter if you already exported that variable in ~/.bashrc, it'll still show the error.
So edit conf/hadoop-env.sh and uncomment the line "export JAVA_HOME" and add a proper filesystem path to it, i.e. the path to your Java JDK.
# The Java implementation to use. Required.
export JAVA_HOME="/path/to/java/JDK/"
Ran into the same issue on ubuntu LTS 16.04. Running bash -vx ./bin/hadoop showed it tested whether java was a directory. So I changed JAVA_HOME to a folder and it worked.
++ [[ ! -d /usr/bin/java ]]
++ hadoop_error 'ERROR: JAVA_HOME /usr/bin/java does not exist.'
++ echo 'ERROR: JAVA_HOME /usr/bin/java does not exist.'
ERROR: JAVA_HOME /usr/bin/java does not exist.
So I changed JAVA_HOME in ./etc/hadoop/hadoop-env.sh to
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/
and hadoop starts fine.
The way to debug this is to put an "echo $JAVA_HOME" in start-all.sh. Are you running your hadoop environment under a different username, or as yourself? If the former, it's very likely that the JAVA_HOME environment variable is not set for that user.
The other potential problem is that you have specified JAVA_HOME incorrectly, and the value that you have provided doesn't point to a JDK/JRE. Note that "which java" and "java -version" will both work, even if JAVA_HOME is set incorrectly.
extract from etc/hadoop/hadoop-env.sh
The only required environment variable is JAVA_HOME. All others are
optional. When running a distributed configuration it is best to
set JAVA_HOME in this file, so that it is correctly defined on
remote nodes.
This means its better and advised to set JAVA_HOME here.. even though the existing definition reads the JAVA_HOME variable. Perhaps its not getting the value of JAVA_HOME from previously set value... standard apache manual does not tell this :( :(
This error is coming from Line 180
if [[ -z $JAVA_HOME ]]; then
echo "Error: JAVA_HOME is not set and could not be found." 1>&2
exit 1
fi
in libexec/hadoop-config.sh.
Try echo $JAVA_HOME in that script. If it doesn't recognize,
Find your JAVA_HOME using this:
$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")
and replace the line
export JAVA_HOME=${JAVA_HOME}
in /etc/hadoop/hadoop-env.sh with JAVA_HOME you got from above command.
I also had faced the similar problem in hadoop 1.1
I had not noticed that the JAVA_HOME was commented in: hadoop/conf/hadoop-env.sh
It was
/#JAVA_HOME=/usr/lib/jvm/java-6-oracle
Had to change it to
JAVA_HOME=/usr/lib/jvm/java-6-oracle
regardless of debian or any linux flavor, just know that ~/.bash_profile belongs to specific user and is not system wide.
in pseudo-distributed environment hadoop works on localhost so the $JAVA_HOME in .bash_profile is no use anymore.
just export the JAVA_HOME in ~/.bashrc and use it system wide.
Check if your alternatives is pointing to the right one, you might actually be pointing to a different version and trying to alter the hadoop-env.sh on another installed version.
-alternatives --install /etc/hadoop/conf [generic_name] [your correct path] priority {for further check man page of alternatives}
to set alternatives manually,
alternatives --set [generic name] [your current path].
Change the JAVA_HOME variable in conf/hadoop-env.sh
export JAVA_HOME=/etc/local/java/<jdk folder>
echo "export JAVA_HOME=/usr/lib/java" >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh
Notice: Do not use export JAVA_HOME=${JAVA_HOME} !
I put it on the first line of file ~/.bashrc, then it works well!
export JAVA_HOME=/usr/lib/jvm/default-java
After start the cygwin shell, it just locate in a wrong home dir:
xfire#codingme.com ~
$ pwd
/cygdrive/c/Users/xfire
But it used to be /home/xfire
xfire#codingme.com /etc
$ cat passwd | grep xfire
xfire:unused:22773:10513:U-CORP\xfire,S-1-5-21-527237240-725345543-682003330-12773:/home/xfire:/bin/bash
And the .bashrc in the /home/xfire was not executed, even I copy it to the /cygdrive/c/Users/xfire, it also doesn't work!
On cygwin, I add this to my ~/.bash_profile:
. ~/.bashrc
Some program add an "HOME" environment in windows registry and set the value to "C:\Users\xfire", that's why cygwin take that directory as the home. cygwin.com/faq-nochunks.html
On my version of cygwin I found that only ~/.profile was being execueted so added
if [ -e "${HOME}/.bash_profile" ]; then
source "${HOME}/.bash_profile"
fi
to the .profile file. My .bash_profile file contains another test for .bashrc and executes that from inside there. I also added the following two lines to my .bashrc file.
export BASH_ENV="${HOME}/.profile"
export ENV="${HOME}/.profile"
The first of these ensures that .profile gets executed in non-interactive terminals and the second ensures it gets executed in POSIX terminals. I found a very useful explanation of what get run and when in the Bash Reference Manual.
In your case it wouldn't help as you have an issue with the value of your HOME environment variable but this page comes up quite high on the list when searching for this issue.
You can also also set BASH_ENV variable, e.g., BASH_ENV='C:\DOCUME~1\dwyttenb\.bashrc'
~/.bash_profile is executed before the initial command prompt is returned to the user. After that, every time a new shell is opened, ~/.bashrc is executed.
Adding . ~/.bashrc at end of ~/.bash_profile has resolved the problem.
Now you can check existing aliases using alias command.
Find more details on github.