Path variables become longer and longer after doing source ~/.bashrc - bash

This might be a stupid question, but I wonder how to avoid this problem.
In my ~/.bashrc file, I add some local paths. The following is an example of the PYTHONPATH. In my work environment, I need to do "source ~/.bashrc" from time to time, and the following PYTHONPATH becomes longer and longer which is quite annoying.
Instead of appending to the existing PYTHONPATH, it might be nicer if I can append it to the "clean" PYTHONPATH. Is there anyway of doing this?
export PYTHONPATH=$PYTHONPATH:$CLIENTS:$EXPERIMENTAL/my_pythonlib:/mnt/src/cloud/chanwcom/chanwcom-speech/mnt/experimental/users/chanwcom/bin:$CK_MEDIA_FRAMEWORK

EDIT: I answered the question about keeping a clean PATH.
#mklement0 commented correctly, that the OP is talking about the PYTHONPATH.
I could correct my answer, but perhaps other readers have the same problem for PATH.
Dear Chanwcom, you can use the methods beneath, you only need to rename the variables.
Add some tests before expanding your path. Choose one of these examples.
if [[ -z "${my_clean_path}" ]]; then
my_clean_path="${PATH}"
fi
# some more commands
PATH="${my_clean_path}:${PYTHONPATH}"
or
if [[ -z "${python_added}" ]]; then
PATH="${PATH}:${PYTHONPATH}" # PATH += also possible here
fi
or (my favorite, without additional variables)
if [[ "${PATH}" != *${PYTHONPATH}* ]]; then
PATH="${PATH}:${PYTHONPATH}" # PATH += also possible here
fi
or open a second shell before changing the path. exit to the first shell and open a new fresh shell.

The "right" way to solve this is as chepner mentioned in a comment -- modify your PYTHONPATH in .bash_profile which gets run at login, rather than in .bashrc which gets run for every shell.
If you're unable to adjust the scripts or tools that append repeated items to your path, you may be able to clean things up by removing non-unique values.
Here's a strategy I use, which involves converting my $PATH to an array, inverting the array (i.e. turning array values into subscripts of an associative array) and then rebuilding the path from the array index:
if [[ ${BASH_VERSINFO[0]} -ge 4 ]]; then
path_a=( ${PATH//:/ } )
declare -A path_b=()
for i in ${path_a[#]}; do path_b[$i]=1; done
IFS=: eval 'PATH="${!path_b[*]}"'
fi
export PATH
You can adjust this for PYTHONPATH easily enough.
Note that associative arrays were introduced with Bash version 4, so they may not be available with the default bash in OSX. You should be fine just about anywhere else that's modern, though.
Note also that this solution will break if any of the directories in the path contain colons. But your path would be broken in that case anyway, I think.
Also, note that this solution uses eval, which some people consider dangerous, unclean, smelly. In the right setting, though, it can be a fine cheese.

Related

bash rematch as chroot

I'm using this in my script and it works fine on my root server and on some others I tested.
But there's a problem when using it on my webhosting (Hosted Plesk): The chrooted shell, it doesn't output anything. Just quits. My webhoster said I need to use the absolute paths, but I dont know how to apply this on the bash rematch.
#!/bin/bash
str='"<result="Abc1234" />"'
regex='<result="([0-9a-zA-Z._-/:]*)" />'
[[ $str =~ $regex ]] && echo ${BASH_REMATCH[1]}
(My first post here, sorry if I forgot something or misformatted this whole post)
In the discussion below the question it turned out that the issue is related to different default locales on the servers. Make sure that you are running the command with the right locale:
LANG=en_US.UTF-8 bash script.sh
(en_US.UTF-8) turned out to be the right locale in your case.
PS: Please also keep in mind what Ondrej K. says.
Instead of:
regex='<result="([0-9a-zA-Z._-/:]*)" />'
say:
regex='<result="([0-9a-zA-Z._/:-]*)" />'
- is moved to where it no longer can be considered meaning a range.
Actually I am a surprised it worked on the other system. I've replaced && echo ${BASH_REMATCH[1]} with ; echo $? (this was another possible debugging step) and was getting 2 which according to man bash means "syntactically incorrect regular expression".
If this doesn't help. Than we must be seeing different reason, way our shell interpret the script, but in any case printing the return status could then be the next step to take.

What's the difference between using env and preceding a command with variable assignments?

When I want to change the environment of a command I execute in bash, I can just precede it with a variable assignment. So for example, if I temporarily want to set the CLICOLOR variable I can do this:
CLICOLOR=1 ls
But I could also do this
env CLICOLOR=1 ls
Both result in the same result, so I wonder if there is any difference? Why do people use one over the other? Is it because of portability, or are there any differences when using output redirection or piping, etc?
This is mainly so that you don't have to run the shell just to set a variable. Many tools allow you to run a single command to perform a specific task (cron job, build system, internal scripting or macro languages for various tools) and you want to minimize the performance impact and security surface for such scenarios.
Both result in the same result, ..
No !
.. , so I wonder if there is any difference?
Yes !
Just some trial and error gave some interesting results. I think this would supplement #tripleee's answer
# Where it differs
# PATH=bingo echo "$PATH"
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
# Here the variable expansion happened before setting the PATH
# env PATH=bingo echo $PATH
env: ‘echo’: No such file or directory
# When an `env` is appended in the beginning, the PATH has changed (even)
# before the the full path of echo is resolved, hence the error
Read below points inline with [ this ] answer.
PATH=bingo echo $PATH starts with PATH=bingo which is an assignment.
env PATH=bingo echo $PATH starts with env which is not an assignment.
Hope this helps.

OS X 10.10 - path_helper causing invalid characters in path?

In an earlier question, I had problems with invalid/non-ASCII characters in my path statement. Thanks to some helpful answers, I was able to fix most of the problem, but I still need some help.
To resolve the problem, I made changes to my /private/etc/profile file and removed these lines:
if [ -x /usr/libexec/path_helper ]; then
eval `/usr/libexec/path_helper -s`
fi
if [ "${BASH-no}" != "no" ]; then
[ -r /etc/bashrc ] && . /etc/bashrc
fi
Once I removed those lines, the corrupted characters in my path went away. So, I suspect that path_helper was picking up some invalid characters and inserting them in my Path. But, I'm very new to all of this, so I'm not sure how to go about investigating how path_helper modifies my path?
path_helper is returning a line of shell code that is being executed with eval.
So if you want to see what it is doing just run it (either by hand or in your profile script) without the eval and backticks.
Try checking the files in /etc/paths.d if they contain any invalid (maybe invisible) character. For me it helped to do a "wc *" in that directory. That lists in the first column the number of lines in each file. I had some files without trailing newline. After adding a newline to the end of each file it worked. I had to call
PATH=''
eval `/usr/libexec/path_helper -s`
To make the change effective.
I could not find a pattern how to force a specific invalid pattern into the path, but the number of errors and the position of errors did not directly relate to the order of my files or the number of files with missing newline.

Deleting a directory contents using shell scripts

I am a newbie to Shell scripting. I want to delete all the contents of a directory which is in HOME directory of the user and deleting some files which are matching with my conditions. After googled for some time, i have created the following script.
#!/bin/bash
#!/sbin/fuser
PATH="$HOME/di"
echo "$PATH";
if [ -d $PATH ]
then
rm -r $PATH/*
fuser -kavf $PATH/.n*
rm -rf $PATH/.store
echo 'File deleted successfully :)'
fi
If I run the script, i am getting error as follows,
/users/dinesh/di
dinesh: line 11: rm: command not found
dinesh: line 12: fuser: command not found
dinesh: line 13: rm: command not found
File deleted successfully :)
Can anybody help me with this?
Thanks in advance.
You are modifying PATH variable, which is used by the OS defines the path to find the utilities (so that you can invoke it without having to type the full path to the binary). The system cannot find rm and fuser in the folders currently specified by PATH (since you overwritten it with the directory to be deleted), so it prints the error.
tl;dr DO NOT use PATH as your own variable name.
PATH is a special variable that controls where the system looks for command executables (like rm, fuser, etc). When you set it to /users/dinesh/di, it then looks there for all subsequent commands, and (of course) can't find them. Solution: use a different variable name. Actually, I'd recommend using lowercase variables in shell scripts -- there are a number of uppercase reserved variable names, and if you try to use any of them you're going to have trouble. Sticking to lowercase is an easy way to avoid this.
BTW, in general it's best to enclose variables in double-quotes whenever you use them, to avoid trouble with some parsing the shell does after replacing them. For example, use [ -d "$path" ] instead of [ -d $path ]. $path/* is a bit more complicated, since the * won't work inside quotes. Solution: rm -r "$path"/*.
Random other notes: the #!/sbin/fuser line isn't doing anything. Only the first line of the script can act as a shebang. Also, don't bother putting ; at the end of lines in shell scripts.
#!/bin/bash
path="$HOME/di"
echo "$path"
if [ -d "$path" ]
then
rm -r "$path"/*
fuser -kavf "$path"/.n*
rm -rf "$path/.store"
echo 'File deleted successfully :)'
fi
This line:
PATH="$HOME/di"
removes all the standard directories from your PATH (so commands such as rm that are normally found in /bin or /usr/bin are 'missing'). You should write:
PATH="$HOME/di:$PATH"
This keeps what was already in $PATH, but puts $HOME/di ahead of that. It means that if you have a custom command in that directory, it will be invoked instead of the standard one in /usr/bin or wherever.
If your intention is to remove the directory $HOME/di, then you should not be using $PATH as your variable. You could use $path; variable names are case sensitive. Or you could use $dir or any of a myriad other names. You do need to be aware of the key environment variables and avoid clobbering or misusing them. Of the key environment variables, $PATH is one of the most key ($HOME is another; actually, after those two, most of the rest are relatively less important). Conventionally, upper case names are reserved for environment variables; use lower case names for local variables in a script.

bash: defining a file-local variable invisible to sourcing script

Say I have a bash script file config.sh. It's meant to be source'd by other scripts and variables defined is used as customization of the upper-level scripts.
The problem is, if config.sh has a temporary variable and its name conflicts with upper-level scripts' variable, it breaks the upper-level one.
config.sh:
TMP1=abc
CONFIG_INPUT_DIR="$TMP1"/in
CONFIG_OUTPUT_DIR="$TMP1"/out
upper-level script:
TMP1=def
source config.sh
echo $TMP1
The last echo prints abc, not def.
Solution 1
My current solution is to append a random string to the temporary variable name to make it almost impossible to conflict. e.g:
TMP1_vFc9Uiew=abc
CONFIG_INPUT_DIR="$TMP1_vFc9Uiew"/in
CONFIG_OUTPUT_DIR="$TMP1_vFc9Uiew"/out
unset TMP1_vFc9Uiew
which is painful and makes the code hard to read, in addition not to be perfect.
Solution 2 using local keyword
After some searching, I've come to know local keyword.
But when I simply declare TMP1 as local, bash complains that config.sh: line 1: local: can only be used in a function.
So my another solution is to enclose whole config script as a function:
function config_func_rZ0Yqkpm() {
local TMP1=abc
CONFIG_INPUT_DIR="$TMP1"/in
CONFIG_OUTPUT_DIR="$TMP1"/out
}
config_func_rZ0Yqkpm
unset config_func_rZ0Yqkpm
which is better than previous solution in maintainability and readability, but there's some possibility to conflict as well as solution 1.
Question
I want to know more robust and smart solution without any possibility to conflict.
Thanks.
A trick I learned from the keychain utility is using one program to build a source-able file containing just the variables that you want to export from your program. You could modify your script to echo the variables you want to set and then source the output from your program:
$ echo $FOO
$ source <(echo FOO=bar)
$ echo $FOO
bar
$
I used echo FOO=bar to simulate the larger script; your program is probably more involved. The important part is that you must modify your program to output the variables and values you would like to set, rather than just setting them. This lets you decide which variables to expose and which ones to hold private at the cost of another shell process.
You could avoid variables and use functions in config.sh to hold your values:
get_dirname() { echo "abc"; }
CONFIG_INPUT_DIR="$(get_dirname)/in"
CONFIG_OUTPUT_DIR="$(get_dirname)/out"
unset -f get_dirname
If you're still concerned about name collision for functions, this doesn't really help you.
The "ssh-agent" method:
config.sh
#!/bin/bash
TMP=abc
printf "CONFIG_INPUT_DIR=%s/in\n" "$TMP"
printf "CONFIG_OUTPUT_DIR=%s/out\n" "$TMP"
main program:
TMP1=def
eval "$(config.sh)"
echo "$TMP1"

Resources