Out of curiosity I called which myscript.sh for a script I have under ~/bin/myscript.sh (in macOS's bash). I can execute the script with no problems from any directory without giving its full path. ~/bin is on my PATH and the script has executable flags set, ls -l outputs -rwxr-xr-x.
I would have expected which to show me the script's full path, but it didn't output anything.
Is this intended behaviour or is something odd happening here?
This happens when you add a literal tilde ~ to your PATH, instead of the actual path to your home directory. Rewriting ~ into /home/youruser is the job of the shell, not of the tool or filesystem. If you e.g. quote the "~", this rewrite doesn't happen and other tools like which get confused.
Here's information on this issue from the shellcheck wiki:
Literal tilde in PATH works poorly across programs.
Problematic code:
PATH="$PATH:~/bin"
Correct code:
PATH="$PATH:$HOME/bin"
Rationale:
Having literal ~ in PATH is a bad idea. Bash handles it, but nothing else does.
This means that even if you're always using Bash, you should avoid it because any invoked program that relies on PATH will effectively ignore those entries.
For example, make may say foo: Command not found even though foo works fine from the shell and Make and Bash both use the same PATH. You'll get similar messages from any non-bash scripts invoked, and whereis will come up empty.
Use $HOME or full path instead.
Related
I am attempting to work with an existing library of code but have encountered an issue. In short, I execute a shell script (let's call this one A) whose first act is to call another script (B). Script B is in my current directory (a requirement of the program I'm using). The software's manual makes reference to bash, however comments in A suggest it was developed in ksh. I've been operating in bash so far.
Inside A, the line to execute B is simply:
. B
It uses the "dot space" syntax to call the program. It doesn't do anything unusual like sudo.
When I call A without dot space syntax, i.e.:
./A
it always errors saying it cannot find the file B. I added pwd, ls, whoami, echo $SHELL, and echo $PATH lines to A to debug and confirmed that B is in fact right there, the script is running with the same $SHELL as I am at the command prompt, the script is the same user as I am, and the script has the same search path $PATH as I do. I also verified if I do:
. B
at the command line, it works just fine. But, if I change the syntax inside A to:
./B
instead, then A executes successfully.
Similarly, if I execute A with dot space syntax, then both . B and ./B work.
Summarizing:
./A only works if A contains ./B syntax.
. A works for A with either ./B or . B syntax.
I understand that using dot space (i.e. . A) syntax executes without forking to a subshell, but I don't see how this could result in the behavior I'm observing given that the file is clearly right there. Is there something I'm missing about the nuances of syntax or parent/child process workspaces? Magic?
UPDATE1: Added info indicating that the script may have been developed in ksh, while I'm using bash.
UPDATE2: Added checking to verify $PATH is the same.
UPDATE3: The script says it was written for ksh, but it is running in bash. In response to Kenster's answer, I found that running bash -posix then . B fails at the command line. That indicates that the difference in environments between the command line and the script is that the latter is running bash in a POSIX-compliant mode, whereas the command line is not. Looking a little closer, I see this in the bash man page:
When invoked as sh, bash enters posix mode after the startup files are read.
The shebang for A is indeed #!/bin/sh.
In summary, when I run A without dot space syntax, it's forking to its own subshell, which is in POSIX-compliant mode because the shebang is #!/bin/sh (instead of, e.g., #!/bin/bash. This is the critical difference between the command line and script runtime environments that leads to A being unable to find B.
Let's start with how the command path works and when it's used. When you run a command like:
ls /tmp
The ls here doesn't contain a / character, so the shell searches the directories in your command path (the value of the PATH environment variable) for a file named ls. If it finds one, it executes that file. In the case of ls, it's usually in /bin or /usr/bin, and both of those directories are typically in your path.
When you issue a command with a / in the command word:
/bin/ls /tmp
The shell doesn't search the command path. It looks specifically for the file /bin/ls and executes that.
Running ./A is an example of running a command with a / in its name. The shell doesn't search the command path; it looks specifically for the file named ./A and executes that. "." is shorthand for your current working directory, so ./A refers to a file that ought to be in your current working directory. If the file exists, it's run like any other command. For example:
cd /bin
./ls
would work to run /bin/ls.
Running . A is an example of sourcing a file. The file being sourced must be a text file containing shell commands. It is executed by the current shell, without starting a new process. The file to be sourced is found in the same way that commands are found. If the name of the file contains a /, then the shell reads the specific file that you named. If the name of the file doesn't contain a /, then the shell looks for it in the command path.
. A # Looks for A using the command path, so might source /bin/A for example
. ./A # Specifically sources ./A
So, your script tries to execute . B and fails claiming that B doesn't exist, even though there's a file named B right there in your current directory. As discussed above, the shell would have searched your command path for B because B didn't contain any / characters. When searching for a command, the shell doesn't automatically search the current directory. It only searches the current directory if that directory is part of the command path.
In short, . B is probably failing because you don't have "." (current directory) in your command path, and the script which is trying to source B is assuming that "." is part of your path. In my opinion, this is a bug in the script. Lots of people run without "." in their path, and the script shouldn't depend on that.
Edit:
You say the script uses ksh, while you are using bash. Ksh follows the POSIX standard--actually, KSH was the basis for the POSIX standard--and always searches the command path as I described. Bash has a flag called "POSIX mode" which controls how strictly it follows the POSIX standard. When not in POSIX mode--which is how people generally use it--bash will check the current directory for the file to be sourced if it doesn't find the file in the command path.
If you were to run bash -posix and run . B within that bash instance, you should find that it won't work.
Consider the following folder structure:
$ tree ~/test_path
test_path
`-- sub_folder
`-- script.sh
1 directory, 1 file
Let's say that you have added test_path to your path by
export PATH=$PATH:~/test_path
$ whereis sub_folder
sub_folder: /home/murtraja/test_path/sub_folder
Now how to execute script.sh by calling sub_folder/script.sh?
$ sub_folder/script.sh
bash: sub_folder/script.sh: No such file or directory
EDIT: I don't want to change the call sub_folder/script.sh because this is called by another script which I cannot (am avoiding to) change.
Short answer: You can't, but depending on the set of constraints you're facing, there might be another way to handle it.
Long answer: When a command name contains at least one slash character, it treated as a path to the executable (i.e. it doesn't search the directories in $PATH). With the command name sub_folder/script.sh, it contains a slash, but doesn't start with a slash, so it'll be resolved relative to the current working directory.
So there are a couple of possibilities for making this work:
If you can cd to ~/test_path before running this, it'll find it directly. Of course, this may break other things (i.e. anything else that uses relative paths and/or plain filenames and expects them to be resolved somewhere else). Also, be sure to check for errors when you cd, or the script could execute in an unexpected directory, with unexpected consequences.
If the script needs to execute from a different working directory, you might be able to create a symbolic link from sub_folder in that working directory to ~/test_path/sub_folder. But depending on where the script's working directory is, this may be impossible or unsafe. I'd avoid using this option if possible.
There's also an option that depends on a weird/nonstandard feature of bash: the ability to define function names with slash in them. But this has weird limitations depending on the version of bash you have:
You can define a function like this:
sub_folder/script.sh() { ~/test_path/sub_folder/script.sh "$#"; }
and then either use export -f sub_folder/script.sh (so bash subprocesses inherit it), or do this in a wrapper script and then source the script you can't change from there (so it's the same shell, and inheritance isn't necessary).
Difficulty: some versions of bash refuse to export functions with weird names, and some refuse to inherit them. So the export method might or might not work (and might break unexpectedly due to an update). The source method might be better, but also might cause other trouble.
If there's any way at all to change the other script, that'd really be the best option.
Since you've added it to your path, you can just call the script by name. It should also let you tab complete as well.
$ ./script.sh
I am trying to execute a shell script for automating the process rather than manually running the python script. But i am getting the error folder not found.
cd /home/gaurav/AndroPyTool
export ANDROID_HOME=$HOME/android-sdk-linux/
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/platform-tools
source ~/.bashrc
source droidbox_env/bin/activate
alias mycd='cd /home/gaurav/AndroPyTool/test'
mycd
pwd
python androPyTool.py -all -csv EXPORTCSV.csv -s mycd
>>>> AndroPyTool -- STEP 1: Filtering apks
Folder not found!
This is the error i am getting because the script is not able to find the path that i have provided above.
The part after "-s" in the code represents the folder path where the file stored.
The issue here is that you are not passing the path to the python program. The python program is not aware of bash aliases and bash will only expand aliases when it is interpreting the token as a command.
When bash reads python androPyTool.py -all -csv EXPORTCSV.csv -s mycd It interprets python as the command and all other space separated tokens are arguments that will be passed into python. Python then invokes androPyTool.py and passes the subsequent arguments to that script. So the program is receiving literally mycd as the argument for -s.
Moreover, even if mycd is expanded, it wouldn't be the correct argument for -s. androPyTool.py is expecting just the /path/to/apks, not cd /path/to/apks/.
I don't really think that using the alias in this script makes much sense. It actually makes the script harder to read and understand. If you want to wrap a command, I recommend defining a function, and occasionally you can use variable expansion (but that mixes code and data which can lead to issues). EDIT: As has been pointed out in the comments, aliases are disabled for scripts.
Finally there are some other suspicious issues with your script. Mainly, why are you sourcing .bashrc? If this script is run by you in your user's environment, .bashrc will already be sourced and there is no need to re-source it. On the other hand, if this is not intended to be run in your environment, and there is something in the .bashrc file that you need in your script, I recommend pulling just that out and nothing else.
But the most immediate issue that I can see is that sourcing .bashrc after you modify path runs the risk of overwriting the changes to PATH you just made. Depending on the contents of the .bashrc file, sourcing it may not be idempotent, meaning that running it more than once could have side effects. Finally, anything could get thrown in that .bashrc file down the road since that's what its for. So now your script may depend on something that likely will be changing. This opens up the possibility that bugs will creep in to your script unexpectedly.
For a long time I was under impression that *NIX shells do not support running executables via path relative to current directory. Also there're a number of posts explaining why having working directory in path is a bad practice. Still it's easy to see that following example works:
% cat<<EOF > temp/a
? #!/bin/sh
? echo "Hello looser!"
? EOF
% chmod 750 temp/a
% temp/a
Hello looser!
(I tested at CentOS and OSX).
Why it works?
Upd: it's not purely academic question, I had cases when binaries run via full path and the way described above worked different way.
Upd 2: libc execv accepts both absolute and relative paths with no problem. Hence, support for subpath/exe while failing to run exe looks as shell feature common to bash and tcsh. There must be some logic behind it??
You can always execute a program by specifying its relative path. There is no requirement that you would have to use an absolute path. Actually, the way you are executing your program, is quite common.
If you write command with no / then the shell searches $PATH for the command.
If you write dir/command with a relative directory, there's a / and the shell does not search $PATH. It interprets it relative to the current directory. dir/command always resolves to ./dir/command.
Putting . in the $PATH is dangerous because you may type command intend ing to run /usr/bin/command but actually get ./command if it exists in the current directory. Indeed, it's not just . that's dangerous: any relative path is dangerous. $PATH should only ever contain absolute paths.
So, you're saying that relative paths shall be implicitly prefixed with current directory, with exceptional case when relative path is empty that shall be treated as direction to search in $PATH?
Yes, you could say that. But I prefer to reframe it. A better way to think of it is to understand what the shell is doing versus what the kernel is doing.
Interpreting relative paths is handled implicitly by the kernel. When the shell performs a system call like execve("dir/command", ...) to execute a program the kernel knows the parent process's current directory and resolves dir/command relative to it. The shell does not have to first convert the path into an absolute one. The kernel can handle both absolute and relative paths.
The kernel knows nothing about $PATH, though. That's a shell construct. Keep in mind that the shell is a higher level piece of software. The shell decides that if you type a simple command name without a / in it, it will not simply pass that command to the kernel. If it did, typing cat would simply execute ./cat. Nobody wants that.
Instead, the shell decides that the lack of a / means it will search for the command in the $PATH. It'll search /bin, /usr/bin, $HOME/bin, etc. If you listed . in the $PATH then it would also search the current directory—don't do that! You don't want it to run ./cat. No sir.
If the shell finds an executable in the $PATH then it converts it to an absolute path name and passes that to the kernel. The kernel sees execve("/usr/bin/cat", ...).
If so and you know it from some formal specification, I'd appreciate the reference. I do need to know precisely.
See the bash man page:
If the command name contains no slashes, the shell attempts to locate it. …
If the name … contains no slashes, bash searches each element of the PATH for a directory containing an executable file by that name. …
If the search is successful, or if the command name contains one or more slashes, the shell executes the named program ….
This as nothing to do with execution. It is due to the fact that you can name a file with an absolute or a relative name, and that executables are just ordinary files.
Why is it dangerous? (.in PATH)
Think about a script that calls a command like cat foo, if you have . in the path then it could be possible to create a command cat in the same directory that would be executed in place of the original one.
Yes, I know that I can run
. my_cd_script.sh
to change my directory directly. However, once I do that, my $PATH is messed up. For instance, when I type ls, the shell will return Command not found.
Anyone encountered this?
I named a variable "path" without a second thought, although I would've expected shell to be case sensitive. – user1836155
If you're running into variable names not seeming to be case-sensitive, then I suspect you're not actually using bash. Maybe csh instead, or some other variant in the csh family... – twalberg
I used the "#!/bin/bash" header though – user1836155
The header means nothing when you source a file with . myscript - it's just a comment in that case. – twalberg