I have been working on a few scripts on CentOS 7 and sometimes I see:
#!/bin/sh -
on the first line. Looking at the man page for sh I see the following under the Special Parameters
- Expands to the current option flags as specified upon invocation,
by the set builtin command, or those set by the shell
itself (such as the -i option).
What exactly does this mean? When do I need to use this special parameter option??
The documentation you are reading has nothing to do with the command line you're looking at: it's referring to special variables. In this case, if you run echo $- you will see "the current option flags as specified upon invocation...".
If you take a look at the OPTIONS part of the bash man page, you will find:
-- A -- signals the end of options and disables further option processing.
Any arguments after the -- are treated as filenames and arguments. An
argument of - is equivalent to --.
In other words, an argument of - simply means "there are no other options after this argument".
You often see this used in situation in which you want to avoid filenames starting with - accidentally being treated as command options: for example, if there is a file named -R in your current directory, running ls * will in fact behave as ls -R and produce a recursive listing, while ls -- * will not treat the -R file specially.
The single dash when used in the #! line is meant as a security precaution. You can read more about that here.
/bin/sh is an executable representing the system shell. Actually, it is usually implemented as a symbolic link pointing to the executable for whichever shell is the system shell. The system shell is kind of the default shell that system scripts should use. In Linux distributions, for a long time this was usually a symbolic link to bash, so much so that it has become somewhat of a convention to always link /bin/sh to bash or a bash-compatible shell. However, in the last couple of years Debian (and Ubuntu) decided to switch the system shell from bash to dash - a similar shell - breaking with a long tradition in Linux (well, GNU) of using bash for /bin/sh. Dash is seen as a lighter, and much faster, shell which can be beneficial to boot speed (and other things that require a lot of shell scripts, like package installation scripts).
Dash is fairly well compatible with bash, being based on the same POSIX standard. However, it doesn't implement the bash-specific extensions. There are scripts in existence that use #!/bin/sh (the system shell) as their shebang, but which require bash-specific extensions. This is currently considered a bug that should be fixed by Debian and Ubuntu, who require /bin/sh to be able to work when pointed to dash.
Even though Ubuntu's system shell is pointing to dash, your login shell as a user continues to be bash at this time. That is, when you log in to a terminal emulator anywhere in Linux, your login shell will be bash. Speed of operation is not so much a problem when the shell is used interactively, and users are familiar with bash (and may have bash-specific customization in their home directory).
Related
Is there a difference in behaviours with respect to the flags passed to the script on the shebang line vs. using the set builtin?
For example:
#!/bin/bash -e
# do stuff
vs.
#!/bin/bash
set -e
# do stuff
(The question is not specific to -e flag but in general for any such flags).
Obviously the set [flags] is effective only from the point it's set. But are there any other differences in functionality/behaviour?
Is behaviour same in a POSIX shell, too?
are there any other differences in functionality/behaviour?
When your file has executable permissions and is executed, then the shebang line is parsed by the kernel.
When your file is executed under the shell like bash ./script.sh then the shebang is just a comment. So it will be ignored, and your script will be run with whatever the callers flags are. Putting your flags after the shebang will make sure proper flags are set in your scripts in either case.
The shebang is parsed by kernel. That basically means that the behavior differs from kernel to kernel, from operating system to operating system. Some operating systems didn't handle arguments in shebang at all and ignored all the arguments. Some kernels parse for example #!/bin/sh -a -b as execl("/bin/sh", "-a -b") some as execl("/bin/sh", "-a", "-b"). The parsing of the shebang line to executable and arguments is done by some other code different that your shell. Sometimes if there is a space after #! like #! /bin/sh utilities don't recognize it as a valid shebang. There's even a recent linux kernel regression with too long shebang line.
The behavior of how shebang is interpreted differs between systems, so you can't be certain, so it's best to set options after the shebang.
Is behaviour same in a POSIX shell, too?
A POSIX shell doesn't (have to) interpret your shebang. If you are asking if executing sh -e and set -e has the same behavior in posix shell, then yes, the option -e on command line has the same behavior as set -e.
I couldn't find a specification of shebang line nor how should it be interpreted in posix specification. I can see in execve documentation:
Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute.
Those "historical implementations" seem to be very widely used still today.
The shebang line is parsed by the kernel after exec* calls. But when you are doing sh <script> or popen or system the shell can (but doesn't have to) interpret the shebang line by itself as a extension and not rely on kernel implementation, from posix:
Shell Introduction
The shell reads its input from a file (see sh), from the −c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-200x. If the first line of a file of shell commands starts with the characters"#!",the results are unspecified.
As for bash, it looks like bash first tries execve, then if it can't find the reason why kernel failed to run the executable, if the file has a shebang, then it parses shebang on its own to find out the interpreter.
Some operating systems may disallow or limit the use of arguments in the shebang. No such limit would apply to a set command in the script itself.
The manpage states that "-e" has different behaviour depending on POSIX mode or not.
Other than that, I have never noticed a difference between setting parameters and extending shebang.
Why are you asking this question?
I was writing some code, navigating my computer (OSX 10.11.6) via the command line, like I always do, and I made a typo! Instead of typing:
cd USB
I typed
Cd USB
Nothing happened, but it didn't register as an invalid command. Perplexed by this, I did some investigating: I checked the man entry. There was no entry. I found the source file (/usr/bin/Cd) using which Cd, and then cated it:
#!/bin/sh
# $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $
# This file is in the public domain.
builtin `echo ${0##*/} | tr \[:upper:] \[:lower:]` ${1+"$#"}
What is this, and why is it here? How does it relate to freeBSD?
Any help would be amazing, thanks!
macOS uses a case-insensitive filesystem by default[1]
, which can be misleading at times:
which Cd is effectively the same as which cd and which CD in terms of returning the (effectively) same file path.
Confusingly, even though all 3 command refer to the same file, they do so in a case-preserving manner, misleadingly suggesting that the actual case of the filename is whatever you specified.
As a workaround, you can see the true case of the filename if you employ globbing (filename expansion):
$ ls "$(which Cd)"* # could match additional files, but the one of interest is among them
/usr/bin/cd # true case of the filename
Bash (the macOS default shell) is internally case-sensitive.
That is, it recognizes cd as builtin cd (its built-in directory-changing command).
By contrast, it does NOT recognize Cd as that, due to the difference in case.
Given that it doesn't recognize Cd as a builtin, it goes looking for an external utility (in the $PATH), and that is when it finds /usr/bin/cd.
/usr/bin/cd is implemented as a shell script, which is mostly useless, because as an external utility it cannot affect the shell's state, so its attempts to change the directory are simply quietly ignored.
(Keith Thompson points out in a comment that you can use it as test whether a given directory can be changed to, because the script's exit code will reflect that).
Matt's answer provides history behind the inclusion of the script in FreeBSD and OSX (which mostly builds on FreeBSD), but it's worth taking a closer look at the rationale (emphasis mine):
From the POSIX spec:
However, all of the standard utilities, including the regular built-ins in the table, but not the special built-ins described in Special Built-In Utilities, shall be implemented in a manner so that they can be accessed via the exec family of functions as defined in the System Interfaces volume of POSIX.1-2008 and can be invoked directly by those standard utilities that require it (env, find, nice, nohup, time, xargs).
In essence, the above means: regular built-ins must (also) be callable stand-alone, as executables (whether as scripts or binaries), nut just as built-ins from within the shell.
The cited regular built-ins table comprises these utilities:
alias bg cd command false fc fg getopts jobs kill newgrp pwd read true umask unalias wait
Note: special built-in utilities are by definition shell-internal only, and their behavior differs from regular built-in utilities.
As such, to be formally POSIX-compliant an OS must indeed provide cd as an external utility.
At the same time, the POSIX spec. does have awareness that at least some of these regular built-ins - notably cd - only makes sense as a built-in:
"Since cd affects the current shell execution environment, it is always provided as a shell regular built-in." - http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cd.html
Among the regular built-in utilities listed, some make sense both as a built-in and as an external utility:
For instance kill needs to be a built-in in order to kill jobs (which are a shell-internal concept), but it is also useful as an external utility, so as to kill processes by PID.
However, among the regular built-in utilities listed, the following never make sense as external utilities, as far as I can tell Do tell me if you disagree
, even though POSIX mandates their presence:
alias bg cd command fc fg getopts jobs read umask unalias
Tip of the hat to Matt for helping to complete the list; he also points that the hash built-in, even though it's not a POSIX utility, also has a pointless script implementation.
[1] As Dave Newton points out in a comment, it is possible to format HFS+, the macOS filesystem, in a case-sensitive manner (even though most people stick with the case-insensitive default). Based on the answer Dave links to, the following command will tell you whether your macOS filesystem is case-insensitive or not:
diskutil info / | grep -iq '^\s*Name.*case-sensitive*' && echo "case-SENSITIVE" || echo "case-INsensitive"
What is this?
The script itself is a portable way to convert a command, even with random upper casing, into the equivalent shell builtin based on the exec paths file name, that is any part of the string after the final / in the $0 variable). The script then runs the builtin command with the same arguments.
As OSX file systems are case insensitive by default, /usr/bin/cd converts running Cd, CD, cD and any form of cd with a / fs path (like /usr/bin/cd) back to the shell builtin command cd. This is largely useless in a script as cd only affects the current shell it is running in, which immediately closes when the script ends.
How does it relate to freeBSD?
A similar file exists in FreeBSD, which Apple adapted to do case conversion. Mac file systems by default are case insensitive (but case preserving).
The $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $ header is the source information in the file.
Most of the underlying OSX system comes directly from FreeBSD or was based on it. The Windowing system on top of this and the Cocoa app layer is where OSX becomes truly Apple. Some of the lower level Apple bits have even made it back into FreeBSD like Clang and LLVM compiler.
Why is it here?
The earlier FreeBSD svn commits shed a bit of light:
A little bit more thought has resulted in a generic script which can
implement any of the useless POSIX-required ``regular shell builtin''
utilities...
Although most builtins aren't very useful when run in a new shell via a script, this compliance script was used for the commands alias bg cd command fc fg getopts hash jobs read type ulimit umask unalias wait. POSIX compliance is fun!
As I recall, MacOS uses a case-insensitive file system by default. The command you saw as /usr/bin/Cd is actually /usr/bin/cd, but it can be referred to by either name.
You can see this by typing
ls /usr/bin/ | grep -i cd
Normally cd is a builtin command in the shell. As you know, it changes the current directory. An external cd command is nearly useless -- but it still exists.
It can be used to detect whether it's possible to change to a specified directory without actually affecting the working directory of your current process.
Your shell (probably bash) tends to assume case-sensitive command names. The builtin command can only be referred to as cd, but since it's able to open the script file named /usr/bin/Cd, it can find and execute it.
my scripts are developed using a shebang line as "#!/bin/ksh" and the default shell is
$ echo $SHELL
/bin/ksh
i am moving all these scripts without changing the shebang line to a new machine where the default shell is
$ echo $SHELL
/bin/bash
Should i worry about this ?
I am guessing there should not be any issue as the shebang line will override the interpreter and use ksh as defined in the scripts and as i want it to be.
Please share your thoughts ..
The default shell does not affect how scripts are executed (unless you're using a shell that does something very strange).
An executable script with no #! line will be executed with /bin/sh. Actually that doesn't appear to be correct, but in any case you don't have to worry about that.
As long as your scripts start with #!/bin/ksh and you execute them normally, the system will execute them by passing them to /bin/ksh.
One thing you might have to worry about is whether /bin/ksh exists, and if it does, just what it is. On my system (Linux Mint 17), /bin/ksh is a symlink to /etc/alternatives/ksh, which in turn is a symlink to /bin/ksh93.
Scripts with #!/bin/ksh are probably common enough that almost all UNIX-like systems will cater to them, and will install something that behaves like ksh at that location.
Note that what you call the "default shell", specified by $SHELL, is not a system-wide default. It's just the value of a particular environment variable. That variable is set for each user on login based on the shell specified in /etc/passwd or equivalent; thus different users can have different default shells. You can change the value of $SHELL after logging in. The entry in /etc/passwd or equivalent is set when the account is created, and can be changed later. Most systems have a default user shell that's set for new accounts if no shell is specified (for example, most Linux systems user /bin/bash).
The supposition given is correct: The shebang line is honored on any execve() call. Only if your scripts are sourced (. yourscript or source yourscript) or lack a valid shebang do you need to care which interpreter they're called from.
If this were not true, scripts in non-shell languages wouldn't work as expected (as the Python interpreter, for instance, is never a system's default shell).
The kernel will use the shebang line to select the appropriate interpreter to use when the script is executed in the default manner, whether it is sh, bash, ksh, expect, python, or whatever. The only real issue to be wary of is scripts written on a system where sh is one specific shell (e.g. bash) that are then moved to another system where sh is a different shell (e.g. dash) since they may use shell features found in the former that do not exist in the latter.
I am using Bash
$ echo $SHELL
/bin/bash
and starting about a year ago I stopped using Shebangs with my Bash scripts. Can
I benefit from using #!/bin/sh or #!/bin/bash?
Update: In certain situations a file is only treated as a script with the
Shebang, example
$ cat foo.sh
ls
$ cat bar.sh
#!/bin/sh
ls
$ file foo.sh bar.sh
foo.sh: ASCII text
bar.sh: POSIX shell script, ASCII text executable
On UNIX-like systems, you should always start scripts with a shebang line. The system call execve (which is responsible for starting programs) relies on an executable having either an executable header or a shebang line.
From FreeBSD's execve manual page:
The execve() system call transforms the calling process into a new
process. The new process is constructed from an ordinary file, whose
name is pointed to by path, called the new process file.
[...]
This file is
either an executable object file, or a file of data for an interpreter.
[...]
An interpreter file begins with a line of the form:
#! interpreter [arg]
When an interpreter file is execve'd, the system actually execve's the
specified interpreter. If the optional arg is specified, it becomes the
first argument to the interpreter, and the name of the originally
execve'd file becomes the second argument
Similarly from the Linux manual page:
execve() executes the program pointed to by filename. filename must be
either a binary executable, or a script starting with a line of the
form:
#! interpreter [optional-arg]
In fact, if a file doesn't have the right "magic number" in it's header, (like an ELF header or #!), execve will fail with the ENOEXEC error (again from FreeBSD's execve manpage):
[ENOEXEC] The new process file has the appropriate access
permission, but has an invalid magic number in its
header.
If the file has executable permissions, but no shebang line but does seem to be a text file, the behaviour depends on the shell that you're running in.
Most shells seem to start a new instance of themselves and feed it the file, see below.
Since there is no guarantee that the script was actually written for that shell, this can work or fail spectacularly.
From tcsh(1):
On systems which do not understand the `#!' script interpreter conven‐
tion the shell may be compiled to emulate it; see the version shell
variable. If so, the shell checks the first line of the file to see if
it is of the form `#!interpreter arg ...'. If it is, the shell starts
interpreter with the given args and feeds the file to it on standard
input.
From FreeBSD's sh(1):
If the program is not a normal executable file (i.e., if it
does not begin with the “magic number” whose ASCII representation is
“#!”, resulting in an ENOEXEC return value from execve(2)) but appears to
be a text file, the shell will run a new instance of sh to interpret it.
From bash(1):
If this execution fails because the file is not in executable format,
and the file is not a directory, it is assumed to be a shell script, a
file containing shell commands. A subshell is spawned to execute it.
You cannot always depend on the location of a non-standard program like bash. I've seen bash in /usr/bin, /usr/local/bin, /opt/fsf/bin and /opt/gnu/bin to name a few.
So it is generally a good idea to use env;
#!/usr/bin/env bash
If you want your script to be portable, use sh instead of bash.
#!/bin/sh
While standards like POSIX do not guarantee the absolute paths of standard utilities, most UNIX-like systems seem to have sh in /bin and env in /usr/bin.
Scripts should always begin with a shebang line. If a script doesn't start with this, then it may be executed by the current shell. But that means that if someone who uses your script is running a different shell than you do, the script may behave differently. Also, it means the script can't be run directly from a program (e.g. the C exec() system call, or find -exec), it has to be run from a shell.
You might be interested in an early description by Dennis M Ritchie (dmr) who invented the #! :
From uucp Thu Jan 10 01:37:58 1980
.>From dmr Thu Jan 10 04:25:49 1980 remote from research
The system has been changed so that if a file
being executed begins with the magic characters #! , the rest of the
line is understood to be the name of an interpreter for the executed
file. Previously (and in fact still) the shell did much of this job;
it automatically executed itself on a text file with executable mode
when the text file's name was typed as a command. Putting the facility
into the system gives the following benefits.
1) It makes shell scripts more like real executable files, because
they can be the subject of 'exec.'
2) If you do a 'ps' while such a command is running, its real name
appears instead of 'sh'. Likewise, accounting is done on the basis of
the real name.
3) Shell scripts can be set-user-ID.
4) It is simpler to have alternate shells available; e.g. if you like
the Berkeley csh there is no question about which shell is to
interpret a file.
5) It will allow other interpreters to fit in more smoothly.
To take advantage of this wonderful opportunity, put
#! /bin/sh
at the left margin of the first line of your shell scripts. Blanks
after ! are OK. Use a complete pathname (no search is done). At the
moment the whole line is restricted to 16 characters but this limit
will be raised.
Hope this helps
If you write bash scripts, i.e. non portable scripts containing bashisms, you should keep using the #!/bin/bash shebang just to be sure the correct interpreter is used. You should not replace the shebang by #!/bin/sh as bash will run in POSIX mode so some of your scripts might behave differently.
If you write portable scripts, i.e. scripts only using POSIX utilities and their supported options, you might keep using #!/bin/sh on your system (i.e. one where /bin/sh is a POSIX shell).
It you write stricly conforming POSIX scripts to be distributed in various platforms and you are sure they will only be launched from a POSIX conforming system, you might and probably should remove the shebang as stated in the POSIX standard:
As it stands, a strictly conforming application must not use "#!" as the first two characters of the file.
The rationale is the POSIX standard doesn't mandate /bin/sh to be the POSIX compliant shell so there is no portable way to specify its path in a shebang. In this third case, to be able to use the 'find -exec' syntax on systems unable to run a shebangless still executable script, you can simply specify the interpreter in the find command itself, eg:
find /tmp -name "*.foo" -exec sh -c 'myscript "$#"' sh {} +
Here, as sh is specified without a path, the POSIX shell will be run.
The header is useful since it specifies which shell to use when running the script. For example, #!/bin/zsh would change the shell to zsh instead of bash, where you can use different commands.
For example, this page specifies the following:
Using #!/bin/sh, the default Bourne shell in most commercial variants
of UNIX, makes the script portable to non-Linux machines, though you
sacrifice Bash-specific features ...
TL;DR: always in scripts; please not in source'd scripts
Always in your parent
FYI: POSIX compliant is #!/bin/bash, not #!/bin/sh
You want to clarify this so that nothing else overrides the interpreter your script is made for.
You don't want a user at the terminal using zsh to have trouble if your script was written for POSIX bash scripts.
You don't want to run source in your #!/bin/bash unrecognized by #!/bin/sh, someone in an sh terminal have it break the script because it is expecting the simple/POSIX . for including source'd files
You don't want e.g. zsh features - not available in other interpreters - to make their way into your bash code. So, put #!/bin/bash in all your script headers. Then, any of your zsh habits in your script will break so you know to remove them before your roll-out.
It's probably best, especially so POSIX-compliant scripts don't break in a terminal like zsh.
Not expected for included source scripts
FYI: POSIX compliant for sourcing text in a BASH script is ., not source
You can use either for sourcing, but I'll do POSIX.
Standard "shebanging" for all scripting:
parent.sh:
#!/bin/bash
echo "My script here"
. sourced.sh # child/source script, below
sourced.sh:
echo "I am a sourced child script"
But, you are allowed to do this...
sourced.sh: (optional)
#!/bin/bash
echo "I am a sourced child script"
There, the #!/bin/bash "shebang" will be ignored. The main reason I would use it is for syntax highlighting in my text editor. However, in the "proper" scripting world, it is expected that your rolled-out source'd script will not contain the shebang.
In addition to what the others said, the shebang also enables syntax highlighting in some text editors, for example vim.
$SHELL and #!/bin/bash or #!/bin/sh are different.
To start, #!/bin/sh is a symlink to /bin/bash on most Linux systems (on Ubuntu it is now /bin/dash)
But on whether to start with /bin/sh or /bin/bash:
Bash and sh are two different shells. Basically bash is sh, with more
features and better syntax. Most commands work the same, but they are
different.
Just assume if you're writing a bash script, stick with /bin/bash and not /sh because problems can arise.
$SHELL does not necessarily reflect the currently running shell.
Instead, $SHELL is the user's preferred shell, which is typically the
one set in /etc/passwd. If you start a different shell after logging
in, you can not necessarily expect $SHELL to match the current shell
anymore.
This is mine for example, but it could also be /root:/bin/dash or /root:/bin/sh depending on which shell you have input in passwd. So to avoid any problems, keep the passwd file at /bin/bash and then using $SHELL vs. #!/bin/bash wouldn't matter as much.
root#kali:~/Desktop# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
Sources:
http://shebang.mintern.net/bourne-is-not-bash-or-read-echo-and-backslash/
https://unix.stackexchange.com/questions/43499/difference-between-echo-shell-and-which-bash
http://man.cx/sh
http://man.cx/bash
I make a complex and long line command to successful login in a site. If I execute it in Console it work. But if I copy and paste the same line in a bash script it not work.
I tried a lot of thing, but accidentally discovery that if I NOT use the line
#!/bin/sh
it work! Why this happens in my mac OSX Lion? What this config line do in a bash script?
A bash script that is run via /bin/sh runs in sh compatibility mode, which means that many bash-specific features (herestrings, process substitution, etc.) will not work.
sh-4.2$ cat < <(echo 123)
sh: syntax error near unexpected token `<'
If you want to be able to use full bash syntax, use #!/bin/bash as your shebang line.
"#!/bin/sh" is a common idiom to insure that the correct interpreter is used to run the script. Here, "sh" is the "Bourne Shell". A good, standard "least common denominator" for shell scripts.
In your case, however, "#!/bin/sh" seems to be the wrong interpreter.
Here's a bit more info:
http://www.unix.com/answers-frequently-asked-questions/7077-what-does-usr-bin-ksh-mean.html
Originally, we only had one shell on unix. When you asked to run a
command, the shell would attempt to invoke one of the exec() system
calls on it. It the command was an executable, the exec would succeed
and the command would run. If the exec() failed, the shell would not
give up, instead it would try to interpet the command file as if it
were a shell script.
Then unix got more shells and the situation became confused. Most
folks would write scripts in one shell and type commands in another.
And each shell had differing rules for feeding scripts to an
interpreter.
This is when the "#! /" trick was invented. The idea was to let the
kernel's exec() system calls succeed with shell scripts. When the
kernel tries to exec() a file, it looks at the first 4 bytes which
represent an integer called a magic number. This tells the kernel if
it should try to run the file or not. So "#! /" was added to magic
numbers that the kernel knows and it was extended to actually be able
to run shell scripts by itself. But some people could not type "#! /",
they kept leaving the space out. So the kernel was exended a bit again
to allow "#!/" to work as a special 3 byte magic number.
So #! /usr/bin/ksh and
#!/usr/bin/ksh now mean the same thing. I always use the former since at least some kernels might still exist that don't understand the
latter.
And note that the first line is a signal to the kernel, and not to the
shell. What happens now is that when shells try to run scripts via
exec() they just succeed. And we never stumble on their various
fallback schemes.
The very first line of the script can be used to select which script interpreter to use.
With
#!/bin/bash
You are telling the shell to invoke /bin/bash interpreter to execute your script.
Assure that there are not spaces or empty lines before #!/bin/bash or it will not work.