My default shell is bash in Ubuntu 14.04. I have a csh script file named clean.sh with the following make command:
#! /bin/csh -f
make -f commande.make del
And commande.make has
CKHOME=../CHEMKIN/DATA_BASES
LIN_DATA=${CKHOME}/LIN_FILES/
LINK_CKTP=${CKHOME}/LINK_CKTP_FILES/
#-----------------------------------------------------
include schema_cinetique.make
LINKFILE=${NAME}_LINK
LINKTPFILE=${NAME}_LINKTP
LINKFILE_OLD=${NAME_OLD}_LINK
LINKFILE_NEW=${NAME_NEW}_LINK
#-----------------------------------------------------
cplink :
${COPY} ${LINK_CKTP}${LINKFILE} LINK
cplink2 :
${COPY} ${LINK_CKTP}${LINKFILE} LINKZ1
tplink :
${COPY} ${LINK_CKTP}${LINKTPFILE} LINKTPZ1
calcul :
${COPY} jobtimp1 LJOBNZ1
${COPY} unsteadyf.dat1 DATZ1
del :
${DELETE} LINKZ1 LINKTPZ1 LJOBNZ1 DATZ1 SOLASUZ1
I opened the terminal and moved to the location and tried
./clean.sh
or
csh clean.sh &
or
csh -f clean.sh
Nothing worked.
I got the following line in the terminal,
LINKZ1 LINKTPZ1 LJOBNZ1 DATZ1 SOLASUZ1
make: LINKZ1: Command not found
make: *** [del] Error 127
So, how to run clean.sh file ?
You are confused. The Csh script contains a single command which actually runs identically in Bash.
#!/bin/bash
make -f commande.make del
Or, for that matter, the same with #!/bin/sh. Or, in this individual case, even sh clean.sh, since the shebang is then just a comment, and the commands in the file are available in sh just as well as in csh.
Once make runs, that is what parses and executes the commands in commande.make. make is not a "Fortran command", it is a utility for building projects (but the makefile named commande.make probably contains some instructions for how to compile Fortran code).
In the general case, Csh and Bash are incompatible, but the differences are in the shell's syntax itself (so, the syntax of loops and conditionals, etc, as well as variable assignments and various other shell builtins).
As an aside, Csh command files should probably not have a .sh extension, as that vaguely implies Bourne shell (sh) syntax. File extensions on Unix are just a hint to human readers, so not technically important; but please don't confuse them/us.
(As a further aside, nobody should be using Csh in 2022. There was a time when the C shell was attractive compared to its competition, but that was on the order of 40 years ago.)
The subsequent errors you are reporting seem to indicate that the makefile depends on some utilities which you have not installed. Figuring that out is a significant enough and separate enough question that you should probably ask a new question about that, probably with more debugging details. But in brief, it seems that make needs to be run with parameters to indicate what NAME and COPY (and probably some other variables) should be. Try with make -f commande.make COPY=cp DELETE=rm NAME=foobar for a start, but it's probably not yet anywhere near sufficient.
(I would actually assume that there will be a README file or similar to actually instruct you how to use commande.make since it seems to have some local conventions of its own.)
It seems the script is written having portability in mind, i.e. the name of the cp and rm binaries is kept in variables rather than hard-coding it. My best guess is that this has been done to make it possible to run the script on non UNIX systems, like Windows.
To make it work, export the respective variables before running the script. For the del action you are calling, only the DELETE variable is needed. It should be set to rm which is the command used to remove files on Linux:
export DELETE=rm
./clean.sh
Note: exporting the variable can also be done in one line when invoking the script, by prepending it to the command line:
DELETE=rm ./clean.sh
This behaviour is described in the bash manual:
The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments, as described in Shell Parameters. These assignment statements affect only the environment seen by that command.
Related
Original Title: Indirect parameter substitution breaks when the script is sourced (zsh)
zsh 5.7.1 (x86_64-apple-darwin19.0)
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
I’m developing a shell script on a Mac and I’m trying to keep it portable between bash & zsh, so array indexing is a consideration. I know that I can set KSH_ARRAYS to get indexing to start at 0, but I decided to query the OS for the shell that’s in use and set the start index accordingly, which led to the issue described below.
It made sense (to me anyway!) to use indirect expansion, which is what led to the problem. Consider the script indirect.sh:
#! /bin/bash
declare -r ARRAY_START_BASH=0
declare -r ARRAY_START_ZSH=1
declare -r SHELL_BASH=0
declare -r SHELL_ZSH=1
# Indirect expansion is used to reference the values of the variables declared
# in this case statement e.g. ${!ARRAY_START}
case $(basename $SHELL) in
"bash" )
declare -r SHELL_ID=SHELL_BASH
declare -r ARRAY_START=ARRAY_START_BASH
;;
"zsh" )
declare -r SHELL_ID=SHELL_ZSH
declare -r ARRAY_START=ARRAY_START_ZSH
;;
* )
return 1
;;
esac
echo "Shell ID: ${!SHELL_ID} Index arrays from: ${!ARRAY_START}"
It works fine when run from the command line while in the same directory:
<my home> ~ % echo "$(./indirect.sh)"
Shell ID: 1 Index arrays from: 1
Problems arise when I source the script:
<my home> ~ % echo "$(. ~/indirect.sh)"
/Users/<me>/indirect.sh:28: bad substitution
I don’t understand why sourcing the script changes the behavior of the parameter expansion.
Is this expected behavior? If so, I’d be grateful if someone could explain it and hopefully, offer a work around.
The problem described in the original post has nothing to do with indirect expansion. The difference in behavior is a result of different shells being invoked depending on whether the script is “executed” or “sourced”. These differences reveal the basic flaw in deriving the shell from the $SHELL variable that underpins the script's design. If the shell defined in $SHELL does not match the shebang, the script will fail either when sourced or executed. An explanation follows.
Indirect expansion doesn’t offer value in the given scenario because values could just as easily be assigned directly. They’ll have to be assigned that way regardless given the different syntax used for indirect expansion between shells. In fact, other syntax differences between shells makes the entire premise for detecting the shell moot! However, putting that aside, the difference in behavior is a result of different shells being invoked based on whether the script is “executed” or “sourced”. The behavior of sourcing is well documented with numerous explanations on the web, but for context here’s how it works:
Executing a Script
Use the “./“ syntax to execute a script.
When run this way, the script executes in a sub-shell. Any changes the
script makes to it’s shell are applied to the sub-shell, not the shell
in which the script was launched, so those changes are lost when the
shell exits because the sub-shell in which it executed is destroyed as
well. For example, if the script changes the working directory, it
does so in the sub-shell. The working directory of the main shell that
launched the script is unchanged when the script terminates. If you
want to make changes to the shell in which the script was launched, it
must be sourced.
Sourcing a Script
Use the “source “ syntax to source a
script. When run this way, the script essentially becomes an argument
for the source command, which handles invoking the appropriate
execution. Some shells (e.g. ksh) use a single period “.” instead of
“source”.
When a script is executed with the “./“ syntax, the shebang at the top of the file is used to determine which shell to use. When a script is sourced, the shebang is ignored and the shell in which the script is launched is used instead. Also note that the period that appears in the “./“ command syntax used to execute a script, is not related to the period that’s occasionally used as an alias for the source command.
The script in the post uses bash in the shebang statement, so it works when executed because it’s run using bash. When it’s sourced from zsh, it encounters the incorrect indirect expansion syntax:
“${!A_VAR}"
The correct syntax is:
"${(P)A_VAR}"
However, correcting the syntax won’t help because it will then fail when executed. The shebang will invoke bash and the syntax will be wrong again. That renders indirection useless for accessing a variable designed to indicate the shell in use. More importantly, a design based on querying an environment variable for the shell is flawed due to differences in the shell that’s ultimately used depending on whether the script is executed or sourced.
To add to your answer (what I'm going to say is too long for a comment), I can not think of any application, why your script could be useful if not sourced. Actually, I came accross the need of such a script by myself in exactly one occasion:
Since I use as interactive shell not only zsh, but also sometimes bash, so I have written my .zshrc and .bashrc to set up everything (including defining variables and shell functions for interactive use). In order to safe work,
I try to put code which works under both bash and zsh into a single file (say: .commonrc), and my .zshrc and .bashrc have inside them a
source .commonrc
While many things are so different in bash and zsh, that I can't put them into .commonrc, some can, provided I do some tweaking. One reason for headache is obviously the different indexing of arrays, which you seemingly try to solve. So I have also a similar feature. However, I don't nee ca case construct for this. Instead, my .bashrc looks like this (using your naming of the variables):
...
declare -r ARRAY_START=0
source .commonrc
...
and my .zshrc looks like this:
...
declare -r ARRAY_START=1
source .commonrc
...
Since it does not happen that the .bashrc is run from a zsh and vice versa, I don't need to query what kind of shell I have.
I have been working on a few scripts on CentOS 7 and sometimes I see:
#!/bin/sh -
on the first line. Looking at the man page for sh I see the following under the Special Parameters
- Expands to the current option flags as specified upon invocation,
by the set builtin command, or those set by the shell
itself (such as the -i option).
What exactly does this mean? When do I need to use this special parameter option??
The documentation you are reading has nothing to do with the command line you're looking at: it's referring to special variables. In this case, if you run echo $- you will see "the current option flags as specified upon invocation...".
If you take a look at the OPTIONS part of the bash man page, you will find:
-- A -- signals the end of options and disables further option processing.
Any arguments after the -- are treated as filenames and arguments. An
argument of - is equivalent to --.
In other words, an argument of - simply means "there are no other options after this argument".
You often see this used in situation in which you want to avoid filenames starting with - accidentally being treated as command options: for example, if there is a file named -R in your current directory, running ls * will in fact behave as ls -R and produce a recursive listing, while ls -- * will not treat the -R file specially.
The single dash when used in the #! line is meant as a security precaution. You can read more about that here.
/bin/sh is an executable representing the system shell. Actually, it is usually implemented as a symbolic link pointing to the executable for whichever shell is the system shell. The system shell is kind of the default shell that system scripts should use. In Linux distributions, for a long time this was usually a symbolic link to bash, so much so that it has become somewhat of a convention to always link /bin/sh to bash or a bash-compatible shell. However, in the last couple of years Debian (and Ubuntu) decided to switch the system shell from bash to dash - a similar shell - breaking with a long tradition in Linux (well, GNU) of using bash for /bin/sh. Dash is seen as a lighter, and much faster, shell which can be beneficial to boot speed (and other things that require a lot of shell scripts, like package installation scripts).
Dash is fairly well compatible with bash, being based on the same POSIX standard. However, it doesn't implement the bash-specific extensions. There are scripts in existence that use #!/bin/sh (the system shell) as their shebang, but which require bash-specific extensions. This is currently considered a bug that should be fixed by Debian and Ubuntu, who require /bin/sh to be able to work when pointed to dash.
Even though Ubuntu's system shell is pointing to dash, your login shell as a user continues to be bash at this time. That is, when you log in to a terminal emulator anywhere in Linux, your login shell will be bash. Speed of operation is not so much a problem when the shell is used interactively, and users are familiar with bash (and may have bash-specific customization in their home directory).
I have a shell script on a mac (OSX 10.9) named msii810161816_TMP_CMD with the following content.
matlab
When I execute it, I get
./msii810161816_TMP_CMD: line 1: matlab: command not found
However, when I type matlab into the shell directly it starts as normal. How can it be that the same command works inside the shell but not inside a shell script? I copy-pasted the command directly from the script into the shell and it worked ...
PS: When I replace the content of the script with
echo matlab
I get the desired result, so I can definitely execute the shell script (I use ./msii810161816_TMP_CMD)
Thanks guys!
By default, aliases are not expanded in non-interactive shells, which is what shell scripts are. Aliases are intended to be used by a person at the keyboard as a typing aid.
If your goal is to not have to type the full path to matlab, instead of creating an alias you should modify your $PATH. Add /Applications/MATLAB_R2014a.app/bin to your $PATH environment variable and then both you and your shell scripts will be able to simply say
matlab
This is because, as commenters have stated, the PATH variable inside of the shell executing the script does not include the directory containing the matlab executable.
When a command name is used, like "matlab", your shell looks at every directory in the PATH in order, searching for one containing an executable file with the name "matlab".
Without going into too much detail, the PATH is determined by the shell being invoked.
When you execute bash, it combines a global setting for basic directories that must be in the PATH with any settings in your ~/.bashrc which alter the PATH.
Most likely, you are not running your script in a shell where the PATH includes matlab's directory.
To verify this, you can take the following steps:
Run which matlab. This will show you the path to the matlab executable.
Run echo "$PATH". This will show you your current PATH settings. Note that the directory from which matlab is included in the colon-separated list.
Add a line to the beginning of your script that does echo "$PATH". Note that the directory from which matlab is not included.
To resolve this, ensure that your script is executed in a shell that has the needed directory in the PATH.
You can do this a few ways, but the two most highly recommended ones would be
Add a shebang line to the start of your script. Assuming that you want to run it with bash, do #!/bin/bash or whatever the path to your bash interpreter is.
The shebang line is not actually fully standardized by POSIX, so BSD-derived systems like OSX will happily handle multiple arguments to the shebanged executable, while Linux systems pass at most one argument.
In spite of this, the shebang is an easy and simple way to document what should be used to execute the script, so it's a good solution.
Explicitly invoke your script with a shell as its interpreter, as in bash myscript.sh or tcsh myscript.sh or even sh myscript.sh
This is not incompatible with using a shebang line, and using both is a common practice.
I believe that the default shell on OSX is always bash, so you should start by trying with that.
If these instructions don't help, then you'll have to dig deeper to find out why or how the PATH is being altered between the calling context and the script's internal context.
Ultimately, this is almost certainly the source of your issue.
I am using Bash
$ echo $SHELL
/bin/bash
and starting about a year ago I stopped using Shebangs with my Bash scripts. Can
I benefit from using #!/bin/sh or #!/bin/bash?
Update: In certain situations a file is only treated as a script with the
Shebang, example
$ cat foo.sh
ls
$ cat bar.sh
#!/bin/sh
ls
$ file foo.sh bar.sh
foo.sh: ASCII text
bar.sh: POSIX shell script, ASCII text executable
On UNIX-like systems, you should always start scripts with a shebang line. The system call execve (which is responsible for starting programs) relies on an executable having either an executable header or a shebang line.
From FreeBSD's execve manual page:
The execve() system call transforms the calling process into a new
process. The new process is constructed from an ordinary file, whose
name is pointed to by path, called the new process file.
[...]
This file is
either an executable object file, or a file of data for an interpreter.
[...]
An interpreter file begins with a line of the form:
#! interpreter [arg]
When an interpreter file is execve'd, the system actually execve's the
specified interpreter. If the optional arg is specified, it becomes the
first argument to the interpreter, and the name of the originally
execve'd file becomes the second argument
Similarly from the Linux manual page:
execve() executes the program pointed to by filename. filename must be
either a binary executable, or a script starting with a line of the
form:
#! interpreter [optional-arg]
In fact, if a file doesn't have the right "magic number" in it's header, (like an ELF header or #!), execve will fail with the ENOEXEC error (again from FreeBSD's execve manpage):
[ENOEXEC] The new process file has the appropriate access
permission, but has an invalid magic number in its
header.
If the file has executable permissions, but no shebang line but does seem to be a text file, the behaviour depends on the shell that you're running in.
Most shells seem to start a new instance of themselves and feed it the file, see below.
Since there is no guarantee that the script was actually written for that shell, this can work or fail spectacularly.
From tcsh(1):
On systems which do not understand the `#!' script interpreter conven‐
tion the shell may be compiled to emulate it; see the version shell
variable. If so, the shell checks the first line of the file to see if
it is of the form `#!interpreter arg ...'. If it is, the shell starts
interpreter with the given args and feeds the file to it on standard
input.
From FreeBSD's sh(1):
If the program is not a normal executable file (i.e., if it
does not begin with the “magic number” whose ASCII representation is
“#!”, resulting in an ENOEXEC return value from execve(2)) but appears to
be a text file, the shell will run a new instance of sh to interpret it.
From bash(1):
If this execution fails because the file is not in executable format,
and the file is not a directory, it is assumed to be a shell script, a
file containing shell commands. A subshell is spawned to execute it.
You cannot always depend on the location of a non-standard program like bash. I've seen bash in /usr/bin, /usr/local/bin, /opt/fsf/bin and /opt/gnu/bin to name a few.
So it is generally a good idea to use env;
#!/usr/bin/env bash
If you want your script to be portable, use sh instead of bash.
#!/bin/sh
While standards like POSIX do not guarantee the absolute paths of standard utilities, most UNIX-like systems seem to have sh in /bin and env in /usr/bin.
Scripts should always begin with a shebang line. If a script doesn't start with this, then it may be executed by the current shell. But that means that if someone who uses your script is running a different shell than you do, the script may behave differently. Also, it means the script can't be run directly from a program (e.g. the C exec() system call, or find -exec), it has to be run from a shell.
You might be interested in an early description by Dennis M Ritchie (dmr) who invented the #! :
From uucp Thu Jan 10 01:37:58 1980
.>From dmr Thu Jan 10 04:25:49 1980 remote from research
The system has been changed so that if a file
being executed begins with the magic characters #! , the rest of the
line is understood to be the name of an interpreter for the executed
file. Previously (and in fact still) the shell did much of this job;
it automatically executed itself on a text file with executable mode
when the text file's name was typed as a command. Putting the facility
into the system gives the following benefits.
1) It makes shell scripts more like real executable files, because
they can be the subject of 'exec.'
2) If you do a 'ps' while such a command is running, its real name
appears instead of 'sh'. Likewise, accounting is done on the basis of
the real name.
3) Shell scripts can be set-user-ID.
4) It is simpler to have alternate shells available; e.g. if you like
the Berkeley csh there is no question about which shell is to
interpret a file.
5) It will allow other interpreters to fit in more smoothly.
To take advantage of this wonderful opportunity, put
#! /bin/sh
at the left margin of the first line of your shell scripts. Blanks
after ! are OK. Use a complete pathname (no search is done). At the
moment the whole line is restricted to 16 characters but this limit
will be raised.
Hope this helps
If you write bash scripts, i.e. non portable scripts containing bashisms, you should keep using the #!/bin/bash shebang just to be sure the correct interpreter is used. You should not replace the shebang by #!/bin/sh as bash will run in POSIX mode so some of your scripts might behave differently.
If you write portable scripts, i.e. scripts only using POSIX utilities and their supported options, you might keep using #!/bin/sh on your system (i.e. one where /bin/sh is a POSIX shell).
It you write stricly conforming POSIX scripts to be distributed in various platforms and you are sure they will only be launched from a POSIX conforming system, you might and probably should remove the shebang as stated in the POSIX standard:
As it stands, a strictly conforming application must not use "#!" as the first two characters of the file.
The rationale is the POSIX standard doesn't mandate /bin/sh to be the POSIX compliant shell so there is no portable way to specify its path in a shebang. In this third case, to be able to use the 'find -exec' syntax on systems unable to run a shebangless still executable script, you can simply specify the interpreter in the find command itself, eg:
find /tmp -name "*.foo" -exec sh -c 'myscript "$#"' sh {} +
Here, as sh is specified without a path, the POSIX shell will be run.
The header is useful since it specifies which shell to use when running the script. For example, #!/bin/zsh would change the shell to zsh instead of bash, where you can use different commands.
For example, this page specifies the following:
Using #!/bin/sh, the default Bourne shell in most commercial variants
of UNIX, makes the script portable to non-Linux machines, though you
sacrifice Bash-specific features ...
TL;DR: always in scripts; please not in source'd scripts
Always in your parent
FYI: POSIX compliant is #!/bin/bash, not #!/bin/sh
You want to clarify this so that nothing else overrides the interpreter your script is made for.
You don't want a user at the terminal using zsh to have trouble if your script was written for POSIX bash scripts.
You don't want to run source in your #!/bin/bash unrecognized by #!/bin/sh, someone in an sh terminal have it break the script because it is expecting the simple/POSIX . for including source'd files
You don't want e.g. zsh features - not available in other interpreters - to make their way into your bash code. So, put #!/bin/bash in all your script headers. Then, any of your zsh habits in your script will break so you know to remove them before your roll-out.
It's probably best, especially so POSIX-compliant scripts don't break in a terminal like zsh.
Not expected for included source scripts
FYI: POSIX compliant for sourcing text in a BASH script is ., not source
You can use either for sourcing, but I'll do POSIX.
Standard "shebanging" for all scripting:
parent.sh:
#!/bin/bash
echo "My script here"
. sourced.sh # child/source script, below
sourced.sh:
echo "I am a sourced child script"
But, you are allowed to do this...
sourced.sh: (optional)
#!/bin/bash
echo "I am a sourced child script"
There, the #!/bin/bash "shebang" will be ignored. The main reason I would use it is for syntax highlighting in my text editor. However, in the "proper" scripting world, it is expected that your rolled-out source'd script will not contain the shebang.
In addition to what the others said, the shebang also enables syntax highlighting in some text editors, for example vim.
$SHELL and #!/bin/bash or #!/bin/sh are different.
To start, #!/bin/sh is a symlink to /bin/bash on most Linux systems (on Ubuntu it is now /bin/dash)
But on whether to start with /bin/sh or /bin/bash:
Bash and sh are two different shells. Basically bash is sh, with more
features and better syntax. Most commands work the same, but they are
different.
Just assume if you're writing a bash script, stick with /bin/bash and not /sh because problems can arise.
$SHELL does not necessarily reflect the currently running shell.
Instead, $SHELL is the user's preferred shell, which is typically the
one set in /etc/passwd. If you start a different shell after logging
in, you can not necessarily expect $SHELL to match the current shell
anymore.
This is mine for example, but it could also be /root:/bin/dash or /root:/bin/sh depending on which shell you have input in passwd. So to avoid any problems, keep the passwd file at /bin/bash and then using $SHELL vs. #!/bin/bash wouldn't matter as much.
root#kali:~/Desktop# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
Sources:
http://shebang.mintern.net/bourne-is-not-bash-or-read-echo-and-backslash/
https://unix.stackexchange.com/questions/43499/difference-between-echo-shell-and-which-bash
http://man.cx/sh
http://man.cx/bash
I make a complex and long line command to successful login in a site. If I execute it in Console it work. But if I copy and paste the same line in a bash script it not work.
I tried a lot of thing, but accidentally discovery that if I NOT use the line
#!/bin/sh
it work! Why this happens in my mac OSX Lion? What this config line do in a bash script?
A bash script that is run via /bin/sh runs in sh compatibility mode, which means that many bash-specific features (herestrings, process substitution, etc.) will not work.
sh-4.2$ cat < <(echo 123)
sh: syntax error near unexpected token `<'
If you want to be able to use full bash syntax, use #!/bin/bash as your shebang line.
"#!/bin/sh" is a common idiom to insure that the correct interpreter is used to run the script. Here, "sh" is the "Bourne Shell". A good, standard "least common denominator" for shell scripts.
In your case, however, "#!/bin/sh" seems to be the wrong interpreter.
Here's a bit more info:
http://www.unix.com/answers-frequently-asked-questions/7077-what-does-usr-bin-ksh-mean.html
Originally, we only had one shell on unix. When you asked to run a
command, the shell would attempt to invoke one of the exec() system
calls on it. It the command was an executable, the exec would succeed
and the command would run. If the exec() failed, the shell would not
give up, instead it would try to interpet the command file as if it
were a shell script.
Then unix got more shells and the situation became confused. Most
folks would write scripts in one shell and type commands in another.
And each shell had differing rules for feeding scripts to an
interpreter.
This is when the "#! /" trick was invented. The idea was to let the
kernel's exec() system calls succeed with shell scripts. When the
kernel tries to exec() a file, it looks at the first 4 bytes which
represent an integer called a magic number. This tells the kernel if
it should try to run the file or not. So "#! /" was added to magic
numbers that the kernel knows and it was extended to actually be able
to run shell scripts by itself. But some people could not type "#! /",
they kept leaving the space out. So the kernel was exended a bit again
to allow "#!/" to work as a special 3 byte magic number.
So #! /usr/bin/ksh and
#!/usr/bin/ksh now mean the same thing. I always use the former since at least some kernels might still exist that don't understand the
latter.
And note that the first line is a signal to the kernel, and not to the
shell. What happens now is that when shells try to run scripts via
exec() they just succeed. And we never stumble on their various
fallback schemes.
The very first line of the script can be used to select which script interpreter to use.
With
#!/bin/bash
You are telling the shell to invoke /bin/bash interpreter to execute your script.
Assure that there are not spaces or empty lines before #!/bin/bash or it will not work.