It’s well known that export masks the return value of command
substitutions in its variable assignments. But, interestingly,
export does not mask the return value of failed substitutions:
$ (set -eu; export FOO="$(bad_command)"); echo $?
bash: bad_command: command not found
0
$ (set -eu; export FOO="${bad_variable}"); echo $?
bash: bad_variable: unbound variable
1
$ (set -eu; export FOO="${}"); echo $? # bad substitution
bash: ${}: bad substitution
1
(Similar behavior in dash.)
What part of the specification indicates that failure of command
substitution does not propagate through export, but failure of
parameter expansion does?
Relevant sections from man bash (GNU Bash 4.4):
set -u
Treat unset variables and parameters other than the
special parameters "#" and "*" as an error when performing parameter
expansion. If expansion is attempted on an unset variable or
parameter, the shell prints an error message, and, if not interactive,
exits with a non-zero status.
and
export [-fn] [name[=word]] ...
export -p
The supplied names are marked for automatic export to the
environment of subsequently executed commands. If the -f option is
given, the names refer to functions. If no names are given, or if the
-p option is supplied, a list of names of all exported variables is
printed. The -n option causes the export property to be removed from
each name. If a variable name is followed by =word, the value of
the variable is set to word. export returns an exit status of 0
unless an invalid option is encountered, one of the names is not a
valid shell variable name, or -f is supplied with a name that is
not a function.
—I don’t see anything here that would distinguish between the two cases.
In particular, export just says that the value of the variable “is set
to” word, which suggests that it goes through the normal expansion
process (which it does) without special treatment.
POSIX specification references:
Shell Command Language
The export special builtin
The set special builtin
Neither case is really up to export
A command substitution just becomes text (empty or not) in a command's argument array. The Unix process model does not have any mechanism for relaying whether the text came from a program, or whether that program was successful.
This means that it's not possible to write an external command that behaves differently when you run foo var="$(true)" vs foo var="$(false)" vs foo var="" and shell builtins like export traditionally follow the same behavior for ease of implementation.
With set -u and unset variables, the command never runs at all. The shell simply skips execution if it encounters this condition while building the argument array, and reports failure instead. A command can't choose to ignore such a failure, since it's never consulted.
It would certainly be possible to implement a new shell mode that similarly skips execution and reports failure if command substitutions fail during the construction of the argument array, but this has not been a traditional feature so it's not in the POSIX spec.
Related
When using bash, there are self-referential variables that don't seem to be available in zsh. For example, $? gets the exit status of the most recent foreground process, $_ gets the last parameter of the previous command, etc.
Are there equivalents to these in zsh? I ask with reference to this question for bash.
bash calls these the special parameters (to distinguish them from ordinary variables and the positional parameters). zsh implements essentially the same set, but documents them along with other parameters set by the shell, though it does also tag each one as special.
See man zshparam:
PARAMETERS SET BY THE SHELL
In the parameter lists that follow, the mark `<S>' indicates that the
parameter is special. `<Z>' indicates that the parameter does not
exist when the shell initializes in sh or ksh emulation mode.
The following parameters are automatically set by the shell:
! <S> The process ID of the last command started in the background
with &, or put into the background with the bg builtin.
# <S> The number of positional parameters in decimal. Note that some
confusion may occur with the syntax $#param which substitutes
the length of param. Use ${#} to resolve ambiguities. In par-
ticular, the sequence `$#-...' in an arithmetic expression is
interpreted as the length of the parameter -, q.v.
ARGC <S> <Z>
Same as #.
$ <S> The process ID of this shell. Note that this indicates the
original shell started by invoking zsh; all processes forked
from the shells without executing a new program, such as sub-
shells started by (...), substitute the same value.
[... etc ...]
Why does the builtin BASH command set print shell functions? My searches on Google, and even Stack Overflow, haven't produced a satisfactory answer. From the manual page, "Without options, the name and value of each shell variable are displayed ... ." With options doesn't change this. Functions aren't variables ... are they?
For reference, this became a problem when I was using set to test for the existence of a variable:
set_version() {
if [[ ! $(set | grep -q PKG_VERSION_STAMP) ]]; then
echo Version not set. Using default
PKG_VERSION_STAMP=0.1
fi
echo Package Version: $PKG_VERSION_STAMP
}
I tested the conditional construction from the interactive shell, and it worked, but that's because I hadn't defined it in a function. My script failed because the function was defined and the grep passed. Consequently, the variable, PKG_VERSION_STAMP, was never defined.
Incidentally, changing from set to env fixes the problem. I'd just like to know why BASH considers functions to be variables and print their contents.
Should you want to have the set builtin not listing the shell functions, you can switch to bash posix mode:
$ set -o posix
$ set
...
While not documented in its manual, it is a well known fact since the shellshock vulnerability exposition that bash is storing its functions as special variables.
I'm using qsub to submit pbs jobs. My job file requires 5 environment variables be defined: qsub -v A=foo,B=bar,C=cat,D=dog,E=qux jobfile.sh. Sometimes, I might forget to define one of the variables on the command line when I submit only later to find out my error.
So, is there a way to catch the condition that not all environment variables are defined when submitting with qsub? Or am I going to have to accept that I have to be very careful when submitting job files with many required environment variables?
Thank you,
Paul
The best you can do is to add a runtime check to the beginning of jobfile.sh that will cause it to exit if any expected variables are not set.
set -u
: $A $B $C $D $E
set +u
set -u tells the shell to exit if an attempt is made to expand an unset parameter. The : command does nothing, but its arguments do undergo the standard expansions first, so if any of the five are unset, the shell will exit. set +u then disables the unset-parameter check for the remainder of the shell.
qsub itself cannot examine the given job to determine what variables it might need.
I really like the brevity of #chepner's answer, but if you don't want to fiddle with toggling set -/+u, then you can use this shell parameter expansion in a similar manner:
: ${A?} ${B?} ${C?} ${D?} ${E?}
If any of the variables are not set, then you'll get an error like this and the script will quit immediately:
./jobfile.sh: line 3: A: parameter null or not set
Optionally, you may override the default error message with your own like this:
${A?Your message here}
From the manual:
${parameter:?word}
If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of parameter is substituted.
I was trying to run a bash shell script in ESXi shell.
Think ESXi is posix compatible, posix compiler threw much syntax errors.
please let me know if anyone has the answer
The manual lists the changed behavior when running in POSIX mode. You should adapt your script accordingly.
The following list is what’s changed when POSIX mode is in effect:
When a command in the hash table no longer exists, Bash will re-search $PATH to find the new location. This is also available with ‘shopt -s checkhash’.
The message printed by the job control code and builtins when a job exits with a non-zero status is ‘Done(status)’.
The message printed by the job control code and builtins when a job is stopped is ‘Stopped(signame)’, where signame is, for example, SIGTSTP.
The bg builtin uses the required format to describe each job placed in the background, which does not include an indication of whether the job is the current or previous job.
Reserved words appearing in a context where reserved words are recognized do not undergo alias expansion.
The POSIX PS1 and PS2 expansions of ‘!’ to the history number and ‘!!’ to ‘!’ are enabled, and parameter expansion is performed on the values of PS1 and PS2 regardless of the setting of the promptvars option.
The POSIX startup files are executed ($ENV) rather than the normal Bash files.
Tilde expansion is only performed on assignments preceding a command name, rather than on all assignment statements on the line.
The default history file is ~/.sh_history (this is the default value of $HISTFILE).
The output of ‘kill -l’ prints all the signal names on a single line, separated by spaces, without the ‘SIG’ prefix.
The kill builtin does not accept signal names with a ‘SIG’ prefix.
Non-interactive shells exit if filename in . filename is not found.
Non-interactive shells exit if a syntax error in an arithmetic expansion results in an invalid expression.
Non-interactive shells exit if there is a syntax error in a script read with the . or source builtins, or in a string processed by the eval builtin.
Redirection operators do not perform filename expansion on the word in the redirection unless the shell is interactive.
Redirection operators do not perform word splitting on the word in the redirection.
Function names must be valid shell names. That is, they may not contain characters other than letters, digits, and underscores, and may not start with a digit. Declaring a function with an invalid name causes a fatal syntax error in non-interactive shells.
POSIX special builtins are found before shell functions during command lookup.
The time reserved word may be used by itself as a command. When used in this way, it displays timing statistics for the shell and its completed children. The TIMEFORMAT variable controls the format of the timing information.
When parsing and expanding a ${...} expansion that appears within double quotes, single quotes are no longer special and cannot be used to quote a closing brace or other special character, unless the operator is one of those defined to perform pattern removal. In this case, they do not have to appear as matched pairs.
The parser does not recognize time as a reserved word if the next token begins with a ‘-’.
If a POSIX special builtin returns an error status, a non-interactive shell exits. The fatal errors are those listed in the POSIX standard, and include things like passing incorrect options, redirection errors, variable assignment errors for assignments preceding the command name, and so on.
A non-interactive shell exits with an error status if a variable assignment error occurs when no command name follows the assignment statements. A variable assignment error occurs, for example, when trying to assign a value to a readonly variable.
A non-interactive shell exists with an error status if a variable assignment error occurs in an assignment statement preceding a special builtin, but not with any other simple command.
A non-interactive shell exits with an error status if the iteration variable in a for statement or the selection variable in a select statement is a readonly variable.
Process substitution is not available.
Assignment statements preceding POSIX special builtins persist in the shell environment after the builtin completes.
Assignment statements preceding shell function calls persist in the shell environment after the function returns, as if a POSIX special builtin command had been executed.
The export and readonly builtin commands display their output in the format required by POSIX.
The trap builtin displays signal names without the leading SIG.
The trap builtin doesn’t check the first argument for a possible signal specification and revert the signal handling to the original disposition if it is, unless that argument consists solely of digits and is a valid signal number. If users want to reset the handler for a given signal to the original disposition, they should use ‘-’ as the first argument.
The . and source builtins do not search the current directory for the filename argument if it is not found by searching PATH.
Subshells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in POSIX mode, Bash clears the -e option in such subshells.
Alias expansion is always enabled, even in non-interactive shells.
When the alias builtin displays alias definitions, it does not display them with a leading ‘alias ’ unless the -p option is supplied.
When the set builtin is invoked without options, it does not display shell function names and definitions.
When the set builtin is invoked without options, it displays variable values without quotes, unless they contain shell metacharacters, even if the result contains nonprinting characters.
When the cd builtin is invoked in logical mode, and the pathname constructed from $PWD and the directory name supplied as an argument does not refer to an existing directory, cd will fail instead of falling back to physical mode.
The pwd builtin verifies that the value it prints is the same as the current directory, even if it is not asked to check the file system with the -P option.
When listing the history, the fc builtin does not include an indication of whether or not a history entry has been modified.
The default editor used by fc is ed.
The type and command builtins will not report a non-executable file as having been found, though the shell will attempt to execute such a file if it is the only so-named file found in $PATH.
The vi editing mode will invoke the vi editor directly when the ‘v’ command is run, instead of checking $VISUAL and $EDITOR.
When the xpg_echo option is enabled, Bash does not attempt to interpret any arguments to echo as options. Each argument is displayed, after escape characters are converted.
The ulimit builtin uses a block size of 512 bytes for the -c and -f options.
The arrival of SIGCHLD when a trap is set on SIGCHLD does not interrupt the wait builtin and cause it to return immediately. The trap command is run once for each child that exits.
What is the meaning of a bash variable used like this:
${Server?}
It works almost the same as (from the bash manpage):
${parameter:?word} Display Error if Null or Unset. If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of parameter is substituted.
That particular variant checks to ensure the variable exists (is both defined and not null). If so, it uses it. If not, it outputs the error message specified by word (or a suitable one if there is no word) and terminates the script.
The actual difference between that and the non-colon version can be found in the bash manpage above the section quoted:
When not performing substring expansion, using the forms documented below, bash tests for a parameter that is unset or null. Omitting the colon results in a test only for a parameter that is unset.
In other words, the section above can be modified to read (basically taking out the "null" bits):
${parameter?word} Display Error if Unset. If parameter is unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of parameter is substituted.
The difference is illustrated thus:
pax> unset xyzzy ; export plugh=
pax> echo ${xyzzy:?no}
bash: xyzzy: no
pax> echo ${plugh:?no}
bash: plugh: no
pax> echo ${xyzzy?no}
bash: xyzzy: no
pax> echo ${plugh?no}
pax> _
There, you can see that while both unset and null variable result in an error with :?, only the unset one errors with ?.
It means that the script should abort if the variable isn't defined
Example:
#!/bin/bash
echo We will see this
${Server?Oh no! server is undefined!}
echo Should not get here
This script will print out the first echo, and the "Oh no! ..." error message.
See all the variable substitutions for bash here:
https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
As per man bash:
${parameter:?word}
If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of parameter is substituted.
So if you omit word and don't pass $1 to your function or shell script then it will exit with this error:
-bash: 1: parameter null or not set
However you can place a user-defined error to let your caller know that a required argument has not been set e.g.:
local arg1="${1:?needs an argument}"
It will continue running script/function only if $1 is set.
:? causes the (non-interactive) shell to exit with an error message if given parameter is null or unset. The script you are looking at is omitting the error message; usually, you'd see something like
foo=${1:?Missing first argument}
For example,
$ foo=${1:?Missing first argument}
bash: 1: Missing first argument
$ set firstarg
$ foo=${1:?Missing first argument}
$ echo $foo
firstarg