This question already has answers here:
What does `set -x` do?
(3 answers)
Closed 4 years ago.
So I have a file deploy.sh, and it has the shell script. Since I know about it, I have a little confusion, that is what does that set -x actually means.
After running the file I have observed that the command written after it in the file gets mentioned in the terminal with a + sign.
Like if I have this,
#!/bin/bash
set -x
ng build
So the output mentions +ng build, and when I comment the set -x from the file, everything executes, but the later commands does not show up in the terminal.
I have researched about it, but specifically couldn't find the real meaning and work of this particular command.
You can read the bash online manual for set:
-x
Print a trace of simple commands, for commands, case commands, select
commands, and arithmetic for commands and their arguments or
associated word lists after they are expanded and before they are
executed. The value of the PS4 variable is expanded and the resultant
value is printed before the command and its expanded arguments.
So it does exactly what you described.
It's a bit hard to find but this is from the bash man page:
set [+abefhkmnptuvxBCEHPT] [+o option-name] [arg ...]
Without options, the name and value of each shell variable are
displayed in a format that can be reused as input for setting or
resetting the currently-set variables. Read-only variables can‐
not be reset. In posix mode, only shell variables are listed.
The output is sorted according to the current locale. When
options are specified, they set or unset shell attributes. Any
arguments remaining after option processing are treated as val‐
ues for the positional parameters and are assigned, in order, to
$1, $2, ... $n. Options, if specified, have the following
meanings:
[...]
-x After expanding each simple command, for command, case
command, select command, or arithmetic for command, dis‐
play the expanded value of PS4, followed by the command
and its expanded arguments or associated word list.
So basically it's a debug option. It does not only execute all the commands but also prints them before it executes them (to stderr).
Related
This question already has answers here:
How do I parse command line arguments in Bash?
(40 answers)
How can I use long options with the Bash getopts builtin?
(9 answers)
Closed 1 year ago.
I have the following bash script:
export USER_AT_HOST="<user#ipaddress>"
export PUBKEYPATH="$HOME/.ssh/<pubkeyfilename>"
ssh-copy-id -i "$PUBKEYPATH" "$USER_AT_HOST" -p 8524
I would like to modify the script, so that <pubkeyfilename> (which is currently hardcoded in the script), becomes a named input argument, and the same for the port number. Ideally, the script should do the following
take two named arguments, -p/--port and --pubkeyfile, respectively an integer and a path
perform input validation, i.e., check that -p/--port is an integer, if not empty, and that --pubkeyfile is a string. Of course, if this is not possible or too complex in bash, you can ignore this request.
if an argument is not passed, use a default value
if a wrong argument or the argument -h/--help is passed, an help message is displayed
This answer is highly regarded:
https://stackoverflow.com/a/14203146
but it doesn't show how to use default values if an argument is not passed, it doesn't perform input validation and it doesn't include an help method. Also, for the getopts option (which wouldn't allow me to use long option names), it refers to this tutorial
https://wiki.bash-hackers.org/howto/getopts_tutorial
which however gives me the error
This site can’t provide a secure connection
wiki.bash-hackers.org sent an invalid response.
ERR_SSL_PROTOCOL_ERROR
EDIT: this question is not a duplicate of How can I use long options with the Bash getopts builtin?, because the answers there either don't show how to use getops for long options (see https://stackoverflow.com/a/12026302/1711271) or don't show how to pass long options which require an argument (https://stackoverflow.com/a/30026641/1711271), or don't show how to set a default value for an option which is not passed (all of them). My OS is MacOS.
EDIT 2: my shell is bash:
> ps -p $$
PID TTY TIME CMD
67809 ttys001 0:00.01 -bash
I'm not going to learn another shell just for the sake of a simple script. However, if I understand correctly, it's possible to write a shell script that runs another shell and then terminate. Thus a solution based on zsh, while not ideal, could be acceptable.
It’s well known that export masks the return value of command
substitutions in its variable assignments. But, interestingly,
export does not mask the return value of failed substitutions:
$ (set -eu; export FOO="$(bad_command)"); echo $?
bash: bad_command: command not found
0
$ (set -eu; export FOO="${bad_variable}"); echo $?
bash: bad_variable: unbound variable
1
$ (set -eu; export FOO="${}"); echo $? # bad substitution
bash: ${}: bad substitution
1
(Similar behavior in dash.)
What part of the specification indicates that failure of command
substitution does not propagate through export, but failure of
parameter expansion does?
Relevant sections from man bash (GNU Bash 4.4):
set -u
Treat unset variables and parameters other than the
special parameters "#" and "*" as an error when performing parameter
expansion. If expansion is attempted on an unset variable or
parameter, the shell prints an error message, and, if not interactive,
exits with a non-zero status.
and
export [-fn] [name[=word]] ...
export -p
The supplied names are marked for automatic export to the
environment of subsequently executed commands. If the -f option is
given, the names refer to functions. If no names are given, or if the
-p option is supplied, a list of names of all exported variables is
printed. The -n option causes the export property to be removed from
each name. If a variable name is followed by =word, the value of
the variable is set to word. export returns an exit status of 0
unless an invalid option is encountered, one of the names is not a
valid shell variable name, or -f is supplied with a name that is
not a function.
—I don’t see anything here that would distinguish between the two cases.
In particular, export just says that the value of the variable “is set
to” word, which suggests that it goes through the normal expansion
process (which it does) without special treatment.
POSIX specification references:
Shell Command Language
The export special builtin
The set special builtin
Neither case is really up to export
A command substitution just becomes text (empty or not) in a command's argument array. The Unix process model does not have any mechanism for relaying whether the text came from a program, or whether that program was successful.
This means that it's not possible to write an external command that behaves differently when you run foo var="$(true)" vs foo var="$(false)" vs foo var="" and shell builtins like export traditionally follow the same behavior for ease of implementation.
With set -u and unset variables, the command never runs at all. The shell simply skips execution if it encounters this condition while building the argument array, and reports failure instead. A command can't choose to ignore such a failure, since it's never consulted.
It would certainly be possible to implement a new shell mode that similarly skips execution and reports failure if command substitutions fail during the construction of the argument array, but this has not been a traditional feature so it's not in the POSIX spec.
This question already has answers here:
What does the line '!/bin/sh -e' do? [closed]
(2 answers)
How to echo shell commands as they are executed
(14 answers)
Closed 4 years ago.
In the repository on GitHub of Tribler, I see a file with the interpreter: #!/bin/sh -xe.
On Mac and Linux man sh redirects to BASH(1) (same as man bash).
In this man file, under the section:
OPTIONS
,there is no mention of an option -x , nor an option -e.
What is the meaning of the interpreter #!/bin/sh -xe ?
I found this:
man sh:
-x xtrace Write each command to standard error (preceded by
a ‘+ ’) before it is executed. Useful for debug‐
ging.
-e errexit If not interactive, exit immediately if any
untested command fails. The exit status of a com‐
mand is considered to be explicitly tested if the
command is used to control an if, elif, while, or
until; or if the command is the left hand operand
of an “&&” or “||” operator.
It seems these are just error control options that make sure nothing worse happens if a command fails.
Per man page: read more
The -a, -b, -C, -e, -f, -m, -n, -o option, -u, -v, and -x options are described as part of the set utility in Special Built-In Utilities. The option letters derived from the set special built-in shall also be accepted with a leading plus sign ( '+' ) instead of a leading hyphen (meaning the reverse case of the option as described in this volume of IEEE Std 1003.1-2001).
when you read set man pages you get:
-x The shell shall write to standard error a trace for each command after it expands the command and before it executes it. It is unspecified whether the command that turns tracing off is traced.
and
-e When this option is on, if a simple command fails for any of the reasons listed in Consequences of Shell Errors or returns an exit status value >0, and is not part of the compound list following a while, until, or if keyword, and is not a part of an AND or OR list, and is not a pipeline preceded by the ! reserved word, then the shell shall immediately exit.
Consider this little shell script.
# Save the first command line argument
cmd="$1"
# Execute the command specified in the first command line argument
out=$($cmd)
# Do something with the output of the specified command
# Here we do a silly thing, like make the output all uppercase
echo "$out" | tr -s "a-z" "A-Z"
The script executes the command specified as the first argument, transforms the output obtained from that command and prints it to standard output. This script may be executed in this manner.
sh foo.sh "echo select * from table"
This does not do what I want. It may print something like the following,
$ sh foo.sh "echo select * from table"
SELECT FILEA FILEB FILEC FROM TABLE
if fileA, fileB and fileC is present in the current directory.
From a user perspective, this command is reasonable. The user has quoted the * in the command line argument, so the user doesn't expect the * to be globbed. But my script astonishes the user by using this argument in a command substitution which causes globbing of * as seen in the above output.
I want the output to be the following instead.
SELECT * FROM TABLE
The entire text in cmd actually comes from command line arguments to the script so I would like to preserve any * symbol present in the argument without globbing them.
I am looking for a solution that works for any POSIX shell.
One solution I have come up with is to disable globbing with set -o noglob just before the command substitution. Here is the complete code.
# Save the first command line argument
cmd="$1"
# Execute the command specified in the first command line argument
set -o noglob
out=$($cmd)
# Do something with the output of the specified command
# Here we do a silly thing, like make the output all uppercase
echo "$out" | tr -s "a-z" "A-Z"
This does what I expect.
$ sh foo.sh "echo select * from table"
SELECT * FROM TABLE
Apart from this, is there any other concept or trick (such as a quoting mechanism) I need to be aware of to disable globbing only within a command substitution without having to use set -o noglob.
I am not against set -o noglob. I just want to know if there is another way. You know, globbing can be disabled for normal command line arguments just by quoting them, so I was wondering if there is anything similar for command substiution.
If I understand correctly, you want the user to provide a shell command as a command-line argument, which will be executed by the script, and is expected to produce an SQL string, which will be processed (upper-cased) and echoed to stdout.
The first thing to say is that there is no point in having the user provide a shell command that the script just blindly executes. If the script applied some kind of modification/preprocessing of the command before it executed it then perhaps it could make sense, but if not, then the user might as well execute the command himself and pass the output to the script as a command-line argument, or via stdin.
But that being said, if you really want to do it this way, then there are two things that need to be said. Firstly, this is the proper form to use:
out=$(eval "$cmd");
A fairly advanced understanding of the shell grammer and expansion rules would be required to fully understand the rationale for using the above syntax, but basically executing $cmd and executing eval "$cmd" have subtle differences that render the $cmd form inappropriate for executing a given shell command string.
Just to give some detail that will hopefully clarify the above point, there are seven kinds of expansion that are performed by the shell in the following order when processing input: (1) brace expansion, (2) tilde expansion, (3) parameter and variable expansion, (4) arithmetic expansion, (5) command substitution, (6) word splitting, and (7) pathname expansion. Notice that variable expansion happens somewhat in the middle of that sequence, and thus the variable-expanded shell command (which was provided by the user) will not receive the benefit of the prior expansion types. Other issues are that leading variable assignments, pipelines, and command list tokens will not be executed correctly under the $cmd form, because they are parsed and processed prior to variable expansion (actually prior to all expansions) as well.
By running the command through eval, properly double-quoted, you ensure that the full shell parsing/processing/execution algorithm will be applied to the shell command string that was given by the user of your script.
The second thing to say is this: If you try the above proper form in your script, you will find that it has not solved your problem. You will still get SELECT FILEA FILEB FILEC FROM TABLE as output.
The reason is this: Since you've decided you want to accept an arbitrary shell command from the user of your script, it is now the user's responsibility to properly quote all metacharacters that may be embedded in that piece of code. It does not make sense for you to accept a shell command as a command-line argument, but somehow change the processing rules for shell commands so that certain metacharacters will no longer be metacharacters when the given shell command is executed. Actually, you could do something like that, perhaps using set -o noglob as you discovered, but then that must become a contract between the script and the user of the script; the user must be made aware of exactly what the precise processing rules will be when the command is executed so that he can properly use the script.
Under this design, the user could call the script as follows (notice the extra layer of quoting for the shell command string evaluation; could alternatively backslash-escape just the asterisk):
$ sh foo.sh "echo 'select * from table'";
I'd like to return to my earlier comment about the overall design; it doesn't really make sense to do it this way. It makes more sense to take the text-to-process itself, not a shell command that is expected to produce the text-to-process.
Here is how that could be done:
## take the text-to-process via a command-line argument
sql="$1";
## process and echo it
echo "$sql"| tr a-z A-Z;
(I also removed the -s option of tr, which really doesn't make sense here.)
Notice that the script is simpler now, and usage is also simpler:
$ sh foo.sh 'select * from table';
This answer, "How to profile a bash shell script?", seems to nearly perfectly cover what I'm trying to accomplish here. I currently have some zsh scripts that modify the prompt, however I think some updates to oh-my-zsh have evoked some issues that I need to hunt down. The sluggishness from time to time is unbearable.
To this end, how would you adapt the prompt sections in this example answer to work with zsh vs bash?
Presently I have modified /etc/zshenv such that it has the initial suggested code from the example:
PS4='+ $(date "+%s.%N")\011 '
exec 3>&2 2>/tmp/bashstart.$$.log
set -x
And my ~/.zshrc has the following appended to it's tail:
set +x
exec 2>&3 3>&-
Of course these are not valid for ZSH shell customization. My prompt rendering code utilizes oh-my-zsh customizations. I could prepend the appropriate code to the prompt I suppose or I'm open to other suggestions.
Calling date for each command will fork and exec, which adds overhead which may interfere with your measurements.
Instead, you could use
PS4=$'+ %D{%s.%6.}\011 '
to log timestamps with lower overhead (up to millisecond precision).
For some notes on processing the resulting logs, see http://blog.xebia.com/profiling-zsh-shell-scripts/
You may need to do
setopt prompt_subst
if it's not already.
Also, in order to interpret the octal escape for tab, use $'':
PS4=$'+ $(date "+%s.%N")\011 '
You may also find some of these escapes to be useful:
%? The return status of the last command executed just before the prompt.
%_ The status of the parser, i.e. the shell constructs (like `if' and `for') that have been started on the command
line. If given an integer number that many strings will be printed; zero or negative or no integer means print as
many as there are. This is most useful in prompts PS2 for continuation lines and PS4 for debugging with the
XTRACE option; in the latter case it will also work non-interactively.
%i The line number currently being executed in the script, sourced file, or shell function given by %N. This is most
useful for debugging as part of $PS4.
%I The line number currently being executed in the file %x. This is similar to %i, but the line number is always a
line number in the file where the code was defined, even if the code is a shell function.
%L The current value of $SHLVL.
%N The name of the script, sourced file, or shell function that zsh is currently executing, whichever was started
most recently. If there is none, this is equivalent to the parameter $0. An integer may follow the `%' to spec‐
ify a number of trailing path components to show; zero means the full path. A negative integer specifies leading
components.
%x The name of the file containing the source code currently being executed. This behaves as %N except that function
and eval command names are not shown, instead the file where they were defined.