I am trying to log the command with its parameters (after evaluating if necessary) in Bash.
I am trying to use set -v:
$ variable=2
$ set -v
$ sleep $variable
sleep $variable
As you can see, it prints sleep $variable. I want to log sleep 2 instead.
My original command is more complex, so I don't want to echo each parameter one by one. (And it is probably more error prone to do so).
set -v (or set -o verbose) will output every command as it is read, without expanding things.
set -x (or set -o xtrace) will output the expanded command line before execution. Each line is prepended by the PS4 prompt (usually a +) and for commands executed as part of command substitutions, the prompt will be "doubled up" (++).
The trace will be written to the file descriptor indicated by $BASH_XTRACEFD (or to the shell's standard error by default).
Related
In my program I need to know the maximum number of process I can run. So I write a script. It works when I run it in shell but but when in program using system("./limit.sh"). I work in bash.
Here is my code:
#/bin/bash
LIMIT=\`ulimit -u\`
ACTIVE=\`ps -u | wc -l \`
echo $LIMIT > limit.txt
echo $ACTIVE >> limit.txt
Anyone can help?
Why The Original Fails
Command substitution syntax doesn't work if escaped. When you run:
LIMIT=\`ulimit -u\`
...what you're doing is running a command named
-u`
...with the environment variable named LIMIT containing the value
`ulimit
...and unless you actually have a command that starts with -u and contains a backtick in its name, this can be expected to fail.
This is because using backticks makes characters which would otherwise be syntax into literals, and running a command with one or more var=value pairs preceding it treats those pairs as variables to export in the environment for the duration of that single command.
Doing It Better
#!/bin/bash
limit=$(ulimit -u)
active=$(ps -u | wc -l)
printf '%s\n' "$limit" "$active" >limit.txt
Leave off the backticks.
Use modern $() command substitution syntax.
Avoid multiple redirections.
Avoid all-caps names for your own variables (these names are used for variables with meaning to the OS or system; lowercase names are reserved for application use).
Doing It Right
#!/bin/bash
exec >limit.txt # open limit.txt as output for the rest of the script
ulimit -u # run ulimit -u, inheriting that FD for output
ps -u | wc -l # run your pipeline, likewise with output to the existing FD
You have a typo on the very first line: #/bin/bash should be #!/bin/bash - this is often known as a "shebang" line, for "hash" (#) + "bang" (!)
Without that syntax written correctly, the script is run through the system's default shell, which will see that line as just a comment.
As pointed out in comments, that also means only the standardised options available to the builtin ulimit command, which doesn't include -u.
This question already has answers here:
What does set -e mean in a bash script?
(10 answers)
Closed 6 years ago.
In a bash script what is the use of
set -e
?
I expect it has something to do with environment variables but I have not come across it before
quoting from help set
-e Exit immediately if a command exits with a non-zero status.
i.e the script or shell would exit as soon as it encounters any command that exited with a non-0(failure) exit code.
Any command that fails would result in the shell exiting immediately.
As an example:
Open up a terminal and type the following:
$ set -e
$ grep abcd <<< "abc"
As soon you hit enter after grep command, the shell exits because grep exited with a non-0 status i.e it couldn't find regex abcd in text abc
Note: to unset this behavior use set +e.
man bash says
Exit immediately if a simple command (see SHELL GRAMMAR above) exits with a non-zero
status. The shell does not exit if the command that fails is part of the command list
immediately following a while or until keyword, part of the test in an if statement,
part of a && or ││ list, or if the command’s return value is being inverted via !. A
trap on ERR, if set, is executed before the shell exits.
It is super convenient way to get "fail-fast" behaviour if you want to avoid testing the return code of every command in a bash script.
Suppose there is no file named trumpet in the current directory below script :
#!/bin/bash
# demonstrates set -e
# set -e means exit immediately if a command exited with a non zero status
set -e
ls trumpet #no such file so $? is non-zero, hence the script aborts here
# you still get ls: cannot access trumpet: No such file or directory
echo "some other stuff" # will never be executed.
You may also combine the e with the x option like set -ex where :
-x Print commands and their arguments as they are executed.
This may help you debugging bash scripts.
Reference:Set Manpage
Bash doesn't seem to pass the "exit on error" environment flag into command substitution shells.
I am using a large number of command substitutions (to get around bash's lack of return values), but I'd still like the whole script to go down if something in the subshell fails.
So, for example:
set -e
function do_internet {
curl not.valid.address
}
answer=$(do_internet)
I'd like the script to stop there and then, and not continue.
(I hoped that setting -e would stop from having to put '|| die' on everything.
Am I doing something wrong; and/or is there any way around this?
Here's a little example:
#!/bin/bash
set -e
echo "You should only see this line, and not any other line."
function foo {
false
echo "The above line is false. Figure that one out, Plato."
}
bar=$(foo)
echo $bar
It prints both lines.
(Using GNU bash, version 4.2.25(1)-release (x86_64-pc-linux-gnu))
There is a difference in handling of -e between subshells created with (...), as in Why doesn't bash flag -e exit when a subshell fails?, and subshells created with command substitution $(...), as in the OP.
According to the section COMMAND EXECUTION ENVIRONMENT in the bash manual (and slightly confusingly):
Subshells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in posix mode, bash clears the -e option in such subshells.
Regardless of the posix setting, the -e only applies to the subshell created for the purposes of command substitution. So:
$ set -e
# The subshell has -e cleared
$ echo $(false; echo foo)
foo
$ set -o posix
# Now the subshell has -e, so it terminates at `false`
$ echo $(false; echo foo)
$
Nonetheless, -e does apply to the execution of a command which only sets a variable. So
set -e
a=$(false)
will terminate the shell.
However, -e does not apply to individual commands in a function. In the case of
fail() {
false
echo "failed"
}
The return value of fail is 0 (i.e. success) because the echo (which was the last command executed) succeeded. Consequently
a=$(fail) && echo ok
will set a to failed and then print ok
Is there a better way to save a command line before it it executed?
A number of my /bin/bash scripts construct a very long command line. I generally save the command line to a text file for easier debugging and (sometimes) execution.
My code is littered with this idiom:
echo >saved.txt cd $NEW_PLACE '&&' command.py --flag $FOO $LOTS $OF $OTHER $VARIABLES
cd $NEW_PLACE && command.py --flag $FOO $LOTS $OF $OTHER $VARIABLES
Obviously updating code in two places is error-prone. Less obvious is that Certain parts need to be quoted in the first line but not the next. Thus, I can not do the update by simple copy-and-paste. If the command includes quotes, it gets even more complicated.
There has got to be a better way! Suggestions?
How about creating a helper function which logs and then executes the command? "$#" will expand to whatever command you pass in.
log() {
echo "$#" >> /tmp/cmd.log
"$#"
}
Use it by simply prepending log to any existing command. It won't handle && or || though, so you'll have to log those commands separately.
log cd $NEW_PLACE && log command.py --flag $FOO $LOTS $OF $OTHER $VARIABLES
are you looking for set -x (or bash -x)? This writes every command to standard out after executing.
use script and you will get archived everything.
use -x for tracing your script, e.g. run them as bash -x script_name args....
use set -x in your current bash (you will get echoed your commands with substitued globs and variables
combine 2 and 3 with the 1
If you just execute the command file immediately after creating it, you will only need to construct the command once, with one level of escapes.
If that would create too many discrete little command files, you could create shell procedures and then run an individual one.
(echo fun123 '()' {
echo echo something important
echo }
) > saved.txt
. saved.txt
fun123
It sounds like your goal is to keep a good log of what your script did so that you can debug it when things go bad. I would suggest using the -x parameter in your shebang like so:
#!/bin/sh -x
# the -x above makes bash print out every command before it is executed.
# you can also use the -e option to make bash exit immediately if any command
# returns a non-zero return code.
Also, see my answer on a previous question about redirecting all of this debug output to a log when --log is passed into your shell script. This will redirect all stdout and stderr. Occasionally, you'll still want to write to the terminal to give the user feedback. You can do this by saving stdout to a new file descriptor and using that with echo (or other programs):
exec 3>&1 # save stdout to fd 3
# perform log redirection as per above linked answer
# now all stdout and stderr will be redirected to the file and console.
# remove the `tee` command if you want it to go just to the file.
# now if you want to write to the original stdout (i.e. terminal)
echo "Hello World" >&3
# "Hello World" will be written to the terminal and not the logs.
I suggest you look into the xargs command. It was made to solve the problem of programtically building up argument lists and passing them off to executables for batch processing
http://en.wikipedia.org/wiki/Xargs
I want to inject a transparent wrappering command on each shell command in a make file. Something like the time shell command. ( However, not the time command. This is a completely different command.)
Is there a way to specify some sort of wrapper or decorator for each shell command that gmake will issue?
Kind of. You can tell make to use a different shell.
SHELL = myshell
where myshell is a wrapper like
#!/bin/sh
time /bin/sh "$0" "$#"
However, the usual way to do that is to prefix a variable to all command calls. While I can't see any show-stopper for the SHELL approach, the prefix approach has the advantage that it's more flexible (you can specify different prefixes for different commands, and override prefix values on the command line), and could be visibly faster.
# Set Q=# to not display command names
TIME = time
foo:
$(Q)$(TIME) foo_compiler
And here's a complete, working example of a shell wrapper:
#!/bin/bash
RESULTZ=/home/rbroger1/repos/knl/results
if [ "$1" == "-c" ] ; then
shift
fi
strace -f -o `mktemp $RESULTZ/result_XXXXXXX` -e trace=open,stat64,execve,exit_group,chdir /bin/sh -c "$#" | awk '{if (match("Process PID=\d+ runs in (64|32) bit",$0) == 0) {print $0}}'
# EOF
I don't think there is a way to do what you want within GNUMake itself.
I have done things like modify the PATH env variable in the Makefile so a directory with my script linked to all name the bins I wanted wrapped was executed rather than the actual bin. The script would then look at how it was called and exec the actual bin with the wrapped command.
ie. exec time "$0" "$#"
These days I usually just update the targets in the Makefile itself. Keeping all your modifications to one file is usually better IMO than managing a directory of links.
Update
I defer to Gilles answer. It's a better answer than mine.
The program that GNU make(1) uses to run commands is specified by the SHELL make variable. It will run each command as
$SHELL -c <command>
You cannot get make to not put the -c in, since that is required for most shells. -c is passed as the first argument ($1) and <command> is passed as a single argument string as the second argument ($2).
You can write your own shell wrapper that prepends the command that you want, taking into account the -c:
#!/bin/sh
eval time "$2"
That will cause time to be run in front of each command. You need eval since $2 will often not be a single command and can contain all sorts of shell metacharacters that need to be expanded or processed.