Bash /usr/bin/time command runs command in subshell - bash

In a bash script (running on REHL5) I use the /usr/bin/time command (not the builtin time command) to record the run times of other commands I run in that same script. The problem I am facing is that some of the commands that I want to record times for are not builtin commands or external scripts but are functions that are either declared in the script or are sourced from a different script. I have found out that the time command fails with error like below:
/usr/bin/time: cannot run shared-func: No such file or directory
Which means that the function shared-func() that I declare elsewhere is not visible in the scope and therefore time cannot run that command. I have run some tests and have verified that the reason behind this error is in fact because the time command tries to execute its command in a new subshell and therefore loses every declared function or variable in its scope. Is there a way to get around this? The ideal solution would be to force the time command to change its behavior and use the current shell for executing its command but I am also interested in any other solutions if that is not possible.
For the record, below is the test I ran. I created two small scripts:
shared.sh:
function shared-func() {
echo "The shared function is visible."
}
test.sh:
#!/bin/bash
function record-timestamp() {
/usr/bin/time -f % -a -o timestamps.csv "$#"
}
source "shared.sh"
record-timestamp shared-func
And this is the test:
$ ./test.sh
/usr/bin/time: cannot run shared-func: No such file or directory
$

A different process, yes. A subshell, no.
A subshell is what you get when your parent shell forks but doesn't exec() -- it's a new process made by copying your current shell instance. Functions are accessible within subshells, though they can't have direct effect on the parent shell (changes to process state die with the subshell when it exits).
When you launch an external program without using exec, the shell first forks, but then calls execve() to invoke the new program. execve() replaces the process image in memory with the program being run -- so it's not the fork(), that is, the creation of the subshell causing this to fail; instead, it's the exec(), the invocation of a separate program.
Even if your new process is also a shell, if it's gone through any exec()-family call it's not a subshell -- it's a whole new process.
tl;dr: You cannot use an external program to wrap a shell function invocation inside the current shell, because an external program's invocation always uses execve(), and execve() always clears process state -- including non-exported shell functions.

function shared-func() {
echo "The shared function is visible."
}
export -f shared-func
echo shared-func | /usr/bin/time -lp /bin/bash
Output:
cat <<EOF
The shared function is visible.
real 0.00
user 0.00
sys 0.00
1040384 maximum resident set size
...
On a 3GHz machine, the overhead of running bash this way is approximately 5ms (u+s time).

/bin/time is a separate program so it will be run in a separate process, and then it will try to run yet another program in yet another separate process.
There's a trick though, a shell script can call itself, or in this case tell /bin/time to call itself.
function record-timestamp() {
/usr/bin/time -f % -a -o timestamps.csv bash "$0" "$#"
}
if [ -n "$1" ]; then
source "shared.sh"
"$#"
else
record-timestamp shared-func
fi
If you call test.sh with arguments then it will try to run that function (which comes from shared.sh), and if you call it without arguments will call the record-timestamp function which in turn will call /bin/tine which in turn will call test.sh with arguments.

Related

Writing a bash script, how do I stop my session from exiting when my script exits?

bash scripting noob here. I've found this article: https://www.shellhacks.com/print-usage-exit-if-arguments-not-provided/ that suggests putting
[ $# -eq 0 ] && { echo "Usage: $0 argument"; exit 1; }
at the top of a script to ensure arguments are passed. Seems sensible.
However, when I do that and test that that line does indeed work (by running the script without supplying any arguments: . myscript.sh) then the script does indeed exit but so does the bash session that I was calling the script from. This is very irritating.
Clearly I'm doing something wrong but I don't know what. Can anyone put me straight?
. myscript.sh is a synonym for source myscript.sh, which runs the script in the current shell (rather than as a separate process). So exit terminates your current shell. (return, on the other hand, wouldn't; it has special behaviour for sourced scripts.)
Use ./myscript.sh to run it "the normal way" instead. If that gives you a permission error, make it executable first, using chmod a+x myscript.sh. To inform the kernel that your script should be run with bash (rather than /bin/sh), add the following as the very first line in the script:
#!/usr/bin/env bash
You can also use bash myscript.sh if you can't make it executable, but this is slightly more error-prone (somebody might do sh myscript.sh instead).
Question seems not clear if you're sourcing script source script_name or . script_name it's interpreted in current bash process, if you're running a function it's the same it's running in same process, otherwise, calling a script, caller bash forks a new bash process and waits until it terminates (so running exit doesn't exit caller process), but when running exit builtin in in current bash it exits current process.

Any reason not to exec in shell script?

I have a bunch of wrapper shell scripts which manipulate command line arguments and do some stuff before invoking another binary at the end. Is there any reason to not always exec the binary at the end? It seems like this would be simpler and more efficient, but I never see it done.
If you check /usr/bin, you will likely find many many shell scripts that end with an exec command. Just as an example, here is /usr/bin/ps2pdf (debian):
#!/bin/sh
# Convert PostScript to PDF.
# Currently, we produce PDF 1.4 by default, but this is not guaranteed
# not to change in the future.
version=14
ps2pdf="`dirname \"$0\"`/ps2pdf$version"
if test ! -x "$ps2pdf"; then
____ps2pdf="ps2pdf$version"
fi
exec "$ps2pdf" "$#"
exec is used because it eliminates the need for keeping the shell process active after it is no longer needed.
My /usr/bin directory has over 150 shell scripts that use exec. So, the use of exec is common.
A reason not to use exec would be if there was some processing to be done after the binary finished executing.
I disagree with your assessment that this is not a common practice. That said, it's not always the right thing.
The most common scenario where I end a script with the execution of another command, but can't reasonably use exec, is if I need a cleanup hook to be run after the command at the end finishes. For instance:
#!/bin/sh
# create a temporary directory
tempdir=$(mktemp -t -d myprog.XXXXXX)
cleanup() { rm -rf "$tempdir"; }
trap cleanup 0
# use that temporary directory for our program
exec myprog --workdir="$tempdir" "$#"
...won't actually clean up tempdir after execution! Changing that exec myprog to merely myprog has some disadvantages -- continued memory usage from the shell, an extra process-table entry, signals being potentially delivered to the shell rather than to the program that it's executing -- but it also ensures that the shell is still around on myprog's exit to run any traps required.

Exiting a shell script with an error

basically I have written a shell script for a homework assignment that works fine however I am having issues with exiting. Essentially the script reads numbers from the user until it reads a negative number and then does some output. I have the script set to exit and output an error code when it receives anything but a number and that's where the issue is.
The code is as follows:
if test $number -eq $number >dev/null 2>&1
then
"do stuff"
else
echo "There was an error"
exit
The problem is that we have to turn in our programs as text files using script and whenever I try to script my program and test the error cases it exits out of script as well. Is there a better way to do this?
The script is being run with the following command in the terminal
script "insert name of program here"
Thanks
If the program you're testing is invoked as a subprocess, then any exit command will only exit the command itself. The fact that you're seeing contrary behavior means you must be invoking it differently.
When invoking your script from the parent testing program, use:
# this runs "yourscript" as its own, external process.
./yourscript
...to invoke it as a subprocess, not
# this is POSIX-compliant syntax to run the commands in "yourscript" in the current shell.
. yourscript
...or...
# this is bash-extended syntax to run the commands in "yourscript" in the current shell.
source yourscript
...as either of the latter will run all the commands -- including exit -- inside your current shell, modifying its state or, in the case of exit, exec or similar, telling it to cease execution.

Can I combine flock and source?

I'd like to source a script (it sets up variables in my shell):
source foo.sh args
But under flock so that only one instance operates at a time (it does a lot of disk access which I'd like to ensure is serialized).
$ source flock lockfile foo.sh args
-bash: source: /usr/bin/flock: cannot execute binary file
and
$ flock lockfile source foo.sh args
flock: source: Success
don't work.
Is there some simple syntax for this I'm missing? Let's assume I can't edit foo.sh to put the locking commands inside it.
You can't source a script directly via flock because it is an external command and source is a shell builtin. You actually have two problems because of this:
flock doesn't know any command called source because it's built into bash
Even if flock could run it, the changes would not affect the state of the calling shell as you'd want with source because it's happening in a child process.
And, passing flock to source won't work because source expects a script. To do what you want, you need to lock by fd. Here's an example
#!/bin/bash
exec 9>lockfile
flock 9
echo whoopie
sleep 5
flock -u 9
Run two instances of this script in the same directory at the same time and you will see one wait for the other.

Timing a sourced bash script

Normally I simply call the time program when I want to profile a script or a command it runs.
But this time (no pun intended), I want to measure the time it takes to execute a sourced script, such as:
/usr/bin/time source myscript.sh
But I get the following error:
/usr/bin/time: cannot run source: No such file or directory
And sourcing it this way:
/usr/bin/time . myscript.sh
Simply gives me this error:
/usr/bin/time: cannot run .: Permission denied
I can call it in the following two ways:
/usr/bin/time ./myscript.sh
/usr/bin/time bash myscript.sh
But it's really important that I source my script. Anyone know how I can do this?
Additional Context:
We have many, many scripts which are somewhat complex and these scripts know how to source a specific 'set' of other 'client' scripts. We'll call these scripts 'bundles'.
bundle-a.sh
bundle-b.sh
...
bundle-z.sh
That's not exactly how they are named, but it serves the purpose of showing 26 of them in this example. Each bundle contain a lot of logic and each bundle invoke a specific set of client scripts.
There are hundreds of client scripts, some with a few lines of code and others that are really complex and take hours to execute.
client-000.sh
client-001.sh
...
client-300.sh
We also provide a script which serves as an API library for both our bundle scripts and our client scripts. Many, many functions in this API.
api.sh
So the bundle scripts source the API library and so does each client script. We have it this way so that a bundle can be called (which call a set of client scripts) or at times we only need to call a specific client script directly. It is implemented in a way that if the bundle is executed, the API library is only sourced once (and not again from the client script).
So all of this has worked well for a long time. Its just that now I'm trying to add the /usr/bin/time to each sourcing of our client scripts from the bundle scripts.
If you really, really need to get the system and user times of a sourced script, you can use strace from a different shell to watch the one you're timing in, trace calls to getrusage, and inspect the return values.
Shell 1 (where you will time the script)
$ echo $$
12345
Shell 2 (where you will trace)
$ strace -etrace=getrusage -v -p 12345
Process 12345 attached - interrupt to quit
Shell 1:
$ time : ; . myscript.sh ; time :
real 0m0.000s
(( time output and script output, then time output again)
Shell 2: will output something like
getrusage(RUSAGE_SELF, {ru_utime={0, 12000}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1551, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=1032, ru_nivcsw=3}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 448028}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1552, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=2141, ru_nivcsw=22}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
Notice how the values in the ru_utime structure element changed? The difference there is how much user CPU time the shell itself used. See man getrusage for the details on exactly what you are seeing. From there, it's just a automatically extracting those and finding the difference.
It's possibly just the lack of a ./ in your reference to myscript.sh (which I note you use in your working examples, but not elsewhere)
Have tested under Ubuntu (Trusty64), and Darwnin/BSD - just using the following works fine:
time source ./myscript.sh
FWIW, I used the following script to just confirm that the source command was actually executing correctly:
#!/usr/bin/env sh
V_FIRST=1
export V_SECOND=2
Execution under darwin:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
RBENV_SHELL=bash
V_FIRST=1
V_SECOND=2
Execution under Ubuntu:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
V_FIRST=1
V_SECOND=2

Resources