$ time (exec -a foo echo hello)
hello
It seems as though stderr (where time writes its output) leaks somewhere; obviously this is not what I intended.
My question could be phrased in generic terms as "why isn't the standard error stream written on the terminal when a subshell executes another program?".
A few notes:
I need to use exec for its -a switch, which changes the zeroth argument of the process. I would appreciate an alternative to exec to do just this, but I don't know of any, and now this behavior got me curious.
Of course, I need a subshell because I want my script to continue. Again, any alternative would be welcome. Is exec in a subshell even a good thing to do?
time'ing a subshell in general works fine, so it really has to do with exec.
Could somebody point me in the right direction? I'm not sure where to begin in any of the reference materials, exec descriptions are pretty terse.
Update: Actually, I was just "lucky" with time here being the bash builtin. It doesn't parse at all with /usr/bin/time or with any other process:
$ env (exec -a foo echo hello)
bash: syntax error near unexpected token `exec'
Actually this makes sense, we can't pass a subshell as an argument. Any idea how to do this any other way?
Update: To summarize, we have four good answers here, all different, and potentially something lacking:
Use actual filesystem links (hard or symbolic) that bash will use by default and time normally. Credits to hek2mgl.
ln $(which echo) foo && time ./foo hello && rm foo
fork for time using bash and exec using a bash subshell without special syntax.
time bash -c 'exec -a foo echo hello'
fork for time using bash but exec using a tiny wrapper.
time launch -a foo echo hello
fork and exec for time using bash with special syntax. Credits to sjnarv.
time { (exec -a foo echo hello); }
I think that solution 1 has the less impact on time as the timer doesn't have to count the exec in the "proxy" program, but isn't very practical (many filesystem links) nor technically ideal. In all other cases, we actually exec two times: once to load the proxy program (subshell for 2 and 4, wrapper for 3), and once to load the actual program. This means that time will count the second exec. While it can be extremely cheap, exec actually does filesystem lookups which can be pretty slow (especially if it searches through PATH, either itself with exec*p or if the proxy process does).
So, the only clean way (as far as what the answers of this question covered) would be to patch bash to modify its time keyword so that it can exec while setting the zeroth argument to a non-zero value. It would probably look like time -a foo echo hello.
I don't think that the timer's output disappears. I think it (the timer) was running in the
sub-shell overlaid by the exec.
Here's a different invocation. Perhaps this produces what you expected initially:
$ time { (exec -a foo echo hello); }
Which for me emits:
hello
real 0m0.002s
user 0m0.000s
sys 0m0.001s
Time is based on the wait system call. From the time man page
Most information shown by time is derived from the wait3(2) system call.
This will only work if time is the father process of the command to be executed. But exec creates a completely new process.
As time requires fork() and wait() I would not attach too much attention on that zeroth argument of exec (what is useful, of course). Just create a symbolic link and then call it like:
time link_name > your.file 2>&1 &
So, I ended up writing that tiny C wrapper, which I call launch:
#include <stdlib.h>
#include <unistd.h>
int main(const int argc, char *argv[])
{
int opt;
char *zeroth = NULL;
while ((opt = getopt(argc, argv, "a:")) != -1)
if (opt == 'a')
zeroth = optarg;
else
abort();
if (optind >= argc) abort();
argv += optind;
const char *const program = *argv;
if (zeroth) *argv = zeroth;
return execvp(program, argv);
}
I obviously simplified it to emphasize only what's essential. It essentially works just like exec -a, except that since it is not a builtin, the shell will fork normally to run the launch program as a separate process. There is thus no issue with time.
The test program in the following sample output is a simple program that only outputs its argument vector, one argument per line.
$ ./launch ./test hello world
./test
hello
world
$ ./launch -a foo ./test hello world
foo
hello
world
$ time ./launch -a foo ./test hello world
foo
hello
world
real 0m0.004s
user 0m0.001s
sys 0m0.002s
$ ./launch -a foo -- ./test -g hello -t world
foo
-g
hello
-t
world
The overhead should be minimal: just what's necessary to load the program, parse its single and optional argument, and manipulate the argument vector (which can be mostly reused for the next execvp call).
The only issue is that I don't know of a good way to signal that the wrapper failed (as opposed to the wrapped program) to the caller, which may happen if it was invoked with erroneous arguments. Since the caller probably expects the status code from the wrapped program and since there is no way to reliably reserve a few codes for the wrapper, I use abort which is a bit more rare, but it doesn't feel appropriate (nor does it make it all OK, the wrapped program may still abort itself, making it harder for the caller to diagnose what went wrong). But I digress, that's probably not interesting for the scope of this question.
Edit: just in case, the C compiler flags and feature test macros (gcc/glibc):
CFLAGS=-std=c11 -pedantic -Wall -D_XOPEN_SOURCE=700
Related
I was writing a question, but finally came up with a solution. As it might be useful for others (my future self, at least), here it is.
Context
To run a single command in parallel in several detached screens that automatically close themselves, this works nicely:
timeslots='00_XX 01_XX 02_XX 03_XX 04_XX 05_XX 06_XX'
for timeslot in $timeslots;
do
screen -dmS $timeslot bash -c "echo '$timeslot' >> DUMP";
done
But what if, for each timeslot, we want to execute in screen not one but several (RAM-heavy) commands, one after the other?
We can write a function (in which everything is run sequentially), with an argument in our bash script.
test_function () {
# Commands to be executed sequentially, one at a time:
echo $1 >> DUMP; # technically we'd put heavy things that shouldn't be executed in parallel
echo $1 $1 >> DUMP; # these are just dummy MWE commands
# ETC
}
But, how to create detached screens that run this function with the $timelot argument?
There are lots of discussions on stackoverflow about running a distinct executable script file or on using stuff, but that's not what I want to do. Here the idea is to avoid unnecessary files, keep it all in the same small bash script, simple and clean.
Function definition (in script.sh)
test_function () {
# Commands to be executed sequentially, one at a time:
echo $1 >> DUMP; # technically we'd put heavy things that shouldn't be executed in parallel
echo $1 $1 >> DUMP; # these are just dummy MWE commands
# ETC
}
export -f test_function # < absolutely crucial bit to enable using this with screen
Usage (further down in script.sh)
Now we can do
timeslots='00_XX 01_XX 02_XX 03_XX 04_XX 05_XX 06_XX'
for timeslot in $timeslots;
do
screen -dmS $timeslot bash -c "test_function $timeslot";
done
And it works.
I have a bunch of wrapper shell scripts which manipulate command line arguments and do some stuff before invoking another binary at the end. Is there any reason to not always exec the binary at the end? It seems like this would be simpler and more efficient, but I never see it done.
If you check /usr/bin, you will likely find many many shell scripts that end with an exec command. Just as an example, here is /usr/bin/ps2pdf (debian):
#!/bin/sh
# Convert PostScript to PDF.
# Currently, we produce PDF 1.4 by default, but this is not guaranteed
# not to change in the future.
version=14
ps2pdf="`dirname \"$0\"`/ps2pdf$version"
if test ! -x "$ps2pdf"; then
____ps2pdf="ps2pdf$version"
fi
exec "$ps2pdf" "$#"
exec is used because it eliminates the need for keeping the shell process active after it is no longer needed.
My /usr/bin directory has over 150 shell scripts that use exec. So, the use of exec is common.
A reason not to use exec would be if there was some processing to be done after the binary finished executing.
I disagree with your assessment that this is not a common practice. That said, it's not always the right thing.
The most common scenario where I end a script with the execution of another command, but can't reasonably use exec, is if I need a cleanup hook to be run after the command at the end finishes. For instance:
#!/bin/sh
# create a temporary directory
tempdir=$(mktemp -t -d myprog.XXXXXX)
cleanup() { rm -rf "$tempdir"; }
trap cleanup 0
# use that temporary directory for our program
exec myprog --workdir="$tempdir" "$#"
...won't actually clean up tempdir after execution! Changing that exec myprog to merely myprog has some disadvantages -- continued memory usage from the shell, an extra process-table entry, signals being potentially delivered to the shell rather than to the program that it's executing -- but it also ensures that the shell is still around on myprog's exit to run any traps required.
I am trying to write a task-runner for command line. No rationale. Just wanted to do it. Basically it just runs a command, stores the output in a file (instead of stdout) and meanwhile prints a progress indicator of sorts on stdout and when its all done, prints Completed ($TIME_HERE).
Here's the code:
#!/bin/bash
task() {
TIMEFORMAT="%E"
COMMAND=$1
printf "\033[0;33m${2:-$COMMAND}\033[0m\n"
while true
do
for i in 1 2 3 4 5
do
printf '.'
sleep 0.5
done
printf "\b\b\b\b\b \b\b\b\b\b"
sleep 0.5
done &
WHILE=$!
EXECTIME=$({ TIMEFORMAT='%E';time $COMMAND >log; } 2>&1)
kill -9 $WHILE
echo $EXECTIME
#printf "\rCompleted (${EXECTIME}s)\n"
}
There are some unnecessarily fancy bits in there I admit. But I went through tons of StackOverflow questions to do different kinds of fancy stuff just to try it out. If it were to be applied anywhere, a lot of fat could be cut off. But it's not.
It is to be called like:
task "ping google.com -c 4" "Pinging google.com 4 times"
What it'll do is print Pinging google.com 4 times in yellow color, then on the next line, print a period. Then print another period every .5 seconds. After five periods, start from the beginning of the same line and repeat this until the command is complete. Then it's supposed to print Complete ($TIME_HERE) with (obviously) the time it took to execute the command in place of $TIME_HERE. (I've commented that part out, the current version would just print the time).
The Issue
The issue is that that instead of the execution time, something very weird gets printed. It's probably something stupid I'm doing. But I don't know where that problem originates from. Here's the output.
$ sh taskrunner.sh
Pinging google.com 4 times
..0.00user 0.00system 0:03.51elapsed 0%CPU (0avgtext+0avgdata 996maxresident)k 0inputs+16outputs (0major+338minor)pagefaults 0swaps
Running COMMAND='ping google.com -c 4';EXECTIME=$({ TIMEFORMAT='%E';time $COMMAND >log; } 2>&1);echo $EXECTIME in a terminal works as expected, i.e. prints out the time (3.559s in my case.)
I have checked and /bin/sh is a symlink to dash. (However that shouldn't be a problem because my script runs in /bin/bash as per the shebang on the top.)
I'm looking to learn while solving this issue so a solution with explanation will be cool. T. Hanks. :)
When you invoke a script with:
sh scriptname
the script is passed to sh (dash in your case), which will ignore the shebang line. (In a shell script, a shebang is a comment, since it starts with a #. That's not a coincidence.)
Shebang lines are only interpreted for commands started as commands, since they are interpreted by the system's command launcher, not by the shell.
By the way, your invocation of time does not correctly separate the output of the time builtin from any output the timed command might sent to stderr. I think you'd be better with:
EXECTIME=$({ TIMEFORMAT=%E; time $COMMAND >log.out 2>log.err; } 2>&1)
but that isn't sufficient. You will continue to run into the standard problems with trying to put commands into string variables, which is that it only works with very simple commands. See the Bash FAQ. Or look at some of these answers:
How to escape a variable in bash when passing to command line argument
bash quotes in variable treated different when expanded to command
Preserve argument splitting when storing command with whitespaces in variable
find command fusses on -exec arg
Using an environment variable to pass arguments to a command
(Or probably hundreds of other similar answers.)
Normally I simply call the time program when I want to profile a script or a command it runs.
But this time (no pun intended), I want to measure the time it takes to execute a sourced script, such as:
/usr/bin/time source myscript.sh
But I get the following error:
/usr/bin/time: cannot run source: No such file or directory
And sourcing it this way:
/usr/bin/time . myscript.sh
Simply gives me this error:
/usr/bin/time: cannot run .: Permission denied
I can call it in the following two ways:
/usr/bin/time ./myscript.sh
/usr/bin/time bash myscript.sh
But it's really important that I source my script. Anyone know how I can do this?
Additional Context:
We have many, many scripts which are somewhat complex and these scripts know how to source a specific 'set' of other 'client' scripts. We'll call these scripts 'bundles'.
bundle-a.sh
bundle-b.sh
...
bundle-z.sh
That's not exactly how they are named, but it serves the purpose of showing 26 of them in this example. Each bundle contain a lot of logic and each bundle invoke a specific set of client scripts.
There are hundreds of client scripts, some with a few lines of code and others that are really complex and take hours to execute.
client-000.sh
client-001.sh
...
client-300.sh
We also provide a script which serves as an API library for both our bundle scripts and our client scripts. Many, many functions in this API.
api.sh
So the bundle scripts source the API library and so does each client script. We have it this way so that a bundle can be called (which call a set of client scripts) or at times we only need to call a specific client script directly. It is implemented in a way that if the bundle is executed, the API library is only sourced once (and not again from the client script).
So all of this has worked well for a long time. Its just that now I'm trying to add the /usr/bin/time to each sourcing of our client scripts from the bundle scripts.
If you really, really need to get the system and user times of a sourced script, you can use strace from a different shell to watch the one you're timing in, trace calls to getrusage, and inspect the return values.
Shell 1 (where you will time the script)
$ echo $$
12345
Shell 2 (where you will trace)
$ strace -etrace=getrusage -v -p 12345
Process 12345 attached - interrupt to quit
Shell 1:
$ time : ; . myscript.sh ; time :
real 0m0.000s
(( time output and script output, then time output again)
Shell 2: will output something like
getrusage(RUSAGE_SELF, {ru_utime={0, 12000}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1551, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=1032, ru_nivcsw=3}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 448028}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1552, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=2141, ru_nivcsw=22}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
Notice how the values in the ru_utime structure element changed? The difference there is how much user CPU time the shell itself used. See man getrusage for the details on exactly what you are seeing. From there, it's just a automatically extracting those and finding the difference.
It's possibly just the lack of a ./ in your reference to myscript.sh (which I note you use in your working examples, but not elsewhere)
Have tested under Ubuntu (Trusty64), and Darwnin/BSD - just using the following works fine:
time source ./myscript.sh
FWIW, I used the following script to just confirm that the source command was actually executing correctly:
#!/usr/bin/env sh
V_FIRST=1
export V_SECOND=2
Execution under darwin:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
RBENV_SHELL=bash
V_FIRST=1
V_SECOND=2
Execution under Ubuntu:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
V_FIRST=1
V_SECOND=2
In the following test script I run an elementary coprocess to which the echo built-in, run in background, attaches its standard-output:
#!/bin/bash
# TEST 1
coproc /bin/sleep 100
echo >&${COPROC[1]} &
The script always fails, for no apparent reason, giving the output:
./test.sh: line 4: ${COPROC[1]}: Bad file descriptor
I wonder if the correct syntax should be rather this one (ampersand moved before redirection):
#!/bin/bash
# TEST 2
coproc /bin/sleep 100
echo & >&${COPROC[1]}
This second example seems to work since it reports no errors during execution, but with this syntax, the redirection is not performed in practice; in fact, consider this other test:
#!/bin/bash
# TEST 3
/bin/echo abc & >xfile
Test 3 creates the file xfile, but does not write anything into it. Curiously, trying again to position the ampersand after the redirection make the echo work fine:
#!/bin/bash
# TEST 4
/bin/echo abc >xfile &
Test 4 creates the file xfile with inside the string abc.
Have some idea on what is causing the coproc redirection error or what the correct syntax is?
As noted elsewhere, coproc arranges for its filedescriptors to be closed in subshells. You can get around that using
coproc { whatever; }
exec {WHATEVER[0]}<&${COPROC[0]}- {WHATEVER[1]}>&${COPROC[1]}-
If using Bash prior to version 4.3 you'll have to use separate variables for the input & output variables:
exec {IN}<&${COPROC[0]}- {OUT}>&${COPROC[1]}-
If using Bash prior to 4.1, you'll have to make do with fixed filedescriptor numbers:
exec 4<&${COPROC[0]}- 5>&${COPROC[1]}- ; IN=4 OUT=5
For an interactive shell you might want to consider disown.
This arrangement also has the benefit that you can use more than one coprocess, even though the Bash man page says that it's not supported.
And as discussed elsewhere, be aware of the limitations of sharing pipes between processes.
You've got the answer elsewhere http://lists.gnu.org/archive/html/bug-bash/2012-06/msg00027.html:
Coproc file descriptors are not available to subshells. They're
implemented using pipes, and leaving pipe file descriptors open in
subshells causes processes to hang and not terminate properly, which
results in very hard-to-track-down-and-reproduce bugs.