Normally I simply call the time program when I want to profile a script or a command it runs.
But this time (no pun intended), I want to measure the time it takes to execute a sourced script, such as:
/usr/bin/time source myscript.sh
But I get the following error:
/usr/bin/time: cannot run source: No such file or directory
And sourcing it this way:
/usr/bin/time . myscript.sh
Simply gives me this error:
/usr/bin/time: cannot run .: Permission denied
I can call it in the following two ways:
/usr/bin/time ./myscript.sh
/usr/bin/time bash myscript.sh
But it's really important that I source my script. Anyone know how I can do this?
Additional Context:
We have many, many scripts which are somewhat complex and these scripts know how to source a specific 'set' of other 'client' scripts. We'll call these scripts 'bundles'.
bundle-a.sh
bundle-b.sh
...
bundle-z.sh
That's not exactly how they are named, but it serves the purpose of showing 26 of them in this example. Each bundle contain a lot of logic and each bundle invoke a specific set of client scripts.
There are hundreds of client scripts, some with a few lines of code and others that are really complex and take hours to execute.
client-000.sh
client-001.sh
...
client-300.sh
We also provide a script which serves as an API library for both our bundle scripts and our client scripts. Many, many functions in this API.
api.sh
So the bundle scripts source the API library and so does each client script. We have it this way so that a bundle can be called (which call a set of client scripts) or at times we only need to call a specific client script directly. It is implemented in a way that if the bundle is executed, the API library is only sourced once (and not again from the client script).
So all of this has worked well for a long time. Its just that now I'm trying to add the /usr/bin/time to each sourcing of our client scripts from the bundle scripts.
If you really, really need to get the system and user times of a sourced script, you can use strace from a different shell to watch the one you're timing in, trace calls to getrusage, and inspect the return values.
Shell 1 (where you will time the script)
$ echo $$
12345
Shell 2 (where you will trace)
$ strace -etrace=getrusage -v -p 12345
Process 12345 attached - interrupt to quit
Shell 1:
$ time : ; . myscript.sh ; time :
real 0m0.000s
(( time output and script output, then time output again)
Shell 2: will output something like
getrusage(RUSAGE_SELF, {ru_utime={0, 12000}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1551, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=1032, ru_nivcsw=3}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 448028}, ru_stime={0, 16001}, ru_maxrss=2364, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1552, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=2141, ru_nivcsw=22}) = 0
getrusage(RUSAGE_CHILDREN, {ru_utime={0, 0}, ru_stime={0, 4000}, ru_maxrss=1216, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=5143, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=57, ru_nivcsw=21}) = 0
Notice how the values in the ru_utime structure element changed? The difference there is how much user CPU time the shell itself used. See man getrusage for the details on exactly what you are seeing. From there, it's just a automatically extracting those and finding the difference.
It's possibly just the lack of a ./ in your reference to myscript.sh (which I note you use in your working examples, but not elsewhere)
Have tested under Ubuntu (Trusty64), and Darwnin/BSD - just using the following works fine:
time source ./myscript.sh
FWIW, I used the following script to just confirm that the source command was actually executing correctly:
#!/usr/bin/env sh
V_FIRST=1
export V_SECOND=2
Execution under darwin:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
RBENV_SHELL=bash
V_FIRST=1
V_SECOND=2
Execution under Ubuntu:
$ time source ./test.sh ; set | grep -i V_
real 0m0.000s
user 0m0.000s
sys 0m0.000s
V_FIRST=1
V_SECOND=2
Related
I have this need to collect\log all the command lines that were used to start a process on my machine during the execution of a Perl script which happens to be a test automation script. This Perl script starts the executable in question (MySQL) multiple times with various command lines and I would like to inspect all of the command lines of those invocations. What would be the right way to do this? One possibility i see is run something like "ps -aux | grep mysqld | grep -v grep" in a loop in a shell script and capture the results in a file but then I would have to do some post processing on this and remove duplicates etc and I could possibly miss some process command lines because of timing issues. Is there a better way to achieve this.
Processing the ps output can always miss some processes. It will only capture the ones currently existing. The best way would be to modify the Perl script to log each command before or after it executes it.
If that's not an option, you can get the child pids of the perl script by running:
pgrep -P $pid -a
-a gives the full process command. $pid is the pid of the perl script. Then process just those.
You could use strace to log calls to execve.
$ strace -f -o strace.out -e execve perl -e 'system("echo hello")'
hello
$ egrep ' = 0$' strace.out
11232 execve("/usr/bin/perl", ["perl", "-e", "system(\"echo hello\")"], 0x7ffc6d8e3478 /* 55 vars */) = 0
11233 execve("/bin/echo", ["echo", "hello"], 0x55f388200cf0 /* 55 vars */) = 0
Note that strace.out will also show the failed execs (where execve returned -1), hence the egrep command to find the successful ones. A successful execve call does not return, but strace records it as if it returned 0.
Be aware that this is a relatively expensive solution because it is necessary to include the -f option (follow forks), as perl will be doing the exec call from forked subprocesses. This is applied recursively, so it means that your MySQL executable will itself run through strace. But for a one-off diagnostic test it might be acceptable.
Because of the need to use recursion, any exec calls done from your MySQL executable will also appear in the strace.out, and you will have to filter those out. But the PID is shown for all calls, and if you were to log also any fork or clone calls (i.e. strace -e execve,fork,clone), you would see both the parent and child PIDs, in the form <parent_pid> clone(......) = <child_pid> so then you should hopefully then have enough information to reconstruct the process tree and decide which processes you are interested in.
I have inherited an application which is written in dozens (maybe hundreds, I haven't counted exactly) of PostgreSQL functions. In order to check the application code into git and be able to easily work on specific functions, I used pg_extractor to export the database into a separate file for each function.
In order to easily apply updates from git (both on developer machines and in production), I wrote a bash script that uses the psql command line client to run all of the function files, which causes the database server to be updated to match the files from git.
The gist of it looks like this (with some initialization code replaced by comments for brevity):
#!/bin/bash
# Check if a .env file is present and load it to set the PGHOST, PGPORT, PGUSER, PGPASSWORD, and PGDATABASE environment variables
# Check that `psql` is on the PATH
# Check the the database set in PGDATABASE exists on the server
GREEN='\033[0;32m'
RED='\033[0;31m'
NC='\033[0m' # No Color
makeFunction () {
echo -n "CREATE OR REPLACE FUNCTION $2"
psql -q -x < "$DIR/$1/functions/$2.sql" > /dev/null
if [ $? -eq 0 ]; then
echo -e " - ${GREEN}COMPLETE${NC}"
else
echo -e " - ${RED}FAILED${NC}"
exit 1
fi
}
for schema in admin admin_internal main main_internal
do
if [ -d "$DIR/$schema/functions" ]; then
for function in $DIR/$schema/functions/*.sql
do
makeFunction $schema $(basename "$function" .sql)
done
fi
done
On most of our Linux machines (development and production, Ubuntu 16.04 and 18.04) this script takes 15-20 seconds. Example:
real 0m14.324s
user 0m6.894s
sys 0m1.742s
However, on our Windows development machines (when run using git-bash) it usually takes around three minutes to run the same script. Example:
real 3m0.825s
user 0m3.525s
sys 0m11.943s
(Thinking the issue might be with Bash on Windows, I tried converting the script to PowerShell, only to see the same issue. Thankfully I saw that it wouldn't make a difference while doing partial testing before spending too much time on it.)
It turns out that the problem is in actually making the connection to the PostgreSQL server. For example, using time psql -lqt to time listing all databases on the server (these are example numbers, but dozens of test runs have shown that they are consistently the similar to these):
On Ubuntu:
real 0m0.055s
user 0m0.032s
sys 0m0.020s
On Windows:
real 0m0.852s
user 0m0.000s
sys 0m0.030s
As you can see, it takes 15 times longer on Windows. Extend that out to all the times we are calling psql in the update script, and it's no wonder that it takes 9 times longer to run the full script on Windows than on Linux.
It is well known that Postgres performance will never be as good on Windows as Linux because of the one-process-per-connection model and the lack of fork() on Windows, but that should be a bottleneck in the creation of the connection, not in the execution of the commands. (That the bottleneck is in the connection and not the query execution is evidenced by the fact that a single example command is consistently 15x slower, but the whole script with much larger queries being run is only 9-12x slower.)
Is there a way to make this faster for our Windows users? Is there some way to reuse an existing psql session and pipe additional files into it? Or is my only option to rewrite this in some other language and write my own database communication code that reads the files and pipes them to PostgreSQL myself?
Running the connection through PgBouncer instead of directly to Postgres makes a huge difference.
Using it, my script run on Windows is reduced to around 30 seconds.
It's still not quite as fast as on Linux, but I can live with "only" a 6x improvement.
real 0m33.232s
user 0m2.740s
sys 0m9.785s
In a bash script (running on REHL5) I use the /usr/bin/time command (not the builtin time command) to record the run times of other commands I run in that same script. The problem I am facing is that some of the commands that I want to record times for are not builtin commands or external scripts but are functions that are either declared in the script or are sourced from a different script. I have found out that the time command fails with error like below:
/usr/bin/time: cannot run shared-func: No such file or directory
Which means that the function shared-func() that I declare elsewhere is not visible in the scope and therefore time cannot run that command. I have run some tests and have verified that the reason behind this error is in fact because the time command tries to execute its command in a new subshell and therefore loses every declared function or variable in its scope. Is there a way to get around this? The ideal solution would be to force the time command to change its behavior and use the current shell for executing its command but I am also interested in any other solutions if that is not possible.
For the record, below is the test I ran. I created two small scripts:
shared.sh:
function shared-func() {
echo "The shared function is visible."
}
test.sh:
#!/bin/bash
function record-timestamp() {
/usr/bin/time -f % -a -o timestamps.csv "$#"
}
source "shared.sh"
record-timestamp shared-func
And this is the test:
$ ./test.sh
/usr/bin/time: cannot run shared-func: No such file or directory
$
A different process, yes. A subshell, no.
A subshell is what you get when your parent shell forks but doesn't exec() -- it's a new process made by copying your current shell instance. Functions are accessible within subshells, though they can't have direct effect on the parent shell (changes to process state die with the subshell when it exits).
When you launch an external program without using exec, the shell first forks, but then calls execve() to invoke the new program. execve() replaces the process image in memory with the program being run -- so it's not the fork(), that is, the creation of the subshell causing this to fail; instead, it's the exec(), the invocation of a separate program.
Even if your new process is also a shell, if it's gone through any exec()-family call it's not a subshell -- it's a whole new process.
tl;dr: You cannot use an external program to wrap a shell function invocation inside the current shell, because an external program's invocation always uses execve(), and execve() always clears process state -- including non-exported shell functions.
function shared-func() {
echo "The shared function is visible."
}
export -f shared-func
echo shared-func | /usr/bin/time -lp /bin/bash
Output:
cat <<EOF
The shared function is visible.
real 0.00
user 0.00
sys 0.00
1040384 maximum resident set size
...
On a 3GHz machine, the overhead of running bash this way is approximately 5ms (u+s time).
/bin/time is a separate program so it will be run in a separate process, and then it will try to run yet another program in yet another separate process.
There's a trick though, a shell script can call itself, or in this case tell /bin/time to call itself.
function record-timestamp() {
/usr/bin/time -f % -a -o timestamps.csv bash "$0" "$#"
}
if [ -n "$1" ]; then
source "shared.sh"
"$#"
else
record-timestamp shared-func
fi
If you call test.sh with arguments then it will try to run that function (which comes from shared.sh), and if you call it without arguments will call the record-timestamp function which in turn will call /bin/tine which in turn will call test.sh with arguments.
I'd like to redirect the time of a process to a file, and the process itself redirects to a file. Here's what I've tried:
time &> ltime.txt echo | ls -l > ls-l.txt
Both files (ltime.txt and ls-l.txt) are created, but ltime.txt is empty.
time still outputs to the screen,
which is fine in my case,
but I expected it to not do that because of the &>:
real 0m0.034s
user 0m0.000s
sys 0m0.004s
(All that I know about this came from http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html)
Any suggestions?
Maybe you're getting bitten by the time that's built-in to the shell? Try this:
/usr/bin/time ls -l > ls-l.txt 2> ltime.txt
From man time:
Users of the bash shell need to use an explicit path in order to run
the external time command and not the shell builtin variant.
[Updated to add]
Apparently the location of time isn't totally standard. For a one-off use (and for general info) which time will tell you where it is. For a portable script you can say command time .... (command is a shell built-in that tells the shell to ignore built-ins.)
Try
{ time ls -l > ls-l.txt ; } &> ltime.txt
The curly braces capture the output even with the builtin.
$ time (exec -a foo echo hello)
hello
It seems as though stderr (where time writes its output) leaks somewhere; obviously this is not what I intended.
My question could be phrased in generic terms as "why isn't the standard error stream written on the terminal when a subshell executes another program?".
A few notes:
I need to use exec for its -a switch, which changes the zeroth argument of the process. I would appreciate an alternative to exec to do just this, but I don't know of any, and now this behavior got me curious.
Of course, I need a subshell because I want my script to continue. Again, any alternative would be welcome. Is exec in a subshell even a good thing to do?
time'ing a subshell in general works fine, so it really has to do with exec.
Could somebody point me in the right direction? I'm not sure where to begin in any of the reference materials, exec descriptions are pretty terse.
Update: Actually, I was just "lucky" with time here being the bash builtin. It doesn't parse at all with /usr/bin/time or with any other process:
$ env (exec -a foo echo hello)
bash: syntax error near unexpected token `exec'
Actually this makes sense, we can't pass a subshell as an argument. Any idea how to do this any other way?
Update: To summarize, we have four good answers here, all different, and potentially something lacking:
Use actual filesystem links (hard or symbolic) that bash will use by default and time normally. Credits to hek2mgl.
ln $(which echo) foo && time ./foo hello && rm foo
fork for time using bash and exec using a bash subshell without special syntax.
time bash -c 'exec -a foo echo hello'
fork for time using bash but exec using a tiny wrapper.
time launch -a foo echo hello
fork and exec for time using bash with special syntax. Credits to sjnarv.
time { (exec -a foo echo hello); }
I think that solution 1 has the less impact on time as the timer doesn't have to count the exec in the "proxy" program, but isn't very practical (many filesystem links) nor technically ideal. In all other cases, we actually exec two times: once to load the proxy program (subshell for 2 and 4, wrapper for 3), and once to load the actual program. This means that time will count the second exec. While it can be extremely cheap, exec actually does filesystem lookups which can be pretty slow (especially if it searches through PATH, either itself with exec*p or if the proxy process does).
So, the only clean way (as far as what the answers of this question covered) would be to patch bash to modify its time keyword so that it can exec while setting the zeroth argument to a non-zero value. It would probably look like time -a foo echo hello.
I don't think that the timer's output disappears. I think it (the timer) was running in the
sub-shell overlaid by the exec.
Here's a different invocation. Perhaps this produces what you expected initially:
$ time { (exec -a foo echo hello); }
Which for me emits:
hello
real 0m0.002s
user 0m0.000s
sys 0m0.001s
Time is based on the wait system call. From the time man page
Most information shown by time is derived from the wait3(2) system call.
This will only work if time is the father process of the command to be executed. But exec creates a completely new process.
As time requires fork() and wait() I would not attach too much attention on that zeroth argument of exec (what is useful, of course). Just create a symbolic link and then call it like:
time link_name > your.file 2>&1 &
So, I ended up writing that tiny C wrapper, which I call launch:
#include <stdlib.h>
#include <unistd.h>
int main(const int argc, char *argv[])
{
int opt;
char *zeroth = NULL;
while ((opt = getopt(argc, argv, "a:")) != -1)
if (opt == 'a')
zeroth = optarg;
else
abort();
if (optind >= argc) abort();
argv += optind;
const char *const program = *argv;
if (zeroth) *argv = zeroth;
return execvp(program, argv);
}
I obviously simplified it to emphasize only what's essential. It essentially works just like exec -a, except that since it is not a builtin, the shell will fork normally to run the launch program as a separate process. There is thus no issue with time.
The test program in the following sample output is a simple program that only outputs its argument vector, one argument per line.
$ ./launch ./test hello world
./test
hello
world
$ ./launch -a foo ./test hello world
foo
hello
world
$ time ./launch -a foo ./test hello world
foo
hello
world
real 0m0.004s
user 0m0.001s
sys 0m0.002s
$ ./launch -a foo -- ./test -g hello -t world
foo
-g
hello
-t
world
The overhead should be minimal: just what's necessary to load the program, parse its single and optional argument, and manipulate the argument vector (which can be mostly reused for the next execvp call).
The only issue is that I don't know of a good way to signal that the wrapper failed (as opposed to the wrapped program) to the caller, which may happen if it was invoked with erroneous arguments. Since the caller probably expects the status code from the wrapped program and since there is no way to reliably reserve a few codes for the wrapper, I use abort which is a bit more rare, but it doesn't feel appropriate (nor does it make it all OK, the wrapped program may still abort itself, making it harder for the caller to diagnose what went wrong). But I digress, that's probably not interesting for the scope of this question.
Edit: just in case, the C compiler flags and feature test macros (gcc/glibc):
CFLAGS=-std=c11 -pedantic -Wall -D_XOPEN_SOURCE=700