I'm trying to write a script that will run and time a given and output that to a file in a .csv format.
So far from looking at SO previous posts, I've found that sh -c "$index_of_command_arg" can be used to invoke that command.
I'm also familiar with time and I know that people use /usr/bin/time for formatting, but I need to format the time given in total seconds (for example, 1.34516) but the only given option to format the real time is %E which return [hours:]minutes:seconds. Is there any way to format it the way I need?
The general idea of my script is:
# ----
# some input validation
# ----
rule=$1
command=$2
execution_time=/usr/bin/time -f "%total_seconds" sh -c "$command" #is this line possible?
echo "$rule,$execution_time" > output_file.csv
Can this be formatted the way I want? and also, the line with the comment after it,
Will this even work the way I wrote it? is the syntax correct?
Say I use the normal time and I get the real 0m2.003 ...
output, how can I take the 2.003 out of it?
The normal time you are mentioning is the bash built-in time. From the Bash Reference Manual:
The TIMEFORMAT variable may be set to a format string
that specifies how the timing information should be displayed.
…
%[p][l]R
The elapsed time in seconds.
So, you could use
execution_time=`TIMEFORMAT=%R bash -c "time $command" 2>&1 >/dev/tty`
You replace:
execution_time=/usr/bin/time -f "%total_seconds" sh -c "$command" `
with:
execution_time=`(time sh -c "$command > /dev/null 2>&1") 2>&1 | grep real |sed "s/.*m//;s/s.*//;"`
or with:
execution_time=`(time sh -c "$command > /dev/null 2>&1") 2>&1 | grep real | cut -c 8-12`
Why > /dev/null - you don't want to see $command output 2>&1 (STDERR > STDOUT)
Why 2>&1 for time command (for the same reason as in previous bullet)
Related
I have some command which launches in this way:
./mycommand -arg=word
I measure execution time and write output to file:
/usr/bin/time -f "%K %e" ./mycommand -arg=word 2>file
However, mycommand has also stderr output. So file contains mixed data from both (time and mycommand). How can I suppress mycommand output and save time data?
If the nature of your process is such that a shell startup won't interfere with the timing too much, one simple approach is to have a shell run the direction as a subprocess of time:
/usr/bin/time -f "%K %e" sh -c '"$0" "$#" >/dev/null 2>&1' ./mycommand -arg=word
I would also suggest using the bash-builtin version of time instead of the external one, which lets you use tricks documented in BashFAQ #32:
TIMEFORMAT='%R' # replace for whatever is equivalent to your "%K %e"
timing=$(time { ./mycommand >/dev/null 2>&1; } 2>&1)
I have this line of code:
{ time cp $PWD/my_file $PWD/new_file ; } 2> my_log.log
I need to know how long it takes to execute the 'cp' command and I also need to get the PID of the 'cp'. I just want to print the PID of the 'cp' process and get the following in the my_log.log file:
<output of time>
I have tried PID=$! but this does not provide PID of the cp process.
First, you need to send your (timed) cp command to the background with a trailing &, so you can inspect the running processes after launching it.
(I suspect you're already doing this, but it's not currently reflected in the question).
$!, the special variable that contains the PID of the most recently launched background job, in this case reflects the subshell that runs the time command, so we know that it is the parent process of the cp command. To get the (one and only, in this case) child process:
If your platform has the nonstandard pgrep utility (comes with many Linux distros and BSD/macOS platforms), use:
pgrep -P $!
Otherwise, use the following POSIX-compliant approach:
ps -o pid=,ppid= | awk -v ppid=$! '$2 == ppid { print $1 }'
To put it all together, using prgep for convenience:
# Send the timed `cp` command to the background with a trailing `&`
{ time cp "$PWD/my_file" "$PWD/new_file"; } 2> my_log.log &
# Get the `cp` comand's PID via its parent PID, $!
cpPid=$(pgrep -P $!)
To know approximately how long takes cp command you can check new file size
size=$(stat -c %s "${old_file}")
cp "${old_file}" "${new_file}" &
cp_pid=$!
while kill -0 ${cp_pid}; do
cpsize=$(stat -c %s "${new_file}")
echo elapsed time $(ps -p${cp_pid} -oetime=)
echo $((100*cpsize/size)) % done so far..
sleep 3
done
EDIT: following comment stat -c %s "${file}" can be replaced by du "${file}" it's POSIX and more suitable command (see man page).
The simplest one I can think of is
pgrep cp
OK -- from the comments: "Contents of my_log.log will be PID of the cp command followed by the timing output of the cp command":
( time cp $PWD/my_file $PWD/new_file & 2>&1; echo $! ) > my_log.log 2>&1
First, you need to use /usr/bin/time explicitly, and pass the options to append to an output file. Then, use pgrep on the name of the file you are copying (cp will get too many hits):
/usr/bin/time --output=my_log.log --append cp $PWD/my_file $PWD/new_file & pgrep -f my_file > my_log.log
You may want to change the output format, because it's kinda ugly:
18400
0.00user 0.30system 0:02.43elapsed 12%CPU (0avgtext+0avgdata 2520maxresident)k
0inputs+496424outputs (0major+141minor)pagefaults 0swaps
Following what written in one of your comment above... the following is a proper answer:
The following code (just an example):
time (sleep 10 & echo -n "$!"; wait)
will return something like:
30406
real 0m10.009s
user 0m0.004s
sys 0m0.005s
In your case:
time (cp $PWD/old_file $PWD/new_file & echo -n "$!"; wait) &> my_log.log
will do the job.
I find this solution quite elegant since it's a "one liner" despite the "completely negligible overhead" you got in the timing (the timing will be related to the entire subshell (also the echo and the wait). That the overhead is negligible is evident from the result of the sleep command.
The &> redirect stdout and stderr to the same file (so you do not need to specify the 1>&2).
Pay attention that doing
(time sleep 10 & echo -n "$!")
You will get the pid of the time process not sleep or cp in your case.
I'm trying to redirect stdin to a FIFO with bash. This way, I will be able to use this stdin on an other part of the script.
However, it doesn't seem to work as I want
script.bash
#!/bin/bash
rm /tmp/in -f
mkfifo /tmp/in
cat >/tmp/in &
# I want to be able to reuse /tmp/in from an other process, for example :
xfce4-terminal --hide-menubar --title myotherterm --fullscreen -x bash -i -c "less /tmp/in"
Here I would expect , when I run ls | ./script.bash, to see the output of ls, but it doesn't work (eg the script exits, without outputing anything)
What am I misunderstanding ?
I am pretty sure that less need additional -f flag when reading from pipe.
test_pipe is not a regular file (use -f to see it)
If that does not help I would also recommend to change order between last two lines of your script:
#!/bin/bash
rm /tmp/in -f
mkfifo /tmp/in
xfce4-terminal --hide-menubar --title myotherterm --fullscreen -x bash -i -c "less -f /tmp/in" &
cat /dev/stdin >/tmp/in
In general, I avoid the use of /dev/stdin, because I get a lot of surprises from what is exactly /dev/stdin, especially when using redirects.
However, what I think that you're seeing is that less finishes before your terminal is completely started. When the less ends, so will the terminal and you won't get any output.
As an example:
xterm -e ls
will also not really display a terminal.
A solution might be tail -f, as in, for example,
#!/bin/bash
rm -f /tmp/in
mkfifo /tmp/in
xterm -e "tail -f /tmp/in" &
while :; do
date > /tmp/in
sleep 1
done
because the tail -f remains alive.
Just a little question about timing programs on Linux: the time command allows to
measure the execution time of a program:
[ed#lbox200 ~]$ time sleep 1
real 0m1.004s
user 0m0.000s
sys 0m0.004s
Which works fine. But if I try to redirect the output to a file, it fails.
[ed#lbox200 ~]$ time sleep 1 > time.txt
real 0m1.004s
user 0m0.001s
sys 0m0.004s
[ed#lbox200 ~]$ cat time.txt
[ed#lbox200 ~]$
I know there are other implementations of time with the option -o to write a file but
my question is about the command without those options.
Any suggestions ?
Try
{ time sleep 1 ; } 2> time.txt
which combines the STDERR of "time" and your command into time.txt
Or use
{ time sleep 1 2> sleep.stderr ; } 2> time.txt
which puts STDERR from "sleep" into the file "sleep.stderr" and only STDERR from "time" goes into "time.txt"
Simple. The GNU time utility has an option for that.
But you have to ensure that you are not using your shell's builtin time command, at least the bash builtin does not provide that option! That's why you need to give the full path of the time utility:
/usr/bin/time -o time.txt sleep 1
Wrap time and the command you are timing in a set of brackets.
For example, the following times ls and writes the result of ls and the results of the timing into outfile:
$ (time ls) > outfile 2>&1
Or, if you'd like to separate the output of the command from the captured output from time:
$ (time ls) > ls_results 2> time_results
If you care about the command's error output you can separate them like this while still using the built-in time command.
{ time your_command 2> command.err ; } 2> time.log
or
{ time your_command 2>1 ; } 2> time.log
As you see the command's errors go to a file (since stderr is used for time).
Unfortunately you can't send it to another handle (like 3>&2) since that will not exist anymore outside the {...}
That said, if you can use GNU time, just do what #Tim Ludwinski said.
\time -o time.log command
Since the output of 'time' command is error output, redirect it as standard output would be more intuitive to do further processing.
{ time sleep 1; } 2>&1 | cat > time.txt
If you are using GNU time instead of the bash built-in, try
time -o outfile command
(Note: GNU time formats a little differently than the bash built-in).
&>out time command >/dev/null
in your case
&>out time sleep 1 >/dev/null
then
cat out
I ended up using:
/usr/bin/time -ao output_file.txt -f "Operation took: %E" echo lol
Where "a" is append
Where "o" is proceeded by the file name to append to
Where "f" is format with a printf-like syntax
Where "%E" produces 0:00:00; hours:minutes:seconds
I had to invoke /usr/bin/time because the bash "time" was trampling it and doesn't have the same options
I was just trying to get output to file, not the same thing as OP
If you don't want to touch the original process' stdout and stderr, you can redirect stderr to file descriptor 3 and back:
$ { time { perl -le "print 'foo'; warn 'bar';" 2>&3; }; } 3>&2 2> time.out
foo
bar at -e line 1.
$ cat time.out
real 0m0.009s
user 0m0.004s
sys 0m0.000s
You could use that for a wrapper (e.g. for cronjobs) to monitor runtimes:
#!/bin/bash
echo "[$(date)]" "$#" >> /my/runtime.log
{ time { "$#" 2>&3; }; } 3>&2 2>> /my/runtime.log
#!/bin/bash
set -e
_onexit() {
[[ $TMPD ]] && rm -rf "$TMPD"
}
TMPD="$(mktemp -d)"
trap _onexit EXIT
_time_2() {
"$#" 2>&3
}
_time_1() {
time _time_2 "$#"
}
_time() {
declare time_label="$1"
shift
exec 3>&2
_time_1 "$#" 2>"$TMPD/timing.$time_label"
echo "time[$time_label]"
cat "$TMPD/timing.$time_label"
}
_time a _do_something
_time b _do_another_thing
_time c _finish_up
This has the benefit of not spawning sub shells, and the final pipeline has it's stderr restored to the real stderr.
If you are using csh you can use:
/usr/bin/time --output=outfile -p $SHELL -c 'your command'
For example:
/usr/bin/time --output=outtime.txt -p csh -c 'cat file'
If you want just the time in a shell variable then this works:
var=`{ time <command> ; } 2>&1 1>/dev/null`
I'm trying to log the time I spend working in vim. I've got a script that works with gvim but when I try to set it up with vim it locks up the terminal session silently or with the message 'Vim: Warning: Output is not to a terminal'
Here is the script that works with gvim:
#!/bin/sh
workfile="/home/na/writing/fiction.txt"
worklog="/home/na/writing/worktime.log"
d=`date --rfc-3339 date`
t=$( { /usr/bin/time -f "%e" /usr/local/bin/gvim -f -S /home/na/.vim/writeroom/writeroom.vim $workfile; } 2>&1 )
w=`wc -w $workfile`
echo $d $t $w >> $worklog
When I close the gvim window I get a logfile containing a date, the number of seconds I spent editing the file, and a word count for the file.
2011-08-15 700.15 238869 /home/na/writing/fiction.txt
I would like the same using vim in a terminal session.
I understand vim talks to the terminal directly instead of to stdout but I don't care about what vim returns, I want the output from the time command.
These don't work:
t=`/usr/bin/time -f "%e" /usr/local/bin/vim -f $workfile`
t=$( { /usr/bin/time -f "%e" /usr/local/bin/vim -f $workfile; } 2>&1 )
t=$( { /usr/bin/time -f "%e" /usr/local/bin/vim -f $workfile; } )
I suspect there's some combination of backticks and paranthesis that will make this work but I haven't stumbled onto it yet.
The backtick or $() operators capture vim's output (what you see on the terminal when opening vim is vim's output from its stdout), so you can't do that.
You could try this instead:
start=$(date +%s)
/usr/local/bin/vim -f $workfile
end=$(date +%s)
duration=$((end-start))
echo "$duration"
Some versions of time (eg: GNU's) support a flag to set the output file. eg:
/usr/bin/time -o /tmp/foo vim
If you're using bash you'll need to use the path for time (or a leading \) as bash has a reserved word time which behaves similar to the time command (but operates on an entire pipeline) that does not support the -o flag.