I have a bash script, call it Exp, which performs a computational experiment, and I want to
have the result of
time Exp
within the script itself: First it needs to be always done (and relying on typing "time Exp"
is not enough -- for the critical case where you (or the user!!) needs it, it will be
forgotten), and second the script Exp itself needs to store it in a file.
Writing a wrapper script (which calls "time Exp") seems to make ordinary work with
Exp impossible due to the destruction of parameters and input/output by the time-command.
But actually all what is needed is to access the data in Exp itself of that universal
record (which can also be accessed by ps) which is just printed by time!
That is why I ask for an "elegant" solution, not just first storing somehow the date
in Exp, and finally, before exit, computing the difference. But just simulating what
the time-command is doing in Exp. I think that would be useful in many other
situations.
Since POSIX defines the time utility to write its result on standard error, the only trick you have to be aware of is that there is a bash built-in time that behaves slightly differently.
$ time sleep 1
real 0m1.004s
user 0m0.001s
sys 0m0.002s
$ time sleep 1 2>xyz
real 0m1.005s
user 0m0.001s
sys 0m0.003s
$ (time sleep 1) 2>xyz
$ cat xyz
real 0m1.005s
user 0m0.001s
sys 0m0.002s
$ /usr/bin/time sleep 1 2>xyz
$ cat xyz
1.00 real 0.00 user 0.00 sys
$
Testing shown on MacOS X 10.7. Note the difference in output format between built-in and external versions of the time command. Note also that in the sub-shell format, the built-in time is redirected normally, but in the simple case, redirecting the output after the built-in time does not send the output to the same place that the rest of the standard error goes to.
So, these observations allow you to write your elegant script. Probably the simplest method is to wrap your existing script as a function, and then invoke that function via the built-in time.
Original script
# This is what was in the Exp script before.
# It echoes its name and its arguments to both stdout and stderr
echo "Exp $*"
echo "Exp $*" 1>&2
Revised script
log=./Exp.log
Exp()
{
# This is what was in the Exp script before.
# It echoes its name and its arguments to both stdout and stderr
echo "Exp $*"
echo "Exp $*" 1>&2
}
(time Exp "$#") 2>> $log
Note the careful use of "$#" in the invocation of the Exp function to preserve the separate arguments passed on the command line, and the equally deliberate use of $* inside the function (which loses the separate arguments, but is only for illustrative purposes).
Possible Issue
The only issue with this is that the error output from the original command also goes to the same log file as timing information. That can be resolved but involves tricky multiple redirections that are more likely to confuse than help. However, for the record, this works nicely:
log=./Exp.log
Exp()
{
# This is what was in the Exp script before.
# It echoes its name and its arguments to both stdout and stderr
echo "Exp $*"
echo "Exp $*" 1>&2
}
exec 3>&2
(time Exp "$#" 2>&3 ) 2>> $log
You can just add the following as the first line of the script:
test -z "$TIMED" && TIMED=yes exec time $0 $#
This will not run time if TIMED is set in the environment, so
it gives you a way to suppress the timing if you want.
The above two solutions invoke the time-command. I have my doubts that even the very
elaborated one by #Jonathan Leffler is really functionally equivalent the original script:
it seems to take care about output to standard output and standard error, but how
it behaves w.r.t. symbolic links one needed to test (and that won't be so simple ---
there are many subtleties regarding paths, especially when containing links).
And I think w.r.t. the nasty quoting-business it definitely changes the semantics
of the original script, making it necessary to add one quotation layer more to the parameters (if for example the script runs remotely, and one needs to have quotation marks).
With these two issues, handling of paths and symbolic links and quotation, we had
a lot of trouble in the past, and these errors are very hard to catch: often the
scripts we write use complicated other scripts from many mathematical/computer science packages, where literally hundreds of especially installed packages have to be handled,
each with its own strange specialities, and so we want to avoid as much as possible
adding further complications.
The solution, which I found with the help of #Arne, is simpler; see
How to use the S-output-modifier with Unix/Linux command ps?
One just needs to add the line
ps p $$ k time S | tail -n 1 | tr -s '[:space:]' | cut -d ' ' -f 4 > log-file
to the script when one wants to store the (total) process-time in log-file.
Don't know where the time-command gets the wall-clock and the system-time, but
for the wall-clock perhaps one needs to do subtraction of time; and don't know
about the system-time, but it must be somewhere.
Related
I time a command that has some output. I want to output the real time from the time command to a file, but leave the output of the command to the console.
For example, if I do time my_command I get this printed in the console:
several lines of output from my_command ...
real 1m25.970s
user 0m0.427s
sys 0m0.518s
In this case, I want to store only 1m25.970s to a file, but still print the output of the command to the console.
The time command is tricky. The POSIX specification of time
doesn't define the default output format, but does define a format for the -p (presumably for 'POSIX') option. Note the (not easily understood) discussion of command sequences in pipelines.
The Bash specification say time prefixes a 'pipeline', which means that time cmd1 | cmd2 times both cmd1 and cmd2. It writes its results to standard error. The Korn shell is similar.
The POSIX format requires a single space between the tags such as real and the time; the default format often uses a tab instead of a space. Note that the /usr/bin/time command may have yet another output format. It does on macOS, for example, listing 3 times on a single line, by default, with the label after the time value; it supports -p to print in an approximation to the POSIX format (but it has multiple spaces between label and time).
You can easily get all the information written to standard error into a file:
(time my_command) 2> log.file
If my_command or any programs it invokes reports any errors to standard error, those will got to the log file too. And you will get all three lines of the output from time written to the file.
If your shell is Bash, you may be able to use process substitution to filter some of the output.
I wouldn't try it with a single command line; the hieroglyphs needed to make it work are ghastly and best encapsulated in shell scripts.
For example, a shell script time.filter to capture the output from time and write only the real time to a log file (default log.file, configurable by providing an alternative log file name as the first argument
#!/bin/sh
output="${1:-log.file}"
shift
sed -E '/^real[[:space:]]+(([0-9]+m)?[0-9]+[.][0-9]+s?)/{ s//\1/; w '"$output"'
d;}
/^(user|sys)[[:space:]]+(([0-9]+m)?[0-9]+[.][0-9]+s?)/d' "$#"
This assumes your sed uses -E to enable extended regular expressions.
The first line of the script finds the line containing the real label and the time after it (in a number of possible formats — but not all). It accepts an optional minutes value such as 60m05.003s, or just a seconds value 5.00s, or just 5.0 (POSIX formats — at least one digit after the decimal point is required). It captures the time part and prints it to the chosen file (by default, log.file; you can specify an alternative name as the first argument on the command line). Note that even GNU sed treats everything after the w command as file name; you have to continue the d (delete) command and the close brace } on a newline. GNU sed does not require the semicolon after d; BSD (macOS) sed does. The second line recognizes and deletes the lines reportin the user and sys times. Everything else is passed through unaltered.
The script processes any files you give it after the log file name, or standard input if you give it none. A better command line notation would use an explicit option (-l logfile) and getopts to specify the log file.
With that in place, we can devise a program that reports to standard error and standard output — my_command:
echo "nonsense: error: positive numbers are required for argument 1" >&2
dribbler -s 0.4 -r 0.1 -i data -t
echo "apoplexy: unforeseen problems induced temporary amnesia" >&2
You could use cat data instead of the dribbler command. The dribbler command as shown reads lines from data, writes them to standard output, with a random delay with a gaussian distribution between lines. The mean delay is 0.4 seconds; the standard deviation is 0.1 seconds. The other two lines are pretending to be commands that report errors to standard error.
My data file contained a nonsense 'poem' called 'The Great Panjandrum'.
With this background in place, we can run the command and capture the real time in log.file, delete (ignore) the user and system time values, while sending the rest of standard error to standard error by using:
$ (time my_command) 2> >(tee raw.stderr | time.filter >&2)
nonsense: error: positive numbers are required for argument 1
So she went into the garden
to cut a cabbage-leaf
to make an apple-pie
and at the same time
a great she-bear coming down the street
pops its head into the shop
What no soap
So he died
and she very imprudently married the Barber
and there were present
the Picninnies
and the Joblillies
and the Garyulies
and the great Panjandrum himself
with the little round button at top
and they all fell to playing the game of catch-as-catch-can
till the gunpowder ran out at the heels of their boots
apoplexy: unforeseen problems induced temporary amnesia
$ cat log.file
0m7.278s
(The time taken is normally between 6 and 8 seconds. There are 17 lines, so you'd expect it to take around 6.8 seconds at 0.4 seconds per line.) The blank line is from time; it is pretty hard to remove that blank line, and only that blank line, especially as POSIX says it is optional. It isn't worth it.
I'm running rman commands from Bash scripts. I pass my commands to rman using here documents. I want to capture the output but also at the same time print it to the console (real-time).
I found this solution, but I don't how to make it to work with here-docs.
VAR=$(ls | tee /dev/tty)
What I currently run is:
output=$(rman <<RMAN
$rman_script
RMAN
)
Do you know how in this RMAN example I could also print stdout to the console apart from storing it in the output variable? Any help is appreciated.
Cheers.
The here document is no different from other redirections, although the syntax is of course slightly different.
var=$(rman <<\... | tee /dev/stderr
$rman_script
...
)
If this is a representative snippet of your code, you might as well
var=$(rman <<<"$rman_script" | tee /dev/stderr)
By the by, if you genuinely need the script multiple times (why else keep it in a variable?) maybe refactor into a function:
rman_script () {
rman <<\____HERE
Actual script
Probably multiple lines
____HERE
}
var=$(rman_script | tee /dev/stderr)
You'll notice that I use /dev/stderr instead of /dev/tty. Having a script require, and muck with, your tty should probably be avoided unless your script is really short and simple, and only makes sense to use interactively (password manipulation comes to mind as one soenario where it's sometimes hard to avoid).
output=$(rman <<RMAN)
$rman_script
RMAN
Note that a HERE-document looks syntactically like a input redirection, only that you have << instead of <. The input will be taken from the subsequent lines.
I am executing a command a which takes as input a script input.sh and runs a binary program, which produces some output file output. input.sh has a line NUMPROCS={some number} which specifies the number of processors program uses. a gives me back control immediately while program runs. The output file has a line at the end telling me how long program took.
I want to run this in a loop to see the time program takes as a function of number of processors. How do I make the control wait for output to be generated in each loop (and therefore have grep get the correct value) before it moves to line 4?
My script so far is below. Line 3 is how I would run it if NUMPROCS had a fixed value.
Some issues: Other users run program too, so I can't use pidof program and wait on it directly.
1 #! /bin/bash
2 for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do
3 a input.sh
4 E=`grep "Elapsed Time" output`
5 sed -i "/NUMPROCS/c\NUMPROCS=${i}" input.sh
6 echo $i $E >> result.dat
7 done
8 cat result.dat
It sounds like your input.sh script (side note, if that's a command-line command, it shouldn't have .sh on the end - on the other hand, if it's read into the "a" program as a shell include with "." or "source", the .sh is fine) backgrounds its jobs internally and immediately exits, leaving all the subjobs running. Ideally you'd want to edit input.sh, to have a "wait" command after spawning the jobs (perhaps a behavior triggerable by a command line option). Of course if you could edit it, you'd obviously want to add a wait to specify the job count on the command line too, and then suddenly everything would be really easy.
Editing the script with sed, on the other hand, really only makes sense if this is indeed a shell-script import, which is an approach that makes it hard to pass in command line parameters (passing them to "a" is easy, but for input.sh, it would need to pick them up from "a"'s internal variables, which is pretty abnormal). Obviously that's a difficult model to do anything tidy with. In such a case, your options are to either parse the output(s) to look for lines indicating the several executions are all complete, or to use "ps" and "grep" to count how many are running and go on to the next loop iteration when they're done (there's also a really sick approach where you loop until the system load bottoms out again, but surely nobody would do that).
Editing the input.sh script each time isn't really good, since it means if you run several tests together they may interfere with each other - in rare cases, dramatically.
So, although this is far uglier than fixing input.sh or "a" itself, you could just add a loop like this after "a input.sh"
while true ; do
if [ $i -eq `grep -c 'Elapsed Time' output | wc -l` ] ; then
break
fi
sleep 10 # avoid zillions of greps in a given time interval.
done
Without more information about input.sh it's hard to give better advice. Fixing input.sh itself, or the "a" program, to have an option to wait until subjobs complete, would normally be the preferred approach.
Other notes:
The "#!/bin/bash" generally shouldn't have a space after the magic number.
Use "for i in {1..16} ; do" instead - much easier to change (assumes halfway modern bash)
The E=grep "Elapsed Time" output is a nested command output capture, meaning that it will try to execute the line output by grep. This probably isn't what you wanted.
You can drop the ">> result.dat" and the cat, and instead change the "done" to "done | tee result.dat" for the same effect.
I have a program that users can run using the command line. Once running, it receives and processes commands from the keyboard. It's possible to start the program with input from disk like so: $ ./program < startScript.sh. However, the program quits as soon as the script finishes running. Is there a way to start the program with input from disk using < that does not quit the program and allows additional keyboard input to be read?
(cat foo.txt; cat) | ./program
I.e., create a subshell (that's what the parentheses do) which first outputs the contents of foo.txt and after that just copies whatever the user types. Then, take the combined output of this subshell and pipe it into stdin of your program.
Note that this also works for other combinations. Say you want to start a program that always asks the same questions. The best approach would be to use "expect" and make sure the questions didn't change, but for a quick workaround, you can also do something like this:
(echo y; echo $file; echo n) | whatever
Use system("pause")(in bash it's just "pause") in your program so that it does not exit immediatly.
There are alternatives such as
dummy read
infinite loop
sigsuspend
many more
Why not try something like this
BODY=`./startScript.sh`
if [ -n "$BODY" ]
then cat "$BODY" |./program
fi
That depends on how the program is coded. This cannot be achieved from writing code in startScript.sh, if that is what you're trying to achieve.
What you could do is write a callingScript.sh that asks for the input first and then calls the program < startScript.sh.
In a shell, I run following commands without problem,
ls -al
!ls
the second invocation to ls also list files with -al flag. However, when I put the above script to a bash script, complaints are thrown,
!ls, command not found.
how to realise the same effects in script?
You would need to turn on both command history and !-style history expansion in your script (both are off by default in non-interactive shells):
set -o history
set -o histexpand
The expanded command is also echoed to standard error, just like in an interactive shell. You can prevent that by turning on the histverify shell option (shopt -s histverify), but in a non-interactive shell, that seems to make the history expansion a null-op.
Well, I wanted to have this working as well, and I have to tell everybody that the set -o history ; set -o histexpand method will not work in bash 4.x. It's not meant to be used there, anyway, since there are better ways to accomplish this.
First of all, a rather trivial example, just wanting to execute history in a script:
(bash 4.x or higher ONLY)
#!/bin/bash -i
history
Short answer: it works!!
The spanking new -i option stands for interactive, and history will work. But for what purpose?
Quoting Michael H.'s comment from the OP:
"Although you can enable this, this is bad programming practice. It will make your scripts (...) hard to understand. There is a reason it is disabled by default. Why do you want to do this?"
Yes, why? What is the deeper sense of this?
Well, THERE IS, which I'm going to demonstrate in the follow-up section.
My history buffer has grown HUGE, while some of those lines are script one-liners, which I really would not want to retype every time. But sometimes, I also want to alter these lines a little, because I probably want to give a third parameter, whereas I had only needed two in total before.
So here's an ideal way of using the bash 4.0+ feature to invoke history:
$ history
(...)
<lots of lines>
(...)
1234 while IFS='whatever' read [[ $whatever -lt max ]]; do ... ; done < <(workfile.fil)
<25 more lines>
So 1234 from history is exactly the line we want. Surely, we could take the mouse and move there, chucking the whole line in the primary buffer? But we're on *NIX, so why can't we make our life a bit easier?
This is why I wrote the little script below. Again, this is for bash 4.0+ ONLY (but might be adapted for bash 3.x and older with the aforementioned set -o ... stuff...)
#!/bin/bash -i
[[ $1 == "" ]] || history | grep "^\s*$1" |
awk '{for (i=2; i<=NF; i++) printf $i" "}' | tr '\n' '\0'
If you save this as xselauto.sh for example, you may invoke
$ ./xselauto.sh 1234
and the contents of history line #1234 will be in your primary buffer, ready for re-use!
Now if anyone still says "this has no purpose AFAICS" or "who'd ever be needing this feature?" - OK, I won't care. But I would no longer want to live without this feature, as I'm just too lazy to retype complex lines every time. And I wouldn't want to touch the mouse for each marked line from history either, TBH. This is what xsel was written for.
BTW, the tr part of the pipe is a dirty hack which will prevent the command from being executed. For "dangerous" commands, it is extremely important to always leave the user a way to look before he/she hits the Enter key to execute it. You may omit it, but ... you have been warned.
P.S. This scriptlet is in fact a workaround, simulating !1234 typed on a bash shell. As I could never make the ! work directly in a script (echo would never let me reveal the contents of history line 1234), I worked around the problem by simply greping for the line I wanted to copy.
History expansion is part of the interactive command-line editing features of a shell, not part of the scripting language. It's not generally available in the context of a script, only when interacting with a (pseudo-)human operator. (pseudo meaning that it can be made to work with things like expect or other keystroke repeating automation tools that generally try to play act a human, not implying that any particular operator might be sub-human or anything).