I was trying to compose a command that would monitor the stability of the script on server by curling it every couple of minutes (actual path to script was replaced):
while :; do date +"%T" >> monitor.txt; time curl -Is http://googel.com | egrep "HTTP|m.\." >> monitor.txt; echo ================ >> monitor.txt; sleep 30; done
The problem is that for some reason part of output is not forwarded to file monitor.txt. So file contains following lines:
$ cat monitor.txt
19:39:10
HTTP/1.1 301 Moved Permanently
================
19:39:40
HTTP/1.1 301 Moved Permanently
================
..while time details go to default output:
$ while :; do date +"%T" >> monitor.txt; time curl -Is http://googel.com | egrep "HTTP|m.\." >> monitor.txt; echo ================ >> monitor.txt; sleep 30; done
real 0m0.075s
user 0m0.005s
sys 0m0.003s
real 0m0.106s
user 0m0.004s
sys 0m0.005s
Could you please point out, what am I missing here? Basically I would run this command in a background and check monitor.txt for results.
Thank you in advance!
The time command sends its output to stderr, not stdout. Your redirection only affects stdout, so the time output ends up going
to the console.
To add to the confusion, bash also has a builtin time command, which
is a bit trickier to redirect. If you use /usr/bin/time instead of
time, you should be able to redirect its output with the 2>&1 syntax.
Or if you prefer the bash builtin version of the command, you can see this answer for a way
to redirect its output.
Related
I'm trying to write a script that will run and time a given and output that to a file in a .csv format.
So far from looking at SO previous posts, I've found that sh -c "$index_of_command_arg" can be used to invoke that command.
I'm also familiar with time and I know that people use /usr/bin/time for formatting, but I need to format the time given in total seconds (for example, 1.34516) but the only given option to format the real time is %E which return [hours:]minutes:seconds. Is there any way to format it the way I need?
The general idea of my script is:
# ----
# some input validation
# ----
rule=$1
command=$2
execution_time=/usr/bin/time -f "%total_seconds" sh -c "$command" #is this line possible?
echo "$rule,$execution_time" > output_file.csv
Can this be formatted the way I want? and also, the line with the comment after it,
Will this even work the way I wrote it? is the syntax correct?
Say I use the normal time and I get the real 0m2.003 ...
output, how can I take the 2.003 out of it?
The normal time you are mentioning is the bash built-in time. From the Bash Reference Manual:
The TIMEFORMAT variable may be set to a format string
that specifies how the timing information should be displayed.
…
%[p][l]R
The elapsed time in seconds.
So, you could use
execution_time=`TIMEFORMAT=%R bash -c "time $command" 2>&1 >/dev/tty`
You replace:
execution_time=/usr/bin/time -f "%total_seconds" sh -c "$command" `
with:
execution_time=`(time sh -c "$command > /dev/null 2>&1") 2>&1 | grep real |sed "s/.*m//;s/s.*//;"`
or with:
execution_time=`(time sh -c "$command > /dev/null 2>&1") 2>&1 | grep real | cut -c 8-12`
Why > /dev/null - you don't want to see $command output 2>&1 (STDERR > STDOUT)
Why 2>&1 for time command (for the same reason as in previous bullet)
Just a little question about timing programs on Linux: the time command allows to
measure the execution time of a program:
[ed#lbox200 ~]$ time sleep 1
real 0m1.004s
user 0m0.000s
sys 0m0.004s
Which works fine. But if I try to redirect the output to a file, it fails.
[ed#lbox200 ~]$ time sleep 1 > time.txt
real 0m1.004s
user 0m0.001s
sys 0m0.004s
[ed#lbox200 ~]$ cat time.txt
[ed#lbox200 ~]$
I know there are other implementations of time with the option -o to write a file but
my question is about the command without those options.
Any suggestions ?
Try
{ time sleep 1 ; } 2> time.txt
which combines the STDERR of "time" and your command into time.txt
Or use
{ time sleep 1 2> sleep.stderr ; } 2> time.txt
which puts STDERR from "sleep" into the file "sleep.stderr" and only STDERR from "time" goes into "time.txt"
Simple. The GNU time utility has an option for that.
But you have to ensure that you are not using your shell's builtin time command, at least the bash builtin does not provide that option! That's why you need to give the full path of the time utility:
/usr/bin/time -o time.txt sleep 1
Wrap time and the command you are timing in a set of brackets.
For example, the following times ls and writes the result of ls and the results of the timing into outfile:
$ (time ls) > outfile 2>&1
Or, if you'd like to separate the output of the command from the captured output from time:
$ (time ls) > ls_results 2> time_results
If you care about the command's error output you can separate them like this while still using the built-in time command.
{ time your_command 2> command.err ; } 2> time.log
or
{ time your_command 2>1 ; } 2> time.log
As you see the command's errors go to a file (since stderr is used for time).
Unfortunately you can't send it to another handle (like 3>&2) since that will not exist anymore outside the {...}
That said, if you can use GNU time, just do what #Tim Ludwinski said.
\time -o time.log command
Since the output of 'time' command is error output, redirect it as standard output would be more intuitive to do further processing.
{ time sleep 1; } 2>&1 | cat > time.txt
If you are using GNU time instead of the bash built-in, try
time -o outfile command
(Note: GNU time formats a little differently than the bash built-in).
&>out time command >/dev/null
in your case
&>out time sleep 1 >/dev/null
then
cat out
I ended up using:
/usr/bin/time -ao output_file.txt -f "Operation took: %E" echo lol
Where "a" is append
Where "o" is proceeded by the file name to append to
Where "f" is format with a printf-like syntax
Where "%E" produces 0:00:00; hours:minutes:seconds
I had to invoke /usr/bin/time because the bash "time" was trampling it and doesn't have the same options
I was just trying to get output to file, not the same thing as OP
If you don't want to touch the original process' stdout and stderr, you can redirect stderr to file descriptor 3 and back:
$ { time { perl -le "print 'foo'; warn 'bar';" 2>&3; }; } 3>&2 2> time.out
foo
bar at -e line 1.
$ cat time.out
real 0m0.009s
user 0m0.004s
sys 0m0.000s
You could use that for a wrapper (e.g. for cronjobs) to monitor runtimes:
#!/bin/bash
echo "[$(date)]" "$#" >> /my/runtime.log
{ time { "$#" 2>&3; }; } 3>&2 2>> /my/runtime.log
#!/bin/bash
set -e
_onexit() {
[[ $TMPD ]] && rm -rf "$TMPD"
}
TMPD="$(mktemp -d)"
trap _onexit EXIT
_time_2() {
"$#" 2>&3
}
_time_1() {
time _time_2 "$#"
}
_time() {
declare time_label="$1"
shift
exec 3>&2
_time_1 "$#" 2>"$TMPD/timing.$time_label"
echo "time[$time_label]"
cat "$TMPD/timing.$time_label"
}
_time a _do_something
_time b _do_another_thing
_time c _finish_up
This has the benefit of not spawning sub shells, and the final pipeline has it's stderr restored to the real stderr.
If you are using csh you can use:
/usr/bin/time --output=outfile -p $SHELL -c 'your command'
For example:
/usr/bin/time --output=outtime.txt -p csh -c 'cat file'
If you want just the time in a shell variable then this works:
var=`{ time <command> ; } 2>&1 1>/dev/null`
I'd like to use time command in a bash script to calculate the elapsed time of some function, but time command gives me error messages when I try to call it to measure one of the script's functions using any of its options, like '--format', which I want to make use of.
The same thing happens if I use 'time [...]' from terminal, with any command. However, using it from terminal, it works when I enter '/usr/bin/time [...]' instead of just 'time [...]', but when I call '/usr/bin/time ANY_FUNCTION_INSIDE_ANY_SCRIPT [arguments]' in a script, it complains about the called function being not a valid command.
test.sh code:
#!/bin/bash
testime ()
{
printf "test message is \"$1\" \n"
printf "just testing...\n"
sleep 5
read -p "enter something: " buffer
echo $buffer
}
time -f "testime() finished after %E" testime "damn!"
printf "Bye bye!\n\n"
exit
Output:
$ ./test.sh
./test.sh: line 12: -f: command not found
real 0m0.001s
user 0m0.000s
sys 0m0.001s
Bye bye!
BUT... If i use time without options, like "-f ...", as here:
[...]
time testime "damn!"
printf "Bye bye!\n\n"
exit
...the script works correctly, as you can see:
$ ./test.sh
test message is "damn!"
just testing...
enter something: SOMETHING_I_HAVE_ENTERED
SOMETHING_I_HAVE_ENTERED
real 0m24.050s
user 0m0.000s
sys 0m0.001s
Bye bye!
AND when I call '/usr/bin/time -f [...]', as it would work when I'm doing stuff from terminal, using '-f [...]' or not, it gives me some output like this:
$ ./test.sh
/usr/bin/time: cannot run testime: No such file or directory
Command exited with non-zero status 127
0.00user 0.00system 0:00.00elapsed ?%CPU (0avgtext+0avgdata 340maxresident)k
0inputs+0outputs (0major+72minor)pagefaults 0swaps
Bye bye!
Does someone know why it happens?
Mention the command and -f option at the end.
Shell> time testime 'damn!' -f 'testime() finished after %E'
test message is "damn!"
just testing...
enter something: abc
abc
real 0m7.513s
user 0m0.000s
sys 0m0.003s
The usage of the test command clearly mentioned that you need to give command before to options.
NAME
time - time a simple command or give resource usage
SYNOPSIS
time [options] command [arguments...]
In [options] only -P is given and remaining are arguments.
Trying to redirect and quite the output to screen on a cURL command? Also what happens with an exception, if cURL can't find the file on the site? Its downloading to a FTP site with an FTP URL. I have many files so its iterating in a loop and I don't want it to stop if file isn't found so will continue if exception isn't found? Can I use the cURL command if I am strictly using a bash script? It does download the files but it outputs to much stuff and also haven't been able to test in situation where it would throw an error so not sure if it would continue.
How can I stop cURL from outputting to screen? This is what I have so far.
echo $DLADDR
curl -o Downloads/$FILECATNAME $DLADDR 2>&1 | tee $LOGFILE
I would like to stop this output and put each output into a LOG file.
Example cURL output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 574k 100 574k 0 0 622k 0 --:--:-- --:--:-- --:--:-- 632k
As mentioned in the comment above, it's tough to exactly figure what you're asking. However, one of the questions is:
How can I stop CURL from outputting to screen?
You are saying:
curl -o Downloads/$FILECATNAME $DLADDR 2>&1 | tee $LOGFILE
which would redirect the response output to the specified file but the STDERR of curl would be merged with STDOUT and teed to the logfile.
Redirect the output right away to stop CURL from outputting to screen, i.e. say:
curl -o Downloads/$FILECATNAME $DLADDR > $LOGFILE 2>&1
The other question appears to be:
I don't want it to stop if file isn't found so will continue if
exception isn't found?
The script shouldn't stop if curl encounters an error unless you've specified to do so, i.e. unless you've said:
set -e
in your script.
You might want to investigate curl's -w (--write-out) option, to get precisely the log information you require, and avoid the unnecessary/unreported bits.
#!/usr/bin/env bash
DLADDR=ftp://example.com/somefile
LOGFILE=/path/to/downloader.log
declare -a bashopts=()
bashopts+=(-s)
bashopts+=("-o 'Downloads/$FILECATNAME'")
bashopts+=("-w '%{size_download} %{speed_download} %{time_total} %{filename_effective}'")
curl "${bashopts[#]}" "$DLADDR" > "$LOGFILE"
We simplify the option management by putting it into an array. Note the options:
-s silences the progress meter.
-o sets an output file per your example.
-w writes output according to the format listed in curl's man page.
Since you're directing the output file using -o, the only information curl sends to stdout will be the log data generated by the -w command.
Note that the rule of thumb for quoting variables in bash is that if you're using a variable ... it should be quoted.
curl -o Downloads/$DESCFILE $TEXT1 > $TEXT 2>&1
I have a bash script with some scp commands inside.
It works very well but, if I try to redirect my stdout with "./myscript.sh >log", only my explicit echos are shown in the "log" file.
The scp output is missing.
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
fi
Ok, what should I do now?
Thank you
scp is using interactive terminal in order to print that fancy progress bar. Printing that output to a file does not make sense at all, so scp detects when its output is redirected to somewhere else other than a terminal and does disable this output.
What makes sense, however, is redirect its error output into the file in case there are errors. You might want to disable standard output if you want.
There are two possible ways of doing this. First is to invoke your script with redirection of both stderr and stdout into the log file:
./myscript.sh >log 2>&1
Second, is to tell bash to do this right in your script:
#!/bin/sh
exec 2>&1
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
fi
...
If you need to check for errors, just verify that $? is 0 after scp command is executed:
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
RET=$?
if [ $RET -ne 0 ]; then
echo SOS 2>&1
exit $RET
fi
fi
Another option is to do set -e in your script which tells bash script to report failure as soon as one of commands in scripts fails:
#!/bin/bash
set -e
...
Hope it helps. Good luck!
You cat simply test your tty with:
[ ~]#echo "hello" >/dev/tty
hello
If that works, try:
[ ~]# scp <user>#<host>:<source> /dev/tty 2>/dev/null
This has worked for me...
Unfortunately SCP's output can't simply be redirected to stdout it seems.
I wanted to get the average transfer speed of my SCP transfer, and the only way that I could manage to do that was to send stderr and stdout to a file, and then to echo the file to stdout again.
For example:
#!/bin/sh
echo "Starting with upload test at `date`:"
scp -v -i /root/.ssh/upload_test_rsa /root/uploadtest.tar.gz speedtest#myhost:/home/speedtest/uploadtest.tar.gz > /tmp/scp.log 2>&1
grep -i bytes /tmp/scp.log
rm -f /tmp/scp.log
echo "Done with upload test at `date`."
Which would result in the following output:
Starting with upload test at Thu Sep 20 13:04:44 SAST 2012:
Transferred: sent 10191920, received 5016 bytes, in 15.7 seconds
Bytes per second: sent 650371.2, received 320.1
Done with upload test at Thu Sep 20 13:05:04 SAST 2012.
I found a rough solution for scp:
$ scp -qv $USER#$HOST:$SRC $DEST
According to the scp man page, -q (quiet) disables the progress meter, as well as disabling all other output. Add -v (verbose) as well, you get heaps of output... and the progress meter is still disabled! Disabling the progress meter allows you to redirect the output to a file.
If you don't need all the authentication debug output, redirect the output to stdout and grep out the bits you don't want:
$ scp -qv $USER#$HOST:$SRC $DEST 2>&1 | grep -v debug
Final output is something like this:
Executing: program /usr/bin/ssh host myhost, user (unspecified), command scp -v -f ~/file.txt
OpenSSH_6.0p1 Debian-4, OpenSSL 1.0.1e 11 Feb 2013
Warning: Permanently added 'myhost,10.0.0.1' (ECDSA) to the list of known hosts.
Authenticated to myhost ([10.0.0.1]:22).
Sending file modes: C0644 426 file.txt
Sink: C0644 426 file.txt
Transferred: sent 2744, received 2464 bytes, in 0.0 seconds
Bytes per second: sent 108772.7, received 97673.4
Plus, this can be redirected to a file:
$ scp -qv $USER#$HOST:$SRC $DEST 2>&1 | grep -v debug > scplog.txt