Till today I was always able to find answer for my all bash questions. But now I stuck. I am testing 20TB RAID6 configuration working on LSI 9265.
I wrote script to create files from /dev/urandom and I am creating second to calculate md5 from all files with two addons.
One is to use time command to calculate md5sum execution time
Second is use pv command to show progress of each md5sum command
My command looks like this:
filename="2017-03-13_12-38-08"
/usr/bin/time -f "real read %E" pv $filename | md5sum | sed "s/-/$filename /"
This is example terminal printout:
/usr/bin/time -f "real read %E" pv $i | md5sum | sed "s/-/$i/"
1GiB 0:00:01 [ 551MiB/s] [==================================================================================================>] 100%
real read 0:01.85
f561af8cc0927967c440fe2b39db894a 2017-03-13_12-38-08
And I want to log it to file. I failed all tries using 2>&1, using tee, using brackets. I know pv uses stdErr but this doesnt help in finding solution. I can only catch "f561af8cc0927967c440fe2b39db894a 2017-03-13_12-38-08_done"
which is not enough.
This is the solution:
(time pv -f $filename | md5sum | sed "s/-/$filename/") 2>&1 | tee output.log
or equivalent but without printing into terminal only file to output.log
(time pv -f $filename | md5sum | sed "s/-/$filename/") > output.log 2>&1
Related
I want to update the download status at every 5 second of a downloading file to my telegram bot. Also here I'm using bash.
aria2c $url --summary-interval=5 2>&1 | tee output.log | grep -oP "(\d+(\.\d+)?(?=%))"
This thing derive me download percentage after each 5 seconds. I want to use this download percentage for my bot to update it regularly. I tried these
aria2c $url --summary-interval=5 2>&1 | tee output.log | grep -oP "(\d+(\.\d+)?(?=%))" | { read text; curl -s "https://api.legram.org/bot${tg_token}/editMessageText" --data "message_id=${msg_id}&text=DOWNLOADED-${text}&chat_id=${ch_id}&parse_mode=HTML&disable_web_page_preview=True"; }
Try 2
aria2c $url --summary-interval=5 2>&1 | tee output.log | text=$(grep -oP "(\d+(\.\d+)?(?=%))") | curl -s "https://api.legram.org/bot${tg_token}/editMessageText" --data "message_id=${msg_id}&text=DOWNLOADED-${text}%&chat_id=${ch_id}&parse_mode=HTML&disable_web_page_preview=True"; }
But none works. Then for testing I tried this
aria2c $url --summary-interval=5 2>&1 | tee output.log | grep -oP "(\d+(\.\d+)?(?=%))" | { read text; echo "$text"; }
I just got one output at last(which might be the first download %), unlike what it should be. Can anyone get me the working code.
The problem is that you only run read (and then updates the status) once, so it reads a single line (and updates the status once). You need a loop, so it'll repeat the read+update process over & over. You can use a while loop to do this. If it should exit when there's no more input to process, make read the while condition:
aria2c $url --summary-interval=5 2>&1 |
tee output.log |
grep -oP "(\d+(\.\d+)?(?=%))" |
while read text; do
curl -s "https://api.legram.org/bot${tg_token}/editMessageText" --data "message_id=${msg_id}&text=DOWNLOADED-${text}&chat_id=${ch_id}&parse_mode=HTML&disable_web_page_preview=True"
done
I have the following in a bash script file watcher.sh.
grep ERROR $ExampleLogFile > $ErrorLogFile
When I run this, it copied the lines from ExampleLogFile to ErrorLogFile that contain ERROR successfully.
I need to make it so it continually monitors the ExampleLogFile for changes and writes those to the ErrorLogFile.
I was thinking of doing the following, but this doesn't work:
tail -f grep ERROR $ExampleLogFile > $ErrorLogFile
It does write some of the lines, but its not the ones containing ERROR.
tail: grep: No such file or directory
tail: ERROR: No such file or directory
Any advise please.
You can use tee command here.
tail -f $ExampleLogFile | grep --line-buffered ERROR | tee $ErrorLogFile
It will store and print to stdout at the same time.
You need:
while :; do grep ERROR $ExampleLogFile > $ErrorLogFile; sleep 2; done
This should achieve what you want without needing the tail command.
If the file will ever be cleared though this will not work as you might expect because the grep will pull only current entries in the $ErrorLogFile.
You can arrange the tail/grep in a pipe
tail -f $ExampleLogFile | grep ERROR > $ErrorLogFile
Remember that this command will never exit by itself (tail will continue to look for additional data). You will have to arrange for some other exit condition (e.g., timeout, explicit kill, etc).
tail -f $ExampleLogFile | grep --line-buffered ERROR > $ErrorLogFile
or paranoic:
stdbuf -oL tail -f $ExampleLogFile | stdbuf -oL grep --line-buffered ERROR > $ErrorLogFile
But most probably you want to include existing lines too. In that case:
tail -n +1 -f $ExampleLogFile | grep --line-buffered ERROR > $ErrorLogFile
I have a shell script that is sending in about 150 files to a python program I wrote.
I have no idea how long it is going to take, so I was wondering if there was a terminal command to either:
a) tell me which file is currently being worked on
b) how many files are left to run
Here's my shell:
#!/bin/bash
ls fp00t*g*k2odfnew.dat | while read line; do
echo $line
python file_editor.py $line
done
PipeViewer will probably do what you need: http://www.catonmat.net/blog/unix-utilities-pipe-viewer/
Something like this might work, putting both ls and pv in line mode:
#!/bin/bash
ls -1 fp00t*g*k2odfnew.dat | pv -l | while read line; do
echo $line
python file_editor.py $line
done
you can also supply a total to pv so it knows how many you're counting up to, so the progress bar works properly:
#!/bin/bash
ls -1 fp00t*g*k2odfnew.dat | pv -l -s`ls -1 fp00t*g*k2odfnew.dat | wc -l` | while read line; do
echo $line
python file_editor.py $line
done
Full pv docs here: http://www.ivarch.com/programs/quickref/pv.shtml
I am facing with the following bash script:
#! /bin/bash
processname=$1
x=1
while [ $x -eq 1 ] ; do
ps -el | grep $processname | awk ' {if($15!="grep") print "The memory consumed by the process " $15 " is = "$9} ' >> out.log
sleep 30
done
and I am running this with :
$ ./myscript.sh Firefox
but when i see the output in the file, apart from the firefox process i am also getting information for /bin/bash process
The memory consumed by the process /Applications/Firefox.app/Contents/MacOS/firefox is = 911328
The memory consumed by the process /bin/bash is = 768
Can some one help me with this so that I only want to get information related to Firefox process and nothing else(/bin.bash etc)
The common trick is to change grep Firefox to grep [F]irefox. In this case, you can achieve it with
ps | grep '['${processname:0:1}']'${processname:1}
This is normal and because the $processname is Firefox. Since your command also monitors it, there is a process that uses it.
For example, try ps -el | grep Firefox and you will get two process lines matching (if you have one instance of Firefox running), one is Firefox, the other is the grep command looking for Firefox.
Piping your output in grep -v /bin/bash' should solve this. Eg:
ps -el | grep $processname | awk ...
becomes:
ps -el | grep $processname | grep -v 'grep' | awk ...
You calling
ps -el | grep $processname | awk ' {if($15!="grep") print "The memory consumed by the process " $15 " is = "$9} '
This means that you run awk and connect it's input with output of grep. Then grep is started and gets output of ps -el as input.
At the moment when bash will start ps you have grep and awk running.
Solution: run ps -el and remember it's output. Then run grep and awk. It should be like this:
ps -el > ps.tmp
grep $processname ps.tmp | awk ' {if($15!="grep") print "The memory consumed by the process " $15 " is = "$9} ' >> out.log
Probably this could be done without using tmp file. Like TMP=$(ps -el). But I don't know how to run grep filter on variable
I want to count the number of lines output from a command in a bash script. i.e.
COUNT=ls | wc -l
But I also want the script to output the original output from ls. How to get this done? (My actual command is not ls and it has side effects. So I can't run it twice.)
The tee(1) utility may be helpful:
$ ls | tee /dev/tty | wc -l
CHANGES
qpi.doc
qpi.lib
qpi.s
4
info coreutils "tee invocation" includes this following example, which might be more instructive of tee(1)'s power:
wget -O - http://example.com/dvd.iso \
| tee >(sha1sum > dvd.sha1) \
>(md5sum > dvd.md5) \
> dvd.iso
That downloads the file once, sends output through two child processes (as started via bash(1) process substitution) and also tee(1)'s stdout, which is redirected to a file.
ls | tee tmpfile | first command
cat tmpfile | second command
Tee is a good way to do that, but you can make something simpler:
ls > __tmpfile
cat __tmpfile | wc -l
cat __tmpfile
rm __tmpfile