I have a long running process running and I want to monitor its RAM usage. I can do this by watching top. However I would like to be able to log out and have a record written, every minute say, to a shared disk space instead.
My solution which works is:
nohup top -b -d 60 -p 10036|grep 10036 >> ramlog.txt &
But I would like to know when each line is outputted too. How can I modify the one-liner to add this information on each line?
I know about screen and tmux but I would like to get this simple one-liner working.
You could add a loop that reads each line from grep and prepends a date. Make sure to use grep --line-buffered to ensure each line is printed without delay.
nohup top -b -d 60 -p 10036 |
grep --line-buffered 10036 |
while read line; do echo "$(date): $line"; done >> ramlog.txt &
Related
I am trying to split a dozen 100MB+ csv files into managable smaller files for a curl post.
I have managed to do it but with a lot of temporary files and IO. It's taking an eternity.
I am hoping someone can show me a way to do this much more effectively; preferably with little to no disk io
#!/bin/sh
for csv in $(ls *.csv); do
tail -n +2 $csv | split -a 5 -l - $RANDOM.split.
done
# chose a file randomly to fetch the header from
header=$(ls *.csv |sort -R |tail -1 | cut -d',' -f1)
mkdir split
for x in $(/usr/bin/find . -maxdepth 1 -type f -name '*.split.*'); do
echo Processing $x
cat header $x >> split/$x
rm -f $x
done
The above script may not entirely work. I basically got it working through a combination of these commands.
I decided to make the curl POST another step entirely in the case of upload failure; I didn't want to lose the data if it were all posted. But, if, say, on error from curl the data could be put into a redo folder then that can work.
#!/bin/sh
# working on a progress indicator as a percentage. Never finished.
count=$(ls -1 | wc -l 2> /dev/null | cut -d' ' -f1)
for file in $(/usr/bin/find . -maxdepth 1 -type f); do
echo Processing $file
curl -XPOST --data-binary #$file -H "Content-Type: text/cms+csv" $1
done
Edit 1 -- why the RANDOM? because split is going to produce the exact same files when it splits the next file as it did for the first. so .. aa ab ac ... will be produced for every file. I need to ensure every file produced by split is unique for the entire run
Not quite sure what you want to accomplish, but it seems to me that you are processing line by line. Thus, if you serialize all your csv files and lines, you can do it without disk I/O. Yet from your descriptions, I can't tell if this script runs many instances or just one instance (multiple processes or one process). Thus I can just try my best to mimic your script to reach as similar results as possible, yet to resolve the disk I/O problem. The codes are provided below, yet please correct script error if any, as I have no way to run/debug/verify it:
for csv in $(ls *.csv | sort -R); do
# first read line skip the first line, since I see your tail -n +2 command.
(read line;
count=0;
while read line; do
Processing $line;
count=$(($count + 1));
echo $csv.$count >> split/$count;
done
) < $csv
done
Your 'Processing' code now should process from a verbose line, rather than a file. Perhaps a pipe and have your Processing to process on STDIN will do the trick:
echo $line | Processing
Your curl can do similar way, to process from STDIN, by replacing #$file with -, and you can print what you want curl to send and then pipe it to curl, similar to this:
ProcessingAndPrint | curl -XPOST --data-binary - -H "Content-Type: text/cms+csv" $1
I wrote a bash script that first do operations on text files, and then execute another script from within, repeating this operation in a while loop. The script that is executed from within at a certain time ask the user
'press to stop'
and wait 5 sec, if the user doesn't press return goes further. If I execute this script alone, it works fine, but if I execute it from within the other, it seems that the return key is pressed automatically and stops the execution.
How can I avoid it?
Here is an example of the script:
#!/bin/bash
pMatrixFile='file.csv'
templateFile='out.txt'
nSim=0
while read line
do
((nSim++))
# ***************Read the input file*****************************************
scale1=$(echo $line | cut -f1 -d\;)
scale2=$(echo $line | cut -f2 -d\;)
# ***************Write the file to be runned*********************************
sed -e "/double Scale_EX2 = / s|scale_DOE|$scale1|g" \
-e "/double Scale_EX6 = / s|scale_DOE|$scale2|g" \
-e "/double Scale_EX7 = / s|scale_DOE|$scale8|g" <$templateFile >$fileName
# ***************Launch the simulation on server*****************************
sed -e "s|simFile|$simFile|g" <$submitTemplateFile >$submitFile
sed -i "s|simVisName|$simVisName|g" $submitFile
# *************At this line we have the issue!***********
chmod a+x $submitFile
. ./$submitFile |tee log
# *******************************************************
# ***************Clean up the temporary files********************************
rm $simFile $fileName $submitFile
done<$pMatrixFile
$submitFile is my external script.
Thanks for the help!
Use a file descriptor that's different from stdin (0). Example:
while read -u 4 line; do
...
done 4< your_file.txt
This would help prevent some parts of your while block to eat input from your_file.txt everytime they ask for one.
I have that generate a history of server's memory and cpu used. But i can't install any software on server.
Then i am trying this script
#!/bin/bash
while true; do
date >> system.log
top -n1 | grep 'Cpu\|Mem\|java\|eservices' >> system.log
echo '' >> system.log
sleep 2
done
but when i try execute tail -500f system.log the logs stopping
You should probably use the -b batch mode parameter. From man top:
Starts top in 'Batch' mode, which could be useful for sending output from top to other programs or to a file. In this mode, top will not accept input and runs until the iterations limit you've set with the '-n' command-line option or until killed.
You might want to use the portable format tail -n 500 -f.
In any case, saving top output to file and then running tail -f on it emulates the way top works. What are you trying to achieve that top does not do already?
To monitor total free memory use on a server, you can
grep -F MemFree: /proc/meminfo
To monitor process memory use:
ps -o rss $pid
For work, I occasionally need to monitor the output logs of services I create. These logs are short lived, and contain a lot of information that I don't necessarily need. Up until this point I've been watching them using:
grep <tag> * | less
where <tag> is either INFO, DEBUG, WARN, or ERROR. There are about 10x as many warns as there are errors, and 10x as many debugs as warns, and so forth. It makes it difficult to catch one ERROR in a sea of relevant DEBUG messages. I would like a way to, for instance, make all 'WARN' messages appear on the left-hand side of the terminal, and all the 'ERROR' messages appear on the right-hand side.
I have tried using tmux and screen, but it doesn't seem to be working on my dev machine.
Try doing this :
FILE=filename.log
vim -O <(grep 'ERR' "$FILE") <(grep 'WARN' "$FILE")
Just use sed to indent the desired lines. Or, use colors. For example, to make ERRORS red, you could do:
$ r=$( printf '\033[1;31m' ) # escape sequence may change depending on the display
$ g=$( printf '\033[1;32m' )
$ echo $g # Set the output color to the default
$ sed "/ERROR/ { s/^/$r/; s/$/$g/; }" *
If these are live logs, how about running these two commands in separate terminals:
Errors:
tail -f * | grep ERROR
Warnings:
tail -f * | grep WARN
Edit
To automate this you could start it in a tmux session. I tend to do this with a tmux script similar to what I described here.
In you case the script file could contain something like this:
monitor.tmux
send-keys "tail -f * | grep ERROR\n"
split
send-keys "tail -f * | grep WARN\n"
Then run like this:
tmux new -d \; source-file monitor.tmux; tmux attach
You could do this using screen. Simply split the screen vertically and run tail -f LOGFILE | grep KEYWORD on each pane.
As a shortcut, you can use the following rc file:
split -v
screen bash -c "tail -f /var/log/syslog | grep ERR"
focus
screen bash -c "tail -f /var/log/syslog | grep WARN"
then launch your screen instance using:
screen -c monitor_log_screen.rc
You can of course extend this concept much further by making more splits and use commands like tail -f and watch to get live updates of different output.
Do also explore screen other screen features such as use of multiple windows (with monitoring) and hardstatus and you can come up with quite a comprehensive "monitoring console".
I am trying to debug some errors in a live Merb app. There are a lot of lined of error code running by, but I jut need to see the first one. I can use grep to select these lines and print them but it closes as soon as it reached the end of the file.
What I would like to do is use grep like the shift-F mode in less where it will keep the file open and report new matching line as they are written to the log.
- or -
Is there a way to do this directly with less that I don't know about?
try this
tail -f dev.log | grep '^ERROR:'
the -f option to tail tells it to wait for more data when it hits EOF.
Can't you do this with watch and tail?
watch -n 30 "grep 'dev.log' '^ERROR:' | tail -n 30"