Running Shell Script having multiple programs dynamically in parallel - bash

I have a shell script which captures the Process ID, CPU and Memory of the JVM every nth second and writes the output to a file. Below is my code:
JVM="aaa001_bcdefx01"
systime=$(date +"%m-%d-%y-%T")
for i in {1..10}
do
PID=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $2}'`
MEM=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $4 }'`
CPU=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $3 }'`
printf '%-5s %-20s %-20s %-20s %-20s \n' "$systime $JVM $PID $CPU $MEM " >> $LOGFILE
sleep 5
done
This run perfectly fine when i have only one JVM in that server. How can i execute the same script in parallel and fetch the details if i have multiple JVM for a server.
I looked for some solutions and found & to be used in the script but couldn't understand how this can be implemented to my above script. Let's say I have 5 JVMs. How can i run the script and fetch the stats in parallel for all the below JVMs in parallel. Kindly guide. Any help would be appreciated.
JVM="aaa001_bcdefx01"
JVM="aaa002_bcdefx01"
JVM="aaa003_bcdefx01"
JVM="aaa004_bcdefx01"
JVM="aaa005_bcdefx01"

GNU Parallel is made for this kind of stuff
doit() {
JVM="$1"
systime=$(date +"%m-%d-%y-%T")
for i in {1..10}
do
PID=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $2}'`
MEM=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $4 }'`
CPU=`ps -auxwww | grep "jdk" | grep $JVM | grep -v grep | cut -c -30 | awk '{print $3 }'`
printf '%-5s %-20s %-20s %-20s %-20s \n' "$systime $JVM $PID $CPU $MEM "
sleep 5
done
}
export -f doit
parallel -j0 --linebuffer --tag doit ::: aaa00{1..5}_bcdefx01 >> $LOGFILE
The function is basically your code. The change is that it takes the JVM as argument and it prints to stdout (standard output). GNU Parallel calls the function with the arguments aaa00N_bcdefx01 where N = 1..5, and saves the output to $LOGFILE. It uses --linebuffer to pass the output as soon as there is a full line, and thus guarantees that you will not get half-a-line from one process mixed with a line from another process. --tag prepends the line with the JVM.

How about using subshell?
Each JVM shell script should go inside '(' and ')'. Put '&' at the end so that it executes in the background.
An example is given here.
#!/bin/bash
echo > testfile.txt
echo "execute subshell 1"
(
#JVM 1 should go here
sleep 10
echo "subshell 1" >> testfile
)&
echo "execute subshell 2"
(
#JVM 2 should go here
sleep 10
echo "subshell 2" >> testfile
)&
echo "execute subshell 3"
(
#JVM 3 should go here
sleep 10
echo "subhsell 3" >> testfile
)&
Here each subshell writes data to testfile.txt after waiting for 10 seconds.

Related

Print output of two commands in one line

I'm got this working:
while sleep 5s
do
lscpu | grep 'CPU MHz:' | cut -d ':' -f 2 | awk '{$1=$1};1' && grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}'
done
And it gives me the following output:
1601.058
3.4811%
1452.514
3.48059%
1993.800
3.48006%
2085.585
3.47955%
2757.776
3.47902%
1370.237
3.47851%
1497.903
3.47798%
But I'd really like to get the two values onto a single line. Every time I try to do this I run into a double / single quote variable issue. Granted I pulled some of this awk stuff from online so I'm not really up to speed on that. I just want to print per line, CPU clock and load ever 5 seconds.
Can you help me find a better way to do that?
You may use process substitution to run lscpu and cat /proc/stat and feed to single command. No need to use pipes.
while sleep 5; do
awk '/CPU MHz:/{printf "%s ", $NF} /cpu /{print ($2+$4)*100/($2+$4+$5)"%"}' <(lscpu) /proc/stat
done
If there is only one input command:
date| awk '{print $1}'
Wed
OR
awk '{print $NF}' <(date)
2019
If more then one command: Example , get the year of of the two date command in same line. (not very useful example, only for sake of demo)
awk '{printf "%s ", $1=NF}END{print ""}' <(date) <(date)
2019 2019
pipe the output of the 2 commands into paste
while sleep 5; do
lscpu | awk -F':[[:blank:]]+' '$1 == "CPU MHz" {print $2}'
awk '$1 == "cpu" {printf "%.4f%%\n", ($2+$4)*100/($2+$4+$5)}' /proc/stat
done | paste - -
The 2 columns will be separated by a tab.
Writing this for readability rather than efficiency, you might consider something like:
while sleep 5; do
cpu_pct=$(lscpu | awk -F': +' '/CPU MHz:/ { print $2 }')
usage=$(awk '/cpu / {usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}' /proc/stat)
printf '%s\n' "$cpu_pct $usage"
done
Command substitutions implicitly trim trailing newlines, so if lscpu | awk has output that ends in a newline, var=$(lscpu | awk) removes it; thereafter, you can use "$var" without that newline showing up.
All you need to do is change the newline on the first line to a different separator. Something like:
lscpu | ... | tr \\n : && grep ...
You can also use echo -n $(command_with_stdout). The -n switch specifies that the new line (\n) will be omitted.
while sleep 5s; do
echo -n $( lscpu | grep 'CPU MHz:' | cut -d ':' -f 2 | awk '{$1=$1};1' )
echo -n ' **** '
echo $( grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}' )
done
Or the same representation in one line:
while sleep 5s; do echo -n $( lscpu | grep 'CPU MHz:' | cut -d ':' -f 2 | awk '{$1=$1};1' ); echo -n ' **** '; echo $( grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}' ); done
EDIT: (remove -n switch from echo according to Charles Duffy's comment)
while sleep 5s; do echo "$( lscpu | grep 'CPU MHz:' | cut -d ':' -f 2 | awk '{$1=$1};1' ) **** $( grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}' )"; done

Why does my awk redirection not work?

Im trying to redirect my output to replace the contents of my file but if I do this it doesn't change my output at all
#!/bin/bash
ssh_config_path="$HOME/.ssh/config"
temp_ssh_config_path="$HOME/.ssh/config_temporary"
new_primary_username=$1
curr_primary_username=`awk '/^Host github\.com$/,/#Username/{print $2}' $ssh_config_path | tail -1`
new_user_name=`awk "/^Host github-$new_primary_username$/,/#Name/{print $2}" $ssh_config_path | tail -1 | sed 's/#Name //' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'`
new_user_email=`awk "/^Host github-$new_primary_username$/,/#Email/{print $2}" $ssh_config_path | tail -1 | sed 's/#Email //' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'`
echo "Switching from $curr_primary_username to $new_primary_username"
echo "Setting name to $new_user_name"
echo "Setting email to $new_user_email"
awk "
!x{x=sub(/github-$new_primary_username/,\"github.com\")}
!y{y=sub(/github\.com/,\"github-$curr_primary_username\")}
1" $ssh_config_path > temp_ssh_config_path && mv temp_ssh_config_path ssh_config_path
but if I do this I get the correct output on my terminal screen
#!/bin/bash
ssh_config_path="$HOME/.ssh/config"
temp_ssh_config_path="$HOME/.ssh/config_temporary"
new_primary_username=$1
curr_primary_username=`awk '/^Host github\.com$/,/#Username/{print $2}' $ssh_config_path | tail -1`
new_user_name=`awk "/^Host github-$new_primary_username$/,/#Name/{print $2}" $ssh_config_path | tail -1 | sed 's/#Name //' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'`
new_user_email=`awk "/^Host github-$new_primary_username$/,/#Email/{print $2}" $ssh_config_path | tail -1 | sed 's/#Email //' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'`
echo "Switching from $curr_primary_username to $new_primary_username"
echo "Setting name to $new_user_name"
echo "Setting email to $new_user_email"
awk "
!x{x=sub(/github-$new_primary_username/,\"github.com\")}
!y{y=sub(/github\.com/,\"github-$curr_primary_username\")}
1" $ssh_config_path
It's disappointing how far you've veered from the answers you were given but in any case here's the correct syntax for your script (untested since you didn't provide any sample input/output):
#!/bin/bash
ssh_config_path="$HOME/.ssh/config"
temp_ssh_config_path="$HOME/.ssh/config_temporary"
new_primary_username="$1"
curr_primary_username=$(awk 'f&&/#Username/{print $2; exit} /^Host github\.com$/{f=1}' "$ssh_config_path")
new_user_name=$(awk -v npu="$new_primary_username" 'f&&/#Name/{print $2; exit} $0~"^Host github-"npu"$"{f=1}' "$ssh_config_path")
new_user_email=$(awk -v npu="$new_primary_username" 'f&&/#Email/{print $2; exit} $0~"^Host github-"npu"$"{f=1}' "$ssh_config_path")
echo "Switching from $curr_primary_username to $new_primary_username"
echo "Setting name to $new_user_name"
echo "Setting email to $new_user_email"
awk -v npu="$new_primary_username" -v cpu="$curr_primary_username" '
!x{x=sub("github-"npu,"github.com")}
!y{y=sub(/github\.com/,"github-"cpu)}
1' "$ssh_config_path" > temp_ssh_config_path && mv temp_ssh_config_path "$ssh_config_path"
By doing that I noticed that your last statement was:
mv temp_ssh_config_path ssh_config_path
when you probably meant:
mv temp_ssh_config_path "$ssh_config_path"
and that would have caused a problem with your expected output file being empty.
The whole thing should, of course, have been written as just 1 simple awk script.

BASH: Remove newline for multiple commands

I need some help . I want the result will be
UP:N%:N%
but the current result is
UP:N%
:N%
this is the code.
#!/bin/bash
UP=$(pgrep mysql | wc -l);
if [ "$UP" -ne 1 ];
then
echo -n "DOWN"
else
echo -n "UP:"
fi
df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t && echo -n ":"
top -bn2 | grep "Cpu(s)" | \sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \awk 'END{print 100 - $1"%"}'
You can use command substitution in your first sentence (notice you're creating a subshell in this way):
echo -n $(df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t ):

Using bash command on a variable that will be used as reference for an array

Short and direct, basically I want to use the value of $command on a variable, instead using it inside the while loop as a command itself. So:
This Works, but I think it's ugly:
#!/bin/bash
IFS=$'\n'
lsof=`which lsof`
whoami=`whoami`
while true ; do
execution_array=($(${lsof} -iTCP -P 2> /dev/null | grep ':' | grep ${whoami} | awk '{print $9}' | cut -f2 -d'>' | sort | uniq ))
for i in ${execution_array[*]}; do
echo $i
done
sleep 1
done
unset IFS
This doesn't work ( no output happens ), but i think is less ugly:
#!/bin/bash
IFS=$'\n'
lsof=`which lsof`
whoami=`whoami`
command="${lsof} -iTCP -P 2> /dev/null | grep ':' | grep ${whoami} | awk '{print $9}' | cut -f2 -d'>' | sort | uniq"
while true ; do
execution_array=($(command))
for i in ${execution_array[*]}; do
echo $i
done
sleep 1
done
unset IFS
This solved my problem:
#!/bin/bash
IFS=$'\n'
lsof=$(which lsof)
list_connections() {
${lsof} -iTCP -P 2> /dev/null | grep ':' | grep $(whoami) | awk '{print $9}' | cut -f2 -d'>' | sort | uniq
}
while true ; do
execution_array=($(list_connections))
for i in ${execution_array[*]}; do
echo $i
done
sleep 1
done
unset IFS

Tail -Fn0 and variable

This follows on from Faulty tail syntax or grep command? but I'm reading a live log entries for given conditions and when they're met continuing the execution of the rest of the script. I'm using this:
tail -Fn0 /var/log/messages | grep -q "CPU utilization" | grep -q "exceeded threshold"
FPC=$(echo $line | awk 'END { print substr($8,1,1) }')
PIC=$(echo $line | awk 'END { print substr($11,1,1) }')
echo FPC $FPC
echo PIC $PIC
echo "Running information gathering"...and rest of script.
Which works perfectly for the conditions detection and further execution, but I don't have the log entry to test for the FPC and PIC variables. I've tried wrapping the tail statement thus:
line=$(tail -Fn0 /var/log/messages | grep -q "CPU utilization" | grep -q "exceeded threshold")
but grep -q exits silently and the $line variable is blank. I've tried:
line=$(tail -Fn0 /var/log/messages | grep -m1 "CPU utilization" | grep -m1 "exceeded threshold")
which doesn't work until I attempt to CONTROL-C out of the script. Then it works fine and continues perfectly. Can someone help please?
I need the variables FPC and PIC later in the script.
Assuming that you don't need these variables later on, you could do something like this:
tail -Fn0 /var/log/messages | \
awk '/CPU utilization/ && /exceeded threshold/ {
print "FPC", substr($8,1,1); print "PIC", substr($11,1,1); exit }'
When the line matches both patterns, print the two parts of it that you are interested in and exit.
If you do need the variables, you could do something like this instead:
line=$(tail -Fn0 /var/log/messages | awk '/CPU utilization/&&/exceeded threshold/{print;exit}')
FPC=$(echo "$line" | awk '{ print substr($8,1,1) }')
PIC=$(echo "$line" | awk '{ print substr($11,1,1) }')

Resources