How to parse a variable in while loop multiple times - bash

Here is my shell script and where 'i' stands for key.txt. Key file consists of around 8 properties and its values separated by a space like shown below.
Key.txt file :
bat slservice
solr slservice
tvs kimservice
ACM kimservice
product kimservice
tax kimservice
tvs kimservice
SNB taxservice
Shell script:
#!/bin/bash
while read i
do
key=$(echo $i | awk '{print $1}')
service=`echo $i | awk '{print $2}'`
ip=`cat IP.txt | grep $service | awk '{print $2}'|awk 'NR==1{print $1}'`
echo "VALIDATING $service properties"
echo "validating $key in $service IP = $ip"
curl -X GET -H "Content-type: application/json" -H "Accept: application/json" http://${ip}/v1/config 2>/dev/null>$service.json
v1_value=$(jq ".\"$key\"" "$service.json" | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b")
echo "$key $v1_value" > file.txt
acutal_value=$(cat file.txt | grep "$key" | awk '{print $2}')
LB_name=$(cat mapping.txt | grep "$key" | awk '{print $2}')
LB_ip=$(cat LB.txt | grep "${LB_name}" | awk '{print $2}')
if [ "${acutal_value}" == "${LB_ip}" ]; then
echo "$key of $service value is matching $v1_value = $LB_ip"
else
echo "$key of $service v1/value ${acutal_value} is notmatching to GCP LB ${LB_ip}"
fi
done < key.txt
The above script is working fine and giving the exact output if we use mapping.txt file alone as a input file for LB_name variable.
But in my current situation I have 4 mapping (mapping1.txt to mapping4.txt)files and each mapping file should run only once for execution of all the props in key file and then followed by 2,3,4 files.
Here is the exact line that we are talking about:
LB_name=$(cat mapping.txt | grep "$key" | awk '{print $2}')
Waiting for suggestions !

Related

How to remove the usage of temp file and read data from the command itself

I have a shell script and I need help to make it efficient. I am using temp files to store and read the data, but I need to read the data in memory.
It collects metrics from the Postgres database using a command and fetches the metrics. My current script fetches the metrics to a temp file, then reads from it.
I want to stop using temp files and use memory instead.
The script works, I just need help to automate more and get rid of reading data from temp files.
List item
INPUT=`mktemp`
#/usr/pgsql-9.5/bin/pgbench -c1 -j1 -t 1000 -S man > $INPUT
TESTTIME=15 #seconds
echo "Waiting $TESTTIME seconds..."
/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man > $INPUT
OLDIFS=$IFS
IFS=" "
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
tps=`cat $INPUT |awk '/^tps/ {print $3}' |awk -F'.' '{print $1}' |head -n1`
update_l=`cat $INPUT |awk '/UPDATE/ {print $1}' |tail -n1`
select_l=`cat $INPUT |awk '/SELECT/ {print $1}' |tail -n1`
insert_l=`cat $INPUT |awk '/INSERT/ {print $1}' |tail -n1`
echo ${PLOTTER_PREFIX}.tps $tps kv
echo ${PLOTTER_PREFIX}.update_latency $update_l kv
echo ${PLOTTER_PREFIX}.select_latency $select_l kv
echo ${PLOTTER_PREFIX}.insert_latency $insert_l kv
#{ while read line; do
# # statsite_buildData ${PLOTTER_PREFIX}.latency average ${latency average} kv
# echo ${PLOTTER_PREFIX}.${line} kv
# done } < $INPUT
statsite_sendData
#echo $Test
IFS=$OLDIFS
rm -f $INPUT
You can capture the output of the command to a variable, like so:
output=$(/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man)
Then just use echo instead of cat and substitute $INPUT with the variable name.
tps=`echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1`
update_l=`echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1`
...
I would also suggest using $() instead of surrounding commands with backticks. So the above would become:
tps=$(echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1)
update_l=$(echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1)
...

Bash Shell Issue

currentDate="20160324"
headerDumpFile="header.txt"
#currentDate="$(date +ā€™%Y%m%dā€™)"
printf "Current date in dd/mm/yyyy format %s\n" $currentDate
contId=""
labelList="c12,playlist-play,play,pause,end,playlist-end,heartbeat,ns_st_cl"
params="corporate=abc&user=abc&password=abc&startdate=$currentDate&site=abc&extralabels=$labelList"
url="https://example.com/v1/start?$params"
a=1
while true
do
curl -D $headerDumpFile -v -k -H "Accept-Encoding:gzip" $url > $a.zip
contId= cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}'
if [ "$contId" ];then
printf "Breaking the Loop.."
break;
fi
url="https://example.com/v1/start?$params&continuationId=${contId}"
a=$((a + 1))
echo $contId
echo $url
done
When i Do echo url its giving value of contId as blank but when i do echo $contId. Its printed correctly .Please suggest
Perhaps is it what you want to achieve:
contId=$(cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}')
Or the simpler:
contId=$(awk '/X-CS-Continuation-Id:/ {print $NF}' $headerDumpFile)
Note that unlike what you were guessing, echo $contId isn't displaying anything in your code. What is displayed is the result of the bogus contId= cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}' line.

BASH: Remove newline for multiple commands

I need some help . I want the result will be
UP:N%:N%
but the current result is
UP:N%
:N%
this is the code.
#!/bin/bash
UP=$(pgrep mysql | wc -l);
if [ "$UP" -ne 1 ];
then
echo -n "DOWN"
else
echo -n "UP:"
fi
df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t && echo -n ":"
top -bn2 | grep "Cpu(s)" | \sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \awk 'END{print 100 - $1"%"}'
You can use command substitution in your first sentence (notice you're creating a subshell in this way):
echo -n $(df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t ):

A generic script for checking the health of application

I have written one shell script (Health_app.sh) which checks the health of the application. And for that it takes the name of the processes from App_Details file and checks for PID (whether it is running or not) and if it is not running and grep for that process in logs (field 3) and send email to the email id mentioned in the App_Details file (field 4).
App_Details is having records like:
process_Name|Process_description|logfile_path|email
abcd|main proceess to invoke the
dataready|/123/456/log|vikas#yahoo.com
pqrs|2nd
process..........................|/123/456/log|vikas#yahoo.com
Here is how my script looks like:
export App_Details=/home/123/sanity/App_Details
while read line
do
export procname=$(echo $line | cut -d " " -f1)
export PROCDES=$(echo $line | cut -d " " -f2)
#if ps -ef |grep [`echo $procname|awk '{print substr($0,1,1)}'`] [`echo $procname|awk '{print substr($0,2,length($0))}'`]> /dev/null
if ps -ef |grep -q [`echo $procname|awk '{print substr($0,1,1)}'`] `echo $procname|awk '{print substr($0,2,length($0))}'`
then
export part1=[`echo $procname|awk '{print substr($0,1,1)}'`]
export part2=`echo $procname|awk '{print substr($0,2,length($0))}'`
export PROCID=`ps -ef |grep $part1$part2|awk -F ' ' '{print $2}'`
else
export PROCID="OFFLINE"
trace_path=$(echo $line | cut -d " " -f3)
export mail=$(echo $line | cut -d " " -f4)
file_name=`ls -rt $trace_path/$procname*.trc 2>/dev/null | tail -1`
#export PROCDES=$(echo `tail -10 $file_name`)
(echo `tail -10 $file_name`) >> send.txt
mailx -s "Please find the alerts for your application OFFLINE services" vikas#domain.com < send.txt
fi
echo $PROCID|awk '{ printf("%-20s", $0)}'
echo $procname|awk '{ printf("%-20s", $0)}'
echo $PROCDES|awk '{ printf("%-20s\n", $0)}'
done<$App_Details
Now the issue is that grep -q is illegal for solaris and it is not working in solaris server.

BASH better way to monitor files

I've made a Bash script to monitor some server log files for certain data and my method probably isn't the most efficient.
One section specifically bugs me is that I have to write a newline to the monitored log so that the same line wont be read over continually.
Feedback would be greatly appreciated!
#!/bin/bash
serverlog=/home/skay/NewWorld/server.log
onlinefile=/home/skay/website/log/online.log
offlinefile=/home/skay/website/log/offline.log
index=0
# Creating the file
if [ ! -f "$onlinefile" ]; then
touch $onlinefile
echo "Name Date Time" >> "$onlinefile"
fi
if [ ! -f "$offlinefile" ]; then
touch $offlinefile
echo "Name Date Time" >> "$offlinefile"
fi
# Functions
function readfile {
# Login Variables
loginplayer=`tail -1 $serverlog | grep "[INFO]" | grep "joined the game" | awk '{print $4}'`
logintime=`tail -1 $serverlog | grep "[INFO]" | grep "joined the game" | awk '{print $2}'`
logindate=`tail -1 $serverlog | grep "[INFO]" | grep "joined the game" | awk '{print $1}'`
# Logout Variables
logoutplayer=`tail -1 $serverlog | grep "[INFO]" | grep "left the game" | awk '{print $4}'`
logouttime=`tail -1 $serverlog | grep "[INFO]" | grep "left the game" | awk '{print $2}'`
logoutdate=`tail -1 $serverlog | grep "[INFO]" | grep "left the game" | awk '{print $1}'`
# Check for Player Login
if [ ! -z "$loginplayer" ]; then
echo "$loginplayer $logindate $logintime" >> "$onlinefile"
echo "Player $loginplayer login detected" >> "$serverlog"
line=`grep -rne "$loginplayer" $offlinefile | cut -d':' -f1`
if [ "$line" > 1 ]; then
sed -i "$line"d $offlinefile
unset loginplayer
unset line
fi
fi
# Check for Player Logout
if [ ! -z "$logoutplayer" ]; then
echo "$logoutplayer $logoutdate $logouttime" >> "$offlinefile"
echo "Player $loginplayer logout detected" >> "$serverlog"
line=`grep -rne "$logoutplayer" $onlinefile | cut -d':' -f1`
if [ "$line" > 1 ]; then
sed -i "$line"d $onlinefile
unset logoutplayer
unset line
fi
fi
}
# Loop
while [ $index -lt 100 ]; do
readfile
done
Thanks!
instead of using multiple
tail -n 1 file
try the following construct:
tail -f file | while read line;do
echo "read: $line"
done
it will be much more reliable...and won't read the same line twice ;)
note: by using new processes of grep/awk/etc you are burning away processes...it's not that it is critical, but usually process creation is expensive...but if new lines occur rarely it's perfectly fine
where i want'ed to get is: if you are intrested, take a look at bash builting string manipulator function replace $(x/aa} ${x//aa} and friends..or try to use extended regexpes with grep

Resources