Bash ping output in csv format - bash

My aim is to transform the output (the last 2 lines) of the ping command in a CSV style.
Here are some examples:
In case there is a packet loss lower than 100% <
URL, PacketLoss, Min, Average, Max, Deviation
In case there is packet loss equal to 100%
URL, 100, -1, -1, -1, -1
My script is below, but when the packet loss is 100% the output is:
URL, 100,
So the problem is at the if statement, as it does not enter in elif, I use the same syntax as checking if the address is full or not (with "www." or not).
Can you please have a look because I tried multiple things and it did not work.
My script:
#!/bin/bash
declare site=''
declare result='';
if [[ "$1" == "www."* ]]; then
site="$1";
else
site="www.$1";
fi
result="$site";
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2);
fl=true;
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
if [ "$fl" == "true" ]; then
result="$result $(echo "$line" | cut -d',' -f3 | cut -d" " -f2 | sed -r 's/%//g')";
fl=false;
elif [[ "$line" == "ms"* ]]; then
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')";
else
result="$result -1 -1 -1 -1";
fi
done <<< "$pingOutput"
echo "$result";

This is a pretty old question but I've just stumbled upon it today. Below I paste a slight modified version of the above script that fixes the if issue and works on Mac OS.
P.S. You can uncomment the # prctg=100.0% line to see the if working.
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q | tail -n2)
fl=true
while IFS= read -r line
do
#echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Eo "[.[:digit:]]{1,10}%")
result="$result,$prctg"
fl=false
# prctg=100.0%
else
if [ "$prctg" == "100.0%" ]
then
result="$result,-1,-1,-1,-1"
else
result="$result,$(echo "$line" | cut -d' ' -f4 | sed -E 's/\//,/g')"
fi
fi
done <<< "$pingOutput"
echo "$result"
I hope it helps someone from the future! :)

Since the second line of the pingOutput was never processed (the loop ended before) the action of adding the -1 to the output was never performed.
Due to this problem I decided to capture the percentage of failure and act when no packets were returned (100%), I also simplified some expressions you used initially.
I investigated the script and came up with the following solution:
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2)
fl=true
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Po "[0-9]{0,3}(?=%)")
result="$result $prctg"
fl=false
fi
if [ "$prctg" == "100" ]
then
result="$result -1 -1 -1 -1"
else
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')"
fi
done <<< "$pingOutput"
echo "$result"

Related

Bash script, if statement in while loop, unwanted duplicate output

I'm doing a script to parse m3u files.
The goal is to retrieve the variable tag and the url.
I tested with this file.
#!/bin/bash
echo "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r line; do
tags_detect="$(echo "$line" | grep -Eo '^#EXTINF:')"
if [[ -n ${tag_detect} ]]; then
get_chno="$(echo "$line" | grep -o 'tvg-chno="[^"]*' | cut -d '"' -f2)"
get_id="$(echo "$line" | grep -o 'tvg-id="[^"]*' | cut -d '"' -f2)"
get_logo="$(echo "$line" | grep -o 'tvg-logo="[^"]*' | cut -d '"' -f2)"
get_grp_title="$(echo "$line" | grep -o 'group-title="[^"]*' | cut -d '"' -f2)"
get_title="$(echo "$line" | grep -o ',[^*]*' | cut -d ',' -f2)"
get_tvg_name="$(echo "$line" | grep -o 'tvg-name="[^"]*' | cut -d '"' -f2)"
get_country="$(echo "$line" | grep -o 'tvg-country="[^"]*' | cut -d '"' -f2)"
get_language="$(echo "$line" | grep -o 'tvg-language="[^"]*' | cut -d '"' -f2)"
phrase="${get_title},${get_id},${get_tvg_name},${get_country},${get_grp_title},${get_language}"
else
url="$line"
fi
echo "${phrase},${url}"
done <"${1}"
So, Without "If" it works but i don't have url.
I add a "IF" and ... :
,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
... It's broken and I don't found my error.
desired output:
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://1.2.3.4/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
I don't understand my mistake.
It's broken and I don't found my error.
Paste you script at https://shellcheck.net for validation/recommendation.
Here is how I would do it in bash.
#!/usr/bin/env bash
printf '%s\n' "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r data; do
[[ $data != '#EXTINF:-1'* ]] && continue
IFS= read -r url && [[ $url != 'http'* ]] && echo "$url" && continue
if [[ "$data" == '#EXTINF:-1'* && "$url" == 'http'* ]]; then
title=${data#*\",}
tvg_id=${data#*tvg-id=\"} tvg_id=${tvg_id%%\"*}
tvg_name=${data#*tvg-name=\"} tvg_name=${tvg_name%%\"*}
tvg_country=${data#*tvg-country=\"} tvg_country=${tvg_country%%\"*}
group_title=${data#*group-title=\"} group_title=${group_title%%\",*}
tvg_language=${data#*tvg-language=\"} tvg_language=${tvg_language%%\"*}
printf '%s,%s,%s,%s,%s,%s,%s\n' "$title" "$tvg_id" "$tvg_name" "$tvg_country" "$group_title" "$tvg_language" "$url"
fi
done < file.txt
Although I'm not sure what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
An ed solution if available/acceptable.
The script, name it anything you like. I'll just name it script.ed
g/^#EXTINF:-1/s/$/ /\
;/^http\(s\)\{0,1\}.*/-1;/^[^#]*$/j
,s/^#EXTINF:-1 tvg-id="\([^"]*\)" tvg-name="\([^"]*\)" tvg-country="\([^"]*\)" tvg-language="\([^"]*\).* group-title="\([^"]*\)",\(.*\) \(http.*\)\{0,1\}/\6,\1,\2,\3,\5,\4,\7/
1c
name,tvg-id,tvg-name,tvg-country,group-title,languages,url
.
,p
Q
Now run it against the file in question.
ed -s file.txt < script.ed
Remove the ,p from the script to silence the output to stdout or if you're satisfied with the output.
Change Q to w from the script if in-place editing is needed.
Should give more or less same result as the bash solution, but since it is still unknown what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
You probably better use a more capable language like Perl.
#! /usr/bin/perl
use strict;
use warnings;
print "name,tvg-id,tvg-name,tvg-country,group-title,languages,url\n";
my %tags;
my $title;
while (<>)
{
next if /^#EXTM3U/;
if (s/^#EXTINF:-1//) {
%tags = ();
$tags{$1} = $2 while (s/\s*(\S+)="([^"]*)"//);
($title) = $_ =~ /,(.*)/;
} else {
print join (',', $title,
$tags{'tvg-id'},
$tags{'tvg-name'},
$tags{'tvg-country'},
$tags{'group-title'},
$tags{'tvg-language'},
$_);
}
}
A quick refactor (untested) -
declare -A tag
while IFS= read -r line; do
case "$line" in
\#EXTINF:*)
if [[ "$line" =~ ,([^*]+) && -n "${BASH_REMATCH[0]}" ]]; then
tag[title]="${BASH_REMATCH[0]}"
phrase="${tag[title]}"
fi
for id in tvg-id tvg-name tvg-country group-title tvg-language tvg-chno tvg-logo; do
pat=$id'="([^"]+)"'
[[ "$line" =~ $pat ]] && tag[$id]="${BASH_REMATCH[0]}";
phrase="$phrase,${tag[$id]}"
done
phrase="${phrase%,${tag[tvg-chno]},${tag[tvg-logo]}}"
;;
*) url="$line"
esac
echo "${phrase},${url}"
done <"${1}"
Needs a lot more error checking...

How does process substitution work with while loops?

I'm reading/editing a bash git integration script
This snippet is supposed to print ${SYMBOL_GIT_PUSH} or ${SYMBOL_GIT_PULL} alongside how many commits i am behind and/or ahead by.
local marks
while IFS= read -r line; do
if [[ $line =~ ^## ]]; then
[[ $line =~ ahead\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PUSH}"
[[ $line =~ behind\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PULL}"
else
marks="${SYMBOL_GIT_MODIFIED}${marks}"
break
fi
done < <(git status --porcelain --branch 2>/dev/null)
printf '%s' "$marks"
Example:
4↑ 10↓
It is working, but i am trying to understand it.
Why is there some IFS and how does it work with process substitution?
I've heard process isn't defined in sh. Is there a way to do this the /bin/sh way or at least more efficiently?
I was provided with a link that should explain what IFS does.
I switched mixed up things and managed to remove the process substitution:
local marks
git status --porcelain --branch 2>/dev/null |
while IFS= read -r line; do
if [[ $line =~ ^## ]]; then
[[ $line =~ ahead\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PUSH}"
[[ $line =~ behind\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PULL}"
else
marks="${SYMBOL_GIT_MODIFIED}${marks}"
break
fi
done
printf '%s\n' "$marks"
But now, the value of $marks isn't saved and it prints nothing.
I was provided with another link that explains why.
Will return and update on what i've found.
I used the command grouping workaround and wrapped the loop and the print statement inside curly braces:
Also, i made the /bin/sh version almost functional (the exception - show how much commits i'm ahead or behind, not hard, i'm sure i'll do something with awk or cut).
I took advantage of fact that grep returns non-0 when nothing matches.
git status --porcelain --branch 2>/dev/null | {
SYMBOL_GIT_PUSH='↑'
SYMBOL_GIT_PULL='↓'
while IFS= read -r line
do
if echo "$line" | egrep -q '^##'
then
echo "$line" | egrep -q 'ahead' && marks="$marks $SYMBOL_GIT_PUSH"
echo "$line" | egrep -q 'behind' && marks="$marks $SYMBOL_GIT_PULL"
else
marks="*$marks"
break
fi
done
printf ' %s' "$marks"
}
This was a fun learning experience! Thanks to everyone who helped. When i find the 100% solution i'll update this.
Here's the bashism-less git info function.
__git() {
git_eng="env LANG=C git"
ref="$($git_eng symbolic-ref --short HEAD 2>/dev/null)"
[ -n "$ref" ] && ref="$SYMBOL_GIT_BRANCH$ref" || ref="$($git_eng describe --tags --always 2>/dev/null)"
[ -n "$ref" ] || return;
git status --porcelain --branch 2>/dev/null | {
SYMBOL_GIT_PUSH='↑'
SYMBOL_GIT_PULL='↓'
while IFS= read -r line
do
if echo "$line" | grep -E -q '^##'
then
echo "$line" | grep -E -q 'ahead' &&
marks="$marks $SYMBOL_GIT_PUSH$(echo "$line" | sed 's/.*\[ahead //g' | sed 's/\].*//g')"
echo "$line" | grep -E -q 'behind' &&
marks="$marks $SYMBOL_GIT_PULL$(echo "$line" | sed 's/.*\[behind //g' | sed 's/\].*//g')"
else
marks="$SYMBOL_GIT_MODIFIED$marks"
break
fi
done
printf ' %s%s' "$ref" "$marks"
}
}
sed searches for [ahead and deletes it, as well as everything before it, then it pipes it into another sed which deletes everything past ]. This way only the number remains.

Bash - disk utilization notification

This script should output a warning notification for the utilization of the main disk if over 50%, but it provides no output. My disk is currently sat at 60% so it should in theory work.
I have added an else statement to identify if the loop is not working but the else statement isnt triggered.
I'm provided no error so its hard to identify where i have gone wrong specifically.
#!/bin/bash
df -H | grep /dev/sda2 | awk '{ printf "%d", $5}' > diskOutput.txt
input="diskOutput.txt"
while IFS= read -r line
do
if [ $line -gt 50 ]
then
up="`uptime | cut -b 1-9`"
output="WARNING UTILISATION $line - $up"
echo "$output"
else
echo "no-in"
fi
done < $input
#rm diskOutput.txt
echo "finished"
Try this.
#!/bin/bash
df -H | grep /dev/sda2 | awk '{ printf "%d", $5}' > diskOutput.txt
echo "" >>diskOutput.txt
input="diskOutput.txt"
while IFS= read -r line
do
if [ $line -gt 50 ]
then
up="`uptime | cut -b 1-9`"
output="WARNING UTILISATION $line - $up"
echo "$output"
else
echo "no-in"
fi
done < $input
#rm diskOutput.txt
echo "finished"
You are setting an internal field separator as space here.
while IFS= read -r line
But when creating file, with %d you are removing all char except digits.

Bash - Extract Matching String from GZIP Files Is Running Very Slow

Complete novice in Bash. Trying to iterate thru 1000 gzip files, may be GNU parallel is the solution??
#!/bin/bash
ctr=0
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
for f in "$dir"/*.gz; do
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr" >> $1
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym
else
ctr=$((ctr+1))
fi
done
done
Any help to speed the process will be greatly appreciated !!!
#!/bin/bash
ctr=0
export ctr
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
export dir
doit() {
f="$1"
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr"
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym >&2
else
ctr=$((ctr+1))
fi
done
}
export -f doit
parallel doit ::: *gz 2>&1 > $1
The Bash while read loop is probably your main bottleneck here. Calling multiple external processes for simple field splitting will exacerbate the problem. Briefly,
while IFS="|" read -r first second third rest; do ...
leverages the shell's built-in field splitting functionality, but you probably want to convert the whole thing to a simple Awk script anyway.
echo "file_name,symbol,record_count" > "$1"
for f in "/data/myfolder"/*.gz; do
gunzip -c "$f" |
awk -F "\|" -v f="$f" -v OFS="," '
/H/ { if(ctr) print f, sym, ctr
ctr=0; sym=$3;
print sym >"/dev/stderr"
next }
{ ++ctr }'
done >>"$1"
This vaguely assumes that printing the lone sym is just for diagnostics. It should hopefully not be hard to see how this can be refactored if this is an incorrect assumption.

Intermittent piping failure in bash

I have a code snippet that looks like this
while grep "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#"; do
grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do
lineno=$(echo $line | cut -d':' -f1)
spaces=$(sed "${lineno}!d" /tmp/kubernetes/$basefile | awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" /tmp/kubernetes/$basefile
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" /tmp/kubernetes/$basefile
lineno=$((lineno+1))
done < "/tmp/secrets.yaml"
done
done
in /tmp/kubernetes/$basefile, the string {{SECRETS}} appears twice 100% of the time.
Almost every single time, this completes fine. However, very infrequently, the script errors on its second loop through the file. like so, according to set -x
...
IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
exit code 1
When it works, the set -x looks like this, and continues processesing the file correctly.
...
+ IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
+ grep '{{SECRETS}}' /tmp/kubernetes/deployment.yaml
+ grep -v '#'
I have no answer for how this can only happen occasionally, so I think there's something about bash piping's parallelism I don't understand. Is there something in grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do that could lead to out-of-order execution somehow? Based on the error, it seems like it's trying to read a line, but can't because previous commands didn't work. But there's no indication of that in the set -x output.
A likely cause of the problem is that the pipeline containing the inner loop both reads and writes the "basefile" at the same time. See How to make reading and writing the same file in the same pipeline always “fail”?.
One way to fix the problem is do a full read of the file before trying to update it. Try:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
while
line=$(grep -n "{{SECRETS}}" "$basepath" | grep -v "#" | head -n1)
[[ -n $line ]]
do
lineno=$(echo "$line" | cut -d':' -f1)
spaces=$(sed "${lineno}!d" "$basepath" \
| awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" "$basepath"
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" "$basepath"
lineno=$((lineno+1))
done < "$secretspath"
done
(I introduced the variables basepath and secretspath to make the code easier to test.)
As an aside, it's also possible to do this with pure Bash code:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
updated_lines=()
is_updated=0
while IFS= read -r line || [[ -n $line ]] ; do
if [[ $line == *'{{SECRETS}}'* && $line != *'#'* ]] ; then
spaces=${line%%[^[:space:]]*}
while IFS= read -r secretline || [[ -n $secretline ]]; do
updated_lines+=( "${spaces}${secretline}" )
done < "$secretspath"
is_updated=1
else
updated_lines+=( "$line" )
fi
done <"$basepath"
(( is_updated )) && printf '%s\n' "${updated_lines[#]}" >"$basepath"
The whole updated file is stored in memory (in the update_lines array) but that shouldn't be a problem because any file that's too big to store in memory will almost certainly be too big to process line-by-line with Bash. Bash is generally extremely slow.
In this code spaces holds the actual space characters for indentation, not the number of them.

Resources