I'm doing a script to parse m3u files.
The goal is to retrieve the variable tag and the url.
I tested with this file.
#!/bin/bash
echo "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r line; do
tags_detect="$(echo "$line" | grep -Eo '^#EXTINF:')"
if [[ -n ${tag_detect} ]]; then
get_chno="$(echo "$line" | grep -o 'tvg-chno="[^"]*' | cut -d '"' -f2)"
get_id="$(echo "$line" | grep -o 'tvg-id="[^"]*' | cut -d '"' -f2)"
get_logo="$(echo "$line" | grep -o 'tvg-logo="[^"]*' | cut -d '"' -f2)"
get_grp_title="$(echo "$line" | grep -o 'group-title="[^"]*' | cut -d '"' -f2)"
get_title="$(echo "$line" | grep -o ',[^*]*' | cut -d ',' -f2)"
get_tvg_name="$(echo "$line" | grep -o 'tvg-name="[^"]*' | cut -d '"' -f2)"
get_country="$(echo "$line" | grep -o 'tvg-country="[^"]*' | cut -d '"' -f2)"
get_language="$(echo "$line" | grep -o 'tvg-language="[^"]*' | cut -d '"' -f2)"
phrase="${get_title},${get_id},${get_tvg_name},${get_country},${get_grp_title},${get_language}"
else
url="$line"
fi
echo "${phrase},${url}"
done <"${1}"
So, Without "If" it works but i don't have url.
I add a "IF" and ... :
,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
... It's broken and I don't found my error.
desired output:
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://1.2.3.4/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
I don't understand my mistake.
It's broken and I don't found my error.
Paste you script at https://shellcheck.net for validation/recommendation.
Here is how I would do it in bash.
#!/usr/bin/env bash
printf '%s\n' "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r data; do
[[ $data != '#EXTINF:-1'* ]] && continue
IFS= read -r url && [[ $url != 'http'* ]] && echo "$url" && continue
if [[ "$data" == '#EXTINF:-1'* && "$url" == 'http'* ]]; then
title=${data#*\",}
tvg_id=${data#*tvg-id=\"} tvg_id=${tvg_id%%\"*}
tvg_name=${data#*tvg-name=\"} tvg_name=${tvg_name%%\"*}
tvg_country=${data#*tvg-country=\"} tvg_country=${tvg_country%%\"*}
group_title=${data#*group-title=\"} group_title=${group_title%%\",*}
tvg_language=${data#*tvg-language=\"} tvg_language=${tvg_language%%\"*}
printf '%s,%s,%s,%s,%s,%s,%s\n' "$title" "$tvg_id" "$tvg_name" "$tvg_country" "$group_title" "$tvg_language" "$url"
fi
done < file.txt
Although I'm not sure what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
An ed solution if available/acceptable.
The script, name it anything you like. I'll just name it script.ed
g/^#EXTINF:-1/s/$/ /\
;/^http\(s\)\{0,1\}.*/-1;/^[^#]*$/j
,s/^#EXTINF:-1 tvg-id="\([^"]*\)" tvg-name="\([^"]*\)" tvg-country="\([^"]*\)" tvg-language="\([^"]*\).* group-title="\([^"]*\)",\(.*\) \(http.*\)\{0,1\}/\6,\1,\2,\3,\5,\4,\7/
1c
name,tvg-id,tvg-name,tvg-country,group-title,languages,url
.
,p
Q
Now run it against the file in question.
ed -s file.txt < script.ed
Remove the ,p from the script to silence the output to stdout or if you're satisfied with the output.
Change Q to w from the script if in-place editing is needed.
Should give more or less same result as the bash solution, but since it is still unknown what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
You probably better use a more capable language like Perl.
#! /usr/bin/perl
use strict;
use warnings;
print "name,tvg-id,tvg-name,tvg-country,group-title,languages,url\n";
my %tags;
my $title;
while (<>)
{
next if /^#EXTM3U/;
if (s/^#EXTINF:-1//) {
%tags = ();
$tags{$1} = $2 while (s/\s*(\S+)="([^"]*)"//);
($title) = $_ =~ /,(.*)/;
} else {
print join (',', $title,
$tags{'tvg-id'},
$tags{'tvg-name'},
$tags{'tvg-country'},
$tags{'group-title'},
$tags{'tvg-language'},
$_);
}
}
A quick refactor (untested) -
declare -A tag
while IFS= read -r line; do
case "$line" in
\#EXTINF:*)
if [[ "$line" =~ ,([^*]+) && -n "${BASH_REMATCH[0]}" ]]; then
tag[title]="${BASH_REMATCH[0]}"
phrase="${tag[title]}"
fi
for id in tvg-id tvg-name tvg-country group-title tvg-language tvg-chno tvg-logo; do
pat=$id'="([^"]+)"'
[[ "$line" =~ $pat ]] && tag[$id]="${BASH_REMATCH[0]}";
phrase="$phrase,${tag[$id]}"
done
phrase="${phrase%,${tag[tvg-chno]},${tag[tvg-logo]}}"
;;
*) url="$line"
esac
echo "${phrase},${url}"
done <"${1}"
Needs a lot more error checking...
Related
example
I want to make something like 'a button, when the user press this button A question will appear to him: Do you want to share your contact with the bot?,' I want like this using curl in bash script:
and I try like this, but it is doesn't working
#!/bin/bash
clear
token=""
old=$(curl -s "https://api.telegram.org/bot$token/getUpdates" | awk 'END{print}' | sed -n -e 's/^.*"date"://p' | cut -d ',' -f1)
while true; do
new=$(curl -s "https://api.telegram.org/bot$token/getUpdates" | awk 'END{print}')
if [[ "$new" =~ \"chat\"\:\{\"id\"\:([\-0-9]+)\, ]]; then
CHATID=${BASH_REMATCH[1]};
fi
if [[ "$new" =~ \"date\"\:([0-9]+)\, ]]; then
DATE=${BASH_REMATCH[1]};
fi
TEXT=$(echo "$new" | sed -n -e 's/^.*"text":"//p' | cut -d '"' -f1)
if ! [ "$DATE" = "$old" ]; then
echo "text: $TEXT, from: $USERNAME, id: $CHATID"
if [ "$TEXT" = "/start" ]; then
curl "https://api.telegram.org/bot$token/KeyboardButton?chat_id=$CHATID&text=flase&request_contact=true"
fi
old="$DATE"
fi
done
And error code is:
{"ok":false,"error_code":404,"description":"Not Found"}%
I try like this but It's now working how can I done this in right way?
https://i.stack.imgur.com/HQ8ga.jpg
Complete novice in Bash. Trying to iterate thru 1000 gzip files, may be GNU parallel is the solution??
#!/bin/bash
ctr=0
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
for f in "$dir"/*.gz; do
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr" >> $1
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym
else
ctr=$((ctr+1))
fi
done
done
Any help to speed the process will be greatly appreciated !!!
#!/bin/bash
ctr=0
export ctr
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
export dir
doit() {
f="$1"
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr"
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym >&2
else
ctr=$((ctr+1))
fi
done
}
export -f doit
parallel doit ::: *gz 2>&1 > $1
The Bash while read loop is probably your main bottleneck here. Calling multiple external processes for simple field splitting will exacerbate the problem. Briefly,
while IFS="|" read -r first second third rest; do ...
leverages the shell's built-in field splitting functionality, but you probably want to convert the whole thing to a simple Awk script anyway.
echo "file_name,symbol,record_count" > "$1"
for f in "/data/myfolder"/*.gz; do
gunzip -c "$f" |
awk -F "\|" -v f="$f" -v OFS="," '
/H/ { if(ctr) print f, sym, ctr
ctr=0; sym=$3;
print sym >"/dev/stderr"
next }
{ ++ctr }'
done >>"$1"
This vaguely assumes that printing the lone sym is just for diagnostics. It should hopefully not be hard to see how this can be refactored if this is an incorrect assumption.
I have a code snippet that looks like this
while grep "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#"; do
grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do
lineno=$(echo $line | cut -d':' -f1)
spaces=$(sed "${lineno}!d" /tmp/kubernetes/$basefile | awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" /tmp/kubernetes/$basefile
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" /tmp/kubernetes/$basefile
lineno=$((lineno+1))
done < "/tmp/secrets.yaml"
done
done
in /tmp/kubernetes/$basefile, the string {{SECRETS}} appears twice 100% of the time.
Almost every single time, this completes fine. However, very infrequently, the script errors on its second loop through the file. like so, according to set -x
...
IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
exit code 1
When it works, the set -x looks like this, and continues processesing the file correctly.
...
+ IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
+ grep '{{SECRETS}}' /tmp/kubernetes/deployment.yaml
+ grep -v '#'
I have no answer for how this can only happen occasionally, so I think there's something about bash piping's parallelism I don't understand. Is there something in grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do that could lead to out-of-order execution somehow? Based on the error, it seems like it's trying to read a line, but can't because previous commands didn't work. But there's no indication of that in the set -x output.
A likely cause of the problem is that the pipeline containing the inner loop both reads and writes the "basefile" at the same time. See How to make reading and writing the same file in the same pipeline always “fail”?.
One way to fix the problem is do a full read of the file before trying to update it. Try:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
while
line=$(grep -n "{{SECRETS}}" "$basepath" | grep -v "#" | head -n1)
[[ -n $line ]]
do
lineno=$(echo "$line" | cut -d':' -f1)
spaces=$(sed "${lineno}!d" "$basepath" \
| awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" "$basepath"
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" "$basepath"
lineno=$((lineno+1))
done < "$secretspath"
done
(I introduced the variables basepath and secretspath to make the code easier to test.)
As an aside, it's also possible to do this with pure Bash code:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
updated_lines=()
is_updated=0
while IFS= read -r line || [[ -n $line ]] ; do
if [[ $line == *'{{SECRETS}}'* && $line != *'#'* ]] ; then
spaces=${line%%[^[:space:]]*}
while IFS= read -r secretline || [[ -n $secretline ]]; do
updated_lines+=( "${spaces}${secretline}" )
done < "$secretspath"
is_updated=1
else
updated_lines+=( "$line" )
fi
done <"$basepath"
(( is_updated )) && printf '%s\n' "${updated_lines[#]}" >"$basepath"
The whole updated file is stored in memory (in the update_lines array) but that shouldn't be a problem because any file that's too big to store in memory will almost certainly be too big to process line-by-line with Bash. Bash is generally extremely slow.
In this code spaces holds the actual space characters for indentation, not the number of them.
My bash script is the following:
#!/bin/bash
if [ ! -f "$1" ]; then
exit
fi
while read line;do
str1="[GAC]*T"
num=$"(echo $line | tr -d -c 'T' | wc -m)"
for((i=0;i<$num;i++))do
echo $line | sed "s/$str1/&\n/" | head -n1 -q
str1="${str1}[GAC]*T"
done
str1="[GAC]*T"
done < "$1
While it works normally as it should (take the filename input and print it line by line until the letter T and next letter T and so on) it prints to the terminal.
Input:
GATTT
ATCGT
Output:
GAT
GATT
GATTT
AT
ATCGT
When I'm using the script with | tee outputfile the outputfile is correct but when using the script with > outputfile the terminal hangs / is stuck and does not finish. Moreover it works with bash -x scriptname inputfile > outputfile but is stuck with bash scriptname inputfile > outputfile.
I made modifications to your original script, please try:
if [ ! -f "$1" ]; then
exit
fi
while IFS='' read -r line || [[ -n "$line" ]];do
str1="[GAC]*T"
num=$(echo $line | tr -d -c 'T' | wc -m)
for((i=0;i<$num;i++));do
echo $line | sed "s/$str1/&\n/" | head -n1 -q
str1="${str1}[GAC]*T"
done
str1="[GAC]*T"
done < "$1"
For input:
GATTT
ATCGT
This script outputs:
GAT
GATT
GATTT
AT
ATCGT
Modifications made to your original script were:
Line while read line; do changed to while IFS='' read -r line || [[ -n "$line" ]]; do. Why I did this is explained here: Read a file line by line assigning the value to a variable
Line num=$"(echo $line | tr -d -c 'T' | wc -m)" changed to num=$(echo $line | tr -d -c 'T' | wc -m)
Line for((i=0;i<$num;i++))do changed to for((i=0;i<$num;i++));do
Line done < "$1 changed to done < "$1"
Now you can do: ./scriptname inputfile > outputfile
Try:
sed -r 's/([^T]*T+)/\1\n/g' gatc.txt > outputfile
instead of your script.
It takes some optional non-Ts, followed by at least one T and inserts a newline after the T.
cat gatc.txt
GATGATTGATTTATATCGT
sed -r 's/([^T]*T+)/\1\n/g' gatc.txt
GAT
GATT
GATTT
AT
AT
CGT
For multiple lines, to delete empty lines in the end:
echo "GATTT
ATCGT" | sed -r 's/([^T]*T+)/\1\n/g;' | sed '/^$/d'
GATTT
AT
CGT
My aim is to transform the output (the last 2 lines) of the ping command in a CSV style.
Here are some examples:
In case there is a packet loss lower than 100% <
URL, PacketLoss, Min, Average, Max, Deviation
In case there is packet loss equal to 100%
URL, 100, -1, -1, -1, -1
My script is below, but when the packet loss is 100% the output is:
URL, 100,
So the problem is at the if statement, as it does not enter in elif, I use the same syntax as checking if the address is full or not (with "www." or not).
Can you please have a look because I tried multiple things and it did not work.
My script:
#!/bin/bash
declare site=''
declare result='';
if [[ "$1" == "www."* ]]; then
site="$1";
else
site="www.$1";
fi
result="$site";
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2);
fl=true;
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
if [ "$fl" == "true" ]; then
result="$result $(echo "$line" | cut -d',' -f3 | cut -d" " -f2 | sed -r 's/%//g')";
fl=false;
elif [[ "$line" == "ms"* ]]; then
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')";
else
result="$result -1 -1 -1 -1";
fi
done <<< "$pingOutput"
echo "$result";
This is a pretty old question but I've just stumbled upon it today. Below I paste a slight modified version of the above script that fixes the if issue and works on Mac OS.
P.S. You can uncomment the # prctg=100.0% line to see the if working.
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q | tail -n2)
fl=true
while IFS= read -r line
do
#echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Eo "[.[:digit:]]{1,10}%")
result="$result,$prctg"
fl=false
# prctg=100.0%
else
if [ "$prctg" == "100.0%" ]
then
result="$result,-1,-1,-1,-1"
else
result="$result,$(echo "$line" | cut -d' ' -f4 | sed -E 's/\//,/g')"
fi
fi
done <<< "$pingOutput"
echo "$result"
I hope it helps someone from the future! :)
Since the second line of the pingOutput was never processed (the loop ended before) the action of adding the -1 to the output was never performed.
Due to this problem I decided to capture the percentage of failure and act when no packets were returned (100%), I also simplified some expressions you used initially.
I investigated the script and came up with the following solution:
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2)
fl=true
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Po "[0-9]{0,3}(?=%)")
result="$result $prctg"
fl=false
fi
if [ "$prctg" == "100" ]
then
result="$result -1 -1 -1 -1"
else
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')"
fi
done <<< "$pingOutput"
echo "$result"