How to convert decimal number to Base58 in Bash - bash

myNumber=$(date +%s) # big number in decimal
myNumberInB58=$(toBase58 $myNumber)
toBase58() {
# <your answer here>
}
What is the most elegant and/or concise way to encode an integer in Base58?

The bitcoin-bash-tools provide the functions {en,de}codeBase58:
decodeBase58() {
echo -n "$1" | sed -e's/^\(1*\).*/\1/' -e's/1/00/g' | tr -d '\n'
dc -e "$dcr 16o0$(sed 's/./ 58*l&+/g' <<<$1)p" |
while read n; do echo -n ${n/\\/}; done
}
encodeBase58() {
echo -n "$1" | sed -e's/^\(\(00\)*\).*/\1/' -e's/00/1/g' | tr -d '\n'
dc -e "16i ${1^^} [3A ~r d0<x]dsxx +f" |
while read -r n; do echo -n "${base58[n]}"; done
}
Those work with the fields dcr and base58 defined directly above in the file.

Would this do?
a=( {1..9} {A..H} {J..N} {P..Z} {a..k} {m..z} )
toBase58() {
# TODO: check that $1 is a valid number
local nb=$1 b58= fiftyeight=${#a[#]}
while ((nb)); do
b58=${a[nb%fiftyeight]}$b58
((nb/=fiftyeight))
done
printf '%s\n' "$b58"
}

Here's another version:
# Order-Preserving Base58 (OPB58).
# Also supports negative numbers.
int2b58() {
# Omit IOlo
local n="$1" i BASE58=$(echo {0..9} {A..H} {J..N} {P..Z} {a..k} {m..n} {p..z} | tr -d ' ')
((n < 0 )) && printf -- '-' && n=$((-n))
for i in $(echo "obase=58; $n" | bc); do
printf ${BASE58:$(( 10#$i )):1}
done; echo
}

Related

Bash script, if statement in while loop, unwanted duplicate output

I'm doing a script to parse m3u files.
The goal is to retrieve the variable tag and the url.
I tested with this file.
#!/bin/bash
echo "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r line; do
tags_detect="$(echo "$line" | grep -Eo '^#EXTINF:')"
if [[ -n ${tag_detect} ]]; then
get_chno="$(echo "$line" | grep -o 'tvg-chno="[^"]*' | cut -d '"' -f2)"
get_id="$(echo "$line" | grep -o 'tvg-id="[^"]*' | cut -d '"' -f2)"
get_logo="$(echo "$line" | grep -o 'tvg-logo="[^"]*' | cut -d '"' -f2)"
get_grp_title="$(echo "$line" | grep -o 'group-title="[^"]*' | cut -d '"' -f2)"
get_title="$(echo "$line" | grep -o ',[^*]*' | cut -d ',' -f2)"
get_tvg_name="$(echo "$line" | grep -o 'tvg-name="[^"]*' | cut -d '"' -f2)"
get_country="$(echo "$line" | grep -o 'tvg-country="[^"]*' | cut -d '"' -f2)"
get_language="$(echo "$line" | grep -o 'tvg-language="[^"]*' | cut -d '"' -f2)"
phrase="${get_title},${get_id},${get_tvg_name},${get_country},${get_grp_title},${get_language}"
else
url="$line"
fi
echo "${phrase},${url}"
done <"${1}"
So, Without "If" it works but i don't have url.
I add a "IF" and ... :
,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,#EXTM3U
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,http://51.210.199.30/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
... It's broken and I don't found my error.
desired output:
4 Turk Music,4TurkMusic.fr,4 Turk Music,FR;TK,Music,Turkish,http://1.2.3.4/hls/stream.m3u8
Alpe d’Huez TV,AlpedHuezTV.fr,Alpe d’Huez TV,FR,,French,https://edge10.vedge.infomaniak.com/livecast/ik:adhtv/chunklist.m3u8
I don't understand my mistake.
It's broken and I don't found my error.
Paste you script at https://shellcheck.net for validation/recommendation.
Here is how I would do it in bash.
#!/usr/bin/env bash
printf '%s\n' "name,tvg-id,tvg-name,tvg-country,group-title,languages,url"
while IFS= read -r data; do
[[ $data != '#EXTINF:-1'* ]] && continue
IFS= read -r url && [[ $url != 'http'* ]] && echo "$url" && continue
if [[ "$data" == '#EXTINF:-1'* && "$url" == 'http'* ]]; then
title=${data#*\",}
tvg_id=${data#*tvg-id=\"} tvg_id=${tvg_id%%\"*}
tvg_name=${data#*tvg-name=\"} tvg_name=${tvg_name%%\"*}
tvg_country=${data#*tvg-country=\"} tvg_country=${tvg_country%%\"*}
group_title=${data#*group-title=\"} group_title=${group_title%%\",*}
tvg_language=${data#*tvg-language=\"} tvg_language=${tvg_language%%\"*}
printf '%s,%s,%s,%s,%s,%s,%s\n' "$title" "$tvg_id" "$tvg_name" "$tvg_country" "$group_title" "$tvg_language" "$url"
fi
done < file.txt
Although I'm not sure what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
An ed solution if available/acceptable.
The script, name it anything you like. I'll just name it script.ed
g/^#EXTINF:-1/s/$/ /\
;/^http\(s\)\{0,1\}.*/-1;/^[^#]*$/j
,s/^#EXTINF:-1 tvg-id="\([^"]*\)" tvg-name="\([^"]*\)" tvg-country="\([^"]*\)" tvg-language="\([^"]*\).* group-title="\([^"]*\)",\(.*\) \(http.*\)\{0,1\}/\6,\1,\2,\3,\5,\4,\7/
1c
name,tvg-id,tvg-name,tvg-country,group-title,languages,url
.
,p
Q
Now run it against the file in question.
ed -s file.txt < script.ed
Remove the ,p from the script to silence the output to stdout or if you're satisfied with the output.
Change Q to w from the script if in-place editing is needed.
Should give more or less same result as the bash solution, but since it is still unknown what should happen at line 233 and 238 those lines starts with #EXTVLCOPT
You probably better use a more capable language like Perl.
#! /usr/bin/perl
use strict;
use warnings;
print "name,tvg-id,tvg-name,tvg-country,group-title,languages,url\n";
my %tags;
my $title;
while (<>)
{
next if /^#EXTM3U/;
if (s/^#EXTINF:-1//) {
%tags = ();
$tags{$1} = $2 while (s/\s*(\S+)="([^"]*)"//);
($title) = $_ =~ /,(.*)/;
} else {
print join (',', $title,
$tags{'tvg-id'},
$tags{'tvg-name'},
$tags{'tvg-country'},
$tags{'group-title'},
$tags{'tvg-language'},
$_);
}
}
A quick refactor (untested) -
declare -A tag
while IFS= read -r line; do
case "$line" in
\#EXTINF:*)
if [[ "$line" =~ ,([^*]+) && -n "${BASH_REMATCH[0]}" ]]; then
tag[title]="${BASH_REMATCH[0]}"
phrase="${tag[title]}"
fi
for id in tvg-id tvg-name tvg-country group-title tvg-language tvg-chno tvg-logo; do
pat=$id'="([^"]+)"'
[[ "$line" =~ $pat ]] && tag[$id]="${BASH_REMATCH[0]}";
phrase="$phrase,${tag[$id]}"
done
phrase="${phrase%,${tag[tvg-chno]},${tag[tvg-logo]}}"
;;
*) url="$line"
esac
echo "${phrase},${url}"
done <"${1}"
Needs a lot more error checking...

Bash - Extract Matching String from GZIP Files Is Running Very Slow

Complete novice in Bash. Trying to iterate thru 1000 gzip files, may be GNU parallel is the solution??
#!/bin/bash
ctr=0
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
for f in "$dir"/*.gz; do
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr" >> $1
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym
else
ctr=$((ctr+1))
fi
done
done
Any help to speed the process will be greatly appreciated !!!
#!/bin/bash
ctr=0
export ctr
echo "file_name,symbol,record_count" > $1
dir="/data/myfolder"
export dir
doit() {
f="$1"
gunzip -c $f | while read line;
do
str=`echo $line | cut -d"|" -f1`
if [ "$str" == "H" ]; then
if [ $ctr -gt 0 ]; then
echo "$f,$sym,$ctr"
fi
ctr=0
sym=`echo $line | cut -d"|" -f3`
echo $sym >&2
else
ctr=$((ctr+1))
fi
done
}
export -f doit
parallel doit ::: *gz 2>&1 > $1
The Bash while read loop is probably your main bottleneck here. Calling multiple external processes for simple field splitting will exacerbate the problem. Briefly,
while IFS="|" read -r first second third rest; do ...
leverages the shell's built-in field splitting functionality, but you probably want to convert the whole thing to a simple Awk script anyway.
echo "file_name,symbol,record_count" > "$1"
for f in "/data/myfolder"/*.gz; do
gunzip -c "$f" |
awk -F "\|" -v f="$f" -v OFS="," '
/H/ { if(ctr) print f, sym, ctr
ctr=0; sym=$3;
print sym >"/dev/stderr"
next }
{ ++ctr }'
done >>"$1"
This vaguely assumes that printing the lone sym is just for diagnostics. It should hopefully not be hard to see how this can be refactored if this is an incorrect assumption.

how will I fix the echo when counting chars in a string

I have an issue with the echo in the for loop, as I want to count string chars especailly for "*" it but it prints all the files in current directory.
clearvar() {
int=0
str=0
uniqchar=0
}
countstring(){
for c in $(echo "${1}" | fold -w1); do
echo "$c"
if [[ $c == [0-9] ]];then
int=$(( $int + 1 ))
elif [[ $c == [a-Z] ]];then
str=$(( $str + 1 ))
else
uniqchar=$(( $uniqchar + 1 ))
fi
done
}
while [ $# -gt 0 ];do
echo "Argument input: $1"
read -p "Input: " string
rmws=$(echo $string | tr -d " ")
mashed=$rmws$1
countstring $mashed
echo -e "int: $int\nstr: $str\nuniquechar: $uniqchar\nWhole string: $mashed"
clearvar
shift
done
Example output:
Argument input: io1
Input: fj^*23
f
j
^
file1
file2
file3
2
3
i
o
1
int: 3
str: 4
uniquechar: 4
Whole string: fj^*2wio1
it interprets as echo * instead of echo "*".
so I expect it to not print out the file names.
rmws=$(echo $string | tr -d " ")
If string=* this just executes echo * and expands the *.
The same happens in:
countstring $mashed
Both these expansions are unquoted. Quote them in double quotes. As a rule of a thumb - always use double quotes.
Also the same happens in the for loop:
for c in $(echo "${1}" | fold -w1)
the expansion, as elsewhere, is unquoted, so * expands. You have to quote. That's why the for i in $(..) is considered bad style - because such bugs happen. You can't do for i in "$(...)" because then you would iterate over one element. To iterate over lines or elements in a stream use a while IFS= read -r loop. You can print every character on each separate line with ex. sed 's/./&\n/g' and iterate over lines, or use bash extension read -n1 to read one character.
while IFS= read -r -n1 c; do
..
done <<<"$1"
The <<<"$1" is a bash's "here string".
You don't need $ in arithmetic expansion. Just:
int=$(( int + 1 ))
str=$(( str + 1 ))
uniqchar=$(( uniqchar + 1 ))
or in bash you can even do:
(( int++ ))
# and so on
Your script could become:
clearvar() {
int=0
str=0
uniqchar=0
}
countstring(){
while IFS= read -r -n1 c; do
echo "$c"
if [[ $c == [0-9] ]];then
(( int++ ))
elif [[ $c == [a-Z] ]];then
(( str++ ))
else
(( uniqchar++ ))
fi
done <<<"$1"
}
while [ $# -gt 0 ]; do
echo "Argument input: $1"
read -p "Input: " string
rmws="$(echo "$string" | tr -d " ")"
mashed="$rmws$1"
countstring "$mashed"
echo "int: $int"
echo "str: $str"
echo "uniquechar: $uniqchar"
echo "Whole string: $mashed"
clearvar
shift
done
Notes:
echo has portability issues. Prefer to use printf instead.
I prefer while (($#)); do in place of while [ $# -eq 0 ]; do.
PS. I would use tr:
countstring() {
printf "%s" "$1" | tr -cd '[0-9]' | wc -c
printf "%s" "$1" | tr -cd '[a-zA-Z]' | wc -c
printf "%s" "$1" | tr -d '[0-9a-zA-Z]' | wc -c
}

extract a numeric substring and add value to it

I have a string like 1001.2001.3001.5001.6001 or 1001-2001-3001-5001-6001. How to extract the 4th string i.e., 5001, add a value like 121 to it and put it back in the same string. The output should be like 1001.2001.3001.5122.6001 or 1001-2001-3001-5122-6001. I have to achieve this in Linux bash scripting.
Try this
#!/bin/bash
str=$1
if [[ $(echo $str | grep '\.' | wc -l) == 1 ]]
then
str1=$(echo $str | cut -d '.' -f 1,2,3)
str2=$(echo $str | cut -d '.' -f 4 | awk {'print $1+121'})
str3=$(echo $str | cut -d '.' -f 5)
echo $str1.$str2.$str3
elif [[ $(echo $str | grep - | wc -l) == 1 ]]
then
str1=$(echo $str | cut -d '-' -f 1,2,3)
str2=$(echo $str | cut -d '-' -f 4 | awk {'print $1+121'})
str3=$(echo $str | cut -d '-' -f 5)
echo $str1-$str2-$str3
else
echo "do nothing"
fi
Pass a string as parameter
No pipes, no forks, no cutting, no awking, just plain POSIX shell:
$ s=1001.2001.3001.5001.6001
$ oldIFS=$IFS
$ IFS=.-
$ set -- $s
$ case $s in
> (*.*) echo "$1.$2.$3.$(($4 + 121)).$5";;
> (*-*) echo "$1-$2-$3-$(($4 + 121))-$5";;
> esac
1001.2001.3001.5122.6001
$ IFS=$oldIFS
One liner
value=121 ; str='1001.2001.3001.5001.6001' ; token="$(echo "$str" | cut -f 4 -d '.')" ; newtoken=$(( $token + $value )) ; newstr="$(echo "$str" | sed -e "s/$token/$newtoken/g" | tr '.' '-')" ; echo "$newstr"
Breakdown:
value=121 # <- Increment
str='1001.2001.3001.5001.6001' # <- Initial String
token="$(echo "$str" | cut -f 4 -d '.')" # <- Extract the 4th field with . sep
newtoken=$(( $token + $value )) # <- Add value and save to $newtoken
newstr="$(echo "$str" \
| sed -e "s/$token/$newtoken/g" \
| tr '.' '-')" # <- Replace 4th field with $newtoken
# and translate "." to "-"
echo "$newstr" # <- Echo new string
Works in:
Bash
sh
FreeBSD
Busybox
Using out of the box tools
If the field separator can either be . or -, then do something like
echo "1001.2001.3001.5001.6001" | awk 'BEGIN{FS="[.-]";OFS="-"}{$4+=121}1'
1001-2001-3001-5122-6001
However, if you need to match the regex FS or field separator with OFS then you need to have gawk installed
echo "1001.2001.3001.5001.6001" |
gawk 'BEGIN{FS="[.-]"}{split($0,a,FS,seps)}{$4+=121;OFS=seps[1]}1'
1001.2001.3001.5122.6001
Though resetting the argument list with the values is probably the preferred way, or by setting IFS to the delimiter and reading the values into an array and adding the desired value to the array index at issue, you can also do it with a simple loop to look for the delimiters and continually skipping characters until the desired segment is found (4 in you case -- when the delimiter count is 3). Then simply appending the digit at each array index until your next delimiter is found will give you the base value. Simply adding your desired 121 to the completed number completes the script, e.g.
#!/bin/bash
str=${1:-"1001.2001.3001.5001.6001"} ## string
ele=${2:-4} ## element to add value to [1, 2, 3, ...]
add=${3:-121} ## value to add to element
cnt=0 ## flag to track delimiters found
num=
## for each character in str
for ((i = 0; i < ${#str}; i++))
do
if [ "${str:$i:1}" = '.' -o "${str:$i:1}" = '-' ] ## is it '.' or '-'
then
(( cnt++ )) ## increment count
(( cnt == ele )) && break ## if equal to ele, break
## check each char is a valid digit 0-9
elif [ "0" -le "${str:$i:1}" -a "${str:$4i:1}" -le "9" ]
then
(( cnt == (ele - 1) )) || continue ## it not one of interest, continue
num="$num${str:$i:1}" ## append digit to num
fi
done
((num += add)) ## add the amount to num
printf "num: %d\n" $num ## print results
Example Use/Output
$ bash parsenum.sh
num: 5122
$ bash parsenum.sh "1001.2001.3001.5001.6001" 2
num: 2122
$ bash parsenum.sh "1001.2001.3001.5001.6001" 2 221
num: 2222
Look things over and let me know if you have any questions.

Tail recursion in Bash

I've tried to write a script to verify that all the stats of a metrics are positive before I make any further changes using the service. The part I'm stuck at is thinking over how to tail the recursion for the following use-case :
function load_cache() {
cacheStat=( $(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="cacheSize" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "cacheSize" | cut -d ':' -f 2) )
# the above gives me the ouput(cacheStat) as -
# 2.0
# 311.0
# 102.0
count=0
for index in ${!cacheStat[*]}
do
if [[ ${cacheStat[$index]} -le 0 ] && [ $count -lt 3 ]]; then
sleep .5
count=$[$count +1];
load_cache
#Wouldn't the above initialise `count` to 0 again.
fi
done
}
What I am trying to do is if any of the elements in the cacheStat is less than or equal to 0, then sleep for .5 secs and query the cacheStat again and perform the check on all its elements again. Though not do this more than 3 times for which I am trying to use `count.
Open to any suggestion to improve the script.
Update -
On modifying the scripts as suggested by #Inian to
RETRY_COUNT=0
function load_cache() {
cacheStat=( $(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="cacheSize" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "cacheSize" | cut -d ':' -f 2) );
for index in ${!cacheStat[*]}
do
echo "Stat - ${cacheStat[$index]}"
if (( ${cacheStat[$index]} <= 0 )) && (( $RETRY_COUNT < 3 )); then
echo "Attempt count - ${RETRY_COUNT}"
sleep .5s
RETRY_COUNT=$((RETRY_COUNT +1));
load_cache
fi
done
}
The logs read -
> > + cacheStat=($(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed
> 's/[{}]//g' | awk -v k="cacheSize"
> > '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed
> > 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w
> > "cacheSize" | cut -d ':' -f 2))
> > ++ curl -s -X GET http://localhost:8181/metrics
> > ++ sed 's/\\\\\//\//g'
> > ++ sed 's/[{}]//g'
> > ++ sed 's/[\,]/ /g'
> > ++ awk -v k=cacheSize '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}'
> > ++ sed 's/\"\:\"/\|/g'
> > ++ cut -d : -f 2
> > ++ sed 's/\"//g'
> > ++ grep -w cacheSize
It doesn't even iterate I guess.
Remove the infinite recursion by moving the count=0 outside the function body.
Also your script has couple of issues, a syntax violation and an outdated construct, lines 12-14 should have been,
if [[ ${cacheStat[$index]} -le 0 ]] && [[ $count -lt 3 ]]; then
sleep .5s
count=$((count +1));
load_cache
fi
or) use a more readable arithmetic operator, (()) in the if-clause as
if (( ${cacheStat[$index]} <= 0 )) && (( $count < 3 )); then
bash does not inherently support floating point arithmetic (comparison in your case), use a third party tool like bc, awk for this,
if (( $(echo "${cacheStat[$index]} <= 0" | bc -l) )) && (( $count < 3 )); then
You can avoid all that ad-hoc JSON parsing by using a JSON parser.
# Avoid using Bash-only "function" keyword
load_cache () {
local try
for try in 1 2 3; do
# Suction: jq doesn't return non-zero exit code for no match
# work around that by piping to grep .
if curl -s -X GET "http://localhost:${MET_PORT}/metrics" |
jq '.[] | select(cacheSize < 0)' |
grep .
then
# Notice also redirection to stderr for diagnostic messages
echo "$0: Attempt $try failed, sleeping before retrying" >&2
sleep 0.5
else
# Return with success, we are done, exit function
return 0
fi
done
# Return failure
return 1
}
I see no reason to prefer recursion over a straightforward for loop for controlling the number of retries.
If you never want to see the offending values, you can use grep -q in the conditional. I'm expecting you would do load_cache >/dev/null if you don't want the output.
If you want to see the non-offending values, the code will need some refactoring, but I'm focusing on getting the central job done elegantly and succinctly. Here's a sketch, mainly to show you the jq syntax for that.
load_cache () {
local try
local results
for try in 1 2 3; do
results=$(curl -s -X GET "http://localhost:${MET_PORT}/metrics" |
jq '.[] | .cacheSize' | tr '\n' ' ')
echo "$0: try $try: cacheSize $results" >&2
# Funky: massage the expression we test againt into a normalized form
# so that we know that the value will always be preceded by a space
case " $results " in
*" 0 "* | *" -"* )
case $try in
3) echo "$0: try $try failed; aborting" >&2 ;;
*) echo "$0: try $try failed; sleeping before retrying" >&2
sleep 0.5 ;;
esac;;
*) return 0
esac
done
return 1
}
The nested case to avoid sleeping on the final iteration isn't particularly elegant, but at least it should ensure that the reader is awake. /-8

Resources