My shell script look like this
i="10 ID:794 A:TX-SPN S:0"
A=`echo $i | cut -d" " -f 3| cut -d":" -f2` # gives TX-SPN
ID=`echo $i | cut -d" " -f 2|cut -d":" -f2` # gives 794
sZeroCount=`echo $i | cut -d" " -f 1` # gives 10
by above commands,I am able to get the values for A,ID,sZeroCount variables, since the value for i contains only one entry, value of i not limited to 1 it may go upto 1000. Is there any better approach in which I can obtain those values.
With an array. Split string i with separator space and : to array a:
i="10 ID:794 A:TX-SPN S:0"
IFS=" :" a=($i)
echo "${a[4]}" # TX-SPN
echo "${a[2]}" # 794
echo "${a[0]}" # 10
With chepner's bugfix:
i="10 ID:794 A:TX-SPN S:0"
IFS=": " read -a a <<< "$i"
echo "${a[4]}" # TX-SPN
echo "${a[2]}" # 794
echo "${a[0]}" # 10
With this piece of code you can convert your line into a proper associative array:
declare -A dict
for token in START:$i # choose a value for START that is not a key
do
IFS=: read key value <<< "$token"
dict["$key"]=$value
done
You can dump the result using declare -p dict:
declare -A dict='([A]="TX-SPN" [S]="0" [ID]="794" [START]="10" )'
And you can access the contents e. g. using this: echo "${dict[A]}"
TX-SPN
The start value (the 10 in your example) can be accessed as "${dict[START]}". Choose a value for START that doesn't appear as key in your input.
If you want to iterate over a lot of lines like your $i, you can do it like this:
while read i
do
declare -A dict
# ... add code from above ...
done < input_file
The advantage of using associative arrays is that this way you can access your values in a much more understandable way, i. e. by using the keys instead of some arbitrary indexes which can easily be mixed up and which need constant maintenance when changing your code.
Related
I want to read into bash associative array the content of one yaml file, which is a simple key value mapping.
Example map.yaml
---
a: "2"
b: "3"
api_key: "somekey:thatcancontainany#chara$$ter"
the key can contain any characters excluding space
the value can contain any characters without limitations $!:=#etc
What will always be constant is the separator between key and value is :
the script proceses.sh
#!/usr/bin/env bash
declare -A map
# how to read here into map variable, from map.yml file
#map=populatesomehowfrommap.yaml
for key in "${!map[#]}"
do
echo "key : $key"
echo "value: ${map[$key]}"
done
I tried to play around with yq tool, similar to json tool jq but did not have success yet.
With the following limitations:
simple YAML key: "value" in single lines
keys cannot contain :
values are always wrapped in "
#!/usr/bin/env bash
declare -A map
regex='^([^:]+):[[:space:]]+"(.*)"[[:space:]]*$'
while IFS='' read -r line
do
if [[ $line =~ $regex ]]
then
printf -v map["${BASH_REMATCH[1]}"] '%b' "${BASH_REMATCH[2]}"
else
echo "skipping: $line" 1>&2
fi
done < map.yaml
Update
Here's a robust solution using yq, which would be simpler if the builtin #tsv filter implemented the lossless TSV escaping rules instead of the CSV ones.
#!/usr/bin/env bash
declare -A map
while IFS=$'\t' read key value
do
printf -v map["$key"] '%b' "$value"
done < <(
yq e '
to_entries | .[] |
[
(.key | sub("\\","\\") | sub("\n","\n") | sub("\r","\r") | sub("\t","\t")),
(.value | sub("\\","\\") | sub("\n","\n") | sub("\r","\r") | sub("\t","\t"))
] |
join(" ")
' map.yaml
)
note: the join needs a literal Tab
One way, is by letting yq output each key/value pair on a single line, in the following syntax:
key#value
Then we can use bash's IFS to split those values.
The # is just an example and can be replaced with any single char
This works, but please note the following limitations:
It does not expect nested values, only a flat list`
The field seperator (# in the example) does not exist in the YAML key/value's
#!/bin/bash
declare -A arr
while IFS="#" read -r key value
do
arr[$key]="$value"
done < <(yq e 'to_entries | .[] | (.key + "#" + .value)' input.yaml)
for key in "${!arr[#]}"
do
echo "key : $key"
echo "value: ${arr[$key]}"
done
$ cat input.yaml
---
a: "bar"
b: "foo"
$
$
$ ./script.sh
key : a
value: bar
key : b
value: foo
$
I used #Fravadona s answer so will mark it as answer
After some modification to my use case, what worked for me looks like:
DEFS_PATH="definitions"
declare -A ssmMap
for file in ${DEFS_PATH}/*
do
filename=$(basename -- "$file")
projectName="${filename%.*}"
regex='^([^:]+):[[:space:]]*"(.*)"[[:space:]]*$'
while IFS='' read -r line
do
if [[ $line =~ $regex ]]
then
value="${BASH_REMATCH[2]}"
value=${value//"{{ ssm_env }}"/$INFRA_ENV}
value=${value//"{{ ssm_reg }}"/$SSM_REGION}
value=${value//"{{ projectName }}"/$projectName}
printf -v ssmMap["${BASH_REMATCH[1]}"] '%b' "$value"
else
echo "skipping: $line" 1>&2
fi
done < "$file"
done
Basically in real use case I have one folder where yaml definitions are located. I iterate over all of them to form the associative array ssmMap
What's the most efficient way to convert a factor vector (not all levels are unique) into a numeric vector in bash? The values in the numeric vector do not matter as long as each represents a unique level of the factor.
To illustrate, this would be the R equivalent to what I want to do in bash:
numeric<-seq_along(levels(factor))[factor]
I.e.:
factor
AV1019A
ABG1787
AV1019A
B77hhA
B77hhA
numeric
1
2
1
3
3
Many thanks.
It is most probably not the most efficient, but maybe something to start.
#!/bin/bash
input_data=$( mktemp )
map_file=$( mktemp )
# your example written to a file
echo -e "AV1019A\nABG1787\nAV1019A\nB77hhA\nB77hhA" >> $input_data
# create a map <numeric, factor> and write to file
idx=0
for factor in $( cat $input_data | sort -u )
do
echo $idx $factor
let idx=$idx+1
done > $map_file
# go through your file again and replace values with keys
while read line
do
key=$( cat $map_file | grep -e ".* ${line}$" | awk '{print $1}' )
echo $key
done < $input_data
# cleanup
rm -f $input_data $map_file
I initially wanted to use associative arrays, but it's a bash 4+ feature only and not available here and there. If you have bash 4 then you have one file less, which is obviously more efficient.
#!/bin/bash
# your example written to a file
input_data=$( mktemp )
echo -e "AV1019A\nABG1787\nAV1019A\nB77hhA\nB77hhA" >> $input_data
# declare an array
declare -a factor_map=($( cat $input_data | sort -u | tr "\n" " " ))
# go through your file replace values with keys
while read line
do
echo ${factor_map[#]/$line//} | cut -d/ -f1 | wc -w | tr -d ' '
done < $input_data
# cleanup
rm -f $input_data
This is probably pretty basic, I want to read in a occurrence file.
Then the program should find all occurrences of "CallTilEdb" in the file Hendelse.logg:
CallTilEdb 8
CallCustomer 9
CallTilEdb 4
CustomerChk 10
CustomerChk 15
CallTilEdb 16
and sum up then right column. For this case it would be 8 + 4 + 16, so the output I would want would be 28.
I'm not sure how to do this, and this is as far as I have gotten with vistid.sh:
#!/bin/bash
declare -t filename=hendelse.logg
declare -t occurance="$1"
declare -i sumTime=0
while read -r line
do
if [ "$occurance" = $(cut -f1 line) ] #line 10
then
sumTime+=$(cut -f2 line)
fi
done < "$filename"
so the execution in terminal would be
vistid.sh CallTilEdb
but the error I get now is:
/home/user/bin/vistid.sh: line 10: [: unary operator expected
You have a nice approach, but maybe you could use awk to do the same thing... quite faster!
$ awk -v par="CallTilEdb" '$1==par {sum+=$2} END {print sum+0}' hendelse.logg
28
It may look a bit weird if you haven't used awk so far, but here is what it does:
-v par="CallTilEdb" provide an argument to awk, so that we can use par as a variable in the script. You could also do -v par="$1" if you want to use a variable provided to the script as parameter.
$1==par {sum+=$2} this means: if the first field is the same as the content of the variable par, then add the second column's value into the counter sum.
END {print sum+0} this means: once you are done from processing the file, print the content of sum. The +0 makes awk print 0 in case sum was not set... that is, if nothing was found.
In case you really want to make it with bash, you can use read with two parameters, so that you don't have to make use of cut to handle the values, together with some arithmetic operations to sum the values:
#!/bin/bash
declare -t filename=hendelse.logg
declare -t occurance="$1"
declare -i sumTime=0
while read -r name value # read both values with -r for safety
do
if [ "$occurance" == "$name" ]; then # string comparison
((sumTime+=$value)) # sum
fi
done < "$filename"
echo "sum: $sumTime"
So that it works like this:
$ ./vistid.sh CallTilEdb
sum: 28
$ ./vistid.sh CustomerChk
sum: 25
first of all you need to change the way you call cut:
$( echo $line | cut -f1 )
in line 10 you miss the evaluation:
if [ "$occurance" = $( echo $line | cut -f1 ) ]
you can then sum by doing:
sumTime=$[ $sumTime + $( echo $line | cut -f2 ) ]
But you can also use a different approach and put the line values in an array, the final script will look like:
#!/bin/bash
declare -t filename=prova
declare -t occurance="$1"
declare -i sumTime=0
while read -a line
do
if [ "$occurance" = ${line[0]} ]
then
sumTime=$[ $sumtime + ${line[1]} ]
fi
done < "$filename"
echo $sumTime
For the reference,
id="CallTilEdb"
file="Hendelse.logg"
sum=$(echo "0 $(sed -n "s/^$id[^0-9]*\([0-9]*\)/\1 +/p" < "$file") p" | dc)
echo SUM: $sum
prints
SUM: 28
the sed extract numbers from a lines containing the given id, such CallTilEdb
and prints them in the format number +
the echo prepares a string such 0 8 + 16 + 4 + p what is calculation in RPN format
the dc do the calculation
another variant:
sum=$(sed -n "s/^$id[^0-9]*\([0-9]*\)/\1/p" < "$file" | paste -sd+ - | bc)
#or
sum=$(grep -oP "^$id\D*\K\d+" < "$file" | paste -sd+ - | bc)
the sed (or the grep) extracts and prints only the numbers
the paste make a string like number + number + number (-d+ is a delimiter)
the bc do the calculation
or perl
sum=$(perl -slanE '$s+=$F[1] if /^$id/}{say $s' -- -id="$id" "$file")
sum=$(ID="CallTilEdb" perl -lanE '$s+=$F[1] if /^$ENV{ID}/}{say $s' "$file")
Awk translation to script:
#!/bin/bash
declare -t filename=hendelse.logg
declare -t occurance="$1"
declare -i sumTime=0
sumtime=$(awk -v entry=$occurance '
$1==entry{time+=$NF+0}
END{print time+0}' $filename)
I have a shell script and a common configuration file where all the generic path, username and other values are stored. I want to get the value from this configuration file while I am running the sh script.
example:
sample.conf
pt_user_name=>xxxx
pt_passwd=>Junly#2014
jrnl_source_folder=>x/y/v
pt_source_folder=>/x/y/r/g
css_source_folder=>/home/d/g/h
Now i want get some thing like this in my sh script.
cd $css_source_folder
this command inside the shell script should take me to the location d/g/h while the script is running.
Is there any way to achieve this other than with grep and awk??
Thanks
Rinu
If you want to read from conf file everytime then grep and cut might help you,
suppose you need value for css_source_folder property
prop1="css_source_folder" (I am assuming you know property name whose value you want)
value_of_prop1=`grep $prop1 sample.conf| cut -f2 -d "=" | cut -f2 -d ">"`
like,
[db2inst2#pegdb2 ~]$ vi con.conf
[db2inst2#pegdb2 ~]$ grep css_source_folder con.conf
css_source_folder=>/home/d/g/h
[db2inst2#pegdb2 ~]$ value=`grep css_source_folder con.conf | cut -f2 -d "="`
[db2inst2#pegdb2 ~]$ echo $value
>/home/d/g/h
[db2inst2#pegdb2 ~]$ value=`grep css_source_folder con.conf | cut -f2 -d "=" | cut -f2 -d ">"`
[db2inst2#pegdb2 ~]$ echo $value
/home/d/g/h
If you want to read all properties at once, then apply loop and this will solve the purpose
Yes, you can get the configuration names and values relatively simple and associate them through array indexes. Reading your config can be done like this:
#!/bin/bash
test -r "$1" || { echo "error: unable to read conf file [$1]\n"; exit 1; }
declare -a tag
declare -a data
let index=0
while read line || test -n "$line"; do
tag[index]="${line%%\=*}"
data[index]="${line##*\>}"
((index++))
done < "$1"
for ((i=0; i<${#tag[#]}; i++)); do
printf " %18s %s\n" "${tag[$i]}" "${data[$i]}"
done
After reading the config file, you then have the config name tags and config values stored in the arrays tag and value, respectively:
pt_user_name xxxx
pt_passwd Junly#2014
jrnl_source_folder x/y/v
pt_source_folder /x/y/r/g
css_source_folder /home/d/g/h
At that point, it is a matter of determining how you will use them, whether as a password or as a directory. You may have to write a couple of functions, but the basic function of given a tag, get the correct data can be done like this:
function getvalue {
test -n "$1" || { echo "error in getvalue, no data supplied"; return 1; }
for ((i=0; i<${#tag[#]}; i++)); do
if test "$1" = "${tag[$i]}"; then
echo " eval cmd ${data[$i]}"
return $i
fi
done
return -1
}
echo -e "\nget value for 'jrnl_source_folder'\n"
getvalue "jrnl_source_folder"
The function will return the index of the data value and can execute any command needed. You seem to have directory paths and passwords, so you may need a function for each. To illustrate, the output of the example is:
get value for jrnl_source_folder
eval cmd x/y/v
You can also use an associative array in later versions of BASH to store the tag and data in a single associative array. You may also be able to use indirect references on the tag and data values to process them. I simply took the straight forward approach in the example.
Try this eval $(awk -F'=>' '{print $1"=\""$2"\";"}' sample.conf):
EX:
eval $(awk -F'=>' '{print $1"=\""$2"\";"}' sample.conf); echo $pt_user_name
xxxx
Using sed :
eval $(sed -re 's/=>/="/g' -e 's/$/";/g' sample.conf); echo $pt_passwd
Junly#2014
Using perl :
eval $(perl -F'=>' -alne 'print "$F[0]=\"$F[1]\";"' sample.conf); echo $pt_source_folder
/x/y/r/g
Using tr :
eval $(tr -d '>' <sample.conf); echo "$css_source_folder"
/home/d/g/h
PS. Using tr blindly to remove > may cause undesirable results depending on the content of sample.conf, but for the one provided works fine.
I'm developing a little script using ash shell (not bash).
Now i have a variable with the following composition:
VARIABLE = "number string status"
where number could be any number (actually between 1 and 18 but in the future that number could be higher) the string is a name and status is or on or off
The name usually is only lowercase letter.
Now my problem is to read only the string content in the variable, removing the number and the status.
How i can obtain that?
Two ways; one is to leverage $IFS and use a while loop - this will work for a single line quite happily - as:
echo "Part1 Part2 Part3" | while read a b c
do
echo $a
done
alternatively, use cut as follows:
a=`echo $var | cut -d' ' -f2`
echo $a
How about using cut?
name=$(echo "$variable" | cut -d " " -f 2)
UPDATE
Apparently, Ash doesn't understand $(...). Hopefully you can do this instead:
name=`echo "$variable" | cut -d " " -f 2`
How about :
name=$(echo "$variable" | awk '{print $2}')
#!/bin/sh
myvar="word1 word2 word3 wordX"
set -- $myvar
echo ${15} # outputs word 15