Is there a way to create an associative array from a text file in bash? [duplicate] - bash

This question already has an answer here:
bash4 read file into associative array
(1 answer)
Closed 3 years ago.
I'm currently creating a list of commands so for example by saying "directory install plugin-name" I can install all needed plugins specified in an external list. This list is just a txt file with all plugin names. But I'm struggling getting all names in an associative array.
I've tried this one:
while IFS=";" read line;
do " communtyList[ $line ]=1 " ;
done < community-list.txt;
The desired output should be
communityList[test1]=1
communityList[test2]=1....
It need to be an associative array because I want to access it by words and not by index. This word will be implemented as parameters/arguments.
For example "install plugin" instead of "1 plugin"
So I can ask for example this way:
if [ ! -z "${!communtyList[$2]}" ];
Update, here the whole code:
#!/usr/bin/env bash
community(){
declare -A communtyList
while IFS= read line;
do communtyList[$line]=1 ;
done < community-list.txt;
# communtyList[test1]=1
# communtyList[test2]=1
# communtyList[test3]=1
# communtyList[test4]=1
if { [ $1 = 'install' ] || [ $1 = 'activate' ] || [ $1 = 'uninstall' ] || [ $1 = 'deactivate' ] ; } && [ ! -z $2 ] ; then
if [ $2 = 'all' ];
then echo "$1 all community plugins....";
while IFS= read -r line; do echo "$1 $line "; done < community-list.txt;
elif [ ! -z "${!communtyList[$2]}" ];
then echo "$1 community plugin '$2'....";
else
echo -e "\033[0;31m Something went wrong";
echo " Plugin '$2' does not exist.";
echo " Here a list of all available community plugins: ";
echo ${!communtyList[#]}
echo -e " \e[m"
fi
else
echo -e "\033[0;31m Something went wrong";
if [ -z $2 ];
then echo -e "[Plugin name] required. [community][action][plugin name] \e[m"
else
echo " Action '$1' does not exist.";
echo -e " Do you mean some of this? \n install \n activate \n uninstall \e[m"
fi
fi
echo ${!communtyList[#]}
}
"$#"

To use asociative array you have to declare it first
declare -A communityList
Then you can add values
communityList[test1]=1
communityList[test2]=2
...
Or with the declaration
declare -A communityList=(
communityList[test1]=1
communityList[test2]=2
...
)

The quotes around " communtyList[ $line ]=1 " mean you try to evaluate a command whose first character is a space. You want to take out those quotes, and probably put quotes around "$line" instead.
It's also unclear why you have IFS=";" -- you are not splitting the line into fields anyway, so this is not doing anything useful. Are there semicolons in your input file? Where and why; what do they mean?
You should probably prefer read -r unless you specifically require read to do odd things with backslashes in the input.
Finally, as suggested by Ivan, you have to declare the array's type as associative before you try to use it.
With those things out of the way, try
declare -A communityList
while read -r line; do
communtyList["$line"]=1
done < community-list.txt

Related

Bash script to download PDF using a CSV with name and url and auto-increment name

I'm trying to create a bash script that reads a CSV with two columns:
first column = name
second column = URL
and try to download a PDF file from the URL on the second column with a random name with letters and numbers .pdf and change the name using the first column.
The PDF name could be duplicate so if is duplicate I want to add numbers like:
Example %20 $5000.pdf
Example %20 $5000.1.pdf
Example %20 $5000.2.pdf
Because if I try to download wget and curl will not auto-increment with the output option.
I tried a lot of things but my limitations are taking too much time.
I created a counter that add the line number to the end, but if I got a larger PDF there will be unnecessary auto-increment numbers. (code below)
There should be a better method, but my lack of knowledge is taking too much time. So any help with that will be really appreciated, I'm a beginner on bash scripts.
Thanks for any help in advance!
CSV example:
Example %20 $5000,HTTP://example.com/djdiede.pdf
Example %20 $5000,HTTP://example.com/djdi42322ede.pdf
Example %30 $1000,HTTP://example.com/djd4234iede.pdf
Example %50 $1000,HTTP://example.com/dj43566diede.pdf
Code so far:
#!/bin/bash -e
COUNTER=1
while IFS=, read -r field1 field2
do
COUNTER=$[$COUNTER +1]
if [ "$field1" == "" ]
then
echo "Line $COUNTER field1 is empty or no value set"
elif [ "$field2" == "" ]
then
echo "Line $COUNTER field2 is empty or no value set"
else
pdf_file=$(echo $field1 | tr '/' ' ')
echo "================================================"
echo "Downloading $COUNTER $pdf_file..."
echo "================================================"
pdf_file_test="$pdf_file.pdf"
if [ -e "$pdf_file_test" ]; then
echo -e "\033[32m ^^^ File already exists!!! Adding line number at the end of the file: $pdf_file.$COUNTER.pdf \033[0m" >&2
wget -q -nc -O "$pdf_file."$COUNTER.pdf $field2
else
wget -q -nc -O "$pdf_file".pdf $field2
fi
fi
done < test.csv
This should help. I tried to stay close to your own coding style:
#!/bin/bash -e
LINECOUNTER=0
while IFS=, read -r field1 field2
do
LINECOUNTER=$[$LINECOUNTER +1]
if [ "$field1" == "" ]
then
echo "Line $LINECOUNTER: field1 is empty or no value set"
elif [ "$field2" == "" ]
then
echo "Line $LINECOUNTER: field2 is empty or no value set"
else
pdf_file=$(echo "$field1" | tr '/' ' ')
echo "================================================"
echo "Downloading $LINECOUNTER: $pdf_file..."
echo "================================================"
pdf_file_saveas="$pdf_file.pdf"
FILECOUNTER=0
while [ -e "$pdf_file_saveas" ]
do
FILECOUNTER=$[$FILECOUNTER +1]
pdf_file_saveas="$pdf_file.$FILECOUNTER.pdf"
done
if [ $FILECOUNTER -gt 0 ]
then
echo -e "\033[32m ^^^ File already exists!!! Adding number at the end of the file: $pdf_file_saveas \033[0m" >&2
fi
wget -q -nc -O "$pdf_file_saveas" "$field2"
fi
done < test.csv
Here's what I did:
use two counters: one for lines, one for files
when a file already exists, use file counter + loop to find the next 'empty slot' (i.e. file named <filename>.<counter-value>.pdf that does not exist)
fixed wrong line numbers (line counter needs to start at 0 instead of 1)
added double quotes where necessary/advisable
If you want to improve your script further, here are some suggestions:
instead of the big if ... elif ... else contruct, you can use if + continue, e.g. if [ "$field1" == "" ]; then continue; fi or even [ "$field1" == "" ] && continue
instead of terminating on error (#!/bin/bash -e), you could add error detection and handling after the wget call, e.g. if [ $? -ne 0 ]; then echo "failed to download ..."; fi

nested if statement not working using while read variable

I have a csv file that i am reading with a "while read" statement and i want to run an if statement on one of the fields in the csv.
====================================
csv file
client1,admin,password,5.9
client2,admin,password,5.8
====================================
this is my script
while read clientid user pass version
do
if [ '$version' = "5.9" ];
then
echo "IS"
else
echo "NOT"
fi
done < $1
The problem is that the if statement does not work.
It does not echo IS when the version is 5.9, it just keeps saying NOT, unless i change it to !=
I have tried using single and double quotes, even without... still doesn't work as expected.
The goal is to run commands until the end of the file.
Is this script correct for doing this?
Obviously the IS and NOT would be replaced by actual command, this is just for testing.
The sample csv file provided has trailing whitespace on the line, which can be removed from the version variable using parameter expansion.
This should work:
while IFS=, read -r clientid user pass version; do
if [ "${version//[[:space:]]/}" = "5.9" ]; then
echo "IS"
else
echo "NOT"
fi
done < $1
And here's another:
while IFS=$' \t\r\n' read -r line; do
IFS=, read -r clientid user pass version __ <<< "$line"
if [[ $version == '5.9' ]]; then
echo "IS"
else
echo "NOT"
fi
done < "$1"
Quote variables in the open always to prevent word splitting and pathname expansion.
Prefer [[ ]] over [ ]. It doesn't do word splitting and pathname expansion.
IFS=$' \t\r\n' trims out leading and trailing spaces.
__ is added to store surplus values just in case.
You can add the IFS value comma and whitespace IFS=', ' . You will get the exact result.
#!/bin/bash
IFS=', '
while read clientid user pass version
do
if [ "$version" == "5.9" ] ; then
echo "IS"
else
echo "NOT"
fi
done < $1

Variables from file

A text file has the following structure:
paa pee pii poo puu
baa bee bii boo buu
gaa gee gii goo guu
maa mee mii moo muu
Reading it line by line in a script is done with
while read LINE; do
ACTION
done < FILE
I'd need to get parameters 3 and 4 of each line into variables for ACTION. If this was manual input, $3 and $4 would do the trick. I assume awk is the tool, but I just can't wrap my head around the syntax. Halp?
read does this just fine. Pass it multiple variables and it will split on $IFS into that many fields.
while read -r one two three four five; do
action "$three" "$four"
done <file
I added the -r option because that is usually what you want. The default behavior is a legacy oddity of limited use.
Thanks tripleee. In the meantime I managed a suitably versatile solution:
#!/bin/sh
if [ ! $1 ]; then
echo "Which inputfile?"
exit
elif [ ! $2 -o ! $3 ]; then
echo "Two position parameters required"
exit
fi
if [ -f outfile ]; then
mv outfile outfile.old
fi
while read -a LINE; do
STRING="${LINE[#]}"
if [ ${LINE[$2-1]} == ${LINE[$3-1]} ]; then # remove comment for strings
# if [ ${LINE[$(($2-1))]} -eq ${LINE[$(($3-1))]} ]; then # remove comment for integers
echo $STRING >> outfile
fi
done < $1

Compare $1 with another string in bash

I've spent 2 hours with an if statement, that never works like I want:
#should return true
if [ "$1" == "355258054414904" ]; then
Here is the whole script:
#!/bin/bash
param=$1
INPUT=simu_900_imei_user_pass.csv
OLDIFS=$IFS
IFS=,
[ ! -f $INPUT ] && { echo "$INPUT ime not found"; exit 99; }
while read imei email pass
do
echo "First Parameter-IMEI: $1"
if [ "$1" == "355258054414904" ]; then
echo "GOOD"
fi
done < $INPUT
IFS=$OLDIFS
This is the output of the script:
First Parameter-IMEI: 355258054414904
First Parameter-IMEI: 355258054414904
First Parameter-IMEI: 355258054414904
I have seen a lot of pages about the subject, but I can't make it work :(
EDIT: I Join the content of csv for better understanding ! Tx for your help !
4790057be1803096,user1,pass1
355258054414904,juju,capp
4790057be1803096,user2,pass2
358854053154579,user3,pass3
The reason $1 does not match is because $1 means the first parameter given to the script on the command line, while you want it to match the first field read from the file. That value is in $imei.
You probably meant:
if [ "$imei" == "355258054414904" ]; then
echo "GOOD"
fi
Since it is inside the loop where you read input file line by line.
To check content of $1 use:
cat -vet <<< "$1"
UPDATE: To strip \r from $1 have this at top:
param=$(tr -d '\r' <<< "$1")
And then use "$param" in rest of your script.
To test string equality with [ you want to use a single '=' sign.

Reading a config file from a shell script

I am looking for a shell script analog to something like Pythons's ConfigParser or Perl's Config::INI. I have sourced files in the past to accomplish this, but I'd prefer to read rather than execute my "config file". Does anyone know of anything comparable to the above modules available for shell (or bash) scripts?
Thanks,
Jerry
You don't want source it, so you should:
1.read the config, 2.verify lines 3.eval them
CONFIGFILE="/path/to/config"
echo "=$ADMIN= =$TODO= =$FILE=" #these variables are not defined here
eval $(sed '/:/!d;/^ *#/d;s/:/ /;' < "$CONFIGFILE" | while read -r key val
do
#verify here
#...
str="$key='$val'"
echo "$str"
done)
echo =$ADMIN= =$TODO= =$FILE= #here are defined
sample of config file
ADMIN: root
TODO: delete
var=badly_formtatted_line_without_colon
#comment
FILE: /path/to/file
if you run the above sample should get (not tested):
== == ==
=root= =delete= =/path/to/file=
sure this is not the best solution - maybe someone post a nicer one.
You might want to take a look at cfget which can be installed with sudo apt-get install cfget.
#!/bin/bash
# Author: CJ
# Date..: 01/03/2013
## sample INI file save below to a file, replace "^I" with tab
#^I [ SECTION ONE ]
#TOKEN_TWO^I ="Value1 two "
#TOKEN_ONE=Value1 One
#TOKEN_THREE=^I"Value1^I three" # a comment string
#TOKEN_FOUR^I=^I"^IValue1 four"
#
#[SECTION_TWO]
#TOKEN_ONE=Value1 One ^I^I^I# another comment string
#TOKEN_TWO^I ="Value1 two "
#TOKEN_THREE=^I"Value1^I three"
#TOKEN_FOUR^I=^I"^IValue1 four"
## sample INI file
export INI= # allows access to the parsed INI values in toto by children
iniParse() {
# Make word separator Linefeed(\n)
OIFS="${IFS}"
IFS=$(echo)
SECTION=_
while read LINE; do {
IFS="${OIFS}"
# Skip blank lines
TMP="$(echo "${LINE}"|sed -e "s/^[ \t]*//")"
if [ 0 -ne ${#TMP} ]; then
# Ignore comment lines
if [ '#' == "${LINE:0:1}" -o '*' == "${LINE:0:1}" ]; then
continue
fi # if [ '#' == "${LINE:0:1}" -o '*' == "${LINE:0:1}" ]; then
# Section label
if [ "[" == "${LINE:0:1}" ]; then
LINE="${LINE/[/}"
LINE="${LINE/]/}"
LINE="${LINE/ /_}"
SECTION=$(echo "${LINE}")_
else
LINE="$(echo "${LINE}"|sed -e "s/^[ \t]*//")"
LINE="$(echo "${LINE}"|cut -d# -f1)"
TOKEN="$(echo "${LINE:0}"|cut -d= -f1)"
EQOFS=${#TOKEN}
TOKEN="$(echo "${TOKEN}"|sed -e "s/[ \t]*//g")"
VALUE="${LINE:${EQOFS}}"
VALUE="$(echo "${VALUE}"|sed -e "s/^[ \t=]*//")"
VALUE="$(echo "${VALUE}"|sed -e "s/[ \t]*$//")"
if [ "${VALUE:0:1}" == '"' ]; then
echo -n "${SECTION}${TOKEN}=${VALUE}"
echo -e "\r"
else
echo -n "${SECTION}${TOKEN}="\"${VALUE}\"""
echo -e "\r"
fi # if [ "${VALUE:0:1}" == '"' ]; then
fi # if [ "[" == "${LINE:0:1}" ]; then
fi # if [ 0 -ne ${#TMP} ]; then
IFS=$(echo)
} done <<< "$1"
IFS="${OIFS}" # restore original IFS value
} # iniParse()
# call this function with the INI filespec
iniReader() {
if [ -z "$1" ]; then return 1; fi
TMPINI="$(<$1)"
TMPINI="$(echo "${TMPINI}"|sed -e "s/\r//g")"
TMPINI="$(echo "${TMPINI}"|sed -e "s/[ \t]*\[[ \t]*/[/g")"
TMPINI="$(echo "${TMPINI}"|sed -e "s/[ \t]*\][ \t]*/]/g")"
INI=`iniParse "${TMPINI}"`
INI="$(echo "${INI}"|sed -e "s/\r/\n/g")"
eval "${INI}"
return 0
} # iniReader() {
# sample usage
if iniReader $1 ; then
echo INI read, exit_code $? # exit_code == 0
cat <<< "${INI}"
cat <<< "${SECTION_ONE_TOKEN_FOUR}"
cat <<< "${SECTION_ONE_TOKEN_THREE}"
cat <<< "${SECTION_TWO_TOKEN_TWO}"
cat <<< "${SECTION_TWO_TOKEN_ONE}"
else
echo usage: $0 filename.ini
fi # if iniReader $1 ; then
grep based alternative seems to be more readable:
CONFIG_FILE='/your/config/file.ini'
eval $(grep '^\[\|^#' CONFIG_FILE -v | while read line
do echo $line
done)
Where:
-v grep option means exclude matching lines
^\[\|^# selects all lines which starts with [ or # (configparser sections and comments)
It will work ONLY if your config file doesn't have spaces around = (if you would like to generate config with Python use space_around_delimiters=False see https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.write)
Supported config example:
FIRST_VAR="a"
[lines started with [ will be ignored]
secondvar="b"
# some comment
anotherVar="c"
You can use bash it-self to interpret ini values, by:
$ source <(grep = file.ini)
Sample file:
[section-a]
var1=value1
var2=value2
See more examples: How do I grab an INI value within a shell script?
Or you can use bash ini-parser which can be found at The Old School DevOps blog site.

Resources