bash associative array value empty using variable key, based on lookup file - bash

I need to lookup data from a lookup file. I am using bash ver 4.1.2 on rhel.
Lookup data (ip_lookup.txt)
1.1.1.1=server A
2.2.2.2=server B
.. and so on
I iterate list from a file (aggregated from access.log) and need to add information from the lookup file.
iterated list (service.txt)
1234 service_a
324 service_b
34 service_c
access.log with x_http_forwarder info
service_a 200 1.1.1.1
service_a 200 1.1.1.1
service_a 200 2.2.2.2
service_b 200 1.1.1.1
service_c 200 2.2.2.2
So the code I made was this:
# import lookup file into bash array
declare -A ary
while IFS== read -r key value; do
ary[$key]=$value
done < "ip_lookup.txt"
# then I nested iterate from list service and access.log
# list service and its caller
while IFS=$'\t' read -r -a line
do
echo -e "${line[0]}\t${line[1]}"
cat "access.log" |
tr -d \" |
sed "s/?wsdl.$PARTITION_COLUMN.\t/\t/g" |
sed '/^[[:space:]]*$/d' |
sed "s/?WSDL.$PARTITION_COLUMN.\t/\t/g" |
grep ${line[1]} |
awk '{ seen[$3] += $1 } END { for (i in seen) print seen[i],i }' |
sort -nr |
sed 's/ /\t/g' |
head -n 2 > "ip_list.txt"
while IFS=$'\t' read -r -a line
do
# the problem lie in this block <<<<<<<<<<<<<<<<<<<
echo -e "> ${line[1]} \t\t ${line[0]} \t\t ${ary[${line[1]}]}"
# it only returns var line 1 and line 0. While lookup var ${ary[${line[1]}] is empty
# fyi, ${line[1]} returns 1.1.1.1 or 2.2.2.2 (on each iteration)
fi
done < "ip_list.txt"
done < "service.txt"
Let me highlight the problem:
ip=${line[1]}
echo $ip #it returns IP such as 1.1.1.1
echo ${ary['1.1.1.1']} # it returns server A
# but if I call the key using variable, it won't work. I've tried some combination but it works nothing.
echo ${ary[$ip]} # it returns nothing
echo ${ary[${line[1]}]} # it returns nothing
echo ${ary[${!line[1]}]} # it returns nothing
Any idea how to return the value using the key variable in bash associative array?
feel free to ask the detail.

Related

writing the same result for the duplicated values of a column

I'm really new to bash. I have a list of domains in a .txt file (URLs.txt). I also want to have a .csv file which consists of 3 columns separated by , (myFile.csv). My code reads each line of URLs.txt (each domain), finds its IP address and then inserts them into myFile.csv (domain in the first column, its IP in the 2nd column.
Name, IP
ex1.com, 10.20.30.40
ex2.com, 20.30.40.30
ex3.com, 10.45.60.20
ex4.com, 10.20.30.40
Here is my code:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
echo "$line,$ipValue" >> myFile.csv
fi
done < URLs.txt
I want to add another column to myFile.csv for keeping open ports of each IP. So output would be like this:
Name, IP, Port
ex1.com, 10.20.30.40, 21/tcp
ex2.com, 20.30.40.30, 20/tcp
ex3.com, 10.45.60.20, 33/tcp
ex4.com, 10.20.30.40, 21/tcp
I want to use Nmap to do this. After I choose an IP address from the 2nd column of myFile.csv and find its open ports using Nmap, I want to write the Nmap result to the corresponding cell of the 3rd column.
Also, if there is another similar IP in the 2nd column I want to write the Nmap result for that line too. I mean I don't want to run Nmap again for the duplicated IP. For example, in my example, there are two "10.20.30.40" in the 2nd column. I want to use Nmap just once and for the 1st "10.20.30.40" (and write the result for the 2nd "10.20.30.40" as well, Nmap should not be run for the duplicated IP).
For this to happen, I changed the first line of my code to this:
echo "Name,IP,Port" > myFile.csv
and also here is the Nmap code to find the open ports:
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
but I don't know what to do next and how to apply these changes to my code.
I updated my code to something like this:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
echo "$line,$ipValue,$port" >> myFile.csv
fi
done < URLs.txt
but this way, Nmap was used for finding the open ports of the duplicated IPs too, but I didn't want this. What should I do?
Here's a modified version of your script that roughly does what you want:
#!/usr/bin/env bash
# cache maps from IP addresses to open ports
declare -A cache
getports() {
local ip=$1
nmap -v -Pn -p 1-100 "$ip" -oG - \
| awk -F '\t' '
/Ports:/ {
n = split($2, a, /,? /)
printf "%s", a[2]
for (i = 3; i <= n; ++i)
printf ":%s", a[i]
}
'
}
{
echo 'Name,IP,Port'
while IFS= read -r url; do
# Read filtered dig output into array
readarray -t ips < <(dig +short "$url" | grep -E '^([0-9]+\.){3}[0-9]+$')
# Build array of open ports
unset ports
for ip in "${ips[#]}"; do
ports+=("${cache["$ip"]:=$(getports "$ip")}")
done
# Output
printf '%s,%s,%s\n' \
"$url" \
"$(IFS='-'; echo "${ips[*]}")" \
"$(IFS='-'; echo "${ports[*]}")"
done < URLs.txt
} > myFile.csv
The readarray line reads the filtered output from dig into an array of IP addresses; if that array has length zero, the rest of the loop is skipped.
Then, for each elements in the ips array, we get the ports. To avoid calling nmap if we've seen the IP address before, we use the ${parameter:=word} parameter expansion: if ${cache["$ip"]} is non-empty, use it, otherwise call the getports function and store the output in the cache associative array.
getports is called for IP addresses we haven't seen before; I've used -oG ("grepable output") to make parsing easier. The awk command filters for lines containing Ports:, which look something like
Host: 52.94.225.242 () Ports: 80/open/tcp//http/// Ignored State: closed (99)
with tab separated fields. We then split the second field on the regular expression /,? / (an optional comma followed by a blank) and store all but the first field of the resulting array, colon separated.
Finally, we print the line of CSV data; if ips or ports contain more than one element, we want to join the elements with -, which is achieved by setting IFS in the command substitution and then printing the arrays with [*].
The initial echo and the loop are grouped within curly braces so output redirection has to happen just once.

Bash script to read from a file and save the information in an array?

I want to read from a file that has host IPs written in it and save it in an array. So far I have tried this:
Host=`cat /home/hp3385/Desktop/config | egrep '^Host' | awk '{print $2}'`
But I don't think that it saves the information in an array. What is the type of the variable 'Host'? If it's not an array how can I convert it into one?
This is a sample data from the file /home/hp3385/Desktop/config:
############# Server1 #################
Host 8.8.8.8
Hostname google
############# Server2 ################
Host 8.8.4.4
Hostname google
The expected output is:
a=($'8.8.8.8' $'8.8.4.4')
You can try this
myarray=()
while read -r line; do
if echo "$line" | grep -q 'Host '; then
myarray+=($(echo "$line" | awk '/^Host/ {print $2}'))
fi
done < /home/hp3385/Desktop/config
Declaring an array:
ARRAY=(0 1 2 3 4 5)
So your array can be declared like this:
HOSTS=($(awk '/^Host/ {print $2}' YOUR_FILE))
If you want to know the amount of values in your array:
echo ${#HOSTS[*]}
To get an output of all values in your array (credit goes to triplee):
printf '%s\n' "${HOSTS[#]}"

awk printing random "e" that is not in string

I am wanting to extract ip and port from a string.
Strings look like this.
destination x.x.x.x:yyyy
where x is ip and y is port
commandout=()
while IFS= read -r line # Read a line
do
commandout+=("$line") # Append line to the array
done < <(tmsh list ltm virtual $vip | grep destination)
for output in "$commandout";
do
if [[ $output == *"destination"* ]];then
#split off ip and port
ipport=$(echo $output | awk 'BEGIN{}{print $2}')
echo $ipport | awk 'BEGIN{FS=":"}{print $1}'
echo $ipport
fi
done
declare -p commandout
for some reason, awk is printing a random "e" after the ip address. But it only appears to do so after 2.
10.10.10.10
10.10.10.10:https
declare -a commandout='([0]=" destination 10.10.10.10:https")'
12.12.12.12e
12.12.12.12:https
declare -a commandout='([0]=" destination 12.12.12.12:https")'
UPDATE:
So I attempted another test. I found strange behavior and I am unsure how to fix it.
I declare the vipip before and after it is set.
declare -p vipip
vipip=$(tmsh list ltm virtual $vip | grep destination | awk 'BEGIN{}{print $2}' | awk 'BEGIN{FS=":"}{print $1}')
echo $vipip
declare -p vipip
echo "cyle loop"
results in the following. Note that the 12.12.12.12 doesn't have an "e" on the end of it
./findvips-final.scr: line 240: declare: vipip: not found
10.10.10.10
declare -- vipip="10.10.10.10"
cyle loop
declare -- vipip="10.10.10.10"
12.12.12.12
declare -- vipip="12.12.12.12"
cyle loop
If I comment out the declare statements, I get an "e"
#declare -p vipip
vipip=$(tmsh list ltm virtual $vip | grep destination | awk 'BEGIN{}{print $2}' | awk 'BEGIN{FS=":"}{print $1}')
echo $vipip
#declare -p vipip
echo "cyle loop"
results in
10.10.10.10
cyle loop
12.12.12.12e
cyle loop
I found the answer. I have a progress meter above this and I was getting the e off of complete.
echo -ne "$((100*$z/$count))% Complete\r"
I wrapped $vipip in qoutes on the echo and it is working like I thought. UGh wait a big waste of time.
You can straightaway set FS like below to extract Ip from your command, no need of loop, awk can search string also
your_command | awk -F'[ :]' '/destination/{gsub(/[^0-9.]/,"",$2); print $2}'
Explanation
-F'[ :]' - set field separator
'/destination/ - search for word destination in line/record/row
gsub(/[^0-9.]/,"",$2) - remove anything other than number and dot from second field ( so that random char like e, what you said above will be removed )
print $2 - print second field

Shell Script : Assign the outputs to different variables

In a shell script I need to assign the output of few values to different varialbes, need help please.
cat file1.txt
uid: user1
cn: User One
employeenumber: 1234567
absJobAction: HIRED
I need to assign the value of each attribute to different variables so that I can call them them in script. For example uid should be assigned to a new variable name current_uid and when $current_uid is called it should give user1 and so forth for all other attributes.
And if the output does not contain any of the attributes then that attribute value should be considered as "NULL". Example if the output does not have absJobAction then the value of $absJobAction should be "NULL"
This is what I did with my array
#!/bin/bash
IFS=$'\n'
array=($(cat /tmp/file1.txt | egrep -i '^uid:|^cn:|^employeenumber|^absJobAction'))
current_uid=`echo ${array[0]} | grep -w uid | awk -F ': ' '{print $2}'`
current_cn=`echo ${array[1]} | grep -w cn | awk -F ': ' '{print $2}'`
current_employeenumber=`echo ${array[2]} | grep -w employeenumber | awk -F ': ' '{print $2}'`
current_absJobAction=`echo ${array[3]} | grep -w absJobAction | awk -F ': ' '{print $2}'`
echo $current_uid
echo $current_cn
echo $current_employeenumber
echo $current_absJobAction
Output from sh /tmp/testscript.sh follows:
user1
User One
1234567
HIRED
#!/usr/bin/env bash
# assuming bash 4.0 or newer: create an associative array
declare -A vars=( )
while IFS= read -r line; do ## See http://mywiki.wooledge.org/BashFAQ/001
if [[ $line = *": "* ]]; then ## skip lines not containing ": "
key=${line%%": "*} ## strip everything after ": " for key
value=${line#*": "} ## strip everything before ": " for value
vars[$key]=$value
else
printf 'Skipping unrecognized line: <%s>\n' "$line" >&2
fi
done <file1.txt # or < <(ldapsearch ...)
# print all variables read, just to demonstrate
declare -p vars >&2
# extract and print a single variable by name
echo "Variable uid has value ${vars[uid]}"
Note that this must be run with bash yourscript, not sh yourscript.
By the way -- if you don't have bash 4.0, you might consider a different approach:
while IFS= read -r line; do
if [[ $line = *": "* ]]; then
key=${line%%": "*}
value=${line#*": "}
printf -v "ldap_$key" %s "$value"
fi
done <file1.txt # or < <(ldapsearch ...)
will create separate variables of the form "$ldap_cn" or "$ldap_uid", as opposed to putting everything in a single associative array.
Here's a simple example of what you are trying to do that should get you started. It assumes 1 set of data in the file. Although a tad brute-force, I believe its easy to understand.
Given a file called file.txt in the current directory with the following contents (absJobAction intentionally left out):
$ cat file1.txt
uid: user1
cn: User One
employeenumber: 1234567
$
This script gets each value into a local variable and prints it out:
# Use /bin/bash to run this script
#!/bin/bash
# Make SOURCEFILE a readonly variable. Make it uppercase to show its a constant. This is the file the LDAP values come from.
typeset -r SOURCEFILE=./file1.txt
# Each line sets a variable using awk.
# -F is the field delimiter. It's a colon and a space.
# Next is the value to look for. ^ matches the start of the line.
# When the above is found, return the second field ($2)
current_uid="$(awk -F': ' '/^uid/ {print $2}' ${SOURCEFILE})"
current_cn="$(awk -F': ' '/^cn/ {print $2}' ${SOURCEFILE})"
current_enbr="$(awk -F': ' '/^employeenumber/ {print $2}' ${SOURCEFILE})"
current_absja="$(awk -F': ' '/^absJobAction/ {print $2}' ${SOURCEFILE})"
# Print the contents of the variables. Note since absJobAction was not in the file,
# it's value is NULL.
echo "uid: ${current_uid}"
echo "cn: ${current_cn}"
echo "EmployeeNumber: ${current_enbr}"
echo "absJobAction: ${current_absja}"
~
When run:
$ ./test.sh
uid: user1
cn: User One
EmployeeNumber: 1234567
absJobAction:
$

Building Dynamic Variable Names in KornShell

I did a search but did not find anything quite like what I am trying to do.
I have a list of Server Hostnames & IPs
Servera | IPa
Serverb | IPb
Servern | IPn
I want to cat this file and put each element into variables
Server_Var_1
IP_Var_1
Server_Var_2
IP_Var_2
Server_Var_n
IP_Var_n
What I currently have is the following KornShell (ksh):
Counter=0
cat hostfile|while read line; do
Server_Var_"$Counter"=echo $line | awk -F"|" '{print $1}'
IP_Var_"$Counter"=echo $line | awk -F"|" '{print $2}'
echo $Server_Var_[*] $IP_Var_[*]
done
Any help is appreciated.
$ cat hostfile
server1 | 192.168.1.101
server2 | 192.168.1.102
server3 | 192.168.1.103
$ cat foo
#!/bin/sh
counter=0
while IFS=" |" read name ip; do
eval Server_VAR_$counter=$name
eval IP_VAR_$counter=$ip
: $(( counter += 1 ))
done < hostfile
echo $Server_VAR_0:$IP_VAR_0
echo $Server_VAR_1:$IP_VAR_1
echo $Server_VAR_2:$IP_VAR_2
$ ./foo
server1:192.168.1.101
server2:192.168.1.102
server3:192.168.1.103
Here's a slight twist to the original question (which was perfectly answered by #William Pursell). So this bit of code will produce the same output, but uses an array of compound variables instead. Note that this is specific to ksh93.
$ cat read_hostvars
#!/bin/sh
counter=0
typeset -a Server
while IFS=" |" read name ip; do
Server[counter].name=$name
Server[counter].ip=$ip
(( counter++ ))
done < hostfile
for n in ${!Server[#]}; do
echo ${Server[n].name}:${Server[n].ip}
done

Resources