I need to lookup data from a lookup file. I am using bash ver 4.1.2 on rhel.
Lookup data (ip_lookup.txt)
1.1.1.1=server A
2.2.2.2=server B
.. and so on
I iterate list from a file (aggregated from access.log) and need to add information from the lookup file.
iterated list (service.txt)
1234 service_a
324 service_b
34 service_c
access.log with x_http_forwarder info
service_a 200 1.1.1.1
service_a 200 1.1.1.1
service_a 200 2.2.2.2
service_b 200 1.1.1.1
service_c 200 2.2.2.2
So the code I made was this:
# import lookup file into bash array
declare -A ary
while IFS== read -r key value; do
ary[$key]=$value
done < "ip_lookup.txt"
# then I nested iterate from list service and access.log
# list service and its caller
while IFS=$'\t' read -r -a line
do
echo -e "${line[0]}\t${line[1]}"
cat "access.log" |
tr -d \" |
sed "s/?wsdl.$PARTITION_COLUMN.\t/\t/g" |
sed '/^[[:space:]]*$/d' |
sed "s/?WSDL.$PARTITION_COLUMN.\t/\t/g" |
grep ${line[1]} |
awk '{ seen[$3] += $1 } END { for (i in seen) print seen[i],i }' |
sort -nr |
sed 's/ /\t/g' |
head -n 2 > "ip_list.txt"
while IFS=$'\t' read -r -a line
do
# the problem lie in this block <<<<<<<<<<<<<<<<<<<
echo -e "> ${line[1]} \t\t ${line[0]} \t\t ${ary[${line[1]}]}"
# it only returns var line 1 and line 0. While lookup var ${ary[${line[1]}] is empty
# fyi, ${line[1]} returns 1.1.1.1 or 2.2.2.2 (on each iteration)
fi
done < "ip_list.txt"
done < "service.txt"
Let me highlight the problem:
ip=${line[1]}
echo $ip #it returns IP such as 1.1.1.1
echo ${ary['1.1.1.1']} # it returns server A
# but if I call the key using variable, it won't work. I've tried some combination but it works nothing.
echo ${ary[$ip]} # it returns nothing
echo ${ary[${line[1]}]} # it returns nothing
echo ${ary[${!line[1]}]} # it returns nothing
Any idea how to return the value using the key variable in bash associative array?
feel free to ask the detail.
I'm getting following nmap output from a scan:
PORT STATE SERVICE
113/tcp closed ident
443/tcp open https
5060/tcp open sip
I want to extract only open ports and save them into a variable while my script progress is below:
#!/bin/bash
echo Please enter your IP/domain address for nmap vulnerability scanning:
read IP
echo Your results against $ip will be output in sometime
nmap -sS -oG output.txt $ip
Grep them
nmap -sS -oG output.txt $ip | grep open
To store in var
open_ports=$(nmap -sS -oG output.txt $ip | grep open)
open_ports=${open_ports//[^0-9]/ } # remove text
Convert to an array
open_ports_arr=( $open_ports )
Here is how you can filter and extract open ports using awk and read the results into a bash array:
#!/usr/bin/env bash
# Read prompt ip address
read \
-p 'Please enter your IP/domain address for nmap vulnerability scanning: ' \
-r ip
# Print format info to user
printf 'Your results against %s will be output in sometime.\n' "$ip"
# Read the output of the nmap | awk commands into the ports array
IFS=$'\n' read -r -d '' -a ports < <(
# Pipe the result of nmap to awk for processing
nmap -sS -oG output.txt "$ip" |
awk -F'/' '
/[[:space:]]+open[[:space:]]+/{
p[$1]++
}
END{
for (k in p)
print k
}'
)
# If the resulting pors array is not empty iterate print format its content
if [ ${#ports[#]} -gt 0 ]; then
printf 'List of open ports on host IP: %s\n' "$ip"
for p in "${ports[#]}"; do
printf '%d\n' "$p"
done
fi
I'm really new to bash. I have a list of domains in a .txt file (URLs.txt). I also want to have a .csv file which consists of 3 columns separated by , (myFile.csv). My code reads each line of URLs.txt (each domain), finds its IP address and then inserts them into myFile.csv (domain in the first column, its IP in the 2nd column.
Name, IP
ex1.com, 10.20.30.40
ex2.com, 20.30.40.30
ex3.com, 10.45.60.20
ex4.com, 10.20.30.40
Here is my code:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
echo "$line,$ipValue" >> myFile.csv
fi
done < URLs.txt
I want to add another column to myFile.csv for keeping open ports of each IP. So output would be like this:
Name, IP, Port
ex1.com, 10.20.30.40, 21/tcp
ex2.com, 20.30.40.30, 20/tcp
ex3.com, 10.45.60.20, 33/tcp
ex4.com, 10.20.30.40, 21/tcp
I want to use Nmap to do this. After I choose an IP address from the 2nd column of myFile.csv and find its open ports using Nmap, I want to write the Nmap result to the corresponding cell of the 3rd column.
Also, if there is another similar IP in the 2nd column I want to write the Nmap result for that line too. I mean I don't want to run Nmap again for the duplicated IP. For example, in my example, there are two "10.20.30.40" in the 2nd column. I want to use Nmap just once and for the 1st "10.20.30.40" (and write the result for the 2nd "10.20.30.40" as well, Nmap should not be run for the duplicated IP).
For this to happen, I changed the first line of my code to this:
echo "Name,IP,Port" > myFile.csv
and also here is the Nmap code to find the open ports:
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
but I don't know what to do next and how to apply these changes to my code.
I updated my code to something like this:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
echo "$line,$ipValue,$port" >> myFile.csv
fi
done < URLs.txt
but this way, Nmap was used for finding the open ports of the duplicated IPs too, but I didn't want this. What should I do?
Here's a modified version of your script that roughly does what you want:
#!/usr/bin/env bash
# cache maps from IP addresses to open ports
declare -A cache
getports() {
local ip=$1
nmap -v -Pn -p 1-100 "$ip" -oG - \
| awk -F '\t' '
/Ports:/ {
n = split($2, a, /,? /)
printf "%s", a[2]
for (i = 3; i <= n; ++i)
printf ":%s", a[i]
}
'
}
{
echo 'Name,IP,Port'
while IFS= read -r url; do
# Read filtered dig output into array
readarray -t ips < <(dig +short "$url" | grep -E '^([0-9]+\.){3}[0-9]+$')
# Build array of open ports
unset ports
for ip in "${ips[#]}"; do
ports+=("${cache["$ip"]:=$(getports "$ip")}")
done
# Output
printf '%s,%s,%s\n' \
"$url" \
"$(IFS='-'; echo "${ips[*]}")" \
"$(IFS='-'; echo "${ports[*]}")"
done < URLs.txt
} > myFile.csv
The readarray line reads the filtered output from dig into an array of IP addresses; if that array has length zero, the rest of the loop is skipped.
Then, for each elements in the ips array, we get the ports. To avoid calling nmap if we've seen the IP address before, we use the ${parameter:=word} parameter expansion: if ${cache["$ip"]} is non-empty, use it, otherwise call the getports function and store the output in the cache associative array.
getports is called for IP addresses we haven't seen before; I've used -oG ("grepable output") to make parsing easier. The awk command filters for lines containing Ports:, which look something like
Host: 52.94.225.242 () Ports: 80/open/tcp//http/// Ignored State: closed (99)
with tab separated fields. We then split the second field on the regular expression /,? / (an optional comma followed by a blank) and store all but the first field of the resulting array, colon separated.
Finally, we print the line of CSV data; if ips or ports contain more than one element, we want to join the elements with -, which is achieved by setting IFS in the command substitution and then printing the arrays with [*].
The initial echo and the loop are grouped within curly braces so output redirection has to happen just once.
I want to read from a file that has host IPs written in it and save it in an array. So far I have tried this:
Host=`cat /home/hp3385/Desktop/config | egrep '^Host' | awk '{print $2}'`
But I don't think that it saves the information in an array. What is the type of the variable 'Host'? If it's not an array how can I convert it into one?
This is a sample data from the file /home/hp3385/Desktop/config:
############# Server1 #################
Host 8.8.8.8
Hostname google
############# Server2 ################
Host 8.8.4.4
Hostname google
The expected output is:
a=($'8.8.8.8' $'8.8.4.4')
You can try this
myarray=()
while read -r line; do
if echo "$line" | grep -q 'Host '; then
myarray+=($(echo "$line" | awk '/^Host/ {print $2}'))
fi
done < /home/hp3385/Desktop/config
Declaring an array:
ARRAY=(0 1 2 3 4 5)
So your array can be declared like this:
HOSTS=($(awk '/^Host/ {print $2}' YOUR_FILE))
If you want to know the amount of values in your array:
echo ${#HOSTS[*]}
To get an output of all values in your array (credit goes to triplee):
printf '%s\n' "${HOSTS[#]}"
I've received some helpful responses in the past and am hoping you can all help me out. I came across some weird behavior that I can't quite nail down. I'm processing configuration files for Cisco switches and want to generate output that lists the VLAN IP Addresses in a format that would show:
Vlan1: 172.31.200.1 255.255.255.0
Vlan10: 172.40.220.1 255.255.255.0
The "Vlan" would be captured in a variable and the IP/Mask is extracted using "sed" and it works for the most part. Occasionally though it refuses to populate the "vlan" variable even though it appears to work great for other configs.
If there's only one VLAN it just handles that one, if there's more than one it handles the additional ones. If the user selects (-v) it includes VLAN1 on the list there are multiple VLANs configured (otherwise it ignores VLAN1).
This input file appears broken (Filename 1.cfg):
!
interface Vlan1
ip address 172.29.96.100 255.255.255.0
!
ip default-gateway 172.29.96.1
ip http server
no ip http secure-server
!
This input file works fine (Filename 2.cfg):
!
interface Vlan1
ip address 172.31.200.111 255.255.255.0
no ip route-cache
!
ip default-gateway 172.31.200.1
ip http server
ip http secure-server
The output that I get is this:
Notice how the first one fails to include the "Vlan1" reference?
Here's my script:
#!/bin/bash
if [ -f getlan.log ];
then
rm getlan.log
fi
TempFile=getlan.log
verbose=0
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-v)
verbose=1
shift
;;
#*)
#exit
#shift
#;;
esac
done
#Start Processing all files
files=$( ls net )
for i in $files; do
#########################################################
# Collect Configured VLAN Interfaces and IP Information #
#########################################################
echo "-------- Configured VLAN Interfaces --------" >> ~/$TempFile
echo "" >> ~/$TempFile
if [ `grep "^interface Vlan" ~/net/$i | awk '{ print $2 }' | wc -l` -gt 1 ];
then
for g in `grep "^interface Vlan" ~/net/$i | awk '{ print $2 }'`;
do
if [ $g == "Vlan1" ];
then
if [ $verbose -gt 0 ];
then
echo "$g": `sed -n '/^interface '$g'/,/!/p' ~/net/$i | head -n 5 | grep -i "ip address" | awk '{ print $3, $4 }'` >> ~/$TempFile
fi
else
echo "$g": `sed -n '/^interface '$g'/,/^!/p' ~/net/$i | grep -i "ip address" | awk '{ print $3, $4 }'` >> ~/$TempFile
fi
done
echo "" >> ~/$TempFile
else
vlanid=`grep "^interface Vlan" ~/net/$i | awk '{ print $2 }'`
echo $vlanid: `sed -n '/^interface 'Vlan'/,/^!/p' ~/net/$i | grep -i "address" | awk '{ print $3, $4 }'` >> ~/$TempFile
echo "" >> ~/$TempFile
fi
done
It would be really great if this was more consistent. Thanks!
A best-practices approach might look more like:
#!/usr/bin/env bash
interface_re='^[[:space:]]*interface[[:space:]]+(.*)'
ip_re='^[[:space:]]*ip address (.*)'
process_file() {
local line interface
interface=
while IFS= read -r line; do
if [[ $line =~ $interface_re ]]; then
interface=${BASH_REMATCH[1]}
continue
fi
if [[ $interface ]] && [[ $line =~ $ip_re ]]; then
echo "${interface}: ${BASH_REMATCH[1]}"
fi
done
}
for file in net/*; do
process_file <"$file"
done
This can be tested as follows:
process_file <<'EOF'
!
interface Vlan1
ip address 172.29.96.100 255.255.255.0
!
ip default-gateway 172.29.96.1
ip http server
no ip http secure-server
!
!
interface Vlan1
ip address 172.31.200.111 255.255.255.0
no ip route-cache
!
ip default-gateway 172.31.200.1
ip http server
ip http secure-server
EOF
...which correctly identifies the ip address lines from both interface blocks, emitting:
Vlan1: 172.29.96.100 255.255.255.0
Vlan1: 172.31.200.111 255.255.255.0