Select value from 11th column as variable - bash

Question

Objective is to extract the ip address from the 11th column and feed it to whois.
The sourcefile it's first line are the headers so they should be ignored.
then I try to select with awk the 11th column.
Since skipping first line seems to be too hard (for me) right now I left it out for now. Any good suggestion is welcome.
The code so far:
while IFS= read -r p
do
DESTIP=$(awk 'BEGIN{FS=OFS=";"} {print $11}' $p)
echo "$DESTIP; $p"
ORGNAME=$(whois $DESTIP|grep 'OrgName')
COUNTRY=$(whois $DESTIP|grep 'Country')
echo "$p;$ORGNAME;$COUNTRY" >>whois-results.txt
done < working-sorted.csv
The first lines of the sourcefile:
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short fo
r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42 has made a TCP connection towards the dest
ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest
to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monitore
d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short f
or Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167 has made a TCP connection towards the de
stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges
t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monito
red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist
Result for now:
awk: cmd. line:1: fatal: cannot open file `2020-01-19' for reading (No such file or directory)
DESTINATION IP=
Variable P= 2020-01-19 20:42:56;43;3;Remote Administration Tool;::ffff:ac8:c8d0/128;101;00:0c:29:4c:20:37;Vmware;172.16.16.100;54552;52.174.64.84;443;TCP;Connection to blacklisted destination
After adjusting the awk to:
DESTIP=$(awk -v TEST='$p' 'BEGIN{FS=OFS=";"} {print $9;}')
I do get the ip addresses from the correct column, but the are in one list and not line by line, nor passed to the whois commands
Desired output:
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country;
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short fo
r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42 has made a TCP connection towards the dest
ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest
to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monitore
d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;SomeName;SomeCountry
For now I'm a bit stuck.
Help would be appriciated.

Answer 1

Consider this approach instead of your shell loop:
$ cat tst.awk
BEGIN {
numFlds = split("OrgName Country",nr2name)
FS=OFS=";"
}
{ delete name2val }
NR == 1 {
for (fldNr=1; fldNr<=numFlds; fldNr++) {
fldName = fldVal = nr2name[fldNr]
name2val[fldName] = fldVal
}
}
NR > 1 {
cmd = "whois \047" $9 "\047"
while ( (cmd | getline line) > 0 ) {
fldName = fldVal = line
sub(/[[:space:]]*:.*/,"",fldName)
sub(/[^:]+:[[:space:]]*/,"",fldVal)
name2val[fldName] = fldVal
}
close(cmd)
}
{
printf "%s%s", $0, OFS
for (fldNr=1; fldNr<=numFlds; fldNr++) {
fldName = nr2name[fldNr]
fldVal = name2val[fldName]
printf "%s%s", fldVal, (fldNr<numFlds ? OFS : ORS)
}
}
.
$ awk -f tst.awk file
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short for Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42 has made a TCP connection towards the dest ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monitore d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;;
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short for Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167 has made a TCP connection towards the de stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monito red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;OVH Hosting, Inc.;CA
since it gives you exactly the output you wanted, it won't fail when Country, for example, appears in one of the values (your current shell script will fail due to a false match given a company name of "Big Country", for example), and with that you can access any of the values output from whois by just referring to their name. So if you wanted to additionally print the "OrgAbuseEmail" all you have to do is change this:
numFlds = split("OrgName Country",nr2name)
to this:
numFlds = split("OrgName Country OrgAbuseEmail",nr2name)
Alternatively, this avoids spawning a shell once per IP address and so MAY be a bit more efficient than the above:
$ cat tst.sh
#!/bin/env bash
file="$1"
awk 'BEGIN{FS=OFS=";"} {print $9, $0}' "$file" |
while IFS=';' read -r ip all; do
whois "$ip"
printf '%s\n---\n' "$all"
done |
awk '
BEGIN {
numFlds = split("OrgName Country",nr2name)
for (fldNr=1; fldNr<=numFlds; fldNr++) {
fldName = nr2name[fldNr]
name2val[fldName] = fldName
}
FS = OFS = ";"
}
/^[[:alpha:]]+:/ {
fldName = fldVal = $0
sub(/[[:space:]]*:.*/,"",fldName)
sub(/[^:]+:[[:space:]]*/,"",fldVal)
name2val[fldName] = fldVal
}
/^---$/ {
printf "%s%s", prev, OFS
for (fldNr=1; fldNr<=numFlds; fldNr++) {
fldName = nr2name[fldNr]
fldVal = name2val[fldName]
printf "%s%s", fldVal, (fldNr<numFlds ? OFS : ORS)
}
delete name2val
}
{ prev = $0 }
'
.
$ ./tst.sh file
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short fo r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42 has made a TCP connection towards the dest ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monitore d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;;
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port: 100% CertaintyHigh Severity Category: SSH Description: Short f or Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167 has made a TCP connection towards the de stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions: 1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected. 2) If you are not expecting SSH traffic from the monito red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;OVH Hosting, Inc.;CA

Answer 2

I found a workaround. maybe not so nice, but at least working:
(edit: the workaround is now a bit nicer and less "workaround" :)
The code below is adjusted on advice of Cyrus.)
while IFS= read -r p
do
DESTIP=$(echo "$p" | awk 'BEGIN{FS=OFS=";"} {print $9;}')
echo "DESTINATION IP= $DESTIP"
echo "Variable P= $p"
ORGNAME=$(whois $DESTIP|grep 'OrgName')
COUNTRY=$(whois $DESTIP|grep 'Country')
echo "$p;$domain;$ORGNAME;$COUNTRY" >>working-whois.csv
done < working-sorted.csv
Thanks to Cyrus in this by making clear it needs an inputfile.