Bash: awk removing parentheses from array items - bash

So I'm creating a small network scan automation tool using nmap as part of the backend, when attempting to get hostnames and IP addresses the IP is formatted like this "(192.168.0.1)" currently I'm using this bit of code to grab hostname and IP and print.
devices=($( nmap $net_addr -sn | grep -E '([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})' | awk {' print $5 "\t" $6)'} ))
printf '%-10s %s\n' "${devices[#]}"
This code will print this
UNKNOWN (192.168.0.1)
UNKNOWN (192.168.0.3)
UNKNOWN (192.168.0.5)
UNKNOWN (192.168.0.6)
UNKNOWN (192.168.0.16)
UNKNOWN (192.168.0.18)
UNKNOWN (192.168.0.19)
UNKNOWN (192.168.0.21)
UNKNOWN (192.168.0.22)
UNKNOWN (192.168.0.23)
UNKNOWN (192.168.0.32)
UNKNOWN (192.168.0.253)
I attempted to use substr($6,2) but this only chops off the first bit and sadly as the nmap IP addresses are variable in length this doesn't help me.
In short, this is the desired output.
UNKNOWN 192.168.0.1
UNKNOWN 192.168.0.3
UNKNOWN 192.168.0.5
UNKNOWN 192.168.0.6
UNKNOWN 192.168.0.16
UNKNOWN 192.168.0.18
UNKNOWN 192.168.0.19
UNKNOWN 192.168.0.21
UNKNOWN 192.168.0.22
UNKNOWN 192.168.0.23
UNKNOWN 192.168.0.32
UNKNOWN 192.168.0.253
here is the output of nmap 192.168.0.0/24 -sn
Starting Nmap 7.40 ( https://nmap.org ) at 2017-12-31 14:42 UTC
Nmap scan report for UNKNOWN (192.168.0.1)
Host is up (0.0014s latency).
Nmap scan report for UNKNOWN (192.168.0.3)
Host is up (0.052s latency).
Nmap scan report for UNKNOWN (192.168.0.5)
Host is up (0.0065s latency).
Nmap scan report for UNKNOWN (192.168.0.6)
Host is up (0.0073s latency).
Nmap scan report for UNKNOWN (192.168.0.7)
Host is up (0.029s latency).
Nmap scan report for UNKNOWN (192.168.0.16)
Host is up (0.011s latency).
Nmap scan report for UNKNOWN (192.168.0.18)
Host is up (0.053s latency).
Nmap scan report for UNKNOWN (192.168.0.21)
Host is up (0.011s latency).
Nmap scan report for UNKNOWN (192.168.0.22)
Host is up (0.012s latency).
Nmap scan report for UNKNOWN (192.168.0.23)
Host is up (0.0061s latency).
Nmap scan report for UNKNOWN (192.168.0.32)
Host is up (0.0013s latency).
Nmap scan report for UNKNOWN (192.168.0.253)
Host is up (0.0011s latency).
Nmap done: 256 IP addresses (12 hosts up) scanned in 3.16 seconds

As per your shown Input you could try following single awk to get the shown expected output.
your_command | awk '/Nmap scan report/{gsub(/\)|\(/,"",$NF);print $(NF-1),$NF}'
EDIT: Eliminating grep from OP's command and trying to solve it within single awk itself now.
devices=$(nmap $net_addr -sn | awk --re-interval '{if(match($0,/\([0-9]{3}\.[0-9]{3}\.[0-9]{1,3}\.[0-9]{1,3}\)/)){gsub(/\)|\(/,"",$NF);print $(NF-1),$NF}}')
Also since my awk is OLD one in case you have latest version of awk then kindly don't use --re-interval in solution.

You can try this sed too
nmap $net_addr -sn | sed '/^Nmap scan report for /!d;s///;s/[)(]//g'

Related

How can I analyze multiple blobs (multi-lines) of text for patterns in a single file?

I have run an nmap scan using the --script ssl-enum-ciphers -p443 192.168.0.0/24 options against multiple IP addresses. I have also run an extremely similar scan using the --script ssh2-enum-algos -p22 options, that produces output in the same format.
I want to quickly analyze this data and zero in on specific matches of specific ciphers or algorithms. The overarching goal is to run an ad-hoc internal vulnerability assessment without access to fancy tools such as Nessus or Rapid7 InsightVM.
While nmap supports the -oX option to output to XML, I have found that neither Microsoft Word, Excel, or a web browser know how to open the file. The Microsoft products keep producing an error that the xml format is incorrect.
So then I tried the nmap-parse-output code on Github. While it will easily group IP addresses by ports, it doesn't appear to be able to take that a step further and analyze the ciphers or algorithms for me.
So now I'm trying to figure out a way to manually parse these blobs of data.
A typical result might contain data for multiple IP addresses in the following format (the following example stdout is edited for brevity):
Nmap scan report for 192.168.1.1
Host is up (0.00064s latency).
PORT STATE SERVICE VERSION
443/tcp open ssl/http lighttpd
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A
Nmap scan report for 192.168.1.2
Host is up (0.00048s latency).
PORT STATE SERVICE VERSION
443/tcp open ssl/http nginx (reverse proxy)
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A
Nmap scan report for 192.168.1.3
....
Nmap scan report for 192.168.1.4
....
How can write a script, using standard applications available to bash, to loop through each blob of text? We don't know the length of each blob, so I need to match on the string "Nmap scan report" or something similar, and extract the data that is in between each of those matches.
Something like this would get me started, but it isn't complete, and doesn't actually separate each blob individually:
for i in $(cat scan-results | grep "Nmap scan report for"); do more data analysis here; done
For example, I might want to search for any IP address that support RC4 ciphers on port 443, so in the "do more data analysis", I would like to run: grep -i rc4
Or in the case of ssh algorithms, I want to ensure all cbc algorithms are disabled, so I could run: grep -i cbc
The resulting goal would be to list anything that matches so that I can quickly attribute the match to the specific IP address. I don't care how the results look, I just care about finding the results quickly.
Any help would be appreciated!
Without more sample data, and going solely on the limited examples, and keeping in mind that output format isn't of importance ...
NOTE: My sample data file - nmap.dat - is a cut-n-paste copy of the sample nmap data provided by the OP.
I'm thinking a multi-pattern grep may suffice, eg:
# search for any IP address that support RC4 ciphers on port 443
$ grep -i "Nmap scan report for|443|RC4" nmap.dat
Nmap scan report for 192.168.1.1
443/tcp open ssl/http lighttpd
Nmap scan report for 192.168.1.2
443/tcp open ssl/http nginx (reverse proxy)
Nmap scan report for 192.168.1.3
Nmap scan report for 192.168.1.4
# want to ensure all cbc algorithms are disabled
$ egrep -i "Nmap scan report for|cbc" nmap.dat
Nmap scan report for 192.168.1.1
Nmap scan report for 192.168.1.2
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
Nmap scan report for 192.168.1.3
Nmap scan report for 192.168.1.4

How can I split parsed text by "blocks" of text?

I am writing a bash script that runs an Nmap scan of the network. After this the scan needs to be examinned and the relevant bits need to be extracted.
I need to extract the IP, MAC and OS from the completed scan. The problem is that Nmap does not always get the OS from the scan and therefore does not put it in the results. I need to associate the IP, MAC and OS in the end result.
Here is an example of a test scan:
Nmap scan report for 192.168.0.1
Host is up (0.0029s latency).
Not shown: 990 closed ports
PORT STATE SERVICE
PORT# STATE XXXXXXX
MAC Address: MA:CA:DR:ES:S0:03 (Unknown)
Device type: general purpose
Running: Linux 2.6.X|3.X
OS CPE: cpe:/o:linux:linux_kernel:2.6 cpe:/o:linux:linux_kernel:3
OS details: Linux 2.6.32 - 3.13
Network Distance: 1 hop
Nmap scan report for 192.168.0.102
Host is up (0.0044s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
PORT# STATE XXXXXXX
MAC Address: MA:CA:DR:ES:S0:02 (Sony Mobile Communications AB)
Too many fingerprints match this host to give specific OS details
Network Distance: 1 hop
Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT STATE SERVICE
PORT# STATE XXXXXX
MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running (JUST GUESSING): Microsoft Windows 2008 (91%)
OS CPE: cpe:/o:microsoft:windows_server_2008::sp1 cpe:/o:microsoft:windows_server_2008:r2
Aggressive OS guesses: Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
Also note how the last one in the example above could not find the OS, in this case the aggress guess is wanted
The end result needs to be a text file that has has something like the following:
192.168.0.1 - MA:CA:DR:ES:S0:03 - Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 - Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2
I did some research but could not find anything that explains how I can associate the IP with the mac addresses and the os in the text blocks.
I have the following commands that work with a simple scan where the IP and Mac addresses are next to each other
while read line; do
Mac="$(grep -oE '[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}' <<< "$line")"
ip="$(grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' <<< "$line")"
echo -e $ip'\t-\t '$Mac >>/path/to/results.txt
done </path/to/testscan.txt
I am fairly new to bash scripting so apologies if I am missing something obvious.
The nmap command for anyone interested is:
nmap -O --osscan-guess 192.168.0.0/24 -oN /path/to/testscan.txt
Sorry for the wall of text, I figured the more information the better!
This would be pretty easy to parse with awk:
BEGIN {os_details="Not found"}
/^Nmap scan report/ {target=$5}
/^MAC Address/ {mac_address=$3}
/^OS details/ {os_details=substr($0, length("OS details: "))}
/^Aggressive OS guesses/ {
os_details=substr($0, length("Aggressive OS guesses: "))
}
# This matches the blank lines between hosts
/^$/ {
printf "%s - %s - %s\n", target, mac_address, os_details
target=""
mac_address=""
os_details="Not found"
}
END {
printf "%s - %s - %s\n", target, mac_address, os_details
}
Running this on your sample data gets you:
192.168.0.1 - MA:CA:DR:ES:S0:03 - Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 - Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)
I had to make one correct to what I believe was an error in your sample data...I removed the blank line before the MAC Address line here:
Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT STATE SERVICE
PORT# STATE XXXXXX
MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)
Using option -oX of the nmap (output to the XML format) the parsing could be more accurate:
nmap -oX /path/to/testscan.xml ...
# or
nmap -oX - ... > /path/to/testscan.xml
Then you could to use, for example, xmllint to parse this XML with XPath:
file="/path/to/testscan.xml"
get_details() {
local file addr mac os
file="$1"
addr=$2
mac=$(xmllint --xpath "string(//address[../address[#addr='$addr']][#addrtype='mac']/#addr)" "$file")
os=$(xmllint --xpath "string(//os[../address[#addr='$addr']]/osmatch/#name)" "$file")
: ${mac:="No data"}
: ${os:="No data"}
printf "%s - %s - %s\n" "$addr" "$mac" "$os"
}
for a in $(xmllint --xpath "//address[#addrtype='ipv4']/#addr" "$file" | grep -Po '\d+\.\d+\.\d+\.\d+'); do
get_details "$file" $a
done

ssh client to show server-supported algorithms

In order to check that all the servers across a fleet aren't supporting deprecated algorithms, I'm (programmatically) doing this:
telnet localhost 22
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.0p1 Ubuntu-6build1
SSH-2.0-Censor-SSH2
4&m����&F �V��curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1Arsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519lchacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.comlchacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com�umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1�umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1none,zlib#openssh.comnone,zlib#openssh.comSSH-2.0-Censor-SSH2
Connection closed by foreign host.
Which is supposed to be a list of supported algorithms for the various phases of setting up a connection. (kex, host key, etc). Every time I run, I get a different piece of odd data at the start - always a different length.
There's an nmap plugin - ssh2-enum-algos - which returns the data in it's complete form, but I don't want to run nmap; I have a go program which opens the port, and sends the query, but it gets the same as telnet. What am I missing, and how do I fix it?
For comparison, here's the top few lines from the output of nmap script:
$ nmap --script ssh2-enum-algos super
Starting Nmap 7.80 ( https://nmap.org ) at 2019-12-27 22:15 GMT
Nmap scan report for super (192.168.50.1)
Host is up (0.0051s latency).
rDNS record for 192.168.50.1: supermaster
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
| ssh2-enum-algos:
| kex_algorithms: (12)
| curve25519-sha256
| curve25519-sha256#libssh.org
| ecdh-sha2-nistp256
| ecdh-sha2-nistp384
| ecdh-sha2-nistp521
Opening a tcp connection to port 22, (in golang, with net.Dial) then accepting and sending connection strings leaves us able to Read() from the Reader for the connection. Thence the data is in a standard format described by the RFC. From this, I can list the algorithms supported in each phase of an ssh connection. This is very useful for measuring what is being offered, rather than what the appears to be configured (it's easy to configure sshd to use a different config file).
It's a useful thing to be able to do from a security POV.
Tested on every version of ssh I can find from 1.x on a very old solaris or AIX box, to RHEL 8.1.
In some cases you can specify an algorithm to use, and if you specify one that is not supported the server will reply with a list of supported algorithms.
For example, to check for supported key exchange algorithms you can use:
ssh 127.0.0.1 -oKexAlgorithms=diffie-hellman-group1-sha1
diffie-hellman-group1-sha1 is insecure and should be missing from most modern servers. The server will probably respond with something like:
Unable to negotiate with 127.0.0.1 port 22: no matching key exchange method found. Their offer: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
Exit 255
Typing: "ssh -Q cipher | cipher-auth | mac | kex | key"
will give you a list of the algorithms supported by your client
Typing: "man ssh"
will let you see what options you can specify with the -o argument, including Cipher, MACs, and KexAlgorithms

Extract only MAC addresses from arp-scan -l

I want to collect only MAC addresses from arp-scan -l, omitting IP addresses and the devices name. How do we do it? I know that these in the middle column are all MAC addresses.
192.168.1.1 bc:98:89:47:20:f8 Fiberhome Telecommunication Technologies Co.,LTD
192.168.1.3 70:18:8b:5e:01:fd Hon Hai Precision Ind. Co.,Ltd.
192.168.1.5 90:e7:c4:da:80:76 HTC Corporation
192.168.1.6 b8:27:eb:b0:4d:25 Raspberry Pi Foundation
I want like this
bc:98:89:47:20:f8
70:18:8b:5e:01:fd
90:e7:c4:da:80:76
b8:27:eb:b0:4d:25
Edit:
arp-scan -l gives the following result
Interface: wlp5s0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9.5 with 256 hosts (https://github.com/royhills/arp-scan)
192.168.1.1 bc:98:89:47:20:f8 Fiberhome Telecommunication Technologies Co.,LTD
192.168.1.5 90:e7:c4:da:80:76 HTC Corporation
192.168.1.3 70:18:8b:5e:01:fd Hon Hai Precision Ind. Co.,Ltd.
192.168.1.5 90:e7:c4:da:80:76 HTC Corporation (DUP: 2)
192.168.1.6 b8:27:eb:b0:4d:25 Raspberry Pi Foundation
192.168.1.6 b8:27:eb:b0:4d:25 Raspberry Pi Foundation (DUP: 2)
192.168.1.4 80:35:c1:4a:a5:dc (Unknown)
9 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9.5: 256 hosts scanned in 3.017 seconds (84.85 hosts/sec). 7 responded
I am assuming that you are using a linux-like shell and awk utility is installed( it is mostly pre-installed )-
arp-scan -l | awk '/.*:.*:.*:.*:.*:.*/{print $2}'
Learn basic Unix shell!
In particular the cut command can be used to extract columns from text. Set the field separator to space (probably default) and select the second field.
Using --plain option of arp-scan can make things easier. It will display plain output showing only responding hosts. The hosts' information are separated by tab, MAC address (2nd column) can be extracted easily through cut, as Anony-Mousse described:
arp-scan -l --plain | cut -f 2

multiline search in grep

I have made a text file of nmap's output and i was trying to find only those ip whose port are open
and when i am using grep i am not getting the desired output only one of the item is i am able to get either ip or the text open
data:
Nmap scan report for xxx.xxx.xxx.83
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.87
Host is up (0.049s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.89
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.90
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
Nmap scan report for xxx.xxx.xxx.96
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.100
Host is up (0.054s latency).
PORT STATE SERVICE
80/tcp filtered http
You could try the below awk command,
$ awk -v RS="" '/ open /{print $5}' file
xxx.xxx.xxx.93
It prints the ip (column no 5) only if the certain block contains the text open
After awk and grep perl too:
perl -00 -lanE 'say $_ if m/open/' < file
prints:
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
or
perl -00 -lanE 'say $F[4] if m/open/' < file
prints
xxx.xxx.xxx.93
If there are always 4 lines per block and the word open is in the last line, you can do:
grep -B4 open file
and it will show the 4 lines before the word open.
What about using awk? This will report the IP address the first time an open port is encountered in the nmap output:
sh$ awk '$3=="report"{ IP = $5 } $2=="open"&&IP { print IP; IP="" }' nmap.out
xxx.xxx.xxx.93

Resources