multiline search in grep - bash

I have made a text file of nmap's output and i was trying to find only those ip whose port are open
and when i am using grep i am not getting the desired output only one of the item is i am able to get either ip or the text open
data:
Nmap scan report for xxx.xxx.xxx.83
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.87
Host is up (0.049s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.89
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.90
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
Nmap scan report for xxx.xxx.xxx.96
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.100
Host is up (0.054s latency).
PORT STATE SERVICE
80/tcp filtered http

You could try the below awk command,
$ awk -v RS="" '/ open /{print $5}' file
xxx.xxx.xxx.93
It prints the ip (column no 5) only if the certain block contains the text open

After awk and grep perl too:
perl -00 -lanE 'say $_ if m/open/' < file
prints:
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
or
perl -00 -lanE 'say $F[4] if m/open/' < file
prints
xxx.xxx.xxx.93

If there are always 4 lines per block and the word open is in the last line, you can do:
grep -B4 open file
and it will show the 4 lines before the word open.

What about using awk? This will report the IP address the first time an open port is encountered in the nmap output:
sh$ awk '$3=="report"{ IP = $5 } $2=="open"&&IP { print IP; IP="" }' nmap.out
xxx.xxx.xxx.93

Related

Have script find all lines containing x, prepend these lines to each line that follows unless the line has another x in it, at which point repeat

I have a file called file.txt which contains either an IP or a FQDN, followed by ports that were found to be open. I want my bash script to prepend all ports with their associated IP/FQDN (always above them in the file), replace the forward slash with a whitespace, and then delete the IP/FQDN line that isn't associated with a port.
Breaking it down, I thought:
Read the next line
If the line contains a "." in it (IP or FQDN), prepend it into all following lines, unless:
If the following line has a "." in it (another IP or FQDN), make that line the new one to prepend and repeat the process for all following lines
Replace all "/" with a " " (one single whitespace)
Remove all lines that are not associated with a port (probably easier just to grep for "tcp" and "udp" as that will display all open ports with associated IP/FQDN
To make it easier, I can easily create a tmp file if necessary within the process. I have tried various iterartions of "while" and "if" and nothing seems to work...!
E.g:
cat file.txt
www.thisisawebsite.com:
80/tcp
443/tcp
500/udp
192.168.1.5:
80/tcp
dev.anothersite.co.uk:
22/tcp
443/tcp
5050/udp
21000/tcp
10.10.10.10:
4000/udp
8000/udp
Then, running the bash script, it should become:
www.thisisawebsite.com:80 tcp
www.thisisawebsite.com:443 tcp
www.thisisawebsite.com:500 udp
192.168.1.5:80 tcp
dev.anothersite.co.uk:22 tcp
dev.anothersite.co.uk:433 tcp
dev.anothersite.co.uk:5050 udp
dev.anothersite.co.uk:21000 tcp
10.10.10.10:4000 udp
10.10.10.10:8000 udp
Looks like a job for awk:
awk -F/ '/\./ {d=$0; next} {print d":"$1, $2}'

How can I analyze multiple blobs (multi-lines) of text for patterns in a single file?

I have run an nmap scan using the --script ssl-enum-ciphers -p443 192.168.0.0/24 options against multiple IP addresses. I have also run an extremely similar scan using the --script ssh2-enum-algos -p22 options, that produces output in the same format.
I want to quickly analyze this data and zero in on specific matches of specific ciphers or algorithms. The overarching goal is to run an ad-hoc internal vulnerability assessment without access to fancy tools such as Nessus or Rapid7 InsightVM.
While nmap supports the -oX option to output to XML, I have found that neither Microsoft Word, Excel, or a web browser know how to open the file. The Microsoft products keep producing an error that the xml format is incorrect.
So then I tried the nmap-parse-output code on Github. While it will easily group IP addresses by ports, it doesn't appear to be able to take that a step further and analyze the ciphers or algorithms for me.
So now I'm trying to figure out a way to manually parse these blobs of data.
A typical result might contain data for multiple IP addresses in the following format (the following example stdout is edited for brevity):
Nmap scan report for 192.168.1.1
Host is up (0.00064s latency).
PORT STATE SERVICE VERSION
443/tcp open ssl/http lighttpd
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A
Nmap scan report for 192.168.1.2
Host is up (0.00048s latency).
PORT STATE SERVICE VERSION
443/tcp open ssl/http nginx (reverse proxy)
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A
Nmap scan report for 192.168.1.3
....
Nmap scan report for 192.168.1.4
....
How can write a script, using standard applications available to bash, to loop through each blob of text? We don't know the length of each blob, so I need to match on the string "Nmap scan report" or something similar, and extract the data that is in between each of those matches.
Something like this would get me started, but it isn't complete, and doesn't actually separate each blob individually:
for i in $(cat scan-results | grep "Nmap scan report for"); do more data analysis here; done
For example, I might want to search for any IP address that support RC4 ciphers on port 443, so in the "do more data analysis", I would like to run: grep -i rc4
Or in the case of ssh algorithms, I want to ensure all cbc algorithms are disabled, so I could run: grep -i cbc
The resulting goal would be to list anything that matches so that I can quickly attribute the match to the specific IP address. I don't care how the results look, I just care about finding the results quickly.
Any help would be appreciated!
Without more sample data, and going solely on the limited examples, and keeping in mind that output format isn't of importance ...
NOTE: My sample data file - nmap.dat - is a cut-n-paste copy of the sample nmap data provided by the OP.
I'm thinking a multi-pattern grep may suffice, eg:
# search for any IP address that support RC4 ciphers on port 443
$ grep -i "Nmap scan report for|443|RC4" nmap.dat
Nmap scan report for 192.168.1.1
443/tcp open ssl/http lighttpd
Nmap scan report for 192.168.1.2
443/tcp open ssl/http nginx (reverse proxy)
Nmap scan report for 192.168.1.3
Nmap scan report for 192.168.1.4
# want to ensure all cbc algorithms are disabled
$ egrep -i "Nmap scan report for|cbc" nmap.dat
Nmap scan report for 192.168.1.1
Nmap scan report for 192.168.1.2
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
Nmap scan report for 192.168.1.3
Nmap scan report for 192.168.1.4

How can I split parsed text by "blocks" of text?

I am writing a bash script that runs an Nmap scan of the network. After this the scan needs to be examinned and the relevant bits need to be extracted.
I need to extract the IP, MAC and OS from the completed scan. The problem is that Nmap does not always get the OS from the scan and therefore does not put it in the results. I need to associate the IP, MAC and OS in the end result.
Here is an example of a test scan:
Nmap scan report for 192.168.0.1
Host is up (0.0029s latency).
Not shown: 990 closed ports
PORT STATE SERVICE
PORT# STATE XXXXXXX
MAC Address: MA:CA:DR:ES:S0:03 (Unknown)
Device type: general purpose
Running: Linux 2.6.X|3.X
OS CPE: cpe:/o:linux:linux_kernel:2.6 cpe:/o:linux:linux_kernel:3
OS details: Linux 2.6.32 - 3.13
Network Distance: 1 hop
Nmap scan report for 192.168.0.102
Host is up (0.0044s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
PORT# STATE XXXXXXX
MAC Address: MA:CA:DR:ES:S0:02 (Sony Mobile Communications AB)
Too many fingerprints match this host to give specific OS details
Network Distance: 1 hop
Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT STATE SERVICE
PORT# STATE XXXXXX
MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running (JUST GUESSING): Microsoft Windows 2008 (91%)
OS CPE: cpe:/o:microsoft:windows_server_2008::sp1 cpe:/o:microsoft:windows_server_2008:r2
Aggressive OS guesses: Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
Also note how the last one in the example above could not find the OS, in this case the aggress guess is wanted
The end result needs to be a text file that has has something like the following:
192.168.0.1 - MA:CA:DR:ES:S0:03 - Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 - Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2
I did some research but could not find anything that explains how I can associate the IP with the mac addresses and the os in the text blocks.
I have the following commands that work with a simple scan where the IP and Mac addresses are next to each other
while read line; do
Mac="$(grep -oE '[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}' <<< "$line")"
ip="$(grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' <<< "$line")"
echo -e $ip'\t-\t '$Mac >>/path/to/results.txt
done </path/to/testscan.txt
I am fairly new to bash scripting so apologies if I am missing something obvious.
The nmap command for anyone interested is:
nmap -O --osscan-guess 192.168.0.0/24 -oN /path/to/testscan.txt
Sorry for the wall of text, I figured the more information the better!
This would be pretty easy to parse with awk:
BEGIN {os_details="Not found"}
/^Nmap scan report/ {target=$5}
/^MAC Address/ {mac_address=$3}
/^OS details/ {os_details=substr($0, length("OS details: "))}
/^Aggressive OS guesses/ {
os_details=substr($0, length("Aggressive OS guesses: "))
}
# This matches the blank lines between hosts
/^$/ {
printf "%s - %s - %s\n", target, mac_address, os_details
target=""
mac_address=""
os_details="Not found"
}
END {
printf "%s - %s - %s\n", target, mac_address, os_details
}
Running this on your sample data gets you:
192.168.0.1 - MA:CA:DR:ES:S0:03 - Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 - Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)
I had to make one correct to what I believe was an error in your sample data...I removed the blank line before the MAC Address line here:
Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT STATE SERVICE
PORT# STATE XXXXXX
MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)
Using option -oX of the nmap (output to the XML format) the parsing could be more accurate:
nmap -oX /path/to/testscan.xml ...
# or
nmap -oX - ... > /path/to/testscan.xml
Then you could to use, for example, xmllint to parse this XML with XPath:
file="/path/to/testscan.xml"
get_details() {
local file addr mac os
file="$1"
addr=$2
mac=$(xmllint --xpath "string(//address[../address[#addr='$addr']][#addrtype='mac']/#addr)" "$file")
os=$(xmllint --xpath "string(//os[../address[#addr='$addr']]/osmatch/#name)" "$file")
: ${mac:="No data"}
: ${os:="No data"}
printf "%s - %s - %s\n" "$addr" "$mac" "$os"
}
for a in $(xmllint --xpath "//address[#addrtype='ipv4']/#addr" "$file" | grep -Po '\d+\.\d+\.\d+\.\d+'); do
get_details "$file" $a
done

display open ports grouped by process

given a netstat output, how can i display the selected open ports grouped by process?
what i got so far:
:~# netstat -tnlp | awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'
haproxy 636
haproxy 993
haproxy 993
haproxy 465
haproxy 465
exim4 25
apache2 80
exim4 25
apache2 443
desired output (in one line):
apache2 (80 443), exim4 (25), haproxy (465 636 993)
please note:
i have duplicated lines because they listen on different IPs, but i only need one (sort -u is ok)
if possible, id like to sort by process and then by port
the main goal is to have this single line displayed to the user on ssh logon, using motd (i got this part covered)
netstat -tnlp|awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'|sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'
try this, Later addition
sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'

Bash: awk removing parentheses from array items

So I'm creating a small network scan automation tool using nmap as part of the backend, when attempting to get hostnames and IP addresses the IP is formatted like this "(192.168.0.1)" currently I'm using this bit of code to grab hostname and IP and print.
devices=($( nmap $net_addr -sn | grep -E '([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})' | awk {' print $5 "\t" $6)'} ))
printf '%-10s %s\n' "${devices[#]}"
This code will print this
UNKNOWN (192.168.0.1)
UNKNOWN (192.168.0.3)
UNKNOWN (192.168.0.5)
UNKNOWN (192.168.0.6)
UNKNOWN (192.168.0.16)
UNKNOWN (192.168.0.18)
UNKNOWN (192.168.0.19)
UNKNOWN (192.168.0.21)
UNKNOWN (192.168.0.22)
UNKNOWN (192.168.0.23)
UNKNOWN (192.168.0.32)
UNKNOWN (192.168.0.253)
I attempted to use substr($6,2) but this only chops off the first bit and sadly as the nmap IP addresses are variable in length this doesn't help me.
In short, this is the desired output.
UNKNOWN 192.168.0.1
UNKNOWN 192.168.0.3
UNKNOWN 192.168.0.5
UNKNOWN 192.168.0.6
UNKNOWN 192.168.0.16
UNKNOWN 192.168.0.18
UNKNOWN 192.168.0.19
UNKNOWN 192.168.0.21
UNKNOWN 192.168.0.22
UNKNOWN 192.168.0.23
UNKNOWN 192.168.0.32
UNKNOWN 192.168.0.253
here is the output of nmap 192.168.0.0/24 -sn
Starting Nmap 7.40 ( https://nmap.org ) at 2017-12-31 14:42 UTC
Nmap scan report for UNKNOWN (192.168.0.1)
Host is up (0.0014s latency).
Nmap scan report for UNKNOWN (192.168.0.3)
Host is up (0.052s latency).
Nmap scan report for UNKNOWN (192.168.0.5)
Host is up (0.0065s latency).
Nmap scan report for UNKNOWN (192.168.0.6)
Host is up (0.0073s latency).
Nmap scan report for UNKNOWN (192.168.0.7)
Host is up (0.029s latency).
Nmap scan report for UNKNOWN (192.168.0.16)
Host is up (0.011s latency).
Nmap scan report for UNKNOWN (192.168.0.18)
Host is up (0.053s latency).
Nmap scan report for UNKNOWN (192.168.0.21)
Host is up (0.011s latency).
Nmap scan report for UNKNOWN (192.168.0.22)
Host is up (0.012s latency).
Nmap scan report for UNKNOWN (192.168.0.23)
Host is up (0.0061s latency).
Nmap scan report for UNKNOWN (192.168.0.32)
Host is up (0.0013s latency).
Nmap scan report for UNKNOWN (192.168.0.253)
Host is up (0.0011s latency).
Nmap done: 256 IP addresses (12 hosts up) scanned in 3.16 seconds
As per your shown Input you could try following single awk to get the shown expected output.
your_command | awk '/Nmap scan report/{gsub(/\)|\(/,"",$NF);print $(NF-1),$NF}'
EDIT: Eliminating grep from OP's command and trying to solve it within single awk itself now.
devices=$(nmap $net_addr -sn | awk --re-interval '{if(match($0,/\([0-9]{3}\.[0-9]{3}\.[0-9]{1,3}\.[0-9]{1,3}\)/)){gsub(/\)|\(/,"",$NF);print $(NF-1),$NF}}')
Also since my awk is OLD one in case you have latest version of awk then kindly don't use --re-interval in solution.
You can try this sed too
nmap $net_addr -sn | sed '/^Nmap scan report for /!d;s///;s/[)(]//g'

Resources