Formating nmap results to get http server - bash

Question

I am trying to take a nmap scan result, determine the http ports (http, https, http-alt ...) and capture them ip and ports in order to automaticly perform web app scans.
I have my nmap results in grepable format. Using grep to delete any lines that do no contain the string "http". But I am now unsure how I can proceed.
Host: 127.0.0.1 (localhost) Ports: 3390/open/tcp//dsc///, 5901/open/tcp//vnc-1///, 8000/open/tcp//http-alt/// Ignored State: closed (65532)
This is my current result. From this I can get the IP of hosts with a http server open by using the cut command and getting the second field. which is the first part of my problem solved.
But now I am looking for a way to only get (from the above example)
8000/open/tcp//http-alt///
(NB: I'm not looking to get it just for the spefic case, using
cut -f 3 -d "," will work for this case, but if the http server was in the first field it would not work.)
after which i can use the cut command to get the port to then add it to a file with the ip, resulting in
127.0.0.1:8000
Could anyone advise a good way to do this?
Code of my simple bash script for doing a basic scan of all ports,the then doing a more advanced one based on the open ports found. Next step and objecive is to automaticly scan web apps with a directory scan and niktoo scan of identified web apps
#!/bin/bash
echo "Welcome to the quick lil tool. This runs a basic nmap scan, collects open ports and does a more advanced scan. reducing the time needed"
echo -e "\nUsage: ./getPorts.sh [Hosts]\n"
if [ $# -eq 0 ]
then
echo "No argument specified. Usage: ./getPorts.sh [Host or host file]"
exit 1
fi
if [[ "$EUID" -ne 0 ]]; then
echo "Not running as root"
exit 1
fi
nmap -iL $1 -p- -oA results
#Replace input file with gnmap scan, It will generate a list of all open ports
cat results.gnmap |awk -F'[/ ]' '{h=$2; for(i=1;i<=NF;i++){if($i=="open"){print h,":",$(i-1)}}}'| awk -F ':' '{print $2}' | sed -z 's/\n/,/g;s/,$/\n/' >> ports.list
#more advanced nmap scan
ports=$(cat ports.list)
echo $ports
nmap -p $ports -sC -sV -iL $1
EDIT: Found a way. Not sure why I was so focused on using the gnmap format for this, If I use the regular .nmap format. I can simple grep the line with http in and use cut to get the first field.
(cat results.nmap | grep 'http' | cut -d "/" -f 1)
EDIT2: I realised the method mentioned in my first edit is not optimal when processing multiple results as I then have a list of IP's from the .nmap, and a list of ports from the .gnmap. I have found a good solution to my problem using a single file. see below:
#!/bin/bash
httpalt=$(cat test.gnmap | awk '/\/http-alt\// {for(i=5;i<=NF;i++)if($i~"/open/.+/http-alt/"){sub("/.*","",$i); print "http://"$2":"$i}}')
if [ -z "$httpalt" ]
then
echo "No http-alt servers found"
else
echo "http-alt servers found"
echo $httpalt
printf "\n"
fi
http=$(cat test.gnmap | awk '/\/http\// {for(i=5;i<=NF;i++)if($i~"/open/.+/http/"){sub("/.*","",$i);print "http://"$2":"$i}}')
if [ -z "$http" ]
then
echo "No http servers found"
else
echo "http servers found"
echo $http
printf "\n"
fi
https=$(cat test.gnmap | awk '/\/https\// {for(i=5;i<=NF;i++)if($i~"/open/.+/https/"){sub("/.*","",$i); print "https://"$2":"$i}}')
if [ -z "$https" ]
then
echo "No http servers found"
else
echo "https servers found"
echo $https
printf "\n"
fi
echo ----
printf "All ip:webapps \n"
webserver=$(echo "$httpalt $http $https" | sed -e 's/\s\+/,/g'|sed -z 's/\n/,/g;s/,$/\n/')
if [[ ${webserver::1} == "," ]]
then
webserver="${webserver#?}"
else
echo 0; fi
for webservers in $webserver; do
echo $webservers
done
echo $https
https=$(echo "$https" | sed -e 's/\s\+/,/g'|sed -z 's/\n/,/g;s/,$/\n/')
echo $https
mkdir https
mkdir ./https/nikto/
mkdir ./https/dirb/
for onehttps in ${https//,/ }
do
echo "Performing Dirb and nikto for https"
dirb $onehttps > ./https/dirb/https_dirb
nikto -url $onehttps > ./https/nikto/https_nitko
done
mkdir http
mkdir ./http/nikto
mkdir ./http/dirb/
for onehttp in ${http//,/ }
do
echo $onehttp
echo "Performing Dirb for http"
dirb $onehttp >> ./http/dirb/http_dirb
nikto -url $onehttp >> ./http/nikto/http_nikto
done
mkdir httpalt
mkdir httpalt/nikto/
mkdir httpalt/dirb/
for onehttpalt in ${httpalt//,/ }
do
echo "Performing Dirb for http-alt"
dirb $onehttpalt >> ./httpalt/dirb/httpalt_dirb
nikto -url $onehttpalt >> ./httpalt/nikto/httpalt_nikto
done
This will check for any http, https, and http-alt servers, store them in a variable, check for duplicates and remove any trailing commas at the begining, It is far from perfect, but is a good solution for now!

Answer 1

Just want to share a brilliant open source tool on GitHub that can be used to easily parse NMAP XML files.
https://github.com/honze-net/nmap-query-xml
I use some of the python code to extract http/https URLs from the nmap xml file.
# pip3 install python-libnmap
from libnmap.parser import NmapParser
def extract_http_urls_from_nmap_xml(file):
    try:
        report = NmapParser.parse_fromfile(file)
        urls = []
    except IOError:
        print("Error: Nmap XML file %s not found. Quitting!" % file)
        sys.exit(1)
    for host in report.hosts:
        for service in host.services:
            filtered_services = "http,http-alt,http-mgmt,http-proxy,http-rpc-epmap,https,https-alt,https-wmap,http-wmap,httpx"
            if (service.state == "open") and (service.service in filtered_services.split(",")):
                line = "{service}{s}://{hostname}:{port}"
                line = line.replace("{xmlfile}", nmap_file)
                line = line.replace("{hostname}", host.address if not host.hostnames else host.hostnames[0]) # TODO: Fix naive code.
                line = line.replace("{hostnames}", host.address if not host.hostnames else ", ".join(list(set(host.hostnames)))) # TODO: Fix naive code.
                line = line.replace("{ip}", host.address)
                line = line.replace("{service}", service.service)
                line = line.replace("{s}", "s" if service.tunnel == "ssl" else "")
                line = line.replace("{protocol}", service.protocol)
                line = line.replace("{port}", str(service.port))
                line = line.replace("{state}", str(service.state))
                line = line.replace("-alt", "")
                line = line.replace("-mgmt", "")
                line = line.replace("-proxy", "")
                line = line.replace("-rpc-epmap", "")
                line = line.replace("-wmap", "")
                line = line.replace("httpx", "http")
                urls.append(line)
    return list(dict.fromkeys(urls))

Answer 2

printf "Host: 127.0.0.1 (localhost) Ports: 3390/open/tcp//dsc///, 5901/open/tcp//vnc-1///, 8000/open/tcp//http-alt/// Ignored State: closed (65532)" > file
cat file | tr -s ' ' | tr ',' '\n' | sed s'#^ ##g' > f2
string=$(sed -n '3p' f2 | cut -d' ' -f1)
It is only horizontal search which is difficult; vertical is easy. You can get any string out of any text you like, as long as you can get the string on its' own line, and then determine which line you need to print.
You only need complex regular expressions if you are relying exclusively on horizontal search. In almost all cases, as long as your substring is on its' own line, cut can take you the rest of the way.