I am trying to take a nmap scan result, determine the http ports (http, https, http-alt ...) and capture them ip and ports in order to automaticly perform web app scans.
I have my nmap results in grepable format. Using grep to delete any lines that do no contain the string "http". But I am now unsure how I can proceed.
Host: 127.0.0.1 (localhost) Ports: 3390/open/tcp//dsc///, 5901/open/tcp//vnc-1///, 8000/open/tcp//http-alt/// Ignored State: closed (65532)
This is my current result. From this I can get the IP of hosts with a http server open by using the cut command and getting the second field. which is the first part of my problem solved.
But now I am looking for a way to only get (from the above example)
8000/open/tcp//http-alt///
(NB: I'm not looking to get it just for the spefic case, using
cut -f 3 -d "," will work for this case, but if the http server was in the first field it would not work.)
after which i can use the cut command to get the port to then add it to a file with the ip, resulting in
127.0.0.1:8000
Could anyone advise a good way to do this?
Code of my simple bash script for doing a basic scan of all ports,the then doing a more advanced one based on the open ports found. Next step and objecive is to automaticly scan web apps with a directory scan and niktoo scan of identified web apps
#!/bin/bash
echo "Welcome to the quick lil tool. This runs a basic nmap scan, collects open ports and does a more advanced scan. reducing the time needed"
echo -e "\nUsage: ./getPorts.sh [Hosts]\n"
if [ $# -eq 0 ]
then
echo "No argument specified. Usage: ./getPorts.sh [Host or host file]"
exit 1
fi
if [[ "$EUID" -ne 0 ]]; then
echo "Not running as root"
exit 1
fi
nmap -iL $1 -p- -oA results
#Replace input file with gnmap scan, It will generate a list of all open ports
cat results.gnmap |awk -F'[/ ]' '{h=$2; for(i=1;i<=NF;i++){if($i=="open"){print h,":",$(i-1)}}}'| awk -F ':' '{print $2}' | sed -z 's/\n/,/g;s/,$/\n/' >> ports.list
#more advanced nmap scan
ports=$(cat ports.list)
echo $ports
nmap -p $ports -sC -sV -iL $1
EDIT: Found a way. Not sure why I was so focused on using the gnmap format for this, If I use the regular .nmap format. I can simple grep the line with http in and use cut to get the first field.
(cat results.nmap | grep 'http' | cut -d "/" -f 1)
EDIT2: I realised the method mentioned in my first edit is not optimal when processing multiple results as I then have a list of IP's from the .nmap, and a list of ports from the .gnmap. I have found a good solution to my problem using a single file. see below:
#!/bin/bash
httpalt=$(cat test.gnmap | awk '/\/http-alt\// {for(i=5;i<=NF;i++)if($i~"/open/.+/http-alt/"){sub("/.*","",$i); print "http://"$2":"$i}}')
if [ -z "$httpalt" ]
then
echo "No http-alt servers found"
else
echo "http-alt servers found"
echo $httpalt
printf "\n"
fi
http=$(cat test.gnmap | awk '/\/http\// {for(i=5;i<=NF;i++)if($i~"/open/.+/http/"){sub("/.*","",$i);print "http://"$2":"$i}}')
if [ -z "$http" ]
then
echo "No http servers found"
else
echo "http servers found"
echo $http
printf "\n"
fi
https=$(cat test.gnmap | awk '/\/https\// {for(i=5;i<=NF;i++)if($i~"/open/.+/https/"){sub("/.*","",$i); print "https://"$2":"$i}}')
if [ -z "$https" ]
then
echo "No http servers found"
else
echo "https servers found"
echo $https
printf "\n"
fi
echo ----
printf "All ip:webapps \n"
webserver=$(echo "$httpalt $http $https" | sed -e 's/\s\+/,/g'|sed -z 's/\n/,/g;s/,$/\n/')
if [[ ${webserver::1} == "," ]]
then
webserver="${webserver#?}"
else
echo 0; fi
for webservers in $webserver; do
echo $webservers
done
echo $https
https=$(echo "$https" | sed -e 's/\s\+/,/g'|sed -z 's/\n/,/g;s/,$/\n/')
echo $https
mkdir https
mkdir ./https/nikto/
mkdir ./https/dirb/
for onehttps in ${https//,/ }
do
echo "Performing Dirb and nikto for https"
dirb $onehttps > ./https/dirb/https_dirb
nikto -url $onehttps > ./https/nikto/https_nitko
done
mkdir http
mkdir ./http/nikto
mkdir ./http/dirb/
for onehttp in ${http//,/ }
do
echo $onehttp
echo "Performing Dirb for http"
dirb $onehttp >> ./http/dirb/http_dirb
nikto -url $onehttp >> ./http/nikto/http_nikto
done
mkdir httpalt
mkdir httpalt/nikto/
mkdir httpalt/dirb/
for onehttpalt in ${httpalt//,/ }
do
echo "Performing Dirb for http-alt"
dirb $onehttpalt >> ./httpalt/dirb/httpalt_dirb
nikto -url $onehttpalt >> ./httpalt/nikto/httpalt_nikto
done
This will check for any http, https, and http-alt servers, store them in a variable, check for duplicates and remove any trailing commas at the begining, It is far from perfect, but is a good solution for now!
Just want to share a brilliant open source tool on GitHub that can be used to easily parse NMAP XML files.
https://github.com/honze-net/nmap-query-xml
I use some of the python code to extract http/https URLs from the nmap xml file.
# pip3 install python-libnmap
from libnmap.parser import NmapParser
def extract_http_urls_from_nmap_xml(file):
try:
report = NmapParser.parse_fromfile(file)
urls = []
except IOError:
print("Error: Nmap XML file %s not found. Quitting!" % file)
sys.exit(1)
for host in report.hosts:
for service in host.services:
filtered_services = "http,http-alt,http-mgmt,http-proxy,http-rpc-epmap,https,https-alt,https-wmap,http-wmap,httpx"
if (service.state == "open") and (service.service in filtered_services.split(",")):
line = "{service}{s}://{hostname}:{port}"
line = line.replace("{xmlfile}", nmap_file)
line = line.replace("{hostname}", host.address if not host.hostnames else host.hostnames[0]) # TODO: Fix naive code.
line = line.replace("{hostnames}", host.address if not host.hostnames else ", ".join(list(set(host.hostnames)))) # TODO: Fix naive code.
line = line.replace("{ip}", host.address)
line = line.replace("{service}", service.service)
line = line.replace("{s}", "s" if service.tunnel == "ssl" else "")
line = line.replace("{protocol}", service.protocol)
line = line.replace("{port}", str(service.port))
line = line.replace("{state}", str(service.state))
line = line.replace("-alt", "")
line = line.replace("-mgmt", "")
line = line.replace("-proxy", "")
line = line.replace("-rpc-epmap", "")
line = line.replace("-wmap", "")
line = line.replace("httpx", "http")
urls.append(line)
return list(dict.fromkeys(urls))
printf "Host: 127.0.0.1 (localhost) Ports: 3390/open/tcp//dsc///, 5901/open/tcp//vnc-1///, 8000/open/tcp//http-alt/// Ignored State: closed (65532)" > file
cat file | tr -s ' ' | tr ',' '\n' | sed s'#^ ##g' > f2
string=$(sed -n '3p' f2 | cut -d' ' -f1)
It is only horizontal search which is difficult; vertical is easy. You can get any string out of any text you like, as long as you can get the string on its' own line, and then determine which line you need to print.
You only need complex regular expressions if you are relying exclusively on horizontal search. In almost all cases, as long as your substring is on its' own line, cut can take you the rest of the way.
I am trying to write a bash script where i will give an IP as an argument. If the IP is not valid I want to print to screen a message.If the IP is valid I print the hostname.
host $1 | cut -d' ' -f5
This is my code if the IP is valid. I don't know hot to compare the output if it's not valid to print the corresponding message
You can do it this way:
if host=$(host "$1" 2>/dev/null); then # ignore stderr
echo "$host"
else
echo "$1 is not a valid host"
fi
If you want to display the error output of host in the else part, then:
if host=$(host "$1" 2>&1); then # merge stdout and stderr
echo "$host"
else
echo "$1 is not a valid host, error=$host"
fi
I am currently writing the following script that logs into a remote server and runs couple of commands to verify the performance of the server and prints a message based on the output of those commands .But the ssh doesn't work and returns the stats of the server that hosts the script instead .
Script
#!/bin/bash
#######################
#Function to add hosts to the array
#the following function takes the ip addresses provided while the script is run and stores them in an array
#######################
Host_storing_func () {
HOST_array=()
for i in $# ;do
HOST_array+=(${i});
done
#echo ${HOST_array[*]}
}
#######################
#Calling above function
#######################
Host_storing_func "$#"
############################################################
#Collect Stats of Ping,memory,iowait time test function
############################################################
b=`expr ${#HOST_array[*]} - 1 `
for i in `seq 0 $b` ;do
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh student35#${HOST_array[${i}]} << HERE
echo `hostname`
iowaittm=`sar 2 2|awk '/^Average/{print $5};'`
if [ $iowaittm > 10 ];then
echo "IO ==> BAD"
else
echo "IO ==> GOOD"
fi
memoryy=`free -m |grep Swap|awk '{if($2 == 0) print 0;else print (($4 / $2 ) * 100)}'`
if [ ${memoryy} < '10' ] ;then
echo "memory ==> good"
elif [[ "${memory}" -ge 0 ]] && [[ "${memory}" -le 10 ]];then
echo "No Swap"
else
echo "memory ==> bad"`enter code here`
fi
ping -w2 -c2 `hostname` | grep "packet loss"|awk -F, '{print $3}'|awk -F% '{print $1}'|sed 's/^ *//'|awk '{if ($1 == 0) print "Yes" ;else print "No"}'
HERE
done
Output : oc5610517603.XXX.com is the name of the source server
[root#oc5610517603 scripts]# ./big_exercise.sh 9.XXX.XXX.XXX 9.XXX.XXX.XXX
Pseudo-terminal will not be allocated because stdin is not a terminal.
oc5610517603.XXX.com
IO ==> GOOD
No Swap
ping: oc5610517603.ibm.com: Name or service not known
Pseudo-terminal will not be allocated because stdin is not a terminal.
oc5610517603.XXX.com
IO ==> GOOD
No Swap
ping: oc5610517603.XXX.com: Name or service not known
thanks for checking the script , I figured out a way to solve the problem
It is the sshpass command that is causing issue , you just have to put the opening HERE in single quotes if you want to use variables with in the HEREdoc but if the variables are calculated before ssh then you don't have to put opening HERE in single quotes
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh -T student35#${i} << 'HERE'
after I changed the sshpass command as above my script worked
I have modified your script a bit.
As suggested by #chepner, I am not using the Host_storing_func.
Heredocs for sshpaas are somewhat tricky. You have to escape every back-tick and $ sign in the heredoc.
Notice the - before the heredoc start, it allows you to indent the heredoc body. Also, try to avoid back-ticks when you can. use $(command) instead.
Hope it helps.
#!/bin/bash
#######################
#Function to add hosts to the array
#the following function takes the ip addresses provided while the script is run and stores them in an array
#######################
array=( "$#" )
user="student35"
############################################################
#Collect Stats of Ping,memory,iowait time test function
############################################################
for host in ${array[#]}; do
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh -l ${user} ${host} <<-HERE
thishost=\$(hostname)
echo "Current Host -> \$thishost";
iowaittm=\`sar 2 2|awk '/^Average/{print \$5}'\`
if [ \$iowaittm > 10 ]; then
echo "IO ==> BAD"
else
echo "IO ==> GOOD"
fi
memory=\$(free -m | grep Swap | awk '{if(\$2 == 0) print 0;else print ((\$4 / \$2 ) * 100)}')
if [ \${memory} < '10' ] ;then
echo "memory ==> good"
elif [[ "\${memory}" -ge 0 ]] && [[ "\${memory}" -le 10 ]]; then
echo "No Swap"
else
echo "memory ==> bad"\`enter code here\`
fi
ping -w2 -c2 \`hostname\` | grep "packet loss"|awk -F, '{print \$3}'|awk -F% '{print \$1}'|sed 's/^ *//'|awk '{if (\$1 == 0) print "Yes" ;else print "No"}'
HERE
done
I want to use fping to ping multiple ips contained in a file and output the failed ips into a file i.e.
hosts.txt
8.8.8.8
8.8.4.4
1.1.1.1
ping.sh
#!/bin/bash
HOSTS="/tmp/hosts.txt"
fping -q -c 2 < $HOSTS
if ip down
echo ip > /tmp/down.log
fi
So I would like to end up with 1.1.1.1 in the down.log file
It seems that parsing the data from fping is somewhat difficult. It allows the parsing of data for hosts that is alive but not dead. As a way round the issue and to allow for multiple host processing simultaneously with -f, all the hosts that are alive are placed in a variable called alive and then the hosts in the /tmp/hosts.txt file are looped through and grepped against the variable alive to decipher whether the host is alive or dead. A return code of 1 signifies that grep cannot find the host in alive and hence an addition to down.log.
alive=$(fping -c 1 -f ipsfile | awk -F: '{ print $1 }')
while read line
do
grep -q -o $line <<<$alive
if [[ "$?" == "1" ]]
then
echo $line >> down.log
fi
done < /tmp/hosts.txt
Here's one way to get the result you want. Note however; i didn't use fping anywhere in my script. If the usage of fping is crucial to you then i might have missed the point entirely.
#!/bin/bash
HOSTS="/tmp/hosts.txt"
declare -i DELAY=$1 # Amount of time in seconds to wait for a packet
declare -i REPEAT=$2 # Amount of times to retry pinging upon failure
# Read HOSTS line by line
while read -r line; do
c=0
while [[ $c < $REPEAT ]]; do
# If pinging an address does not return the word "0 received", we assume the ping has succeeded
if [[ -z $(ping -q -c $REPEAT -W $DELAY $line | grep "0 received") ]]; then
echo "Attempt[$(( c + 1))] $line : Success"
break;
fi
echo "Attempt[$(( c + 1))] $line : Failed"
(( c++ ))
done
# If we failed the pinging of an address equal to the REPEAT count, we assume address is down
if [[ $c == $REPEAT ]]; then
echo "$line : Failed" >> /tmp/down.log # Log the failed address
fi
done < $HOSTS
Usage: ./script [delay] [repeatCount] -- 'delay' is the total amount of seconds we wait for a response from a ping, 'repeatCount' is how many times we retry pinging upon failure before deciding the address is down.
Here we are reading the /tmp/hosts.txt line by line and evaluating each adress using ping. If pinging an address succeeds, we move on to the next one. If an address fails, we try again for as many times as the user has specified. If the address fails all of the pings, we log it in our /tmp/down.log.
The conditions for checking whether a ping failed/succeeded may not be accurate for your use-cases, so maybe you will have to edit that. Still, i hope this gets the general idea across.
I'm new to shell programming and I have created a script that opens a connection to a server of mine. I want to have this script listen for an input from a client node and use that to run a function.
This is my process.
Run script > opens listener > on second computer use netcat to connect > run a function in the script on the server called nodefunction
I have server_port coded to '4444'
File name: run_hangman
nc -l -k -v -p 4444 | bash hangman
File name: hangman
#!/bin/bash
msg_timeout=0
host_ows=1
server_port=4444
dubOws=xxx.xxx.xxx.xxx
initServer() {
hostIP=`ip -o addr show dev "eth0" | awk '$3 == "inet" {print $4}' | sed -r 's!/.*!!; s!.*\.!!'`
hostOws=`echo $hostIP | cut -d . -f 4`
}
servermsg(){ #message //if = --n, echos on same line
if [ "$1" != "--n" ]
then
echo `date +"%T"` "[SERVER] "$1
else
echo -n `date +"%T"` "[SERVER] "
fi
}
owsmsg(){ #message //if = --n, echos on same line
if [ "$1" != "--n" ]
then
echo `date +"%T"` "[OWS] "$1
else
echo -n `date +"%T"` "[OWS] "
fi
}
playermsg() {
if [ "$1" != "--n" ]
then
echo `date +"%T"` "[PLAYER] "$1
else
echo -n `date +"%T"` "[PLAYER] "
fi
}
question(){ #question, read, example
servermsg "$1"
if [ -n "$3" ]
then
servermsg "$3"
fi
read $2
echo ""
}
owsArray(){ #
for targetOws in $player_list
do
owsArray+=("OWS"$targetOws)
done
echo -n ${owsArray[*]}
echo
}
openSocket() {
servermsg "Starting the Game Listener"
servermsg "Opening Listener on port "$server_port
#nc -k -l $server_port |bash
#nc -kl -q 1 -p $server_port # This should create the listener & This is where everything stops.
servermsg "Now listening on port "$server_port
}
initServer
owsmsg "Starting server on OWS"$hostOws"..."
question "Enter all the OWSs that will play:" player_list "Example: 1 9 14 23"
echo $player_list
question "Type a category hint:" game_cat "Example: Type of Animal"
question "Type your word:" game_word "Example: zebra"
question "How many guesses:" game_guesses "Example: 7"
servermsg "OWS"$host_ows "has created a Hangman session"
servermsg "Players are:"; servermsg --n; owsArray
servermsg "Your word is "${#game_word}" letters long and players have "$game_guesses" guesses"
question "If this is all correct press enter, or CTRL+C to cancel"
openSocket
# I think I need a While script here to read the RAW input and run the playermsg function with the input?
I run the run_hangman file and then I connect to it via my node computer. I enter the following line and echo "1 2 3" because that is what I need. I also can't enter "1 2 3" directly into the window running "run_hangman" as if I press enter it goes to a new line.
echo "1 2 3" >/dev/tcp/xxx.xxx.xxx.xxx/4444
The server shows that it connected
Listening on [0.0.0.0] (family 0, port 4444)
14:52:24 [OWS] Starting server on OWS225...
14:52:24 [SERVER] Enter all the OWSs that will play:
14:52:24 [SERVER] Example: 1 9 14 23
Connection from [xxx.xxx.xxx.xxx] port 4444 [tcp/*] accepted (family 2, sport 41564)
Connection closed, listening again.
1 2 3
Now once it gets to openSocket it will allow me to send one more echo and then it closes on the server. I need to get what I presume is a while statement and have it listen for an input like "playermsg 'has started a game'" and have it actually run that function on the server.
Will I be able to get this to run, almost seems like it has to be in the background? I've been using nc(1) for reference and some websites said to try -d and that didn't work either.
I got it figured out. I did indeed need a while statement.
openSocket
while read -r value; do
val=${value:10}
if [[ "$value" == playermsg* ]]; then
val=${value:10}
playermsg $val
elif [[ "$value" == servermsg* ]]; then
val=${value:10}
servermsg $val
else
echo "Returned "$value
echo "Value was "$val
fi
done
So now on the second computer I simply run
echo "playermsg testing" >/dev/tcp/xxx.xxx.xxx.xxx/4444
Then the server displays the following:
15:29:36 [SERVER] Starting the Game Listener
15:29:36 [SERVER] Opening Listener on port 4440
15:29:36 [SERVER] Now listening on port 4440
15:29:37 [PLAYER] testing