I have a file called file.txt which contains either an IP or a FQDN, followed by ports that were found to be open. I want my bash script to prepend all ports with their associated IP/FQDN (always above them in the file), replace the forward slash with a whitespace, and then delete the IP/FQDN line that isn't associated with a port.
Breaking it down, I thought:
Read the next line
If the line contains a "." in it (IP or FQDN), prepend it into all following lines, unless:
If the following line has a "." in it (another IP or FQDN), make that line the new one to prepend and repeat the process for all following lines
Replace all "/" with a " " (one single whitespace)
Remove all lines that are not associated with a port (probably easier just to grep for "tcp" and "udp" as that will display all open ports with associated IP/FQDN
To make it easier, I can easily create a tmp file if necessary within the process. I have tried various iterartions of "while" and "if" and nothing seems to work...!
E.g:
cat file.txt
www.thisisawebsite.com:
80/tcp
443/tcp
500/udp
192.168.1.5:
80/tcp
dev.anothersite.co.uk:
22/tcp
443/tcp
5050/udp
21000/tcp
10.10.10.10:
4000/udp
8000/udp
Then, running the bash script, it should become:
www.thisisawebsite.com:80 tcp
www.thisisawebsite.com:443 tcp
www.thisisawebsite.com:500 udp
192.168.1.5:80 tcp
dev.anothersite.co.uk:22 tcp
dev.anothersite.co.uk:433 tcp
dev.anothersite.co.uk:5050 udp
dev.anothersite.co.uk:21000 tcp
10.10.10.10:4000 udp
10.10.10.10:8000 udp
Looks like a job for awk:
awk -F/ '/\./ {d=$0; next} {print d":"$1, $2}'
Related
given a netstat output, how can i display the selected open ports grouped by process?
what i got so far:
:~# netstat -tnlp | awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'
haproxy 636
haproxy 993
haproxy 993
haproxy 465
haproxy 465
exim4 25
apache2 80
exim4 25
apache2 443
desired output (in one line):
apache2 (80 443), exim4 (25), haproxy (465 636 993)
please note:
i have duplicated lines because they listen on different IPs, but i only need one (sort -u is ok)
if possible, id like to sort by process and then by port
the main goal is to have this single line displayed to the user on ssh logon, using motd (i got this part covered)
netstat -tnlp|awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'|sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'
try this, Later addition
sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'
I'm going to try to describe my problem and my end goal as best as I can, here it goes:
I have a script that fetches AWS ELB information (elb name + ports that's associated with a specific certifican arn).
So, in the end I have a text file (I call it elb_ports) and it looks something like this:
ccds-lb 636
cf-router 443 4443
dev-cf-router 443 4443
eng-jenkins-monit 443
gitlab-lb 443
gitlab-mattermost-elb 443
jenkins-np-elb 443
saml 443
uaa 443
I have another script that comes after that which I want it to go through that elb_ports file and replace the certificates with a new one, but according to Amazon's documentation: It says in order to replace the certificates, I need two things from that elb_ports file. The load balancer name and the load balancer port.
So basically their command looks like this
aws elb set-load-balancer-listener-ssl-certificate \
--load-balancer-name my-load-balancer \
--load-balancer-port 443 \
--ssl-certificate-id arn:aws:iam::123456789012:server-certificate/my-new-certificate
I want to be able to loop through the file and execute the command above to each elb and port, but my problem is with the elbs that has multiple ports associated with the cert like: cf-router 443 4443 for example.
So my idea was to split that into two lines, so like this:
cf-router 443
cf-router 4443
But I'm not sure how to add cf-router (for example) to the ports that come after the first one (there could be more than two ports using the same cert).
I hope I was able to explain my problem and end goal clearly, if this isn't a good method, I'm open to suggestions also.
EDIT: Perhaps something like this is beneficial, but not sure how to tailor it to my needs.. Like put each line in an array and the space as a delimiter and then loop through each line putting arr(1) (load balancer name) and then the load balancer port, but not sure how to count and go through >arr(2) in bash.
To split out your extra columns into separate lines:
while read -r lb_name lb_ports_str; do ## split line into lb name and port list string
read -r -a lb_ports <<<"$lb_ports_str" ## split out port list string into an array
for port in "${lb_ports[#]}"; do ## iterate through that array
printf '%s %s\n' "$lb_name" "$port" ## handle each port separately
done
done <elb_ports ## reading lines from elb_ports
Of course, that printf could be any other line referring to $lb_name and $port -- meaning you could potentially run your code that's installing new certificates here.
I have ssh config file which contains so many servers from google,aws and some remote servers.I wanted to have a bash function that will output only the Host and the HostName of the server so I dont have to remember there public DNS to check my webapps.
Sample Server Config in my ssh config looks like this
Host aws_server
User rand
HostName 65.2.25.152
Port 8000
IdentityFile PEM PATH
ServerAliveInterval 120
ServerAliveCountMax 30
I want the output like
aws_server 65.2.25.152
for all the servers
Using sed
sed '/^Host/{s/[^ ]* //;:1;N;s/\n.*HostName */\t/;t2;b1;:2;p};d' file
aws_server 65.2.25.152
Modified version that's more robust for multiple hosts,missing HostNames
sed ':1;s/\(.*\n\|^\)Host *//;N;s/\n.*HostName */\t/;t2;$!{b1;:2;p};d' file
using awk
awk '{if($1=="Host")k=$2;if($1=="HostName")printf("%s\t%s\n",k,$2)}' file
I would use awk for this:
awk '
# Save the host when we see it.
/^Host/ {
host=$2
next
}
# If we have a host and are on a HostName line
host && $1 == "HostName" {
# Print the Host and HostName values
printf "%s"OFS"%s\n", host, $2
host=""
next
}
# If we have a HostName without a Host clear the host (should not happen but just to be safe)
$1 == "HostName" {
host=""
}
' .ssh/config
I have made a text file of nmap's output and i was trying to find only those ip whose port are open
and when i am using grep i am not getting the desired output only one of the item is i am able to get either ip or the text open
data:
Nmap scan report for xxx.xxx.xxx.83
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.87
Host is up (0.049s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.89
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.90
Host is up (0.050s latency).
PORT STATE SERVICE
80/tcp closed http
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
Nmap scan report for xxx.xxx.xxx.96
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp filtered http
Nmap scan report for xxx.xxx.xxx.100
Host is up (0.054s latency).
PORT STATE SERVICE
80/tcp filtered http
You could try the below awk command,
$ awk -v RS="" '/ open /{print $5}' file
xxx.xxx.xxx.93
It prints the ip (column no 5) only if the certain block contains the text open
After awk and grep perl too:
perl -00 -lanE 'say $_ if m/open/' < file
prints:
Nmap scan report for xxx.xxx.xxx.93
Host is up (0.051s latency).
PORT STATE SERVICE
80/tcp open http
or
perl -00 -lanE 'say $F[4] if m/open/' < file
prints
xxx.xxx.xxx.93
If there are always 4 lines per block and the word open is in the last line, you can do:
grep -B4 open file
and it will show the 4 lines before the word open.
What about using awk? This will report the IP address the first time an open port is encountered in the nmap output:
sh$ awk '$3=="report"{ IP = $5 } $2=="open"&&IP { print IP; IP="" }' nmap.out
xxx.xxx.xxx.93
Have a text file w/ around 3 million URL's of sites I want to block.
Trying to ping them one by one (yes, I know it is going to take some time).
Have a script (yes, I am a bit slow in BASH) which reads the lines one at a time from text file.
Obviously cannot print text file here. Text file was created >> w/ Python some time ago.
Problem is that ping returns "unknown host" w/ every entry. If I make a smaller file by hand using the same entries the script works. I thought it may be a white space or end of line issue so tried addressing that in script. What could the issue possibly be?
#!/bin/bash
while read line
do
li=$(echo $line|tr -d '\n')
li2=$(echo $li|tr -d ' ')
if [ ${#line} -lt 2 ]
then
continue
fi
ping -c 2 -- $li2>>/dev/null
if [ $? -gt 0 ]
then
echo 'bad'
else
echo 'good'
fi
done<'temp_file.txt'
Does the file contains URLs or hostnames ?
If it contains URLs you must extract the hostname from URLs before pinging:
hostname=$(echo "$li2"|cut -d/ -f3);
ping -c 2 -- "$hostname"
Ping is used to ping hosts. If you have URLs of websites, then it will not work. Check that you have hosts in your file , example www.google.com or an IP address and not actual full website urls. If you want to check actual URLs, use a tool like wget and another tool like grep/awk to grab for errors like 404 or others. Last but not least, people who are security conscious will sometimes block pinging from the outside, so take note.
C heck if the file contains windows-style \r\n line endings: head file | od -c
If so, to fix it: dos2unix filename filename
I wouldn't use ping for this. It can easily be blocked, and it's not the best way to check for either ip addresses or if a server presents web pages.
If you just want to find the corresponding IP, use host:
$ host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 209.85.149.106
www.l.google.com has address 209.85.149.147
www.l.google.com has address 209.85.149.99
www.l.google.com has address 209.85.149.103
www.l.google.com has address 209.85.149.104
www.l.google.com has address 209.85.149.105
As you see, you get all the IPs registered to a host. (Note that this requires you to parse the hostname from your urls!)
If you want to see if a URL points at a web server, use wget:
wget --spider $url
The --spider flag makes wget not save the page, just check that it exists. You could look at the return code, or add the -S flag (which prints the HTTP headers returned)