Print only if field is not empty - bash

I have a text file that I want to pull host and IP info from only if the IP exists in column 4. For example:
cat hostlist.txt
Server One 255.255.255.255 123.123.123.123
Server Two 255.255.255.255
Server Three 255.255.255.255 123.123.123.123
In this case I would only want to see Server One and Three as Server Two has no data in the fourth column.

awk '{if ($4) print $0;}' < hostlist.txt
does the trick. It's functionally equivalent to the earlier solution but is simpler since you only check for existence rather than matching a regex.

If you can live with lines where field 4 has value 0 not being printed, you can simplify to
$ awk '$4' hostlists.txt
Server One 255.255.255.255 123.123.123.123
Server Three 255.255.255.255 123.123.123.123
This is functionally equivalent to {if ($4) print $0;}, but reduced to just a pattern and using the default action of print $0.

awk approach:
awk 'NF>=4 && $4~/^([0-9]{1,3}\.){3}[0-9]{1,3}$/' hostlist.txt
The output:
Server One 255.255.255.255 123.123.123.123
Server Three 255.255.255.255 123.123.123.123
NF>=4 - ensures that a record has at least 4 fields
$4~/^([0-9]{1,3}\.){3}[0-9]{1,3}$/ - ensures that the 4th field contains IPv4 address (in most simple form. Real IPv4 validation requires an additional conditions)

Related

Have script find all lines containing x, prepend these lines to each line that follows unless the line has another x in it, at which point repeat

I have a file called file.txt which contains either an IP or a FQDN, followed by ports that were found to be open. I want my bash script to prepend all ports with their associated IP/FQDN (always above them in the file), replace the forward slash with a whitespace, and then delete the IP/FQDN line that isn't associated with a port.
Breaking it down, I thought:
Read the next line
If the line contains a "." in it (IP or FQDN), prepend it into all following lines, unless:
If the following line has a "." in it (another IP or FQDN), make that line the new one to prepend and repeat the process for all following lines
Replace all "/" with a " " (one single whitespace)
Remove all lines that are not associated with a port (probably easier just to grep for "tcp" and "udp" as that will display all open ports with associated IP/FQDN
To make it easier, I can easily create a tmp file if necessary within the process. I have tried various iterartions of "while" and "if" and nothing seems to work...!
E.g:
cat file.txt
www.thisisawebsite.com:
80/tcp
443/tcp
500/udp
192.168.1.5:
80/tcp
dev.anothersite.co.uk:
22/tcp
443/tcp
5050/udp
21000/tcp
10.10.10.10:
4000/udp
8000/udp
Then, running the bash script, it should become:
www.thisisawebsite.com:80 tcp
www.thisisawebsite.com:443 tcp
www.thisisawebsite.com:500 udp
192.168.1.5:80 tcp
dev.anothersite.co.uk:22 tcp
dev.anothersite.co.uk:433 tcp
dev.anothersite.co.uk:5050 udp
dev.anothersite.co.uk:21000 tcp
10.10.10.10:4000 udp
10.10.10.10:8000 udp
Looks like a job for awk:
awk -F/ '/\./ {d=$0; next} {print d":"$1, $2}'

display open ports grouped by process

given a netstat output, how can i display the selected open ports grouped by process?
what i got so far:
:~# netstat -tnlp | awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'
haproxy 636
haproxy 993
haproxy 993
haproxy 465
haproxy 465
exim4 25
apache2 80
exim4 25
apache2 443
desired output (in one line):
apache2 (80 443), exim4 (25), haproxy (465 636 993)
please note:
i have duplicated lines because they listen on different IPs, but i only need one (sort -u is ok)
if possible, id like to sort by process and then by port
the main goal is to have this single line displayed to the user on ssh logon, using motd (i got this part covered)
netstat -tnlp|awk '/25|80|443|465|636|993/ {proc=split($7,pr,"/"); port=split($4,po,":"); print pr[2], po[port]}'|sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'
try this, Later addition
sort|uniq|awk '{a[$1]=a[$1](" "$2" "$3)}END{for (i in a) printf "%s (%s),",i,a[i]}'

logstash parsing IPV6 address

I am a newbie to logstash / grok patterns.
In my logfile i have a line in this format as below:
::ffff:172.19.7.180 - - [10/Oct/2016:06:40:26 +0000] 1 "GET /authenticator/users HTTP/1.1" 200 7369
When I try to use a simple IP pattern matching %{IP}, using grok constructor, it shows only partial match:
after match: .19.7.180 - - [10/Oct/2016:06:33:58 +0000] 1 "POST /authenticator/searchUsers HTTP/1.1" 200 280
So, only a part of the ip address matched, as the portion 'after match' still shows remaining portion of ip address.
Queries:
1. What is this format of IP address ::ffff:172.19.7.180?
2. How to resolve this issue, to ensure IP address is correctly parsed?
BTW, I am using nodejs middleware morgan logger, which is printing IP address in this format.
Note that the log contains both IPv4 and IPv6 addresses separated by a colon, so the correct pattern you need to use is the following one:
%{IPV6:ipv6}:%{IPV4:ipv4}
Then in your event you'll have two fields:
"ipv6" => "::ffff"
"ipv4" => "172.19.7.180"
This will work until this issue is resolved.
These IP addresses are in the IPv4-Embedded IPv6 Format and the %{IP} doesn't match it. The only way to go is to either use %{DATA} or write your own regex.

Recursive text search whithin lines before the grep match in shell

I need help working with router config backup database. I need to get a list of interfaces that don't have vrf or shutdown in their configuration.
I get the list of all interfaces config passing the config file through awk '/^interface/,/!/'. This gives me the output below:
interface TenGigE0/3/0/0
description
service-policy output QOS
ipv4 mtu 1500
ipv4 address 13.24.15.3 255.255.255.252
carrier-delay up 3000 down 0
load-interval 30
dampening
!
interface TenGigE0/3/0/1
description Link To
!
interface TenGigE0/3/0/1.302
description
vrf 1671
ipv4 address 13.24.14.11 255.255.255.254
encapsulation dot1q 302
Now, i am stuck trying to exclude the interfaces that contain vrf line. What i was trying to do is to grep for vrf, and when there is a match, remove the line that contains the word "interface" above. Unfortunately with no luck. Maybe someone has a more sophisticated solution.
If you have the structured records awk can solve this problem. Given your intermediate file
2$ awk 'BEGIN{RS=ORS="!\n"} !/vrf/' interface
will print the records without "vrf"
interface TenGigE0/3/0/0
description
service-policy output QOS
ipv4 mtu 1500
ipv4 address 13.24.15.3 255.255.255.252
carrier-delay up 3000 down 0
load-interval 30
dampening
!
interface TenGigE0/3/0/1
description Link To
!

How to create network file based on IP address from file similar to /etc/hosts

I have a file which maps IP address to hostnames. Its format is similar to hosts file and contains a list of IP address to hostname mapping.
eg.
10.200.99.1 master1
10.200.99.2 master2
10.200.99.3 master3
10.200.99.4 slave1
10.200.99.5 slave2
10.200.99.6 slave3
...
...
...
I would like to create network file (/etc/sysconfig/network) for all the IP address mentioned in the file.
The format of network file is where the hostname is mentioned based on hosts file.
NETWORKING=yes
HOSTNAME=master1
NOZEROCONF=yes
For every IP Address the network file is created in a directory named by IP Address i.e. network/{IPAddress}.
For example for master1 the path of file should be network/10.200.99.1 and for master2 the path should be network/10.200.99.2.
How can i do so?
Till now I have obtained IPAdress by following command echo $(<hosts) | awk '{print $1}' and Hostname by echo $(<hosts) | awk '{print $2}. But this only prints the contents of first line of the hosts file.
Reads ip and hostname from hosts and writes desired output to network/$ip.
while read ip hostname; do
printf '%s\n' "NETWORKING=yes" "HOSTNAME=$hostname" "NOZEROCONF NF=yes" > network/$ip
done < hosts

Resources