Counting and sorting multiple values in textfile columns - sorting

Question

I want to sift through text files looking for suspicious activity. I'm a little familiar with bash scripting, including grep, sed and awk. My research on stackoverflow.com, tldp.org, etc, and talk with colleagues shows that perl and python are best suited for my task, but I have zero experience with those scripting languages.
Inputs with various scripting, compiled or interpreted languages are welcome. Due to my limitations, please add comments to the code, enabling me to quickly understand and learn the language.
Ok, the task is sorting and counting items sorted in columns. I /can/ accomplish part of this this using grep, awk and sed. Unfortunately, the recursive aspects (as I perceive the problem) has me stumped.
The input text is sorted, two columns of ip addresses (simplified in my example, below) and a column for a destination port (of all possible values). The size of this file may be several megabytes, but probably never more than 250MB at at time, so absolute efficiency isn't necessary. Simplicity is.
SIP DIP DPt
111.100 200.150 80
111.100 200.150 443
111.100 200.155 22
111.100 200.155 80
111.100 200.155 443
111.100 200.160 80
111.100 200.165 139
111.100 200.165 443
111.100 200.165 512
115.102 225.150 80
115.102 225.150 137
115.102 225.150 443
120.125 250.175 23
120.135 250.145 23
125.155 250.165 80
125.155 250.165 139
125.155 250.175 1023
The code I have working (drafting this from memory ... not currently at my linux box) is similar to this ...
#!/bin/bash
declare -i counter=0
SIP=null # current source ip.
SIP_last=null # for last ip address processed.
SIP_next=null # not found a use for this, yet.
# sorting usually reqs three vars, so here it is.
for SIP in `zcat textfile.gz | awk '{ if ($3 <1024) print $1,$2,$3}'` do
# Ensure I count the first item. This was problematic at first.
if [[ "$SIP_last" == null ]] then
SIP_last=$SIP
counter=counter+1 # counter=+ didn't work reliably.
# Do something useful. As shown, it works.
if [[ "$SIP" == "$SIP_last" ]] then
counter=counter+1
if [[ "$SIP != "$SIP_last" ]] then
echo SIP: $SIP_last Counter: $counter # DIP code has not yet been added.
SIP_last=$SIP
# Ensure I always catch the last item. Still working on this issue.
# XXX
done
Using the input provided above, the output should look something like this ...
SIP DIP Ct Ports
> 2 < 1024
111.100 200.150 80, 443
111.100 200.155 20, 80, 443
111.100 200.165 139, 443, 512
115.102 225.150 80, 137, 443
Looking at the output you can see the crux of the matter is only reporting DIP counts > 2 and Ports < 1024. Limiting the ports to < 1024 is simple enough using the provided awk statement. It's matching up the DIPs to the SIPs and keeping a running tally of the DPts that's the kicker.
Again, this is from memory, so forgive the coding errors. Thanks for your assistance.
Allen.

Answer 1

With your posted sample input file:
$ awk '
NR==1 { print; next }
$3 < 1024 {
key = $1 "\t" $2
if (!seen[key,$3]++) {
cnt[key]++
vals[key] = vals[key] sep[key] $3
sep[key] = ", "
}
}
END { for (key in cnt) if (cnt[key] > 1) print key "\t" vals[key] }
' file
SIP DIP DPt
111.100 200.155 22, 80, 443
111.100 200.165 139, 443, 512
125.155 250.165 80, 139
115.102 225.150 80, 137, 443
111.100 200.150 80, 443
If that's not what you're looking for, please clarify.