Parsing unstructured text file with grep - bash

Question

I am trying to analyze this IDS log file from MIT, found here.
Summarized attack: 41.084031
IDnum Date StartTime Duration Destination Attackname insider? manual? console?success? aDump? oDump iDumpBSM? SysLogs FSListing StealthyNew? Category OS
41.08403103/29/1999 08:18:35 00:04:07 172.016.112.050ps out rem succ aDmp oDmp iDmp BSM SysLg FSLst Stlth Old llU2R
41.08403103/29/1999 08:19:37 00:01:56 209.154.098.104ps out rem succ aDmp oDmp iDmp BSM SysLg FSLst Stlth Old llU2R
41.08403103/29/1999 08:29:27 00:00:43 172.016.112.050ps out rem succ aDmp oDmp iDmp BSM SysLg FSLst Stlth Old llU2R
41.08403103/29/1999 08:40:14 00:24:26 172.016.112.050ps out rem succ aDmp oDmp iDmp BSM SysLg FSLst Stlth Old llU2R
I am trying to write commands that do two things:
First, parse through the entire file and determine the amount of distinct "summarized attacks" that begin with 4x.xxxxx. I have accomplished this with:
grep -o -E "Summarized attack: 4". It returns 80.
Second, for each of the "Summarized Attacks" found by the above command, parse the table and determine the amount of IDnum rows, and return the total amount of rows (i.e., attacks) across all "Summarized attack" finds. I would imagine that number is somewhere around 200.
However, I am struggling to get the individual number of IDs, i.e., that are in the IDnum column of this text file.
Since it is a text file with technically no structure, how can I parse this .txt file as if it had a tabular structure to retrieve the total entries in the IDnum column, for each Summarized attack that follows the above grep command's search text?
Desired output would be a count of all IDnum's for the Summarized attacks found by the above command. I don't know the count, but I would imagine an integer output, similar to the return of 80 for grep -o -E "Summarized attack: 4". The output would be <int> where <int> is the # of "attacks" as defined by rows in the IDnum column across all 80 of the found "Summarized attacks" by the above grep command.
If another command other than grep is better suited, that is OK.

Answer 1

to count matches you can use grep -c
grep -cE '(^Summarized.attack:.4[0-9]\.[0-9]+$)'
you can use colon as delimiter for cut -d
(if you loop over results the leading whitespace does not care)
grep -oE '(^Summarized.attack:.4[0-9]\.[0-9]+$)' | cut -d: -f2
example loop
file="path/to/master-listfile-condensed.txt"
for var in $(grep -oE '(^Summarized.attack:.4[0-9]\.[0-9]+$)' "$file" | cut -d: -f2)
do
printf "Summarized attacks: %s: %s\n" $var \
$(grep -cE "(^.${var}[0-9]+/[0-9]{2}/[0-9]{4})" "$file")
done
^ start of line
$ end of line
. any byte (in this case single whitespace)
\. single dot (escaped)
[0-9] single digit
+ one (or more) occurrence
{4} four occurrence

Answer 2

Assuming you have more than one "Summarized attack:" in your input file this may be what you're looking for:
$ cat tst.awk
/^Summarized attack:/ {
prt()
atk = ($3 ~ /^4/ ? $3 : 0)
cnt = 0
}
atk { cnt++ }
END {
prt()
print "TOTAL", tot
}
function prt() {
if ( atk ) {
cnt -= 2
print atk, cnt
}
tot += cnt
}
.
$ awk -f tst.awk file

Answer 3

For your first part, fgrep -c "Summarized attacks: 4" or fgrep -F "Summarized attacks: 4" is sufficient.
If I understand your second part, for each of those blocks, you want to add up the attack rows and print a grand total. You can do that with
gawk '/^Summarized attack: 4/ { on=1; next} /^ 4[0-9.]*/ { if (on) ++ids; next} /^ IDnum/ {next} /^ */ {next} { on=0} END {print ids;}'< master-listfile-condensed.txt
The first statement says, search (/.../) for every line that begins with (^) "Summarized attack: 4", and upon finding it, turn on the "on" flag, and go to the next line. The second statement says, if this is an attack record (i.e. begins with 4 followed by a string [*] of digits), then check the flag; if it is on, count it. Basically, we want the flag to be on when we are in a stanza of target attack records. The next two statements say for every line that starts with " IDnum" or are all whitespace (sometimes blank lines are inserted), go to the next line; this is needed to counteract the next statement, which says that if this is not a line that matches any of the previous statements, turn off the "on" flag. This prevents us from counting attacks outside the target. Finally, END means at the end, print the grand total. I get 757 which is pretty far out of your range. But I think it is correct.
But a far easier way, assuming the Summarized timestamp is always repeated in the IDnum at least to the first significant digit, would be to use
grep -Ec '^ 4' master-listfile-condensed.txt
That means count all the lines that begin with space-4. In this case it gives us the correct result.