Optimal way to parse a log file - bash

Question

I have a log file that looks something like this:
Client connected with ID 8127641241
< multiple lines of unimportant log here>
Client not responding
Total duration: 154.23583
Sent: 14
Received: 9732
Client lost
Client connected with ID 2521598735
< multiple lines of unimportant log here>
Client not responding
Total duration: 12.33792
Sent: 2874
Received: 1244
Client lost
The log contains lots of these blocks starting with Client connected with ID 1234 and ending with Client lost. They are never mixed up (only 1 client at a time).
How would I parse this file and generate statistics like this:
I'm mainly asking about the parsing process, not the formatting.
I guess I could loop over all the lines, set a flag when finding a Client connected line and save the ID in a variable. Then grep the lines, save the values until I find the Client lost line. Is this a good approach? Is there a better one?

Answer 1

Here's a quick way using awk:
awk 'BEGIN { print "ID Duration Sent Received" } /^(Client connected|Total duration:|Sent:)/ { printf "%s ", $NF } /^Received:/ { print $NF }' file | column -t
Results:
ID Duration Sent Received
8127641241 154.23583 14 9732
2521598735 12.33792 2874 1244

Answer 2

A solution in perl
#!/usr/bin/perl
use warnings;
use strict;
print "\tID\tDuration\tSent\tReceived\n";
while (<>) {
chomp;
if (/Client connected with ID (\d+)/) {
print "$1\t";
}
if (/Total duration: ([\d\.]+)/) {
print "$1\t";
}
if (/Sent: (\d+)/) {
print "$1\t";
}
if (/Received: (\d+)/) {
print "$1\n";
}
}
Sample output:
ID Duration Sent Received
8127641241 154.23583 14 9732
2521598735 12.33792 2874 1244

Answer 3

If you're sure that the logfile can't have errors, and if the fields are always in the same order, you can use something like the following:
#!/bin/bash
ids=()
declare -a duration
declare -a sent
declare -a received
while read _ _ _ _ id; do
ids+=( "$id" )
read _ _ duration[$id]
read _ sent[$id]
read _ received[$id]
done < <(grep '\(^Client connected with ID\|^Total duration:\|^Sent:\|Received:\)' logfile)
# printing the data out, for control purposes only
for id in "${ids[#]}"; do
printf "ID=%s\n\tDuration=%s\n\tSent=%s\n\tReceived=%s\n" "$id" "${duration[$id]}" "${sent[$id]}" "${received[$id]}"
done
Output is:
$ ./parsefile
ID=8127641241
Duration=154.23583
Sent=14
Received=9732
ID=2521598735
Duration=12.33792
Sent=2874
Received=1244
but the data is stored in the corresponding associative arrays. It's fairly efficient. It would probably be slightly more efficient in another programming language (e.g., perl), but since you only tagged your post with bash, sed and grep, I guess I fully answered your question.
Explanation: grep only filters the lines we're interested in, and bash only reads the fields we're interested in, assuming they always come in the same order. The script should be easy to understand and modify to your needs.

Answer 4

awk:
awk 'BEGIN{print "ID Duration Sent Received"}/with ID/&&!f{f=1}f&&/Client lost/{print a[1],a[2],a[3],a[4];f=0}f{for(i=1;i<=NF;i++){
if($i=="ID")a[1]=$(i+1)
if($i=="duration:")a[2]=$(i+1)
if($i=="Sent:")a[3]=$(i+1)
if($i=="Received:")a[4]=$(i+1)
}}'log
if there is always an empty line between your data blocks, the awk script above could be simplified to:
awk -vRS="" 'BEGIN{print "ID Duration Sent Received"}
{for(i=1;i<=NF;i++){
if($i=="ID")a[1]=$(i+1)
if($i=="duration:")a[2]=$(i+1)
if($i=="Sent:")a[3]=$(i+1)
if($i=="Received:")a[4]=$(i+1)
}print a[1],a[2],a[3],a[4];}' log
output:
ID Duration Sent Received
8127641241 154.23583 14 9732
2521598735 12.33792 2874 1244
if you want to get better format, pipe the output to |column -t
you get:
ID Duration Sent Received
8127641241 154.23583 14 9732
2521598735 12.33792 2874 1244

Answer 5

Use Paragraph Mode to Slurp Files
Using Perl or AWK, you can slurp in records using a special paragraph mode that uses blank lines between records as a separator. In Perl, use -00 to use paragraph mode; in AWK, you set the RS variable to the empty string (e.g. "") to do the same thing. Then you can parse fields within each record.
Use Line-Oriented Statements
Alternatively, you can use a shell while-loop to read each line at a time, and then use grep or sed to parse each line. You may even be able to use a case statement, depending on the complexity of your parsing.
For example, assuming you always have 5 matching fields in a record, you could do something like this:
while read; do
grep -Eo '[[:digit:]]+'
done < /tmp/foo | xargs -n5 | sed 's/ /\t/g'
The loop would yield:
23583 14 9732 2521598735 33792
2874 1244 8127641241 23583 14
9732 2521598735 33792 2874 1244
You can certainly play with the formatting, and add header lines, and so forth. The point is that you have to know your data.
AWK, Perl, or even Ruby are better options for parsing record-oriented formats, but the shell is certainly an option if your needs are basic.

Answer 6

A short snippet of Perl:
perl -ne '
BEGIN {print "ID Duration Sent Received\n";}
print "$1 " if /(?:ID|duration:|Sent:|Received:) (.+)$/;
print "\n" if /^Client lost/;
' filename | column -t

Answer 7

awk -v RS= -F'\n' '
BEGIN{ printf "%15s%15s%15s%15s\n","ID","Duration","Sent","Received" }
{
for (i=1;i<=NF;i++) {
n = split($i,f,/ /)
if ( $i ~ /^(Client connected|Total duration:|Sent:|Received:)/ ) {
printf "%15s",f[n]
}
}
print ""
}'