Processing multi line logs with AWK to gather SQL statements - bash

Question

I have the following entries in a log file:
2016-01-25 21:12:41 UTC:172.31.21.125(56665):user#production:[21439]:ERROR: bind message supplies 1 parameters, but
prepared statement "" requires 0
2016-01-25 21:12:41 UTC:172.31.21.125(56665):user#production:[21439]:STATEMENT: SELECT count(*) AS total FROM (
SELECT 1 AS count
FROM leads_search_criteria_entities
INNER JOIN entities e on entity_id = e.viq_id
LEFT JOIN companies_user cu ON cu.entity_id = e.viq_id
WHERE criterium_id = 644 AND ((
( cu.udef_type IS NULL -- if not set by user, check calculated value
AND is_university >= 50
) OR (
cu.udef_type IS NOT NULL -- if set by user, use it
AND cu.udef_type = 'university'
)
))
GROUP BY e.viq_id
ORDER BY e.viq_id
) x
2016-01-25 21:14:11 UTC::#:[2782]:LOG: checkpoint starting: time
2016-01-25 21:14:16 UTC::#:[2782]:LOG: checkpoint complete: wrote 51 buffers (0.0%); 0 transaction log file(s) added, 0 remov
ed, 0 recycled; write=5.046 s, sync=0.038 s, total=5.091 s; sync files=18, longest=0.008 s, average=0.002 s
2016-01-25 21:19:11 UTC::#:[2782]:LOG: checkpoint starting: time
I would like to capture the SQL statements but I am not sure how can I do that with AWK.
Update:
Expected outcome:
SELECT count(*) AS total FROM ( SELECT 1 AS count FROM leads_search_criteria_entities INNER JOIN entities e on entity_id = e.viq_id LEFT JOIN companies_user cu ON cu.entity_id = e.viq_id WHERE criterium_id = 644 AND (( ( cu.udef_type IS NULL -- if not set by user, check calculated value AND is_university >= 50 ) OR ( cu.udef_type IS NOT NULL -- if set by user, use it AND cu.udef_type = 'university' ) )) GROUP BY e.viq_id ORDER BY e.viq_id ) x
My current almost working solution uses sed but this is where I got stuck, it just helps filtering the lines that have a select (multiple lines by itself) and the next line after that. Any suggestion is appreciated
sed -n "/:STATEMENT:/,/2016/p" out

Answer 1

I don't recommend using sed for this. First thought for an awk solution might look like this:
/^2016/&&line~/:STATEMENT:/ {
sub(/.*:STATEMENT:/,"",line)
print line
}
/^2016/ {
line=""
}
{
$1=$1
line=sprintf("%s %s",line,$0)
}
END {
if (line~/:STATEMENT:/) {
sub(/.*:STATEMENT:/,"",line)
print line
}
}
Obviously you could shrink this. I wrote and ran it (for testing) as a one-liner.
The idea here is that:
we'll append to a variable, resetting it every time our input line starts with the year. (You could replace this with a regexp matching the date if you want to run this next year without modification),
when we get to a new log line (or the end), we strip off the cruft before the SQL statement and print the result.
Note the $1=$1. The purpose of this is to change your line's whitespace, so that newlines and tabs and multiples spaces are collapsed into single spaces. Experiment with removing it to see the impact.

Answer 2

Update
Howabout a combination of sed and tr
sed 's/^[0-9][^S]*//' INPUT.txt | sed '/^[0-9a-z]/d' | tr -s ' ' | tr -d '\n'
output:
STATEMENT: SELECT count(*) AS total FROM ( SELECT 1 AS count FROM leads_search_criteria_entities INNER JOIN entities e on entity_id = e.viq_id LEFT JOIN companies_user cu ON cu.entity_id = e.viq_id WHERE criterium_id = 644 AND (( ( cu.udef_type IS NULL -- if not set by user, check calculated value AND is_university >= 50 ) OR ( cu.udef_type IS NOT NULL -- if set by user, use it AND cu.udef_type = 'university' ) )) GROUP BY e.viq_id ORDER BY e.viq_id ) x

Answer 3

$ cat log.awk
f && /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/ {f=0; print ""}
sub(/^.*:STATEMENT:[[:space:]]+/,"") {f=1}
f { $1=$1; printf "%s ", $0 }
$ awk -f log.awk log.txt
SELECT count(*) AS total FROM ( SELECT 1 AS count FROM leads_search_criteria_entities INNER JOIN entities e on entity_id = e.viq_id LEFT JOIN companies_user cu ON cu.entity_id = e.viq_id WHERE criterium_id = 644 AND (( ( cu.udef_type IS NULL -- if not set by user, check calculated value AND is_university >= 50 ) OR ( cu.udef_type IS NOT NULL -- if set by user, use it AND cu.udef_type = 'university' ) )) GROUP BY e.viq_id ORDER BY e.viq_id ) x
(2nd line) This turns on printing (f=1) when :STATEMENT: is found, and as a side-effect, removes everything up until the start of the SELECT statement.
(3rd line) Then it keeps printing until printing is turned off (see below), cleaning up by replacing sequences of multiple spaces by a single space. (EDIT: Thanks to #ghoti for suggesting the elegant $1=$1 for that.)
(1st line) Turn off printing at the start of the next log, identified by starting with a date. Print a courtesy newline to end the SELECT.