Find cc attack IPs use shell scripts by log files - bash

Question

I have a history web log files like this:
157.15.14.19 - - 06 Sep 2016 09:13:10 +0300 "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:13:11 +0300 "GET /index.php?id=2 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:13:12 +0300 "GET /index.php?id=3 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:14:13 +0300 "GET /index.php?id=4 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:14:14 +0300 "GET /index.php?id=5 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:15:15 +0300 "GET /index.php?id=6 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:15:16 +0300 "GET /index.php?id=7 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:15:17 +0300 "GET /index.php?id=8 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:16:10 +0300 "GET /index.php?id=9 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:16:10 +0300 "GET /index.php?id=10 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
8.8.8.8 - - 06 Sep 2016 09:17:10 +0300 "GET /index.php?id=11 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
9.9.9.9 - - 06 Sep 2016 09:17:10 +0300 "GET /index.php?id=12 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:18:10 +0300 "GET /index.php?id=13 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:19:10 +0300 "GET /index.php?id=14 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:19:10 +0300 "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:20:10 +0300 "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
123.123.123.123 - - 06 Sep 2016 09:21:10 +0300 "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - - 06 Sep 2016 09:22:10 +0300 "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
I want find out the cc attack IPs,only through the yesterday web log files
This example , I sign a cc attack :
every 5 minutes,The same remote ip request counts more than 5, the ip will a cc attack and print it.
The log file is all day,and only use bash scripts, just like awk,cat,gawk,sed and so..
Please me some suggest， Thanks a lot.
Update:
I try wite the test script (per 2minutes the same request count>5)
yy#yy:/tmp/tb$ cat 5.txt |awk '{print $7,$1}' |awk -F: '{print $1*60+int($2/2),$0}' |sort |uniq -c -f2 |awk '{if($1>5){print $0}}'
10 546 09:13:10 157.15.14.19
But, the code is so badly, It will be optimization.

Answer 1

awk -v Interval=5 -v Trig=5 -F '[[:blank:]]*|:' '
{
# using format log
# 157.15.14.19 - - 06 Sep 2016 09:13:10 +0300 "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
# $1 2 3 4 5 6 7 8 9 10 11 ...
ThisTime = $7 * 60 + $8
#if new cycle (so this line is not in the cycle)
if ( ThisTime > ( LastTic + Interval ) ) {
# check and print last cycle hit
for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
# reset reference
split( "", IPCounts)
LastTime = $4 " " $5 " " $6 " " $7 ":" sprintf( "%2d", ( $8 - ( $8 % Interval) )) ":00"
LastTic = $7 * 60 + ( $8 - ( $8 % Interval) )
}
# add this line to new cycle
IPCounts[ $1]++
}
END {
# print last cycle
for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
}
' YourFile
# for format of log
# op.g.cc 124.145.36.121 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.238:80 "POST ...
# $1 2 3 4 5 6 7 8 9 10 11 ...
# change:
# $7 by $6, $8 by $7
# LastTime = $5 ":" $6 ":" sprintf( "%2d", ( $7 - ( $7 % Interval) )) ":00 +800]"
# IPCounts[ $2]++
Note:
work quick and dirty for time selection (you mention 1 log per day). If more precision is needed, use mkftime to use real epoch time reference
Trig is the count trigger level (5 times) and Interval is the time of the cycle (5 minutes)