I am currently using goaccess-1.0.2. I have installed it on an Amazon Linux box. The box which it resides has customized logs that were forwarded from an Apache WebApp Server. What I have tried to accomplish but can't seem to figure out is how to get GoAccess to parse our customized log.
Here is an example of the custom forwarded WebApp Log entry:
Jun 24 00:00:41 directory1 httpd-access: 55.117.170.95 www.URLaddress.com - [24/Jun/2016:00:00:41 -0700] "GET /sites/all/themes/somthing_on_demand/js/fancybox/jquery.fancybox-1.3.4.css HTTP/1.1" 304 - "ht
tps://www.IPaddress.com/my_account/yum" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "SESSb9948a0b21e4d377a7d82f6adbf86c91=l
on7pgjlikml7q4tq954ejiao1; cookie_js=1; __utma=23285183.1119616966.1452095139.1468883973.1468963151.39; __utmb=23285183.500.10.1468963151; __utmc=23285183; __utmz=23285183.1468963151.39.39.utmcsr=fyi.URLaddress.com|utm
ccn=(r/INFOSEC-MAXLEN-256" "-" 57630
Here are a few log-formats I have tried:
log-format %^ %^ %^ "%h %^ %u %t \"%r\" %>s %b \"%R\" \"%u\" \"%^\" \"%^\" %D"
log-format "%h %{Host}i %{SSL_CLIENT_S_DN_CN}x %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
log-format "%h %{Host}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
I thought I would ignore the date and time format using %^ then use date format %m %d and time format %T .
I am very new at this and could really use help. Thank you for your feedback in advance.
Please try this, it works for me:
goaccess -f access.log --log-format='%^:%^:%^: %h %v %^[%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%^" %D' --date-format='%d/%b/%Y' --time-format='%T'
Related
I am sending logs from a spring boot app to rabbitMq but I can't find a way to send these logs in json format (because i want to get them in elasticsearch as documents)
My log4j.properties are:
log4j.rootLogger=INFO, consoleAppender, amqp
log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=%d %p %t [%c] <%m>%n
log4j.appender.amqp=org.springframework.amqp.rabbit.log4j.AmqpAppender
log4j.appender.amqp.host=localhost
log4j.appender.amqp.port=5672
log4j.appender.amqp.username=guest
log4j.appender.amqp.password=guest
log4j.appender.amqp.virtualHost=/
log4j.appender.amqp.exchangeName=logs_exchange_fin
log4j.appender.amqp.exchangeType=direct
log4j.appender.amqp.routingKeyPattern=service-2
log4j.appender.amqp.declareExchange=true
log4j.appender.amqp.durable=true
log4j.appender.amqp.autoDelete=false
log4j.appender.amqp.contentType=text/plain
log4j.appender.amqp.generateId=false
log4j.appender.amqp.deliveryMode=PERSISTENT
log4j.appender.amqp.layout=org.apache.log4j.PatternLayout
log4j.appender.amqp.layout.ConversionPattern=%d %p %t [%c] <%m>%n
The pattern of the logs is: %d %p %t [%c] <%m>%n
Here is my script. I want to replace text in multiple dest. How I can use wildcard in (dest=/home/*/conf/server.xml).
- hosts: 192.168.8.11
user: mohitmehral
sudo: yes
tasks:
- replace:
dest=/home/5/conf/server.xml
#dest=/home/1/conf/server.xml
#dest=/home/2/conf/server.xml
#dest=/home/3/conf/server.xml
#dest=/home/4/conf/server.xml
#dest=/home/5/conf/server.xml
regexp='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q "%{Referer}i" %m "%U" "%S" "%{User-agent}i" %b %s %D"/>'
replace='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q"%{Referer}i" %m "%U" "%{User-agent}i" "%b" "%s" "%D""/>'
backup=yes
If the regex and replace pattern are same, then you can do like this:
- hosts: 192.168.8.11
user: mohitmehral
sudo: yes
tasks:
- replace:
dest="/home/{{ item }}/conf/server.xml"
regexp='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q "%{Referer}i" %m "%U" "%S" "%{User-agent}i" %b %s %D"/>'
replace='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q"%{Referer}i" %m "%U" "%{User-agent}i" "%b" "%s" "%D""/>'
backup=yes
with_items: [1,2,3,4,5]
I'm reading the goaccess man page but I'm missing simple examples. I have a customised nginx with the following config:
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'$ssl_protocol/$ssl_cipher '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';
Here is an example log entry:
66.249.76.120 - - [20/Dec/2016:19:04:03 +0100]
TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 200 27232
"-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)" 0.026 0.026 .
How do I have to configure the .goaccessrc to read that format?
You can add this to your config file or ~/.goaccessrc
log-format %h %^[%d:%t %^] %^"%r" %s %b "%R" "%u" %T %^
date-format %d/%b/%Y
time-format %H:%M:%S
I have this log format for my Apache server
LogFormat "%h %l %u %t \"%r\" status:%>s %O \"%{Referer}i\" \"%{User-Agent}i\" %v %h %D %A %>s %T" combined
I am trying to filter out these logs based on the value of %T more than 3, so that I can send them to Loggly. I have a huge volume of logs to manage, and cannot send all of them directly to Loggly as it is an overkill for the plan I am on.
I ended up writing this CustomLog directive in my VirtualHost
CustomLog "|/bin/bash -c '{ read ENTRY <&0; if [ ${ENTRY##* } -gt 3 ]; then echo $ENTRY | sed -r \'s/status:(\d*)//\' >> /var/log/apache2/slow.log; fi; }'" combined
This appends the correct log entries to the slow.log file but also gives me an error message
AH00106: piped log program '/bin/bash -c '{ read ENTRY <&0; if [ ${ENTRY##* } -gt 3 ]; then echo $ENTRY | sed -r \\'s/status:(\\d*)//\\' >> /var/log/apache2/slow.log; fi; }'' failed unexpectedly
This is the output in slow.log if it helps
XXX.YYY.ZZZ.WWW - - [06/Oct/2016:10:06:08 +0000] "GET / HTTP/1.1" 200 4575 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0" YYY.ZZZ.WWW.XXX ZZZ.WWW.XXX.YYY 4032879 ZZZ.WWW.XXX.YYY 200 4
Note I had to use status:%>s to filter out for other logs. I also tried without sed but the error persists.
I am searching for one line command to search for few words in 'myfile.txt' and if pattern match then cut that words and print it in new file.
myfile.txt.
162.23.55.222 - - [07/Dec/2013:00:40:35 +0000] 0.033 POST /view/SBEventListComponentController?componentUid=comp_00008OM8 HTTP/1.1 200 77282 Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11 http://sportsbeta.ladbrokes.com/homepage 6476CC940C83EDF031FF2564EE108993.ecomprodsw012
162.16.87.1973 - - [07/Dec/2013:00:40:34 +0000] 0.131 POST /view/SBEventListComponentController?componentUid=comp_000080KW HTTP/1.1 200 82707 Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11 http://sportsbeta.ladbrokes.com/homepage 6476CC940C83EDF031FF2564EE108993.ecomprodsw012
162.23.22.542, 10.32.30.1 - - [07/Dec/2013:00:40:35 +0000] 0.224 GET /view/content/homepage HTTP/1.1 200 66233 Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11 - 6476CC940C83EDF031FF2564EE108993.ecomprodsw012
My output.txt must contains.
162.23.55.222 07/Dec/2013:00:40:35 http://sportsbeta.ladbrokes.com/homepage 6476CC940C83EDF031FF2564EE108993.ecomprodsw012
162.16.87.1973 07/Dec/2013:00:40:34 http://sportsbeta.ladbrokes.com/homepage
162.23.22.542, 10.32.30.1 07/Dec/2013:00:40:35
How can I search for more patterns and redirect it to another file.
I din't get why this down vote. I tried cut, grep and sed commands. But din't get expected results.
Thanks in advance.
Here's how to more or less do what I think you are asking. The first row of your output example includes the field at the end of the line but the second doesn't - I can't immediately see why
perl -l -n -a -e '($u)=/(http:\S+)/; if ($F[0] =~/,$/) { print "#F[0..1] ", substr($F[4],1) } else { print $F[4]," ", substr($F[3],1)," ", $u}' infile.txt infile.txt > outfile.txt