Combining log entries with logstash - elasticsearch

Question

I want to collect and process logs from dnsmasq and I´ve decided to use ELK. Dnsmasq is used as a DHCP Server and as a DNS Resolver and hence it creates log entries for both services.
My goal is to send to Elasticsearch all DNS Queries with the requester IP, requester hostname (if available) and requester mac address. That will allow me to group the request per mac address regardless if the device IP changed or not, and display the host name.
What I would like to do is the following:
1) Read the entries like:
Mar 30 21:55:34 dnsmasq-dhcp[346]: 3806132383 DHCPACK(eth0) 192.168.0.80 04:0c:ce:d1:af:18 air
2) Store temporarily the relationship:
192.168.0.80 => 04:0c:ce:d1:af:18
192.168.0.80 => air
3) Enrich the entries like the one below adding the mac address and hostname. If the hostname was empty I would add the mac address.
Mar 30 22:13:05 dnsmasq[346]: query[A] imap.gmail.com from 192.168.0.80
I found a module called “memorize” that would allow me to store them but unfortunately does not work with the latest version of Logstash
The versions I´m using:
ElastiSearch 2.3.0
Kibana 4.4.2
Logstash 2.2.2
And the logstash filter (this is my first attempt with logstash and hence I´m sure the configuration file can be improved)
input {
file {
path => "/var/log/dnsmasq.log"
start_position => "beginning"
type => "dnsmasq"
}
}
filter {
if [type] == "dnsmasq" {
grok {
match => [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: ?(%{NONNEGINT:num} )?%{NOTSPACE:action} %{IP:clientip} %{MAC:clientmac} ?(%{HOSTNAME:clientname})?"]
match => [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: ?(%{NONNEGINT:num} )?%{USER:action}?(\[%{USER:subaction}\])? %{NOTSPACE:domain} %{NOTSPACE:function} %{IP:clientip}"]
match => [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: %{NOTSPACE:action} %{DATA:data}"]
}
if [action] =~ "DHCPACK" {
}else if [action] == "query" {
}else
{
drop{}
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
Questions:
1) Is there an alternative to the plugin “memorize” working with the latest logstash version? Either another plugin or different procedure.
2) Shall I downgrade logstash to a version before 2 (I think the previous is 1.5.4)? If so, is there any known sever issue or incompatibility with elasticsearch 2.2.1?
3) Or shall I modify the plugin “memorize” allowing logstash 2.x (if so I´ll appreciate any pointer on how to start)?

Answer 1

There's no need to repack the memorize plugin for this in my opinion. You can use the aggregate filter to achieve what you want.
...
# record host/mac in temporary map
if [action] =~ "DHCPACK" {
aggregate {
task_id => "%{clientip}"
code => "map['clientmac'] = event['clientmac']; map['clientname'] = event['clientname'];"
map_action => "create_or_update"
# timeout set to 48h
timeout => 172800
}
}
# add host/mac where/when needed
else if [action] == "query" {
aggregate {
task_id => "%{clientip}"
code => "event['clientmac'] = map['clientmac']; event['clientname'] = map['clientname']"
map_action => "update"
}
}

Answer 2

So to use memorize with logstash >2.0
Clone the repository.
Open file logstash-filter-memorize.gemspec
Change s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0' as s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 3.0.0'
Build plugin via: gem build logstash-filter-memorize.gemspec
Install it via: $ bin/logstash-plugin install /path/to/memorize/logstash-filter-memorize-0.9.1.gem
I tried it and seems to work.