ELK run in containers
I setup iptables send all input/forward/output logs to logstash.
example log seen on kibana discover pane.
#version:1 host:3.3.3.3 #timestamp:March 3rd 2018, 12:14:45.220 message:<4>Mar 3 20:14:47 myhost kernel: [2242132.946331] LOG_ALL_TRAF public INPUT IN=public OUT= MAC=00:1e:67:f2:db:28:00:1e:67:f2:d9:7c:08:00 SRC=1.1.1.1 DST=2.2.2.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=17722 DF PROTO=TCP SPT=3504 DPT=8080 WINDOW=512 RES=0x00 ACK URGP=0 type:rsyslog tags:_jsonparsefailure _id:AWHtgJ_qYRe3mIjckQsb _type:rsyslog _index:logstash-2018.03.03 _score: -
The entire log is categorized as 'message' field.
I want to use SRC, DST, SPT, DPT etc as each individual field and then also use them to visualize.
Any guidance is much appreciated.
You will need to learn about Grok filter plugin that will enable you split the message into named fields.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
The list of common patterns is available here.
And you can test your patterns here.
Related
I'm trying filter for particular words in a log file using regex, the goal is that any log line that matches the regex in custom_pattern will go into influxdb, log lines that do not match willbe ignored. When I tested the regex it works, even in golang playground (https://play.golang.org/p/_apzOVwwgl2). But when I use it in the telegraf conf file as it is below, it doesn't work, there's no input into influxdb. Is there something I'm missing that should added to the configuration?
I've tested the regex on http://grokdebug.herokuapp.com/ and https://play.golang.org/p/_apzOVwwgl2 it works but not in the custom_patterns under [inputs.logparser.grok].
Here is my grok config
[[inputs.logparser]]
files = ["/var/log/test1"]
from_beginning = true
[inputs.logparser.grok]
patterns = ["%{FAIL_LOG}"]
custom_patterns = '''FAIL_LOG ^.*?\b(multipathd?)\b.*?\b(failed|failing|(remaining active paths))\b.*?$'''
The pattern is supposed to match first 2 log lines like below and ignore the third line.
Oct 29 03:29:03 dc-as-5p multipath: checker failed interface 8:0 in map 150gb
Oct 29 03:29:03 dc-as-5p multipathd: checker failing interface 8:0 in map 150gb
Oct 29 03:26:03 dc-as-5p link: checker down remaining active paths interface 8:0 in map 150gb
What am I doing wrong?
I summarised how I got custom log parsing in Telegraf/GROK to work in the following post: Custom log parsing with Telegraf/Tail Plugin/GROK. Maybe it helps you or others debug similar problems.
Maybe interessting for others reading this in 2020, that Telegraf's logparser is now replaced by the Tail plugin. Example in my post above.
PS: My approach for your problem would be to not use regex at all, but to define three different patterns for each of the lines. This of course will only work if you have a low number of possible log errors/lines.
If you run telegraf with the --debug flag, you will see that it is having an issue parsing the logs.
$ telegraf --debug --config ./telegraf.conf
...
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:29:03 dc-as-5p multipath: checker failed interface 8:0 in map 150gb"
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:29:03 dc-as-5p multipathd: checker failing interface 8:0 in map 150gb value=3"
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:26:03 dc-as-5p link: checker down remaining active paths interface 8:0 in map 150gb"
This error message is misleading because, as your testing has shown, your regex pattern is correct. The real issue is that you have not included a value to be logged in your regex.
A version of your regex to store the error message and timestamp might be:
custom_patterns = '''FAIL_LOG %{SYSLOGTIMESTAMP:timestamp}.*(multipath).?: %{GREEDYDATA:message:string}'''
The value pattern can be found between ${}. Additional premade patterns can be found here. This will eliminate the first two errors above. The results of these can be seen using the --test flag.
$telegraf --test --config ./telegraf.conf
...
> logparser,host=pop-os,path=./test1 message="checker failed interface 8:0 in map 150gb",timestamp="Oct 29 03:29:03 " 1573968174161853621
For some reason the --test flag did not always output the results. I would have to run the command multiple times before getting the above output.
I am using UFTP to transfer files within the subnetwork computers.
But when I used -H to send only particular computers instead of sending to all computers, it is not working as expected.
Let me explain in detail :
I have two windows machines in same network of IP's 172.21.170.198,172.21.181.216 respectively.
From one of the system, I used below mentioned command to send the file
uftp.exe -R 100000 -H 172.21.170.198,172.21.181.216 e:\setup.exe
But both machines won't receive those file.
But if I use this command both machines will receive the file.
uftp.exe -R 100000 E:\setup.exe
I want to know whether I made any mistake.
Please correct me if I am wrong.
Thanks in Advance.
Kindly revert back for any clarifications.
Regards,
Thiyagu
If ipv6 isn't enabled, it would look like this, converting the ipv4 addresses to hex (with a converter like http://www.kloth.net/services/iplocate.php):
uftp.exe -R 100000 -H 0xAC15AAC6,0xAC15B5D8 e:\setup.exe
But if you have an ipv6 address on the client, the client id sort of comes from the end of it backwards. Like if the address was "fe80::e5ca:e3ca:fea3:153f%5", the command would look like:
uftp.exe -R 100000 -H 0x3f15a3fe e:\setup.exe
(coming from "fe a3 15 3f")
When you execute ansible on some host, it will write to syslog on that host, something like this:
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:56 run-tools python: ansible-<stdin> Invoked with filter=* fact_path=/etc/ansible/facts.d
Dec 1 15:09:56 run-tools python: ansible-<stdin> Invoked with checksum_algorithm=sha1 mime=False get_checksum=True path=/usr/local/bin/check_open_files_generic.sh checksum_algo=sha1 follow=False get_md5=False
Dec 1 15:09:56 run-tools python: ansible-<stdin> Invoked with directory_mode=None force=False remote_src=None path=/usr/local/bin/check_open_files_generic.sh owner=root follow=False group=root state=None content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None dest=/usr/local/bin/check_open_files_generic.sh selevel=None original_basename=check_open_files_generic.sh regexp=None validate=None src=check_open_files_generic.sh seuser=None recurse=False delimiter=None mode=0755 backup=None
Dec 1 15:20:03 run-tools python: ansible-<stdin> Invoked with warn=True executable=None _uses_shell=False _raw_params=visudo -c removes=None creates=None chdir=None
Is there any documentation or explanation of these logs that would help me understand how to read them? Specifically I would like to be able to see what exactly ansible did, which files it touched etc. Is it possible to find it there? Or reconfigure ansible so that it writes this kind of information in there?
Is it possible to configure these logs at all? How?
I am not aware of documentation that explains the contents of syslog messages specifically. However, you can look at some of the logging code in AnsibleModule.log() to see what's going on. Basically, it's reporting module names and the parameters they were called with.
For configuring logs, there are some good suggestions in response to this related question. The summary is that you can get more information - including your request about what ansible did - by specifying a log path and running with the verbose -v flag. For more fine-grained control, you can attack the problem from two different angles:
From the playbook side, you can use the debug module or tailor your handling of changed/failed results to suit your needs. Both of those changes can add useful context to your log output.
Outside of playbooks, you can use Ansible callback plugins to control logging. Here is an example of a callback plugin which intercepts logs and outputs something more human readable.
My router sends its 'DROP' packets to my server via syslog to be logged to a file in the following manner:
Oct 30 13:01:02 192.168.1.1 kernel: DROP IN=vlan2 OUT=
MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
SRC=93.108.197.92 DST=192.168.2.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51
ID=44828 DF PROTO=TCP SPT=55552 DPT=33248 WINDOW=7300 RES=0x00 SYN
URGP=0 OPT (020405840402080A0035BAC40000000001030300)
Oct 30 13:01:06
192.168.1.1 kernel: DROP IN=vlan2 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
SRC=93.108.197.92 DST=192.168.2.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51
ID=44829 DF PROTO=TCP SPT=55552 DPT=33248 WINDOW=7300 RES=0x00 SYN
URGP=0 OPT (020405840402080A0035BEAE0000000001030300)
Oct 30 13:01:07
192.168.1.1 kernel: DROP IN=vlan2 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
SRC=189.175.171.76 DST=192.168.2.10 LEN=44 TOS=0x00 PREC=0x00 TTL=53
ID=260 PROTO=TCP SPT=14779 DPT=23 WINDOW=50523 RES=0x00 SYN URGP=0 OPT
(020405AC)
Oct 30 13:01:09 192.168.1.1 kernel:
DROP IN=vlan2 OUT=
MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
SRC=125.211.218.39 DST=192.168.1.1 LEN=88 TOS=0x00 PREC=0x00 TTL=48
ID=39896 DF PROTO=ICMP TYPE=8 CODE=0 ID=29389 SEQ=1
Oct 30 13:01:14 192.168.1.1 kernel: DROP IN=vlan2 OUT=
MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
SRC=93.108.197.92 DST=192.168.2.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51
ID=44830 DF PROTO=TCP SPT=55552 DPT=33248 WINDOW=7300 RES=0x00 SYN
URGP=0 OPT (020405840402080A0035C6800000000001030300)
I'd like to put each field into a mysql db so I view and analyze later. I'm trying to think of the best way to parse/filter/do this. I'd like to do it in bash but I'm open to other alternatives/langs if it makes it way more efficient or easier.
My iptables log files are rotated every so often and I was going to create a bash/sed/awk script to look through each line of the log(s) and create an sql file so I can use a 'LOAD DATA INFILE' command to load all data into one INSERT statement.
As you can see above, an ICMP and TCP type of packet will differ from the way it is written to the file (number of fields after ID)
I have a few different ways to complete this:
Search by PROTO and the remaining awk 'print' commands are used to grab all relevant information.
Search for all [PARAM]=[VALUE] in every line, regardless of PROTO and just shove them in mysql and analyze later.
So far I have (I know this is basic, but I'm wondering if I should approach it differently before investing more time into it):
cat "$fw_file" | while read line; do
type=$(grep -oP 'PROTO=\w+\s' | cut -d= -f2)
df=$(grep -oP 'ID=\w+\sDF\s' | cut -d' ' -f2)
# continuing on for all fields....
# ......
done
If there a better, more efficient, way for me to do this instead of just grabbing all fields?
Before your begin working on your script (Reinventing the Wheel), For parsing logs and analyzing log data there are opensource tools that do the job efficently. Please consider them!
For your exact usecase, You Better use Elasticsearch,Logstash, Kibana (ELK Stack).
Some Advantages of ELK Stack comparing to script and relational database approach:
Easier to store logs from multiple sources ( one can be your router)
Scalable ( you probably get slowed pretty soon depending on your logs input rate)
Very easy to visualize data in web interface with various charts using kibana.
Elasticsearch has REST API, So your developers can do their own thing too!
There are many tutorials online to get you going pretty fast.
I'm shipping Windows DNS debug logs via json into Elasticsearch and I need to parse them.
As with Microsoft nothing is easy. The DNS debug log is not a CSV. The only useful thing in that file is that it has fixed lengths of columns.
Here is a sample of the DNS logs:
11/21/2014 5:59:13 PM 0458 PACKET 00000000039ED750 UDP Rcv 192.168.1.98 600c Q [0001 D NOERROR] A (9)grokdebug(9)herokuapp(3)com(0)
11/21/2014 5:59:13 PM 0458 PACKET 00000000039EF460 UDP Snd 192.168.1.1 e044 Q [0001 D NOERROR] A (9)grokdebug(9)herokuapp(3)com(0)
11/21/2014 5:59:13 PM 0458 PACKET 00000000039F85B0 UDP Rcv 192.168.1.1 e044 R Q [8081 DR NOERROR] A (9)grokdebug(9)herokuapp(3)com(0)
11/21/2014 5:59:13 PM 0458 PACKET 00000000039F85B0 UDP Snd 192.168.1.98 600c R Q [8081 DR NOERROR] A (9)grokdebug(9)herokuapp(3)com(0)
I looked at this Stackoverflow answer: Logstash grok filter help - fixed position file
and was trying to set up a grok filter to parse the columns but it's not working for me.
I understand I have a syntax issue but I can't seem to find a good example that would steer me in correct direction.
Here is my grok filter:
grok {
match => [ "message", "(?<dns_date_n_time>.{21}) (?<dns_field_1>.{5}) (?dns_type>.{8}) (?<dns_field_2>.{19}) (?<dns_protocol>.{4}) (?<dns_direction>.{4}) (?<dns_ip>.{16}) (?<dns_field_3>.{4}) (?<dns_query_type>.{5}) (?<dns_field_5>.{7}) (?<dns_field_6>.{3}) (?<dns_flag>.{9}) (?<dns_field_7>.{2}) (?<dns_record>.{5}) (?<dns_domain>.{255})" ]
}
Can anyone help?
Don't get hung up on the fact that the logfile happens to have a fixed-width format. It doesn't really help here since. Parse the file like it's any old logfile using relevant grok patterns. This works for the input you provided:
(?<timestamp>%{DATE_US} %{TIME} (?:AM|PM))\s+%{NUMBER}\s+%{WORD:dns_type}\s+
%{BASE16NUM}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+
%{BASE16NUM}\s+%{WORD:dns_query_type}\s+\[%{BASE16NUM}\s+%{WORD}\s+
%{WORD:dns_result}\]\s+%{WORD:dns_record}\s+%{GREEDYDATA:dns_domain}
That said, since I don't know what each column in the logfile means some patterns used here might be too sloppy or too strict. I've inserted linebreaks to make the answer more readable but make sure you concatenate thing correctly when you insert it into your configuration file.