This is my AWS module setting in K8S
filebeat.modules:
- module: aws
cloudtrail:
enabled: true
var.queue_url:
input:
fields:
cloud.service.name: cloudtrail
cloud.service.type: cloudtrail
fields_under_root: true
But since I view the log in the Kibana
{
"_index": ".log-aws-1",
"fields": {
"fields.cloud.service.name": [
"cloudtrail"
],
...
}
}
This is issues
not work adding 'cloud.service.type' field.
I changed 'input' to 'inputs', but not work.
not work 'fields_under_root', it seems 'fields.fields.cloud.service.name'
What should I do to apply two cases?
Related
I would like to add COLD data node (NOT data node) to my elasticsearch cluster using helm:
My values.yaml:
...
roles:
master: "false"
ingest: "false"
data: "false"
remote_cluster_client: "false"
ml: "false"
data_cold: "true"
...
but when deploy it, i got this error:
java.lang.IllegalArgumentException: unknown setting [node.data_cold] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
Any idea please ?
Thank you in advance!
Assuming you're using the Elastic helm charts, I accomplished this by setting the following in my values.yml:
extraEnvs:
- name: 'node.attr.data'
value: '{{ ilm_phase }}'
and setting the following in my vars.yml for each individual data tier:
ilm_phase: 'cold' # ...or hot, or whatever...
And then finally, using a custom node attribute in my ILM policy.
It's not ideal, but it works well, even if it's not as nuanced as using node.roles. If someone else has a better method, I'm open to it.
Edit
I forgot that I also added the following template, which applies to all new indices created. This forces all new indices to be created on the hot data nodes.
PUT _template/ilm-set-index-ilm-hot
{
"order": 127,
"index_patterns": [ "*" ],
"settings": {
"index": {
"routing": {
"allocation": {
"require": {
"data": "hot"
}
}
}
}
},
"mappings": {},
"aliases": {}
}
I have run into a brick wall considering searching in my logged events. I am using an elasticsearch solution, filebeat to load messages from logs to elasticsearch, and Kibana front end.
I currently log the messages into a field message and exception stacktrace (if present) into error.message. So the logged event's snippet may look like:
{
"message": "Thrown exception: CustomException (Exception for testing purposes)"
"error" : {
"message" : "com.press.controller.CustomException: Exception for testing purposes\n at
com.press.controller....<you get the idea at this point>"
}
}
Of course there are other fields like timestamp, but those are not important. What is important is this:
When I search message : customException, I can find the events I logged. When I search error.message : customException, I do not get the events. I need to be able to fulltext search all fields.
Is there a way how to tell elasticsearch to enable the fulltext search in the fields?
And why has the "message" field enabled it by default? None of my colleagues are aware that any indexing command was run on the field in the console after deployment and our privileges do not allow me or other team members to run indexing or analysis commands on any field. So it has to be in the config somewhere.
So far I was unable to find the solution. Please push me in the right direction.
Edit:
The config of fields is as follows:
We use a modified ECS, and both messages are declared as
level: core
type: text
in file fields.yml.
in filebeat, the config snippet is as such:
filebeat.inputs:
- type: log
enabled: true
paths: .....
...
...
processors:
- rename:
fields:
- from: "msg"
to: "message"
- from: "filepath"
to: "log.file.name"
- from: "ex"
to: "error.message"
ignore_missing: true
fail_on_error: true
logging.level: debug
logging.to_files: true
For security requirements, I cannot disclose full files. Also, I need to write all the snippets by hand, so misspells are probably my fault.
Thanks
Problem is with the analyzer associated with your field, by default for text fields in ES, standard analyzer is used which doesn't create separate tokens if text contains . for ex: foo.bar would result in just 1 token as foo.bar while if you want both foo and bar should match in foo.bar then you need to genrate 2 tokens as foo and bar.
What you need is a custom analyzer which creates token as above as your error.message text contains . which I explained in my example:
PUT /my_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "standard",
"char_filter": ["replace_dots"]
}
},
"char_filter": {
"replace_dots": {
"type": "mapping",
"mappings": [
". => \\u0020"
]
}
}
}
}
}
POST /my_index/_analyze
{
"analyzer": "my_analyzer",
"text": "foo.bar"
}
The above example creates 2 tokens as foo and bar and same should happen with you when you create and test it with these API.
Let me know if you face any issue with it.
Elastic Search indexes all fields by default, here you did not define the mapping hence all fields should be indexed by default.
Also for your case I doubt if the data is properly going in elastic search as the log doesn't seem to be proper json.
Do you see proper logs in Kibana if yes please send a sample log/screenshot
Iam trying to index in elastichsearch with the help of filebeat and logstash. Here is the filebeat.yml :
filebeat.inputs:
- type: docker
combine_partial: true
containers:
path: "/usr/share/dockerlogs/data"
stream: "stdout"
ids:
- "*"
exclude_files: ['\.gz$']
ignore_older: 10m
processors:
# decode the log field (sub JSON document) if JSON encoded, then maps it's fields to elasticsearch fields
- decode_json_fields:
fields: ["log", "message"]
target: ""
# overwrite existing target elasticsearch fields while decoding json fields
overwrite_keys: true
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
# setup filebeat to send output to logstash
output.logstash:
hosts: ["xxx.xx.xx.xx:5044"]
# Write Filebeat own logs only to file to avoid catching them with itself in docker log files
logging.level: info
logging.to_files: false
logging.to_syslog: false
loggins.metrice.enabled: false
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
ssl.verification_mode: none
And here is the logstash.conf:
input
{
beats {
port => 5044
host => "0.0.0.0"
}
}
output
{
stdout {
codec => dots
}
elasticsearch {
hosts => "http://xxx.xx.xx.x:9200"
index => "%{[docker][container][labels][com][docker][swarm][service][name]}-%{+xxxx.ww}"
}
}
Iam trying to index with the docker name so it would be more readable and more clear than the usual pattern we see all the time like "filebeat-xxxxxx.some-date".
I tried several things:
- index => "%{[docker][container][labels][com][docker][swarm][service][name]}-%{+xxxx.ww}"
- index => "%{[docker][container][labels][com][docker][swarm][service][name]}-%{+YYYY.MM}"
- index => "%{[docker][swarm][service][name]}-%{+xxxx.ww}"
But nothing worked. What am i doing wrong ? Maybe iam doing something wrong or missing anthing in filebeat.yml file. It could be that too.
Thanks for any help or any lead.
Looks like you're unsure of what docker metadata fields are being added. It might be a good idea to just get successful indexing first with the default index name (ex. "filebeat-xxxxxx.some-date" or whatever) and then view the log events to see the format of your docker metadata fields.
I don't have the same setup as you, but for reference, I'm on AWS ECS so the format of my docker fields are:
"docker": {
"container": {
"name": "",
"labels": {
"com": {
"amazonaws": {
"ecs": {
"cluster": "",
"container-name": "",
"task-definition-family": "",
"task-arn": "",
"task-definition-version": ""
}
}
}
},
"image": "",
"id": ""
}
}
After seeing the format and fields available, I was able to add a custom "application_name" field using the above. This field is being generated in my input plugin which is redis in my case, but all input plugins should have the add_field option (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-add_field):
input {
redis {
host => "***"
data_type => "list"
key => "***"
codec => json
add_field => {
"application_name" => "%{[docker][container][labels][com][amazonaws][ecs][task-definition-family]}"
}
}
}
After getting getting this new custom field, I was able to run specific filters (grok, json, kv, etc) for different "application_name" fields as they had different log formats, but the important part for you is that you could use it in your output to Elasticsearch for index names:
output {
elasticsearch {
user => ***
password => ***
hosts => [ "***" ]
index => "logstash-%{application_name}-%{+YYY.MM.dd}"
}
}
I am writing logs into log file from my Django app, from there I am shipping those logs to elasticsearch. Because I want to split the fields as well, I am using logstash between filebeat and elasticsearch.
Here is sample log field:
2019-03-19 13:39:06 logfile INFO save_data {'field1': None, 'time':
'13:39:06', 'mobile': '9876543210', 'list_item': "[{'item1': 10,
'item2': 'path/to/file'}]", 'response': '{some_complicated_json}}',
'field2': 'some data', 'date': '19-03-2019', 'field3': 'some other
data'}
I tried to write a GROK match pattern but all the fields are going into message field :
%{TIMESTAMP_ISO8601:temp_date}%{SPACE} %{WORD:logfile} %{LOGLEVEL:level} %{WORD:save_data} %{GREEDYDATA:message}
How can I write GROK match pattern which can decompose above log entry.
I don't know how you could do this with Grok, but the way we do it is with a json processor in elastic ingest node pipeline. Something like this:
{
"my-log-pipeline": {
"description": "My log pipeline",
"processors": [{
"json": {
"field": "message",
"target_field": "messageFields"
}
}]
}
}
Then you just need to tell your source (filebeat/logstash) to use this pipeline when ingesting.
I am using ELK with kibana.
I am also using filebeat for sending data to Logstash.
The i have created look like this
{
"mappings": {
"_default_": {
"properties": {
"msg":{"type":"string", "index":"not_analyzed"}
}
},
"log": {
"properties": {
"#timestamp":{"type":"date","format":"strict_date_optional_time||epoch_millis"},
"#version":{"type":"string"},
"beat": {
"properties": {
"hostname":{"type":"string"},
"name":{"type":"string"},
}
},
"count":{"type":"long"},
"host":{"type":"string"},
"input_type":{"type":"string"},
"message":{"type":"string"},
"msg":{"type":"string","index":"not_analyzed"},
"offset":{"type":"long"},
"source":{"type":"string"},
"type":{"type":"string"}
}
}
}
}';
I want to know that just like beat has 2 fields like hostname and name. Is it possible to have add more fields like environment: dev which i can see in kibana so that i can filter messages based on that
Yes, you can specify additional fields in your filebeat.yml configuration. Those new fields will be created. You have two options, you can either specify fields and/or fields_under_root.
If you use the former (see below), a new fields subgroup with your custom fields will appear in your document and you will be able to filter messages with fields.environment: dev in Kibana.
fields:
environment: dev
If you use the latter (see below), your custom fields will appear at the top-level in your document and you will be able to filter messages with environment: dev in Kibana.
fields_under_root: true