After processing data with: input | filter | output > ElasticSearch the format it's get stored in is somewhat like:
"_index": "logstash-2012.07.02",
"_type": "stdin",
"_id": "JdRaI5R6RT2do_WhCYM-qg",
"_score": 0.30685282,
"_source": {
"#source": "stdin://dist/",
"#type": "stdin",
"#tags": [
"tag1",
"tag2"
],
"#fields": {},
"#timestamp": "2012-07-02T06:17:48.533000Z",
"#source_host": "dist",
"#source_path": "/",
"#message": "test"
}
I filter/store most of the important information in specific fields, is it possible to leave out the default fields like: #source_path and #source_host? In the near future it's going to store 8 billion logs/month and I would like to run some performance tests with this default fields excluded (I just don't use these fields).
This removes fields from output:
filter {
mutate {
# remove duplicate fields
# this leaves timestamp from message and source_path for source
remove => ["#timestamp", "#source"]
}
}
Some of that will depend on what web interface you are using to view your logs. I'm using Kibana, and a customer logger (c#) that indexes the following:
{
"_index": "logstash-2013.03.13",
"_type": "logs",
"_id": "n3GzIC68R1mcdj6Wte6jWw",
"_version": 1,
"_score": 1,
"_source":
{
"#source": "File",
"#message": "Shalom",
"#fields":
{
"tempor": "hit"
},
"#tags":
[
"tag1"
],
"level": "Info"
"#timestamp": "2013-03-13T21:47:51.9838974Z"
}
}
This shows up in Kibana, and the source fields are not there.
To exclude certain fields you can use prune filter plugin.
filter {
prune {
blacklist_names => [ "#timestamp", "#source" ]
}
}
Prune filter is not a logstash default plugin and must be installed first:
bin/logstash-plugin install logstash-filter-prune
Related
My events in Elasticsearch look something like that (simplified version):
{
"_index": "greatest_index-2023.01",
"_type": "_doc",
"_id": "5BQ8yIUBtpR1CBn8kFyo",
"_version": 1,
"_score": 0,
"_source": {
"#version": "1",
"#timestamp": "2023-01-18T09:35:50.251Z",
"id": "4e80c00dd8e003c8",
"action": "action1"
},
"fields": {
"#timestamp": [
"2023-01-18T09:35:50.251Z"
]
}
}
Basically, the "id" field is common to multiple events. Each id goes through a few "action" field values through time (action1, action2, action3) - only once for each action value.
I'm trying to create a visualization in Kibana that would display the actions each id went through.
If it were a table, it could look something like this :
id
actions
5BQ8yIUBtpR1CBn8kFyo
action1, action 2
pISQ9VDSJVlkqklv9VQ9
action1
cohqBHSQC85AHB67AB2h
action1, action 2, action 3
I tried to use Transforms in the Elasticsearch section of Kibana (v 7.5.0), but it doesn't seem to be the right way.
How would you recommend doing that ?
I'm trying to sort the results based on a numeric field,
Here is my mapping:
{
"elasticie": {
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"number": {
"type": "long"
}
}
}
}
}
I'm using Python, and this is my testing data:
data = [
{'name': 'sElwUYiLXGHaQCKbdxtnvVzqIehfFWkJcPSTurgNoRD', 'number': 8583},
{'name': 'XJEtNsIFfcwHTMhqAvRkiygjbUGzZQPdS', 'number': 8127},
{'name': 'ZIeAGosUKJbjOdylM', 'number': 5862},
{'name': 'HYvcafoXkC', 'number': 7458},
{'name': 'tATJCjNuizOlGckXBpyVqSQL', 'number': 530},
{'name': 'TFYixotjhXzNZPvHnkraRDpAMEImJfqdcVGLC', 'number': 7052},
{'name': 'JCEGfoKDHRrcIkPQSqiVgNshZOBaMdXjAlxwUzmeWLy', 'number': 6168},
{'name': 'IpCTwUAQynSizJtcsuDmbX', 'number': 6492},
{'name': 'fTrcoXSBJNFhAkzWpDMxsEiLmZRvgnC', 'number': 382},
{'name': 'ulVNmqKTpPXfEIdiykhDjMrUGOYazLBFvgnWwsRtJoQbxSe', 'number': 2061}
]
Using following code, I'm creating the index and inserting the data:
from elasticsearch import Elasticsearch
from data import data # the data I've shown above
INDEX = 'elasticie'
es = Elasticsearch('http://127.0.0.1:9200')
for _ in data:
es.index(index=INDEX, body=_)
I'm trying to sort data based on number, asc or desc
Here is what I tried so far:
es.search(index=INDEX, params={'sort': {'number': {'order': 'asc'}})
es.search(index=INDEX, params={'sort': {'number': 'asc'})
es.search(index=INDEX, params={'sort': [('number', 'asc')]})
es.search(index=INDEX, params={'sort': {'number': {'order': 'asc', 'ignore_unmapped': True}})
es.search(index=INDEX, params={'sort': {'number': {'order': 'asc', 'unmapped_type': 'integer'}})
es.search(index=INDEX, params={'sort': {'number': {'order': 'asc', 'unmapped_type': 'long'}})
es.search(index=INDEX, params={'sort': {'number.raw': 'asc'})
Not of the above methods worked for me, The result is the same as the inserted data,
If I assign the above lines to a variable named search_result and print the result using the following code:
for index, result in enumerate(search_result['hits']['hits']):
print(f'{index}. {result["_source"]["number"]}')
I'll get the following result:
0. 8583
1. 8127
2. 5862
3. 7458
4. 530
5. 7052
6. 6168
7. 6492
8. 382
9. 2061
Which is obviously not sorted using number field!!
I don't know what I'm doing wrong, I'm using ElasticSearch 7.6 and Python 3.8
How can I make the sorting results work?
Update
Based on debugging logs, Python is sending a GET request to the following URL using the first method:
http://127.0.0.1:9200/elasticie/_search?sort={%27number%27%3A+{%27order%27%3A+%27asc%27}}
I am not familiar with python, but here is the Elasticsearch JSON query which would sort your documents according to the numbers in desc order. I've tried with your data set and it gives proper results.
Sort Search query
{
"sort": [
{
"number": {
"order": "desc"
}
}
]
}
Results
"hits": [
{
"_index": "so-60598395-sort",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"name": "sElwUYiLXGHaQCKbdxtnvVzqIehfFWkJcPSTurgNoRD",
"number": 8583
},
"sort": [
8583
]
},
{
"_index": "so-60598395-sort",
"_type": "_doc",
"_id": "2",
"_score": null,
"_source": {
"name": "XJEtNsIFfcwHTMhqAvRkiygjbUGzZQPdS",
"number": 8127
},
"sort": [
8127
]
},
{
"_index": "so-60598395-sort",
"_type": "_doc",
"_id": "4",
"_score": null,
"_source": {
"name": "HYvcafoXkC",
"number": 7862
},
"sort": [
7862
]
},
{
"_index": "so-60598395-sort",
"_type": "_doc",
"_id": "3",
"_score": null,
"_source": {
"name": "ZIeAGosUKJbjOdylM",
"number": 5862
},
"sort": [
5862
]
}
EDIT:- Based on the OP comments, python library which he is using supports the POST method of search endpoint, using which he solved the issue. Refer to the comments on the question for more details.
My mistake, I read the documentation and the code functionality using help and dir functions
There is no parameter named sort defined on the Elasticsearch.search method, That's why I thought I should use it as a key within the params dict that it takes,
Thanks to #OpsterElasticSearchNinja and his comment, I realized there is something wrong with either the library or how I'm using it
Sending POST request with sort key as post body worked well,
So I decided to read the whole source code and find out what's going wrong?
#query_params(
#...
"size",
"sort",
#...
)
def search(self, body=None, index=None, doc_type=None, params=None):
# ...
This is how the sort parameter is defined, using a decorator on runtime!!
That's when I tried this code, and somehow it worked!
es.search(index=INDEX, sort=['number:asc'])
I have to upsert bulk records in elastic search index with _id being combination of more than one field from the message. Can I do so. if that can be done then please give me a sample json for the same.
Regards
A sample _id field I am looking for some thing like below
{
"_index": "kpi_aggr",
"_type": "KPIBackChannel",
"_id": "<<<combination of name , period_type>>>",
"_score": 1,
"_source": {
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
}
Naturally, you can specify your own Elasticsearch document ids during a call to the Index API:
PUT kpi_aggr/KPIBackChannel/kpi-v1,w
{
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
You can also do so during a _bulk API call:
POST _bulk
{ "index" : { "_index" : "kpi_aggr", "_type" : "KPIBackChannel", "_id" : "kpi-v1,w" } }
{"name":"kpi-v1","period_type":"w","country":"AL","pg_name":"DENTAL CARE","panel_type":"retail","number_of_records_with_proposal":10000,"number_of_proposals":80000,"overall_number_of_records":2000,"#timestamp":1442162810}
Notice that Elasticsearch will replace the document with the new version.
If you execute these two queries on an empty index, then querying by document id:
GET kpi_aggr/KPIBackChannel/kpi-v1,w
will give you the following:
{
"_index": "kpi_aggr",
"_type": "KPIBackChannel",
"_id": "kpi-v1,w",
"_version": 2,
"found": true,
"_source": {
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
}
Notice "_version": 2, which in our case indicates that a document has been indexed twice, hence performed an "upsert" (but in general is meant to be used for Optimistic Concurrency Control).
Hope that helps!
I search for key word machine4 in my ES . My python client is simply:
result = es.search('machine4', index='machines')
Result look like this
[
{
"_score": 0.13424811,
"_type": "person",
"_id": "2",
"_source": {
"date": "**20180601**",
"deleted": [],
"changed": [
"machine1",
"machine2",
"machine3"
],
"**added**": [
"**machine4**",
"machine5"
]
},
"_index": "contacts"
},
{
"_score": 0.13424811,
"_type": "person",
"_id": "3",
"_source": {
"date": "**20180701**",
"deleted": [
"machine2"
],
"**changed**": [
"machine1",
"**machine4**",
"machine3"
],
"added": [
"machine7"
]
},
"_index": "contacts"
}
]
So we can easily see:
In date 20180601 , machine4 belonged to added.
In date 20180701 , machine4 belonged to changed.
I can write another function to analyze the result. Basically loop through every key,value of each items and check if the searched keyword belong, like this:
for result in search_results['hits']['hits']:
source_result = result['_source']
for key,value in source_result.items():
if 'machine4' in value:
print key
However, I wonder if ES having API to detect which key/mapping/field that the searched keywords belonged to ? In this case is added of the 1st result, and changed in 2nd result
Thank you so much
Alex
The simple answer seems to be that no, Elasticsearch doesn't have a way to do this out of the box, because Lucene doesn't have it, as per this thread
Elasticsearch has the concept of highlights, however. These could be useful, but they do require you to have some idea about which fields the match may be in.
The ES Python search documentation suggests there's no way to do that as a parameter to search, but you could create a custom query and pass it on as the q argument. It would look something like:
q = {"query" : {"match": { "content": "'machine4'" }}, "highlight" : {"fields" : {"added" : {}, "updated": {}}}}
result = es.search(index='machines', q=q)
Hope this is helpful!
I'd like to import a text file in Elasticsearch. The text file contains 3 values per line. After spending several hours of struggling, I didn't get it done. Help is greatly appreciated.
Elasticsearch 5.4.0 with Logstash installed.
Sample data:
username email hash
username email hash
username email hash
username email hash
username email hash
also built a python script but its too slow:
import requests
import json
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
i = 1
with open("my2") as fileobject:
for line in fileobject:
username, email, hash = line.strip('\n').split(' ')
body = {"username": username, "email": email, "password": hash}
es.index(index='dbs', doc_type='db1', id=i, body=body)
i += 1
edit:
Thanks its work but i guess my filter is bad because i want it to look like this:
{
"_index": "logstash-2017.06.01",
"_type": "db",
"_id": "AVxinqK5XRvft8kN7Q6M",
"_version": 1,
"_score": null,
"_source": {
"username": "Marlb0ro",
"email": "Marlb0ro#site.com",
"hash": "123456",
}
and it put the data like this:
{
"_index": "logstash-2017.06.01",
"_type": "logs",
"_id": "AVxinqK5XRvft8kN7Q6M",
"_version": 1,
"_score": null,
"_source": {
"path": "C:/Users/user/Desktop/user/log.txt",
"#timestamp": "2017-06-01T07:46:22.488Z",
"#version": "1",
"host": "DESKTOP-FNGSJ6C",
"message": "username email password",
"tags": [
"_grokparsefailure"
]
},
"fields": {
"#timestamp": [
1496303182488
]
},
"sort": [
1496303182488
]
}
Simply put this in a file called grok.conf:
input {
file {
path => "/path/to/your/file.log"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => {"message" => "%{WORD:username} %{WORD:email} %{WORD:hash}" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
Then run Logstash with bin/logstash -f grok.conf and you should be ok.