I have json data as follows
{
"_index": "project",
"_type": "ts_order_snapshot",
"_id": "1",
"_version": 1,
"exists": true,
"_source": {
"order_id": "100000001",
"order_data": "Order Data:{\"entity_id\":\"1\",\"state\":\"canceled\"}"
}
}
Now I would like to get the list of all the order ids having state as canceled. Any suggestions?
order_data is a serialized, json-encoded string, not a structured json map. If you promoted it to actual json, you could construct a query like ?q=order_data.state:canceled&fields=order_id.
Related
My events in Elasticsearch look something like that (simplified version):
{
"_index": "greatest_index-2023.01",
"_type": "_doc",
"_id": "5BQ8yIUBtpR1CBn8kFyo",
"_version": 1,
"_score": 0,
"_source": {
"#version": "1",
"#timestamp": "2023-01-18T09:35:50.251Z",
"id": "4e80c00dd8e003c8",
"action": "action1"
},
"fields": {
"#timestamp": [
"2023-01-18T09:35:50.251Z"
]
}
}
Basically, the "id" field is common to multiple events. Each id goes through a few "action" field values through time (action1, action2, action3) - only once for each action value.
I'm trying to create a visualization in Kibana that would display the actions each id went through.
If it were a table, it could look something like this :
id
actions
5BQ8yIUBtpR1CBn8kFyo
action1, action 2
pISQ9VDSJVlkqklv9VQ9
action1
cohqBHSQC85AHB67AB2h
action1, action 2, action 3
I tried to use Transforms in the Elasticsearch section of Kibana (v 7.5.0), but it doesn't seem to be the right way.
How would you recommend doing that ?
I have to upsert bulk records in elastic search index with _id being combination of more than one field from the message. Can I do so. if that can be done then please give me a sample json for the same.
Regards
A sample _id field I am looking for some thing like below
{
"_index": "kpi_aggr",
"_type": "KPIBackChannel",
"_id": "<<<combination of name , period_type>>>",
"_score": 1,
"_source": {
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
}
Naturally, you can specify your own Elasticsearch document ids during a call to the Index API:
PUT kpi_aggr/KPIBackChannel/kpi-v1,w
{
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
You can also do so during a _bulk API call:
POST _bulk
{ "index" : { "_index" : "kpi_aggr", "_type" : "KPIBackChannel", "_id" : "kpi-v1,w" } }
{"name":"kpi-v1","period_type":"w","country":"AL","pg_name":"DENTAL CARE","panel_type":"retail","number_of_records_with_proposal":10000,"number_of_proposals":80000,"overall_number_of_records":2000,"#timestamp":1442162810}
Notice that Elasticsearch will replace the document with the new version.
If you execute these two queries on an empty index, then querying by document id:
GET kpi_aggr/KPIBackChannel/kpi-v1,w
will give you the following:
{
"_index": "kpi_aggr",
"_type": "KPIBackChannel",
"_id": "kpi-v1,w",
"_version": 2,
"found": true,
"_source": {
"name": "kpi-v1",
"period_type": "w",
"country": "AL",
"pg_name": "DENTAL CARE",
"panel_type": "retail",
"number_of_records_with_proposal": 10000,
"number_of_proposals": 80000,
"overall_number_of_records": 2000,
"#timestamp": 1442162810
}
}
Notice "_version": 2, which in our case indicates that a document has been indexed twice, hence performed an "upsert" (but in general is meant to be used for Optimistic Concurrency Control).
Hope that helps!
I search for key word machine4 in my ES . My python client is simply:
result = es.search('machine4', index='machines')
Result look like this
[
{
"_score": 0.13424811,
"_type": "person",
"_id": "2",
"_source": {
"date": "**20180601**",
"deleted": [],
"changed": [
"machine1",
"machine2",
"machine3"
],
"**added**": [
"**machine4**",
"machine5"
]
},
"_index": "contacts"
},
{
"_score": 0.13424811,
"_type": "person",
"_id": "3",
"_source": {
"date": "**20180701**",
"deleted": [
"machine2"
],
"**changed**": [
"machine1",
"**machine4**",
"machine3"
],
"added": [
"machine7"
]
},
"_index": "contacts"
}
]
So we can easily see:
In date 20180601 , machine4 belonged to added.
In date 20180701 , machine4 belonged to changed.
I can write another function to analyze the result. Basically loop through every key,value of each items and check if the searched keyword belong, like this:
for result in search_results['hits']['hits']:
source_result = result['_source']
for key,value in source_result.items():
if 'machine4' in value:
print key
However, I wonder if ES having API to detect which key/mapping/field that the searched keywords belonged to ? In this case is added of the 1st result, and changed in 2nd result
Thank you so much
Alex
The simple answer seems to be that no, Elasticsearch doesn't have a way to do this out of the box, because Lucene doesn't have it, as per this thread
Elasticsearch has the concept of highlights, however. These could be useful, but they do require you to have some idea about which fields the match may be in.
The ES Python search documentation suggests there's no way to do that as a parameter to search, but you could create a custom query and pass it on as the q argument. It would look something like:
q = {"query" : {"match": { "content": "'machine4'" }}, "highlight" : {"fields" : {"added" : {}, "updated": {}}}}
result = es.search(index='machines', q=q)
Hope this is helpful!
I'm use elastic search for about one month and i've found one thing one query fuzzie that i can't understand.
The scenario is i've a set of users on a type and index almost 10.000 items, and i want to search for username, and return all the items that match with search string in a fuzzy mode, for example my user is "masterviana" if i search by only with text "mastervi" i expect to see the masterviana at the top of results using a fuzzy query right?
"fuzzy" : {
"public_name" : {
"value" : "mastervi",
"boost" : 1.0,
"fuzziness" : 2,
"prefix_length" : 0,
"max_expansions": 100
}
}
However i'm not seeing my username (masterviana) at the first page and also i see usernames that are "less similar" like my query string, i'll show the only the first 5 hits for not extended to much the post
{
"_index": "username",
"_type": "username",
"_id": "2061|FZ4y1t042482S3EqobiVllmv00",
"_score": 9.198499,
"_source": {
"public_name": "masterv",
"bbid": "FZ4y1t042482S3EqobiVllmv00",
"hash": 2061,
"avata": "http://goo.gl/4CRt3v"
}
},
{
"_index": "username",
"_type": "username",
"_id": "2048|r0I5XZ31076phruMS1gu9Hjv00",
"_score": 5.9688096,
"_source": {
"public_name": "project--master",
"bbid": "r0I5XZ31076phruMS1gu9Hjv00",
"hash": 2048,
"avata": "http://goo.gl/4CRt3vr"
}
},
{
"_index": "username",
"_type": "username",
"_id": "1980|W5Wal166832UV5oCqUH9Vjcv00",
"_score": 5.7984095,
"_source": {
"public_name": "masterjv",
"bbid": "W5Wal166832UV5oCqUH9Vjcv00",
"hash": 1980,
"avata": "http://goo.gl/4CRt3v"
}
},
{
"_index": "username",
"_type": "username",
"_id": "2108|Kufhm899338GPWHsuoei1HOv00",
"_score": 5.7984095,
"_source": {
"public_name": "master25",
"bbid": "Kufhm899338GPWHsuoei1HOv00",
"hash": 2108,
"avata": "http://goo.gl/4CRt3v"
}
},
{
"_index": "username",
"_type": "username",
"_id": "1952|AtPw2a97575sC5JT406msOXv00",
"_score": 5.7984095,
"_source": {
"public_name": "masterpiz",
"bbid": "AtPw2a97575sC5JT406msOXv00",
"hash": 1952,
"avata": "http://goo.gl/4CRt3v"
}
},
AS you can see i'm getting at top 1. masterv 2. project-master i think my query "mastervi" is more close to "masterviana" that for example "masterv" or "project-master"
One more thing if i search with exactly the same text "masterviana" i'm getting only this item
The ranking is a blend of edit distance and (often unhelpfully) how rare a term is.
I'm not sure which of these is to blame in this case but the term scarcity ranking is a long-standing Lucene issue. There is a work-around in elasticsearch with FuzzyLikeThisQuery but that might not be around for much longer so this has accelerated the need to fix Lucene (see here for background https://github.com/elastic/elasticsearch/pull/10391 )
After processing data with: input | filter | output > ElasticSearch the format it's get stored in is somewhat like:
"_index": "logstash-2012.07.02",
"_type": "stdin",
"_id": "JdRaI5R6RT2do_WhCYM-qg",
"_score": 0.30685282,
"_source": {
"#source": "stdin://dist/",
"#type": "stdin",
"#tags": [
"tag1",
"tag2"
],
"#fields": {},
"#timestamp": "2012-07-02T06:17:48.533000Z",
"#source_host": "dist",
"#source_path": "/",
"#message": "test"
}
I filter/store most of the important information in specific fields, is it possible to leave out the default fields like: #source_path and #source_host? In the near future it's going to store 8 billion logs/month and I would like to run some performance tests with this default fields excluded (I just don't use these fields).
This removes fields from output:
filter {
mutate {
# remove duplicate fields
# this leaves timestamp from message and source_path for source
remove => ["#timestamp", "#source"]
}
}
Some of that will depend on what web interface you are using to view your logs. I'm using Kibana, and a customer logger (c#) that indexes the following:
{
"_index": "logstash-2013.03.13",
"_type": "logs",
"_id": "n3GzIC68R1mcdj6Wte6jWw",
"_version": 1,
"_score": 1,
"_source":
{
"#source": "File",
"#message": "Shalom",
"#fields":
{
"tempor": "hit"
},
"#tags":
[
"tag1"
],
"level": "Info"
"#timestamp": "2013-03-13T21:47:51.9838974Z"
}
}
This shows up in Kibana, and the source fields are not there.
To exclude certain fields you can use prune filter plugin.
filter {
prune {
blacklist_names => [ "#timestamp", "#source" ]
}
}
Prune filter is not a logstash default plugin and must be installed first:
bin/logstash-plugin install logstash-filter-prune