I have the following schema elastic search:
I have abbreviated most of it for ease of reading
{
"took": 56,
"timed_out": false,
"hits": {
"hits": [
{
"_id": "2FREAL%2FShaarHanegev%2Faxis-ACCC8E43E0C6%2F20160314%2F16%2F20160314_164253_E23D_ACCC8E43E0C6%2F20160314_16%2F20160314_164253_8DA7_ACCC8E43E0C6.mkv%3A26.153_1508_1439_1763_1440",
"_source": {
"path": "2FREAL%2FShaar%2Faxis-ACCC8E43E0C6%2F20160314%2F16%2F20160314_164253_E23D_ACCC8E43E0C6%2F20160314_16%2F20160314_164253_8DA7_ACCC8E43E0C6.mkv%3A26.153_1508_1439_1763_1440",
"frameAttributes": {
"trackerId": "OB7E600",
"identities": [
{
"_id": "Abel_Nachos_034841460",
"_score": 0.451906
},
{
"_id": "judas_acorn_10000897",
"_score": 0.430024
},
{
"_id": "regenald_barbish_10000452",
"_score": 0.41979
},
{
"_id": "Matthew_Gordon_Douglas_0631B#1",
"_score": 0.412086
},
{
"_id": "NewYork_NYC_Gina_Lob_10000566",
"_score": 0.407909
}
]
},
"originalVideoFullPath": "REAL/Shaar/axis-ACCC8E43E0C6/20160314/16/20160314_164253_E23D_ACCC8E43E0C6/20160314_16/20160314_164253_8DA7_ACCC8E43E0C6.mkv"
}
},
{
"_id": "2FREAL%2FShaar%2Faxis-ACCC8E43E0C6%2F20160314%2F16%2F20160314_164253_E23D_ACCC8E43E0C6%2F20160314_16%2F20160314_164253_8DA7_ACCC8E43E0C6.mkv%3A26.393_1454_1457_1711_1454",
"_source": {
"path": "2FREAL%2FShaarHanegev%2Faxis-ACCC8E43E0C6%2F20160314%2F16%2F20160314_164253_E23D_ACCC8E43E0C6%2F20160314_16%2F20160314_164253_8DA7_ACCC8E43E0C6.mkv%3A26.393_1454_1457_1711_1454",
"frameAttributes": {
"trackerId": "OB7E600",
"identities": [
{
"_id": "Levon_Ayrapetyan_10000036",
"_score": 0.432837
},
{
"_id": "Patrick_sole_10001145",
"_score": 0.425161
},
{
"_id": "TAI_YUNG_LEE_LOUIE_10002450",
"_score": 0.404628
},
{
"_id": "Izak_Gold_10000159",
"_score": 0.400651
},
{
"_id": "Takahiro_Friend_10001663",
"_score": 0.392012
}
]
},
"clockwiseRotation": 90,
"numberOfFrames": 0,
"originalVideoFullPath": "REAL/Shaar/axis-ACCC8E43E0C6/20160314/16/20160314_164253_E23D_ACCC8E43E0C6/20160314_16/20160314_164253_8DA7_ACCC8E43E0C6.mkv"
}
}
]
}
}
Edit: I added the abbreviated mapping as requested
{
"index1": {
"mappings": {
"objects": {
"properties": {
"_results": {
"type": "object"
},
"cameraId": {
"type": "string"
},
"cameraType": {
"type": "string"
},
"date": {
"type": "date",
"format": "dateOptionalTime"
},
"frameAttributes": {
"properties": {
"identities": {
"properties": {
"_id": {
"type": "string"
},
"_score": {
"type": "double"
}
}
},
"testPassed": {
"type": "boolean"
}
}
},
"fraudType": {
"type": "string"
},
"index": {
"type": "long"
},
"path": {
"type": "string"
},
"siteName": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
I am trying to write a query that will find all the videos (id) where the Person {NewYork_NYC_Gina_Lob_10000566} is located.
I don't see mapping, so this is a query ignoring possible nested objects or parent-child relationships.
GET your_index/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"frameAttributes.identities._id": "NewYork_NYC_Gina_Lob_10000566"
}
}
]
}
}
}
Related
I want to apply order for sorting in Elasticsearch. Unfortunately, common standard sort can't sort in next way:
special chars
numbers
alphabetic(aA-zZ)
My mapping:
PUT /sorting_by_name
{
"mappings": {
"sorting_by_name": {
"properties": {
"name": {
"type": "keyword"
}
}
}
}
}
I am querying Elasticsearch as follows:
GET /sorting_by_name/_search
{
"query": {
"match_all": {}
},
"sort": [
{ "name": { "order": "asc" } }
]
}
And I would want to get the following result:
[
{
"_id": "AXe1Y7vv7OkJxgqzrpj9",
"_source": {
"name": "{{{{ A {{{ sdasd"
}
},
{
"_id": "AXe1Y7vv7OkJxgqzrpj9",
"_source": {
"name": "{{{{ A {{{ sdasd"
}
},
{
"_id": "AXe1QBZz7OkJxgqzrpj2",
"_source": {
"name": "11asdasd"
}
}
{
"_id": "AXe1Y6id7OkJxgqzrpj8",
"_source": {
"name": "A {{{ sdasd"
}
},
{
"_id": "AXe1QFwn7OkJxgqzrpj4",
"_source": {
"name": "asdasd"
}
},
{
"_id": "AXe1ZOGe7OkJxgqzrpkC",
"_source": {
"name": "Ooopopl"
}
},
{
"_id": "AXe1ZM4V7OkJxgqzrpkB",
"_source": {
"name": "ooopopl"
}
}
]
But now I get the following result:
[
{
"_id": "AXe1Y7vv7OkJxgqzrpj9",
"_source": {
"name": "{{{{ A {{{ sdasd"
}
},
{
"_id": "AXe1QBZz7OkJxgqzrpj2",
"_source": {
"name": "11asdasd"
}
},
{
"_id": "AXe1Y6id7OkJxgqzrpj8",
"_source": {
"name": "A {{{ sdasd"
}
},
{
"_id": "AXe1ZOGe7OkJxgqzrpkC",
"_source": {
"name": "Ooopopl"
}
},
{
"_id": "AXe1QFwn7OkJxgqzrpj4",
"_source": {
"name": "asdasd"
}
},
{
"_id": "AXe1ZM4V7OkJxgqzrpkB",
"_source": {
"name": "ooopopl"
}
},
{
"_id": "AXe1Y7vv7OkJxgqzrpj9",
"_source": {
"name": "{{{{ A {{{ sdasd"
}
}
]
How would I get the expected result?
we have a mapping with nested objects.
We use it for storing variants in an ecommerce environment.
To build filters in ecommerce we check agg bucket counts.
What happens now is that we get agg counts for every variant that this matching.
But i would like to count only once per parent (even if more than one children match).
here is our mapping:
{
"products": {
"mappings": {
"dynamic": "false",
"properties": {
"product_id": {
"type": "keyword"
},
"variants": {
"type": "nested",
"properties": {
"brand": {
"type": "keyword"
},
"color": {
"type": "keyword"
}
}
}
}
}
}
example of documents:
{
"_index": "products",
"_type": "_doc",
"_id": "5e42f759de235d5b42e6e35524141bd2cfcbc5d0",
"_score": 1.0,
"_source": {
"product_id": "5e42f759de235d5b42e6e35524141bd2cfcbc5d0",
"variants": [
{
"color": [
"Black"
]
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann",
},
{
"color": [],
"brand": "Fackelmann"
}
]
}
},
{
"_index": "products",
"_type": "_doc",
"_id": "768af68018654b04f20d32003348ee9bd81d9f65",
"_score": 1.0,
"_source": {
"product_id": "768af68018654b04f20d32003348ee9bd81d9f65",
"variants": [
{
"color": [
"Grey"
]
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann",
},
{
"color": [],
"brand": "IKEA"
}
]
}
}
in this example we get those counts for aggs:
brand:Fackelmann[7]
brand:IKEA[1]
color:Grey[1]
color:Black[1]
What i expect to get is:
brand:Fackelmann[2]
brand:IKEA[1]
color:Grey[1]
color:Black[1]
I am trying to search for all the unique names in the index test_nested.
GET test_nested/_mappings
{
"test_nested": {
"mappings": {
"my_type": {
"properties": {
"group": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user": {
"type": "nested",
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
}
GET test_nested/_search
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 1,
"hits": [
{
"_index": "test_nested",
"_type": "my_type",
"_id": "AWG5iVBz4bQsVnslc9gL",
"_score": 1,
"_source": {
"group": "fans",
"user": [
{
"name": "Linux"
},
{
"name": "Android (operating system)"
},
{
"name": "Widows 10"
}
]
}
},
{
"_index": "test_nested",
"_type": "my_type",
"_id": "AWG5ieKW4bQsVnslc9gM",
"_score": 1,
"_source": {
"group": "fans",
"user": [
{
"name": "Bitcoin"
},
{
"name": "PHP"
},
{
"name": "Microsoft Windows"
}
]
}
},
{
"_index": "test_nested",
"_type": "my_type",
"_id": "AWG5irrV4bQsVnslc9gN",
"_score": 1,
"_source": {
"group": "fans",
"user": [
{
"name": "Windows XP"
}
]
}
},
{
"_index": "test_nested",
"_type": "my_type",
"_id": "1",
"_score": 1,
"_source": {
"group": "fans",
"user": [
{
"name": "iOS"
},
{
"name": "Android (operating system)"
},
{
"name": "Widows 10"
},
{
"name": "Widows XP"
}
]
}
}
]
}
}
I want all the unique names for a term. i.e. if I search for "wi"* then I should get [Microsoft Windows, Widows 10, Windows XP]
I don't know exactly what you mean but I use that query to list all statuses:
GET order/default/_search
{
"size": 0,
"aggs": {
"status_terms": {
"terms": {
"field": "status.keyword",
"missing": "N/A",
"min_doc_count": 0,
"order": {
"_key": "asc"
}
}
}
}
}
My model has status field and that query lists all statuses.
This is bucket aggregations
One of fields in result is:
sum_other_doc_count - Elastic returns the top unique terms. So if you have many different terms then some of them will not appear in the results. This field is a sum of documents which will not be a part of the response.
For nested objects try to read and use Nested Query docs
I found the solution. Hope it helps someone.
GET record_new/_search
{
"size": 0,
"query": {
"term": {
"software_tags": {
"value": "windows"
}
}
},
"aggs": {
"software_tags": {
"terms": {
"field": "software_tags.keyword",
"include" : ".*Windows.*",
"size": 10000,
"order": {
"_count": "desc"
}
}
}
}
}
Can somebody help me with Alerting Via X-Pack for Energy monitoring system project? The main problem here is I can't collect the 'Value' data from the database, as I want to compare it later with the upper and the lower threshold.
So here is the index:
PUT /test-1
{
"mappings": {
"Test1": {
"properties": {
"Value": {
"type": "integer"
},
"date": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
},
"UpperThreshold": {
"type": "integer"
},
"LowerThreshold": {
"type": "integer"
}
}
}
}
}
Here is the example of the input:
POST /test-1/Test1
{
"Value": "500",
"date": "2017-06-13T16:20:00.000Z",
"UpperThreshold":"450",
"LowerThreshold": "380"
}
This is my alerting code
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"types": [],
"body": {
"query": {
"match": {
"message": "error"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"<account#gmail.com>"
],
"subject": "Watcher Notification",
"body": {
"text": "{{ctx.payload.hits.total}} error logs found"
}
}
}
}
}
Here is the response I got from the alerting plugin
{
"watch_id": "Alerting-Test",
"state": "execution_not_needed",
"_status": {
"state": {
"active": true,
"timestamp": "2017-07-26T15:27:35.497Z"
},
"last_checked": "2017-07-26T15:27:38.625Z",
"actions": {
"logging": {
"ack": {
"timestamp": "2017-07-26T15:27:35.497Z",
"state": "awaits_successful_execution"
}
}
}
},
"trigger_event": {
"type": "schedule",
"triggered_time": "2017-07-26T15:27:38.625Z",
"schedule": {
"scheduled_time": "2017-07-26T15:27:38.175Z"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"test-1"
],
"types": [
"Test1"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.hits.0.Value": {
"gt": 450
}
}
},
"metadata": {
"name": "Alerting-Test"
},
"result": {
"execution_time": "2017-07-26T15:27:38.625Z",
"execution_duration": 0,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 5,
"failed": 0,
"successful": 5
},
"hits": {
"hits": [
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-22T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "350",
"UpperThreshold": "450"
},
"_id": "AV1-1P3lArbJ1tbnct4e",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-22T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "4100",
"UpperThreshold": "450"
},
"_id": "AV1-1Sq0ArbJ1tbnct4v",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-24T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "450",
"UpperThreshold": "450"
},
"_id": "AV1-1eLJArbJ1tbnct6G",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T00:00:00.000Z",
"LowerThreshold": "380",
"Value": "400",
"UpperThreshold": "450"
},
"_id": "AV1-1VUzArbJ1tbnct5A",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1X4FArbJ1tbnct5R",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1YySArbJ1tbnct5T",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-26T00:00:00.000Z",
"LowerThreshold": "380",
"Value": "4700",
"UpperThreshold": "450"
},
"_id": "AV1-1mflArbJ1tbnct67",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-26T06:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1oluArbJ1tbnct7M",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-21T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "400",
"UpperThreshold": "450"
},
"_id": "AV1-1IrZArbJ1tbnct3r",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-21T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "440",
"UpperThreshold": "450"
},
"_id": "AV1-1LwzArbJ1tbnct38",
"_score": 1
}
],
"total": 20,
"max_score": 1
},
"took": 1,
"timed_out": false
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"test-1"
],
"types": [
"Test1"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
**"ctx.payload.hits.hits.0.Value": null**
}
}
},
"actions": []
},
"messages": []
}
Really appreciate for your help!!
I have a list of console platforms that I'm sorting in Elasticsearch.
Here is the mapping for the "name" field:
{
"name": {
"type": "multi_field",
"fields": {
"name": {
"type": "string",
"index": "analyzed"
},
"sort_name": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
When I execute the following query
{
"query": {
"match_all": {}
},
"sort": [
{
"name.sort_name": { "order": "asc" }
}
],
"fields": ["name"]
}
I get these results:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"failed": 0
},
"hits": {
"total": 17,
"max_score": null,
"hits": [
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602489",
"_score": null,
"fields": {
"name": "GameCube"
},
"sort": [
"GameCube"
]
},
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602490",
"_score": null,
"fields": {
"name": "Gameboy Advance"
},
"sort": [
"Gameboy Advance"
]
},
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602498",
"_score": null,
"fields": {
"name": "Nintendo 3DS"
},
"sort": [
"Nintendo 3DS"
]
},
...remove for brevity ...
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602493",
"_score": null,
"fields": {
"name": "Xbox 360"
},
"sort": [
"Xbox 360"
]
},
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602502",
"_score": null,
"fields": {
"name": "Xbox One"
},
"sort": [
"Xbox One"
]
},
{
"_index": "platforms",
"_type": "platform",
"_id": "1393602497",
"_score": null,
"fields": {
"name": "iPhone/iPod"
},
"sort": [
"iPhone/iPod"
]
}
]
}
Everything is sorted as expected except the iPhone/iPod result is at the end (instead of after GameBoy Advance) - why does the / in the name have an effect on the sorting?
Thanks
Okay so I discovered the reason wasn't anything to do with the /. ES will sort by capital letters then lower case letters.
I added a custom analyzer to the settings of the index creation:
{
"analysis": {
"analyzer": {
"sortable": {
"tokenizer": "keyword",
"filter": [
"lowercase"
]
}
}
}
}
Then in the field mapping I added 'analyzer': 'sortable' to the sort_name multi field.
Use Normalizer with keyword to handle the sort
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-normalizers.html#analysis-normalizers
PUT index_name
{
"settings": {
"analysis": {
"normalizer": {
"my_normalizer": {
"type": "custom",
"char_filter": ["quote"],
"filter": ["lowercase", "asciifolding"]
}
}
}
},
"mappings": {
"properties": {
"name": {
"type": "keyword",
"normalizer": "my_normalizer"
}
}
}
}
Search query may be modified like this
{
"query": {
"match_all": {}
},
"sort": [
{
"name.sort_name": { "order": "asc" }
}
],
"fields": "name.keyword"
}
According to https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-normalizers.html (ElasticSearch 7.16) ...
Elasticsearch ships with a lowercase built-in normalizer.
So you can define an additional field (in the example below named "lowersortable"):
PUT /myindex/_mapping
{
"properties": {
"myproperty": {
"type": "text",
"fields": {
"lowersortable": {
"type": "keyword",
"normalizer": "lowercase"
}
}
}
}
}
... and use this field myproperty.lowersortable for sorting in the search query.