I have the following mapping in ES:
"mappings": {
"products": {
"properties": {
"product": {
"type" : "nested",
"properties": {
"features": {
"type": "nested"
},
"sitedetails": {
"type": "nested"
}
}
}
}
}
}
and then 3 products like this:
"hits": [
{
"_index": "catalog",
"_type": "products",
"_id": "AVNE8F4mFYOWvB4rMqdO",
"_score": 1,
"_source": {
"product": {
"ean": "abc",
"features": {
"productType": "DVD player"
},
"color": "Black",
"manufacturer": "Sony",
"sitedetails": [
{
"name": "amazon.com",
"sku": "zzz",
"url": "http://www.amazon.com/dp/zzz"
}
],
"category": "Portable DVD Players"
}
}
},
{
"_index": "catalog",
"_type": "products",
"_id": "AVNE8XkXFYOWvB4rMqdQ",
"_score": 1,
"_source": {
"product": {
"ean": "def",
"features": {
"ProductType": "MP3 player"
},
"color": "Black",
"manufacturer": "LG",
"sitedetails": [
{
"name": "amazon.com",
"sku": "aaa",
"url": "http://www.amazon.com/dp/aaa"
}
],
"category": "MP3 Players"
}
}
},
{
"_index": "catalog",
"_type": "products",
"_id": "AVNIh-xVWwxj6Cz_r8AT",
"_score": 1,
"_source": {
"product": {
"ean": "abc",
"features": {
"productType": "DVD player"
},
"color": "White",
"manufacturer": "Sony",
"sitedetails": [
{
"name": "amazon.com",
"sku": "ggg",
"url": "http://www.amazon.com/dp/ggg"
}
],
"category": "Portable DVD Players"
}
}
}
]
I need to display on the UI side 2 filters, one for Manufacturer and one for website.
How can I aggregate on product.manufacturer and product.sitedetails.name?
tnx!
Figured it out:
GET /catalog/products/_search
{
"aggs": {
"byManufacturer": {
"nested": {
"path": "product"
},
"aggs": {
"byManufacturer": {
"terms": {
"field": "product.manufacturer"
}
}
}
},
"bySeller": {
"nested": {
"path": "product.sitedetails"
},
"aggs": {
"bySeller": {
"terms": {
"field": "product.sitedetails.name"
}
}
}
}
}
}
Related
I have products in my index. Documents are basically structured like these:
{
"_id": "product",
"_source": {
...
"type": "product",
"id": 1,
"mainTaxon": {
"name": "T-SHIRT",
},
"attributes": [
{
"code": "name",
"name": "Name",
"value": [
"BANANA T-SHIRT"
],
"score": 50
},
]
}
},
{
"_id": "product",
"_source": {
...
"type": "product",
"id": 2,
"mainTaxon": {
"name": "JEANS",
},
"attributes": [
{
"code": "name",
"name": "Name",
"value": [
"BANANA JEANS"
],
"score": 50
},
]
}
}
}
When I search for 'BANANA' I would prioritize products with mainTaxon different from JEANS. So, every product with the mainTaxon name T_SHIRT or something else would be listed before products with mainTaxon JEANS.
You can use boosting query to prioritize documents
{
"query": {
"boosting": {
"positive": {
"match": {
"attributes.value": "banana"
}
},
"negative": {
"match": {
"mainTaxon.name": "JEANS"
}
},
"negative_boost": 0.5
}
}
}
Search Result will be
"hits": [
{
"_index": "67164768",
"_type": "_doc",
"_id": "1",
"_score": 0.5364054,
"_source": {
"type": "product",
"id": 1,
"mainTaxon": {
"name": "T-SHIRT"
},
"attributes": [
{
"code": "name",
"name": "Name",
"value": [
"BANANA T-SHIRT"
],
"score": 50
}
]
}
},
{
"_index": "67164768",
"_type": "_doc",
"_id": "2",
"_score": 0.32743764,
"_source": {
"type": "product",
"id": 2,
"mainTaxon": {
"name": "JEANS"
},
"attributes": [
{
"code": "name",
"name": "Name",
"value": [
"BANANA JEANS"
],
"score": 50
}
]
}
}
]
I have created the index
PUT ten2
{
"mappings": {
"documents": {
"properties": {
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},"uid": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"publish_details": {
"type": "nested",
"properties": {
"environment": {
"type": "keyword"
},
"locale": {
"type": "keyword"
},
"time": {
"type": "date"
},
"version": {
"type": "integer"
}
}
}
}
}
}
}
and added documents into it. here is the list of documents:
[{
"_index": "ten2",
"_type": "documents",
"_id": "blt69b62b48bbed1fb6_en-us",
"_source": {
"publish_details": [{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:11:25.276Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
},
{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:11:25.276Z",
"locale": "hi-in",
"user": "bltaadab2f531206e9d",
"version": 1
}
],
"title": "Entry 1",
"uid": "blt69b62b48bbed1fb6"
}
},
{
"_index": "ten2",
"_type": "documents",
"_id": "blt69b62b48bbed1fb6_mr-in",
"_source": {
"publish_details": [{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:12:35.467Z",
"locale": "mr-in",
"user": "bltaadab2f531206e9d",
"version": 1
}],
"title": "Entry 3",
"uid": "blt69b62b48bbed1fb6"
}
},
{
"_index": "ten2",
"_type": "documents",
"_id": "blt4044c5198122a3ed_en-us",
"_source": {
"publish_details": [{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
},{
"environment": "blt603fe91adbdcff6690",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
}],
"title": "Entry 10",
"uid": "blt4044c5198122a3ed"
}
}
]
and I want the following result
[
{
"_index": "ten2",
"_type": "documents",
"_id": "blt4044c5198122a3ed_en-us",
"_source": {
"publish_details": [{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
},{
"environment": "blt603fe91adbdcff6690",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
}],
"title": "Entry 10",
"uid": "blt4044c5198122a3ed"
}
}
]
I am using the following query to get the result
GET ten2/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"bool": {
"must": [{
"nested": {
"path": "publish_details",
"query": {
"term": {
"publish_details.environment": "blt603fe91adbdcff66"
}
}
}
}, {
"nested": {
"path": "publish_details",
"query": {
"term": {
"publish_details.locale": "en-us"
}
}
}
}, {
"nested": {
"path": "publish_details",
"query": {
"term": {
"publish_details.locale": "hi-in"
}
}
}
}, {
"nested": {
"path": "publish_details",
"query": {
"term": {
"publish_details.locale": "mr-in"
}
}
}
}]
}
}]
}
}
}
}
}
kindly help me a query to get expected result. First two dicuemtns having same uid only publish_details.locale is different.I am using must query within must_not to get result, currently I am getting all three documents but I want only last one. I have million documwnts.
To know more about Bool queries refer to this official documentation
Adding a working example with your mapping, index data, and with the search query
Search Query:
{
"query": {
"nested": {
"path": "publish_details",
"query": {
"bool": {
"must": [
{
"match": {
"publish_details.locale": "en-us"
}
}
],
"must_not": [
{
"match": {
"publish_details.environment": "blt603fe91adbdcff66"
}
},
{
"match": {
"publish_details.locale": "hi-in"
}
},
{
"match": {
"publish_details.locale": "mr-in"
}
}
]
}
},
"inner_hits": {
}
}
}
}
Search Result :
"hits": [
{
"_index": "test",
"_type": "_doc",
"_id": "3",
"_score": 0.53899646,
"_source": {
"publish_details": [
{
"environment": "blt603fe91adbdcff66",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
},
{
"environment": "blt603fe91adbdcff6690",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
}
],
"title": "Entry 10",
"uid": "blt4044c5198122a3ed"
},
"inner_hits": {
"publish_details": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.53899646,
"hits": [
{
"_index": "test",
"_type": "_doc",
"_id": "3",
"_nested": {
"field": "publish_details",
"offset": 1
},
"_score": 0.53899646,
"_source": {
"environment": "blt603fe91adbdcff6690",
"time": "2020-06-24T12:10:46.430Z",
"locale": "en-us",
"user": "bltaadab2f531206e9d",
"version": 1
}
}
]
}
}
}
}
]
To know more about inner hits refer to this documentation
The above query returns only the third document, thus satisfying the conditions of the search query. In the Inner Hits, only one part of the third document is returning, and the part which is matching blt603fe91adbdcff66 is discarded.
we have a mapping with nested objects.
We use it for storing variants in an ecommerce environment.
To build filters in ecommerce we check agg bucket counts.
What happens now is that we get agg counts for every variant that this matching.
But i would like to count only once per parent (even if more than one children match).
here is our mapping:
{
"products": {
"mappings": {
"dynamic": "false",
"properties": {
"product_id": {
"type": "keyword"
},
"variants": {
"type": "nested",
"properties": {
"brand": {
"type": "keyword"
},
"color": {
"type": "keyword"
}
}
}
}
}
}
example of documents:
{
"_index": "products",
"_type": "_doc",
"_id": "5e42f759de235d5b42e6e35524141bd2cfcbc5d0",
"_score": 1.0,
"_source": {
"product_id": "5e42f759de235d5b42e6e35524141bd2cfcbc5d0",
"variants": [
{
"color": [
"Black"
]
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann",
},
{
"color": [],
"brand": "Fackelmann"
}
]
}
},
{
"_index": "products",
"_type": "_doc",
"_id": "768af68018654b04f20d32003348ee9bd81d9f65",
"_score": 1.0,
"_source": {
"product_id": "768af68018654b04f20d32003348ee9bd81d9f65",
"variants": [
{
"color": [
"Grey"
]
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann"
},
{
"color": [],
"brand": "Fackelmann",
},
{
"color": [],
"brand": "IKEA"
}
]
}
}
in this example we get those counts for aggs:
brand:Fackelmann[7]
brand:IKEA[1]
color:Grey[1]
color:Black[1]
What i expect to get is:
brand:Fackelmann[2]
brand:IKEA[1]
color:Grey[1]
color:Black[1]
Can somebody help me with Alerting Via X-Pack for Energy monitoring system project? The main problem here is I can't collect the 'Value' data from the database, as I want to compare it later with the upper and the lower threshold.
So here is the index:
PUT /test-1
{
"mappings": {
"Test1": {
"properties": {
"Value": {
"type": "integer"
},
"date": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
},
"UpperThreshold": {
"type": "integer"
},
"LowerThreshold": {
"type": "integer"
}
}
}
}
}
Here is the example of the input:
POST /test-1/Test1
{
"Value": "500",
"date": "2017-06-13T16:20:00.000Z",
"UpperThreshold":"450",
"LowerThreshold": "380"
}
This is my alerting code
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"types": [],
"body": {
"query": {
"match": {
"message": "error"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"<account#gmail.com>"
],
"subject": "Watcher Notification",
"body": {
"text": "{{ctx.payload.hits.total}} error logs found"
}
}
}
}
}
Here is the response I got from the alerting plugin
{
"watch_id": "Alerting-Test",
"state": "execution_not_needed",
"_status": {
"state": {
"active": true,
"timestamp": "2017-07-26T15:27:35.497Z"
},
"last_checked": "2017-07-26T15:27:38.625Z",
"actions": {
"logging": {
"ack": {
"timestamp": "2017-07-26T15:27:35.497Z",
"state": "awaits_successful_execution"
}
}
}
},
"trigger_event": {
"type": "schedule",
"triggered_time": "2017-07-26T15:27:38.625Z",
"schedule": {
"scheduled_time": "2017-07-26T15:27:38.175Z"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"test-1"
],
"types": [
"Test1"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.hits.0.Value": {
"gt": 450
}
}
},
"metadata": {
"name": "Alerting-Test"
},
"result": {
"execution_time": "2017-07-26T15:27:38.625Z",
"execution_duration": 0,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 5,
"failed": 0,
"successful": 5
},
"hits": {
"hits": [
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-22T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "350",
"UpperThreshold": "450"
},
"_id": "AV1-1P3lArbJ1tbnct4e",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-22T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "4100",
"UpperThreshold": "450"
},
"_id": "AV1-1Sq0ArbJ1tbnct4v",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-24T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "450",
"UpperThreshold": "450"
},
"_id": "AV1-1eLJArbJ1tbnct6G",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T00:00:00.000Z",
"LowerThreshold": "380",
"Value": "400",
"UpperThreshold": "450"
},
"_id": "AV1-1VUzArbJ1tbnct5A",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1X4FArbJ1tbnct5R",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-23T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1YySArbJ1tbnct5T",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-26T00:00:00.000Z",
"LowerThreshold": "380",
"Value": "4700",
"UpperThreshold": "450"
},
"_id": "AV1-1mflArbJ1tbnct67",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-26T06:00:00.000Z",
"LowerThreshold": "380",
"Value": "390",
"UpperThreshold": "450"
},
"_id": "AV1-1oluArbJ1tbnct7M",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-21T12:00:00.000Z",
"LowerThreshold": "380",
"Value": "400",
"UpperThreshold": "450"
},
"_id": "AV1-1IrZArbJ1tbnct3r",
"_score": 1
},
{
"_index": "test-1",
"_type": "Test1",
"_source": {
"date": "2017-07-21T18:00:00.000Z",
"LowerThreshold": "380",
"Value": "440",
"UpperThreshold": "450"
},
"_id": "AV1-1LwzArbJ1tbnct38",
"_score": 1
}
],
"total": 20,
"max_score": 1
},
"took": 1,
"timed_out": false
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"test-1"
],
"types": [
"Test1"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
**"ctx.payload.hits.hits.0.Value": null**
}
}
},
"actions": []
},
"messages": []
}
Really appreciate for your help!!
I tried to get my head wrapped around nested queries but I can't get this to work.
I have 2 items in ES that look like this
{
"_index": "catalog",
"_type": "products",
"_source": {
"product": {
"ean": "abc",
"features": {
"Product Type": "DVD player",
},
"color": "Black",
"manufacturer": "Sony",
"sitedetails": [
{
"name": "amazon.com",
"sku": "zzz",
"url": "http://www.amazon.com/dp/zzz"
}
],
"category": "Portable DVD Players"
}
}
},
{
"_index": "catalog",
"_type": "products",
"_source": {
"product": {
"ean": "def",
"features": {
"Product Type": "MP3 player",
},
"color": "Black",
"manufacturer": "LG",
"sitedetails": [
{
"name": "amazon.com",
"sku": "aaa",
"url": "http://www.amazon.com/dp/aaa"
}
],
"category": "MP3 Players"
}
}
}
2 questions:
What is the curl to get sku = zzz?
What is the curl to get both items on a search for "players"?
tnx!
Heyy bro, lets do the magic.
First , you need an mapping including your nested objects, like this
curl -XPUT "http://192.168.99.100:9200/catalog" -d'
{
"mappings": {
"products": {
"properties": {
"product": {
"type": "nested",
"properties": {
"features": {
"type":"nested"
},
"sitedetails": {
"type": "nested"
}
}
}
}
}
}
}'
After that, lets insert your data (change your Product Type to product_type)
curl -XPOST "http://192.168.99.100:9200/catalog/products" -d'
{
"product": {
"ean": "abc",
"features": {
"product_type": "DVD player"
},
"color": "Black",
"manufacturer": "Sony",
"sitedetails": [
{
"name": "amazon.com",
"sku": "zzz",
"url": "http://www.amazon.com/dp/zzz"
}
],
"category": "Portable DVD Players"
}
}'
Now, lets do the query
curl -XPOST "http://192.168.99.100:9200/catalog/products/_search" -d'
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "product.features",
"query": {
"match": {
"product.features.product_type": "player"
}
}
}
},
{
"nested": {
"path": "product.sitedetails",
"query": {
"match": {
"product.sitedetails.sku": "zzz"
}
}
}
}
]
}
}
}'
And the response will be:
"hits": {
"total": 1,
"max_score": 1.4054651,
"hits": [
{
"_index": "catalog",
"_type": "products",
"_id": "AVM_fcYgvVoSi3OfqPTX",
"_score": 1.4054651,
"_source": {
"product": {
"ean": "abc",
"features": {
"Product Type": "DVD player"
},
"color": "Black",
"manufacturer": "Sony",
"sitedetails": [
{
"name": "amazon.com",
"sku": "zzz",
"url": "http://www.amazon.com/dp/zzz"
}
],
"category": "Portable DVD Players"
}
}
}
]
}
Hope it help :D
Use:
curl 'http://localhost:9200/catalog/products/_search?q=sku:"zzz"&pretty=true'
curl 'http://localhost:9200/catalog/products/_search?q=sku:*&pretty=true'. like my thinking, you want to get data within sku:"zzz" and sku:"aaa".
Referer:
http://joelabrahamsson.com/elasticsearch-101/
http://www.elasticsearchtutorial.com/elasticsearch-in-5-minutes.html