How to search in a list like an indexed array and I'm new on Elasticsearch please let me know what is the name of some concepts. I don't know which concept is true, for example: Is it better to say array, object, properties, or list in my question?
This is my result in Kibana when I run GET car/_doc/4
{
"_index" : "car",
"_type" : "_doc",
"_id" : "4",
"_version" : 2,
"_seq_no" : 7,
"_primary_term" : 1,
"found" : true,
"_source" : {
"id" : 4,
"user_id" : 7,
"ads" : {
"0" : {
"id" : 1,
"priority" : 1,
"city_id" : 83,
"model_id" : 13
},
"2" : {
"id" : 4,
"priority" : 2,
"city_id" : 54,
"model_id" : 23
}
},
"status" : 1
}
}
And this is my result for GET car/_doc/15
{
"_index" : "car",
"_type" : "_doc",
"_id" : "15",
"_version" : 2,
"_seq_no" : 27,
"_primary_term" : 1,
"found" : true,
"_source" : {
"id" : 15,
"user_id" : 24,
"ads" : [
{
"id" : 5,
"priority" : 4,
"city_id" : 42,
"model_id" : 11
}
],
"status" : 1
}
}
As you see, I have 2 types of ads. My question is how to search When status is 1 and (ads.city_id = 83 OR ads.0.city_id = 83)
I can use:
GET car/_search
{
"query": {
"bool": {
"must": [
{
"terms": {
"ads.city.slug": ["LA"]
}
}
]
}
}
}
But it doesn't work for the other type of ads and I need to use something like this:
GET car/_search
{
"query": {
"bool": {
"must": [
{
"terms": {
"ads.2.city.slug": ["NewYork"]
}
}
]
}
}
}
How to write a query without writing an index of ads (2)?
As SagarPatel suggested in comment, do normalize the ads field as an array of objects (like it is for doc with id: 15 in your example). After that create an index with defined static mapping like this:
PUT /your-index-name
{
"mappings": {
"properties": {
"ads": {
"type": "nested"
}
}
}
}
(BTW it is advisable to define static mappings for other fields as well)
Nested type will index ads objects as separate inner documents. See elastic docs
After adding documents to the created index you can run queries as follows:
GET /your-index-name/_search
"query": {
"bool": {
"must": [
{
"nested": {
"path": "ads",
"query": {
"term": {
"ads.city_id": {
"value": 83
}
}
}
}
},
{
"nested": {
"path": "ads",
"query": {
"term": {
"ads.city_id": {
"value": 94
}
}
}
}
}
],
"minimum_should_match" : 1 // OR clause
}
}
Related
I'm using "match-Boolean-prefix query but I can't get the exact match of the query.I can't use prefix queries because I also need "not exact match" results and I also need the fuzziness and word completion.I get every thing I need by match-boo-prefix query(the fuzziness not work that good though) but my problem is when I'm looking for exact match like "apple" it shows everything that includes "apple" I need the exact match gets higher ranking than others.
GET /_search
{
"query": {
"bool": {
"must": [
{
"match_bool_prefix": {
"name": {
"query": "apple",
"fuzziness": "auto"
}
}
},
{
"bool": {
"must_not": [
{
"match": {
"type": "3"
}
},
{
"match": {
"type": "4"
}
}
]
}
},
{
"match": {
"status": "A"
}
}
],
"should": [
{
"exists": {
"field": "",
"boost": 10
}
}
]
}
},
"indices_boost": [
{
"index1": 3
},
{
"index2": 1.3
},
{
"index3": 1.5
}
],
"size": 20
}
the result I'm getting with this query is :
{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 20,
"successful" : 20,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4970,
"relation" : "eq"
},
"max_score" : 14.451834,
"hits" : [
{
"_index" : "index",
"_id" : "11434",
"_score" : 14.451834,
"_source" : {
"name" : "Apple Slices With Peanut Butter".
is there any solution for this?
Let's say I have index like this:
{
"id": 6,
"name": "some name",
"users": [
{
"id": 1,
"name": "User1",
"isEnabled": false,
},
{
"id": 2,
"name": "User2",
"isEnabled": false,
},
{
"id": 3,
"name": "User3,
"isEnabled": true,
},
]
}
what I need is to return that index while user searching for the name some name, but also I want to filter out all not enabled users, and if there is not enabled users omit that index.
I tried to use filters like this:
{
"query": {
"bool": {
"must": {
"match": {
"name": "some name"
}
},
"filter": {
"term": {
"users.isEnabled": true
}
}
}
}
}
but in such a case I see index with all users no matter if user is enabled or not. I'm a bit new but is there a way to do so??? I can filter out all that in code after getting data from elasticsearch but in such a case it can break pagination if I remove some index without enabled users from result set.
I'm a bit new to elasticsearch, but as far I can't find how to do it. Thank you in advice!
Elasticsearch will return whole document if there is any match. If you update your mapping and make users array nested, you can achieve this by using inner hits. This is a basic example mapping that works:
{
"mappings": {
"properties": {
"name": {
"type": "text"
},
"users": {
"type": "nested"
}
}
}
}
And if you send a query like following, response will contain id and name from the parent document, and it will contain inner_hits that match to your user's isEnabled query.
{
"_source": ["id", "name"],
"query": {
"bool": {
"must": [
{
"match": {
"name": "some name"
}
},
{
"nested": {
"path": "users",
"query": {
"term": {
"users.isEnabled": {
"value": true
}
}
},
"inner_hits": {}
}
}
]
}
}
}
This is an example response
{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.9375811,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.9375811,
"_source" : {
"name" : "some name",
"id" : 6
},
"inner_hits" : {
"users" : {
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.540445,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_nested" : {
"field" : "users",
"offset" : 2
},
"_score" : 1.540445,
"_source" : {
"id" : 3,
"name" : "User3",
"isEnabled" : true
}
}
]
}
}
}
}
]
}
}
Then you can do the mapping in the application.
The goal is to filter products with multiple prices.
The data looks like this:
{
"name":"a",
"price":[
{
"membershipLevel":"Gold",
"price":"5"
},
{
"membershipLevel":"Silver",
"price":"50"
},
{
"membershipLevel":"Bronze",
"price":"100"
}
]
}
I would like to filter by membershipLevel and price. For example, if I am a silver member and query price range 0-10, the product should not appear, but if I am a gold member, the product "a" should appear. Is this kind of query supported by Elasticsearch?
You need to make use of nested datatype for price and make use of nested query for your use case.
Please see the below mapping, sample document, query and response:
Mapping:
PUT my_price_index
{
"mappings": {
"properties": {
"name":{
"type":"text"
},
"price":{
"type":"nested",
"properties": {
"membershipLevel":{
"type":"keyword"
},
"price":{
"type":"double"
}
}
}
}
}
}
Sample Document:
POST my_price_index/_doc/1
{
"name":"a",
"price":[
{
"membershipLevel":"Gold",
"price":"5"
},
{
"membershipLevel":"Silver",
"price":"50"
},
{
"membershipLevel":"Bronze",
"price":"100"
}
]
}
Query:
POST my_price_index/_search
{
"query": {
"nested": {
"path": "price",
"query": {
"bool": {
"must": [
{
"term": {
"price.membershipLevel": "Gold"
}
},
{
"range": {
"price.price": {
"gte": 0,
"lte": 10
}
}
}
]
}
},
"inner_hits": {} <---- Do note this.
}
}
}
The above query means, I want to return all the documents having price.price range from 0 to 10 and price.membershipLevel as Gold.
Notice that I've made use of inner_hits. The reason is despite being a nested document, ES as response would return the entire set of document instead of only the document specific to where the query clause is applicable.
In order to find the exact nested doc that has been matched, you would need to make use of inner_hits.
Below is how the response would return.
Response:
{
"took" : 128,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.9808291,
"hits" : [
{
"_index" : "my_price_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.9808291,
"_source" : {
"name" : "a",
"price" : [
{
"membershipLevel" : "Gold",
"price" : "5"
},
{
"membershipLevel" : "Silver",
"price" : "50"
},
{
"membershipLevel" : "Bronze",
"price" : "100"
}
]
},
"inner_hits" : {
"price" : {
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.9808291,
"hits" : [
{
"_index" : "my_price_index",
"_type" : "_doc",
"_id" : "1",
"_nested" : {
"field" : "price",
"offset" : 0
},
"_score" : 1.9808291,
"_source" : {
"membershipLevel" : "Gold",
"price" : "5"
}
}
]
}
}
}
}
]
}
}
Hope this helps!
Let me take show you how to do it, using the nested fields and query and filter context. I will take your example to show, you how to define index mapping, index sample documents, and search query.
It's important to note the include_in_parent param in Elasticsearch mapping, which allows us to use these nested fields without using the nested fields.
Please refer to Elasticsearch documentation about it.
If true, all fields in the nested object are also added to the parent
document as standard (flat) fields. Defaults to false.
Index Def
{
"mappings": {
"properties": {
"product": {
"type": "nested",
"include_in_parent": true
}
}
}
}
Index sample docs
{
"product": {
"price" : 5,
"membershipLevel" : "Gold"
}
}
{
"product": {
"price" : 50,
"membershipLevel" : "Silver"
}
}
{
"product": {
"price" : 100,
"membershipLevel" : "Bronze"
}
}
Search query to show Gold with price range 0-10
{
"query": {
"bool": {
"must": [
{
"match": {
"product.membershipLevel": "Gold"
}
}
],
"filter": [
{
"range": {
"product.price": {
"gte": 0,
"lte" : 10
}
}
}
]
}
}
}
Result
"hits": [
{
"_index": "so-60620921-nested",
"_type": "_doc",
"_id": "1",
"_score": 1.0296195,
"_source": {
"product": {
"price": 5,
"membershipLevel": "Gold"
}
}
}
]
Search query to exclude Silver, with same price range
{
"query": {
"bool": {
"must": [
{
"match": {
"product.membershipLevel": "Silver"
}
}
],
"filter": [
{
"range": {
"product.price": {
"gte": 0,
"lte" : 10
}
}
}
]
}
}
}
Above query doesn't return any result as there isn't any matching result.
P.S :- this SO answer might help you to understand nested fields and query on them in detail.
You have to use Nested fields and nested query to archive this: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html
Define you Price property with type "Nested" and then you will be able to filter by every property of nested object
GET _search
{
"query": {
"match": {
"content": "this test"
}
}
}
This gave me below result:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 6,
"successful" : 6,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.5753642,
"hits" : [
{
"_index" : "inbox",
"_type" : "mailbox",
"_id" : "6bb174ab-a4ce-4409-a626-c9a42c98b89e",
"_score" : 0.5753642,
"_source" : {
"user_id" : 13,
"content" : "This is a test"
}
},
{
"_index" : "inbox",
"_type" : "mailbox",
"_id" : "1304cf2e-a1d4-40ca-9876-9abb08c4474d",
"_score" : 0.36464313,
"_source" : {
"user_id" : 10,
"content" : "This is a test"
}
},
{
"_index" : "inbox",
"_type" : "mailbox",
"_id" : "623c093c-4408-445e-abb1-460d2c5004cd",
"_score" : 0.36464313,
"_source" : {
"user_id" : 15,
"content" : "This is a test"
}
}
]
}
}
Which is good. However, I need to filter them by user_id. I mean I need to score only specific user and their content.
GET _search
{
"query": {
"match": {
"content": "this test",
"user_id": 10
}
}
}
When I add user_id i get this error:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[match] query doesn't support multiple fields, found [content] and [user_id]",
"line": 5,
"col": 18
}
],
"type": "parsing_exception",
"reason": "[match] query doesn't support multiple fields, found [content] and [user_id]",
"line": 5,
"col": 18
},
"status": 400
}
Why? And How to properly filter based on user_id?
You can use term query to filter the result by user_id.
Update your query as below:
{
"query": {
"bool": {
"must": [
{
"match": {
"content": "this test"
}
}
],
"filter": [
{
"term": {
"user_id": 10
}
}
]
}
}
}
The query should be like this:
{
"query": {
"bool": {
"must": [
{
"match": {
"content": "this test"
}
},
{
"match": {
"user_id": 10
}
}
]
}
}
}
Use bool query to combine filters
{
"query": {
"bool": {
"must": [
{
"match": {
"content": "this is content"
}
},
{
"term": {
"user_id": {
"value": 47545
}
}
}
]
}
}
}
I am trying to implement a search query over records stored in elasticsearch.
The record structure looks something like this.
{
"_index" : "box_info_store",
"_type" : "boxes",
"_id" : "pWjQLWkBIJk0ORjd0X2P",
"_score" : null,
"_source" : {
"transactionID" : "60ab66cf24c9924f562bf1a2b5d92305d0a6",
"boxNumber" : "Box3",
"createDate" : "2013-09-17T00:00:00",
"itemNumber" : "Item1",
"address" : "Sample Address"
}
}
one box can contain multiple items. For example Box3 can have Item1, Item2 and Item3. So in elasticsearch i will have 3 different documents. Also at the same time, same box and same item can also exist but with different address. The transactionID may or maynot be the same for these documents.
My requirement is to fetch last n recent and distinct transactionIDs, along with their records.
I tried following query to fetch last 7 distinct transactionIDs
GET /box_info_store/boxes/_search?size=7
{
"query": {
"bool": {
"must": [
{"match":{"boxNumber":"Box3"}},
{"match":{"itemNumber":"Item1"}}
]
}
},
"sort": [
{
"createDate": {
"order": "desc"
}
}
],
"aggs": {
"distinct_transactions": {
"terms": { "field": "transactionID"}
}
}
}
This fetched me last 7 documents where boxNumber is Box3 and itemNumber is Item1, but not 7 distinct transactionIDs, two out of these seven documents have the same transactionID(both having separate address though).
But my requirement is to get 7 distinct transactionIds, no matter how many document it returns.
Hope i was able to explain myself.
Appreciate any kind of help here
Thanks
------Edited #gaurav9620, i ran the first query and got count as 32, then i ran the second query with distinct count as 3 i got the following result
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 32,
"max_score" : null,
"hits" : [
{
"_index" : "box_info_store",
"_type" : "boxes",
"_id" : "RWjRLWkBIJk0ORjdEX-L",
"_score" : null,
"_source" : {
"transactionID" : "3087e106244f6247a5290fb21ce64254529c",
"boxNumber" : "Box3",
"createDate" : "2017-11-15T00:00:00",
"itemNumber" : "Item1",
"address" : "sampleAddress12",
},
"sort" : [
1510704000000
]
},
{
"_index" : "box_info_store",
"_type" : "boxes",
"_id" : "MGjQLWkBIJk0ORjdwX0M",
"_score" : null,
"_source" : {
"transactionID" : "60ab66cf24c9924f562bf1a2b5d92305d0a6",
"boxNumber" : "Box3",
"createDate" : "2016-04-03T00:00:00",
"itemNumber" : "Item1",
"address" : "sampleAddress321",
},
"sort" : [
1459641600000
]
},
..........
..........
..........
{
"_index" : "box_info_store",
"_type" : "boxes",
"_id" : "AGjRLWkBIJk0ORjdK4CJ",
"_score" : null,
"_source" : {
"transactionID" : "3087e106244f6247a5290fb21ce64254529c",
"boxNumber" : "Box3",
"createDate" : "1996-02-16T00:00:00",
"itemNumber" : "Item1",
"address" : "sampleAddress4324",
},
"sort" : [
824428800000
]
}
]
},
"aggregations" : {
"unique_transactions" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 16,
"buckets" : [
{
"key" : "3087e106244f6247a5290fb21ce64254529c",
"doc_count" : 6
},
{
"key" : "27c5f3422f4482495d29e7b2c15c0e311743",
"doc_count" : 5
},
{
"key" : "c40e53212e74e24bf02a5bd2b134cf92bffb",
"doc_count" : 5
}
]
}
}
}
The size which you have used : represents number of raw documents that are retrieved.
If your case what you need to do is :
Mention size as 0 -> which will return you no raw documents
Include a size parameter in aggregation which will return you unique 7 ids.
GET /box_info_store/boxes/_search?size=7
{
"query": {
"bool": {
"must": [
{
"match": {
"boxNumber": "Box3"
}
},
{
"match": {
"itemNumber": "Item1"
}
}
]
}
},
"sort": [
{
"createDate": {
"order": "desc"
}
}
],
"aggs": {
"distinct_transactions": {
"terms": {
"field": "transactionID",
"size": 7
}
}
}
}
EDIT-------------------------------------
First fire this query
GET /box_info_store/boxes/_search?size=0
{
"query": {
"bool": {
"must": [
{
"match": {
"boxNumber": "Box3"
}
},
{
"match": {
"itemNumber": "Item1"
}
}
]
}
}
}
Here you will find total number of documents matching your query which you can set as n
After this fire your query as below
GET /box_info_store/boxes/_search?size=**n**
{
"query": {
"bool": {
"must": [
{
"match": {
"boxNumber": "Box3"
}
},
{
"match": {
"itemNumber": "Item1"
}
}
]
}
},
"sort": [
{
"createDate": {
"order": "desc"
}
}
],
"aggs": {
"distinct_transactions": {
"terms": {
"field": "transactionID",
"size": NUMBER_OF_UNIQUE_TRANSACTION_IDS_TO_BE_FETCHED
}
}
}
}