I need a query where the results will exclude any userIds if they have at least 1 document with the tag set to a value within an 'excluded' list i.e. TAG A or TAG B.
I have an index with data like below:
{
"_index": "tags-3",
"_type": "_doc",
"_id": "YYYYYYY",
"_score": 10.272416,
"_source": {
"id": "YYYYYYY",
"userId": "User1",
"tag": "TAG A"
}
},
{
"_index": "tags-3",
"_type": "_doc",
"_id": "ZZZZZZ",
"_score": 10.272416,
"_source": {
"id": "ZZZZZZ",
"userId": "User1",
"tag": "TAG B"
},
{
"_index": "tags-3",
"_type": "_doc",
"_id": "ZZZZZZ",
"_score": 10.272416,
"_source": {
"id": "ZZZZZZ",
"userId": "User2",
"tag": "TAG A"
},
{
"_index": "tags-3",
"_type": "_doc",
"_id": "ZZZZZZ",
"_score": 10.272416,
"_source": {
"id": "ZZZZZZ",
"userId": "User2",
"tag": "TAG D"
},
{
"_index": "tags-3",
"_type": "_doc",
"_id": "ZZZZZZ",
"_score": 10.272416,
"_source": {
"id": "ZZZZZZ",
"userId": "User4",
"tag": "TAG D"
}
For the input above, I would expect an output of:
{
"_index": "tags-3",
"_type": "_doc",
"_id": "ZZZZZZ",
"_source": {
"userId": "User4"
}
since User4 has no documents with the tag set to TAG A or TAG B.
User4 is the only other user with a document with the tag set to TAG D however since it has another document with TAG B, it is excluded.
One way to do this would be to:
Aggregate (group) on the user IDs - this would give you all the user IDs
Then, aggregate the documents for each user ID (nested aggregation) with a filter for the multiple (or single) tag values you want to exclude - this would give you the total sum of documents with the tag set to an excluded tag for each user ID
Finally, perform a bucket selector aggregation, only including user IDs which have a count of 0 for any excluded documents; this would give you the users who don't have any documents with any excluded tag values
This query should work, for an excluded tag list of A, B & C:
{
"aggs": {
"user-ids": {
"terms": {
"field": "userId.keyword",
"size": 10000
},
"aggs": {
"excluded_tags_agg": {
"filter": {
"bool": {
"should": [
{
"match_phrase": {
"tag.keyword": "TAG A"
}
},
{
"match_phrase": {
"tag.keyword": "TAG B"
}
},
{
"match_phrase": {
"tag.keyword": "TAG C"
}
}
],
"minimum_should_match": 1
}
}
},
"filter_userids_which_do_not_have_any_docs_with_excluded_tags": {
"bucket_selector": {
"buckets_path": {
"doc_count": "excluded_tags_agg > _count"
},
"script": "params.doc_count == 0"
}
}
}
}
},
"size": 0
}
I want to perform both exact word match and partial word/sub string match. For example, if I search for "test product" then I should be able to find "test" and "product" related text in the result. I'm searching Elasticsearch with the below match query, which is not giving me the exact match, instead its giving some more irrelevant match.
I'm using Elasticsearch 6.3
My query for GET /_search:
{
"must": {
"query_string": {
"query": "title:*test product*"
}
}
}
Search Result:
"hits": [
{
"_index": "67107104",
"_type": "_doc",
"_id": "1",
"_score": 0.6931471,
"_source": {
"title": "testing"
}
},
{
"_index": "67107104",
"_type": "_doc",
"_id": "2",
"_score": 0.6931471,
"_source": {
"title": "product good"
}
},
{
"_index": "67107104",
"_type": "_doc",
"_id": "3",
"_score": 0.6931471,
"_source": {
"title": "sample"
}
}
]
Expected Search Result:
"hits": [
{
"_index": "67107104",
"_type": "_doc",
"_id": "1",
"_score": 0.6931471,
"_source": {
"title": "testing"
}
},
{
"_index": "67107104",
"_type": "_doc",
"_id": "2",
"_score": 0.6931471,
"_source": {
"title": "product good"
}
}
]
In the search query above, you are searching in the review field, whereas in the search result you are getting data for title field
Adding a working example with index data, search query, and search result
Index Data:
{
"review": "testing"
}
{
"review": "product good"
}
{
"review": "sample"
}
Search Query:
{
"query": {
"match": {
"review": "test product"
}
}
}
Search Result:
"hits": [
{
"_index": "67119314",
"_type": "_doc",
"_id": "2",
"_score": 0.2876821,
"_source": {
"review": "product good"
}
}
]
My results for the following term query gets rendered like this. But we would want the search results where "BC" appears after "Bar", since we are trying to perform a alphabetical search. What should be done to get this working
Adam
Buck
BC
Bar
Car
Far
NativeSearchQuery query = new NativeSearchQueryBuilder()
.withSourceFilter(new FetchSourceFilterBuilder().withIncludes().build())
.withQuery(QueryBuilders.termQuery("type", field))
.withSort(new FieldSortBuilder("name").order(SortOrder.ASC))
.withPageable(pageable).build();
To sort the result in alphabetical order you can define a normalizer with a lowercase filter, lowercase filter will ensure that all the letters are changed to lowercase before indexing the document and searching.
Modify your index mapping as
{
"settings": {
"analysis": {
"normalizer": {
"my_normalizer": {
"type": "custom",
"filter": [
"lowercase"
]
}
}
}
},
"mappings": {
"properties": {
"name": {
"type": "keyword",
"normalizer": "my_normalizer"
}
}
}
}
Indexed the same sample documents as given in the question.
Search Query:
{
"sort":{
"name":{
"order":"asc"
}
}
}
Search Result:
"hits": [
{
"_index": "66064809",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"name": "Adam"
},
"sort": [
"adam"
]
},
{
"_index": "66064809",
"_type": "_doc",
"_id": "4",
"_score": null,
"_source": {
"name": "Bar"
},
"sort": [
"bar"
]
},
{
"_index": "66064809",
"_type": "_doc",
"_id": "3",
"_score": null,
"_source": {
"name": "BC"
},
"sort": [
"bc"
]
},
{
"_index": "66064809",
"_type": "_doc",
"_id": "2",
"_score": null,
"_source": {
"name": "Buck"
},
"sort": [
"buck"
]
},
{
"_index": "66064809",
"_type": "_doc",
"_id": "5",
"_score": null,
"_source": {
"name": "Car"
},
"sort": [
"car"
]
},
{
"_index": "66064809",
"_type": "_doc",
"_id": "6",
"_score": null,
"_source": {
"name": "Far"
},
"sort": [
"far"
]
}
]
}
The below is the records in my test-data index and using the elastic search version 5.6.
[
{
"_index": "test-data",
"_type": "log",
"_id": "123",
"_score": 2,
"_source": {
"request": "/test-url/poll?request_ids=1",
"user": "test1"
}
},
{
"_index": "test-data",
"_type": "log",
"_id": "126",
"_score": 2,
"_source": {
"request": "/test-url/poll?request_ids=2",
"user": "test1"
}
},
{
"_index": "test-data",
"_type": "log",
"_id": "124",
"_score": 2,
"_source": {
"request": "/test-url/poll?request_ids=2",
"user": "test1"
}
},
{
"_index": "test-data",
"_type": "log",
"_id": "125",
"_score": 2,
"_source": {
"request": "/test-url/poll?request_ids=2",
"user": "test1"
}
},
{
"_index": "test-data",
"_type": "log",
"_id": "128",
"_score": 2,
"_source": {
"request": "/test-url/poll?request_ids=2",
"user": "test2"
}
}
]
I need to find the number of distinct records which are having the unique combination of request and user and tried the below query. I expect 3 as the result, but getting 5.
{
"query": {
"bool": {
"must": [
{
"exists": {
"field": "request"
}
},
{
"regexp": {
"request.keyword": "/test-url/poll\\?request_ids=.*"
}
}
]
}
},
"_source": ["request.keyword", "user.keyword","request", "user"],
"aggs": {
"request_count": {
"cardinality": {
"script": {
"lang": "painless",
"source": "[doc['request.keyword'], doc['user.keyword']]"
}
}
}
}
}
Can somebody suggest what is wrong with the query or some other option to solve this issue?
I think you should try the following:
“[doc['request.keyword'].value + ' ' + doc['user.keyword']].value”
This would calculate the hashes of the field, which would be a concatenated string from a two values - request and user
Caveat - this would be a significant performance hit, since it’s calculating and extracting field values on the fly
One of the possibility to avoid this would be to alter your indexing process to create this synthetic field being a concatenation, so later you could use normal cardinality aggregation, not the script one.
I have written the following lucene query in elasticsearch for getting documents with Id field as mentioned:
GET requirements_v3/_search
{
"from": 0,
"size": 10,
"query": {
"bool": {
"filter": {
"bool": {
"should": [
{"match": {
"Id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b"
}},
{
"match": {
"Id": "048b7907-2b5a-438a-ace9-f1e1fd67ca69"
}
},
{
"match": {
"Id": "3b385896-1207-4f6d-8ae9-f3ced84cf1fa"
}
},
{
"match": {
"Id": "0aa1db52-c0fb-4bf6-9223-00edccc32703"
}
},
{
"match": {
"Id": "8c399993-f273-4ee0-a1ab-3a85c6848113"
}
},
{
"match": {
"Id": "4461eb37-487e-4899-a7be-914640fab0e0"
}
},
{
"match": {
"Id": "07052261-b904-4bfc-a6fd-3acd28114c6a"
}
},
{
"match": {
"Id": "95816ff0-9eae-4196-99fc-86c6f43395fd"
}
},
{
"match": {
"Id": "ea8a59a6-2b2f-467a-9beb-e281b1581a0a"
}
},
{
"match": {
"Id": "33f87d98-024f-4893-aa1c-8d438a98cd1f"
}
}
]
}
}
}
}
The response for the above query is:
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 18,
"max_score": 0,
"hits": [
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "9d8060da-c3e2-4f6d-b4e2-17e65b266c76",
"_score": 0,
"_source": {
"Id": "9d8060da-c3e2-4f6d-b4e2-17e65b266c76",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "4461eb37-487e-4899-a7be-914640fab0e0",
"_score": 0,
"_source": {
"Id": "4461eb37-487e-4899-a7be-914640fab0e0",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "33f87d98-024f-4893-aa1c-8d438a98cd1f",
"_score": 0,
"_source": {
"Id": "33f87d98-024f-4893-aa1c-8d438a98cd1f",
"Name": "Create Configurator"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "d75d9a7c-e145-487e-922f-102c16d0026f",
"_score": 0,
"_source": {
"Id": "d75d9a7c-e145-487e-922f-102c16d0026f",
"Name": "Create Configurator"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "007eadb7-adda-487e-b7fe-6f6b5648de2e",
"_score": 0,
"_source": {
"Id": "007eadb7-adda-487e-b7fe-6f6b5648de2e",
"Name": "Detail Page - Build"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "95816ff0-9eae-4196-99fc-86c6f43395fd",
"_score": 0,
"_source": {
"Id": "95816ff0-9eae-4196-99fc-86c6f43395fd",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "07052261-b904-4bfc-a6fd-3acd28114c6a",
"_score": 0,
"_source": {
"Id": "07052261-b904-4bfc-a6fd-3acd28114c6a",
"Name": "HUC"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "d60daf3a-4681-4bfc-a3a9-b04b5b005f73",
"_score": 0,
"_source": {
"Id": "d60daf3a-4681-4bfc-a3a9-b04b5b005f73",
"Name": "DAMS UpsertUnenrollPrice" }
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "c1b367f2-a57a-487e-994c-84470e0f9db4",
"_score": 0,
"_source": {
"Id": "c1b367f2-a57a-487e-994c-84470e0f9db4",
"Name": "Item Setup"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b",
"_score": 0,
"_source": {
"Id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b",
"Name": "Installments"
}
}
]
}
}
This mentions totalHits as '18'. Why is it returning more items than 10? I believe match query should be used for 'exact' matches, so why more documents are returned here?
P.S.: I know I can use the Ids query for this, but I want to know why is this not returning the correct response
Update: Setting the size to 20 returns the following response:
{
"took": 195,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 18,
"max_score": 0,
"hits": [
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "9d8060da-c3e2-4f6d-b4e2-17e65b266c76",
"_score": 0,
"_source": {
"Id": "9d8060da-c3e2-4f6d-b4e2-17e65b266c76",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "4461eb37-487e-4899-a7be-914640fab0e0",
"_score": 0,
"_source": {
"Id": "4461eb37-487e-4899-a7be-914640fab0e0",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "33f87d98-024f-4893-aa1c-8d438a98cd1f",
"_score": 0,
"_source": {
"Id": "33f87d98-024f-4893-aa1c-8d438a98cd1f",
"Name": "Create Configurator"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "d75d9a7c-e145-487e-922f-102c16d0026f",
"_score": 0,
"_source": {
"Id": "d75d9a7c-e145-487e-922f-102c16d0026f",
"Name": "Create Configurator"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "007eadb7-adda-487e-b7fe-6f6b5648de2e",
"_score": 0,
"_source": {
"Id": "007eadb7-adda-487e-b7fe-6f6b5648de2e",
"Name": "Detail Page - Build"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "95816ff0-9eae-4196-99fc-86c6f43395fd",
"_score": 0,
"_source": {
"Id": "95816ff0-9eae-4196-99fc-86c6f43395fd",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "07052261-b904-4bfc-a6fd-3acd28114c6a",
"_score": 0,
"_source": {
"Id": "07052261-b904-4bfc-a6fd-3acd28114c6a",
"Name": "HUC"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "d60daf3a-4681-4bfc-a3a9-b04b5b005f73",
"_score": 0,
"_source": {
"Id": "d60daf3a-4681-4bfc-a3a9-b04b5b005f73",
"Name": "DAMS UpsertUnenrollPrice"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "c1b367f2-a57a-487e-994c-84470e0f9db4",
"_score": 0,
"_source": {
"Id": "c1b367f2-a57a-487e-994c-84470e0f9db4",
"Name": "Item Setup"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b",
"_score": 0,
"_source": {
"Id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b",
"Name": "Installments"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "b9437079-47c4-487e-abf0-1ff076f69e0f",
"_score": 0,
"_source": {
"Id": "b9437079-47c4-487e-abf0-1ff076f69e0f",
"Name": "Detail Page - Strings "
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "0aa1db52-c0fb-4bf6-9223-00edccc32703",
"_score": 0,
"_source": {
"Id": "0aa1db52-c0fb-4bf6-9223-00edccc32703",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "ea8a59a6-2b2f-467a-9beb-e281b1581a0a",
"_score": 0,
"_source": {
"Id": "ea8a59a6-2b2f-467a-9beb-e281b1581a0a",
"Name": "Create Configurator"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "fd259359-4f6d-4530-ac29-fcebe00d66a6",
"_score": 0,
"_source": {
"Id": "fd259359-4f6d-4530-ac29-fcebe00d66a6",
"Name": "Invite Platform"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "1b2ba0bb-3e7f-46fb-b904-07460b84848b",
"_score": 0,
"_source": {
"Id": "1b2ba0bb-3e7f-46fb-b904-07460b84848b",
"Name": "Training"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "8c399993-f273-4ee0-a1ab-3a85c6848113",
"_score": 0,
"_source": {
"Id": "8c399993-f273-4ee0-a1ab-3a85c6848113",
"Name": "Configure ASIN for Reporting"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "3b385896-1207-4f6d-8ae9-f3ced84cf1fa",
"_score": 0,
"_source": {
"Id": "3b385896-1207-4f6d-8ae9-f3ced84cf1fa",
"Name": "Create Extended/Limited Warranty Configuration"
}
},
{
"_index": "requirements_v3",
"_type": "_doc",
"_id": "048b7907-2b5a-438a-ace9-f1e1fd67ca69",
"_score": 0,
"_source": {
"Id": "048b7907-2b5a-438a-ace9-f1e1fd67ca69",
"Name": "Invite Platform"
}
}
]
}
}
Lets understand this by the following mapping e.g:
{
"_doc": {
"properties": {
"Id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
The above mapping is created dynamically by elasticsearch. Lets us now focus on Id field. Its type is text. By default the analyzer for text datatype is standard analyzer. When this analyzer is applied on the input for this field it get tokenized into terms. So for example if you input value for Id is 33f87d98-024f-4893-aa1c-8d438a98cd1f following tokens get generated:
33f87d98
024f
4893
aa1c
8d438a98cd1f
As you can see the input value is splitted by - being used as delimiter. This is because standard analyzer is applied on it.
There is another sub-field under Id which is keyword and its type is keyword. For type keyword the input is indexed as it is without applying any modification.
Now lets understand why more documents get matched and result count is more than expected. In your query you used match query on Id field as below:
{
"match": {
"Id": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b"
}
}
By default match query uses the same analyzer that is applied on the field in mapping. So on the Id value in the query again the same analyzer is applied and the input is splitted into tokens in a similar way as above. The default operator that is applied between tokens of match query input string is OR and hence your query actually becomes:
b8bf49a4 OR 960b OR 4fa8 OR 8c5f OR a3fce4b4d07b
There if any of the above tokens match to any of the indexed terms stored in Id field, the document is considered a match.
Solution for the above based on above mapping:
Use the keyword field instead. So the query becomes:
{
"match": {
"Id.keyword": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b"
}
}
More on how match works see here.
Also as mention by #Curious_MInd in his answer its better to use terms than using multiple match in should.
As you said that your Id is text as well as keyword so you should use Id.keyword for matching exact values like
GET requirements_v3/_search
{
"from": 0,
"size": 10,
"query": {
"bool": {
"filter": {
"bool": {
"should": [
{"match": {
"Id.keyword": "b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b"
}},
{
"match": {
"Id.keyword": "048b7907-2b5a-438a-ace9-f1e1fd67ca69"
}
}
]
}
}
}
}
But I guess you should use terms if you wants to match multiple exact values. Have a look here. For an example:
{
"terms" : {
"Id" : ["b8bf49a4-960b-4fa8-8c5f-a3fce4b4d07b", "048b7907-2b5a-438a-ace9-f1e1fd67ca69"]
}
}