I am trying to combine wildcard with date range in elastic search query but is not giving response based upon the wildcard search. It is returning response with items which have incorrect date range.
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"wildcard": {
"hostName": "*abc*"
}
},
{
"range": {
"requestDate": {
"gte": "2019-10-01T08:00:00.000Z"
}
}
}
]
}
}
]
}
}
}
The index mapping looks as below:
{
"index_history": {
"mappings": {
"applications_datalake": {
"properties": {
"query": {
"properties": {
"term": {
"properties": {
"server": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
},
"index-data-type": {
"properties": {
"attributes": {
"properties": {
"wwnListForServer": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"hostName": {
"type": "keyword"
},
"requestDate": {
"type": "date"
},
"requestedBy": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
}
}
You missed minimum_should_match parameter,
Check this out :
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html.
I think your query should looklike this:
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"wildcard": {
"hostName": "*abc*"
}
},
{
"range": {
"requestDate": {
"gte": "2019-10-01T08:00:00.000Z"
}
}
}
],
"minimum_should_match" : 2
}
}
]
}
}
}
From the documentation :
You can use the minimum_should_match parameter to specify the number
or percentage of should clauses returned documents must match.
If the bool query includes at least one should clause and no must or
filter clauses, the default value is 1. Otherwise, the default value
is 0.
According to your mappings, you have to call-out the fully qualified property for hostName and requestDate fields. Example:
"wildcard": {
"index-data-type.hostName": {
"value": "..."
}
}
Also, could also consider reducing your compound queries to just the main bool query, using the must clause, and apply a filter. Example:
{
"from": 0,
"size": 20,
"query": {
"bool": {
"must": [
{
"wildcard": {
"index-data-type.hostName": {
"value": "*abc*"
}
}
}
],
"filter": {
"range": {
"index-data-type.requestDate": {
"gte": "2019-10-01T08:00:00.000Z"
}
}
}
}
}
}
The filter context doesn't contribute to the _score yet it reduces your number of hits.
Warnining:
Using the leading asterisk (*) on a wildcard query can have severe performance impacts to your queries.
Related
I have two questions regarding mapping and querying a java hashmap in elasticsearch.
Does this mapping make sense in elasticsearch (is it the correct way to map a hashmap)?:
{
"properties": {
"itemsMap": {
"type": "nested",
"properties": {
"key": {
"type": "date",
"format": "yyyy-MM-dd"
},
"value": {
"type": "nested",
"properties": {
"itemVal1": {
"type": "double"
},
"itemVal2": {
"type": "double"
}
}
}
}
}
}
}
Here is some example data:
{
"itemsMap": {
"2021-12-31": {
"itemVal1": 100.0,
"itemVal2": 150.0,
},
"2021-11-30": {
"itemVal1": 200.0,
"itemVal2": 50.0,
}
}
}
My queries don't seem to work. For example:
{
"query": {
"nested": {
"path": "itemsMap",
"query": {
"bool": {
"must": [
{
"match": {
"itemsMap.key": "2021-11-30"
}
}
]
}
}
}
}
}
Am I doing something wrong? How can I query such a structure? I have the possibility to change the mapping if it's necessary.
Thanks
TLDR;
The way you are uploading your data, nothing is stored in key.
You will have fields named 2021-11-30 ... and key is going to be empty.
Either you have a limited amount of "dates" and this is a viable options (less than 1000) else your format is not viable on the long run.
If you don't want to change your doc, here is the query
GET /71525899/_search
{
"query": {
"nested": {
"path": "itemsMap",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "itemsMap.2021-12-31"
}
}
]
}
}
}
}
}
To understand
If you inspect the mapping by querying the index
GET /<index_name>/_mapping
You will see that the number of fields name after your date is going to grow.
And in all your doc, itemsMap.key is going to be empty. (this explain why my previous answer did not work.
A more viable option
Keep your mapping, update the shape of your docs.
They will look like
{
"itemsMap": [
{
"key": "2021-12-31",
"value": { "itemVal1": 100, "itemVal2": 150 }
},
{
"key": "2021-11-30",
"value": { "itemVal1": 200, "itemVal2": 50 }
}
]
}
DELETE /71525899
PUT /71525899/
{
"mappings": {
"properties": {
"itemsMap": {
"type": "nested",
"properties": {
"key": {
"type": "date",
"format": "yyyy-MM-dd"
},
"value": {
"type": "nested",
"properties": {
"itemVal1": {
"type": "double"
},
"itemVal2": {
"type": "double"
}
}
}
}
}
}
}
}
POST /_bulk
{"index":{"_index":"71525899"}}
{"itemsMap":[{"key":"2021-12-31", "value": {"itemVal1":100,"itemVal2":150}},{"key":"2021-11-30", "value":{"itemVal1":200,"itemVal2":50}}]}
{"index":{"_index":"71525899"}}
{"itemsMap":[{"key":"2022-12-31", "value": {"itemVal1":100,"itemVal2":150}},{"key":"2021-11-30", "value":{"itemVal1":200,"itemVal2":50}}]}
{"index":{"_index":"71525899"}}
{"itemsMap":[{"key":"2021-11-31", "value": {"itemVal1":100,"itemVal2":150}},{"key":"2021-11-30", "value":{"itemVal1":200,"itemVal2":50}}]}
GET /71525899/_search
{
"query": {
"nested": {
"path": "itemsMap",
"query": {
"bool": {
"must": [
{
"match": {
"itemsMap.key": "2021-12-31"
}
}
]
}
}
}
}
}
I have the following query that works as expected:
GET <index_name>/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
The problem is that I need to further filter the data, but the field I need to filter on is a text field. I have tried many different ways of putting a must match into my query but everything is either malformed or filters out all hits when I know it should only filter out half. How can I add a must match "irStatus":"COMPLETE" to this query? Thanks in advance.
What you're after is a term query on, preferably, the keyword of irStatus. That is to say:
GET index/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"term": {
"irStatus.keyword": {
"value": "COMPLETE"
}
}
}
],
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
Assuming your mapping looks something like this:
{
"mappings": {
"properties": {
"irFileCreateTime": {
"type": "date"
},
"fileId": {
"type": "integer"
},
"irStatus": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
The reason it's apparently failing on your end is that "COMPLETE" has been lowercased due to standard analyzer.
Alternatively, you could do:
{
"must":[
{
"query_string":{
"query":"irStatus:COMPLETE AND (fileId:(46704 OR 46706 OR 46719))"
}
}
]
}
I have a nested type field in my mapping. When I use Term search query on my nested field no result is returned from Elasticsearch whereas when I change Term to Match query, it works fine and Elasticsearch returns expected result
here is my mapping, imagine I have only one nested field in my type mapping
{
"homing.estatefiles": {
"mappings": {
"estatefile": {
"properties": {
"DynamicFields": {
"type": "nested",
"properties": {
"Name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ValueBool": {
"type": "boolean"
},
"ValueDateTime": {
"type": "date"
},
"ValueInt": {
"type": "long"
}
}
}
}
}
}
}
}
And here is my term query (which returns no result)
{
"from": 50,
"size": 50,
"query": {
"bool": {
"filter": [
{
"nested": {
"query": {
"bool": {
"must": [
{
"term": {
"DynamicFields.Name":{"value":"HasParking"}
}
},
{
"term": {
"DynamicFields.ValueBool": {
"value": true
}
}
}
]
}
},
"path": "DynamicFields"
}
}
]
}
}
}
And here is my query which returns expected result (by changing Term query to Match query)
{
"from": 50,
"size": 50,
"query": {
"bool": {
"filter": [
{
"nested": {
"query": {
"bool": {
"must": [
{
"match": {
"DynamicFields.Name":"HasParking"
}
},
{
"term": {
"DynamicFields.ValueBool": {
"value": true
}
}
}
]
}
},
"path": "DynamicFields"
}
}
]
}
}
}
This is happening because the capital letters with the analyzer of elastic.
When you are using term the elastic is looking for the exact value you gave.
up until now it sounds good, but before it tries to match the term, the value you gave go through an analyzer of elastic which manipulate your value.
For example in your case it also turn the HasParking to hasparking.
And than it will try to match it and of course will fail. They have a great explanation in the documentation in the "Why doesn’t the term query match my document" section. This analyzer not being activated on the value when you query using match and this why you get your result.
I have a user document with a field called experiences which is an array of objects, like:
{
"experiences": [
{
"end_date": "2017-03-02",
"is_valid": false
},
{
"end_date": "2015-03-02",
"is_valid": true
}
]
}
With this document I have to search users where end date is in last year and is_valid is true.
At this time I have a query -> bool and I add two must there, one range for the end_date and one term for the is_valid.
{
"query": {
"bool": {
"must": {
"term": {
"experiences.is_valid": true
},
"range": {
"experiences.end_date": {
"gte": "now-1y",
"lte": "now"
}
},
}
}
}
}
The result is that this user is selected because he has an end_date in the last year (the first exp.) and another exp. with is_valid true.
Of course this is not what I need, because I need that end_date and is_valid must be referenced to the same object, but how can we do this on Elasticsearch?
Mapping:
"experiences": {
"properties": {
"comment": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"end_date": {
"type": "date"
},
"id": {
"type": "long"
},
"is_valid": {
"type": "boolean"
},
"start_date": {
"type": "date"
}
}
}
You need to change experiences type to Nested data type.
Then apply nested query :
{
"query": {
"nested": {
"path": "experiences",
"query": {
"bool": {
"must": [
{
"term": {
"experiences.is_valid": true
}
},
{
"range": {
"experiences.end_date": {
"gte": "now-1y",
"lte": "now"
}
}
}
]
}
}
}
}
}
This is due to the way arrays of objects are flattened in Elasticsearch.
Study more here
I have two documents in my index (same type) :
{
"first_name":"John",
"last_name":"Doe",
"age":"24",
"phone_numbers":[
{
"contract_number":"123456789",
"phone_number":"987654321",
"creation_date": ...
},
{
"contract_number":"123456789",
"phone_number":"012012012",
"creation_date": ...
}
]
}
{
"first_name":"Roger",
"last_name":"Waters",
"age":"36",
"phone_numbers":[
{
"contract_number":"546987224",
"phone_number":"987654321",
"creation_date": ...,
"expired":true
},
{
"contract_number":"87878787",
"phone_number":"55555555",
"creation_date": ...
}
]
}
Clients would like to perform a full text search. Okay no problem here
My problem :
In this full text search, sometimes user will search by phone_number. In this case there is a parameter like expired=true.
Example :
First client search request : "987654321" with expired absent or set to false
--> Result : Only first document
Second client search request : "987654321" with expired set to true
--> Result : The two documents
How can I achieve that ?
Here is my mapping :
{
"user": {
"_all": {
"auto_boost": true,
"omit_norms": true
},
"properties": {
"phone_numbers": {
"type": "nested",
"properties": {
"phone_number": {
"type": "string"
},
"creation_date": {
"type": "string",
"index": "no"
},
"contract_number": {
"type": "string"
},
"expired": {
"type": "boolean"
}
}
},
"first_name":{
"type": "string"
},
"last_name":{
"type": "string"
},
"age":{
"type": "string"
}
}
}
}
Thanks !
MC
EDIT :
I tried this query :
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "987654321",
"analyze_wildcard": "true"
}
},
"filter": {
"nested": {
"path": "phone_numbers",
"filter": {
"bool": {
"should":[
{
"bool": {
"must": [
{
"term": {
"phone_number": "987654321"
}
},
{
"missing": {
"field": "expired"
}
}
]
}
},
{
"bool": {
"must_not": [
{
"term": {
"phone_number": "987654321"
}
}
]
}
}
]
}
}
}
}
}
}}
But I get the two documents instead of get only the first one
You're very close. Try using a combination of must and should, where the must clause ensures the phone_number matches the search value, and the should clause ensures that either the expired field is missing or set to false. For example:
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "987654321",
"analyze_wildcard": "true"
}
},
"filter": {
"nested": {
"path": "phone_numbers",
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"phone_number": "987654321"
}
}
],
"should": [
{
"missing": {
"field": "expired"
}
},
{
"term": {
"expired": false
}
}
]
}
}
}
}
}
}
}
}
}
I ran this query using your mapping and sample documents and it returned the one document for John Doe, as expected.