Elasticsearch : filter results based on the date range - elasticsearch

I'm using Elasticsearch 6.6, trying to extract multiple results/records based on multiple values (email_address) passed to the query (Bool) on a date range. For ex: I want to extract information about few employees based on their email_address (annie#test.com, charles#test.com, heman#test.com) and from the period i.e project_date (2019-01-01).
I did use should expression but unfortunately it's pulling all the records from elasticsearch based on the date range i.e. it's even pulling other employees information from project_date 2019-01-01.
{
"query": {
"bool": {
"should": [
{ "match": { "email_address": "annie#test.com" }},
{ "match": { "email_address": "chalavadi#test.com" }}
],
"filter": [
{ "range": { "project_date": { "gte": "2019-08-01" }}}
]
}
}
}
I also tried must expression but getting no result. Could you please help me on finding employees using their email_address with the date range?
Thanks in advance.

Should(Or) clauses are optional
Quoting from this article.
"In a query, if must and filter queries are present, the should query occurrence then helps to influence the score. However, if bool query is in a filter context or has neither must nor filter queries, then at least one of the should queries must match a document."
So in your query should is only influencing the score and not actually filtering the document. You must wrap should in must, or move it in filter(if scoring not required).
GET employeeindex/_search
{
"query": {
"bool": {
"filter": {
"range": {
"projectdate": {
"gte": "2019-01-01"
}
}
},
"must": [
{
"bool": {
"should": [
{
"term": {
"email.raw": "abc#text.com"
}
},
{
"term": {
"email.raw": "efg#text.com"
}
}
]
}
}
]
}
}
}
You can also replace should clause with terms clause as in #AlwaysSunny's answer.

You can do it with terms and range along with your existing query inside filter in more shorter way. Your existing query doesn't work as expected because of should clause, it makes your filter weaker. Read more here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html
{
"query": {
"bool": {
"filter": [
{
"terms": {
"email_address.keyword": [
"annie#test.com", "chalavedi#test.com"
]
}
},
{
"range": {
"project_date": {
"gte": "2019-08-01"
}
}
}
]
}
}
}

Related

multi fields search query for elasticsearch golang

I have a situation where I need to do elastic search based on multi-field. For Example: I have multiple fields in my postindex and I want to apply condition on four these fields (i.e. userid, channelid, createat, teamid) to meet my search requirement. When value of all these fields matched then search query displays results and if one of these is not match with values in postindex then it display no result.
I am trying to make a multifield search query for go-elasticsearch to search data from my post index. For the searcquery result four field must match otherwise it display 0 hit/no-result.
So, I think you need to write a following query :
GET postindex/_search
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"must": [
{
"term": {
"userid": {
"value": "mcqmycxpyjrddkie9mr13txaqe"
}
}
},
{
"term": {
"channelid": {
"value": "dnoihmrinins3qrm6bb9175ume"
}
}
},
{
"range": {
"createat": {
"gt": 1672909114890
}
}
}
]
}
},
{
"term": {
"teamid": {
"value": "qomrg11o8b8ijxoy8hrcnweoay"
}
}
}
]
}
}
}
In here, there is a bool query with should in parent scope, which is like OR. And inside the should there is another bool query with must which is like AND. We can also write the query shorter, but this will be better for you to understand.

Is it possible to access a query term in a script field?

I would like to construct an elasticsearch query in which I can search for a term and on-the-fly compute a new field for each found document, which is calculated based on some existing fields as well as the query term. Is this possible?
For example, let's say in my EL query I am searching for documents which have the keyword "amsterdam" in the "text" field.
"filter": [
{
"match_phrase": {
"text": {
"query": "amsterdam"
}
}
}]
Now I would also like to have a script field in my query, which computes some value based on other fields as well as the query.
So far, I have only found how to access the other fields of a document though, using doc['someOtherField'], for example
"script_fields" : {
"new_field" : {
"script" : {
"lang": "painless",
"source": "if (doc['citizens'].value > 10000) {
return "large";
}
return "small";"
}
}
}
How can I integrate the query term, e.g. if I wanted to add to the if statement "if the query term starts with a-e"?
You're on the right track but script_fields are primarily used to post-process your documents' attributes — they won't help you filter any docs because they're run after the query phase.
With that being said, you can use scripts to filter your documents through script queries. Before you do that, though, you should explore alternatives.
In other words, scripts should be used when all other mechanisms and techniques have been exhausted.
Back to your example. I see three possibilities off the top of my head.
Match phrase prefix queries as a group of bool-should subqueries:
POST your-index/_search
{
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"match_phrase_prefix": {
"text_field": "a"
}
},
{
"match_phrase_prefix": {
"text_field": "b"
}
},
{
"match_phrase_prefix": {
"text_field": "c"
}
},
... till the letter "e"
]
}
}
]
}
}
}
A regexp query:
POST your-index/_search
{
"query": {
"bool": {
"must": [
{
"regexp": {
"text_field": "[a-e].+"
}
}
]
}
}
}
Script queries using .charAt comparisons:
POST your-index/_search
{
"query": {
"bool": {
"must": [
{
"script": {
"script": {
"source": """
char c = doc['text_field.keyword'].value.charAt(0);
return c >= params.gte.charAt(0) && c <= params.lte.charAt(0);
""",
"params": {
"gte": "a",
"lte": "e"
}
}
}
}
]
}
}
}
If you're relatively new to ES and would love to see real-world examples, check out my recently released Elasticsearch Handbook. One chapter is dedicated to scripting and as it turns out, you can achieve a lot with scripts (if of course executed properly).

Elastic search query using python list

How do I pass a list as query string to match_phrase query?
This works:
{"match_phrase": {"requestParameters.bucketName": {"query": "xxx"}}},
This does not:
{
"match_phrase": {
"requestParameters.bucketName": {
"query": [
"auditloggingnew2232",
"config-bucket-123",
"web-servers",
"esbck-essnap-1djjegwy9fvyl",
"tempexpo",
]
}
}
}
match_phrase simply does not support multiple values.
You can either use a should query:
GET _search
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"requestParameters.bucketName": {
"value": "auditloggingnew2232"
}
}
},
{
"match_phrase": {
"requestParameters.bucketName": {
"value": "config-bucket-123"
}
}
}
]
},
...
}
}
or, as #Val pointed out, a terms query:
{
"query": {
"terms": {
"requestParameters.bucketName": [
"auditloggingnew2232",
"config-bucket-123",
"web-servers",
"esbck-essnap-1djjegwy9fvyl",
"tempexpo"
]
}
}
}
that functions like an OR on exact terms.
I'm assuming that 1) the bucket names in question are unique and 2) that you're not looking for partial matches. If that's the case, plus if there are barely any analyzers set on the field bucketName, match_phrase may not even be needed! terms will do just fine. The difference between term and match_phrase queries is nicely explained here.

Searching upon multiple date fields in ElasticSearch

I have a document like this:
{
"listings": {
"mappings": {
"listing": {
"properties": {
"auctionOn": {
"type": "date"
},
"inspections": {
"type": "nested",
"properties": {
"endsOn": {
"type": "date"
},
"startsOn": {
"type": "date"
}
}
},
// more fields.. snipped for brevity
}
}
}
}
}
and i would like to perform the following search: (needs to be a bool filter.. no scoring req'd)
return documents any of the inspections.startsOn matches any of the dates provided (if they are provided)
OR
return documents where auctionOn matches the date provided (if it's provided)
they can also specify to search for a) inspections only, b) auctions only. if not provided, either of the dates need to match.
So in other words, possible searches:
Search where there are any inspections/auctions
Search where there are any inspections
Search where there are any auctions
Search where there are any inspections/auctions on the dates provided
Search where there are any inspections on the dates provided
Search where there are any auctions on the dates provided
Now, i'm already in a bool query filter:
{
"query":
{
"bool":
{
"filter":[{"terms":{"location.suburb":["Camden"]}}
}
}
}
and i need this new filter to be seperate. so.. this is like a nested or filter, within a main bool filter?
So if provided "Suburb = Camden, Dates = ['2018-11-01','2018-11-02']'
then it should return documents where the suburb = Camden and either the inspections or auction date includes one of the dates provided.
I'm kinda stumped on how to do it, so any help would be much appreciated!
There will lot of bool query combinations for the cases you mentioned in the question. Taking the example you mention i.e.
So if provided "Suburb = Camden, Dates = ['2018-11-01','2018-11-02']'
then it should return documents where the suburb = Camden and either
the inspections or auction date includes one of the dates provided.
Assuming your location filter is working as expected, for dates part in the above e.g. additions to the query will be:
{
"query": {
"bool": {
"filter": [
{
"terms": {
"location.suburb": [
"Camden"
]
}
},
{
"bool": {
"should": [
{
"terms": {
"auctionOn": [
"2018-11-01",
"2018-11-02"
]
}
},
{
"nested": {
"path": "inspections",
"query": {
"bool": {
"should": [
{
"terms": {
"inspections.startsOn": [
"2018-11-01",
"2018-11-02"
]
}
},
{
"terms": {
"inspections.endsOn": [
"2018-11-01",
"2018-11-02"
]
}
}
]
}
}
}
}
]
}
}
]
}
}
}

Elasticsearch query to match two different fields with exact values

I want to find the record in my elasticsearch index where it should match field "connectorSpecific.hostname.keyword" with value "tyco-fire.com" and field "hasForms" with value true.
Below is my elasticsearch query:
GET index1/_search
{
"query": {
"bool": {
"should": [
{ "match": { "connectorSpecific.hostname.keyword": "tyco-fire.com" }},
{ "match": { "hasForms": true }}
]
}
}
}
This query is returning records which also has field "hasForms" with value false. Not sure why.I am using a boolean should query.Any help is appreciated
If you want both constraints to match, then you should use bool/filter (or bool/must would work as well but since you're doing exact matching, you don't need scoring at all), like this:
GET index1/_search
{
"query": {
"bool": {
"filter": [
{ "match": { "connectorSpecific.hostname.keyword": "tyco-fire.com" }},
{ "match": { "hasForms": true }}
]
}
}
}

Resources