I'm currently quite confuse about the structuring of queries in elastic. Let me explain what I mean with the following template that works fine for me:
{
"template" : {
"query" : {
"filtered" : {
"query" : {
"bool" : {
"must" : [
{ "match" : {
"user" : "{{param_user}}"
} },
{ "match" : {
"session" : "{{param_session}}"
} },
{ "range" : {
"date" : {
"gte" : "{{param_from}}",
"lte" : "{{param_to}}"
}
} }
]
}
}
}
}
}
}
Ok so I want to get entries of a specific session of a user in a certain time period. Now if you take a llok at this link http://www.elastic.co/guide/en/elasticsearch/guide/current/combining-filters.html you can find the following query:
{
"query" : {
"filtered" : {
"filter" : {
"bool" : {
"should" : [
{ "term" : {"price" : 20}},
{ "term" : {"productID" : "XHDK-A-1293-#fJ3"}}
],
"must_not" : {
"term" : {"price" : 30}
}
}
}
}
}
}
In this example we have right after the "filtered" the "filter" keyword. However if I exchange my second "query" with a "filter" as in the example , my template won't work anymore. This is really counterintuitive and I payed alot of time to figure this out. A̶l̶s̶o̶ ̶I̶ ̶d̶o̶n̶'̶t̶ ̶u̶n̶d̶e̶r̶s̶t̶a̶n̶d̶ ̶w̶h̶y̶ ̶w̶e̶ ̶n̶e̶e̶d̶ ̶t̶o̶ ̶p̶u̶t̶ ̶e̶v̶e̶r̶y̶ ̶f̶i̶l̶t̶e̶r̶ ̶i̶n̶ ̶s̶e̶p̶a̶r̶a̶t̶e̶ ̶̶{̶ ̶}̶̶ ̶e̶v̶e̶n̶ ̶t̶h̶o̶u̶g̶h̶ ̶t̶h̶e̶y̶ ̶a̶r̶e̶ ̶a̶l̶r̶e̶a̶d̶y̶ ̶s̶e̶p̶a̶r̶a̶t̶e̶d̶ ̶b̶y̶ ̶t̶h̶e̶ ̶a̶r̶r̶a̶y̶ ̶s̶y̶n̶t̶a̶x̶.̶
Another issue I had was that I suggested to match several fields I can just type smth like:
{
"query" : {
"match" : {
"user" : "{{param_user}}",
"session" : "{{param_session}}"
}
}
}
but it seemed that I have to use a bool query which I didn't know of, so I searched for 'elastic multi match' but got something completely different.
My question: where can I find how to structure a query properly (smth like a PEG)? The documentation only give basic examples but doesn't state what we can actually do and how.
Best regards,
Jan
Edit: Ok I just found by accident that I cannot exchange "query" with "filter" as "match" is a query and not a filter. But then again what about "range"? It seems to be a query as well as a filter... Is there a summary of keywords specifying in which context they can be used?
Is there a summary of keywords specifying in which context they can be used?
I wouldn't consider that as keywords. It's just there are both queries and filters with the same names (but not all of them).
Here is everything you need. For example there are both range query and filter. All you need is to understand the difference between filters and queries.
For example, if you want to move range section from query to filter, you can do that like shown in the code below (not tested). Since your code already contains filtered type of query, you can just create filter section right after query section.
{
"template": {
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"user": "{{param_user}}"
}
},
{
"match": {
"session": "{{param_session}}"
}
}
]
}
},
"filter": {
"range": {
"date": {
"gte": "{{param_from}}",
"lte": "{{param_to}}"
}
}
}
}
}
}
}
Just remember that you can filter only not analyzed fields.
Related
Say that I have to filter cars constructors in a Elastic Search Index (ES 7.15), where the field car_maker is mapped to keyword, having it a limited number of possibilities among car makers string names:
{
"mappings": {
"properties": {
"car_maker": {
"type": "keyword"
}
}
}
}
GET /cars/_search
{
"query": {
"bool": {
"filter": [{
"term": {
"car_maker": "Honda"
}
}]
}
}
}
This, along with a matching query will work ok. The filter will not participate to score calculation as desired.
Now I would like to to filter more car makers for that query (let's say a should query):
{
"query": {
"bool": {
"filter" : [
{"term" : { "car_maker" : "Honda"}},
{"term" : { "car_maker" : "Ferrari"}}
]
}
}
}
this is not going to work. I will have any error from ES query engine, but any result too. Of course is always possibile to apply more filters to different fields like car_maker and car_color, but how to do the opposite: apply more values (Honda, Ferrari, etc.) to the same filter field car_maker like in the example above, without conditioning the score calculation?
You might want to try the following filter query:
{
"query" : {
"bool" : {
"filter" : {
"terms" : {
"car_maker" : ["Honda", "Ferrari"]
}
}
}
}
}
Each document in my Elasticsearch index has two access control lists containing user ids. One is an allow list, the other is a deny list. I am trying to add a filter to a given query that considers these ACLs. I thought I could use a bool query with a must clause for the given query, a filter clause for the allow list, and a must_not clause for the deny list. What I have so far (example for user 1):
{
"bool" : {
"must" : {
[given query]
},
"filter" : [ {
"match" : {
"acl.allow" : {
"query" : "/user/1",
"type" : "boolean"
}
}
}],
"must_not" : [ {
"match" : {
"acl.deny" : {
"query" : "/user/1",
"type" : "boolean"
}
}
}]
}
}
Unfortunately, this query does not return the desired result. It returns objects that have not listed user 1 in their allow list (a behavior I don't understand). Also, it (obviously) ignores objects with empty access control lists (which should be visible to anyone). Any suggestions to fix that?
I figured it out. First of all, using match isn't really a good solution for that kind of query—due to its analyzer. Using term though left me puzzled why I did not get any results. Term queries only return results if the corresponding field is set to not_analyzed. Thus I changed my mapping:
"acl": {
"properties": {
"allow": {
"type": "string",
"index": "not_analyzed"
},
"deny": {
"type": "string",
"index": "not_analyzed"
}
}
}
My second problem—treating objects with empty ACLs as visible to anyone—was solved using exists nested in must_not nested in bool. This is recommended as substitute for the deprecated missing query. My final query looks like this and passed all ACL related tests I could think of.
{
"bool" : {
"must" : {
[given query]
},
"filter" : {
"bool" : {
"should" : [ {
"terms" : {
"acl.allow" : [ "/user/1" ]
}
}, {
"bool" : {
"must_not" : {
"exists" : {
"field" : "acl.allow"
}
}
}
} ]
}
},
"must_not" : {
"terms" : {
"acl.deny" : [ "/user/1" ]
}
}
}
}
I have the following filtered query:
{
"query": {
"filtered" : {
"query": {
....
},
"filter" : {
"bool" : {
"should" : [
{
bool : {
"must" : [
{ "term1" : { "name1" : "value1" } },
{ "term2" : { "name2" : "value2" } }
]
}
},
{
"bool" : {
"must" : [
......
]
}
},
{
"bool" : {
"must" : [
......
]
}
},
{
"bool" : {
"must" : [
.......
]
}
}
]
}
}
}
}
}
Is there any room for me to improve the complex filters without harming performance? How?
I feel the filter part can be simplified, but not sure. Also not sure about any performance impact.
Thanks!
UPDATE
Under each must clause, there is a GROUP of clauses. See the first must clause for an example.
UPDATE 2
According to Duc's input and Mario's updated answer, I am very happy that what I have so far is the right way. No changes are is needed.
I choose Mario's response as the answer because it confirms that mine is right.
You don't need Bool with only one must clause. You put your conditions directly within the outer should.
{
"query": {
"filtered" : {
"query": {
....
},
"filter" : {
"bool" : {
"should" : [
{
.......
},
{
.......
},
{
.......
},
{
.......
}
]
}
}
}
}
}
UPDATE: since I see from your update that you have several clauses within your nested must statements, my suggestion is no longer valid. Your query filter is ok like it is (except that term1 and term2 aren't valid, but that's not the point of the question)
Im trying to use the suggest-api in elasticsearch. And i would like to get total hits, as you get when you are doing a regular query.
As it is now, if i ask this of elasticsearch
/_suggest
{
"name_suggest": {
"text": "derp",
"completion": {
"size": 10,
"field": "name.sugest"
}
}
}
I get 10 answers, but no information on how many other matches there is.
So the question is, is there a way yo get hold of this information, using the suggest feature? for example using facets? (i have tried but not got anything working)
You can get your 10 answers and build aggregation query like this:
{
"query" : {
"query_string" : {
"query" : "answer1 OR answer2 OR ...",
"fields" : [ "name.sugest" ]
}
},
"aggregations" : {
"name_sugest" : {
"terms" : {
"field" : "name.sugest"
}
}
}
}
I'm new to Elasticsearch. I don't think I fully understand the concept of query and filters. In my case I just want to use filters as I don't want to use advance feature like scoring.
How would I convert the following SQL statement into elasticsearch query?
SELECT * FROM advertiser
WHERE company like '%com%'
AND sales_rep IN (1,2)
What I have so far:
curl -XGET 'localhost:9200/advertisers/advertiser/_search?pretty=true' -d '
{
"query" : {
"bool" : {
"must" : {
"wildcard" : { "company" : "*com*" }
}
}
},
"size":1000000
}'
How to I add the OR filters on sales_rep field?
Thanks
Add a "should" clause after your must clause. In a bool query, one or more should clauses must match by default. Actually, you can set the "minimum_number_should_match" to be any number, Check out the bool query docs.
For your case, this should work.
"should" : [
{
"term" : { "sales_rep_id" : "1" }
},
{
"term" : { "sales_rep_id" : "2" }
}
],
The same concept works for bool filters. Just change "query" to "filter". The bool filter docs are here.
I come across this post 4 years too late...
Anyways, perhaps the following code could be useful...
{
"query": {
"filtered": {
"query": {
"wildcard": {
"company": "*com*"
}
},
"filter": {
"bool": {
"should": [
{
"terms": {
"sales_rep_id": [ "1", "2" ]
}
}
]
}
}
}
}
}