ElasticSearch using wildcard and term queries - elasticsearch

I'm new using Elastic Search, and i never used Lucene too.
I build this query:
{
"query" : {
"wildcard" : { "referer" : "*.domain.com*" }
},
"filter" : {
"query" : {
"term" : { "first" : "1" }
}
},
"facets" : {
"site_id" : {
"terms" : {
"field" : "site",
"size" : "70"
}
}
}
}
The wildcard is working great, but the term filter was ignored, what i did wrong?
I need to filter the results with both wildcard and term
Thanks!

Assuming what you are trying to do is applying the filter on the wildcard query results,
you can use a FilteredQuery. However, your case might fit better for a filter.
You use a query filter. Instead of that you may directly use a TermFilter in a FilteredQuery rather than making a filter out of a TermQuery. TermFilter should be faster as it directly uses the TermsEnum.
Note that results of Filters are cached in a FilterCache and Filters are faster because they do not do any scoring of documents. In your case, even though the filter part of the FilteredQuery will work fast, but the wildcard query will be unnecessarily do scoring. You may try to use an AND Filter to club both queryfilter(wildcard query) and term filter instead of a FilteredQuery.
To make just the filter work as required by you, try something like below. (Not tried myself)
{
"filtered" : {
"query" : {
"wildcard" : { "referer" : "*.domain.com*" }
},
"filter" : {
"term" : { "first" : "1" }
}
},
"facets" : {
"site_id" : {
"terms" : {
"field" : "site",
"size" : "70"
}
}
}
}

Related

Elasticsearch bool query join order

Raising this question to know the order in which ES executes query clauses (must, should, filter, must_not) that are part of bool query. Sharing the sample query from ES docs -
{ "query": {
"bool" : {
"must" : {
"term" : { "user.id" : "kimchy" }
},
"filter": {
"term" : { "tags" : "production" }
},
"must_not" : {
"range" : {
"age" : { "gte" : 10, "lte" : 20 }
}
},
"should" : [
{ "term" : { "tags" : "env1" } },
{ "term" : { "tags" : "deployed" } }
],
"minimum_should_match" : 1,
"boost" : 1.0
} } }
From the documentation it looks like query-clauses are joined using AND condition. For example, above search DSL's SQL counterpart would look like (rough translation) -
select * from user where user_id like 'kimchy' and tags in ('production') and not (10 <= range <= 20) and tags in ('env1', 'deployed');
I actually wasn't able to find official documentation around this, but did see some texts that ES query-evaluation heavily depends on certain cost approximations. Wondering how to map the ordering to SQL like syntax so, we can develop a clear mental picture when authoring ES queries. It also feels like ordering might have some affect for deeply nested boolean AND OR queries.

Sum of all the values of a field in Kibana using elasticsearch query DSL

I have seen quite a few similar questions answered but they are all for older versions of Kibana, or do not actually help with my particular question.
I want to find the sum of all values in a specific field,the kibana docs give the following example code for creating the sum of a field.
POST /sales/_search?size=0
{
"query" : {
"constant_score" : {
"filter" : {
"match" : { "type" : "hat" }
}
}
},
"aggs" : {
"hat_prices" : { "sum" : { "field" : "price" } }
}
}
Based on this, the following should sum all the values in the field "tweetSentiment.polarity"
(POST /sales/_search?size=0 was removed because the UI gives an "unexpected 'p'" error with that line in.)
{
"query" : {
"constant_score" : {
"filter" : {
"match" : { "type" : "number" }
}
}
},
"aggs" : {
"hat_prices" : { "sum" : { "field" : "tweetSentiment.polarity" } }
}
}
Changing around the values for "type" and "field" between all the possible combinations of things they could be did not solve the issue either. My best guess is that this is not actually the code I want, especially after digging deep into how to create the query I am looking for.

elasticsearch percolator filter fails

I'm using a document query against a percolator that works ok. When I try to filter the percolator queries against which document percolate using queries ids, it doesn't return any result. For example:
{
"doc" : {
"text" : "This is the text within my document"
},
"highlight" : {
"order" : "score",
"pre_tags" : ["<example>"],
"post_tags" : ["</example>"],
"fields" : {
"text" : { "number_of_fragments" : 0 }
}
},
"filter":{"ids":{"values":[11,15]}}
,
"size" : 100
}
I know for sure that those ids are correct, but allways obtain "matches" : [ ]. When I don't use filter, ES retrieves correct matches.
Thanks for your help.
I think I've solved it. It seems that the filter only works on the "metadata" fields, meaning that you have to add customized fields to the queries indexed in the percolator in order to use them to filter when you need.
Using my previous example, I would have to index in percolator queries like:
{
"query" : {
"match_phrase" : {
"text" : "document"
}
},
"id" : 11
}
Adding "manually" a redundant id field in order to use it later as filter reference.
At percolation time, you have to use something like:
{
"doc" : {
"text" : "This is the text within my document"
},
"filter":{"match":{"id":11}},
"highlight" : {
"order" : "score",
"pre_tags" : ["<example>"],
"post_tags" : ["</example>"],
"fields" : {
"text" : { "number_of_fragments" : 0 }
}
},
"size" : 100
}
In order to use only that percolator query. Complementary information can be found here.

How to query on multiple fields in elasticsearch?

i have tried the multiple field query and it works fine. But I would like to know what other options are generally used to query multiple fields in elasticsearch?
Structured queries with multiple terms, for finding exact values, the same as SQL
https://www.elastic.co/guide/en/elasticsearch/guide/current/_finding_multiple_exact_values.html
"bool" : {
"must" : [
{ "term" : { "tags" : "search" } },
{ "term" : { "tag_count" : 1 } }
]
}
For example, consider following sql query,
SELECT product
FROM products
WHERE (price = 20 OR productID = "XHDK-A-1293-#fJ3")
AND (price != 30)
In these situations, you will need the bool filter. This is a compound filter that accepts other filters as arguments, combining them in various Boolean combinations.
The Query DSL would be,
GET /my_store/products/_search
{
"query" : {
"filtered" : {
"filter" : {
"bool" : {
"should" : [
{ "term" : {"price" : 20}},
{ "term" : {"productID" : "XHDK-A-1293-#fJ3"}}
],
"must_not" : {
"term" : {"price" : 30}
}
}
}
}
}
}
Follow the below link for documentation
https://www.elastic.co/guide/en/elasticsearch/guide/current/combining-filters.html

Do query results impact elasticsearch phrase suggestions?

I'd like to know whether Elasticsearch users query results to populate phrase suggestions for direct generator or not?
Or it simply picks tokens from given index?
My queries are based on some permission sets.
So for instance, that'd be my query:
{
"size" : 0,
"query" : {
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"bool" : {
"must" : [{
"terms" : {
"Permissions" : ["permission1", "permission2", "permission3"
]
}
}
]
}
}
}
},
"suggest" : {
"DidYouMean" : {
"text" : "{{SearchPhrase}}",
"phrase" : {
"field" : "_all",
"analyzer" : "simple",
"size" : 1,
"real_word_error_likelihood" : 0.96,
"max_errors" : 5,
"gram_size" : 3,
"direct_generator" : [{
"field" : "_all",
"suggest_mode" : "popular",
"min_word_length" : 3
}
]
}
}
}
}
How would I ensure that direct generator creates suggestions and doesn't violate my permissions clause?
Is this even possible?
The term suggester and phrase suggester feeds on the tokens for generating suggest results. The query does not affect the suggest results. The suggester directly works on the reverse index and get the tokens from them. So its scope is global and never the query

Resources