In my index I have inserted fields without changing the case of values(Upper case or Lower case), like in my elasticsearch document a field name contains value Hello World. And i have made name field as not_analyzed for exact match. But in that case, when i search by hello world this document don’t returned by elasticsearch, might be due to case sensitivity. I have tried by using term query and match query but haven't found a luck.
Please suggest, if there is a way.
Thanks
The only way you can do this in Elasticsearch is by analyzing the field and using token filters. There is a lowercase token filter available that you should use but this can't really be done on-the-fly like SQL where you wrap the field to be queried against in something like LOWER().
To get the effect you desire I would use something like the Keyword tokenizer with the Lowercase token filter. If you set this analyzer to be the default analyzer for indexing and searching then your searches will also be case insensitive too.
Related
Let's say I have two texts:
Text 1 - "The fox has been living in the wood cabin for days."
Text 2 - "The wooden hammer is a dangerous weapon."
And I would like to search for the word "wood", without it matching me "wooden hammer". How would I do that in Elastic Search or nest?
Term query is used for exact matches search. However it's not recommended to use it against text fields, the following quote from term query documentation:
To better search text fields, the match query also analyzes your
provided search term before performing a search. This means the match
query can search text fields for analyzed tokens rather than an exact
term.
The term query does not analyze the search term. The term query only
searches for the exact term you provide. This means the term query may
return poor or no results when searching text fields.
The problem with text exact matches, as described in the Term query documentation:
By default, Elasticsearch changes the values of text fields as part of
analysis. This can make finding exact matches for text field values
difficult.
So, the documents data is modified (i.e., analyzed) before indexing. This depends on the index mapping definition for each field, defaults to the default index analyzer, or the standard analyzer.
But the default standard analyzer will not change the token "Wooden" to "Wood", this might happen if you used stemming for this field.
This means, if you don't use a different analyzer or stemming, querying with "Wood" shouldn't match "Wooden" token.
To summarize: Indexed data is modified/analyzed before indexing (based on the field mapping definition). Match query analyze the search query, while Term query doesn't analyze the search query. So you have to properly chose the field mapping and the search query to better suit your use case
For some use cases, like storing email addressed, phone numbers or keyword fields that always have the same value, consider using the Keyword type, which is suitable for exact matches in these use cases. However, ES recommends:
Avoid using keyword fields for full-text search. Use the text field
type instead.
So for better visibility and practical solution for your use case, it's better to elaborate more the field mapping you use and what you want to achieve.
I would like to create SQL query on some text field (not keyword) for example "name" field and send that query to elastic server.
my problem is that I need to use the standard SQL language (not the MATCH and QUERY operators which are specials for elastic SQL) of text fields.
when I tried to use JDBC driver or when I tried to use high-level-java-client with LIKE operatorI got the following error
"No keyword/multi-field defined exact matches for [name]; define one or use MATCH/QUERY instead"
I also tried to use the translate API of elasticsearch- but even there I couldn't use the "LIKE" operator on text fields only on keyword fields.
does anyone have any solution for me? I want to use the LIKE operator on text fields instead of the full text operators which are unique to elastic sql.
Please check the this documentation. they have clearly mentioned in document that it is not possible.
One significant difference between LIKE/RLIKE and the full-text search
predicates is that the former act on exact fields while the latter
also work on analyzed fields. If the field used with LIKE/RLIKE
doesn’t have an exact not-normalized sub-field (of keyword type)
Elasticsearch SQL will not be able to run the query. If the field is
either exact or has an exact sub-field, it will use it as is, or it
will automatically use the exact sub-field even if it wasn’t
explicitly specified in the statement.
If you still want to used text field then you need to enabled multi-field as mentioned here. or you can try out to enable fielddata on text field but i am not sure that it will work SQL or not.
The Wildcard Query offers a rewrite parameter to influence how Lucene calculates the relevance scores. On keyword fields this works as expected but it does not seem to work with the new wildcard field type which belongs to the keyword family. Is this an expected behavior or a bug?
As confirmed by Elastic staff, the rewrite parameter is unsupported. Unlike keyword fields, the wildcard field doesn't have a single indexed token for each term so it has no pre-built count for the document frequency of whole values. Instead it uses an ngram index which obviously has different frequencies for the multiple terms a search string can be broken down into.
I have a use case which is a bit similar to the ES example of dynamic_template where I want certain strings to be analyzed and certain not.
My document fields don't have such a convention and the decision is made based on an external schema. So currently my flow is:
I grab the inputs document from the DB
I grab the approrpiate schema (same database, currently using logstash for import)
I adjust the name in the document accordingly (using logstash's ruby mutator):
if not analyzed I don't change the name
if analyzed I change it to ORIGINALNAME_analyzed
This will handle the analyzed/not_analyzed problem thanks to dynamic_template I set but now the user doesn't know which fields are analyzed so there's no easy way for him to write queries because he doesn't know what's the name of the field.
I wanted to use field name aliases but apparently ES doesn't support them. Are there any other mechanisms I'm missing I could use here like field rename after indexation or something else?
For example this ancient thread mentions that field.sub.name can be queried as just name but I'm guessing this has changed when they disallowed . in the name some time ago since I cannot get it to work?
Let the user only create queries with the original name. I believe you have some code that converts this user query to Elasticsearch query. When converting to Elasticsearch query, instead of using the field name provided by the user alone use both the field names ORIGINALNAME as well as ORIGINALNAME_analyzed. If you are using a match query, convert it to multi_match. If you are using a term query, convert it to a bool should query. I guess you get where I am going with this.
Elasticsearch won't mind if a field does not exists. This can be a problem if there is already a field with _analyzed appended in its original name. But with some tricks that can be fixed too.
I have an analyser called autocomplete_analyser defined on a field name. When I run the query
http://localhost:9200/courses/course/_search?q=name:dav&pretty=true
it runs the analyser and returns the correct results. When I run
http://localhost:9200/courses/course/_search?q=dav&pretty=true
it does not.
How can I make ES run the analyser without me specifying the fields being searched on?
I need to use this analyser across a number of fields so its important that I can search all of them.
By default, queryString queries are applied on _all field which have its own analyzer.
You can define your specific analyzer for the _all field using the Put Mapping API.
Does it help?