Can I have multiple filters in an Elasticsearch index's settings? - elasticsearch

I want an Elasticsearch index that simply stores "names" of features. I want to be able to issue phonetic queries and also type-ahead style queries separately. I would think I would be able to create one index with two analyzers and two filters; each analyzer could use one of the filters. But I do not seem to be able to do this.
Here is the index settings json I'm trying to use:
{
"settings": {
"number_of_shards": 1,
"analysis": {
"analyzer": {
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["standard", "lowercase", "ngram"]
}
},
"analyzer": {
"phonetic_analyzer": {
"tokenizer": "standard",
"filter": "double_metaphone_filter"
}
},
"filter": {
"double_metaphone_filter": {
"type": "phonetic",
"encoder": "double_metaphone"
}
},
"filter": {
"ngram": {
"type": "ngram",
"min_gram": 2,
"max_gram": 15
}
}
}
}
}
When I attempt to create an index with these settings:
http://hostname:9200/index/type
I get an HTTP 400, saying
Custom Analyzer [phonetic_analyzer] failed to find filter under name [double_metaphone_filter]
Don't get me wrong, I fully realize what that sentence means. I looked and looked for an erroneous comma or quote but I don't see any. Otherwise, everything is there and formatted correctly.
If I delete the phonetic analyzer, the index is created but ONLY with the autocomplete analyzer and ngram filter.
If I delete the ngram filter, the index is created but ONLY with the phonetic analyzer and phonetic filter.
I have a feeling I'm missing a fundamental concept of ES, like only one analyzer per index, or one filter per index, or I must have some other logical dependencies set up correctly, etc. It sure would be nice to have a logical diagram or complete API spec of the Elasticsearch infrastructure, i.e. any index can have 1..n analyzers, only 1 filter, query must need any one of bool, match, etc. But that unicorn does not seem to exist.
I see tons of documentation, blog posts, etc on how to do each of these functionalities, but with only one analyzer and one filter on the index. I'd really like to do this dual functionality on one index (for reasons out of scope).
Can someone offer some help, advice here?

You are just missing the proper formatting for your settings object. You cannot have two analyzer or filter keys, as there can only be one value per key in this settings map object. Providing a list of your filters seems to work just fine. When you were creating your index object, the second key was overriding the first.
Look here:
"settings": {
"number_of_shards": 1,
"analysis": {
"filter": {
"double_metaphone_filter": {
"type": "phonetic",
"encoder": "double_metaphone"
},
"ngram": {
"type": "ngram",
"min_gram": 2,
"max_gram": 15
}
},
"analyzer": {
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["standard", "lowercase", "ngram"]
},
"phonetic_analyzer": {
"tokenizer": "standard",
"filter": "double_metaphone_filter"
}
}
}
}
I downloaded the plugin to confirm this works.
You can now test this out at the _analyze enpoint with a payload:
{
"analyzer":"autocomplete_analyzer",
"text":"Jonnie Smythe"
}

Related

Elastic search with Java: exclude matches with random leading characters in a letter

I am new to using elastic search. I managed to get things working somewhat close to what I intended. I am using the following configuration.
{
"analysis": {
"filter": {
"shingle_filter": {
"type": "shingle",
"min_shingle_size": 2,
"max_shingle_size": 3,
"output_unigrams": true,
"token_separator": ""
},
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"shingle_search": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase"
]
},
"shingle_index": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"shingle_filter",
"autocomplete_filter"
]
}
}
}
}
I have this applied over multiple fields and doing a multi match query.
Following is the java code:
NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.multiMatchQuery(i)
.field("title")
.field("alias")
.fuzziness(Fuzziness.ONE)
.type(MultiMatchQueryBuilder.Type.BEST_FIELDS))
.build();
The problem is it matches with fields that have letters with some leading characters.
For example, if my search input is "ron" I want it to match with "ron mathews", but I don't want it match with "iron". How can I make sure that I am matching with letters having no leading characters?
Update-1
Turning off fuzzy transposition seems to improve search results. But I think we can make it better.
You probably want to score "ron" higher than "ronaldo" and the exact match of complete field "ron" even higher so the best option here would be to use few subfields with standard and keyword analyzers and boost those fields in your multi_match query.
Also, as you figured out yourself, be careful with the fuzziness. Might make sense to run 2 queries in a should with one being fuzzy and another boosted so that exact matches are ranked higher.

Auto Suggestions in Elastic Search after 3 letters

I've a search query which does basic search after a complete word is typed in. I'm looking for auto suggestions after 3 letters.
For Example,
Title- samsung galaxy s4
I want to see auto suggestions after "sam" instead of complete word "samsung".
while the ngram filter works, there is a dedicated suggester for this use-case, called the completion suggester, which uses another data structure internal, which will allow you to execute suggestions in the millisecond range, thus being much faster than a regular query use edgengram. Check out the documentation here
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/search-suggesters-completion.html
You need to use an edgeNGram filter for this.
{
"analysis": {
"tokenizer": {
"autocomplete_tokenizer": {
"type": "edgeNGram",
"min_gram": "3",
"max_gram": "20"
}
},
"analyzer": {
"autocomplete_edge_ngram": {
"filter": ["lowercase"],
"type": "custom",
"tokenizer": "autocomplete_tokenizer"
}
}
}
}
and mapping will be
{
"title_edge_ngram": {
"type": "text",
"analyzer": "autocomplete_edge_ngram",
"search_analyzer": "standard"
}
Or you can use the completion suggester in elasticsearch.
For three character check, you have to do it in your client side itself.

Tokenize a big word into combination of words

Suppose I have Super Bowl is the value of a document's property in the elasticsearch. How can the term query superbowl match Super Bowl?
I read about letter tokenizer and word delimiter but both don't seem to solve my problem. Basically I want to be able to convert combination of a large word into meaningful combination of words.
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-letter-tokenizer.html
I know this is quite late but you could use synonym filter
You could define that super bowl is the same as "s bowl", "SuperBowl" etc.
There are ways to do this without changing what you actually index. For example, if you are using at least 5.2 (where normalizers were introduced), but it can also be earlier version but 5.x makes it easier, you can define a normalizer to lowercase your text and not change it and then use a fuzzy query at search time to account for the space between super and bowl. My solution though is specific to this example you have given. As it is with Elasticsearch most of time, one needs to think about what kind of data goes into Elasticsearch and what it is required at search time.
In any case, if you are interested in an approach here it is:
DELETE test
PUT /test
{
"settings": {
"analysis": {
"normalizer": {
"my_normalizer": {
"type": "custom",
"char_filter": [],
"filter": ["lowercase", "asciifolding"]
}
}
}
},
"mappings": {
"test": {
"properties": {
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"normalizer": "my_normalizer"
}
}
}
}
}
}
}
POST test/test/1
{"title":"Super Bowl"}
GET /test/_search
{
"query": {
"fuzzy": {
"title.keyword": "superbowl"
}
}
}

Elasticsearch and Drupal : how to add filter lowercase and asciifolding

I created an index in Drupal, and my queries works.
Now I try to add filters lowercase and asciifolding in the elasticsearch.yml file, but unsuccessfully:
I add these lines :
index:
analysis:
analyzer:
default:
filter : [standard, lowercase, asciifolding]
I have an error : IndexCreationException: [myindex] failed to create index.
But 'myindex' already exist, I just try to add filters to this existing index.
How I can add these filters so that the indexation is correct for me?
Thank you very much for your help.
The reason that you get this exception is because it's not possible to update the settings of an index by calling the general create index endpoint. In order to update analyzers you will have to call the '_settings' endpoint.
I've made a small example for you on how to do this:
PUT test
{
"settings": {
"analysis": {
"analyzer": {
"new_analyzer": {
"tokenizer": "standard"
}
}
}
}
}
GET test/_analyze
{
"analyzer": "new_analyzer",
"text": "NoLowercasse"
}
POST test/_close
PUT test/_settings
{
"analysis": {
"analyzer": {
"new_analyzer": {
"tokenizer": "standard",
"filter": [
"asciifolding",
"lowercase"
]
}
}
}
}
POST test/_open
GET test/_analyze
{
"analyzer": "new_analyzer",
"text": "LowerCaseAdded"
}
Response:
{
"tokens": [
{
"token": "lowercaseadded",
"start_offset": 0,
"end_offset": 14,
"type": "<ALPHANUM>",
"position": 0
}
]
}
You can see that after the second analysis, the lowercase filter is being applied. The reason that you have to close your index is because it needs to rebuild the analyzer. You will notice that the new analyzer won't work as expected since the previously added documents weren't indexed with this analyzer, but rather the one without the asciifolding and lowercase.
In order to fix this, you'll have to rebuild your index (with the Reindex-API for example)
Hope this helps!
Edit: I maybe was a bit too quick in responding, as this is not a Drupal-Elastic solution, but it might point you in the right direction. To be honest I'm not familiar with running ES in combination with Drupal.

best setup for live data in elasticsearch

I am trying to use elasticsearch for live data filtering. Right now I use a single machine which gets constantly pushed new data (every 3 seconds via _bulk). Even so I did set up a ttl the index gets quite big after a day or so and then elasticsearch hangs. My current mapping:
curl -XPOST localhost:9200/live -d '{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"lowercase_keyword": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase"
]
},
"no_keyword": {
"type": "custom",
"tokenizer": "whitespace",
"filter": []
}
}
}
},
"mappings": {
"log": {
"_timestamp": {
"enabled": true,
"path": "datetime"
},
"_ttl":{
"enabled":true,
"default":"8h"
},
"properties": {
"url": {
"type": "string",
"search_analyzer": "lowercase_keyword",
"index_analyzer": "lowercase_keyword"
},
"q": {
"type": "string",
"search_analyzer": "no_keyword",
"index_analyzer": "no_keyword"
},
"datetime" : {
"type" : "date"
}
}
}
}
}'
I think a problem is purging the old documents but I could be wrong. Any ideas on how to optimize my setup?
To avoid elasticsearch hanging, you might want to increase amount of memory available to java process.
If all your documents have the same 8 hour life span, it might be more efficient to use rolling aliases instead of ttl. The basic idea is to create a new index periodically (every hour, for example) and use aliases to keep track of current indices. As time goes, you can update the list of indices in the alias that you search and simply delete indices that are more than 8 hour long. Deleting an index is much quicker than removing indices using ttl. A sample code that demonstrates how to create rolling aliases setup can be found here.
I am not quite sure how much live data you are trying to keep, but if you are just testing incoming data against a set of queries, you might also consider using Percolate API instead of indexing data.

Resources