How to remove stop words with attachment processor in Elastisearch? - elasticsearch

Question

I have to following index template:
PUT _index_template/aclimdb_1
{
"index_patterns": [
"aclimdb-*"
],
"template": {
"settings": {
"number_of_shards": 1,
"index": {
"max_result_window": "25000"
}
},
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"attachment.content": {
"type": "text",
"analyzer": "english"
},
"label": {
"type": "integer"
},
"class": {
"type": "integer"
}
}
}
},
"version": 1,
"priority": 500,
"composed_of": [
],
"_meta": {
"description": "aclImdb index"
}
}
and the following mappings:
{
"mappings": {
"_doc": {
"properties": {
"attachment": {
"properties": {
"content": {
"type": "text",
"analyzer": "english"
},
"content_length": {
"type": "long"
},
"content_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"language": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"class": {
"type": "integer"
},
"data": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"filename": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"label": {
"type": "integer"
}
}
}
}
}
and the following ingest pipeline:
[
{
"attachment": {
"field": "data"
}
},
{
"html_strip": {
"field": "attachment.content"
}
}
]
I would like to remove the stop words from the terms list when I upload the documents.How can I do that?
Do I have to do it in the ingest pipeline or in the settings of the index template?
Thank you very much for your help
Edit:
Thanks to the answer below I found a way to solve the problem. Here is my new index template.
PUT _index_template/aclimdb_1
{
"index_patterns": [
"aclimdb-*"
],
"template": {
"settings": {
"number_of_shards": 1,
"index": {
"max_result_window": "25000"
},
"analysis": {
"filter": {
"english_stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
},
"english_possessive_stemmer": {
"type": "stemmer",
"language": "possessive_english"
}
},
"analyzer": {
"english_analyzer": {
"tokenizer": "standard",
"filter": [
"english_possessive_stemmer",
"lowercase",
"english_stop",
"english_stemmer"
]
}
}
}
},
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"attachment.content": {
"type": "text",
"analyzer": "english_analyzer"
},
"label": {
"type": "integer"
},
"class": {
"type": "integer"
}
}
}
},
"version": 1,
"priority": 500,
"composed_of": [],
"_meta": {
"description": "aclImdb index"
}
}
Is there a cleaner way how I could achieve the same result?
Now I have the next challenge. I would like to filter out all numeric values. How would a filter to achieve that look like?

Answer 1

You can create a field with the analyzer stop word english. Then, when you index your token not have the stop words.
Read more about stop words filter.
Example:
PUT idx_stop_words
{
"settings": {
"analysis": {
"filter": {
"english_stop_filter": {
"type": "stop",
"stopwords": "_english_"
}
},
"analyzer": {
"english_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"english_stop_filter"
]
}
}
}
},
"mappings": {
"properties": {
"content": {
"type": "text",
"analyzer": "english_analyzer"
}
}
}
}
GET idx_stop_words/_analyze
{
"field": "content",
"text": "the woman in the window"
}
Token:
{
"tokens": [
{
"token": "woman",
"start_offset": 4,
"end_offset": 9,
"type": "<ALPHANUM>",
"position": 1
},
{
"token": "window",
"start_offset": 17,
"end_offset": 23,
"type": "<ALPHANUM>",
"position": 4
}
]
}