Elasticsearch using special characters - elasticsearch

Question

I'm indexing pdf's into Elasticsearch.
When I search for content like "§ 123", than the "§" is ignored.
What do I have to do so that "§" is also included in the search?
Here the Indices.Create:
CreateIndexResponse createIndexResponse = elasticClient.Indices.Create(sep.DefaultIndex, c => c
.Settings(s => s
.Analysis(a => a
.Analyzers(ad => ad
.Custom("windows_path_hierarchy_analyzer", ca => ca
.Tokenizer("windows_path_hierarchy_tokenizer")
)
)
.Tokenizers(t => t
.PathHierarchy("windows_path_hierarchy_tokenizer", ph => ph
.Delimiter('\\')
)
.NGram("ngram_tokenizer", td => td
.MinGram(2)
.MaxGram(20)
.TokenChars(
TokenChar.Letter,
TokenChar.Digit,
TokenChar.Symbol)
)
)
)
)
.Map<ElasticDocument>(mp => mp
.AutoMap()
.Properties(ps => ps
.Text(s => s
.Name(n => n.Path)
.Analyzer("windows_path_hierarchy_analyzer")
)
.Object<Attachment>(a => a
.Name(n => n.Attachment)
.AutoMap()
)
)
)
);
Here the mappings:
{
"attachments": {
"mappings": {
"properties": {
"attachment": {
"properties": {
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content_length": {
"type": "long"
},
"content_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"creator_tool": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"date": {
"type": "date"
},
"description": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"format": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"keywords": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"language": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"metadata_date": {
"type": "date"
},
"modified": {
"type": "date"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"id": {
"type": "long"
},
"instance": {
"type": "long"
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

Answer 1

Tldr;
As pointed out by #llermaly in the mapping you do not seems to be using the analyser you are creating with your code.
Be default text fields are going to be analysed with the standard analyser.
Solution
You will need to specify the analyser you want to be used on text fields
{
"mappings": {
"properties": {
"title": {
"type": "text",
"analyzer": "windows_path_hierarchy_analyzer"
}
}
}
}
else the standard analyser is used.
Which is going to delete special char.
POST _analyze
{
"analyzer": "standard",
"text": "§ 123"
}
gives you:
{
"tokens": [
{
"token": "123",
"start_offset": 2,
"end_offset": 5,
"type": "<NUM>",
"position": 0
}
]
}
The § has been removed.