I need to get result below with NEST (Elastic Search .NET client)
"detailVal": {
"name": "detailVal",
"type": "multi_field",
"fields": {
"detailVal": {
"type": "string"
},
"untouched": { // <== FOCUS 2
"type": "string",
"index": "not_analyzed",
"omit_norms": true,
"include_in_all": false,
"index_options": "docs" // <== FOCUS 1
}
}
}
I have done so far
[ElasticProperty(OmitNorms = true, Index = FieldIndexOption.not_analyzed, IncludeInAll = false, AddSortField = true)]
public string DetailVal { get; set; }
which gets me
"detailVal": {
"name": "detailVal",
"type": "multi_field",
"fields": {
"detailVal": {
"type": "string",
"index": "not_analyzed",
"omit_norms": true,
"include_in_all": false
},
"sort": { // <== FOCUS 2
"type": "string",
"index": "not_analyzed"
}
}
}
so, any idea how to
add "index_options": "docs" (I found IndexOptions.docs but it is not valid as Attribute)
change sort to untouched
The attribute based mapping only gets you so far. It's good enough if you only need to change names and set simple properties.
The recommended approach is to use client.MapFluent()
See https://github.com/Mpdreamz/NEST/blob/master/src/Nest.Tests.Unit/Core/Map/FluentMappingFullExampleTests.cs#L129
For an example how to set index_options
And line 208:
https://github.com/Mpdreamz/NEST/blob/master/src/Nest.Tests.Unit/Core/Map/FluentMappingFullExampleTests.cs#L208
To see how you can create your own multi_field mapping.
You can even combine both approaches:
client.MapFluent<MyType>(m=>m
.MapFromAttributes()
//Map what you can't with attributes here
);
client.Map() and client.MapFromAttributes() will most likely be removed at some point.
Related
I want my elasticsearch index to match the exact value for all the fields. How do I map my index to "not_analysed" for all the fields.
I'd suggest making use of multi-fields in your mapping (which would be default behavior if you aren't creating mapping (dynamic mapping)).
That way you can switch to traditional search and exact match searches when required.
Note that for exact matches, you would need to have keyword datatype + Term Query. Sample examples are provided in the links I've specified.
Hope it helps!
You can use dynamic_templates mapping for this. As a default, Elasticsearch is making the fields type as text and index: true like below:
{
"products2": {
"mappings": {
"product": {
"properties": {
"color": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
As you see, also it creates a keyword field as multi-field. This keyword fields indexed but not analyzed like text. if you want to drop this default behaviour. You can use below configuration for the index while creating it :
PUT products
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"product": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"index": false
}
}
}
]
}
}
}
After doing this the index will be like below :
{
"products": {
"mappings": {
"product": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"index": false
}
}
}
],
"properties": {
"color": {
"type": "keyword",
"index": false
},
"type": {
"type": "keyword",
"index": false
}
}
}
}
}
}
Note: I don't know the case but you can use the multi-field feature as mentioned by #Kamal. Otherwise, you can not search on the not analyzed fields. Also, you can use the dynamic_templates mapping set some fields are analyzed.
Please check the documentation for more information :
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html
Also, I was explained the behaviour in this article. Sorry about that but it is Turkish. You can check the example code samples with google translate if you want.
I have created the index in elasticsearch with the following mapping:
{
"test": {
"mappings": {
"documents": {
"properties": {
"fields": {
"type": "nested",
"properties": {
"uid": {
"type": "keyword"
},
"value": {
"type": "text",
"copy_to": [
"fulltext"
]
}
}
},
"fulltext": {
"type": "text"
},
"tags": {
"type": "text"
},
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"url": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
}
While searching I want to set the preference of fields for example if search text found in title or url then that document comes first then other documents.
Can we set a field preference for search result sequence(in my case preference like title,url,tags,fields)?
Please help me into this?
This is called "boosting" . Prior to elasticsearch 5.0.0 - boosting could be applied in indexing phase or query phase( added as part of field mapping ). This feature is deprecated now and all mappings after 5.0 are applied in query time .
Current recommendation is to to use query time boosting.
Please read this documents to get details on how to use boosting:
1 - https://www.elastic.co/guide/en/elasticsearch/guide/current/_boosting_query_clauses.html
2 - https://www.elastic.co/guide/en/elasticsearch/guide/current/_boosting_query_clauses.html
I am trying to sort a search by the length of a list. I have a field defined by
"authors": {
"type": "text", "norms": { "enabled": False }, "analyzer": "autocomplete_with_asciifolding", "search_analyzer": "with_and_without_accent_search", "position_increment_gap": 100,
"fields": {
"raw" : { # This field is needed for your_paper suggestions search
"type" : "string",
"analyzer" : 'lower_keyword'
}
}
}
Now I want to sort by this field, which might look like ['John Doe', 'Mike Smith']
Following the elasticsearch documentation and some google searches I found
doc['sort'] = {
"_script": {
"type": "number",
"script": "doc['authors'].length",
"order": "asc"
}
}
I also tried doc['authors'].values.length() and doc['authors'].values.size(), but all of them result in
TransportError: TransportError(500, u'search_phase_execution_exception')
any idea how to sort by the length of a list?
You can't access to doc['authors'] because field data is disabled in text fields by default for performance reasons.
You can, for example, set fielddata=true in your inner field raw:
"authors": {
"type": "text", "norms": { "enabled": False }, "analyzer": "autocomplete_with_asciifolding", "search_analyzer": "with_and_without_accent_search", "position_increment_gap": 100,
"fields": {
"raw" : { # This field is needed for your_paper suggestions search
"type" : "string",
"analyzer" : 'lower_keyword',
"fielddata": True
}
}
}
And sort by authors.raw instead:
doc['sort'] = {
"_script": {
"type": "number",
"script": "doc['authors.raw'].length",
"order": "asc"
}
}
Another solution would be to create a new inner field of type keyword and sort by it. But I suggested to use raw because it looks that analyzer just lower the case.
I've got the following simple mapping:
"element": {
"dynamic": "false",
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"group": { "type": "string", "index": "not_analyzed" },
"type": { "type": "string", "index": "not_analyzed" }
}
}
Which basically is a way to store Group object:
{
id : "...",
elements : [
{id: "...", type: "..."},
...
{id: "...", type: "..."}
]
}
I want to find how many different groups exist sharing the same set of element types (ordered, including repetitions).
An obvious solution would be to change the schema to:
"element": {
"dynamic": "false",
"properties": {
"group": { "type": "string", "index": "not_analyzed" },
"concatenated_list_of_types": { "type": "string", "index": "not_analyzed" }
}
}
But, due to the requirements, we need to be able to exclude some types from group by (aggregation) :(
All fields of the document are mongo ids, so in SQL I would do something like this:
SELECT COUNT(id), concat_value FROM (
SELECT GROUP_CONCAT(type_id), group_id
FROM table
WHERE type_id != 'some_filtered_out_type_id'
GROUP BY group_id
) T GROUP BY concat_value
In Elastic with given mapping it's really easy to filter out, its also not a problem to count assuming we have a concated value. Needless to say, sum aggregation does not work for strings.
How can I get this working? :)
Thanks!
Finally I solved this problem with scripting and by changing the mapping.
{
"mappings": {
"group": {
"dynamic": "false",
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"elements": { "type": "string", "index": "not_analyzed" }
}
}
}
}
There are still some issues with duplicate elements in array (ScriptDocValues.Strings) for some reason strips out dups, but here's an aggregation that counts by string concat:
{
"aggs": {
"path": {
"scripted_metric": {
"map_script": "key = doc['elements'].join('-'); _agg[key] = _agg[key] ? _agg[key] + 1 : 1",
"combine_script": "_agg",
"reduce_script": "_aggs.collectMany { it.entrySet() }.inject( [:] ) { result, e -> result << [ (e.key):e.value + ( result[ e.key ] ?: 0 ) ]}"
}
}
}
}
The result would be as follows:
"aggregations" : {
"path" : {
"value" : {
"5639abfb5cba47087e8b457e" : 362,
"568bfc495cba47fc308b4567" : 3695,
"5666d9d65cba47701c413c53" : 14,
"5639abfb5cba47087e8b4571-5639abfb5cba47087e8b457b" : 1,
"570eb97abe529e83498b473d" : 1
}
}
}
I have approximately 3300 documents with geo_point typed field filled.
When I try to visualize my documents on the tile map, kibana says "no results found".
I've already tried putting coordinates as:
- geohash in string
- [lon, lat] array
- object with "lat" and "lon" properties
- string "lat,lon"
All these ways of setting geo_point are allowed according to ES docs.
Kibana detects this field as geo_point (there is a globe icon near field name), but nothing shows up on tile map.
What's wrong with me?
I'm using Kibana 4.2, elasticsearch 2.0.0
I've managed it.
It was happening because I had my geo_point typed field inside of the field with "type": "nested" parameter.
I've changed this outer field to "dynamic": "true" and now I can visualize my locations!
I was able to have a nested geo_point by removing the "type": "nested" from the mapping. No "dynamic":"true" needed. My mapping looks like this:
"mappings": {
"_default_": {
"_all": {
"enabled": true
},
"_ttl": {
"enabled": true,
"default": "12m"
},
"dynamic_templates": [{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "analyzed",
"omit_norms": true,
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
}
}
}
}],
"properties": {
"#version": {
"type": "string",
"index": "not_analyzed"
},
"user_data": {
"properties": {
"user_geolocation": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
}
}
}