[ElasticSearch]: Get all indices containing a specific value - elasticsearch

Following is the use case:
I am trying to list all indices in elasticsearch that contain a particular value.
For illustration purposes let us consider following to be the index template:
{
"order": 0,
"template": "sample-*",
"settings": {
"index.refresh_interval": "300",
"index.number_of_replicas": "1",
"index.number_of_shards": "10"
},
"mappings": {
"digital": {
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"properties": {
"website": {
"index": "not_analyzed",
"store": false,
"type": "string",
"doc_values": true
},
"iab_codes": {
"store": false,
"type": "long",
"doc_values": true
},
"audiences": {
"store": false,
"type": "long",
"doc_values": true
}
}
}
},
"aliases": {
}
}
The audiences field in this template is a sequence of long eg. [1,2,3]. I create one index per day based on this template. How can I get the list of all the indices that contain a specific value in the audiences array field.
Something like list all the indices where audience array contains the value 3.
Thank you.

You can make a search query similar to
http://localhost:9200/sample-*/_search?q=audiences:3&pretty
and using the Java API, you can try getting the value of hits.hits._index
Or rather use filter_path :
http://localhost:9200/sample*/_search?q=audiences:3&filter_path=hits.hits._index&pretty
Result will look like this

Related

Elasticsearch basic mapping fails

I've installed the Docker containers for Elasticsearch 5.5.2 and Kibana. I started to learn about mapping types, and created an index with the following code through xcurl:
{
"mappings": {
"user": {
"_all": { "enabled": false },
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" }
}
}
}
The index was created successfully and I decided to insert some data. When I try to add a string into an integer field i.e. {"age": "hello"}, Elastic shows an error (this means mappings is working OK). The problem is with other data types:
1.It accepts integers and floats in string fields (I think this could be because of implicit casts).
2.It accepts floats like 22.4 in the agefield (when I search with Kibana or xcurl the agefield content is shown as float and not as an integer, that means is not doing casts from float to integer)
What I'm doing bad?
Have you tried to disable coercion? It can be done at field level:
{
"mappings": {
"user": {
"_all": { "enabled": false },
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer",
"coerce": false}
}
}
}
Or at index level for all fields:
"settings": {
"index.mapping.coerce": false
},
"mappings": {
...

Only allow fields that are in the index template

I have logstash pushing docs into an elasticsearch cluster.
And I apply a template to the indices with logstash:
elasticsearch {
hosts => 1.1.1.1.,2.2.2.2.
index => "logstash-myindex-%{+YYYY-MM-dd}"
template_name => "mytemplate"
template => "/etc/logstash/index_templates/mytemplate.json"
template_overwrite => true
}
Is there a way I can have only the fields defined in the template get added to the docs? Because sometimes the docs have a bunch of other fields I don't care about and I don't want to manually filter out each one. I want to be able to say if field not in index template do not add.
edit:
I did this in my index template but fields not specified in the template are still getting added to docs:
{
"template": "logstash-myindex*",
"order": 10,
"mappings": {
"_default_": {
"dynamic": "scrict",
"_all": {
"enabled": false
},
"properties": {
"#timestamp": {
"type": "date",
"include_in_all": false
},
"#version": {
"type": "keyword",
"include_in_all": false
},
"bytesReceived": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
},
.... etc
I'm not familiar with logstash - but I'm assuming this is just like creating an index in ElasticSearch.
In ElasticSearch you can disabled the dynamic creation of fields by adding:
"dynamic": false
to the mapping.
This would look something like this:
{
"mappings": {
"_default_": {
"dynamic": false
}
}
}

Kibana doesn't show results on tile map

I have approximately 3300 documents with geo_point typed field filled.
When I try to visualize my documents on the tile map, kibana says "no results found".
I've already tried putting coordinates as:
- geohash in string
- [lon, lat] array
- object with "lat" and "lon" properties
- string "lat,lon"
All these ways of setting geo_point are allowed according to ES docs.
Kibana detects this field as geo_point (there is a globe icon near field name), but nothing shows up on tile map.
What's wrong with me?
I'm using Kibana 4.2, elasticsearch 2.0.0
I've managed it.
It was happening because I had my geo_point typed field inside of the field with "type": "nested" parameter.
I've changed this outer field to "dynamic": "true" and now I can visualize my locations!
I was able to have a nested geo_point by removing the "type": "nested" from the mapping. No "dynamic":"true" needed. My mapping looks like this:
"mappings": {
"_default_": {
"_all": {
"enabled": true
},
"_ttl": {
"enabled": true,
"default": "12m"
},
"dynamic_templates": [{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "analyzed",
"omit_norms": true,
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
}
}
}
}],
"properties": {
"#version": {
"type": "string",
"index": "not_analyzed"
},
"user_data": {
"properties": {
"user_geolocation": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
}
}
}

ElasticSearch - Reindex to add doc_value

What am I trying to do?
Add doc_type to an existing index.
What have I tried?
Created index and document
POST /my_index-1/my_type/1
{
"my_prop": "my_value"
}
Added a template
PUT /_template/my_template
{
"id": "my_template",
"template": "my_index-*",
"mappings": {
"_default_": {
"dynamic_templates": [
{
"my_prop_template": {
"mapping": {
"index": "not_analyzed",
"doc_values": true,
"fielddata": {
"format": "doc_values"
},
"type": "string"
},
"match": "my_prop",
"match_mapping_type": "string"
}
}
]
}
}
}
Reindexed
./stream2es es --source http://localhost:9200/my_index-1 --target http://localhost:9200/my_index-2
What went wrong?
In the new index my_index-2 the property did not receive "doc_values": true:
...
"properties": {
"my_prop": {
"type": "string"
}
}
...
Just for the sanity, I have also tried adding the same document to my_index-3, and it got "doc_values": true.
My question
How can I reindex my old index with "doc_values": true?
Thanks #Val! Logstash indeed solved the problem.
Both stream2es and elasticsearch-reindex created new mapping without "doc_values": true.

How to set IndexOption = docs

I need to get result below with NEST (Elastic Search .NET client)
"detailVal": {
"name": "detailVal",
"type": "multi_field",
"fields": {
"detailVal": {
"type": "string"
},
"untouched": { // <== FOCUS 2
"type": "string",
"index": "not_analyzed",
"omit_norms": true,
"include_in_all": false,
"index_options": "docs" // <== FOCUS 1
}
}
}
I have done so far
[ElasticProperty(OmitNorms = true, Index = FieldIndexOption.not_analyzed, IncludeInAll = false, AddSortField = true)]
public string DetailVal { get; set; }
which gets me
"detailVal": {
"name": "detailVal",
"type": "multi_field",
"fields": {
"detailVal": {
"type": "string",
"index": "not_analyzed",
"omit_norms": true,
"include_in_all": false
},
"sort": { // <== FOCUS 2
"type": "string",
"index": "not_analyzed"
}
}
}
so, any idea how to
add "index_options": "docs" (I found IndexOptions.docs but it is not valid as Attribute)
change sort to untouched
The attribute based mapping only gets you so far. It's good enough if you only need to change names and set simple properties.
The recommended approach is to use client.MapFluent()
See https://github.com/Mpdreamz/NEST/blob/master/src/Nest.Tests.Unit/Core/Map/FluentMappingFullExampleTests.cs#L129
For an example how to set index_options
And line 208:
https://github.com/Mpdreamz/NEST/blob/master/src/Nest.Tests.Unit/Core/Map/FluentMappingFullExampleTests.cs#L208
To see how you can create your own multi_field mapping.
You can even combine both approaches:
client.MapFluent<MyType>(m=>m
.MapFromAttributes()
//Map what you can't with attributes here
);
client.Map() and client.MapFromAttributes() will most likely be removed at some point.

Resources