I have 2 loggers from 2 different clusters logging into my elasticsearch. logger1 uses indices mydata-cluster1-YYYY.MM.DD and logger2 uses indices mydata-cluster2-YYYY.MM.DD.
I have no way of touching the loggers. So i would like to add a field on the ES side when the data is indexed to show which cluster the data belongs to. Can i use mappings to do this?
Thanks
What if you use the PUT mapping API, in order to add a field to your index:
PUT mydata-cluster1-YYYY.MM.DD/_mapping/mappingtype <-- change the mapping type according to yours
{
"properties": {
"your_field": {
"type": "text" <--- type of the field
}
}
}
This SO could come in handy. Hope it helps!
Related
I was trying to find how flattened type in Elasticsearch works under the hood, the documentation specifies that all leaf values will be indexed into a single field as a keyword, as a result, there will be a dedicated index for all those flattened keywords.
From documentation:
By default, Elasticsearch indexes all data in every field and each indexed field has a dedicated, optimized data structure. For example, text fields are stored in inverted indices, and numeric and geo fields are stored in BKD trees.
The specific case that I am trying to understand:
If I have flattened field and index object with nested objects there is the ability to query a specific nested key in the flattened object. See how to query by labels.release:
PUT bug_reports
{
"mappings": {
"properties": {
"labels": {
"type": "flattened"
}
}
}
}
POST bug_reports/_doc/1
{
"labels": {
"priority": "urgent",
"release": ["v1.2.5", "v1.3.0"]
}
}
POST bug_reports/_search
{
"query": {
"term": {"labels.release": "v1.3.0"}
}
}
Would flattened field have the same index structure as the keyword field, and how it is able to reference the specific child key of flattened object?
The initial design and implementation of the flattened field type is described in this issue. The leaf keys are also indexed along with the leaf values, which is how they are allowing the search for a specific sub-field.
There are some ongoing improvements to the flattened field type and Elastic would also like to support numeric values, but that's not yet released.
I am using https://github.com/babenkoivan/scout-elasticsearch-driver to implement Elasticsearch with Laravel Scout. Ivan mentions this on Github:
Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type. Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. Mapping types will be completely removed in Elasticsearch 7.0.0.
If I understood right here: https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html I either need to use:
PUT index?include_type_name=true
or, better:
2)
PUT index/_doc/1
{
"foo": "baz"
}
I am stuck since I have no idea how to use either 1) or 2)
How can I add the parameter include_type_name=true?
How can I create the right mapping without using the include_type_name parameter?
class TestIndexConfigurator extends IndexConfigurator
{
use Migratable;
/**
* #var array
*/
protected $settings = [
];
protected $name = 'test';
}
Earlier versions of Elasticsearch (<= 5) supported multiple types per index. That meant that you could have different data mappings for each type. With Elasticsearch 6, this was removed and you can only have single mapping type.
Therefore, for Elasticsearch 7 (latest release), you can add an index, setup mappings and add document like this:
Create an index
PUT user
Add mapping
PUT user/_mapping
{
"properties": {
"name": {
"type": "keyword"
},
"loginCount": {
"type": "long"
}
}
}
Add document(s)
PUT user/_doc/1
{
"name": "John",
"loginCount": 4
}
Check data in the index
GET user/_search
Now, regarding the scout-elasticsearch-driver that you use, after reading the documentation you mentioned, it is simply saying that you need to create separate index configurator for each searchable model, as multiple models cannot be stored inside the same index.
So to create the index, run
php artisan make:index-configurator MyIndexConfigurator
and then
php artisan elastic:create-index App\\MyIndexConfigurator
which will create the index in the elasticsearch for you.
To learn more about elasticsearch, I suggest you install both elasticsearch and kibana to your development machine and then play around with it in kibana - the interface is quite nice and supports autocomplete to ease the learning curve.
When I tried GET product/default/_mapping in Kibana console.
I kept getting this error.
"Types cannot be provided in get mapping requests, unless
include_type_name is set to true"
This is happening in elastic search 7.3.0.
Looks like the above command is no longer supported in latest versions of elastic search.
It worked for me when I remove the default from the above command.
GET product/_mapping
I getting same error like "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true"
you have to add "include_type_name:true" inside the object
fix this problem above code
let type = true
return await esClient.indices.putMapping({
index:indexName,
type:mappingType,
body:mapping,
include_type_name:type
});
PUT busloggw4/_doc/_mapping?include_type_name=true
{
"properties": {
"log_flag": {
"type":"long"
}
}
}
I have two TYPES in my elasticsearch index. Both have same mapping. I am using one for active documents, while the other for archived ones.
Now, i want to archive a document i.e. change its _type from active to archived. Both are in same index, so i cannot reindex them as well.
Is there a way to do this in Elasticsearch 5.0 ?
Changing the type is tricky. You would have to remove and then index the document with the new type.
Why not have a field in your document indicating "activeness". Then you can use a bool query to filter by what you want:
{"query": {
"bool": {
"filter": [{"term": {"status", "active"}}],
"query": { /* your query object here */ }
}
}
}
Agree with having a field which indicates the activeness of the document.
(Or)
Use two different indices for "active" and "inactive" types.
Use aliases which map to these indices.
Aliases will give you flexibility to change your indices without downtimes.
I have some extra inner fields on a geo-shape type field. For example, "shape" is a geo-shape type field which has the regular required fields like "coordinates", "radius" etc., but it may also have other fields like "metadata" which I want elasticsearch to not parse and not store in the index. For example:
"shape": {
"coordinates":[6.77,8.99]
"radius": 500
"metadata": "some value"
}
Mapping schema looks like this:
"shape":{
"type":"geo_shape"
}
How can I achieve this ? By using "dynamic": false on mapping schema does not seem to be working.
Setting dynamic to false in your root mapping, like you did, is the way to go : are your sure it desn't work? Or are you saying that because it appears in your result hit _source?
Actually, by default, the _source attribute will contains the exact same document that you submitted.
However, it doesn't mean the extra metadata field has been indexed and/or stored.
If you want to check this, request specifically that field in your search like this :
POST _search
{
"fields": ["shape.metadata"]
}
You should have your search hits but without any fields value.
If it still bother you, disabled the _source attribute in your mapping.
I use multi-fields in a lot of my mappings. In the doc of Elastic Search there is an indication that multi-fields should be replaced with the "fields" parameter. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_multi_fields.html#_multi_fields
This works fine. However, to access a multi-field as a single field the documentation recommends to specify the copy_to parameter instead of the path parameter (see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#_accessing_fields)
Can somebody provide an example of such a mapping definition (thus using the "fields" parameter combined with "copy_to").
I have the impression that if you use the fields parameter you still need to specify the path parameter. And if you use copy_to, you no longer need to use a multi-fields approach; the fields just become separate fields and data of one field is copied to another at index time.
Hope somebody can help.
thx
Marc
I think that the copy_to option can be viewed as a cleaner variant of the Multi-fields feature (that is, the fields option). Both of these are easy to use when you want to "copy" values of a field to one or more other fields (to apply different mapping rules). However, if you need to "copy" values from multiple fields to the same field (that is, when you want a custom _all field), you must add the path option to the mapping, if you're using Multi-fields. On the other hand, with the copy_to option, you can simply point multiple source fields to the same destination field.
See this: https://www.elastic.co/guide/en/elasticsearch/reference/1.6/_multi_fields.html
copy_to would allow you to merge different fields like first_name and last_name into full_name
while multi field is used when you want to define several ways to index your field. For example
// Document mapping
{
"properties": {
"name": {
"fields": {
"name_metaphone": {
"type": "string",
"analyzer": "mf_analyzer"
},
"name_exact": {
"index": "not_analyzed",
"type": "string"
}
},
"type": "multi_field"
}
}
}