I am in the process of upgrading Elasticsearch. I upgraded elasticsearch from 6.8 to 7.17 and I upgraded the javascript client to #elastic/elasticsearch 7.17.0. I then deleted my old indices, put the mappings in place and tried to reindex the data coming from another database.
Now I am struggling with the current state of types in elasticsearch 7.17. I know that an index can only have one type of document and it looks like the type parameter of the javascript client is deprecated, but it still seems to be required. When I make a call to client.index() it complains about a missing type parameter:
ConfigurationError: Missing required parameter: type
And the error stack points to this block of code:
await client.index({
index: indexName,
id: obj.id,
body: obj.body,
});
My mappings look like this:
{
"author_index" : {
"mappings" : {
"dynamic" : "false",
"properties" : {
"articleCount" : {
"type" : "integer"
}
// ,,,
}
}
}
}
Should I still be specifying the type? Why does the client require it when its deprecated? What am I missing?
Replace your type with _doc and it should work as type is deprecated but you still need to give the _doc placeholder in the API calls.
Related
I am attempting to create a new Elasticsearch document, in an index called uploads, with an automatically generated document ID. I am doing that by POST-ing the new document to /uploads/_doc/. Elasticsearch however is rejecting the upload, because it returns an HTTP response with a status of 400 Bad Request.
Elasticsearch thinks I am attempting to create a new mapping type, because the error message in the body of the response is
Rejecting mapping update to [uploads] as the final mapping would have more than 1 type: [_doc, uploads]
I am not trying to create a new mapping type. I know that each index may now hold only one mapping type.
I am using Elasticsearch 6.4.2 on Debian. I am using libcurl to send the HTTP request and receive the HTTP response.
Why is Elasticsearch interpreting my request as an attempt to add a mapping type? How do I change what I am doing to instead create a new document.
Using curl shows that the index currently has one mapping type, called uploads:
curl http://localhost:9200/uploads?pretty
{
"uploads" : {
"aliases" : { },
"mappings" : {
"uploads" : {
"properties" : {
...
}
}
},
"settings" : {
"index" : {
...
"provided_name" : "uploads"
}
}
}
}
You are POST-ing to the wrong URL. The URL must give the index name and the document type, rather than using _doc.
According your mapping your uploads index has an uploads document type:
{
"uploads" : { <--- index name
"aliases" : { },
"mappings" : {
"uploads" : { <-----doc_type name
"properties" : {
...
}
}
}
So you should use this document type and not _doc. That is, POST to /uploads/uploads, not to /uploads/_doc.
I have index mapping for type 'T1' as below:
"T1" : {
"properties" : {
"prop1" : {
"type" : "text"
}
}
}
And now I want to change the type of prop1 from text to keyword. I don't want to delete index. I have also read people suggesting to create another property with new type and replace it. But then I have to update old documents which I am not interested into. I tried to use PUT api as below but I never works.
PUT /indexName/T1/_mapping -d
{
"T1" : {
"properties" : {
"prop1" : {
"type" : "keyword"
}
}
}
}
Is there any way to achieve this?
Mapping cannot be modified, hence the PUT api you have used will not work. The new index will have to be created with the updated mapping to be used and reindexing all the data to new index.
To prevent downtime you can always use alias:
https://www.elastic.co/blog/changing-mapping-with-zero-downtime
A mapping cannot be updated once it is persisted. The only option is to create a new index with the correct mappings and reindex your data using the reindex API provided by ES.
You can read about the reindex API here:
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html
I am going to define mapping with nested fields. according to this documentation, payload to /order-statistics/_mapping/order looks like:
{
"mappings" : {
"order": {
"properties" : {
"order_no" : {
"type" : "string"
},
"order_products" : {
"type" : "nested",
"properties" : {
"order_product_no" : {
"type" : "int"
},
"order_product_options" : {
"type" : "nested",
"properties" : {
"order_product_option_no" : {
"type" : "int"
}
}
}
}
}
}
}
}
}
I've already created the order-statistics index with a call to curl -XPUT 'localhost:9200/order-statistics' and I'm using predefined types such as int, string, double, But I get the following error and can't find what wrong with.
{
"error":{
"root_cause":[
{
"type":"mapper_parsing_exception",
"reason":"Root mapping definition has unsupported parameters: [mappings : {order={properties={order_no={type=string}, order_products={type=nested, properties={order_product_no={type=int}, order_product_options={type=nested, properties={order_product_option_no={type=int}}}}}}}}]"
}
],
"type":"mapper_parsing_exception",
"reason":"Root mapping definition has unsupported parameters: [mappings : {order={properties={order_no={type=string}, order_products={type=nested, properties={order_product_no={type=int}, order_product_options={type=nested, properties={order_product_option_no={type=int}}}}}}}}]"
},
"status":400
}
could someone explain why this not work?
You are using int as type for some fields which is not a valid type in either 2.x or 5.x. For integer values, please use integer or long depending on the values you want to store. For details, please see the docs on core mapping types.
Which version of elasticsearch are you using - 2.x or 5.x? If you are on 5.x already, you should go with keyword or text for your string fields instead of using just string which was the naming up to 2.x. But this is still only a warning.
Additionally, you should be aware of the implications when using nested instead of just object. Using a nested type is necessary if you store an array of objects and want to query for more than one property of such an object with the guarantee that only these documents match where one of the nested objects in the array matches all your conditions. But this comes at a cost, so consider using the simple object type, if this works for you. For more details, please see the docs on nested data type and especially the warning at the end.
I am using grafana 3.1 on maxOs Sierra and I use elasticsearch as datasource.
I have set my mapping on ES:
"properties" : {
"gasLeak" : {
"type" : "boolean"
},
"gasConsumption" : {
"type" : "boolean"
},
"electricityConsumption" : {
"type" : "boolean"
}
}
I have also a timestamp filed name alert.timestamp which is typed as a date on ES.
Everytime an alert is triggered, I add an entry into the ES index. There are 3 alerts type as shown in the mapping.
Now, on grafana, I have my dashboard with a graph panel. On this graph I want to display the annotation which represents alerts.
I have configured my alert as follow:
Grafana annotation config
But then on my graph, the annotation is displayed aheah of my measures: it is always on the right at the current datetime:
Issue on graph
Am I configuring the annoation wrongly? Do I need other configuration on ES side?
Thank you.
I found why it was not working: in the "field mapping" part, the filed "Time" has to be "timestamp"...
I used to put alert.timestamp and when I updated my ES document structure using timestamp instead of alert.timestamp as field name, after updating the field in grafana, it worked.
I also added the field "desc" and "tags" in my ES document so maybe it is also related, I did not dig that.
Hope this could help someone someday.
We're currently using a Couchbase Plugin (transport-couchbase) to transport and index the data into ElasticSearch (http://docs.couchbase.com/couchbase-elastic-search/)
I've taken a look at ElasticSearch's mapping documentation here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping.html
My understanding is that if you rely on defaults for ElasticSearch, once a document gets indexed, ElasticSearch will create a dynamic mapping for that document type. This is what we've defaulted to.
We ran into issues where after adding a specific document type, and when the transport plugin inserts an "invalid" document (the document's field type is now different -- from string -> array), ElasticSearch throws an exception and essentially breaks the replication from Couchbase to ElasticSearch. The exception looks like this:
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property
[xyz]
java.lang.RuntimeException: indexing error MapperParsingException[failed to parse
[doc.myfield]]; nested: ElasticsearchIllegalArgumentException[unknown property[xyz]]
Is there a way we can configure ElasticSearch so that "invalid" documents simply get filtered without throwing exception and breaking the replication?
Thanks.
{
"tweet" : {
"dynamic": "strict",
"properties" : {
"message" : {"type" : "string", "store" : true }
}
}
}
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-dynamic-mapping.html