Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true - laravel

I am using https://github.com/babenkoivan/scout-elasticsearch-driver to implement Elasticsearch with Laravel Scout. Ivan mentions this on Github:
Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type. Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. Mapping types will be completely removed in Elasticsearch 7.0.0.
If I understood right here: https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html I either need to use:
PUT index?include_type_name=true
or, better:
2)
PUT index/_doc/1
{
"foo": "baz"
}
I am stuck since I have no idea how to use either 1) or 2)
How can I add the parameter include_type_name=true?
How can I create the right mapping without using the include_type_name parameter?
class TestIndexConfigurator extends IndexConfigurator
{
use Migratable;
/**
* #var array
*/
protected $settings = [
];
protected $name = 'test';
}

Earlier versions of Elasticsearch (<= 5) supported multiple types per index. That meant that you could have different data mappings for each type. With Elasticsearch 6, this was removed and you can only have single mapping type.
Therefore, for Elasticsearch 7 (latest release), you can add an index, setup mappings and add document like this:
Create an index
PUT user
Add mapping
PUT user/_mapping
{
"properties": {
"name": {
"type": "keyword"
},
"loginCount": {
"type": "long"
}
}
}
Add document(s)
PUT user/_doc/1
{
"name": "John",
"loginCount": 4
}
Check data in the index
GET user/_search
Now, regarding the scout-elasticsearch-driver that you use, after reading the documentation you mentioned, it is simply saying that you need to create separate index configurator for each searchable model, as multiple models cannot be stored inside the same index.
So to create the index, run
php artisan make:index-configurator MyIndexConfigurator
and then
php artisan elastic:create-index App\\MyIndexConfigurator
which will create the index in the elasticsearch for you.
To learn more about elasticsearch, I suggest you install both elasticsearch and kibana to your development machine and then play around with it in kibana - the interface is quite nice and supports autocomplete to ease the learning curve.

When I tried GET product/default/_mapping in Kibana console.
I kept getting this error.
"Types cannot be provided in get mapping requests, unless
include_type_name is set to true"
This is happening in elastic search 7.3.0.
Looks like the above command is no longer supported in latest versions of elastic search.
It worked for me when I remove the default from the above command.
GET product/_mapping

I getting same error like "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true"
you have to add "include_type_name:true" inside the object
fix this problem above code
let type = true
return await esClient.indices.putMapping({
index:indexName,
type:mappingType,
body:mapping,
include_type_name:type
});

PUT busloggw4/_doc/_mapping?include_type_name=true
{
"properties": {
"log_flag": {
"type":"long"
}
}
}

Related

How can I let ES support mixed type of a field?

I am saving logs to Elasticsearch for analysis but I found there are mixed types of a particular field which causing error when indexing the document.
For example, I may save below log to the index where uuid is an object.
POST /index-000001/_doc
{
"uuid": {"S": "001"}
}
but from another event, the log would be:
POST /index-000001/_doc
{
"uuid": "001"
}
the second POST will fail because the type of uuid is not an object. so I get this error: object mapping for [uuid] tried to parse field [uuid] as object, but found a concrete value
I wonder what the best solution for that? I can't change the log because they are from different application. The first log is from the data of dynamodb while the second one is the data from application. How can I save both types of logs into ES?
If I disable dynamic mapping, I will have to specify all fields in the index mapping. For any new fields, I am not able to search them. so I do need dynamic mapping.
There will be many cases like that. so I am looking for a solution which can cover all conflict fields.
It's perfectly possible using ingest pipelines which are run before the indexing process.
The following would be a solution for your particular use case, albeit somewhat onerous:
create a pipeline
PUT _ingest/pipeline/uuid_normalize
{
"description" : "Makes sure uuid is a hash map",
"processors" : [
{
"script": {
"source": """
if (ctx.uuid != null && !(ctx.uuid instanceof java.util.HashMap)) {
ctx.uuid = ['S': ctx.uuid]; // hash map init
}
"""
}
}
]
}
run the pipeline when ingesting a new doc
POST /index-000001/_doc
{
"uuid": {"S": "001"}
}
POST /index-000001/_doc?pipeline=uuid_normalize <------
{
"uuid": "001"
}
You could now extend this to be as generic as you like but it is assumed that you know what you expect as input in each and every doc. In other words, unlike dynamic templates, you need to know what you want to safeguard against.
You can read more about painless script operators here.
You just cannot.
You should either normalize all your field in a way or another.
Or use 2 separate field.
I can suggest to use a field like this :
"uuid": {"key": "S", "value": "001"}
and skip the key when not necessary.
But you will have to preprocess your value before ingestion.

Can I add a field automatically to an elastic search index when the data is being indexed?

I have 2 loggers from 2 different clusters logging into my elasticsearch. logger1 uses indices mydata-cluster1-YYYY.MM.DD and logger2 uses indices mydata-cluster2-YYYY.MM.DD.
I have no way of touching the loggers. So i would like to add a field on the ES side when the data is indexed to show which cluster the data belongs to. Can i use mappings to do this?
Thanks
What if you use the PUT mapping API, in order to add a field to your index:
PUT mydata-cluster1-YYYY.MM.DD/_mapping/mappingtype <-- change the mapping type according to yours
{
"properties": {
"your_field": {
"type": "text" <--- type of the field
}
}
}
This SO could come in handy. Hope it helps!

Add typed additional attributes to an existing document elasticsearch

I added a field to the document:
POST /erection/shop/1/_update
{
"doc": {
"my_field":""
}
}
The new field is assigned to the type of "String". how can I create a new field with the type "Boolean"/"Integer"?
and 2nd question:
is it possible to add one field in all documents using one query? (without updating each document)
1) Explicitly define a mapping prior to the first update you do.
2) No, you can't. You can do it in your application using "scan" and then "bulk update"

Partial update to an index template

In ElasticSearch, is it possible to make a partial update to an index template?
For example, I tried changing only the default analyzer with the following (using Sense):
PUT /_template/testtemplate/_update
{
"settings": {
"index":{
"analysis": {
"analyzer": {
"default": {
"type": "whitespace"
}
}
}
}
}
}
Bu that didn't work, I get an error that says Invalid index name [_template], must not start with '_'.
Do I have to pass the full template again using a PUT or there's some other way to do a partial update?
Partial updates only work for documents (and to some extent to mapping types), not for index templates. So, I confirm you'll need to store the full index template again and not only the small bit of it that you want to update.
As you can see in the source code for RestUpdateAction.java, the _update REST endpoint expects an {index} name, a {type} name and an {id}. So in your example above, that endpoint thinks that _template is an index name and complains.
Similarly, in the REST endpoint for creating index templates, RestPutIndexTemplateAction.java, you can see that the _template path doesn't support the _update endpoint at all.

Exclude setting on integer field in term query

My documents contain an integer array field, storing the id of tags describing them. Given a specific tag id, I want to extract a list of top tags that occur most frequently together with the provided one.
I can solve this problem associating a term aggregation over the tag id field to a term filter over the same field, but the list I get back obviously always starts with the album id I provide: all documents matching my filter have that tag, and it is thus the first in the list.
I though of using the exclude field to avoid creating the problematic bucket, but as I'm dealing with an integer field, that seems not to be possible: this query
{
"size": 0,
"query": {
"term": {
"tag_ids": "00001"
}
},
"aggs": {
"tags": {
"terms": {
"size": 3,
"field": "tag_ids",
"exclude": "00001"
}
}
}
}
returns an error saying that Aggregation [tags] cannot support the include/exclude settings as it can only be applied to string values.
Is it possible to avoid getting back this bucket?
This is, as of Elasticsearch 1.4, a shortcoming of ES itself.
After the community proposed this change, the functionality has been added and will be included in Elasticsearch 1.5.0.
It's supposed to be fixed since version 1.5.0.
Look at this: https://github.com/elasticsearch/elasticsearch/pull/7727
While it is enroute to being fixed: My workaround is to have the aggregation use a script instead of direct access to the field, and let that script use the value as string.
Works well and without measurable performance loss.

Resources