Elasticsearch how to exclude some fields in indexing - elasticsearch

I have a JSON data that maps automatically by the elastic search when I'm indexing the data. How can I exclude some fields in the mapping. I already tried to define the map manually but when I'm doing bulk index, It automatically maps the other fields.
ex. my JSON data looks like this
[
{
"id": "232",
"name": "Lorem",
"description": "Ipsum Dolor",
"image": [
{"key": "asadasd.jpg"},
{"key": "asasd2d.jpg"}
],
"is_active": true
},
...
My map when I'm defining it manually
PUT myindex
{
"mappings": {
"product": {
"properties": {
"id": { "type": "text },
"name": { "type": "text"},
"description": { "type": "text" },
"is_active": { "type": "boolean" }
}
}
}
}
What I want to achieve is the data still remain I just want to exclude the image property to be not included in the indexing.
So that When I query in the elastic search is still I get the data with image
Is that possible?
Thank you guys. I'm new in elasticsearch
{
"id": "232",
"name": "Lorem",
"description": "Ipsum Dolor",
"image": [
{"key": "asadasd.jpg"},
{"key": "asasd2d.jpg"}
],
"is_active": true
}

Yes, that's possible simply by adding dynamic: false to your mapping, like this:
PUT myindex
{
"mappings": {
"product": {
"dynamic": false, <-- add this line
"properties": {
"id": {
"type": "text"
},
"name": {
"type": "text"
},
"description": {
"type": "text"
},
"is_active": {
"type": "boolean"
}
}
}
}
}
The image array will still be in the source, but the mapping won't be modified.

The problem with the accepted answer is, that you need to explicitly add mappings for all fields, which is not always wanted (e.g. for array types).
You could disable the field like this:
PUT myindex
{
"mappings": {
"product": {
"properties": {
"id": { "type": "text },
"name": { "type": "text"},
"description": { "type": "text" },
"is_active": { "type": "boolean" },
"image": { "type": "object, "enabled": false }
}
}
}
}
The image array is still going to be in the _source.
Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

Related

AWS Open Search/Elastic search wild card search on full index

For example I have below 1 json data sample where multiple fields having the value '1001'. Like this I have many Json document. I want to search particular keyword like '1001' across any field (can be nested json field as well). I have gone through multiple document where they are suggesting to put the particular field name to search. Is there a way to achieve this without knowing which field has the search text?
URL: https://linuxhint.com/wildcard-query-elasticsearch/
{
"id": "1001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "1001" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "1001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
}

How can i do nested field queries in Elastic search using Lucene query syntax

Here is the simple usecase,
I have a system that sends the Lucene query to my elastic search. I have a mapping
{
"mappings": {
"properties": {
"grocery_name":{
"type": "text"
},
"items": {
"type": "nested",
"properties": {
"name": {
"type": "text"
},
"stock": {
"type": "integer"
},
"category": {
"type": "text"
}
}
}
}
}
}
and the data looks like
{
"grocery_name": "Elastic Eats",
"items": [
{
"name": "banana",
"stock": "12",
"category": "fruit"
},
{
"name": "peach",
"stock": "10",
"category": "fruit"
},
{
"name": "carrot",
"stock": "9",
"category": "vegetable"
},
{
"name": "broccoli",
"stock": "5",
"category": "vegetable"
}
]
}
How can I query to get all items where the item name is banana and stock > 10, In KQL i can write something like items:{ name:banana and stock > 10 }
The Lucene expression language doesn't support querying nested documents. That's why the KQL language fills that gap.
That's currently the only way to query nested documents via the Kibana search bar.

Index Mapping for nested json for schema registry

I have mapped and successfully indexed json files of different formats. But this particular json file which is created from schema registry is weird and i am unable to crack the mapping for it. I have also referenced a lot of articles, but of no help.
Please find below mapping and json file which i am trying to index. It does get indexed, i am not getting the output as i want it to look like.
Mapping:
"mappings":
{
"_doc":
{
"dynamic" : "true",
"properties":
{
"schema":
{
"type": "nested",
"properties":
{
"type":
{
"type": "text"
},
"name":
{
"type": "text"
},
"namespace":
{
"type": "text"
},
"fields":
{
"type": "nested",
"properties":
{
"name":
{
"type": "text"
},
"type":
{
"type": "text"
}
}
}
}
},
"deleted":
{
"type": "boolean"
},
"subject":
{
"type": "text"
},
"id":
{
"type": "integer"
},
"version":
{
"type": "integer"
}
}
}
}
the json file looks like.
{
"schema": "{\"type\":\"record\",\"name\":\"F3253\",\"namespace\":\"ADMIN\",\"fields\":[{\"name\":\"table\",\"type\":[\"null\",\"string\"]},{\"name\":\"op_type\",\"type\":[\"null\",\"string\"]},{\"name\":\"op_ts\",\"type\":[\"null\",\"string\"]},{\"name\":\"current_ts\",\"type\":[\"null\",\"string\"]},{\"name\":\"pos\",\"type\":[\"null\",\"string\"]},{\"name\":\"primary_keys\",\"type\":[\"null\",{\"type\":\"array\",\"items\":\"string\"}]},{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"row\",\"namespace\":\"\",\"fields\":[{\"name\":\"C1\",\"type\":[\"null\",\"string\"]},{\"name\":\"C2\",\"type\":[\"null\",\"string\"]},{\"name\":\"C3\",\"type\":[\"null\",\"long\"]},{\"name\":\"C4\",\"type\":[\"null\",\"string\"]},{\"name\":\"C5\",\"type\":[\"null\",\"string\"]},{\"name\":\"C6\",\"type\":[\"null\",\"long\"]},{\"name\":\"C7\",\"type\":[\"null\",\"long\"]},{\"name\":\"C8\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940003025\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940003026\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940003027\",\"type\":[\"null\",\"string\"]},{\"name\":\"C800502536\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005001\",\"type\":[\"null\",\"string\"]},{\"name\":\"C536870914\",\"type\":[\"null\",\"string\"]},{\"name\":\"C536870918\",\"type\":[\"null\",\"long\"]},{\"name\":\"C536870919\",\"type\":[\"null\",\"long\"]},{\"name\":\"C536870920\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005060\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005061\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005063\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005064\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005082\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005083\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005085\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005089\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005092\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005093\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005094\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005095\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005096\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005097\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005099\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005100\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005101\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005102\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005104\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005105\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005106\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005107\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005108\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005109\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005110\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005111\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005112\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005113\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005114\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005115\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005122\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005123\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005124\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005125\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005126\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005127\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005128\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005129\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005130\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005141\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005142\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940005143\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005144\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005145\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005509\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005510\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005511\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005512\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005513\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005514\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005515\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005518\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005519\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005520\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005521\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940005522\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940007513\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008202\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008203\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940006953\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008052\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008053\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008054\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008055\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008056\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008057\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008058\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008060\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008061\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940008062\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008063\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008064\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008065\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008066\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940008067\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008068\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008069\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008070\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008071\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008073\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940008075\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008076\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940008352\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009041\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009042\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009043\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009044\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009045\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009046\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009047\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009048\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009049\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009050\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009051\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009052\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940009053\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009054\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009055\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009056\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009057\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009058\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009059\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009062\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009277\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009278\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009279\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009280\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009281\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940009282\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940009295\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009296\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009297\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009760\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940009761\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009762\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009763\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940009764\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009765\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009766\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009767\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009772\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010169\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940007906\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940007907\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940007909\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009512\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009513\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009514\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009516\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009517\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009519\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009520\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009521\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009528\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009530\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009532\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940009533\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010354\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010355\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010356\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010357\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010358\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010359\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010360\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010361\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010362\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010363\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010364\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010365\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010366\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010367\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010370\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010375\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010376\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010377\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010378\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010380\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010382\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010390\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010391\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010392\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010394\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010395\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010396\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010404\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010878\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010879\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010880\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010881\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010882\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940010883\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010884\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940010885\",\"type\":[\"null\",\"string\"]},{\"name\":\"C800502452\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011556\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011557\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011558\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011559\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011560\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011561\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011562\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011563\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011564\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011565\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011566\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011571\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011572\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011573\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011574\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011575\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011576\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011577\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011578\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011579\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011580\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011581\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011582\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011583\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011584\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011585\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011586\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011587\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011621\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011623\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940011629\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011634\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011635\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011636\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011637\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011638\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011639\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011640\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011641\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011642\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011665\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011666\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011672\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011815\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011816\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011821\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011822\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011823\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011824\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011825\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011826\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011827\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940011828\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012160\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012161\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012162\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012163\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940012164\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940012165\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012169\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012170\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012171\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012172\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012648\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940012649\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940012650\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940012651\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013752\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013753\",\"type\":[\"null\",\"long\"]},{\"name\":\"F940013754\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013755\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013756\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013757\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940013758\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940014875\",\"type\":[\"null\",\"string\"]},{\"name\":\"F940014876\",\"type\":[\"null\",\"long\"]}],\"connect.name\":\"row\"}]},{\"name\":\"after\",\"type\":[\"null\",\"row\"]}],\"connect.name\":\"ADMIN.F3253\"}",
"deleted": false,
"subject": "app-goldenGate-ass-ADMIN_F3253-value",
"id": 157,
"version": 1
}
I want something where the index shows
metadata.schema.namespace : ....
metadata.schema.name : ....
Not like the below as i am currently getting -
metadata.schema {"type":"record","name":"Envelope","namespace":Bla

Elasticsearch mapping parser exception?

I'm trying to create a nested document in Elasticsearch.
Structure:
title,name,comments
comments is a nested document - inside that - Comment & Star_Rating.
Inside Comment, name and address.
Here is the query mentioned below.
PUT /sounduu
{
"mappings": {
"blogpost": {
"properties": {
"title": {
"type": "string"
},
"name": {
"type": "string"
},
"comments": {
"properties": {
"comment": {
"properties": {
"name": {
"type": "string"
},
"address": {
"type": "string"
}
}
},
"star_rating": {
"type": "long"
}
}
}
}
}
}
}
PUT /sounduu/blogpost/1
{
"title": "someh_title",
"name":"soundy",
"comments": {
"comment":"kuu",
[{
"name":"juwww",
"address":"eeeey"
},
{
"name":"jj",
"address":oo"
}]
},
"star_rating":6
}
Error :
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "object mapping for [comments.comment] tried to parse field [comment] as object, but found a concrete value"
}
],
"type": "mapper_parsing_exception",
"reason": "object mapping for [comments.comment] tried to parse field [comment] as object, but found a concrete value"
},
"status": 400
}
Can anyone help with this?
In your PUT /sounduu/blogpost/1 request you are attempting to treat the "comment" property as both a nested object and a string.
Formatting your request's JSON, you can observe the issue:
{
"title": "someh_title",
"name": "soundy",
"comments": {
"comment": "kuu",
[{
"name": "juwww",
"address": "eeeey"
},
{
"name": "jj",
"address": oo"
}]
},
"star_rating":6
}
You either need to update your mapping to include a "text" property, and move the "comment": "kuu" content accordingly, or omit it from your request to work with your current mapping.
Example here - For me it seems logical to group everything like so:
PUT /sounduu
{
"mappings": {
"blogpost": {
"properties": {
"title": {
"type": "string"
},
"name": {
"type": "string"
},
"comments": {
"properties": {
"text" : {
"type": "string"
},
"name": {
"type": "string"
},
"address": {
"type": "string"
}
}
},
"star_rating": {
"type": "long"
}
}
}
}
}
The indexing request would then look like:
{
"title": "someh_title",
"name": "soundy",
"comments": [
{
"text": "kuu",
"name": "juwww",
"address": "eeeey"
},
{
"text": "kuu2",
"name": "jj",
"address": oo"
}
],
"star_rating":6
}
If you are using elasticSearch higher version, then it is recommended to replace 'string' data type with 'text'. ElasticSearch community has discarded 'string'.
Reformed request should be :
`PUT /sounduu
{
"mappings": {
"blogpost": {
"properties": {
"title": {
"type": "text"
},
"name": {
"type": "text"
},
"comments": {
"properties": {
"text" : {
"type": "text"
},
"name": {
"type": "text"
},
"address": {
"type": "text"
}
}
},
"star_rating": {
"type": "long"
}
}
}
}
}`

ElasticSearch - how to not store fields that are not defined in the static index?

I have set a static index with user entity in ES using
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}
When I post a document with more fields than in the index it saves them to the ES.
It doesn't update the mapping but it saves the new fields.
Is there a way to remove the fields that are not in the index?
I dont want to store un-indexed fields.
In your mapping you need to use _source filtering:
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"_source": {
"includes": [
"id","name","age"
]
},
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}

Resources