Elasticsearch version conflict - elasticsearch

I have updated document in the elastic search. after update using I am fetching the same document by using their ID. It is giving me following response:
{
"_index": "b123456",
"_type": "documents",
"_id": "bltde56dd11ba998bab",
"_version": 3,
"found": true,
"_source": {
"title": "index.json",
"url": "/index1",
"tags": [],
"created_at": "2018-06-19T05:02:38.174Z",
"updated_at": "2018-06-19T05:07:57.155Z",
"version": 1,
"fields": [{
"uid": "fname",
"value": "john"
},
{
"uid": "lname",
"value": "test"
}
],
"class": "first"
}
}
After I am using update_by_query to update document I am sending following request to update_by_query:
{
"script": {
"source": "ctx._source.title = params.title;ctx._source.url = params.url;ctx._source.created_at = params.created_at;ctx._source.updated_at = params.updated_at;ctx._source.version = params.version;ctx._source.fields = params.fields",
"params": {
"title": "Demo title",
"url": "/demo",
"created_at": "2018-06-19T05:02:38.174Z",
"updated_at": "2018-06-19T05:07:57.155Z",
"version": 2,
"fields": [{
"uid": "fname",
"value": "vicky"
},
{
"uid": "lname",
"value": "test"
}
]
}
},
"query": {
"bool": {
"must": [{
"term": {
"_id": "bltde56dd11ba998bab"
}
},
{
"range": {
"version": {
"lt": 2
}
}
}
]
}
}
}
But it is giving me status code:409 and following error:
[documents][bltde56dd11ba998bab]: version conflict, current version
[3] is different than the one provided [2]
My document also contain custom version key.
Can anyone help me into this

For the sake of posterity, I'll submit an answer to this old question. The issue is occurring because ElasticSearch's internal version value in the _version field is actually 3 in your initial response, not 1.
You are then trying to update the document to using external version value 2, Elastic sees this as a conflict, as internally it thinks version 3 is the most up-to-date version, not version 1. Effectively, something as caused your external version scheme and Elastic's internal version scheme to become out-of-sync.
Also note, the following parameter should be included in your update calls to indicate that the operation should follow the rules for external versioning as opposed to Elastic's internal versioning scheme.
"version_type":external
There is a subtle but important distinction that needs to be made by specifying this parameter.
With version_type set to external, Elasticsearch will store the
version number as given and will not increment it. Also, instead of
checking for an exact match, Elasticsearch will only return a version
collision error if the version currently stored is greater or equal to
the one in the indexing command.
More information can be on Elastic's version can be found in their blog post

for me, it was document id. I am using node js elastic-search client, when I create a document I need to pass a document Id,
I was getting version conflict because I was trying to create multiple documents with the same id.
await elasticWrapper.client.create({
index: ElasticIndexs.Payments,
id: data.id, // <-- id should be unique
body: {
...data,
},
});
``

Related

Objects in array is not well supported error observed for ELK docker image

I'm using the latest elk image for kibana dashboard and I have json file which is having list of array[] and I'm not able to show those as field in kibana and It's showing that the object in array is not well supported error message.
As per the document in kibana I just went through the below link but I didn't find anything useful for elk docker image.
https://github.com/istresearch/kibana-object-format
I just tried to run the command
Run bin/kibana-plugin install <package.zip>
but it returned as run is unknown command removed run and ran remaining command but It says that's invalid.
I'm using linux box and Kibana 7.3 version.
Is it possible to overcome this issue? how to deploy that plugin for elk image else is there any other way to make those arrays object as fields in kibana.
I'm not sure how can I proceed. Please help me.
Sample Data:
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 4,
"issues": [{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "1999875",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/1999875",
"key": "KINDLEAMZ-67578",
"fields": {
"summary": "contingency is displaying for confirmed card.",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2019428",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2019428",
"key": "KINDLEAMZ-68661",
"fields": {
"summary": "card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "2010958",
"self": "https://amazon.kindle.com/jira/rest/api/2/issue/2010958",
"key": "KINDLEAMZ-68167",
"fields": {
"summary": "Test Card",
"priority": {
"name": "P1",
"id": "1"
},
"created": "2019-09-23T11:25:21.000+0000"
}
}
]
}
I just want to fetch KEY, Summary, Priority from each of the above array. But its not working as expected when I tried to make a field its showing as array in kibana. If this is not working with 7.3.0 should I downgrade to lower version? the steps are missing for docker user in that document. Is there any way to get those details?
Checking here: https://github.com/istresearch/kibana-object-format/releases it looks like the plugin latest release was for Elasticsearch 6.3. I guess that is the reason for your error.
I'm not sure there's a fix for this in kibana. There are many issues on this subject, open for a long time, like: https://github.com/elastic/kibana/issues/3333.

YQL console gives me result but rest query gives me error

I'm trying to use YQL Console to get currency rates, the YQL statement is
select * from yahoo.finance.xchange where pair in ("EURUSD","GBPUSD")
the console results give me
{
"query": {
"count": 2,
"created": "2017-10-26T02:42:44Z",
"lang": "en-US",
"results": {
"rate": [
{
"id": "EURUSD",
"Name": "EUR/USD",
"Rate": "1.1829",
"Date": "10/26/2017",
"Time": "3:42am",
"Ask": "1.1829",
"Bid": "1.1829"
},
{
"id": "GBPUSD",
"Name": "GBP/USD",
"Rate": "1.3269",
"Date": "10/26/2017",
"Time": "3:42am",
"Ask": "1.3269",
"Bid": "1.3269"
}
]
}
}
}
but the rest query gives me error
{"error":{"lang":"en-US","diagnostics":{"cache":{"execution-start-time":"0","execution-stop-time":"0","execution-time":"0","method":"GET","type":"MEMCACHED","content":"ENV.queryyahooapiscomproductionsg3.store://datatables.org/alltableswithkeys.15a841ff462a38eb6175e73b4dc747ef"},"env":"Failed to read from storage: store://datatables.org/alltableswithkeys: Invalid store url: store://datatables.org/alltableswithkeys","warning":"Invalid environment specified: store://datatables.org/alltableswithkeys"},"description":"No definition found for Table yahoo.finance.xchange"}}
The yahoo.finance.xchange is a community table. In the YQL console there should be a checkbox saying Show Community Tables select that and you should have access to it. The REST call here works. Let me know if you have any questions.

Children are not mapping properly in elastic to parents

"chods": {
"mappings": {
"chod": {
"properties": {
"state": {
"type": "text"
}
}
},
"chods": {},
"variant": {
"_parent": {
"type": "chod"
},
"_routing": {
"required": true
},
"properties": {
"percentage": {
"type": "double"
}
}
}
}
},
When I execute:
PUT /chods/variant/565?parent=36442
{ // some data }
It returns:
{
"_index":"chods",
"_type":"variant",
"_id":"565",
"_version":6,
"result":"updated",
"_shards":{
"total":2,
"successful":1,
"failed":0
},
"created":false
}
But when I run this query:
GET /chods/variant/565?parent=36442
It returns variant with parent=36443
{
"_index": "chods",
"_type": "variant",
"_id": "565",
"_version": 7,
"_routing": "36443",
"_parent": "36443",
"found": true,
"_source": {
...
}
}
Why it returns with parent 36443 and not 36442?
When I tried to reproduce this with your steps, I got the expected result (version=36442). I noticed that after your PUT of the document with "_parent": "36442" the output is "_version":6. In your GET of the document, "_version": 7 is returned. Is it possible that you posted another version of the document?
I also noticed that GET /chods/variant/565?parent=36443 would not actually filter by the parent id - the query parameter is disregarded. If you actually want to filter by parent id, this is the query you're looking for:
GET /chods/_search
{
"query": {
"parent_id": {
"type": "variant",
"id": "36442"
}
}
}
As #fylie pointed out the main problem is that if you use same id of the document you will get your document overridden by last version - sort of
Lets say that we have index /tests and type "a" which is child of type "test" and we do following commands:
PUT /tests/a/50?parent=25
{
"item": "C"
}
PUT /tests/a/50?parent=26
{
"item": "D"
}
PUT /tests/a/50?parent=50
{
"item": "E",
"item2": "F",
}
What the result will be? Well it can result in creating 1 - 3 documents.
If it will route to the same shard, you will end up with one document, which will have 3 versions.
If it will route to 3 different shards, you will end up with 3 new documents.

Accessing array values in Nifi

I have a json array which looks like
{
[
{
"name": "ABCDEF",
"details": {
"code": null,
"description": null,
"range": {
"fromNo": "00174",
"toNo": "99999"
}
}
}, {
"name": "ABCDEF",
"details": {
"code": null,
"description": null,
"range": {
"fromSerialNo": "00001",
"toSerialNo": "00060"
}
}, {
"name": "ABCDEF",
"details": {
"code": null,
"description": null,
"range": {
"fromSerialNo": "00061",
"toSerialNo": "00173"
}
}]
}
and I want the output as
{
"name": "ABCDEF",
"details": {
"code": null,
"description": null,
"range": {
"fromSerialNo": "00001",
"toSerialNo": "99999"
}
But the problem is I cannot put the variable in EvaluateJsonPath as it does not accept nifi expression language. I have to loop through all the values and then get final range. Does anyone know if this can be done in nifi.
Using the out of the box processors there is currently no easy way to do this but NiFi was designed to be easily extensible. What you'll need to do it either write a custom java processor or script to solve this.
For an intro tutorial on writing custom java processors for NiFi check out this link: http://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
As of version 0.5.0 there are the ExecuteScript and InvokeScripted processors. The creator of the processors has written a number of blogs on them here: http://funnifi.blogspot.com/

Filter by languages in 2nd and 4th level keys of a couchdb document

Given the following document in CouchDB....
{
"_id": "002bafd55b353692a7ab2968074310cc2cbff258",
"_rev": "1-bc853056ac61d817ae3c4ecb4f81322b",
"names": [
{ "locale": "en", "value": "Example" },
{ "locale": "de", "value": "Beispiel" },
{ "locale": "fr", "value": "Exemple" }
],
"details": [
{ "locale": "en", "value": "An Example is here" },
{ "locale": "de", "value": "Ein Beispiel ist heir" }
{ "locale": "en", "value": "Un exemple est ici" }
]
}
...how can I write a view that will allow me to return a partial document with
the undesired languages filtered out?
curl ..snip.. '_design/locale_filter/?locale=en,de,fr,it'
curl ..snip.. '_design/locale_filter/?locale=en,fr'
curl ..snip.. '_design/locale_filter/?locale=en'
Should return something looking like this:
{
"_id": "002bafd55b353692a7ab2968074310cc2cbff258",
"_rev": "1-bc853056ac61d817ae3c4ecb4f81322b",
"names": [
{ "locale": "en", "value": "Example" },
],
"details": [
{ "locale": "en", "value": "An Example is here" },
]
}
There's also a sub-case, where the documents have a further deeper structure,
which repeats the names and details structure, these would also be
filtered in an ideal world:
{
"_id": "002bafd55b353692a7ab2968074310cc2cbff258",
"_rev": "1-bc853056ac61d817ae3c4ecb4f81322b",
"names": [ ... snip ... ],
"details": [ ... snip ... ]
"deeper": {
"names": [
{ "locale": "en", "value": "Sub-Example" },
],
"details": [
{ "locale": "en", "value": "The Sub-Example is here" },
}
}
I also note that this might not be a view, but rather a show, from the
documentation couchdb says that a show is for transforming documents into any
format.
The final query from a beginner is whether there's some way to make it easier
to work on couchdb views and design docs, right now I'm experimenting with
erica which feels like overkill as I'm
pretty sure I don't want a couch app, I just want to easily maintain my views
in files on the disk, and sync them with the couch database whenever I've made
significant enough changes.
I was able to implement this using a show function, I implemented two show functions, one for convenience:
(doc, req) ->
all_locales = []
for name in doc.names
all_locales.push name.locale
toJSON(all_locales)
(I also implemented it on details, and remove duplicate locales in my real code)
This allows me to do the following:
GET /_design/dbname/_show/list_locales/c0db9ad..snip..
and returns ["en", "de", "fr"], for example - whatever locales the language happens to have.
I can then follow up with the function to retrieve the filtered document:
(doc, req) ->
locales = req.query.locales.split(",")
doc.names = doc.names.filter (name) ->
locales.indexOf(name.locale) > -1
doc.overviews = doc.details.filter (overview) ->
locales.indexOf(overview.locale) > -1
return toJSON(doc) + "\n"
The usage pattern for this is:
GET /_design/dbname/_show/restrict_locales/c0db9ad..snip..?locales=en,fr
GET /_design/dbname/_show/restrict_locales/c0db9ad..snip..?locales=fr
GET /_design/dbname/_show/restrict_locales/c0db9ad..snip..?locales=en,fr,de,it,hu,zh
It works quite remarkably well, and was much faster than I expected. I believe the show function results are aggressively cached by CouchDB.

Resources